Snowpipe Streaming supports partitioned Iceberg tables

Snowpipe Streaming now allows ingesting data directly into partitioned Snowflake-managed Iceberg tables using the high-performance architecture with support for both v2 and v3 format versions.

Use cases

Imagine you need to stream continuous event data directly into partitioned open table formats without dealing with batch bottlenecks. With this update, you can:

• Ingest real-time data into both partitioned and non-partitioned Iceberg tables.
• Use either Elastic Channels or Named Channels.
• Work with Iceberg tables using Snowflake-managed storage or your own external volumes.

How to use

You can create a Snowflake-managed Iceberg table with internal storage and define the Iceberg version before starting ingestion via the SDK.

CREATE OR REPLACE ICEBERG TABLE my_iceberg_table (
    event_id NUMBER,
    event_type STRING,
    event_data VARIANT,
    event_timestamp TIMESTAMP_NTZ
)
    CATALOG = 'SNOWFLAKE'
    EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
    ICEBERG_VERSION = 3;

What happens in this code:

1. Sets up the creation of a new Iceberg table named my_iceberg_table.
2. Specifies the catalog as SNOWFLAKE and uses internal managed storage.
3. Configures the table to use version 3 of the Iceberg standard.

Limitations and requirements

• Generally available (GA) for Snowflake-managed tables.
• Length-constrained VARCHAR columns (e.g., VARCHAR(100)) are not supported for Iceberg tables; use STRING or unconstrained VARCHAR instead.
• Snowpipe Streaming Classic and externally managed Iceberg tables are not included in this release.
• Snowflake must act as the Iceberg catalog (external catalogs like AWS Glue or Hive Metastore are not supported for streaming).

Using it in a pipeline

This feature fits into the ingestion layer (Bronze) of lakehouse architectures, capturing streaming data from applications and writing it directly into partitioned Iceberg tables ready for downstream analytics.