Skip to content
Lucas Lessa
Voltar para as notícias
SnowflakeGAImpacto: Alto

Data mirroring for Snowflake Postgres is now generally available

Replicate data continuously from Snowflake Postgres into Snowflake with sub-minute lag, removing third-party CDC tools and ETL pipelines, with native or Iceberg support.

O que mudou

Data mirroring for Snowflake Postgres is now generally available (GA). The feature continuously replicates tables from a Snowflake Postgres instance into a Snowflake database with minimal lag, eliminating the need for third-party CDC connectors or batch ETL pipelines.

Use cases

Imagine an OLTP application recording transactions in Snowflake Postgres where analytics teams need immediate access to this data in Snowflake OLAP without impacting the transactional workload.

  • Replicate individual tables or entire schemas without operating separate CDC infrastructure.
  • Automatically propagate DDL operations (such as column additions or renames) without breaking replication.
  • Query pending changes with approximately 30-second lag through the $live companion view.
  • Audit or consume the last 7 days of row-level changes (inserts, updates, deletes) via the $changes table.
  • Target either standard Snowflake tables or Snowflake-managed Apache Iceberg tables.

How to use

Data mirroring uses Snowflake serverless compute to apply changes, recording change logs and metadata into Iceberg via the pg_lake extension. To monitor the compute consumption of mirror apply tasks:

-- Inspect serverless task execution history for mirroring
SELECT *
FROM SNOWFLAKE.ACCOUNT_USAGE.SERVERLESS_TASK_HISTORY
WHERE TASK_NAME LIKE 'APPLY_MIRROR_%';

What happens in this code:

  1. Accesses the SERVERLESS_TASK_HISTORY view in SNOWFLAKE.ACCOUNT_USAGE.
  2. Filters tasks by the APPLY_MIRROR_ prefix assigned to the serverless task running the mirror apply procedure.
  3. Outputs execution metrics and credit consumption for that specific mirror.

To inspect storage usage incurred by the change logs and metalog on the Postgres side:

-- Query storage usage history for Snowflake Postgres change logs
SELECT *
FROM SNOWFLAKE.ACCOUNT_USAGE.POSTGRES_STORAGE_USAGE_HISTORY
ORDER BY START_TIME DESC;

What happens in this code:

  1. Queries the Postgres storage account usage view.
  2. Returns storage consumption incurred by $changes and metalog tables written via pg_lake.

Limitations and requirements

  • Maturity: Generally Available (GA).
  • Cloud support: Supported on AWS and Microsoft Azure across all regions where Snowflake Postgres is offered.
  • Boundaries: Mirrors cannot be set up across different accounts, cloud providers, or regions.
  • Membership changes: Adding or removing replicated objects requires alter_mirror with add_tables/remove_tables or add_schemas/remove_schemas; replacing the full object set in a single command is unsupported.

Using it in a pipeline

Data mirroring serves as the zero-ETL ingestion layer landing data into Snowflake Bronze/Raw tables. Downstream pipelines can query the $live view for sub-minute operational reports or read the $changes log table to drive incremental transformations toward Silver and Gold layers.

Ver nota oficial do Snowflake

Data mirroring for Snowflake Postgres (General availability)

Resumo e explicação gerados por IA a partir da release note oficial. Em caso de dúvida, a documentação oficial da plataforma prevalece.

Relacionadas