Monitor Snowpipe Streaming using event tables

Monitor Snowpipe Streaming pipelines directly within Snowflake to check if data is arriving, identify processing delays, and investigate ingestion failures using event tables. Snowflake provides a default event table to collect this telemetry, recording metrics such as latency, throughput, row counts, and channel errors.

Use cases

Imagine you need to ensure streaming data is flowing into your tables without delays and want to set up alerts for write failures. With this feature you can:

• Track processed and rejected row counts in real time.
• Measure server-side latency between data receipt and commit.
• Investigate row errors and channel-specific failures.
• Build dashboards and configure alerts using SQL queries.

How to use

First, verify that event collection is enabled by setting the LOG_EVENT_LEVEL parameter to INFO on the schema containing your target tables:

ALTER SCHEMA my_database.my_schema SET LOG_EVENT_LEVEL = INFO;

What happens in this code:

1. Updates the event logging level of the schema to INFO.
2. Enables the recording of all Snowpipe Streaming event types in the configured event table.

Next, query recent commit events to check ingestion throughput and progress:

SELECT
  timestamp,
  resource_attributes:"snow.table.name"::STRING AS table_name,
  value:"channel_name"::STRING AS channel_name,
  value:"row_count"::NUMBER AS rows_ingested,
  value:"error_count"::NUMBER AS error_count
FROM SNOWFLAKE.TELEMETRY.EVENTS_VIEW
WHERE scope:"name"::STRING = 'snow.snowpipe.streaming'
  AND record_type = 'EVENT'
  AND record:"name"::STRING = 'commit'
  AND resource_attributes:"snow.database.name"::STRING = 'my_database'
  AND resource_attributes:"snow.schema.name"::STRING = 'my_schema'
  AND resource_attributes:"snow.table.name"::STRING = 'my_table'
  AND timestamp > DATEADD('minute', -15, CURRENT_TIMESTAMP())
ORDER BY timestamp DESC;

What happens in this code:

1. Filters records by the Snowpipe Streaming specific scope and the commit event type.
2. Restricts the search to the specified database, schema, and target table.
3. Returns data from the last 15 minutes ordered by descending timestamp.

Limitations and requirements

• Requires a Snowpipe Streaming pipeline using the high-performance architecture.
• Requires an active event table and a role with query permissions (such as the SNOWFLAKE.EVENTS_VIEWER role for the default table).
• The LOG_EVENT_LEVEL parameter must be set to INFO or ERROR on the target schema.