Snowflake Cortex AI Function Evaluation for measuring quality

Snowflake Cortex AI Function Evaluation enables measuring output quality, cost, and token usage of AI calls and custom functions against labeled datasets. It uses the AI_FUNCTION_EVALUATION experiment type to establish consistent comparison baselines via SQL or the Cortex AI Function Studio.

Use cases

Imagine you have a sentiment classification model in production and want to test new prompt instructions safely:

• Establish a measurable quality baseline before moving an AI function to production.
• Compare different prompts, models, and implementations using the same dataset and metric.
• Evaluate different output types using rule-based metrics, llm_judge, or custom UDFs.
• Measure stability and variance in nondeterministic outputs by repeating runs.

How to use

Create the experiment defining the specification inside a YAML document and then execute it:

CREATE OR REPLACE EXPERIMENT my_db.my_schema.sentiment_eval
  TYPE = 'AI_FUNCTION_EVALUATION'
  FROM SPECIFICATION $$
query_text: "AI_COMPLETE('claude-sonnet-4-5',
  'Reply with exactly one word - POSITIVE, NEGATIVE, or NEUTRAL - '
  || 'for the sentiment of: ' || review)"
metrics:
  - name: exact_match
dataset:
  name: my_db.my_schema.reviews_labeled
  version: v1
  ground_truth: expected_sentiment
$$;

EXECUTE EXPERIMENT my_db.my_schema.sentiment_eval;

What happens in this code:

1. Creates an experiment object named sentiment_eval of type AI_FUNCTION_EVALUATION in the specified database and schema.
2. Defines the query under test using the AI function and corresponding model.
3. Sets the evaluation metric to exact_match against the ground truth column (expected_sentiment) of the dataset version.
4. Triggers asynchronous execution of the experiment using serverless compute.

Limitations and requirements

• Public Preview feature, available to all accounts.
• The dataset must be a Snowflake dataset (SNOWFLAKE.ML.DATASET) containing at most 1,000 rows (50–200 recommended).
• Exactly one metric is allowed per evaluation.
• The num_eval_runs parameter accepts integer values from 1 to 20.
• Requires database role SNOWFLAKE.CORTEX_USER (or SNOWFLAKE.AI_FUNCTIONS_USER) and usage privileges on the dataset and creation schema.