Build document pipelines with Genie Code

Genie Code now automates the creation of unstructured document processing pipelines by orchestrating parsing, classification, extraction, and validation steps. It uses AI functions to turn PDFs, contracts, and invoices into structured data saved in Unity Catalog. ### Use cases Imagine you need to process thousands of contracts or PDF invoices and save them into structured tables. With Genie Code, you can: - Discover and track source files in Unity Catalog volumes. - Parse and extract structured content, text, tables, and layouts from documents. - Classify documents into custom categories. - Evaluate extraction quality using labeled data and iterate to improve results. ### How to use To create a pipeline or interact with your data, open Genie Code on the Jobs & Pipelines page or the SQL editor and state your goal using natural language prompts. Example prompts: ```sql -- Build an end-to-end pipeline from a volume "Contracts land in @my_catalog.raw.contracts. Build a daily job that classifies each into affiliate, marketing, consulting, hosting, and escrow; extracts the fields that matter for each type of contract; and writes gold tables."

-- Evaluate and improve extraction quality "How do I know this is good enough to run on millions of rows? Evaluate the extraction against my labeled table @my_catalog.default.eval_table and tell me which fields are weakest." ``` What happens in this code: 1. Genie Code interprets the natural language prompt to understand your goal and specified data sources. 2. It automatically creates or adjusts the Lakeflow job structure and applies the required AI functions. 3. The system evaluates extraction quality or runs the workflow as requested, letting you review before scaling up. ### Limitations and requirements - Meets the requirements for Genie Code agentic capabilities and AI Functions (ai_parse_document, ai_extract, and ai_classify), which run on serverless compute or a SQL warehouse in a supported region. - Requires Unity Catalog access to a volume for source documents and a schema for results.