Scalar and Batch Unity Catalog Python UDFs are now generally available

Scalar and Batch Python UDFs in Unity Catalog are now generally available across all supported compute types, introducing named handlers, dependency installation, and secret access.

Use cases

Imagine you need to apply custom Python logic directly within SQL queries or data pipelines while leveraging external libraries, secured secrets, and structured handlers. With this release, you can:

• Run scalar and batch UDFs without the previous five-call limit per query.
• Use named handlers to structure entry points cleanly.
• Access secure secrets and external custom dependencies.

How to use

You can register and use a Python UDF with a named handler using SQL.

CREATE OR REPLACE FUNCTION my_catalog.my_schema.greet(name STRING)
RETURNS STRING
LANGUAGE PYTHON
HANDLER 'greet_handler'
ENVIRONMENT (
  environment_version = '6'
)
AS $$
greeting_prefix = "Hello"

def greet_handler(name):
    return f"{greeting_prefix}, {name}!"
$$;

What happens in this code:

1. The command creates or replaces the UDF in the specified catalog and schema.
2. The HANDLER clause points to greet_handler as the execution entry point.
3. The ENVIRONMENT clause sets the environment version to '6'.
4. The function body initializes variables once and processes the input parameters.

Limitations and requirements

• Available on all supported compute types, with specific runtime and environment version requirements depending on the feature.
• Advanced features like secrets and named handlers require explicitly setting the environment_version to '6' or higher.