VARIANT cast extends to ARRAY and MAP columns in ABAC column masks

Databricks now allows VARIANT-based masking functions to be applied to ARRAY and MAP columns, in addition to STRUCT. This simplifies governance by reducing the number of UDFs needed to mask complex data.

Use cases

Imagine you need to hide sensitive data in tables containing lists or dictionaries of information, and want to reuse the same security logic without writing separate functions for every data type.

• Mask ARRAY and MAP columns using a single VARIANT-based function.
• Reduce the number of user-defined functions (UDFs) and access policies to maintain.
• Rebuild complex data structures while preserving the original schema after masking.

How to use

Create a SQL function that accepts and returns VARIANT, using schema_of_variant() to identify the type and rebuild the value using array() or map().

CREATE OR REPLACE FUNCTION generic_mask(val VARIANT)
RETURNS VARIANT
RETURN CASE
  WHEN schema_of_variant(val) = 'ARRAY<STRING>' THEN
    to_variant_object(array('redacted'))
  WHEN schema_of_variant(val) = 'OBJECT<key1: STRING, key2: STRING>' THEN
    to_variant_object(map('key1', 'redacted', 'key2', 'redacted'))
  ELSE NULL::VARIANT
END;

What happens in this code:

1. The function receives the column automatically cast to VARIANT format.
2. The schema_of_variant(val) command checks the exact structure of each row at runtime.
3. The CASE statement applies the appropriate redaction logic for each detected schema.
4. The modified value is wrapped in to_variant_object() and returned, then cast back to the original column type by Databricks.

Limitations and requirements

• Requires Databricks Runtime 19 or above for ARRAY and MAP columns.
• Complex columns containing CHAR, VARCHAR, GEOMETRY, GEOGRAPHY, or TIME cannot be masked using VARIANT.
• MAP keys must be strictly of type STRING.
• This casting works exclusively within Unity Catalog column mask and row filter policies (ABAC), not in general SQL queries.