AI Search now supports struct and map columns and filter-only queries

Databricks AI Search has expanded its indexing and retrieval capabilities, allowing you to work with complex data structures and perform purely relational queries without vectors or keywords.

Use cases

Imagine you need to search technical documents containing nested metadata and mapped attributes, or retrieve records by applying conditional business rules only. With this feature, you can:

• Index and filter struct and map columns.
• Run queries that return only rows matching a filter without vector or keyword retrieval.

How to use

You can query the indexes using the AI Search Python SDK.

from databricks.ai_search.client import AISearchClient

client = AISearchClient()
index = client.get_index(index_name="meu_catalogo.meu_schema.meu_indice")

results = index.similarity_search(
    query_vector=[0.2, 0.33, 0.19, 0.52],
    columns=["id", "texto"],
    filters="status = 'ativo' AND categoria = 'tecnologia'",
    num_results=10
)

What happens in this code:

1. The AI Search client connects to the specified index in Unity Catalog.
2. The search function executes the query using the provided vector.
3. The filters parameter applies a SQL-like conditional clause to restrict the results.
4. Up to 10 matching records are returned.

Limitations and requirements

• To query the index, the user must have USE CATALOG, USE SCHEMA, and SELECT privileges on the object if they are not the owner.
• Overfetching of results may occur on storage-optimized endpoints.