Apache Arrow Flight Support in Zerobus Ingest

Apache Arrow Flight support in Zerobus Ingest is now generally available. It lets you send Apache Arrow columnar record batches directly over gRPC, reducing CPU serialization overhead for analytics workloads.### Use casesImagine your application builds in-memory DataFrames or tables using Pyarrow, Polars, or DataFusion and you need to ingest them rapidly into a Delta table. You can:- Send data in columnar batches (RecordBatch or Table) instead of row by row.- Lower CPU overhead on wide and numeric schemas.- Optionally compress IPC payloads in transit using LZ4 or ZSTD.### How to useInstall the package with Arrow support and open an ingestion stream using the Python or Rust SDK.Bashpip install "databricks-zerobus-ingest-sdk[arrow]"Python```import pyarrow as pa from zerobus.sdk.sync import ZerobusSdk

SERVER_ENDPOINT = "https://1234567890123456.zerobus.us-west-2.cloud.databricks.com" DATABRICKS_WORKSPACE_URL = "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com" TABLE_NAME = "main.default.air_quality" CLIENT_ID = "your-client-id" CLIENT_SECRET = "your-client-secret"

schema = pa.schema( [ ("device_name", pa.large_utf8()), ("temp", pa.int32()), ("humidity", pa.int64()), ] )

sdk = ZerobusSdk(SERVER_ENDPOINT, DATABRICKS_WORKSPACE_URL) stream = sdk.create_arrow_stream(TABLE_NAME, schema, CLIENT_ID, CLIENT_SECRET)

try: batch = pa.record_batch( { "device_name": ["sensor-1"], "temp": [22], "humidity": [60], }, schema=schema, ) stream.ingest_batch(batch) stream.flush() finally: stream.close()```What happens in this code:

1. Defines the Arrow schema matching the target Delta table.
2. Initializes the Zerobus SDK pointing to the endpoint and workspace.
3. Creates an Arrow stream and ingests a record batch.
4. Flushes the stream to ensure durability and closes it.### Limitations and requirementsGenerally available (GA). Each individual row inside a RecordBatch must fit within the 10 MB limit, while overall batches can be larger. Arrow fields must match the target Delta table schema's relative order, nullability, and types.