Databricks (Zerobus Ingest)

Datafly Signal streams events directly into a Databricks Delta Lake table via the Zerobus Ingest API — a serverless, real-time ingestion path that bypasses SQL warehouses. Pick this mode for the lowest-latency delivery and to keep SQL warehouses free for analytics workloads.

Prerequisites

Before configuring this integration you need:

• A Databricks workspace with Unity Catalog
• Zerobus Ingest enabled on the workspace (request via your Databricks account team if not enabled)
• A target Delta Lake table
• A service principal with OAuth client credentials and MODIFY permission on the table

CREATE CATALOG IF NOT EXISTS datafly;
CREATE SCHEMA IF NOT EXISTS datafly.events;
 
CREATE TABLE IF NOT EXISTS datafly.events.stream (
  event_name STRING,
  event_type STRING,
  message_id STRING,
  user_id STRING,
  anonymous_id STRING,
  event_timestamp TIMESTAMP,
  ip_address STRING,
  user_agent STRING,
  locale STRING,
  timezone STRING,
  page_url STRING,
  page_referrer STRING,
  source_id STRING
)
USING DELTA;

GRANT USE CATALOG ON CATALOG datafly TO `<service_principal_application_id>`;
GRANT USE SCHEMA ON SCHEMA datafly.events TO `<service_principal_application_id>`;
GRANT SELECT, MODIFY ON TABLE datafly.events.stream TO `<service_principal_application_id>`;

Configuration

Authentication uses OAuth 2.0 client credentials against {{workspace_url}}/oidc/v1/token with scope all-apis. Events are POSTed in batches to /api/2.0/streaming-ingestion/ingest.

Signal Setup

Quick Setup

1. Navigate to Integrations in the sidebar.
2. Open the Integration Library tab.
3. Find Databricks (Zerobus Ingest) or filter by Warehouse.
4. Click Install, select a variant if available, and fill in the required fields.
5. Click Install Integration to create the integration with the default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/databricks_zerobus/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Databricks (Zerobus Ingest)",
    "variant": "default",
    "config": {
      "workspace_url": "https://abc-12345.cloud.databricks.com",
      "client_id": "00000000-0000-0000-0000-000000000000",
      "client_secret": "YOUR_CLIENT_SECRET",
      "catalog": "datafly",
      "schema": "events",
      "table": "stream"
    },
    "delivery_mode": "server_side"
  }'

Schema

The default blueprint maps Signal’s canonical event envelope to a column-per-field shape suited to Delta Lake. Every event becomes one row with the columns below:

Per-event types add their own columns (e.g. page adds page_title; Products Searched adds search_query). You can extend or override mappings in your pipeline blueprint.

Consent

Databricks is a first-party destination under your control. The default blueprint forwards all events. Apply consent filtering in pipeline transforms, or via downstream views over the context columns if your governance requires it.

Testing

1. Enable the integration in Signal and trigger a test event on your website.
2. In the Databricks SQL editor, query the target table:

SELECT * FROM datafly.events.stream
ORDER BY event_timestamp DESC
LIMIT 10;

3. Verify rows are appearing — Zerobus typically commits within a few seconds.
4. In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

See also

• Databricks (SQL Warehouse)
• Data Warehouses overview
• Snowflake
• Apache Kafka