Databricks (SQL Warehouse)

Datafly Signal writes events into a Databricks Delta Lake table using the SQL Statement Execution API, executed against a SQL warehouse. Pick this mode when you already have a running SQL warehouse and want batch-friendly inserts into Unity Catalog.

For lower-latency real-time ingestion, see Databricks (Zerobus Ingest), which bypasses SQL warehouses and writes directly via the Zerobus streaming API.

Prerequisites

Before configuring this integration, you need a Databricks workspace with Unity Catalog, a SQL warehouse, a target table, and a personal access token (or service principal token).

Create a Databricks Workspace

Sign up at databricks.com or provision a workspace through your cloud provider (AWS, Azure, or GCP).
Complete the workspace setup wizard.
Note the Workspace URL (e.g. https://abc-12345.cloud.databricks.com).

Create a SQL Warehouse

In the Databricks workspace, go to SQL Warehouses in the left sidebar.
Click Create SQL warehouse.
Enter a name (e.g. datafly-warehouse).
Choose the Size (2X-Small is sufficient for low to moderate event volume).
Set Auto stop to manage cost.
Click Create.
Note the Warehouse ID from the warehouse details page (visible in the URL and under Connection details > HTTP path — the last segment).

Create a Catalog, Schema, and Table

In the SQL editor or a notebook, run:

CREATE CATALOG IF NOT EXISTS datafly;
CREATE SCHEMA IF NOT EXISTS datafly.events;
 
CREATE TABLE IF NOT EXISTS datafly.events.raw (
  message_id      STRING,
  event_type      STRING,
  event_name      STRING,
  anonymous_id    STRING,
  user_id         STRING,
  ip_address      STRING,
  user_agent      STRING,
  page_url        STRING,
  page_path       STRING,
  page_referrer   STRING,
  org_id          STRING,
  pipeline_id     STRING,
  event_timestamp TIMESTAMP,
  received_at     TIMESTAMP,
  properties_json STRING,
  context_json    STRING
);

This is the canonical Datafly warehouse schema — the same column set Signal writes to BigQuery, Snowflake, and Redshift, so a single DDL is portable across destinations. Every blueprint-mapped property is carried inside properties_json (queryable with Databricks JSON path), so you never have to alter the table when a blueprint changes. Databricks stores the table in Delta Lake format, providing ACID transactions, time travel, and schema evolution.

⚠️

Each event is delivered as one parameterised INSERT executed on the SQL warehouse. This is ideal for low-to-moderate volume and pilots, but high-volume streams generate many small Delta files and sustained warehouse cost — for those, use Databricks (Zerobus Ingest) or land events in object storage and ingest with Auto Loader.

Generate a Personal Access Token

Click your profile icon (top right) > Settings.
Go to Developer > Access tokens.
Click Generate new token.
Enter a Comment (e.g. Datafly Signal) and a Lifetime.
Click Generate.
Copy the token immediately — it is only shown once.

⚠️

For production deployments, prefer a service principal token over a personal access token. Grant the service principal USE CATALOG, USE SCHEMA, and MODIFY on the target table via Unity Catalog.

Configuration

Field	Type	Required	Description
`workspace_url`	string	Yes	Workspace URL (e.g. `https://abc-12345.cloud.databricks.com`).
`access_token`	secret	Yes	Personal access token or service principal token.
`warehouse_id`	string	Yes	The SQL warehouse ID to execute statements against.
`catalog`	string	Yes	The Unity Catalog name.
`schema`	string	Yes	The schema within the catalog.
`table`	string	Yes	The target table name.

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Databricks (SQL Warehouse) or filter by Warehouse.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/databricks_sql/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Databricks (SQL Warehouse)",
    "variant": "default",
    "config": {
      "workspace_url": "https://abc-12345.cloud.databricks.com",
      "access_token": "dapi1234567890abcdef",
      "warehouse_id": "abc123def456",
      "catalog": "datafly",
      "schema": "events",
      "table": "raw"
    },
    "delivery_mode": "server_side"
  }'

Schema

Signal writes the canonical event envelope. Columns whose value is empty are omitted from the INSERT and take the table default (NULL):

Column	Databricks type	Notes
`message_id`	`STRING`	Unique per delivered event.
`event_type`	`STRING`	`track`, `page`, `identify`, etc.
`event_name`	`STRING`	Event name (snake_case).
`anonymous_id`	`STRING`	First-party visitor identifier.
`user_id`	`STRING`	Logged-in user identifier (nullable).
`ip_address`	`STRING`	Client IP (nullable).
`user_agent`	`STRING`	Client user agent (nullable).
`page_url`	`STRING`	Page URL (nullable).
`page_path`	`STRING`	Page path (nullable).
`page_referrer`	`STRING`	Page referrer (nullable).
`org_id`	`STRING`	Datafly org identifier.
`pipeline_id`	`STRING`	Signal pipeline identifier.
`event_timestamp`	`TIMESTAMP`	Client event time.
`received_at`	`TIMESTAMP`	Time Signal received the event.
`properties_json`	`STRING`	JSON document — all event properties (including blueprint-mapped fields).
`context_json`	`STRING`	JSON document — page, device, consent metadata.

Query the JSON columns with : notation or parse_json() (e.g. properties_json:total::DOUBLE, context_json:consent). For native JSON typing, declare properties_json / context_json as VARIANT (Databricks Runtime 15.3+) — Signal sends a JSON string either way.

Databricks is a first-party destination under your control. The default blueprint forwards all events. Apply consent filtering in pipeline transforms, or use downstream views that gate on context.consent if your governance requires it.

Testing

Enable the integration in Signal and trigger a test event on your website.
In the Databricks SQL editor, query the target table:

SELECT * FROM datafly.events.raw ORDER BY event_timestamp DESC LIMIT 10;

Verify that event rows are appearing with correct data.
In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

Problem	Solution
Events not appearing in the table	Verify the workspace URL, catalog, schema, table, and warehouse ID.
`Unauthorized` (401)	The access token is invalid or expired. Generate a new token.
`Forbidden` (403)	The user lacks `MODIFY` on the table. Grant Unity Catalog permissions: `USE CATALOG`, `USE SCHEMA`, `MODIFY`.
Warehouse not running	The SQL warehouse auto-stopped. It will auto-start on the next request — the first event may have higher latency.
`TABLE_OR_VIEW_NOT_FOUND`	Verify the fully-qualified path `catalog.schema.table`.
`UNRESOLVED_COLUMN` / column mismatch	The target table does not match the canonical schema above. Recreate it with the exact column names from the Schema section.
Connection timeout	Ensure Signal can reach the workspace URL on port 443. Check network policies and IP allowlists.
Token expired	Rotate the personal access token or refresh the service principal token.

Visit Databricks SQL Statement Execution API docs for endpoint reference and rate-limit guidance.