Databricks (SQL Warehouse)

Datafly Signal writes events into a Databricks Delta Lake table using the SQL Statement Execution API, executed against a SQL warehouse. Pick this mode when you already have a running SQL warehouse and want batch-friendly inserts into Unity Catalog.

Why route Databricks (SQL Warehouse) through Datafly Signal

• The data lands in infrastructure you own. Signal writes to your Databricks (SQL Warehouse) directly from your own deployment. Datafly operates no intermediary and never holds a copy of your event data.
• Email is hashed on the way in. Signal normalises (trim and lowercase) and applies SHA-256 inside your own deployment, so the table holds hashed identifiers rather than raw personal data.
• Client IP and user agent are recorded as event context. Both come from the original visitor request rather than from your server, so geography and device analysis done in the warehouse reflects the actual visitor.
• Presets for Retail, Media & Publishing, B2B / SaaS and General. The mapping starts from a working configuration for your vertical instead of a blank field map.
• One consent decision governs every destination. Databricks (SQL Warehouse) is gated by the same consent state as the rest of your stack, so a withdrawal applies everywhere at once.

Prerequisites

Before configuring this integration, you need a Databricks workspace with Unity Catalog, a SQL warehouse, a target table, and a personal access token (or service principal token).

CREATE CATALOG IF NOT EXISTS datafly;
CREATE SCHEMA IF NOT EXISTS datafly.events;
 
CREATE TABLE IF NOT EXISTS datafly.events.raw (
  message_id      STRING,
  event_type      STRING,
  event_name      STRING,
  anonymous_id    STRING,
  user_id         STRING,
  ip_address      STRING,
  user_agent      STRING,
  page_url        STRING,
  page_path       STRING,
  page_referrer   STRING,
  org_id          STRING,
  pipeline_id     STRING,
  event_timestamp TIMESTAMP,
  received_at     TIMESTAMP,
  properties_json STRING,
  context_json    STRING
);

Configuration

Signal Setup

Quick Setup

1. Navigate to Integrations in the sidebar.
2. Open the Integration Library tab.
3. Find Databricks (SQL Warehouse) or filter by Warehouse.
4. Click Install, select a variant if available, and fill in the required fields.
5. Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/databricks_sql/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Databricks (SQL Warehouse)",
    "variant": "default",
    "config": {
      "workspace_url": "https://abc-12345.cloud.databricks.com",
      "access_token": "dapi1234567890abcdef",
      "warehouse_id": "abc123def456",
      "catalog": "datafly",
      "schema": "events",
      "table": "raw"
    },
    "delivery_mode": "server_side"
  }'

Schema

Signal writes the canonical event envelope. Columns whose value is empty are omitted from the INSERT and take the table default (NULL):

Query the JSON columns with : notation or parse_json() (e.g. properties_json:total::DOUBLE, context_json:consent). For native JSON typing, declare properties_json / context_json as VARIANT (Databricks Runtime 15.3+) — Signal sends a JSON string either way.

Stored event context

Alongside the fields your blueprint maps (where email, phone, names and address are stored as normalised SHA-256 by default), each row carries the event’s collected context exactly as Signal received it. The blueprint does not filter or hash the context: this is deliberate (decision 2026-09-23), so the context is complete for attribution, fraud and debugging queries, and it is listed here so you can decide what to keep.

They are written to the ip_address, user_agent, page_url, page_path and page_referrer columns and, in full, to the context_json column, beside anonymous_id and user_id. Click and collected IDs outside the context are stored only where the blueprint maps them.

The IP address, device and advertising IDs, and cookie and click identifiers are personal data under UK and EU GDPR. To keep any of them out of storage, strip, mask or hash the field for every integration with the organisation data layer, apply URL redaction to page URLs, or remove it in the installed blueprint; and set the storage’s retention to match your privacy notice. See Personal data by integration.

Consent

Databricks is a first-party destination under your control. The default blueprint forwards all events. Apply consent filtering in pipeline transforms, or use downstream views that gate on context.consent if your governance requires it.

Testing

1. Enable the integration in Signal and trigger a test event on your website.
2. In the Databricks SQL editor, query the target table:

SELECT * FROM datafly.events.raw ORDER BY event_timestamp DESC LIMIT 10;

3. Verify that event rows are appearing with correct data.
4. In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

See also

• Databricks (Zerobus Ingest)
• Data Warehouses overview
• Snowflake
• Google BigQuery