Google BigQuery

Datafly Signal writes events directly into a BigQuery table using the BigQuery Storage Write API (batched appends on the default stream; the legacy streaming insert API is available as a fallback), so you can analyse first-party behaviour at warehouse scale.

Why route Google BigQuery through Datafly Signal

• The data lands in infrastructure you own. Signal writes to your Google BigQuery directly from your own deployment. Datafly operates no intermediary and never holds a copy of your event data.
• Personal data is hashed on the way in. Email, phone (shaped to E.164 first), first and last name, full name and postcode are normalised (trim and lowercase) and hashed with SHA-256 inside your own deployment, so the table holds email_sha256, phone_sha256 and the other digests rather than raw personal data. A raw column is an opt-in edit to the installed blueprint.
• Client IP and user agent are recorded as event context. Both come from the original visitor request rather than from your server, so geography and device analysis done in the warehouse reflects the actual visitor.
• Presets for Retail, Insurance, Travel, Media & Streaming, Publishing, B2B / SaaS, Automotive and General. Each vertical preset names one field per property in the matching event spec, with canonical names (transaction_id, value, items), legacy aliases (order_id, revenue, products) read automatically, and numeric properties typed as numbers. Every other event still lands with its properties.
• One consent decision governs every destination. Google BigQuery is gated by the same consent state as the rest of your stack, so a withdrawal applies everywhere at once.

Prerequisites

Before configuring Google BigQuery in Signal, you need a GCP project with BigQuery enabled, a dataset and table, and a service account with the BigQuery Data Editor role.

CREATE TABLE `your-project.datafly_events.events` (
  message_id      STRING,
  event_type      STRING,
  event_name      STRING,
  anonymous_id    STRING,
  user_id         STRING,
  timestamp       TIMESTAMP,
  received_at     TIMESTAMP,
  ip              STRING,
  user_agent      STRING,
  page_url        STRING,
  page_path       STRING,
  page_referrer   STRING,
  org_id          STRING,
  pipeline_id     STRING,
  properties_json JSON
)
PARTITION BY DATE(timestamp)
CLUSTER BY event_type, event_name;

Configuration

Signal Setup

Quick Setup

1. Navigate to Integrations in the sidebar.
2. Open the Integration Library tab.
3. Find Google BigQuery or filter by Warehouse.
4. Click Install, select a variant if available, and fill in the required fields.
5. Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/google_bigquery/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Google BigQuery",
    "variant": "default",
    "config": {
      "project_id": "datafly-analytics",
      "dataset_id": "datafly_events",
      "table_id": "events",
      "service_account_json": "{\"type\": \"service_account\", \"project_id\": \"datafly-analytics\", ...}",
      "location": "US"
    },
    "delivery_mode": "server_side"
  }'

Schema

Signal writes one row per event: a fixed envelope plus properties_json. Define the table schema to match. With the Storage Write API (the default) a column the table does not have is skipped, and a column the row does not fill is NULL.

Partition by DATE(timestamp) and cluster by event_type, event_name to keep scan costs predictable.

What properties_json contains

properties_json holds only what the integration’s blueprint maps for the event, and nothing else:

• A property or identify trait the blueprint does not map is never written. That includes the traits a visitor gave in an earlier identify() call, which Signal attaches to later events so a blueprint can read them.
• Personal data is written as the blueprint writes it. The catalogue blueprints hash email and phone with normalised_sha256 (trimmed, lower-cased, SHA-256), so email_sha256 holds a digest. A field you map without a hashing transform is stored raw: that is a choice you make in the blueprint.
• Fields mapped in the blueprint’s global section sit under an @root object, for example JSON_VALUE(properties_json, '$."@root".email_sha256'). Fields mapped for a single event sit at the top level.

To store a trait, map it in the blueprint (hashed unless you have a reason to keep it raw).

A BigQuery integration that already existed before this rule keeps writing every property it wrote before, including unmapped properties and pass-through events, with personal data removed (the attached identify traits, unmapped identify traits, context.traits and raw email, phone, name, address and similar fields the blueprint does not map). Set strict_output_contract to true in its connection settings to switch it to only what the blueprint maps. See Integrations installed before this change.

Where mapped fields land

The table layout is fixed: the envelope columns above plus properties_json. The blueprint shapes properties_json: event fields from the preset (for example transaction_id, value, insurance_type, policy_term_months) sit at its top level, and the preset’s envelope block (identity digests, UTM, consent flags) sits under @root:

SELECT
  timestamp,
  JSON_VALUE(properties_json, '$.transaction_id') AS policy_number,
  CAST(JSON_VALUE(properties_json, '$.value') AS NUMERIC) AS premium,
  JSON_VALUE(properties_json, '$."@root".email_sha256') AS email_sha256
FROM `your-project.datafly_events.events`
WHERE event_name = 'purchase';

Signal does not write mapped fields to their own typed columns yet. Create views over properties_json for the columns you query often.

Stored event context

Alongside the fields your blueprint maps (where email, phone, names and address are stored as normalised SHA-256 by default), each row carries the event’s collected context exactly as Signal received it. The blueprint does not filter or hash the context: this is deliberate (decision 2026-09-23), so the context is complete for attribution, fraud and debugging queries, and it is listed here so you can decide what to keep.

They are written to the ip_address, user_agent, page_url, page_path and page_referrer columns and, in full, to the context_json column, beside anonymous_id and user_id. Click and collected IDs outside the context are stored only where the blueprint maps them.

The IP address, device and advertising IDs, and cookie and click identifiers are personal data under UK and EU GDPR. To keep any of them out of storage, strip, mask or hash the field for every integration with the organisation data layer, apply URL redaction to page URLs, or remove it in the installed blueprint; and set the storage’s retention to match your privacy notice. See Personal data by integration.

Consent

BigQuery is a first-party destination you own. The default blueprint forwards all events; if you need consent filtering, apply it inside the pipeline transform or via downstream views that gate on the preset’s consent flags, JSON_VALUE(properties_json, '$."@root".consent_analytics') and consent_marketing (the BigQuery row has no context column).

Testing

1. Enable the integration in Signal and trigger a test event on your website.
2. Open the BigQuery console and navigate to your dataset and table.
3. Click Preview to see recently inserted rows, or run a query:

SELECT * FROM `your-project.datafly_events.events`
ORDER BY timestamp DESC
LIMIT 10;

4. Verify that event rows are appearing with correct data.
5. In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

See also

• Data Warehouses overview
• Snowflake
• Amazon Redshift
• Google Cloud Storage