IntegrationsStreamingGoogle Pub/Sub

Google Pub/Sub

Datafly Signal publishes first-party events to Google Cloud Pub/Sub topics for real-time messaging, event-driven architectures, and stream processing with downstream subscribers.

Why route Google Pub/Sub through Datafly Signal

  • The data lands in infrastructure you own. Signal writes to your Google Pub/Sub directly from your own deployment. Datafly operates no intermediary and never holds a copy of your event data.
  • Client IP and user agent are recorded as event context. Both come from the original visitor request rather than from your server, so geography and device analysis done in the warehouse reflects the actual visitor.
  • Presets for eight verticals. General (the default: page, identify and the key cross-vertical events), Retail, Insurance, Travel, Media & Streaming, Publishing, B2B / SaaS and Automotive, each writing one named field per documented event property. Email, phone and names are written only as SHA-256 digests unless you deliberately change the installed blueprint.
  • One consent decision governs every destination. Google Pub/Sub is gated by the same consent state as the rest of your stack, so a withdrawal applies everywhere at once.

Prerequisites

Before configuring Google Pub/Sub in Signal, you need a GCP project with a Pub/Sub topic and a service account with the Pub/Sub Publisher role.

Create a GCP Account and Project

  1. Sign up at cloud.google.com if you don’t already have an account.
  2. Create a new project or select an existing one in the GCP Console.
  3. Note the Project ID.

Enable the Pub/Sub API

  1. Go to APIs & Services > Library.
  2. Search for Cloud Pub/Sub API.
  3. Click Enable.

Create a Pub/Sub Topic

  1. Go to the Pub/Sub console.
  2. Click Create topic.
  3. Enter a Topic ID (e.g. datafly-events).
  4. Optionally add a default subscription (creates a pull subscription automatically).
  5. Click Create.

If you want consumers to receive events, you need at least one subscription attached to the topic. Subscriptions can be created later and can be pull-based, push-based, or BigQuery subscriptions.

Create a Subscription (Optional)

  1. In the Pub/Sub console, click on your topic.
  2. Click Create subscription.
  3. Enter a Subscription ID (e.g. datafly-events-sub).
  4. Choose the delivery type:
    • Pull — consumers poll for messages.
    • Push — Pub/Sub pushes messages to an HTTPS endpoint.
    • BigQuery — writes directly to a BigQuery table.
  5. Set Acknowledgement deadline and Message retention as needed.
  6. Click Create.

Create a Service Account

  1. Go to IAM & Admin > Service Accounts > Create Service Account.
  2. Enter a name (e.g. datafly-signal-pubsub).
  3. Grant the Pub/Sub Publisher role (roles/pubsub.publisher).
  4. Click Done.

Generate a Service Account Key

  1. Click on the service account.
  2. Go to Keys > Add Key > Create new key > JSON.
  3. The key file will download. Store it securely.
⚠️

Store the JSON key file securely. Do not commit it to version control.

Configuration

FieldTypeRequiredDescription
project_idstringYesThe Google Cloud project that holds the topic.
topic_idstringYesThe topic to publish to. It must already exist.
service_account_jsonsecretNoJSON key of a service account with roles/pubsub.publisher on the topic. Leave empty to use Application Default Credentials (for example GKE Workload Identity).

Signal Setup

Quick Setup

  1. Navigate to Integrations in the sidebar.
  2. Open the Integration Library tab.
  3. Find Google Pub/Sub or filter by Streaming.
  4. Click Install, select a variant if available, and fill in the required fields.
  5. Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/google_pubsub/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Google Pub/Sub",
    "variant": "default",
    "config": {
      "project_id": "datafly-analytics",
      "topic_id": "datafly-events",
      "service_account_json": "{\"type\": \"service_account\", ...}"
    },
    "delivery_mode": "server_side"
  }'

What Signal writes

Each event is one Pub/Sub message whose data is the Signal event as JSON:

FieldContents
event, typeEvent name (purchase, page, …) and type (track, page, identify, …).
anonymous_id, user_id, device_idVisitor identifiers (user_id and device_id only when set).
message_id, event_idSignal’s message ID (use it to deduplicate) and the client event ID.
timestamp, sent_at, received_atRFC 3339 UTC times: event, client send, Signal receipt.
org_id, pipeline_id, vendor_typeWhere the event came from and which destination type wrote it.
contextThe collected context as sent: ip, user_agent, locale, page, campaign, consent, geo, device, library, and any partner attributes. Written in full; the blueprint does not filter it (#1414).
click_ids, collected_idsAd click IDs and ID-provider values, when present.
propertiesOnly the fields the blueprint maps (below).
  • Attributes: datafly_message_id, datafly_event_type, datafly_event_name, so subscriptions can filter without parsing the body.
  • Ordering key: user_id, else anonymous_id, published with message ordering on, so one visitor’s events stay in order on a subscription created with message ordering enabled (other subscriptions ignore the key). After a failed publish the key is resumed, so the retry and the visitor’s next event can publish. Each key is limited to 1 MBps. The event debugger’s test send publishes the same message live delivery does.
  • Size: a message can be up to 10 MB.

Only what the blueprint maps is written. The properties object holds the fields your installed blueprint maps and nothing else (the row contract): the global block under @root (copies of the envelope fields, UTM parameters, the analytics and marketing consent flags, and email_sha256, phone_sha256, first_name_sha256, last_name_sha256, name_sha256), and the event section’s fields at the top level (transaction_id, value, currency, items, …). Identify traits and client properties the blueprint does not map are never written. An event with no section in the blueprint still arrives, with the @root block only: add a section for any other event whose properties you need.

A purchase from an identified visitor, as written by the default (General) preset. The raw email, phone and name in the visitor’s identify traits are not in it; only their digests are. Empty context objects are trimmed here for brevity.

{
  "anonymous_id": "anon-s1-7f3a",
  "user_id": "crm-4711",
  "event": "purchase",
  "type": "track",
  "properties": {
    "@root": {
      "anonymous_id": "anon-s1-7f3a",
      "email_sha256": "86e0b9e56c17cc4d12387e1949b85053fbe73bc3ce5a1188713a9d300cc6133d",
      "event_id": "evt-s1-2",
      "event_name": "purchase",
      "event_timestamp": "2026-09-23T09:30:02.000Z",
      "event_type": "track",
      "first_name_sha256": "81f8f6dde88365f3928796ec7aa53f72820b06db8664f5fe76a7eb13e24546a2",
      "ip_address": "203.0.113.9",
      "last_name_sha256": "4f887a0887ca68593c8acd4e83e0b6bc5f42cd5aff7d3a159041c84230ed1cf6",
      "locale": "en-GB",
      "message_id": "msg-s1-google_pubsub-2",
      "name_sha256": "ed37d99b1445238af3386f81a77a2caf3fffc0cd610be39b4f3ba53943dc66bf",
      "page_url": "https://shop.example/checkout",
      "phone_sha256": "a8acc3a90a7b4e4dc65e93db9240ed26523050ef754d63b75b5161de76781436",
      "user_agent": "Mozilla/5.0 (Macintosh)",
      "user_id": "crm-4711",
      "utm_medium": "cpc",
      "utm_source": "google"
    },
    "coupon": "AUTUMN10",
    "currency": "GBP",
    "items": [{ "item_id": "SKU-1", "item_name": "Trail shoe", "price": 80, "quantity": 1 }],
    "shipping": 4.5,
    "tax": 14.08,
    "transaction_id": "SIG-ORDER-1001",
    "value": 84.5
  },
  "context": {
    "ip": "203.0.113.9",
    "user_agent": "Mozilla/5.0 (Macintosh)",
    "locale": "en-GB",
    "page": { "url": "https://shop.example/checkout", "title": "Checkout", "path": "/checkout" },
    "campaign": { "source": "google", "medium": "cpc" }
  },
  "timestamp": "2026-09-23T09:30:02Z",
  "sent_at": "2026-09-23T09:30:02Z",
  "message_id": "msg-s1-google_pubsub-2",
  "org_id": "org-s1",
  "pipeline_id": "pl-s1",
  "received_at": "2026-09-23T09:30:02Z",
  "event_id": "evt-s1-2",
  "vendor_type": "google_pubsub"
}

The message layout on this page was produced by Signal’s Pub/Sub writer against the Pub/Sub emulator (the message shown was published by the event debugger’s path) on 2026-09-23 (batch S1 review), not written from memory.

Stored event context

Alongside the fields your blueprint maps (where email, phone, names and address are stored as normalised SHA-256 by default), each message carries the event’s collected context exactly as Signal received it. The blueprint does not filter or hash the context: this is deliberate (decision 2026-09-23), so the context is complete for attribution, fraud and debugging queries, and it is listed here so you can decide what to keep.

StoredWhat it contains
Visitor requestIP address, user agent, Accept-Language, locale and timezone
PageURL including its query string, path, title and referrer
CampaignUTM source, medium, campaign, term and content
ConsentThe visitor’s consent state
LocationGeoIP country, region and city, and the network (ASN)
DeviceScreen size; for app events the OS and version, manufacturer, model, advertising ID and IDFV when the app collected them
Vendor IDsCookie and click identifiers in vendor_ids (for example the _fbp browser ID, gclid, ttclid, the GA client ID)
Session and librarySession ID, number and engagement time; the collector’s name and version
Partner attributesProfile attributes from partner enrichment (partner_attrs)

Each message is the event JSON: context, click_ids and collected_ids, and the anonymous_id, user_id, device_id, message_id and event_id identifiers, sit beside the mapped properties.

The IP address, device and advertising IDs, and cookie and click identifiers are personal data under UK and EU GDPR. To keep any of them out of storage, strip, mask or hash the field for every integration with the organisation data layer, apply URL redaction to page URLs, or remove it in the installed blueprint; and set the storage’s retention to match your privacy notice. See Personal data by integration.

Consent

The integration’s consent category in Signal decides which events reach Google Pub/Sub; events that fail it are never written. Every message also carries the visitor’s consent state (context.consent, and consent_analytics / consent_marketing in @root) so downstream consumers can apply their own rules.

Testing

  1. Enable the integration in Signal and trigger a test event on your website.
  2. In the Pub/Sub console, click on a subscription attached to the topic.
  3. Click Messages > Pull to retrieve messages.
  4. Inspect the message body to verify the event data.
  5. In Signal, check the Live Events view to confirm delivery status shows as successful.

Alternatively, use the gcloud CLI:

gcloud pubsub subscriptions pull datafly-events-sub --limit=10 --auto-ack

Troubleshooting

ProblemSolution
Events not appearing in subscriptionsVerify the project ID and topic ID are correct. Ensure at least one subscription exists on the topic.
Permission denied (403)The service account lacks the Pub/Sub Publisher role. Add it in IAM & Admin > IAM.
NOT_FOUND: Topic not foundThe topic does not exist in the specified project. Verify the topic ID.
Invalid service account JSONEnsure you pasted the complete JSON key file content.
Messages not being consumedCheck that a subscription exists and consumers are actively pulling or the push endpoint is reachable.
RESOURCE_EXHAUSTEDYou have exceeded Pub/Sub publish quotas. Check quotas and request an increase.
High publish latencyEnsure the service account and topic are in the same region as your Signal deployment.

Visit Google Pub/Sub documentation for full API reference, subscription types, and dead-letter topic configuration.

See also

Confluent Cloud (deprecated)Apache Kafka