Google Pub/Sub

Datafly Signal publishes events to Google Cloud Pub/Sub topics for real-time messaging, event-driven architectures, and stream processing with downstream subscribers.

Prerequisites

Before configuring Google Pub/Sub in Signal, you need a GCP project with a Pub/Sub topic and a service account with the Pub/Sub Publisher role.

Create a GCP Account and Project

Sign up at cloud.google.com if you don’t already have an account.
Create a new project or select an existing one in the GCP Console.
Note the Project ID.

Enable the Pub/Sub API

Go to APIs & Services > Library.
Search for Cloud Pub/Sub API.
Click Enable.

Create a Pub/Sub Topic

Go to the Pub/Sub console.
Click Create topic.
Enter a Topic ID (e.g. datafly-events).
Optionally add a default subscription (creates a pull subscription automatically).
Click Create.

If you want consumers to receive events, you need at least one subscription attached to the topic. Subscriptions can be created later and can be pull-based, push-based, or BigQuery subscriptions.

Create a Subscription (Optional)

In the Pub/Sub console, click on your topic.
Click Create subscription.
Enter a Subscription ID (e.g. datafly-events-sub).
Choose the delivery type:
- Pull — consumers poll for messages.
- Push — Pub/Sub pushes messages to an HTTPS endpoint.
- BigQuery — writes directly to a BigQuery table.
Set Acknowledgement deadline and Message retention as needed.
Click Create.

Create a Service Account

Go to IAM & Admin > Service Accounts > Create Service Account.
Enter a name (e.g. datafly-signal-pubsub).
Grant the Pub/Sub Publisher role (roles/pubsub.publisher).
Click Done.

Generate a Service Account Key

Click on the service account.
Go to Keys > Add Key > Create new key > JSON.
The key file will download. Store it securely.

⚠️

Store the JSON key file securely. Do not commit it to version control.

Configuration

Field	Type	Required	Description
`project_id`	string	Yes	The Google Cloud project ID that contains the Pub/Sub topic.
`topic_id`	string	Yes	The Pub/Sub topic ID to publish messages to. The topic must already exist.
`service_account_json`	secret	Yes	The full JSON key file content for a GCP service account with Pub/Sub Publisher permissions.

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Google Pub/Sub or filter by Cloud Storage.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/google_pubsub/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Google Pub/Sub",
    "variant": "default",
    "config": {
      "project_id": "datafly-analytics",
      "topic_id": "datafly-events",
      "service_account_json": "{\"type\": \"service_account\", ...}"
    },
    "delivery_mode": "server_side"
  }'

Testing

Enable the integration in Signal and trigger a test event on your website.
In the Pub/Sub console, click on a subscription attached to the topic.
Click Messages > Pull to retrieve messages.
Inspect the message body to verify the event data.
In Signal, check the Live Events view to confirm delivery status shows as successful.

Alternatively, use the gcloud CLI:

gcloud pubsub subscriptions pull datafly-events-sub --limit=10 --auto-ack

Troubleshooting

Problem	Solution
Events not appearing in subscriptions	Verify the project ID and topic ID are correct. Ensure at least one subscription exists on the topic.
`Permission denied` (403)	The service account lacks the Pub/Sub Publisher role. Add it in IAM & Admin > IAM.
`NOT_FOUND: Topic not found`	The topic does not exist in the specified project. Verify the topic ID.
Invalid service account JSON	Ensure you pasted the complete JSON key file content.
Messages not being consumed	Check that a subscription exists and consumers are actively pulling or the push endpoint is reachable.
`RESOURCE_EXHAUSTED`	You have exceeded Pub/Sub publish quotas. Check quotas and request an increase.
High publish latency	Ensure the service account and topic are in the same region as your Signal deployment.

Visit Google Pub/Sub documentation for full API reference, subscription types, and dead-letter topic configuration.

Google Cloud Storage Apache Kafka