Google Cloud Spanner

Datafly Signal delivers events to Google Cloud Spanner for globally distributed, strongly consistent, horizontally scalable relational database storage.

Prerequisites

Before configuring Google Cloud Spanner in Signal, you need a GCP project with a Spanner instance, a database with a target table, and a service account.

Create a GCP Account and Project

Sign up at cloud.google.com if you don’t already have an account.
Create a new project or select an existing one in the GCP Console.
Note the Project ID.

Enable the Cloud Spanner API

Go to APIs & Services > Library.
Search for Cloud Spanner API.
Click Enable.

Create a Spanner Instance

Go to the Spanner console.
Click Create instance.
Enter an Instance name (e.g. datafly-events) and Instance ID.
Choose a Configuration:
- Regional — data in a single region (lower latency, lower cost).
- Multi-region — data replicated across regions (higher availability).
Set the Compute capacity (processing units or nodes). 1 node = 1000 processing units.
Click Create.

For development and testing, you can use the free trial instance (1 node, limited to specific regions). For production, size the instance based on your expected write throughput.

Create a Database

In the Spanner console, click on your instance.
Click Create database.
Enter a Database name (e.g. events_db).
Click Create.

Create a Table

In the Spanner console, open the database and run the following DDL:

CREATE TABLE Events (
  event_id STRING(64) NOT NULL,
  type STRING(20),
  event STRING(256),
  anonymous_id STRING(64),
  user_id STRING(256),
  timestamp TIMESTAMP,
  received_at TIMESTAMP,
  context JSON,
  properties JSON,
  traits JSON,
  source_id STRING(64),
  integration_id STRING(64),
) PRIMARY KEY (event_id);

Spanner uses the primary key for data distribution. Using event_id (a UUID) as the primary key ensures even distribution across splits. Avoid monotonically increasing keys like timestamps as primary keys — they cause hotspots.

Create a Service Account

Go to IAM & Admin > Service Accounts > Create Service Account.
Enter a name (e.g. datafly-signal-spanner).
Grant the Cloud Spanner Database User role (roles/spanner.databaseUser).
Click Done.

Generate a Service Account Key

Click on the service account.
Go to Keys > Add Key > Create new key > JSON.
The key file will download. Store it securely.

⚠️

Store the JSON key file securely. Do not commit it to version control.

Configuration

Field	Type	Required	Description
`project_id`	string	Yes	The Google Cloud project ID that contains the Spanner instance.
`instance_id`	string	Yes	The Spanner instance ID.
`database_id`	string	Yes	The Spanner database ID.
`table_name`	string	Yes	The target table name to insert rows into.
`service_account_json`	secret	Yes	The full JSON key file content for a GCP service account with Spanner Database User permissions.

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Google Cloud Spanner or filter by Cloud Storage.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/google_spanner/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Google Cloud Spanner",
    "variant": "default",
    "config": {
      "project_id": "datafly-analytics",
      "instance_id": "datafly-events",
      "database_id": "events_db",
      "table_name": "Events",
      "service_account_json": "{\"type\": \"service_account\", ...}"
    },
    "delivery_mode": "server_side"
  }'

Testing

Enable the integration in Signal and trigger a test event on your website.
Open the Spanner console and navigate to your database.
Go to Query and run:

SELECT * FROM Events ORDER BY timestamp DESC LIMIT 10;

Verify that event rows are appearing with correct data.
In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

Problem	Solution
Events not appearing in the table	Verify the project ID, instance ID, database ID, and table name are correct.
`Permission denied` (403)	The service account lacks the Cloud Spanner Database User role. Add it in IAM & Admin > IAM.
`NOT_FOUND: Database not found`	The database does not exist. Verify the database ID in the Spanner console.
`NOT_FOUND: Table not found`	The table does not exist in the database. Verify the table name (case-sensitive in Spanner).
Invalid service account JSON	Ensure you pasted the complete JSON key file content.
`RESOURCE_EXHAUSTED`	The instance is at capacity. Increase the number of processing units or nodes.
Write hotspots	Avoid sequential primary keys. Use UUIDs or add a hash prefix to distribute writes evenly across splits.

Visit Google Cloud Spanner documentation for full SQL reference, schema design best practices, and performance tuning guides.

Google Firestore MongoDB