Google Bigtable

Datafly Signal writes first-party events as rows into a Google Cloud Bigtable table — high-throughput, low-latency NoSQL storage suited to time-series workloads.

Prerequisites

Before configuring Google Bigtable in Signal, you need a GCP project with a Bigtable instance, a table with a column family, and a service account.

Create a GCP Account and Project

Sign up at cloud.google.com if you don’t already have an account.
Create a new project or select an existing one in the GCP Console.
Note the Project ID.

Enable the Bigtable API

Go to APIs & Services > Library.
Search for Cloud Bigtable API and Cloud Bigtable Admin API.
Click Enable for both.

Create a Bigtable Instance

Go to the Bigtable console.
Click Create instance.
Enter an Instance name (e.g. datafly-events) and Instance ID.
Choose the Storage type:
- SSD — lowest latency (recommended for real-time workloads).
- HDD — lower cost for large-volume storage.
Configure clusters — select a region and the number of nodes (minimum 1 for development, 3+ for production).
Click Create.

Bigtable pricing is based on the number of nodes and storage. A single-node development instance is suitable for testing but should not be used in production.

Create a Table with Column Family

In the Bigtable console, click on your instance.
Go to Tables > Create table.
Enter a Table ID (e.g. events).
Add a Column family named events (or your preferred name).
Click Create.

Alternatively, use the cbt CLI tool:

cbt -project your-project -instance datafly-events createtable events
cbt -project your-project -instance datafly-events createfamily events events

Create a Service Account

Go to IAM & Admin > Service Accounts > Create Service Account.
Enter a name (e.g. datafly-signal-bigtable).
Grant the Bigtable User role (roles/bigtable.user).
Click Done.

Generate a Service Account Key

Click on the service account.
Go to Keys > Add Key > Create new key > JSON.
The key file will download. Store it securely.

⚠️

Store the JSON key file securely. Do not commit it to version control.

Configuration

Field	Type	Required	Description
`project_id`	string	Yes	The Google Cloud project ID that contains the Bigtable instance.
`instance_id`	string	Yes	The Bigtable instance ID.
`table`	string	Yes	The Bigtable table ID to write rows into. Also accepts `table_id`.
`column_family`	string	Yes	The column family to write event columns to.
`service_account_json`	secret	Yes	The full JSON key file content for a service account with `roles/bigtable.user`.

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Google Bigtable or filter by Database.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/google_bigtable/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Google Bigtable",
    "variant": "default",
    "config": {
      "project_id": "datafly-analytics",
      "instance_id": "datafly-events",
      "table": "events",
      "column_family": "events",
      "service_account_json": "{\"type\": \"service_account\", ...}"
    },
    "delivery_mode": "server_side"
  }'

Schema

Each event becomes one Bigtable row. The row key is <reversed-timestamp>#<event_id> (reversed timestamp prefix prevents hotspotting on monotonically increasing keys). The columns are written under the configured column family, with one column per envelope field:

Column	Type	Notes
`event_id`	bytes	Unique per event.
`type`	bytes	Event type.
`event`	bytes	Event name.
`anonymous_id`	bytes	First-party visitor identifier.
`user_id`	bytes	Logged-in user identifier (optional).
`timestamp`	bytes	ISO-8601 client event time.
`received_at`	bytes	ISO-8601 time Signal received the event.
`sent_at`	bytes	ISO-8601 time the row was written.
`context`	bytes	JSON document.
`properties`	bytes	JSON document.
`traits`	bytes	JSON document.
`source_id`	bytes	Pipeline source identifier.
`integration_id`	bytes	Signal integration identifier.

Bigtable stores all values as bytes — apply type conversion in your reader.

Bigtable is a first-party destination in your own GCP project. The default blueprint forwards all events. Apply consent filtering via pipeline transforms or filter-on-read at the application layer over the context column if needed.

Testing

Enable the integration in Signal and trigger a test event on your website.
Use the cbt CLI to read recent rows:

cbt -project your-project -instance datafly-events read events count=10

Or use the Bigtable console’s Query tab to inspect rows.
In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

Problem	Solution
Events not appearing in the table	Verify the project ID, instance ID, table ID, and column family are correct.
`Permission denied` (403)	The service account lacks the Bigtable User role. Add it in IAM & Admin > IAM.
`Table not found`	The table does not exist in the specified instance. Verify the table ID.
`Column family not found`	The column family name does not match. List column families with `cbt -project ... -instance ... ls <table>`.
Invalid service account JSON	Ensure you pasted the complete JSON key file content, including all fields.
High latency	Check the number of nodes in your Bigtable cluster. Add more nodes for higher throughput.
Row key hotspotting	If all writes go to the same node, consider using a row key with better distribution (e.g. reversed timestamp prefix).

Visit Google Bigtable documentation for full API reference, schema design best practices, and monitoring guides.