Google Bigtable

Datafly Signal delivers events to Google Cloud Bigtable for high-throughput, low-latency NoSQL storage at massive scale, ideal for time-series data and analytics.

Prerequisites

Before configuring Google Bigtable in Signal, you need a GCP project with a Bigtable instance, a table with a column family, and a service account.

Create a GCP Account and Project

Sign up at cloud.google.com if you don’t already have an account.
Create a new project or select an existing one in the GCP Console.
Note the Project ID.

Enable the Bigtable API

Go to APIs & Services > Library.
Search for Cloud Bigtable API and Cloud Bigtable Admin API.
Click Enable for both.

Create a Bigtable Instance

Go to the Bigtable console.
Click Create instance.
Enter an Instance name (e.g. datafly-events) and Instance ID.
Choose the Storage type:
- SSD — lowest latency (recommended for real-time workloads).
- HDD — lower cost for large-volume storage.
Configure clusters — select a region and the number of nodes (minimum 1 for development, 3+ for production).
Click Create.

Bigtable pricing is based on the number of nodes and storage. A single-node development instance is suitable for testing but should not be used in production.

Create a Table with Column Family

In the Bigtable console, click on your instance.
Go to Tables > Create table.
Enter a Table ID (e.g. events).
Add a Column family named events (or your preferred name).
Click Create.

Alternatively, use the cbt CLI tool:

cbt -project your-project -instance datafly-events createtable events
cbt -project your-project -instance datafly-events createfamily events events

Create a Service Account

Go to IAM & Admin > Service Accounts > Create Service Account.
Enter a name (e.g. datafly-signal-bigtable).
Grant the Bigtable User role (roles/bigtable.user).
Click Done.

Generate a Service Account Key

Click on the service account.
Go to Keys > Add Key > Create new key > JSON.
The key file will download. Store it securely.

⚠️

Store the JSON key file securely. Do not commit it to version control.

Configuration

Field	Type	Required	Description
`project_id`	string	Yes	The Google Cloud project ID that contains the Bigtable instance.
`instance_id`	string	Yes	The Bigtable instance ID.
`table_id`	string	Yes	The Bigtable table ID to write rows into.
`column_family`	string	Yes	The column family to write data to. Defaults to `events`.
`service_account_json`	secret	Yes	The full JSON key file content for a GCP service account with Bigtable User role.

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Google Bigtable or filter by Cloud Storage.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/google_bigtable/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Google Bigtable",
    "variant": "default",
    "config": {
      "project_id": "datafly-analytics",
      "instance_id": "datafly-events",
      "table_id": "events",
      "column_family": "events",
      "service_account_json": "{\"type\": \"service_account\", ...}"
    },
    "delivery_mode": "server_side"
  }'

Testing

Enable the integration in Signal and trigger a test event on your website.
Use the cbt CLI to read recent rows:

cbt -project your-project -instance datafly-events read events count=10

Or use the Bigtable console’s Query tab to inspect rows.
In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

Problem	Solution
Events not appearing in the table	Verify the project ID, instance ID, table ID, and column family are correct.
`Permission denied` (403)	The service account lacks the Bigtable User role. Add it in IAM & Admin > IAM.
`Table not found`	The table does not exist in the specified instance. Verify the table ID.
`Column family not found`	The column family name does not match. List column families with `cbt -project ... -instance ... ls <table>`.
Invalid service account JSON	Ensure you pasted the complete JSON key file content, including all fields.
High latency	Check the number of nodes in your Bigtable cluster. Add more nodes for higher throughput.
Row key hotspotting	If all writes go to the same node, consider using a row key with better distribution (e.g. reversed timestamp prefix).

Visit Google Bigtable documentation for full API reference, schema design best practices, and monitoring guides.

Elasticsearch Google Cloud SQL