IntegrationsStreamingConfluent Cloud

Confluent Cloud

Datafly Signal publishes first-party events to Confluent Cloud topics over the standard Kafka wire protocol with SASL/PLAIN authentication.

Why route Confluent Cloud through Datafly Signal

  • The data lands in infrastructure you own. Signal writes to your Confluent Cloud directly from your own deployment. Datafly operates no intermediary and never holds a copy of your event data.
  • Client IP and user agent are recorded as event context. Both come from the original visitor request rather than from your server, so geography and device analysis done in the warehouse reflects the actual visitor.
  • One consent decision governs every destination. Confluent Cloud is gated by the same consent state as the rest of your stack, so a withdrawal applies everywhere at once.

Prerequisites

Before configuring Confluent Cloud in Signal, you need a Confluent Cloud account with a Kafka cluster, a topic, and an API key pair.

Create a Confluent Cloud Account

  1. Sign up at confluent.io/confluent-cloud.
  2. Complete the onboarding wizard.

Create a Kafka Cluster

  1. In the Confluent Cloud console, go to Environments and select your environment (or create a new one).
  2. Click Add cluster.
  3. Choose a cluster type:
    • Basic — multi-tenant, pay-as-you-go (suitable for development and low-volume production).
    • Standard — single-tenant, higher throughput limits.
    • Dedicated — fully isolated, custom configurations.
  4. Select a Cloud provider (AWS, GCP, or Azure) and Region.
  5. Click Launch cluster.
  6. Note the Bootstrap server address from the cluster settings (e.g. pkc-abc123.us-east-1.aws.confluent.cloud:9092).

Create a Topic

  1. In your cluster, go to Topics > Create topic.
  2. Enter a Topic name (e.g. datafly-events).
  3. Set the number of Partitions (6 is a good default for moderate throughput).
  4. Configure Retention settings (default 7 days).
  5. Click Create with defaults or customise further.

Generate an API Key Pair

  1. In your cluster, go to API Keys (under Cluster settings or Data integration).
  2. Click Create key.
  3. Choose Global access or specify a service account.
  4. Click Create.
  5. Copy the API Key and API Secret immediately — the secret is only shown once.
⚠️

Store the API secret securely. If you lose it, you must create a new key pair.

Configuration

FieldTypeRequiredDescription
bootstrap_serversstringYesThe Confluent Cloud bootstrap server address (e.g. pkc-abc123.us-east-1.aws.confluent.cloud:9092).
topicstringYesThe Kafka topic to produce messages to.
security_protocolselectYesSet to SASL_SSL for Confluent Cloud.
sasl_mechanismselectYesSet to PLAIN.
sasl_usernamestringYesThe Confluent Cloud API key.
sasl_passwordsecretYesThe Confluent Cloud API secret.

Signal Setup

Quick Setup

  1. Navigate to Integrations in the sidebar.
  2. Open the Integration Library tab.
  3. Find Confluent Cloud or filter by Streaming.
  4. Click Install, select a variant if available, and fill in the required fields.
  5. Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/confluent_cloud/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Confluent Cloud",
    "variant": "default",
    "config": {
      "bootstrap_servers": "pkc-abc123.us-east-1.aws.confluent.cloud:9092",
      "topic": "datafly-events",
      "security_protocol": "SASL_SSL",
      "sasl_mechanism": "PLAIN",
      "sasl_username": "YOUR_API_KEY",
      "sasl_password": "YOUR_API_SECRET"
    },
    "delivery_mode": "server_side"
  }'

Schema

Each event is produced as one Kafka message — same wire format as the Apache Kafka integration:

  • Key — anonymous_id (or user_id if set).
  • Value — JSON document of the canonical event envelope.
  • Headers — content-type, event_type, event_name.

Consent

Confluent Cloud is a first-party destination under your control. The default blueprint forwards all events. Apply consent filtering in pipeline transforms or downstream stream processors (ksqlDB, Kafka Streams) on context.consent if needed.

Testing

  1. Enable the integration in Signal and trigger a test event on your website.
  2. In the Confluent Cloud console, go to your cluster > Topics > your topic.
  3. Click Messages to view recently produced messages.
  4. Inspect the message value to verify the event data.
  5. In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

ProblemSolution
Events not appearing in the topicVerify the bootstrap server address and topic name are correct.
Authentication failureThe API key or secret is incorrect. Verify the credentials. Generate a new key pair if needed.
TopicAuthorizationExceptionThe API key lacks produce permission on the topic. Create a new key with appropriate ACLs or use a global access key.
UnknownTopicOrPartitionThe topic does not exist. Verify the topic name in the Confluent Cloud console.
Connection timeoutEnsure Signal can reach the bootstrap server on port 9092. Check any firewall or network policies.
MessageSizeTooLargeIndividual messages exceed the topic’s max message size (default 1 MB). Check event payload size.
API key expired or revokedGenerate a new API key pair in the Confluent Cloud console.

Visit Confluent Cloud documentation for full API reference, Schema Registry setup, and connector configuration.

See also

Azure Event HubsGoogle Pub/Sub