Confluent Cloud

Datafly Signal publishes first-party events to Confluent Cloud topics over the standard Kafka wire protocol with SASL/PLAIN authentication.

Why route Confluent Cloud through Datafly Signal

• The data lands in infrastructure you own. Signal writes to your Confluent Cloud directly from your own deployment. Datafly operates no intermediary and never holds a copy of your event data.
• Client IP and user agent are recorded as event context. Both come from the original visitor request rather than from your server, so geography and device analysis done in the warehouse reflects the actual visitor.
• Presets for eight verticals. General (the default: page, identify and the key cross-vertical events), Retail, Insurance, Travel, Media & Streaming, Publishing, B2B / SaaS and Automotive, each writing one named field per documented event property. Email, phone and names are written only as SHA-256 digests unless you deliberately change the installed blueprint.
• One consent decision governs every destination. Confluent Cloud is gated by the same consent state as the rest of your stack, so a withdrawal applies everywhere at once.

Prerequisites

Before configuring Confluent Cloud in Signal, you need a Confluent Cloud account with a Kafka cluster, a topic, and an API key pair.

Configuration

Signal Setup

Quick Setup

1. Navigate to Integrations in the sidebar.
2. Open the Integration Library tab.
3. Find Confluent Cloud or filter by Streaming.
4. Click Install, select a variant if available, and fill in the required fields.
5. Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/confluent_cloud/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Confluent Cloud",
    "variant": "default",
    "config": {
      "bootstrap_servers": "pkc-abc123.us-east-1.aws.confluent.cloud:9092",
      "topic": "datafly-events",
      "security_protocol": "SASL_SSL",
      "sasl_mechanism": "PLAIN",
      "sasl_username": "YOUR_API_KEY",
      "sasl_password": "YOUR_API_SECRET"
    },
    "delivery_mode": "server_side"
  }'

What Signal writes

Confluent Cloud uses the same writer as the Apache Kafka integration. Each event is one message whose value is the Signal event as JSON:

Each event is one Kafka message:

• Key: user_id, or anonymous_id when there is no user ID. The writer’s hash balancer maps a key to one partition, so one visitor’s events stay in order on one partition (an anonymous visitor’s events move to the user ID’s partition once they log in).
• Headers: datafly_message_id, datafly_event_type, datafly_event_name.
• Timestamp: the event time.
• Producer settings: acks=all, Snappy compression, batches of up to 100 messages or 50 ms per partition.
• Delivery: at least once. A retry after a timeout can write a message twice; deduplicate on message_id.

A purchase from an identified visitor, as written by the default (General) preset. The raw email, phone and name in the visitor’s identify traits are not in it; only their digests are. Empty context objects are trimmed here for brevity.

{
  "anonymous_id": "anon-s1-7f3a",
  "user_id": "crm-4711",
  "event": "purchase",
  "type": "track",
  "properties": {
    "@root": {
      "anonymous_id": "anon-s1-7f3a",
      "email_sha256": "86e0b9e56c17cc4d12387e1949b85053fbe73bc3ce5a1188713a9d300cc6133d",
      "event_id": "evt-s1-2",
      "event_name": "purchase",
      "event_timestamp": "2026-09-23T09:30:02.000Z",
      "event_type": "track",
      "first_name_sha256": "81f8f6dde88365f3928796ec7aa53f72820b06db8664f5fe76a7eb13e24546a2",
      "ip_address": "203.0.113.9",
      "last_name_sha256": "4f887a0887ca68593c8acd4e83e0b6bc5f42cd5aff7d3a159041c84230ed1cf6",
      "locale": "en-GB",
      "message_id": "msg-s1-confluent_cloud-2",
      "name_sha256": "ed37d99b1445238af3386f81a77a2caf3fffc0cd610be39b4f3ba53943dc66bf",
      "page_url": "https://shop.example/checkout",
      "phone_sha256": "a8acc3a90a7b4e4dc65e93db9240ed26523050ef754d63b75b5161de76781436",
      "user_agent": "Mozilla/5.0 (Macintosh)",
      "user_id": "crm-4711",
      "utm_medium": "cpc",
      "utm_source": "google"
    },
    "coupon": "AUTUMN10",
    "currency": "GBP",
    "items": [{ "item_id": "SKU-1", "item_name": "Trail shoe", "price": 80, "quantity": 1 }],
    "shipping": 4.5,
    "tax": 14.08,
    "transaction_id": "SIG-ORDER-1001",
    "value": 84.5
  },
  "context": {
    "ip": "203.0.113.9",
    "user_agent": "Mozilla/5.0 (Macintosh)",
    "locale": "en-GB",
    "page": { "url": "https://shop.example/checkout", "title": "Checkout", "path": "/checkout" },
    "campaign": { "source": "google", "medium": "cpc" }
  },
  "timestamp": "2026-09-23T09:30:02Z",
  "sent_at": "2026-09-23T09:30:02Z",
  "message_id": "msg-s1-confluent_cloud-2",
  "org_id": "org-s1",
  "pipeline_id": "pl-s1",
  "received_at": "2026-09-23T09:30:02Z",
  "event_id": "evt-s1-2",
  "vendor_type": "confluent_cloud"
}

Stored event context

Alongside the fields your blueprint maps (where email, phone, names and address are stored as normalised SHA-256 by default), each message carries the event’s collected context exactly as Signal received it. The blueprint does not filter or hash the context: this is deliberate (decision 2026-09-23), so the context is complete for attribution, fraud and debugging queries, and it is listed here so you can decide what to keep.

Each message is the event JSON: context, click_ids and collected_ids, and the anonymous_id, user_id, device_id, message_id and event_id identifiers, sit beside the mapped properties.

The IP address, device and advertising IDs, and cookie and click identifiers are personal data under UK and EU GDPR. To keep any of them out of storage, strip, mask or hash the field for every integration with the organisation data layer, apply URL redaction to page URLs, or remove it in the installed blueprint; and set the storage’s retention to match your privacy notice. See Personal data by integration.

Consent

The integration’s consent category in Signal decides which events reach Confluent Cloud; events that fail it are never written. Every message also carries the visitor’s consent state (context.consent, and consent_analytics / consent_marketing in @root) so downstream consumers can apply their own rules.

Testing

1. Enable the integration in Signal and trigger a test event on your website.
2. In the Confluent Cloud console, go to your cluster > Topics > your topic.
3. Click Messages to view recently produced messages.
4. Inspect the message value to verify the event data.
5. In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

See also

• Apache Kafka
• Amazon Kinesis
• Azure Event Hubs
• Google Pub/Sub