Apache Kafka

Datafly Signal publishes first-party events to Apache Kafka topics for real-time stream processing, event sourcing, and integration with downstream consumers and microservices.

Why route Apache Kafka through Datafly Signal

• The data lands in infrastructure you own. Signal writes to your Apache Kafka directly from your own deployment. Datafly operates no intermediary and never holds a copy of your event data.
• Client IP and user agent are recorded as event context. Both come from the original visitor request rather than from your server, so geography and device analysis done in the warehouse reflects the actual visitor.
• Presets for eight verticals. General (the default: page, identify and the key cross-vertical events), Retail, Insurance, Travel, Media & Streaming, Publishing, B2B / SaaS and Automotive, each writing one named field per documented event property. Email, phone and names are written only as SHA-256 digests unless you deliberately change the installed blueprint.
• One consent decision governs every destination. Apache Kafka is gated by the same consent state as the rest of your stack, so a withdrawal applies everywhere at once.

Prerequisites

Before configuring Apache Kafka in Signal, you need a Kafka cluster with a topic and optional SASL credentials for authentication.

kafka-topics.sh --create \
  --topic datafly-events \
  --partitions 6 \
  --replication-factor 3 \
  --bootstrap-server broker1:9092

kafka-configs.sh --alter \
  --add-config 'SCRAM-SHA-256=[password=your_password]' \
  --entity-type users \
  --entity-name datafly-signal \
  --bootstrap-server broker1:9092

Configuration

Signal Setup

Quick Setup

1. Navigate to Integrations in the sidebar.
2. Open the Integration Library tab.
3. Find Apache Kafka or filter by Streaming.
4. Click Install, select a variant if available, and fill in the required fields.
5. Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/kafka/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Apache Kafka",
    "variant": "default",
    "config": {
      "bootstrap_servers": "broker1:9092,broker2:9092",
      "topic": "datafly-events",
      "security_protocol": "SASL_SSL",
      "sasl_mechanism": "SCRAM-SHA-256",
      "sasl_username": "datafly-signal",
      "sasl_password": "your_password"
    },
    "delivery_mode": "server_side"
  }'

What Signal writes

Each event is one message whose value is the Signal event as JSON:

Each event is one Kafka message:

• Key: user_id, or anonymous_id when there is no user ID. The writer’s hash balancer maps a key to one partition, so one visitor’s events stay in order on one partition (an anonymous visitor’s events move to the user ID’s partition once they log in).
• Headers: datafly_message_id, datafly_event_type, datafly_event_name.
• Timestamp: the event time.
• Producer settings: acks=all, Snappy compression, batches of up to 100 messages or 50 ms per partition.
• Delivery: at least once. A retry after a timeout can write a message twice; deduplicate on message_id.

A purchase from an identified visitor, as written by the default (General) preset. The raw email, phone and name in the visitor’s identify traits are not in it; only their digests are. Empty context objects are trimmed here for brevity.

{
  "anonymous_id": "anon-s1-7f3a",
  "user_id": "crm-4711",
  "event": "purchase",
  "type": "track",
  "properties": {
    "@root": {
      "anonymous_id": "anon-s1-7f3a",
      "email_sha256": "86e0b9e56c17cc4d12387e1949b85053fbe73bc3ce5a1188713a9d300cc6133d",
      "event_id": "evt-s1-2",
      "event_name": "purchase",
      "event_timestamp": "2026-09-23T09:30:02.000Z",
      "event_type": "track",
      "first_name_sha256": "81f8f6dde88365f3928796ec7aa53f72820b06db8664f5fe76a7eb13e24546a2",
      "ip_address": "203.0.113.9",
      "last_name_sha256": "4f887a0887ca68593c8acd4e83e0b6bc5f42cd5aff7d3a159041c84230ed1cf6",
      "locale": "en-GB",
      "message_id": "msg-s1-kafka-2",
      "name_sha256": "ed37d99b1445238af3386f81a77a2caf3fffc0cd610be39b4f3ba53943dc66bf",
      "page_url": "https://shop.example/checkout",
      "phone_sha256": "a8acc3a90a7b4e4dc65e93db9240ed26523050ef754d63b75b5161de76781436",
      "user_agent": "Mozilla/5.0 (Macintosh)",
      "user_id": "crm-4711",
      "utm_medium": "cpc",
      "utm_source": "google"
    },
    "coupon": "AUTUMN10",
    "currency": "GBP",
    "items": [{ "item_id": "SKU-1", "item_name": "Trail shoe", "price": 80, "quantity": 1 }],
    "shipping": 4.5,
    "tax": 14.08,
    "transaction_id": "SIG-ORDER-1001",
    "value": 84.5
  },
  "context": {
    "ip": "203.0.113.9",
    "user_agent": "Mozilla/5.0 (Macintosh)",
    "locale": "en-GB",
    "page": { "url": "https://shop.example/checkout", "title": "Checkout", "path": "/checkout" },
    "campaign": { "source": "google", "medium": "cpc" }
  },
  "timestamp": "2026-09-23T09:30:02Z",
  "sent_at": "2026-09-23T09:30:02Z",
  "message_id": "msg-s1-kafka-2",
  "org_id": "org-s1",
  "pipeline_id": "pl-s1",
  "received_at": "2026-09-23T09:30:02Z",
  "event_id": "evt-s1-2",
  "vendor_type": "kafka"
}

Stored event context

Alongside the fields your blueprint maps (where email, phone, names and address are stored as normalised SHA-256 by default), each message carries the event’s collected context exactly as Signal received it. The blueprint does not filter or hash the context: this is deliberate (decision 2026-09-23), so the context is complete for attribution, fraud and debugging queries, and it is listed here so you can decide what to keep.

Each message is the event JSON: context, click_ids and collected_ids, and the anonymous_id, user_id, device_id, message_id and event_id identifiers, sit beside the mapped properties.

The IP address, device and advertising IDs, and cookie and click identifiers are personal data under UK and EU GDPR. To keep any of them out of storage, strip, mask or hash the field for every integration with the organisation data layer, apply URL redaction to page URLs, or remove it in the installed blueprint; and set the storage’s retention to match your privacy notice. See Personal data by integration.

Consent

The integration’s consent category in Signal decides which events reach Apache Kafka; events that fail it are never written. Every message also carries the visitor’s consent state (context.consent, and consent_analytics / consent_marketing in @root) so downstream consumers can apply their own rules.

Testing

1. Enable the integration in Signal and trigger a test event on your website.
2. Consume messages from the topic to verify:

kafka-console-consumer.sh \
  --topic datafly-events \
  --from-beginning \
  --max-messages 10 \
  --bootstrap-server broker1:9092

3. Inspect the message values to verify the event data.
4. In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

See also

• Confluent Cloud
• Amazon Kinesis
• Azure Event Hubs
• Google Pub/Sub