Apache Kafka

Datafly Signal publishes events to Apache Kafka topics for real-time stream processing, event sourcing, and integration with downstream consumers and microservices.

Prerequisites

Before configuring Apache Kafka in Signal, you need a Kafka cluster with a topic and optional SASL credentials for authentication.

Set Up a Kafka Cluster

You have several options:

Option A: Self-Hosted Kafka

Install Apache Kafka using the official quickstart.
Start ZooKeeper (or use KRaft mode for ZooKeeper-less setups).
Start one or more Kafka brokers.
Note the Bootstrap servers addresses (e.g. broker1:9092,broker2:9092).

Option B: Managed Kafka

Use a managed Kafka service such as Amazon MSK, Aiven, Redpanda, or Strimzi on Kubernetes. Follow the provider’s setup guide and note the bootstrap servers.

Create a Topic

Create the topic that Signal will produce to:

kafka-topics.sh --create \
  --topic datafly-events \
  --partitions 6 \
  --replication-factor 3 \
  --bootstrap-server broker1:9092

Or if using a managed service, create the topic through the provider’s console.

Choose the number of partitions based on expected throughput and consumer parallelism. 6 partitions is a good starting point for moderate workloads. Increasing partitions later requires careful planning.

Configure SASL Authentication (If Required)

If your Kafka cluster requires authentication:

Create SASL credentials for Signal. The exact process depends on your Kafka deployment:
- SASL/PLAIN: Create a username and password in the JAAS configuration.
- SASL/SCRAM: Use kafka-configs.sh to create SCRAM credentials.
- SASL/OAUTHBEARER: Configure an OAuth provider and obtain client credentials.
Note the Security protocol (SASL_SSL for encrypted + authenticated connections, SASL_PLAINTEXT for authenticated without encryption).
Note the SASL mechanism (e.g. PLAIN, SCRAM-SHA-256, SCRAM-SHA-512).

Example SCRAM credential creation:

kafka-configs.sh --alter \
  --add-config 'SCRAM-SHA-256=[password=your_password]' \
  --entity-type users \
  --entity-name datafly-signal \
  --bootstrap-server broker1:9092

⚠️

Always use SASL_SSL in production to encrypt data in transit. SASL_PLAINTEXT transmits credentials in cleartext and should only be used in development environments.

Verify Network Access

Ensure your Signal deployment can reach the Kafka bootstrap servers on the configured port (default 9092 for plaintext, 9093 for SSL). Check firewall rules, security groups, and DNS resolution.

Configuration

Field	Type	Required	Description
`bootstrap_servers`	string	Yes	Comma-separated list of Kafka broker addresses in `host:port` format.
`topic`	string	Yes	The Kafka topic to produce messages to. The topic must already exist or auto-creation must be enabled.
`security_protocol`	select	Yes	The protocol for broker communication: `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL`.
`sasl_mechanism`	select	No	The SASL mechanism: `PLAIN`, `SCRAM-SHA-256`, or `SCRAM-SHA-512`. Required when using SASL protocols.
`sasl_username`	string	No	The SASL username. Required when a SASL mechanism is selected.
`sasl_password`	secret	No	The SASL password. Required when a SASL mechanism is selected.

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Apache Kafka or filter by Cloud Storage.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/kafka/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Apache Kafka",
    "variant": "default",
    "config": {
      "bootstrap_servers": "broker1:9092,broker2:9092",
      "topic": "datafly-events",
      "security_protocol": "SASL_SSL",
      "sasl_mechanism": "SCRAM-SHA-256",
      "sasl_username": "datafly-signal",
      "sasl_password": "your_password"
    },
    "delivery_mode": "server_side"
  }'

Testing

Enable the integration in Signal and trigger a test event on your website.
Consume messages from the topic to verify:

kafka-console-consumer.sh \
  --topic datafly-events \
  --from-beginning \
  --max-messages 10 \
  --bootstrap-server broker1:9092

Inspect the message values to verify the event data.
In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

Problem	Solution
Events not appearing in the topic	Verify the bootstrap servers and topic name are correct.
Connection timeout	Ensure Signal can reach the Kafka brokers. Check DNS, firewalls, and security groups. Verify the port matches the security protocol.
`SASL authentication failed`	The SASL username or password is incorrect. Verify the credentials and SASL mechanism.
`TopicAuthorizationException`	The SASL user lacks produce permission on the topic. Configure ACLs with `kafka-acls.sh`.
`UnknownTopicOrPartitionException`	The topic does not exist and auto-creation is disabled. Create the topic manually.
`MessageSizeTooLarge`	The event payload exceeds `message.max.bytes` on the broker or topic. Increase the limit or reduce payload size.
SSL handshake failure	If using `SSL` or `SASL_SSL`, ensure the broker’s SSL certificate is trusted. Check the CA certificate configuration.
Wrong security protocol	The security protocol must match the broker’s listener configuration. `PLAINTEXT` for port 9092, `SSL` for 9093 is a common pattern but varies by deployment.

Visit Apache Kafka documentation for full producer configuration reference, ACL setup, and security configuration.

Google Pub/Sub Snowflake