Configuration

All Datafly Signal services share a common configuration model loaded from environment variables. This page is the complete reference.

Shared Environment Variables

These variables are read by every Go service via the shared config.Load() function in shared/config/config.go.

General

Kafka

Redis

PostgreSQL

Authentication & Security

GeoIP

Per-Service Configuration

Each service reads the same shared config but uses only the fields relevant to its function. Below is which variables matter to which service.

Ingestion Gateway (port 8080)

Receives events from Datafly.js and server-side sources. Publishes to Kafka.

• PORT — defaults to 8080
• KAFKA_BROKERS — writes to raw-events topic
• REDIS_ADDR — vendor ID lookups, rate limiting
• DATABASE_URL — source and pipeline key validation
• HMAC_SECRET — pipeline key HMAC verification
• GEOIP_PATH — IP geolocation enrichment

Event Processor (port 8081)

Consumes from raw-events, applies Org Data Layer and Pipeline transformations, publishes to delivery-* topics.

• PORT — defaults to 8081
• KAFKA_BROKERS — reads raw-events, writes delivery-{integration_id}
• KAFKA_GROUP_ID — defaults to event-processor
• REDIS_ADDR — vendor ID injection, identity graph lookups
• DATABASE_URL — pipeline configs, transformation files, org data layer rules

Delivery Workers (port 8082)

Consume from delivery-* topics and deliver events to vendor APIs.

• PORT — defaults to 8082
• KAFKA_BROKERS — reads delivery-{integration_id} topics
• KAFKA_GROUP_ID — defaults to delivery-workers
• DATABASE_URL — integration credentials and configuration
• REDIS_ADDR — rate limiting, delivery state

Identity Hub (port 8083)

Cross-domain identity resolution via encrypted tokens.

• PORT — defaults to 8083
• REDIS_ADDR — identity graph storage
• ENCRYPTION_KEY — token encryption/decryption

Management API (port 8084)

REST + WebSocket API for the admin dashboard.

• PORT — defaults to 8084
• DATABASE_URL — all CRUD operations (orgs, users, sources, integrations, pipelines)
• REDIS_ADDR — session cache, real-time event stream
• KAFKA_BROKERS — real-time event debugger (WebSocket consumer)
• JWT_SECRET — JWT token signing and verification

Management UI (port 3000)

The Next.js frontend connects to the Management API. Its configuration is set via Next.js environment variables.

Example .env File

Create an .env file in the application/ directory for local overrides. This is not required for default Docker Compose development.

# =============================================================================
# Datafly Signal — Local Development Environment
# =============================================================================
 
# General
DATAFLY_ENV=dev
LOG_LEVEL=debug
 
# Kafka
KAFKA_BROKERS=localhost:9092
 
# Redis
REDIS_ADDR=localhost:6379
REDIS_PASSWORD=
REDIS_DB=0
 
# PostgreSQL
DATABASE_URL=postgres://datafly:datafly@localhost:5432/datafly?sslmode=disable
 
# Auth (leave empty for dev, required for prod)
JWT_SECRET=
HMAC_SECRET=
ENCRYPTION_KEY=
 
# GeoIP (optional, for IP geolocation enrichment)
GEOIP_PATH=/data/GeoLite2-City.mmdb
 
# Timeouts
READ_TIMEOUT=10s
WRITE_TIMEOUT=10s
 
# Management UI
NEXT_PUBLIC_API_URL=http://localhost:8084

Production Configuration

For production deployments, set all variables via your orchestration layer (Kubernetes secrets, Helm values, or your CI/CD pipeline). Key differences from development:

• DATAFLY_ENV=prod — enables strict validation
• JWT_SECRET and HMAC_SECRET — must be set to strong, unique values
• ENCRYPTION_KEY — should be a cryptographically random 32-byte hex string
• KAFKA_BROKERS — point to your production Kafka cluster
• REDIS_ADDR — point to your production Redis (consider Redis Cluster mode)
• DATABASE_URL — use sslmode=require and strong credentials
• LOG_LEVEL=info or warn — reduce log volume in production