Environment Variables
All Datafly Signal services are configured via environment variables. This page lists every variable, its default value, and which services use it.
Common Variables
These variables are used by all Go services.
| Variable | Default | Required | Description |
|---|---|---|---|
KAFKA_BROKERS | localhost:9092 | Yes | Comma-separated list of Kafka broker addresses |
KAFKA_TLS_ENABLED | false | No | Enable TLS for Kafka connections |
REDIS_URL | redis://localhost:6379 | Yes | Redis connection URL. Use rediss:// for TLS |
DATABASE_URL | postgresql://datafly:datafly@localhost:5432/datafly?sslmode=disable | Yes | PostgreSQL connection string |
JWT_SECRET | — | Yes | Secret key for signing and verifying JWT tokens (256-bit minimum) |
LOG_LEVEL | info | No | Log level: debug, info, warn, error |
PORT | Service-specific | No | HTTP server port (see per-service defaults below) |
JWT_SECRET has no default and must be explicitly set. Use a cryptographically random string of at least 32 characters. All services that validate JWTs must share the same secret.
Ingestion Gateway
Default port: 8080
| Variable | Default | Required | Description |
|---|---|---|---|
PORT | 8080 | No | HTTP server port |
PIPELINE_KEY_CACHE_TTL | 300 | No | Pipeline key cache TTL in seconds. Keys are cached in Redis to avoid repeated database lookups |
COOKIE_DOMAIN | — | No | Domain for the _dfid cookie (e.g., .example.com). If unset, uses the request host |
CORS_ORIGINS | * | No | Comma-separated list of allowed origins for CORS. Use * for development only |
MAX_BATCH_SIZE | 500 | No | Maximum number of events in a single batch request |
MAX_PAYLOAD_SIZE | 524288 | No | Maximum request body size in bytes (default 512 KB) |
RATE_LIMIT_RPS | 1000 | No | Maximum requests per second per pipeline key |
In production, always set CORS_ORIGINS to the specific domains that should be allowed to send events. Using * disables CORS protection.
Event Processor
Default port: 8081
| Variable | Default | Required | Description |
|---|---|---|---|
PORT | 8081 | No | HTTP server port (health and metrics endpoints) |
KAFKA_CONSUMER_GROUP | event-processor | No | Kafka consumer group ID |
PROCESSING_WORKERS | 4 | No | Number of concurrent event processing goroutines |
KAFKA_INPUT_TOPIC | raw-events | No | Kafka topic to consume raw events from |
KAFKA_OUTPUT_TOPIC_PREFIX | delivery- | No | Prefix for per-integration delivery topics |
Delivery Workers
Default port: 8082
| Variable | Default | Required | Description |
|---|---|---|---|
PORT | 8082 | No | HTTP server port (health and metrics endpoints) |
KAFKA_CONSUMER_GROUP | delivery-workers | No | Kafka consumer group ID |
RETRY_MAX_ATTEMPTS | 5 | No | Maximum number of retry attempts for failed deliveries |
RETRY_BACKOFF_BASE | 4 | No | Base for exponential backoff in seconds (delay = base^attempt) |
RATE_LIMIT_PER_SECOND | 100 | No | Default rate limit for vendor API requests (per integration) |
DLQ_TOPIC_PREFIX | dlq-delivery- | No | Prefix for dead letter queue topics |
DLQ_RETENTION_DAYS | 7 | No | Days to retain events in the dead letter queue |
Identity Hub
Default port: 8083
| Variable | Default | Required | Description |
|---|---|---|---|
PORT | 8083 | No | HTTP server port |
ENCRYPTION_KEY | — | Yes | 32-byte hex-encoded AES-256 key for encrypting cross-domain identity tokens and vendor credentials |
TOKEN_TTL | 60 | No | Cross-domain identity token TTL in seconds |
IDENTITY_CACHE_TTL | 3600 | No | Identity mapping cache TTL in seconds (Redis) |
ENCRYPTION_KEY has no default and must be explicitly set. It must be exactly 64 hex characters (32 bytes). Losing this key makes all encrypted data (vendor credentials, identity tokens) unrecoverable.
Management API
Default port: 8084
| Variable | Default | Required | Description |
|---|---|---|---|
PORT | 8084 | No | HTTP server port |
ACCESS_TOKEN_TTL | 900 | No | Access token time-to-live in seconds (default 15 minutes) |
REFRESH_TOKEN_TTL | 604800 | No | Refresh token time-to-live in seconds (default 7 days) |
RATE_LIMIT_PER_MINUTE | 100 | No | API rate limit per authenticated user per minute |
LOGIN_MAX_ATTEMPTS | 5 | No | Maximum consecutive failed login attempts before account lockout |
LOGIN_LOCKOUT_DURATION | 900 | No | Account lockout duration in seconds after max failed attempts |
Management UI
Default port: 3000
The Management UI is a Next.js application with its own environment variables:
| Variable | Default | Required | Description |
|---|---|---|---|
PORT | 3000 | No | HTTP server port |
NEXT_PUBLIC_API_URL | http://localhost:8084 | Yes | Management API URL (used by the browser) |
NEXT_PUBLIC_WS_URL | ws://localhost:8084 | No | WebSocket URL for real-time event debugger |
Example .env File (Local Development)
# Common
KAFKA_BROKERS=localhost:9092
REDIS_URL=redis://localhost:6379
DATABASE_URL=postgresql://datafly:datafly@localhost:5432/datafly?sslmode=disable
JWT_SECRET=your-local-dev-jwt-secret-at-least-32-chars
LOG_LEVEL=debug
# Ingestion Gateway
CORS_ORIGINS=http://localhost:3000,http://localhost:5173
COOKIE_DOMAIN=localhost
# Identity Hub
ENCRYPTION_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
TOKEN_TTL=60
# Management API
ACCESS_TOKEN_TTL=900
REFRESH_TOKEN_TTL=604800
# Management UI
NEXT_PUBLIC_API_URL=http://localhost:8084
NEXT_PUBLIC_WS_URL=ws://localhost:8084Never commit .env files containing secrets to version control. The example above uses placeholder values suitable only for local development.