Kubernetes

Datafly Signal is deployed to production using Kubernetes with Helm charts. Each customer deployment runs in its own namespace with dedicated service instances, providing complete data isolation.

Helm Charts

Helm charts are located at deployments/helm/datafly/ in the repository. The chart deploys all six Datafly services plus the Management UI.

Install

helm install datafly ./deployments/helm/datafly \
  --namespace customer-acme \
  --create-namespace \
  --values values-acme.yaml

Upgrade

helm upgrade datafly ./deployments/helm/datafly \
  --namespace customer-acme \
  --values values-acme.yaml

Uninstall

helm uninstall datafly --namespace customer-acme

⚠️

Always run database migrations before upgrading services. The Helm chart does not run migrations automatically.

Namespace-Per-Customer

Each customer gets a dedicated Kubernetes namespace:

namespace: customer-acme
  ├── ingestion-gateway (Deployment)
  ├── event-processor (Deployment)
  ├── delivery-workers (Deployment)
  ├── identity-hub (Deployment)
  ├── management-api (Deployment)
  ├── management-ui (Deployment)
  ├── ingestion-gateway-svc (Service)
  ├── management-api-svc (Service)
  ├── management-ui-svc (Service)
  └── ingress (Ingress)

This model ensures:

Data isolation — no shared Kafka topics, database schemas, or Redis keyspaces between customers.
Independent scaling — each customer’s workload scales independently.
Independent upgrades — customers can be upgraded on different schedules.
Blast radius containment — issues in one deployment do not affect others.

Services

Service	Replicas (default)	Resource Requests	Resource Limits
Ingestion Gateway	2	256Mi / 250m	512Mi / 500m
Event Processor	2	512Mi / 500m	1Gi / 1000m
Delivery Workers	2	256Mi / 250m	512Mi / 500m
Identity Hub	1	128Mi / 100m	256Mi / 250m
Management API	1	256Mi / 250m	512Mi / 500m
Management UI	1	128Mi / 100m	256Mi / 250m

All values are configurable in the Helm values file.

Infrastructure

The Kubernetes deployment expects managed infrastructure services provisioned outside the Helm chart:

Service	Recommended Provider	Notes
Kafka	AWS MSK, Confluent Cloud	Dedicated cluster or shared with topic isolation
Redis	AWS ElastiCache, GCP Memorystore	Single node sufficient for most deployments
PostgreSQL	AWS RDS, GCP Cloud SQL	PostgreSQL 16+, dedicated database per customer

Connection details are provided via Helm values or Kubernetes secrets.

Ingress

The Ingestion Gateway must be accessible from the public internet so browsers can send events. A typical setup uses a customer subdomain pointing to the cluster’s ingress controller:

data.customer.com → Ingress Controller → ingestion-gateway-svc:8080
app.customer.com  → Ingress Controller → management-ui-svc:3000
api.customer.com  → Ingress Controller → management-api-svc:8084

The Helm chart includes an Ingress resource. Configure the hostname and TLS certificate in the values file.

ingress:
  enabled: true
  className: nginx
  hosts:
    - host: data.acme.com
      paths:
        - path: /v1
          service: ingestion-gateway
          port: 8080
        - path: /d.js
          service: ingestion-gateway
          port: 8080
    - host: app.acme.com
      paths:
        - path: /
          service: management-ui
          port: 3000
    - host: api.acme.com
      paths:
        - path: /
          service: management-api
          port: 8084
  tls:
    - secretName: acme-tls
      hosts:
        - data.acme.com
        - app.acme.com
        - api.acme.com

Horizontal Pod Autoscaling

The Helm chart includes HorizontalPodAutoscaler (HPA) resources for the Ingestion Gateway, Event Processor, and Delivery Workers. Autoscaling is based on CPU utilisation and Kafka consumer lag.

autoscaling:
  ingestionGateway:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
  eventProcessor:
    enabled: true
    minReplicas: 2
    maxReplicas: 8
    targetCPUUtilizationPercentage: 70
    # Custom metric: Kafka consumer lag
    kafkaLagThreshold: 10000
  deliveryWorkers:
    enabled: true
    minReplicas: 2
    maxReplicas: 12
    targetCPUUtilizationPercentage: 70

Kafka consumer lag-based autoscaling requires a metrics adapter such as KEDA or a Prometheus-based custom metrics pipeline. The Helm chart supports both approaches via the autoscaling.metricsProvider setting.

Example values.yaml

# values-acme.yaml
global:
  namespace: customer-acme
  image:
    registry: ghcr.io/datafly
    tag: v1.2.0
    pullPolicy: IfNotPresent
 
kafka:
  brokers: "b-1.msk-cluster.amazonaws.com:9092,b-2.msk-cluster.amazonaws.com:9092"
 
redis:
  url: "redis://acme-redis.xxxxx.cache.amazonaws.com:6379"
 
database:
  url: "postgresql://datafly:secret@acme-db.xxxxx.rds.amazonaws.com:5432/datafly?sslmode=require"
 
jwt:
  secret: "your-256-bit-jwt-secret"
 
ingestionGateway:
  replicas: 2
  env:
    CORS_ORIGINS: "https://www.acme.com,https://shop.acme.com"
    COOKIE_DOMAIN: ".acme.com"
 
eventProcessor:
  replicas: 2
 
deliveryWorkers:
  replicas: 3
 
identityHub:
  replicas: 1
  env:
    ENCRYPTION_KEY: "your-32-byte-encryption-key-hex"
 
managementApi:
  replicas: 1
 
managementUi:
  replicas: 1
 
ingress:
  enabled: true
  className: nginx
  hosts:
    - host: data.acme.com
      paths:
        - path: /
          service: ingestion-gateway
          port: 8080
    - host: app.acme.com
      paths:
        - path: /
          service: management-ui
          port: 3000
    - host: api.acme.com
      paths:
        - path: /
          service: management-api
          port: 8084
  tls:
    - secretName: acme-tls
      hosts:
        - data.acme.com
        - app.acme.com
        - api.acme.com
 
autoscaling:
  ingestionGateway:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
  eventProcessor:
    enabled: true
    minReplicas: 2
    maxReplicas: 8
    targetCPUUtilizationPercentage: 70
  deliveryWorkers:
    enabled: true
    minReplicas: 3
    maxReplicas: 12
    targetCPUUtilizationPercentage: 70

Secrets Management

Sensitive values (database passwords, JWT secrets, encryption keys, vendor API credentials) should be stored in Kubernetes Secrets rather than in the values file:

kubectl create secret generic datafly-secrets \
  --namespace customer-acme \
  --from-literal=DATABASE_URL="postgresql://datafly:secret@..." \
  --from-literal=JWT_SECRET="your-jwt-secret" \
  --from-literal=ENCRYPTION_KEY="your-encryption-key"

Reference the secret in the values file:

global:
  existingSecret: datafly-secrets

⚠️

Never commit secrets to version control. Use a secrets manager (AWS Secrets Manager, HashiCorp Vault, or Kubernetes External Secrets) for production deployments.

Docker Compose Customer-Hosted