Azure Deployment

This guide walks through deploying Datafly Signal on Microsoft Azure using AKS, Event Hubs (Kafka-compatible), Azure Database for PostgreSQL, and Azure Cache for Redis.

Prerequisites

Tool	Version	Purpose
Azure CLI	v2.55+	Azure account access
kubectl	v1.28+	Kubernetes management
Helm	v3.14+	Chart installation

az login
az account set --subscription YOUR_SUBSCRIPTION_ID

Architecture

                    Internet
                       │
                ┌──────▼──────┐
                │ nginx/AGIC  │
                │  (Ingress)  │
                └──────┬──────┘
                       │
              ┌────────▼────────┐
              │   Azure AKS     │
              │  ┌────────────┐ │
              │  │ Datafly    │ │
              │  │ Services   │ │
              │  └──────┬─────┘ │
              └─────────┼───────┘
           ┌────────────┼────────────┐
           │            │            │
    ┌──────▼──────┐ ┌───▼───┐ ┌─────▼─────┐
    │ Event Hubs  │ │Azure  │ │Azure Cache│
    │ (Kafka API) │ │DB for │ │for Redis  │
    │             │ │PgSQL  │ │           │
    └─────────────┘ └───────┘ └───────────┘

Step 1: Create the AKS Cluster

Choose a VM size based on your sizing tier:

Tier	VM Size	Node Count
Small	Standard_D4s_v5	3
Medium	Standard_D4s_v5	3
Large	Standard_D8s_v5	5
XL	Standard_D16s_v5	8

# Create a resource group
az group create --name datafly-rg --location westeurope
 
# Create the AKS cluster
az aks create \
  --resource-group datafly-rg \
  --name datafly-cluster \
  --node-count 3 \
  --node-vm-size Standard_D4s_v5 \
  --kubernetes-version 1.30 \
  --enable-managed-identity \
  --network-plugin azure \
  --generate-ssh-keys

Get credentials:

az aks get-credentials --resource-group datafly-rg --name datafly-cluster

Install nginx Ingress Controller

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz

Alternatively, use the Application Gateway Ingress Controller (AGIC) for native Azure integration. Set ingress.className: azure-application-gateway in the Helm values.

Step 2: Provision Managed Services

Azure Event Hubs (Kafka-Compatible)

Event Hubs provides a Kafka-compatible endpoint. Create a namespace with the Kafka protocol enabled:

az eventhubs namespace create \
  --name datafly-eventhubs \
  --resource-group datafly-rg \
  --location westeurope \
  --sku Standard \
  --enable-kafka true \
  --capacity 2
 
# Create the required topics (Event Hubs entities)
az eventhubs eventhub create --name events.raw \
  --namespace-name datafly-eventhubs --resource-group datafly-rg \
  --partition-count 6 --message-retention 7
 
az eventhubs eventhub create --name events.processed \
  --namespace-name datafly-eventhubs --resource-group datafly-rg \
  --partition-count 6 --message-retention 7
 
az eventhubs eventhub create --name events.delivery.webhook \
  --namespace-name datafly-eventhubs --resource-group datafly-rg \
  --partition-count 6 --message-retention 7
 
az eventhubs eventhub create --name events.dlq \
  --namespace-name datafly-eventhubs --resource-group datafly-rg \
  --partition-count 3 --message-retention 14

Get the connection string:

az eventhubs namespace authorization-rule keys list \
  --resource-group datafly-rg \
  --namespace-name datafly-eventhubs \
  --name RootManageSharedAccessKey \
  --query primaryConnectionString -o tsv

⚠️

Event Hubs uses SASL_SSL for Kafka connections. Set KAFKA_SECURITY_PROTOCOL=SASL_SSL and KAFKA_SASL_MECHANISM=PLAIN in your environment configuration.

Azure Database for PostgreSQL

az postgres flexible-server create \
  --resource-group datafly-rg \
  --name datafly-postgres \
  --location westeurope \
  --admin-user datafly \
  --admin-password "your-db-password" \
  --sku-name Standard_B2s \
  --tier Burstable \
  --storage-size 32 \
  --version 16 \
  --yes
 
az postgres flexible-server db create \
  --resource-group datafly-rg \
  --server-name datafly-postgres \
  --database-name datafly

Azure Cache for Redis

az redis create \
  --resource-group datafly-rg \
  --name datafly-redis \
  --location westeurope \
  --sku Basic \
  --vm-size c0 \
  --enable-non-ssl-port false \
  --minimum-tls-version 1.2

Get the Redis connection details:

az redis show --name datafly-redis --resource-group datafly-rg --query hostName -o tsv
az redis list-keys --name datafly-redis --resource-group datafly-rg --query primaryKey -o tsv

Step 3: Configure Secrets

Option A: Azure Key Vault (Recommended)

az keyvault create \
  --name datafly-kv \
  --resource-group datafly-rg \
  --location westeurope
 
az keyvault secret set --vault-name datafly-kv \
  --name database-url \
  --value "postgresql://datafly:password@datafly-postgres.postgres.database.azure.com:5432/datafly?sslmode=require"
 
az keyvault secret set --vault-name datafly-kv \
  --name jwt-secret \
  --value "$(openssl rand -hex 32)"
 
az keyvault secret set --vault-name datafly-kv \
  --name encryption-key \
  --value "$(openssl rand -hex 32)"
 
az keyvault secret set --vault-name datafly-kv \
  --name hmac-secret \
  --value "$(openssl rand -hex 32)"
 
az keyvault secret set --vault-name datafly-kv \
  --name licence-key \
  --value "lic_your_licence_key"

Install the External Secrets Operator and configure it for Azure Key Vault:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: azure-key-vault
spec:
  provider:
    azurekv:
      tenantId: YOUR_TENANT_ID
      vaultUrl: https://datafly-kv.vault.azure.net
      authType: ManagedIdentity

Option B: Kubernetes Secrets

kubectl create namespace datafly
 
kubectl create secret generic datafly-secrets \
  --namespace datafly \
  --from-literal=DATABASE_URL="postgresql://datafly:password@datafly-postgres.postgres.database.azure.com:5432/datafly?sslmode=require" \
  --from-literal=REDIS_URL="rediss://:access-key@datafly-redis.redis.cache.windows.net:6380" \
  --from-literal=KAFKA_BROKERS="datafly-eventhubs.servicebus.windows.net:9093" \
  --from-literal=KAFKA_SECURITY_PROTOCOL="SASL_SSL" \
  --from-literal=KAFKA_SASL_MECHANISM="PLAIN" \
  --from-literal=JWT_SECRET="$(openssl rand -hex 32)" \
  --from-literal=ENCRYPTION_KEY="$(openssl rand -hex 32)" \
  --from-literal=HMAC_SECRET="$(openssl rand -hex 32)" \
  --from-literal=DATAFLY_LICENCE_KEY="lic_your_licence_key"
 
kubectl create configmap datafly-config \
  --namespace datafly \
  --from-literal=ENVIRONMENT="prod" \
  --from-literal=LOG_LEVEL="info"

Step 4: Configure DNS and TLS

Azure DNS

# Get the ingress controller's external IP
kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
 
# Create DNS records in Azure DNS (or your DNS provider):
az network dns record-set a add-record \
  --resource-group dns-rg --zone-name yourdomain.com \
  --record-set-name data --ipv4-address EXTERNAL_IP
 
az network dns record-set a add-record \
  --resource-group dns-rg --zone-name yourdomain.com \
  --record-set-name app --ipv4-address EXTERNAL_IP
 
az network dns record-set a add-record \
  --resource-group dns-rg --zone-name yourdomain.com \
  --record-set-name api --ipv4-address EXTERNAL_IP

TLS with cert-manager

helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true

Step 5: Install Datafly Signal

helm install datafly oci://ghcr.io/datafly/charts/datafly \
  --namespace datafly --create-namespace \
  --values values-azure.yaml \
  --set licenceKey=lic_your_licence_key

Key values to customise in values-azure.yaml:

ingress:
  className: nginx
  hosts:
    - host: data.yourdomain.com
      paths:
        - path: /v1
          pathType: Prefix
          service: ingestion-gateway
          port: 8080
        - path: /d.js
          pathType: Exact
          service: ingestion-gateway
          port: 8080
    - host: app.yourdomain.com
      paths:
        - path: /
          pathType: Prefix
          service: management-ui
          port: 3000
    - host: api.yourdomain.com
      paths:
        - path: /
          pathType: Prefix
          service: management-api
          port: 8083
 
externalSecrets:
  enabled: true
  provider: azure
  secretStore: azure-key-vault
  keys:
    databaseUrl: database-url
    jwtSecret: jwt-secret
    encryptionKey: encryption-key
    hmacSecret: hmac-secret
    licenceKey: licence-key

Step 6: Verify the Deployment

Check pod status

kubectl get pods -n datafly

Test event ingestion

curl -X POST https://data.yourdomain.com/v1/t \
  -H "Content-Type: application/json" \
  -d '{"type":"track","event":"Test Event","properties":{"source":"deployment-test"}}'

Access the Management UI

Open https://app.yourdomain.com in your browser.

Cost Estimate (Small Tier)

Service	Spec	Monthly Cost
AKS (free tier)	—	$0
VMs (3x Standard_D4s_v5)	On-Demand	~$420
Event Hubs (Standard, 2 TU)	—	~$160
Azure DB for PostgreSQL (B2s)	—	~$50
Azure Cache for Redis (Basic C0)	—	~$15
Load Balancer	—	~$25
Total		~$670/mo

Azure pricing varies by region. Reserved instances can reduce VM costs by 30-72%. Use the Sizing Calculator for detailed estimates.

Next Steps

Set up Observability for monitoring
Review Upgrades for the upgrade process
Configure Backup & DR for disaster recovery
Check Troubleshooting if you encounter issues

GCP Observability