Docker Compose

Docker Compose is the recommended way to run Datafly Signal locally during development. It starts the required infrastructure services (Kafka, Redis, PostgreSQL) and provides a Kafka UI for inspecting topics and messages.

Prerequisites

Tool	Minimum Version
Docker	24+
Docker Compose	v2
Go	1.25
Node.js	20 LTS
golang-migrate	v4
psql	16+

Starting Infrastructure

From the Application/ directory, start all infrastructure services:

make docker-up

This starts:

Service	Port	Description
PostgreSQL	5432	Database (db=`datafly`, user=`datafly`, password=`datafly`)
Redis	6379	Cache and session store
Kafka	9092	Message broker
Kafka UI	8090	Web UI for inspecting Kafka topics

To stop all infrastructure:

make docker-down

Running Database Migrations

After starting infrastructure for the first time (or after pulling new changes), run migrations:

make migrate-up

To seed the database with development data:

make seed

The seed command creates a default organisation, admin user (admin@datafly.local / datafly), and a sample source with a pipeline key. See the seed SQL files in database/migrations/ for details.

Building and Running Services

Build all Go services

make build

This compiles all Go services into the bin/ directory.

Build a single service

make build-ingestion-gateway
make build-event-processor
make build-delivery-workers
make build-identity-hub
make build-management-api

Run a service

After building, run each service directly. Each service reads configuration from environment variables:

# Terminal 1 - Ingestion Gateway
./bin/ingestion-gateway
 
# Terminal 2 - Event Processor
./bin/event-processor
 
# Terminal 3 - Delivery Workers
./bin/delivery-workers
 
# Terminal 4 - Identity Hub
./bin/identity-hub
 
# Terminal 5 - Management API
./bin/management-api

Default environment variables are configured for local development. Override them as needed (see Environment Variables).

Running the Management UI

make build-ui

Or for development with hot reloading:

cd management-ui && npm install && npm run dev

The UI runs on port 3000 and connects to the Management API on port 8084.

Building Datafly.js

make build-datafly-js

This produces UMD and ESM bundles in datafly-js/dist/.

Port Reference

Service	Port	URL
Ingestion Gateway	8080	`http://localhost:8080`
Event Processor	8081	`http://localhost:8081`
Delivery Workers	8082	`http://localhost:8082`
Identity Hub	8083	`http://localhost:8083`
Management API	8084	`http://localhost:8084`
Management UI	3000	`http://localhost:3000`
PostgreSQL	5432	`postgresql://datafly:datafly@localhost:5432/datafly`
Redis	6379	`redis://localhost:6379`
Kafka	9092	`localhost:9092`
Kafka UI	8090	`http://localhost:8090`

Troubleshooting

Port already in use

If a port is already occupied, identify the process and stop it:

lsof -i :8080
kill -9 <PID>

Or change the service port via the PORT environment variable:

PORT=9080 ./bin/ingestion-gateway

Kafka not ready

Services may fail to start if Kafka is not yet ready. Kafka can take 15-30 seconds to initialise. If you see connection errors, wait and retry:

# Check Kafka health
docker compose logs kafka | tail -20

The Go services use retry logic on startup and will attempt to reconnect to Kafka automatically. If a service exits on startup, simply restart it after Kafka is healthy.

Database connection refused

Ensure PostgreSQL is running and accepting connections:

docker compose ps
psql -h localhost -U datafly -d datafly -c "SELECT 1"

If the database exists but migrations have not been run, you will see errors about missing tables. Run make migrate-up.

Redis connection refused

docker compose ps
redis-cli -h localhost ping

Should return PONG.

Resetting everything

To tear down all infrastructure and delete all data:

make docker-down
docker volume prune -f
make docker-up
make migrate-up
make seed

⚠️

docker volume prune deletes all Docker volumes not currently in use, including PostgreSQL data. Only use this if you want a completely fresh start.

Overview Kubernetes