Installation

This page covers system requirements, infrastructure setup, and building every component from source.

System Requirements

macOS (Homebrew)

brew install go node@20 golang-migrate postgresql@16 golangci-lint
brew install --cask docker

Linux (Ubuntu/Debian)

# Go
wget https://go.dev/dl/go1.25.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.25.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
 
# Node.js 20
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs
 
# golang-migrate
curl -L https://github.com/golang-migrate/migrate/releases/download/v4.17.0/migrate.linux-amd64.tar.gz | tar xvz
sudo mv migrate /usr/local/bin/
 
# PostgreSQL client
sudo apt-get install -y postgresql-client
 
# Docker
# Follow https://docs.docker.com/engine/install/ubuntu/

Infrastructure Setup (Docker Compose)

The Docker Compose file at deployments/docker-compose.yml provides the full infrastructure stack for local development.

Start Infrastructure

cd application
make docker-up

This starts:

Verify Health

docker compose -f deployments/docker-compose.yml ps

All services should show healthy status. Kafka typically takes 30-45 seconds to become healthy after starting.

Stop Infrastructure

make docker-down

To stop and remove all data volumes (clean slate):

docker compose -f deployments/docker-compose.yml down -v

Database Setup

Run Migrations

Apply all pending migrations to create the schema:

make migrate-up

This connects to PostgreSQL at localhost:5432 with the default credentials (datafly/datafly) and applies every SQL migration in database/migrations/.

Roll Back

To roll back the most recent migration:

make migrate-down

Create a New Migration

make migrate-create name=create_my_table

This generates a pair of files in database/migrations/: NNNNNN_create_my_table.up.sql and NNNNNN_create_my_table.down.sql.

Seed Development Data

make seed

The seed script (database/seeds/dev_seed.sql) inserts a sample organisation, admin user, source, and integrations suitable for local development.

Building from Source

All build commands are run from the application/ directory.

Build All Go Services

make build

This compiles all six Go services and outputs binaries to the bin/ directory:

bin/
  ingestion-gateway
  event-processor
  delivery-workers
  identity-hub
  management-api

Build a Single Service

make build-ingestion-gateway
make build-event-processor
make build-delivery-workers
make build-identity-hub
make build-management-api

Go Module Structure

Each service is a separate Go module with its own go.mod. The shared library at shared/ is referenced via a replace directive:

application/
  shared/              # github.com/datafly/shared
  ingestion-gateway/   # github.com/datafly/ingestion-gateway
  event-processor/     # github.com/datafly/event-processor
  delivery-workers/    # github.com/datafly/delivery-workers
  identity-hub/        # github.com/datafly/identity-hub
  management-api/      # github.com/datafly/management-api

Each service’s go.mod includes:

replace github.com/datafly/shared => ../shared

This allows local development without publishing the shared module. In CI/CD and production Docker builds, the replace directive is removed and replaced with a versioned dependency.

Build the Management UI

make build-ui

This runs npm run build inside management-ui/, producing a Next.js standalone output suitable for Docker deployment.

For local development with hot reload:

cd management-ui
npm install
npm run dev

The UI runs on http://localhost:3000.

Build Datafly.js

make build-datafly-js

This runs the Rollup build inside datafly-js/, producing:

• dist/datafly.umd.js — UMD bundle for script tag usage
• dist/datafly.esm.js — ES module bundle for npm/bundler usage

The built file is under 8KB gzipped (currently ~4.2KB).

Docker Images

Build Docker images for all services:

make docker-build

Images are tagged with the ghcr.io/datafly/ registry prefix by default. Override with:

make docker-build DOCKER_REGISTRY=your-registry.com/datafly IMAGE_TAG=v1.0.0

Running Tests

# All Go services
make test
 
# Lint all Go services
make lint
 
# Format all Go services
make fmt

Environment Variables

All services share a common configuration pattern loaded from environment variables. See the Configuration reference for the full list.

For local development, the defaults connect to the Docker Compose infrastructure with no additional configuration required.

Cleanup

Remove all build artefacts:

make clean

This deletes the bin/ directory, Next.js build caches, and Datafly.js dist output.