Pipelines

Pipelines are the primary data collection and delivery units in Datafly Signal. Each pipeline has its own pipeline key, domain allowlist, rate limits, JS collector configuration, and attached integrations.

Pipeline Object

{
  "id": "pip_abc123",
  "org_id": "org_xyz",
  "name": "Marketing Website",
  "slug": "marketing-website",
  "type": "web",
  "pipeline_key": "dk_pip_abc123",
  "pipeline_key_secret_prefix": "sk_a1b2",
  "domain_allowlist": ["www.example.com", "example.com"],
  "labels": ["env:production", "brand:marketing"],
  "rate_limit_rps": 1000,
  "challenge_token_required": false,
  "parameters": {
    "ga4_measurement_id": "G-XXXXXXX",
    "ga4_api_secret": "xxxxxxxxxxx"
  },
  "config": {},
  "js_builder_config": {
    "script_filename": "d.js",
    "global_variable": "_df",
    "delivery_method": "cdn"
  },
  "identity_config": {
    "cross_domain_enabled": false,
    "cross_domain_domains": []
  },
  "enabled": true,
  "created_at": "2026-02-20T10:00:00Z",
  "updated_at": "2026-02-25T14:30:00Z"
}

Field	Type	Description
`id`	string	Unique pipeline identifier
`org_id`	string	Organisation identifier
`name`	string	Display name
`slug`	string	URL-safe slug (auto-generated from name if not provided)
`type`	string	Pipeline type: `web`, `app`, or `server`
`pipeline_key`	string	Public pipeline key used by Datafly.js or the server-side API
`pipeline_key_secret_prefix`	string	First 4 characters of the pipeline key secret (for identification)
`domain_allowlist`	string[]	Domains permitted to send events (empty = allow all)
`labels`	string[]	Labels for filtering and grouping. Use `prefix:value` format for groupable labels (e.g. `env:production`, `country:uk`, `brand:acme`). Labels without colons are plain tags.
`rate_limit_rps`	integer	Maximum events per second (default 1000)
`challenge_token_required`	boolean	Whether events must include a valid challenge token
`parameters`	object	Key-value string pairs used as pipeline-scoped variables (see Pipeline Parameters)
`config`	object	Pipeline-specific configuration
`js_builder_config`	object	JS collector build configuration (see below)
`identity_config`	object	Cross-domain identity configuration (see below)
`enabled`	boolean	Whether the pipeline is accepting events
`created_at`	string	ISO 8601 creation timestamp
`updated_at`	string	ISO 8601 last-updated timestamp

JS Builder Configuration

Field	Type	Default	Description
`script_filename`	string	`d.js`	Filename for the generated JS collector
`global_variable`	string	`_df`	Global variable name exposed by the collector
`delivery_method`	string	`cdn`	How the script is served: `download`, `cdn`, or `cluster`
`vendor_syncs`	object	`{}`	Vendor-specific sync configuration
`consent_config`	object	`{}`	Consent management integration settings
`cross_domain`	boolean	`false`	Enable cross-domain tracking in the collector

Identity Configuration

Field	Type	Default	Description
`cross_domain_enabled`	boolean	`false`	Enable cross-domain identity stitching
`cross_domain_domains`	string[]	`[]`	Domains to share identity across
`vendor_sync`	object	`{}`	Per-vendor identity sync settings

Pipeline Parameters

Pipeline parameters are key-value string pairs stored on a pipeline. They allow a single integration template to be reused across multiple pipelines, with each pipeline supplying its own values for vendor credentials, measurement IDs, or other configuration.

Integration revision configs can reference pipeline parameters using the {{pipeline_param.KEY}} syntax. For example, a GA4 integration revision might set measurement_id to {{pipeline_param.ga4_measurement_id}}. When the event processor loads the config (every 5 seconds), it substitutes each placeholder with the corresponding value from the pipeline’s parameters object.

{
  "parameters": {
    "ga4_measurement_id": "G-XXXXXXX",
    "ga4_api_secret": "xxxxxxxxxxx"
  }
}

Parameters are resolved at config load time (every 5s poll), not per-event. If a referenced parameter is missing from the pipeline, the placeholder is left intact and a warning is logged.

Label Convention

Labels use a prefix:value format to enable grouping in the Management UI:

Label	Prefix	Value
`env:production`	`env`	`production`
`country:uk`	`country`	`uk`
`brand:acme`	`brand`	`acme`
`team:checkout`	`team`	`checkout`

The Management UI’s grouping selector discovers available prefixes from available_labels and allows users to group pipelines by any prefix dimension (e.g. group by env shows Production, Test, Dev sections). Labels without colons are treated as plain tags.

A pipeline can have multiple labels with the same prefix (e.g. brand:acme and brand:widgets). In grouped view, the pipeline will appear in both groups.

List Pipelines

`GET /v1/admin/pipelines`

Returns a paginated list of pipelines for the current organisation.

Query parameters:

Parameter	Type	Default	Description
`search`	string	—	Search by name or slug (case-insensitive)
`type`	string	—	Filter by pipeline type (`web`, `app`, `server`)
`label`	string	—	Filter by label (pipelines containing this label)
`enabled`	boolean	—	Filter by enabled status
`limit`	integer	50	Items per page (max 100)
`offset`	integer	0	Offset for pagination

Request:

curl -X GET "http://localhost:8084/v1/admin/pipelines?type=web&label=env:production" \
  -H "Authorization: Bearer {access_token}"

Response:

{
  "pipelines": [
    {
      "id": "pip_abc123",
      "name": "Marketing Website",
      "slug": "marketing-website",
      "type": "web",
      "pipeline_key": "dk_pip_abc123",
      "labels": ["env:production", "brand:marketing"],
      "enabled": true,
      "created_at": "2026-02-20T10:00:00Z",
      "updated_at": "2026-02-25T14:30:00Z"
    }
  ],
  "total": 1,
  "limit": 50,
  "offset": 0
}

Required role: viewer or above.

Create Pipeline

`POST /v1/admin/pipelines`

Create a new pipeline. A pipeline key and secret are generated automatically.

Request:

{
  "name": "Marketing Website",
  "slug": "marketing-website",
  "type": "web",
  "domain_allowlist": ["www.example.com", "example.com"],
  "labels": ["env:production", "brand:marketing"],
  "rate_limit_rps": 1000,
  "parameters": {
    "ga4_measurement_id": "G-XXXXXXX",
    "ga4_api_secret": "xxxxxxxxxxx"
  },
  "js_builder_config": {
    "script_filename": "d.js",
    "global_variable": "_df"
  }
}

Response (201 Created):

{
  "id": "pip_abc123",
  "org_id": "org_xyz",
  "name": "Marketing Website",
  "slug": "marketing-website",
  "type": "web",
  "pipeline_key": "dk_pip_abc123",
  "pipeline_key_secret": "sk_a1b2c3d4e5f6g7h8i9j0...",
  "pipeline_key_secret_prefix": "sk_a1b2",
  "domain_allowlist": ["www.example.com", "example.com"],
  "labels": ["env:production", "brand:marketing"],
  "rate_limit_rps": 1000,
  "enabled": true,
  "created_at": "2026-02-25T10:00:00Z",
  "updated_at": "2026-02-25T10:00:00Z"
}

⚠️

The pipeline_key_secret is only returned once on creation (and on secret rotation). Store it securely. It cannot be retrieved again.

Required role: editor or above.

Get Pipeline

`GET /v1/admin/pipelines/{id}`

Retrieve a single pipeline by ID.

Request:

curl -X GET http://localhost:8084/v1/admin/pipelines/pip_abc123 \
  -H "Authorization: Bearer {access_token}"

Response: Full pipeline object (without pipeline_key_secret).

Required role: viewer or above.

Update Pipeline

`PUT /v1/admin/pipelines/{id}`

Update an existing pipeline. Only include the fields you want to change.

Request:

{
  "name": "Marketing Website (Updated)",
  "domain_allowlist": ["www.example.com", "example.com", "blog.example.com"],
  "labels": ["env:production", "brand:marketing", "site:blog"],
  "parameters": {
    "ga4_measurement_id": "G-NEWIDXX",
    "ga4_api_secret": "new_secret_value"
  },
  "enabled": false
}

Response: Updated pipeline object.

Required role: editor or above.

Delete Pipeline

`DELETE /v1/admin/pipelines/{id}`

Soft-delete a pipeline. Events will no longer be accepted on this pipeline’s pipeline key.

Request:

curl -X DELETE http://localhost:8084/v1/admin/pipelines/pip_abc123 \
  -H "Authorization: Bearer {access_token}"

Response:

{
  "status": "deleted"
}

⚠️

Deleting a pipeline stops event ingestion immediately. Any integrations assigned to this pipeline will stop receiving events. This is a soft delete — the pipeline can be recovered by contacting support.

Required role: admin or above.

Rotate Pipeline Key Secret

`POST /v1/admin/pipelines/{id}/rotate-secret`

Generate a new pipeline key secret. The old secret is immediately invalidated.

Request:

curl -X POST http://localhost:8084/v1/admin/pipelines/pip_abc123/rotate-secret \
  -H "Authorization: Bearer {access_token}"

Response:

{
  "pipeline_key": "dk_pip_abc123",
  "pipeline_key_secret": "sk_new_secret_value...",
  "pipeline_key_secret_prefix": "sk_x9y8"
}

⚠️

Rotating the secret immediately invalidates the old one. Update all clients using HMAC authentication before rotating.

Required role: admin or above.

Generate Challenge Token

`POST /v1/admin/pipelines/{id}/tokens`

Generate a short-lived challenge token for pipelines with challenge_token_required enabled.

Request:

curl -X POST http://localhost:8084/v1/admin/pipelines/pip_abc123/tokens \
  -H "Authorization: Bearer {access_token}"

Response:

{
  "token": "ct_abc123def456...",
  "expires_at": "2026-02-25T11:00:00Z"
}

Challenge tokens are required when challenge_token_required is set to true on the pipeline. Datafly.js obtains these tokens automatically when configured.

Required role: editor or above.

Collector Build

`POST /v1/admin/pipelines/{id}/collector/build`

Trigger a build of the Datafly.js collector for this pipeline. The build includes all vendor syncs and consent configuration specified in js_builder_config.

Request:

curl -X POST http://localhost:8084/v1/admin/pipelines/pip_abc123/collector/build \
  -H "Authorization: Bearer {access_token}"

Response:

{
  "status": "building",
  "pipeline_id": "pip_abc123"
}

`GET /v1/admin/pipelines/{id}/collector/status`

Check the build status of the collector.

Response:

{
  "status": "ready",
  "pipeline_id": "pip_abc123",
  "build_id": "bld_xyz789",
  "version_hash": "a1b2c3d4",
  "file_size_bytes": 4200,
  "built_at": "2026-02-25T10:05:00Z"
}

Status	Description
`no_builds`	No collector has been built yet
`building`	Build is in progress
`ready`	Build complete and available for download

`GET /v1/admin/pipelines/{id}/collector/download`

Download the built collector JavaScript file.

Response: JavaScript file content (application/javascript).

Required role: viewer or above (for status/download), editor or above (for triggering builds).

Authentication Integration Catalog