Integrations

Datafly Signal delivers events server-to-server to vendor APIs. The Integration Library provides 95+ pre-built vendor integrations with one-click setup, industry-specific variants, and full customisation.

How Integrations Work

When you set up an integration, you create an integration template and assign it to one or more pipelines. Events collected by Datafly.js flow through the full pipeline before reaching the vendor:

Datafly.js → Ingestion Gateway → Kafka (raw-events)
  → Event Processor (Org Data Layer → Pipeline Transformations)
  → Kafka (delivery-{integration_id})
  → Delivery Worker → Vendor API

1. The Event Processor applies your Org Data Layer enrichments and per-pipeline transformations.
2. Processed events are published to a dedicated Kafka topic per integration (delivery-{integration_id}).
3. The Delivery Worker for that vendor consumes from the topic, formats the payload to match the vendor’s API specification, and sends it server-to-server.

Each vendor has its own Delivery Worker implementation that understands the target API’s authentication, payload format, rate limits, and error handling.

Integration Library

The Integration Library is a browsable catalog of 95+ pre-built vendor integrations. Each catalog entry includes:

• Configuration fields with labels, types, and help text
• Industry variants (e.g. Retail, Travel, Media) with tuned default configurations
• One-click install that creates a template and first revision automatically

Using the Library (Management UI)

1. Navigate to Integrations in the sidebar.
2. Click the Add Integration button (from My Integrations) or switch to the Integration Library tab.
3. Browse by category or search for a vendor. Each vendor card shows its brand logo for quick recognition.
4. Click Install on the vendor you want.
5. Choose a variant (if available), select a Blueprint for a pre-built configuration (if one exists for your vertical), fill in the required credentials, and click Install Integration.

Blueprints

Blueprints are pre-built V2 schema-mapping configurations for common vendor + vertical combinations. When installing an integration that has Blueprints available, you can select one to start with a ready-made configuration instead of building from scratch.

Available Blueprint verticals include Retail, Travel, and Media, with pre-mapped event types (purchase, add_to_cart, page_view, etc.) and field mappings specific to each vendor’s API format.

All Blueprint mappings are fully editable after install — they serve as a starting point, not a constraint.

Custom Integrations

For vendors not in the library, or for internal APIs, you can create a Custom Integration from the My Integrations tab. Custom integrations support:

• Freeform vendor display name
• Full V2 schema-mapping builder access
• Webhook delivery to any HTTP endpoint

Using the Library (API)

# List the catalog
curl -X GET "http://localhost:8084/v1/admin/integration-catalog?category=advertising" \
  -H "Authorization: Bearer YOUR_TOKEN"
 
# Install from catalog
curl -X POST http://localhost:8084/v1/admin/integration-catalog/meta_capi/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Meta CAPI Production",
    "variant": "retail",
    "config": {
      "pixel_id": "1234567890",
      "access_token": "your_access_token"
    },
    "delivery_mode": "server_side"
  }'

See the Integration Catalog API for full documentation.

Templates & Revisions

Integrations use a template + revision model for versioned configuration management:

• Template: An org-level reusable integration definition (e.g. “Meta CAPI Production”)
• Revision: An immutable configuration snapshot (v1, v2, v3…)
• Pipeline Integration: An assignment of a template at a specific revision to a pipeline

Template "Meta CAPI Production"
  ├── Revision v1 — initial setup
  ├── Revision v2 — updated access token
  └── Revision v3 — increased batch size (current)

Pipeline "Marketing Website" → uses v3
Pipeline "Blog" → uses v2

To change configuration, you create a new revision. This ensures a complete audit trail and allows different pipelines to use different versions.

See the Integration Templates API and Pipeline Integrations API for full documentation.

V2 Schema-Mapping Builder

The integration builder uses a schema-aware point-and-click interface organised into four categories:

Each mapping row maps a source field (from the canonical Datafly event schema) to a vendor field (from the vendor’s API schema), with optional transforms (lowercase, sha256, to_unix_seconds, etc.).

The builder includes:

• Autocomplete for both source and vendor fields from known schemas
• Coverage indicator showing how many required/optional vendor fields are mapped
• Code editor with YAML ↔ visual sync for advanced users
• Preview with dry-run against sample events

The legacy V1 step-based builder is retained for backward compatibility. New integrations default to V2.

Consent Categories

When consent categories are configured on an integration, the Delivery Worker only delivers events where the user has granted consent for all listed categories. Consent state is read from the event’s context.consent object, which Datafly.js populates from your consent management platform.

{
  "context": {
    "consent": {
      "analytics": true,
      "marketing": false,
      "advertising": true
    }
  }
}

In this example, an integration requiring ["analytics", "marketing"] would not receive this event because marketing is false.

Retry Behaviour

When a delivery attempt fails (network error, 5xx response, rate limit hit), the Delivery Worker retries with exponential backoff:

• Maximum retries: 5 attempts (plus the original attempt, for 6 total tries).
• Backoff formula: delay = base^attempt where base = 4 seconds and the first retry uses a 1-second floor.
• Jitter: A random jitter of up to 25% is added to each delay to prevent thundering herd issues.

Dead Letter Queue

After all retry attempts are exhausted, the event is published to a dead letter topic (dlq-delivery-{integration_id}). Dead-lettered events are retained for 7 days and can be:

• Inspected via the Management UI under Integrations > {name} > Dead Letter Queue.
• Replayed manually once the underlying issue is resolved.
• Exported for analysis via the Management API.

Rate Limiting

Each vendor integration has configurable rate limits to avoid exceeding vendor API quotas:

When the rate limit is reached, events are buffered in Kafka and delivered as capacity becomes available. No events are dropped due to rate limiting — delivery is simply delayed.

Supported Integrations

Advertising

Analytics

Attribution

CDP

Cloud Storage

Other