Vendor Identity Sync

The Vendor Identity Sync Engine is the orchestration layer that coordinates all identity resolution methods — self-generation, click ID capture, and server-proxied enrichment — into a unified identity store. It determines which vendor IDs to collect for each source, manages storage and TTLs in Redis, and ensures the correct IDs are attached to events during processing.

Why Passive Cookie Reading Does Not Work

In a traditional client-side tag management setup, each vendor’s JavaScript tag creates its own cookies. A server-side tag manager could theoretically just read those cookies from inbound requests and forward the values to vendor APIs.

With Signal, this approach fails because vendor tags never fire. The Google Analytics script never loads, so the _ga cookie is never created. The Meta pixel never fires, so _fbp never exists. There are no vendor cookies to passively read.

Signal must actively create and manage these identities itself.

Three Sync Methods

The sync engine uses three methods to establish vendor identity, applied in combination:

Each method is independent. A visitor can have self-generated IDs, captured click IDs, and enrichment results all stored simultaneously. The sync engine does not require all three to be present — it uses whatever IDs are available for each vendor.

Per-Source Vendor Module Selection

Not every source needs every vendor’s identity. The sync engine only generates and manages IDs for vendors that have an active integration configured on the source:

Source: "Main Website" (src_abc123)
  Integrations: GA4, Meta CAPI, TikTok Events API

  → Sync engine generates: _ga, _fbp, _ttp
  → Sync engine captures: gclid, fbclid, ttclid
  → No Pinterest, Snapchat, or LinkedIn IDs are generated

This means the Ingestion Gateway only sets cookies that are actually needed, minimising cookie overhead and avoiding unnecessary Redis storage.

When a new integration is added to a source, the sync engine begins generating the relevant vendor ID on the next request from that visitor. Existing visitors receive the new vendor ID on their next page load — there is no need to wait for a new session.

Vendor ID Storage

All vendor IDs are stored in a Redis hash keyed by the visitor’s anonymous ID:

Key:    identity:{anonymous_id}
Type:   Hash

Fields:
  ga_client_id    → "1234567890.1708876543"
  fbp             → "fb.1.1708876543000.9876543210"
  fbc             → "fb.1.1708876543000.IwAR3x..."
  ttp             → "a1b2c3d4e5f6g7h8i9j0k1l2m3n"
  pin_unauth      → "f47ac10b-58cc-4372-a567-0d02b2c3d479"
  scid            → "b82d5c91-3f7a-4e28-9c8f-1a2b3c4d5e6f"
  gclid           → "CjwKCAjw..."
  fbclid          → "IwAR3x..."
  ttclid          → "E.CP..."
  user_id         → "user_98765"
  ramp_id         → "XY1000bGluZWRpbjANCm..."
  uid2_token      → "AgAAAAN..."

Each field in the hash represents a single vendor identity value. This structure allows the Event Processor to retrieve all IDs for a visitor in a single HGETALL call, or fetch specific fields with HMGET when preparing events for individual vendors.

TTL Management

Different vendor IDs have different expected lifetimes. The sync engine manages TTLs at two levels:

Cookie TTL

When a vendor ID is set as a browser cookie, its Max-Age matches the vendor’s expected cookie lifetime:

Redis TTL

Vendor ID fields in Redis use per-field expiry via a companion key pattern:

identity:{anonymous_id}           → Hash with all IDs (TTL = max of any field)
identity:{anonymous_id}:ttl:fbc   → Expiry marker for _fbc (90 days)
identity:{anonymous_id}:ttl:gclid → Expiry marker for gclid (90 days)

A background process periodically cleans up expired fields from the hash. The hash itself persists as long as any field is still valid.

How IDs Flow Through the System

The full lifecycle of a vendor ID, from creation to delivery:

1. COLLECTION (Ingestion Gateway)
   Browser request → Gateway checks for existing cookies
   → Missing vendor IDs are generated (self-generation)
   → URL parameters are scanned for click IDs
   → All IDs are set as cookies (Set-Cookie header)
   → All IDs are stored in Redis hash

2. STORAGE (Redis)
   identity:{anonymous_id} hash holds all vendor IDs
   → Each field has an independent TTL
   → Updated on every request (TTLs refreshed for active visitors)

3. ENRICHMENT (Event Processor)
   Event arrives from Kafka (raw-events topic)
   → Processor reads anonymous_id from the event
   → Looks up identity:{anonymous_id} in Redis
   → For each target integration, attaches the relevant vendor IDs
   → Publishes enriched event to delivery topic

4. DELIVERY (Delivery Workers)
   Delivery Worker consumes from delivery-{integration_id}
   → Formats the vendor ID into the API payload
   → Sends server-to-server to vendor API
   → Vendor matches the ID to its own identity graph

Example: GA4 Delivery

{
  "client_id": "1234567890.1708876543",
  "events": [{
    "name": "purchase",
    "params": {
      "transaction_id": "T-12345",
      "value": 79.99,
      "currency": "USD"
    }
  }]
}

The client_id value 1234567890.1708876543 was self-generated by Signal, set as the _ga cookie on the browser, stored in Redis, retrieved during processing, and included in the Measurement Protocol payload. GA4 accepts it as a valid client ID and can stitch this server-sent event with any client-side GA4 data (if the standard GA4 tag also runs), or use it as a standalone identity for pure server-side implementations.

Sync Timing

Identity sync operations happen at different points depending on the method:

Self-generation and click ID capture are synchronous operations that happen during request handling. The generated IDs are immediately available in the event payload and in Redis. Server-proxied enrichment is asynchronous — the enrichment request is made after the initial event is processed, and the result is available for subsequent events.