Management APIDead Letter Queue

Dead Letter Queue

When a delivery to a destination fails, Datafly Signal keeps the event and the failure in the dead letter queue (DLQ). Transient failures are retried automatically on a backoff schedule (status retrying). When the schedule is exhausted, or the failure is permanent, the event waits for an operator (status pending). Nothing expires automatically.

Each DLQ entry is one event and one destination. If an event failed to two destinations, it has two entries, and replaying one sends the event only to that destination.

An event the processor could not process never reaches this queue: it is held as a processor dead letter and replayed with the event processor’s dlq-replay command.

Statuses

StatusMeaning
retryingOn the automatic retry schedule, or queued for an operator replay
pendingNeeds an operator: replay or discard
replayingA single replay is in flight
resolvedDelivered on retry or replay
discardedDiscarded by an operator

When an entry is not retried

An entry returns to pending without being sent, whatever its retry schedule, when it can never be delivered as stored. Its error code says why, and the original failure is kept at the end of the error message.

Error codeMeaning
integration_removedThe destination is no longer in its pipeline’s published version, or was deleted
replay_no_destinationThe entry names no destination, for example a message that could not be read
replay_unparseableThe stored event cannot be read
replay_no_eventThe entry has no stored event

An entry whose destination is switched off, whose pipeline is paused, or whose connection settings cannot be read stays on the retry schedule and is delivered once that is put right. It goes to pending when the schedule runs out. An entry is never marked resolved unless it was delivered.

While a destination is backlogged behind its rate limit, a retry that is skipped without being attempted does not use up the schedule: it waits and keeps its place. It waits at most as long as the whole schedule would have taken (about 31 hours by default), or the destination’s retry window if it sets one, then goes to pending.

Alerting

The delivery workers export these Prometheus gauges every minute, from one instance at a time. Alert on max() across instances: if the instances cannot agree which one exports, each exports the same values.

MetricLabelsMeaning
datafly_dlq_pending_rowspipeline_id, integration_idEntries waiting for an operator (pending)
datafly_dlq_pending_oldest_secondspipeline_id, integration_idAge of the oldest pending entry, since it last failed
datafly_dlq_depthpipelinepending entries per pipeline
datafly_dlq_stranded_rowsEntries left replaying for more than 15 minutes with no retry scheduled

A destination with no pending entries has no series.

GET /v1/admin/dlq

All parameters are optional and combine with AND.

ParameterDescription
qExact match on the event’s message_id, anonymous_id or user_id, or on the DLQ entry ID
statusOne status, a comma-separated list, or all. Default pending
integration_idThe destination (integration) the delivery failed to
vendorThe destination’s vendor key, for example meta or ga4
pipeline_idThe pipeline the event came from
event_nameThe event name, for example purchase
event_typetrack, page, screen, identify or group
error_codeThe failure code, for example permanent or delivery_failed
from, toRFC 3339 time range. from is inclusive, to exclusive
time_fieldlast_failed_at (default) or first_failed_at
limit, offsetPaging. limit up to 200

The response includes total, the number of entries matching the whole filter (not only the page).

{ "events": [ ... ], "total": 1840, "limit": 50, "offset": 0 }

Replay or discard everything matching a filter

POST /v1/admin/dlq/replay-matching and POST /v1/admin/dlq/discard-matching

Both take the search parameters above as a filter object. First ask for the count with dry_run, then confirm with that count. If the filter matches a different number by then, nothing changes and the response is 409 with the current count.

{ "filter": { "vendor": "meta", "event_name": "purchase", "from": "2026-09-20T00:00:00Z" }, "dry_run": true }
{ "matched": 1840, "dry_run": true, "max_per_request": 50000 }
{ "filter": { "vendor": "meta", "event_name": "purchase", "from": "2026-09-20T00:00:00Z" }, "confirm_count": 1840 }
{ "matched": 1840, "queued": 1840, "remaining": 0 }
  • Replay applies to pending entries only. Each is re-sent only to the destination that failed, with its original payload. Replays are queued and delivered gradually by the delivery workers, through the same rate limits and circuit breakers as live traffic, so a large replay never floods a destination. A replay that fails again returns to pending.
  • Discard applies to pending and retrying entries (status: "all" means both).
  • Up to 50,000 entries are processed per request. remaining says how many still match; repeat the request for the rest.
  • Every call is recorded in the audit log as replay_matching or discard_matching, with the filter and the count.

Replay and discard require write access to pipelines.