Dead Letter Queue
When a delivery to a destination fails, Datafly Signal keeps the event and the failure in the dead letter queue (DLQ). Transient failures are retried automatically on a backoff schedule (status retrying). When the schedule is exhausted, or the failure is permanent, the event waits for an operator (status pending). Nothing expires automatically.
Each DLQ entry is one event and one destination. If an event failed to two destinations, it has two entries, and replaying one sends the event only to that destination.
An event the processor could not process never reaches this queue: it is held as a processor dead letter and replayed with the event processor’s dlq-replay command.
Statuses
| Status | Meaning |
|---|---|
retrying | On the automatic retry schedule, or queued for an operator replay |
pending | Needs an operator: replay or discard |
replaying | A single replay is in flight |
resolved | Delivered on retry or replay |
discarded | Discarded by an operator |
When an entry is not retried
An entry returns to pending without being sent, whatever its retry schedule, when it can never be delivered as stored. Its error code says why, and the original failure is kept at the end of the error message.
| Error code | Meaning |
|---|---|
integration_removed | The destination is no longer in its pipeline’s published version, or was deleted |
replay_no_destination | The entry names no destination, for example a message that could not be read |
replay_unparseable | The stored event cannot be read |
replay_no_event | The entry has no stored event |
An entry whose destination is switched off, whose pipeline is paused, or whose connection settings cannot be read stays on the retry schedule and is delivered once that is put right. It goes to pending when the schedule runs out. An entry is never marked resolved unless it was delivered.
While a destination is backlogged behind its rate limit, a retry that is skipped without being attempted does not use up the schedule: it waits and keeps its place. It waits at most as long as the whole schedule would have taken (about 31 hours by default), or the destination’s retry window if it sets one, then goes to pending.
Alerting
The delivery workers export these Prometheus gauges every minute, from one instance at a time. Alert on max() across instances: if the instances cannot agree which one exports, each exports the same values.
| Metric | Labels | Meaning |
|---|---|---|
datafly_dlq_pending_rows | pipeline_id, integration_id | Entries waiting for an operator (pending) |
datafly_dlq_pending_oldest_seconds | pipeline_id, integration_id | Age of the oldest pending entry, since it last failed |
datafly_dlq_depth | pipeline | pending entries per pipeline |
datafly_dlq_stranded_rows | Entries left replaying for more than 15 minutes with no retry scheduled |
A destination with no pending entries has no series.
Search
GET /v1/admin/dlq
All parameters are optional and combine with AND.
| Parameter | Description |
|---|---|
q | Exact match on the event’s message_id, anonymous_id or user_id, or on the DLQ entry ID |
status | One status, a comma-separated list, or all. Default pending |
integration_id | The destination (integration) the delivery failed to |
vendor | The destination’s vendor key, for example meta or ga4 |
pipeline_id | The pipeline the event came from |
event_name | The event name, for example purchase |
event_type | track, page, screen, identify or group |
error_code | The failure code, for example permanent or delivery_failed |
from, to | RFC 3339 time range. from is inclusive, to exclusive |
time_field | last_failed_at (default) or first_failed_at |
limit, offset | Paging. limit up to 200 |
The response includes total, the number of entries matching the whole filter (not only the page).
{ "events": [ ... ], "total": 1840, "limit": 50, "offset": 0 }Replay or discard everything matching a filter
POST /v1/admin/dlq/replay-matching and POST /v1/admin/dlq/discard-matching
Both take the search parameters above as a filter object. First ask for the count with dry_run, then confirm with that count. If the filter matches a different number by then, nothing changes and the response is 409 with the current count.
{ "filter": { "vendor": "meta", "event_name": "purchase", "from": "2026-09-20T00:00:00Z" }, "dry_run": true }{ "matched": 1840, "dry_run": true, "max_per_request": 50000 }{ "filter": { "vendor": "meta", "event_name": "purchase", "from": "2026-09-20T00:00:00Z" }, "confirm_count": 1840 }{ "matched": 1840, "queued": 1840, "remaining": 0 }- Replay applies to
pendingentries only. Each is re-sent only to the destination that failed, with its original payload. Replays are queued and delivered gradually by the delivery workers, through the same rate limits and circuit breakers as live traffic, so a large replay never floods a destination. A replay that fails again returns topending. - Discard applies to
pendingandretryingentries (status: "all"means both). - Up to 50,000 entries are processed per request.
remainingsays how many still match; repeat the request for the rest. - Every call is recorded in the audit log as
replay_matchingordiscard_matching, with the filter and the count.
Replay and discard require write access to pipelines.