IntegrationsStreamingAmazon SQS

Amazon SQS

Datafly Signal sends first-party events to Amazon SQS queues for decoupled, serverless event processing and integration with downstream AWS services like Lambda, ECS, and Step Functions.

Why route Amazon SQS through Datafly Signal

  • The data lands in infrastructure you own. Signal writes to your Amazon SQS directly from your own deployment. Datafly operates no intermediary and never holds a copy of your event data.
  • Client IP and user agent are recorded as event context. Both come from the original visitor request rather than from your server, so geography and device analysis done in the warehouse reflects the actual visitor.
  • Presets for eight verticals. General (the default: page, identify and the key cross-vertical events), Retail, Insurance, Travel, Media & Streaming, Publishing, B2B / SaaS and Automotive, each writing one named field per documented event property. Email, phone and names are written only as SHA-256 digests unless you deliberately change the installed blueprint.
  • One consent decision governs every destination. Amazon SQS is gated by the same consent state as the rest of your stack, so a withdrawal applies everywhere at once.

Prerequisites

Before configuring Amazon SQS in Signal, you need an AWS account with an SQS queue and an IAM user with send permissions.

Create an AWS Account

If you don’t already have one, sign up at aws.amazon.com.

Create an SQS Queue

  1. Open the SQS console.
  2. Click Create queue.
  3. Choose the queue type:
    • Standard — maximum throughput, at-least-once delivery, best-effort ordering (recommended for most use cases).
    • FIFO — exactly-once processing, strict ordering (lower throughput). FIFO queue names must end with .fifo.
  4. Enter a Queue name (e.g. datafly-events).
  5. Configure Visibility timeout (default 30 seconds) and Message retention period (default 4 days, max 14 days).
  6. Click Create queue.
  7. Note the Queue URL from the queue details page (e.g. https://sqs.us-east-1.amazonaws.com/123456789012/datafly-events).

Standard queues are recommended for most Signal use cases. Use FIFO queues only if you need strict event ordering and exactly-once processing.

Create an IAM User for Signal

  1. Open the IAM console.
  2. Go to Users > Create user.
  3. Enter a username (e.g. datafly-signal-sqs).
  4. Attach a custom policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:SendMessageBatch"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:datafly-events"
    }
  ]
}
  1. Replace the region, account ID, and queue name with your values.

Generate Access Keys

  1. On the IAM user detail page, go to Security credentials.
  2. Under Access keys, click Create access key.
  3. Select Application running outside AWS.
  4. Copy the Access Key ID and Secret Access Key immediately — the secret is only shown once.
⚠️

Store these credentials securely. If you lose the secret access key, you must create a new key pair.

Configuration

FieldTypeRequiredDescription
queue_urlstringYesThe full queue URL. A URL ending in .fifo turns on FIFO mode.
regionselectYesThe AWS region of the queue.
access_key_idsecretYesAccess key with sqs:SendMessage on the queue.
secret_access_keysecretYesSecret for the access key.
session_tokensecretNoSTS session token for temporary credentials.

Signal Setup

Quick Setup

  1. Navigate to Integrations in the sidebar.
  2. Open the Integration Library tab.
  3. Find Amazon SQS or filter by Streaming.
  4. Click Install, select a variant if available, and fill in the required fields.
  5. Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/amazon_sqs/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Amazon SQS",
    "variant": "default",
    "config": {
      "queue_url": "https://sqs.us-east-1.amazonaws.com/123456789012/datafly-events",
      "region": "us-east-1",
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "delivery_mode": "server_side"
  }'

What Signal writes

Each event is one SQS message (SendMessage) whose body is the Signal event as JSON:

FieldContents
event, typeEvent name (purchase, page, …) and type (track, page, identify, …).
anonymous_id, user_id, device_idVisitor identifiers (user_id and device_id only when set).
message_id, event_idSignal’s message ID (use it to deduplicate) and the client event ID.
timestamp, sent_at, received_atRFC 3339 UTC times: event, client send, Signal receipt.
org_id, pipeline_id, vendor_typeWhere the event came from and which destination type wrote it.
contextThe collected context as sent: ip, user_agent, locale, page, campaign, consent, geo, device, library, and any partner attributes. Written in full; the blueprint does not filter it (#1414).
click_ids, collected_idsAd click IDs and ID-provider values, when present.
propertiesOnly the fields the blueprint maps (below).
  • Standard queue: no message group or deduplication ID. Delivery is at least once; deduplicate on message_id.
  • FIFO queue (.fifo): MessageGroupId is user_id, or anonymous_id when there is no user ID, so one visitor’s events are processed in order. MessageDeduplicationId is the Signal message_id, so a retry within SQS’s five-minute deduplication interval is accepted but not delivered twice.
  • Size: an SQS message body can be up to 1 MiB.

Only what the blueprint maps is written. The properties object holds the fields your installed blueprint maps and nothing else (the row contract): the global block under @root (copies of the envelope fields, UTM parameters, the analytics and marketing consent flags, and email_sha256, phone_sha256, first_name_sha256, last_name_sha256, name_sha256), and the event section’s fields at the top level (transaction_id, value, currency, items, …). Identify traits and client properties the blueprint does not map are never written. An event with no section in the blueprint still arrives, with the @root block only: add a section for any other event whose properties you need.

A purchase from an identified visitor, as written by the default (General) preset. The raw email, phone and name in the visitor’s identify traits are not in it; only their digests are. Empty context objects are trimmed here for brevity.

{
  "anonymous_id": "anon-s1-7f3a",
  "user_id": "crm-4711",
  "event": "purchase",
  "type": "track",
  "properties": {
    "@root": {
      "anonymous_id": "anon-s1-7f3a",
      "email_sha256": "86e0b9e56c17cc4d12387e1949b85053fbe73bc3ce5a1188713a9d300cc6133d",
      "event_id": "evt-s1-2",
      "event_name": "purchase",
      "event_timestamp": "2026-09-23T09:30:02.000Z",
      "event_type": "track",
      "first_name_sha256": "81f8f6dde88365f3928796ec7aa53f72820b06db8664f5fe76a7eb13e24546a2",
      "ip_address": "203.0.113.9",
      "last_name_sha256": "4f887a0887ca68593c8acd4e83e0b6bc5f42cd5aff7d3a159041c84230ed1cf6",
      "locale": "en-GB",
      "message_id": "msg-s1-amazon_sqs-2",
      "name_sha256": "ed37d99b1445238af3386f81a77a2caf3fffc0cd610be39b4f3ba53943dc66bf",
      "page_url": "https://shop.example/checkout",
      "phone_sha256": "a8acc3a90a7b4e4dc65e93db9240ed26523050ef754d63b75b5161de76781436",
      "user_agent": "Mozilla/5.0 (Macintosh)",
      "user_id": "crm-4711",
      "utm_medium": "cpc",
      "utm_source": "google"
    },
    "coupon": "AUTUMN10",
    "currency": "GBP",
    "items": [{ "item_id": "SKU-1", "item_name": "Trail shoe", "price": 80, "quantity": 1 }],
    "shipping": 4.5,
    "tax": 14.08,
    "transaction_id": "SIG-ORDER-1001",
    "value": 84.5
  },
  "context": {
    "ip": "203.0.113.9",
    "user_agent": "Mozilla/5.0 (Macintosh)",
    "locale": "en-GB",
    "page": { "url": "https://shop.example/checkout", "title": "Checkout", "path": "/checkout" },
    "campaign": { "source": "google", "medium": "cpc" }
  },
  "timestamp": "2026-09-23T09:30:02Z",
  "sent_at": "2026-09-23T09:30:02Z",
  "message_id": "msg-s1-amazon_sqs-2",
  "org_id": "org-s1",
  "pipeline_id": "pl-s1",
  "received_at": "2026-09-23T09:30:02Z",
  "event_id": "evt-s1-2",
  "vendor_type": "amazon_sqs"
}

The message layout on this page was produced by Signal’s SQS writer against LocalStack 4.4 (a standard and a FIFO queue) on 2026-09-23 (batch S1 review), not written from memory.

Stored event context

Alongside the fields your blueprint maps (where email, phone, names and address are stored as normalised SHA-256 by default), each message carries the event’s collected context exactly as Signal received it. The blueprint does not filter or hash the context: this is deliberate (decision 2026-09-23), so the context is complete for attribution, fraud and debugging queries, and it is listed here so you can decide what to keep.

StoredWhat it contains
Visitor requestIP address, user agent, Accept-Language, locale and timezone
PageURL including its query string, path, title and referrer
CampaignUTM source, medium, campaign, term and content
ConsentThe visitor’s consent state
LocationGeoIP country, region and city, and the network (ASN)
DeviceScreen size; for app events the OS and version, manufacturer, model, advertising ID and IDFV when the app collected them
Vendor IDsCookie and click identifiers in vendor_ids (for example the _fbp browser ID, gclid, ttclid, the GA client ID)
Session and librarySession ID, number and engagement time; the collector’s name and version
Partner attributesProfile attributes from partner enrichment (partner_attrs)

Each message is the event JSON: context, click_ids and collected_ids, and the anonymous_id, user_id, device_id, message_id and event_id identifiers, sit beside the mapped properties.

The IP address, device and advertising IDs, and cookie and click identifiers are personal data under UK and EU GDPR. To keep any of them out of storage, strip, mask or hash the field for every integration with the organisation data layer, apply URL redaction to page URLs, or remove it in the installed blueprint; and set the storage’s retention to match your privacy notice. See Personal data by integration.

Consent

The integration’s consent category in Signal decides which events reach Amazon SQS; events that fail it are never written. Every message also carries the visitor’s consent state (context.consent, and consent_analytics / consent_marketing in @root) so downstream consumers can apply their own rules.

Testing

  1. Enable the integration in Signal and trigger a test event on your website.
  2. Open the SQS console and select your queue.
  3. Click Send and receive messages > Poll for messages to retrieve queued events.
  4. Click on a message to inspect the event payload body.
  5. In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

ProblemSolution
Events not appearing in the queueVerify the queue URL and region are correct. Ensure the queue exists and has not been deleted.
AccessDeniedThe IAM user lacks sqs:SendMessage permission on the queue. Update the IAM policy.
NonExistentQueueThe queue URL is incorrect or the queue has been deleted. Verify the URL in the SQS console.
InvalidParameterValueCheck the message format. SQS message bodies must be valid UTF-8 strings up to 256 KB.
Messages disappearing quicklyAnother consumer may be processing and deleting messages. Check the visibility timeout and verify no unexpected consumers are active.
FIFO deduplication issuesFIFO queues deduplicate messages within a 5-minute window. If you see missing events, check that message deduplication IDs are unique.
Credential errorsVerify the access key ID and secret access key are correct and the IAM user has not been deactivated.

Visit Amazon SQS documentation for full API reference, dead-letter queue configuration, and Lambda trigger setup.

See also

Amazon KinesisAzure Event Hubs