IntegrationsStreamingAmazon SQS

Amazon SQS

Datafly Signal delivers events to Amazon SQS queues for decoupled, serverless event processing and integration with downstream AWS services like Lambda, ECS, and Step Functions.

Prerequisites

Before configuring Amazon SQS in Signal, you need an AWS account with an SQS queue and an IAM user with send permissions.

Create an AWS Account

If you don’t already have one, sign up at aws.amazon.com.

Create an SQS Queue

  1. Open the SQS console.
  2. Click Create queue.
  3. Choose the queue type:
    • Standard — maximum throughput, at-least-once delivery, best-effort ordering (recommended for most use cases).
    • FIFO — exactly-once processing, strict ordering (lower throughput). FIFO queue names must end with .fifo.
  4. Enter a Queue name (e.g. datafly-events).
  5. Configure Visibility timeout (default 30 seconds) and Message retention period (default 4 days, max 14 days).
  6. Click Create queue.
  7. Note the Queue URL from the queue details page (e.g. https://sqs.us-east-1.amazonaws.com/123456789012/datafly-events).

Standard queues are recommended for most Signal use cases. Use FIFO queues only if you need strict event ordering and exactly-once processing.

Create an IAM User for Signal

  1. Open the IAM console.
  2. Go to Users > Create user.
  3. Enter a username (e.g. datafly-signal-sqs).
  4. Attach a custom policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:SendMessageBatch"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:datafly-events"
    }
  ]
}
  1. Replace the region, account ID, and queue name with your values.

Generate Access Keys

  1. On the IAM user detail page, go to Security credentials.
  2. Under Access keys, click Create access key.
  3. Select Application running outside AWS.
  4. Copy the Access Key ID and Secret Access Key immediately — the secret is only shown once.
⚠️

Store these credentials securely. If you lose the secret access key, you must create a new key pair.

Configuration

FieldTypeRequiredDescription
queue_urlstringYesThe full SQS queue URL.
regionselectYesThe AWS region where the queue is located.
access_key_idsecretYesThe AWS access key ID for authentication.
secret_access_keysecretYesThe AWS secret access key for authentication.

Signal Setup

Quick Setup

  1. Navigate to Integrations in the sidebar.
  2. Open the Integration Library tab.
  3. Find Amazon SQS or filter by Cloud Storage.
  4. Click Install, select a variant if available, and fill in the required fields.
  5. Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/amazon_sqs/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Amazon SQS",
    "variant": "default",
    "config": {
      "queue_url": "https://sqs.us-east-1.amazonaws.com/123456789012/datafly-events",
      "region": "us-east-1",
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "delivery_mode": "server_side"
  }'

Testing

  1. Enable the integration in Signal and trigger a test event on your website.
  2. Open the SQS console and select your queue.
  3. Click Send and receive messages > Poll for messages to retrieve queued events.
  4. Click on a message to inspect the event payload body.
  5. In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

ProblemSolution
Events not appearing in the queueVerify the queue URL and region are correct. Ensure the queue exists and has not been deleted.
AccessDeniedThe IAM user lacks sqs:SendMessage permission on the queue. Update the IAM policy.
NonExistentQueueThe queue URL is incorrect or the queue has been deleted. Verify the URL in the SQS console.
InvalidParameterValueCheck the message format. SQS message bodies must be valid UTF-8 strings up to 256 KB.
Messages disappearing quicklyAnother consumer may be processing and deleting messages. Check the visibility timeout and verify no unexpected consumers are active.
FIFO deduplication issuesFIFO queues deduplicate messages within a 5-minute window. If you see missing events, check that message deduplication IDs are unique.
Credential errorsVerify the access key ID and secret access key are correct and the IAM user has not been deactivated.

Visit Amazon SQS documentation for full API reference, dead-letter queue configuration, and Lambda trigger setup.

Amazon KinesisAzure Event Hubs