Amazon Data Firehose

Datafly Signal delivers events to Amazon Data Firehose (formerly Kinesis Data Firehose) for automatic batching, transformation, and loading into S3, Redshift, Elasticsearch, Splunk, and other downstream destinations.

Prerequisites

Before configuring Amazon Data Firehose in Signal, you need an AWS account with a Firehose delivery stream and IAM credentials.

Create an AWS Account

If you don’t already have one, sign up at aws.amazon.com. Ensure billing is configured and you have console access.

Create a Firehose Delivery Stream

Open the Amazon Data Firehose console.
Click Create Firehose stream.
For Source, select Direct PUT (Signal writes directly to Firehose).
Enter a Firehose stream name (e.g. datafly-events-stream).
Choose your Destination (e.g. Amazon S3, Amazon Redshift, Elasticsearch, Splunk, or HTTP endpoint).
Configure the destination settings:
- For S3: specify the bucket, prefix, and compression settings.
- For Redshift: specify the cluster, database, table, and COPY options.
Under Buffer settings, configure buffer size (1-128 MB) and buffer interval (60-900 seconds).
Click Create Firehose stream.

The buffer settings control how Firehose batches data before delivering to the destination. Smaller buffers reduce latency; larger buffers improve throughput and reduce costs.

Create an IAM User for Signal

Open the IAM console.
Go to Users > Create user.
Enter a username (e.g. datafly-signal-firehose).
Select Attach policies directly and create a custom policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecord",
        "firehose:PutRecordBatch"
      ],
      "Resource": "arn:aws:firehose:us-east-1:123456789012:deliverystream/datafly-events-stream"
    }
  ]
}

Replace the region, account ID, and stream name with your values.
Attach the policy to the user.

Generate Access Keys

On the IAM user detail page, go to Security credentials.
Under Access keys, click Create access key.
Select Application running outside AWS.
Copy the Access Key ID and Secret Access Key immediately — the secret is only shown once.

⚠️

Store these credentials securely. If you lose the secret access key, you must create a new key pair.

Configuration

Field	Type	Required	Description
`delivery_stream_name`	string	Yes	The name of the Firehose delivery stream to send records to.
`region`	select	Yes	The AWS region where your Firehose delivery stream is configured.
`access_key_id`	secret	Yes	The AWS access key ID with firehose:PutRecord and firehose:PutRecordBatch permissions.
`secret_access_key`	secret	Yes	The AWS secret access key associated with the access key ID.

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Amazon Data Firehose or filter by Cloud Storage.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/amazon_firehose/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Amazon Data Firehose",
    "variant": "default",
    "config": {
      "delivery_stream_name": "datafly-events-stream",
      "region": "us-east-1",
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "delivery_mode": "server_side"
  }'

Testing

Enable the integration in Signal and trigger a test event on your website.
Open the Firehose console and select your delivery stream.
Check the Monitoring tab for incoming record counts and delivery success metrics.
Verify data is arriving at the configured destination (e.g. check the S3 bucket for new files, or query the Redshift table).
In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

Problem	Solution
Events not appearing in the destination	Check the Firehose Monitoring tab for errors. Verify the delivery stream name and region are correct.
`AccessDeniedException`	The IAM user lacks `firehose:PutRecord` or `firehose:PutRecordBatch` permission. Update the IAM policy.
`ResourceNotFoundException`	The delivery stream does not exist in the specified region. Verify the stream name and region.
Data delayed in destination	Firehose buffers data before delivery. Check the buffer size and interval settings on the stream. Minimum interval is 60 seconds.
`ServiceUnavailableException`	Firehose is temporarily unavailable. Signal will automatically retry. Check the AWS Service Health Dashboard.
Destination delivery failures	Check the Firehose error log (S3 error prefix or CloudWatch logs) for destination-specific errors.
Credential errors	Verify the access key ID and secret access key are correct and the IAM user has not been deactivated.

Visit Amazon Data Firehose documentation for full API reference and destination configuration guides.

Data Warehouses (Overview)Amazon Kinesis