Amazon Data Firehose

Datafly Signal delivers first-party events to Amazon Data Firehose (formerly Kinesis Data Firehose) for automatic batching, transformation, and loading into S3, Redshift, OpenSearch, Splunk, and other downstream destinations.

Why route Amazon Data Firehose through Datafly Signal

The data lands in infrastructure you own. Signal writes to your Amazon Data Firehose directly from your own deployment. Datafly operates no intermediary and never holds a copy of your event data.
Client IP and user agent are recorded as event context. Both come from the original visitor request rather than from your server, so geography and device analysis done in the warehouse reflects the actual visitor.
Presets for eight verticals. General (the default: page, identify and the key cross-vertical events), Retail, Insurance, Travel, Media & Streaming, Publishing, B2B / SaaS and Automotive, each writing one named field per documented event property. Email, phone and names are written only as SHA-256 digests unless you deliberately change the installed blueprint.
One consent decision governs every destination. Amazon Data Firehose is gated by the same consent state as the rest of your stack, so a withdrawal applies everywhere at once.

Prerequisites

Before configuring Amazon Data Firehose in Signal, you need an AWS account with a Firehose delivery stream and IAM credentials.

Create an AWS Account

If you don’t already have one, sign up at aws.amazon.com. Ensure billing is configured and you have console access.

Create a Firehose Delivery Stream

Open the Amazon Data Firehose console.
Click Create Firehose stream.
For Source, select Direct PUT (Signal writes directly to Firehose).
Enter a Firehose stream name (e.g. datafly-events-stream).
Choose your Destination (e.g. Amazon S3, Amazon Redshift, Elasticsearch, Splunk, or HTTP endpoint).
Configure the destination settings:
- For S3: specify the bucket, prefix, and compression settings.
- For Redshift: specify the cluster, database, table, and COPY options.
Under Buffer settings, configure buffer size (1-128 MB) and buffer interval (60-900 seconds).
Click Create Firehose stream.

The buffer settings control how Firehose batches data before delivering to the destination. Smaller buffers reduce latency; larger buffers improve throughput and reduce costs.

Create an IAM User for Signal

Open the IAM console.
Go to Users > Create user.
Enter a username (e.g. datafly-signal-firehose).
Select Attach policies directly and create a custom policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecord",
        "firehose:PutRecordBatch"
      ],
      "Resource": "arn:aws:firehose:us-east-1:123456789012:deliverystream/datafly-events-stream"
    }
  ]
}

Replace the region, account ID, and stream name with your values.
Attach the policy to the user.

Generate Access Keys

On the IAM user detail page, go to Security credentials.
Under Access keys, click Create access key.
Select Application running outside AWS.
Copy the Access Key ID and Secret Access Key immediately — the secret is only shown once.

⚠️

Store these credentials securely. If you lose the secret access key, you must create a new key pair.

Configuration

Field	Type	Required	Description
`delivery_stream_name`	string	Yes	The Firehose stream name (Direct PUT source).
`region`	select	Yes	The AWS region of the stream.
`access_key_id`	secret	Yes	Access key with `firehose:PutRecord` on the stream.
`secret_access_key`	secret	Yes	Secret for the access key.
`session_token`	secret	No	STS session token for temporary credentials.

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Amazon Data Firehose or filter by Streaming.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/amazon_firehose/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Amazon Data Firehose",
    "variant": "default",
    "config": {
      "delivery_stream_name": "datafly-events-stream",
      "region": "us-east-1",
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "delivery_mode": "server_side"
  }'

What Signal writes

Each event is one Firehose record (PutRecord) whose data is the Signal event as JSON followed by a newline, so the objects Firehose writes to S3 are newline-delimited JSON:

Field	Contents
`event`, `type`	Event name (`purchase`, `page`, …) and type (`track`, `page`, `identify`, …).
`anonymous_id`, `user_id`, `device_id`	Visitor identifiers (`user_id` and `device_id` only when set).
`message_id`, `event_id`	Signal’s message ID (use it to deduplicate) and the client event ID.
`timestamp`, `sent_at`, `received_at`	RFC 3339 UTC times: event, client send, Signal receipt.
`org_id`, `pipeline_id`, `vendor_type`	Where the event came from and which destination type wrote it.
`context`	The collected context as sent: `ip`, `user_agent`, `locale`, `page`, `campaign`, `consent`, `geo`, `device`, `library`, and any partner attributes. Written in full; the blueprint does not filter it (#1414).
`click_ids`, `collected_ids`	Ad click IDs and ID-provider values, when present.
`properties`	Only the fields the blueprint maps (below).

Size: a record can be up to 1,000 KiB.
Delivery: at least once. Firehose documents that a retried PutRecord can create duplicates; deduplicate on message_id.

Only what the blueprint maps is written. The properties object holds the fields your installed blueprint maps and nothing else (the row contract): the global block under @root (copies of the envelope fields, UTM parameters, the analytics and marketing consent flags, and email_sha256, phone_sha256, first_name_sha256, last_name_sha256, name_sha256), and the event section’s fields at the top level (transaction_id, value, currency, items, …). Identify traits and client properties the blueprint does not map are never written. An event with no section in the blueprint still arrives, with the @root block only: add a section for any other event whose properties you need.

A purchase from an identified visitor, as written by the default (General) preset. The raw email, phone and name in the visitor’s identify traits are not in it; only their digests are. Empty context objects are trimmed here for brevity.

{
  "anonymous_id": "anon-s1-7f3a",
  "user_id": "crm-4711",
  "event": "purchase",
  "type": "track",
  "properties": {
    "@root": {
      "anonymous_id": "anon-s1-7f3a",
      "email_sha256": "86e0b9e56c17cc4d12387e1949b85053fbe73bc3ce5a1188713a9d300cc6133d",
      "event_id": "evt-s1-2",
      "event_name": "purchase",
      "event_timestamp": "2026-09-23T09:30:02.000Z",
      "event_type": "track",
      "first_name_sha256": "81f8f6dde88365f3928796ec7aa53f72820b06db8664f5fe76a7eb13e24546a2",
      "ip_address": "203.0.113.9",
      "last_name_sha256": "4f887a0887ca68593c8acd4e83e0b6bc5f42cd5aff7d3a159041c84230ed1cf6",
      "locale": "en-GB",
      "message_id": "msg-s1-amazon_firehose-2",
      "name_sha256": "ed37d99b1445238af3386f81a77a2caf3fffc0cd610be39b4f3ba53943dc66bf",
      "page_url": "https://shop.example/checkout",
      "phone_sha256": "a8acc3a90a7b4e4dc65e93db9240ed26523050ef754d63b75b5161de76781436",
      "user_agent": "Mozilla/5.0 (Macintosh)",
      "user_id": "crm-4711",
      "utm_medium": "cpc",
      "utm_source": "google"
    },
    "coupon": "AUTUMN10",
    "currency": "GBP",
    "items": [{ "item_id": "SKU-1", "item_name": "Trail shoe", "price": 80, "quantity": 1 }],
    "shipping": 4.5,
    "tax": 14.08,
    "transaction_id": "SIG-ORDER-1001",
    "value": 84.5
  },
  "context": {
    "ip": "203.0.113.9",
    "user_agent": "Mozilla/5.0 (Macintosh)",
    "locale": "en-GB",
    "page": { "url": "https://shop.example/checkout", "title": "Checkout", "path": "/checkout" },
    "campaign": { "source": "google", "medium": "cpc" }
  },
  "timestamp": "2026-09-23T09:30:02Z",
  "sent_at": "2026-09-23T09:30:02Z",
  "message_id": "msg-s1-amazon_firehose-2",
  "org_id": "org-s1",
  "pipeline_id": "pl-s1",
  "received_at": "2026-09-23T09:30:02Z",
  "event_id": "evt-s1-2",
  "vendor_type": "amazon_firehose"
}

The message layout on this page was produced by Signal’s Firehose writer against LocalStack 4.4 (a Direct PUT stream delivering to S3) on 2026-09-23 (batch S1 review), not written from memory.

Stored event context

Alongside the fields your blueprint maps (where email, phone, names and address are stored as normalised SHA-256 by default), each message carries the event’s collected context exactly as Signal received it. The blueprint does not filter or hash the context: this is deliberate (decision 2026-09-23), so the context is complete for attribution, fraud and debugging queries, and it is listed here so you can decide what to keep.

Stored	What it contains
Visitor request	IP address, user agent, `Accept-Language`, locale and timezone
Page	URL including its query string, path, title and referrer
Campaign	UTM source, medium, campaign, term and content
Consent	The visitor’s consent state
Location	GeoIP country, region and city, and the network (ASN)
Device	Screen size; for app events the OS and version, manufacturer, model, advertising ID and IDFV when the app collected them
Vendor IDs	Cookie and click identifiers in `vendor_ids` (for example the `_fbp` browser ID, `gclid`, `ttclid`, the GA client ID)
Session and library	Session ID, number and engagement time; the collector’s name and version
Partner attributes	Profile attributes from partner enrichment (`partner_attrs`)

Each message is the event JSON: context, click_ids and collected_ids, and the anonymous_id, user_id, device_id, message_id and event_id identifiers, sit beside the mapped properties.

The IP address, device and advertising IDs, and cookie and click identifiers are personal data under UK and EU GDPR. To keep any of them out of storage, strip, mask or hash the field for every integration with the organisation data layer, apply URL redaction to page URLs, or remove it in the installed blueprint; and set the storage’s retention to match your privacy notice. See Personal data by integration.

The integration’s consent category in Signal decides which events reach Amazon Data Firehose; events that fail it are never written. Every message also carries the visitor’s consent state (context.consent, and consent_analytics / consent_marketing in @root) so downstream consumers can apply their own rules.

Testing

Enable the integration in Signal and trigger a test event on your website.
Open the Firehose console and select your delivery stream.
Check the Monitoring tab for incoming record counts and delivery success metrics.
Verify data is arriving at the configured destination (e.g. check the S3 bucket for new files, or query the Redshift table).
In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

Problem	Solution
Events not appearing in the destination	Check the Firehose Monitoring tab for errors. Verify the delivery stream name and region are correct.
`AccessDeniedException`	The IAM user lacks `firehose:PutRecord` or `firehose:PutRecordBatch` permission. Update the IAM policy.
`ResourceNotFoundException`	The delivery stream does not exist in the specified region. Verify the stream name and region.
Data delayed in destination	Firehose buffers data before delivery. Check the buffer size and interval settings on the stream. Minimum interval is 60 seconds.
`ServiceUnavailableException`	Firehose is temporarily unavailable. Signal will automatically retry. Check the AWS Service Health Dashboard.
Destination delivery failures	Check the Firehose error log (S3 error prefix or CloudWatch logs) for destination-specific errors.
Credential errors	Verify the access key ID and secret access key are correct and the IAM user has not been deactivated.

Visit Amazon Data Firehose documentation for full API reference and destination configuration guides.