Amazon Kinesis

Datafly Signal publishes first-party events to Amazon Kinesis Data Streams for real-time stream processing, analytics, and event-driven architectures.

Why route Amazon Kinesis through Datafly Signal

The data lands in infrastructure you own. Signal writes to your Amazon Kinesis directly from your own deployment. Datafly operates no intermediary and never holds a copy of your event data.
Client IP and user agent are recorded as event context. Both come from the original visitor request rather than from your server, so geography and device analysis done in the warehouse reflects the actual visitor.
Presets for eight verticals. General (the default: page, identify and the key cross-vertical events), Retail, Insurance, Travel, Media & Streaming, Publishing, B2B / SaaS and Automotive, each writing one named field per documented event property. Email, phone and names are written only as SHA-256 digests unless you deliberately change the installed blueprint.
One consent decision governs every destination. Amazon Kinesis is gated by the same consent state as the rest of your stack, so a withdrawal applies everywhere at once.

Prerequisites

Before configuring Amazon Kinesis in Signal, you need an AWS account with a Kinesis data stream and IAM credentials.

Create an AWS Account

If you don’t already have one, sign up at aws.amazon.com.

Create a Kinesis Data Stream

Open the Kinesis console.
Click Create data stream.
Enter a Data stream name (e.g. datafly-events).
Choose the Capacity mode:
- On-demand — automatically scales to handle your throughput (recommended for variable workloads).
- Provisioned — set a fixed number of shards. Each shard supports 1 MB/s write and 2 MB/s read.
Click Create data stream.

On-demand mode automatically manages shard count and scales up to 200 MB/s write throughput. For predictable, high-volume workloads, provisioned mode gives more control over costs.

Create an IAM User for Signal

Open the IAM console.
Go to Users > Create user.
Enter a username (e.g. datafly-signal-kinesis).
Attach a custom policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/datafly-events"
    }
  ]
}

Replace the region, account ID, and stream name with your values.

Generate Access Keys

On the IAM user detail page, go to Security credentials.
Under Access keys, click Create access key.
Select Application running outside AWS.
Copy the Access Key ID and Secret Access Key immediately — the secret is only shown once.

⚠️

Store these credentials securely. If you lose the secret access key, you must create a new key pair.

Configuration

Field	Type	Required	Description
`stream_name`	string	Yes	The data stream name.
`region`	select	Yes	The AWS region of the stream.
`access_key_id`	secret	Yes	Access key with `kinesis:PutRecord` on the stream.
`secret_access_key`	secret	Yes	Secret for the access key.
`session_token`	secret	No	STS session token for temporary credentials.
`partition_key`	string	No	The field whose value is the partition key: `user_id`, `anonymous_id`, `message_id`, `device_id`, or a top-level field the blueprint maps (for example `transaction_id`). Empty, or empty on an event: `user_id`, else `anonymous_id`, else `message_id`.
The partition key is set automatically to the event’s `anonymous_id` (or `user_id` if available), giving consistent per-user shard placement and ordering.

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Amazon Kinesis or filter by Streaming.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/amazon_kinesis/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Amazon Kinesis",
    "variant": "default",
    "config": {
      "stream_name": "datafly-events",
      "region": "us-east-1",
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "delivery_mode": "server_side"
  }'

What Signal writes

Each event is one Kinesis record (PutRecord) whose data is the Signal event as JSON:

Field	Contents
`event`, `type`	Event name (`purchase`, `page`, …) and type (`track`, `page`, `identify`, …).
`anonymous_id`, `user_id`, `device_id`	Visitor identifiers (`user_id` and `device_id` only when set).
`message_id`, `event_id`	Signal’s message ID (use it to deduplicate) and the client event ID.
`timestamp`, `sent_at`, `received_at`	RFC 3339 UTC times: event, client send, Signal receipt.
`org_id`, `pipeline_id`, `vendor_type`	Where the event came from and which destination type wrote it.
`context`	The collected context as sent: `ip`, `user_agent`, `locale`, `page`, `campaign`, `consent`, `geo`, `device`, `library`, and any partner attributes. Written in full; the blueprint does not filter it (#1414).
`click_ids`, `collected_ids`	Ad click IDs and ID-provider values, when present.
`properties`	Only the fields the blueprint maps (below).

Partition key: by default user_id, else anonymous_id, else message_id, so one visitor’s records land on one shard in order. Set partition_key to spread load differently: message_id spreads evenly but loses per-visitor order.
Size: a record (data plus partition key) can be up to 10 MiB; a shard takes 1,000 records or 10 MiB per second.
Delivery: at least once; deduplicate on message_id.

Only what the blueprint maps is written. The properties object holds the fields your installed blueprint maps and nothing else (the row contract): the global block under @root (copies of the envelope fields, UTM parameters, the analytics and marketing consent flags, and email_sha256, phone_sha256, first_name_sha256, last_name_sha256, name_sha256), and the event section’s fields at the top level (transaction_id, value, currency, items, …). Identify traits and client properties the blueprint does not map are never written. An event with no section in the blueprint still arrives, with the @root block only: add a section for any other event whose properties you need.

A purchase from an identified visitor, as written by the default (General) preset. The raw email, phone and name in the visitor’s identify traits are not in it; only their digests are. Empty context objects are trimmed here for brevity.

{
  "anonymous_id": "anon-s1-7f3a",
  "user_id": "crm-4711",
  "event": "purchase",
  "type": "track",
  "properties": {
    "@root": {
      "anonymous_id": "anon-s1-7f3a",
      "email_sha256": "86e0b9e56c17cc4d12387e1949b85053fbe73bc3ce5a1188713a9d300cc6133d",
      "event_id": "evt-s1-2",
      "event_name": "purchase",
      "event_timestamp": "2026-09-23T09:30:02.000Z",
      "event_type": "track",
      "first_name_sha256": "81f8f6dde88365f3928796ec7aa53f72820b06db8664f5fe76a7eb13e24546a2",
      "ip_address": "203.0.113.9",
      "last_name_sha256": "4f887a0887ca68593c8acd4e83e0b6bc5f42cd5aff7d3a159041c84230ed1cf6",
      "locale": "en-GB",
      "message_id": "msg-s1-amazon_kinesis-2",
      "name_sha256": "ed37d99b1445238af3386f81a77a2caf3fffc0cd610be39b4f3ba53943dc66bf",
      "page_url": "https://shop.example/checkout",
      "phone_sha256": "a8acc3a90a7b4e4dc65e93db9240ed26523050ef754d63b75b5161de76781436",
      "user_agent": "Mozilla/5.0 (Macintosh)",
      "user_id": "crm-4711",
      "utm_medium": "cpc",
      "utm_source": "google"
    },
    "coupon": "AUTUMN10",
    "currency": "GBP",
    "items": [{ "item_id": "SKU-1", "item_name": "Trail shoe", "price": 80, "quantity": 1 }],
    "shipping": 4.5,
    "tax": 14.08,
    "transaction_id": "SIG-ORDER-1001",
    "value": 84.5
  },
  "context": {
    "ip": "203.0.113.9",
    "user_agent": "Mozilla/5.0 (Macintosh)",
    "locale": "en-GB",
    "page": { "url": "https://shop.example/checkout", "title": "Checkout", "path": "/checkout" },
    "campaign": { "source": "google", "medium": "cpc" }
  },
  "timestamp": "2026-09-23T09:30:02Z",
  "sent_at": "2026-09-23T09:30:02Z",
  "message_id": "msg-s1-amazon_kinesis-2",
  "org_id": "org-s1",
  "pipeline_id": "pl-s1",
  "received_at": "2026-09-23T09:30:02Z",
  "event_id": "evt-s1-2",
  "vendor_type": "amazon_kinesis"
}

The message layout on this page was produced by Signal’s Kinesis writer against kinesis-mock 0.4.9 on 2026-09-23 (batch S1 review), not written from memory.

Stored event context

Alongside the fields your blueprint maps (where email, phone, names and address are stored as normalised SHA-256 by default), each message carries the event’s collected context exactly as Signal received it. The blueprint does not filter or hash the context: this is deliberate (decision 2026-09-23), so the context is complete for attribution, fraud and debugging queries, and it is listed here so you can decide what to keep.

Stored	What it contains
Visitor request	IP address, user agent, `Accept-Language`, locale and timezone
Page	URL including its query string, path, title and referrer
Campaign	UTM source, medium, campaign, term and content
Consent	The visitor’s consent state
Location	GeoIP country, region and city, and the network (ASN)
Device	Screen size; for app events the OS and version, manufacturer, model, advertising ID and IDFV when the app collected them
Vendor IDs	Cookie and click identifiers in `vendor_ids` (for example the `_fbp` browser ID, `gclid`, `ttclid`, the GA client ID)
Session and library	Session ID, number and engagement time; the collector’s name and version
Partner attributes	Profile attributes from partner enrichment (`partner_attrs`)

Each message is the event JSON: context, click_ids and collected_ids, and the anonymous_id, user_id, device_id, message_id and event_id identifiers, sit beside the mapped properties.

The IP address, device and advertising IDs, and cookie and click identifiers are personal data under UK and EU GDPR. To keep any of them out of storage, strip, mask or hash the field for every integration with the organisation data layer, apply URL redaction to page URLs, or remove it in the installed blueprint; and set the storage’s retention to match your privacy notice. See Personal data by integration.

The integration’s consent category in Signal decides which events reach Amazon Kinesis; events that fail it are never written. Every message also carries the visitor’s consent state (context.consent, and consent_analytics / consent_marketing in @root) so downstream consumers can apply their own rules.

Testing

Enable the integration in Signal and trigger a test event on your website.
Open the Kinesis console and select your data stream.
Check the Monitoring tab for incoming put record metrics.
Use the Data Viewer tab to inspect individual records in the stream.
In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

Problem	Solution
Events not appearing in the stream	Verify the stream name and region are correct. Check that the stream status is Active.
`AccessDeniedException`	The IAM user lacks `kinesis:PutRecord` or `kinesis:PutRecords` permission. Update the IAM policy.
`ResourceNotFoundException`	The stream does not exist in the specified region. Check the stream name and region.
`ProvisionedThroughputExceededException`	The shard write capacity has been exceeded. Add more shards or switch to on-demand mode.
Records landing on the same shard	The default partition key keeps a visitor on one shard. If a few heavy visitors skew load, set `partition_key` to `message_id` (even spread, no per-visitor order).
High latency on consumers	Increase the number of shards to improve read parallelism. Each shard supports 2 MB/s read throughput.
Credential errors	Verify the access key ID and secret access key are correct and the IAM user has not been deactivated.

Visit Amazon Kinesis documentation for full API reference and stream management guides.