Amazon S3

Datafly Signal writes batches of first-party events as objects into an S3 bucket — ready for query with Athena, Glue, Spark, Presto, or any downstream lakehouse tooling.

Why route Amazon S3 through Datafly Signal

The data lands in infrastructure you own. Signal writes to your Amazon S3 directly from your own deployment. Datafly operates no intermediary and never holds a copy of your event data.
Client IP and user agent are recorded as event context. Both come from the original visitor request rather than from your server, so geography and device analysis done in the warehouse reflects the actual visitor.
Presets for eight verticals. General (the default: page, identify and the key cross-vertical events), Retail, Insurance, Travel, Media & Streaming, Publishing, B2B / SaaS and Automotive, each writing one named field per documented event property. Email, phone and names are written only as SHA-256 digests unless you deliberately change the installed blueprint.
One consent decision governs every destination. Amazon S3 is gated by the same consent state as the rest of your stack, so a withdrawal applies everywhere at once.

Prerequisites

Before configuring Amazon S3 in Signal, you need an AWS account with an S3 bucket and an IAM user with write permissions.

Create an AWS Account

If you don’t already have one, sign up at aws.amazon.com.

Create an S3 Bucket

Open the S3 console.
Click Create bucket.
Enter a Bucket name (e.g. datafly-events-production). Bucket names must be globally unique.
Select the AWS Region closest to your Signal deployment.
Leave Block Public Access settings enabled (recommended).
Optionally enable Bucket Versioning for data protection.
Click Create bucket.

Choose a region close to your Signal infrastructure to minimise latency and data transfer costs.

Create an IAM User for Signal

Open the IAM console.
Go to Users > Create user.
Enter a username (e.g. datafly-signal-s3).
Attach a custom policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::datafly-events-production/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::datafly-events-production"
    }
  ]
}

Replace the bucket name with your value.

Generate Access Keys

On the IAM user detail page, go to Security credentials.
Under Access keys, click Create access key.
Select Application running outside AWS.
Copy the Access Key ID and Secret Access Key immediately — the secret is only shown once.

⚠️

Store these credentials securely. If you lose the secret access key, you must create a new key pair.

Configuration

Field	Type	Required	Description
`bucket_name`	string	Yes	The S3 bucket. Also accepts `bucket`.
`region`	select	Yes	The AWS region of the bucket.
`access_key_id`	secret	No	Access key with `s3:PutObject` on the bucket. Leave empty (with the secret) to use the default AWS credential chain, for example an IAM role for the delivery worker’s service account on EKS.
`secret_access_key`	secret	No	Secret for the access key.
`session_token`	secret	No	STS session token for temporary credentials.
`prefix`	string	No	Folder prepended to every object name (for example `events`).
`storage_class`	select	No	S3 storage class, for example `STANDARD`, `STANDARD_IA`, `INTELLIGENT_TIERING`. Default `STANDARD`.
`server_side_encryption`	select	No	`AES256` or `aws:kms`.
`kms_key_id`	string	No	KMS key ARN, with `aws:kms`.
`file_format`	select	Yes	`json` (newline-delimited, default), `csv` or `parquet`.
`parquet_compression`	select	No	`snappy` (default), `zstd`, `gzip` or `none`.
`schema_mode`	select	No	`canonical` (default, 16 fixed columns) or `blueprint` (typed columns).
`schema_columns`	textarea	No	Declared columns for `blueprint` mode, one per line as `name[:type[:source]]`.
`batch_size`	number	No	Events per file; above 1 consolidates events into one file per partition-hour. Typical: `1000`.
`batch_max_wait_ms`	number	No	Flush a batch after this long even if it is not full. Default `60000`.
Leave `access_key_id` and `secret_access_key` empty to use the default AWS credential chain (for example an IAM role for the delivery worker service account on EKS).

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Amazon S3 or filter by Cloud Storage.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/amazon_s3/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Amazon S3",
    "variant": "default",
    "config": {
      "bucket_name": "datafly-events-production",
      "region": "us-east-1",
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
      "prefix": "events/",
      "storage_class": "STANDARD",
      "server_side_encryption": "AES256"
    },
    "delivery_mode": "server_side"
  }'

What Signal writes

By default each event is one object (set batch_size to consolidate events into one file per partition-hour). Object names use Hive-style partitions of the event time in UTC:

s3://<bucket>/<prefix>/yyyy=2026/mm=09/dd=23/hh=09/<pipeline_id>/<message_id>.ndjson

The extension and content type follow file_format: .ndjson (application/x-ndjson), .csv (text/csv) or .parquet (application/vnd.apache.parquet).

JSON (default)

One line per event: the Signal event as JSON, the same document the streaming destinations send.

Field	Contents
`event`, `type`	Event name (`purchase`, `page`, …) and type (`track`, `page`, `identify`, …).
`anonymous_id`, `user_id`, `device_id`	Visitor identifiers (`user_id` and `device_id` only when set).
`message_id`, `event_id`	Signal’s message ID (use it to deduplicate) and the client event ID.
`timestamp`, `sent_at`, `received_at`	RFC 3339 UTC times: event, client send, Signal receipt.
`org_id`, `pipeline_id`, `vendor_type`	Where the event came from and which destination type wrote it.
`context`	The collected context as sent: `ip`, `user_agent`, `locale`, `page`, `campaign`, `consent`, `geo`, `device`, `library`, and any partner attributes. Written in full; the blueprint does not filter it (#1414).
`click_ids`, `collected_ids`	Ad click IDs and ID-provider values, when present.
`properties`	Only the fields the blueprint maps (below).

Only what the blueprint maps is written. The properties object holds the fields your installed blueprint maps and nothing else (the row contract): the global block under @root (copies of the envelope fields, UTM parameters, the analytics and marketing consent flags, and email_sha256, phone_sha256, first_name_sha256, last_name_sha256, name_sha256), and the event section’s fields at the top level (transaction_id, value, currency, items, …). Identify traits and client properties the blueprint does not map are never written. An event with no section in the blueprint still arrives, with the @root block only: add a section for any other event whose properties you need.

CSV and Parquet, canonical schema

A header row and one row per event in the 16 fixed columns the SQL warehouses use, all text (timestamps ISO-8601 in UTC): message_id, event_type, event_name, anonymous_id, user_id, ip_address, user_agent, page_url, page_path, page_referrer, org_id, pipeline_id, event_timestamp, received_at, properties_json (the mapped properties object above) and context_json. A purchase (digests shortened):

message_id,event_type,event_name,anonymous_id,user_id,ip_address,user_agent,page_url,page_path,page_referrer,org_id,pipeline_id,event_timestamp,received_at,properties_json,context_json
msg-s1-amazon_s3-2,track,purchase,anon-s1-7f3a,crm-4711,203.0.113.9,Mozilla/5.0 (Macintosh),https://shop.example/checkout,/checkout,,org-s1,pl-s1,2026-09-23T09:30:02.000Z,2026-09-23T09:30:02.000Z,"{""@root"":{""anonymous_id"":""anon-s1-7f3a"",""email_sha256"":""86e0b9e5..."", ...},""currency"":""GBP"",""items"":[...],""transaction_id"":""SIG-ORDER-1001"",""value"":84.5}","{""ip"":""203.0.113.9"",""page"":{...},""campaign"":{...}}"

Blueprint schema mode

With schema_mode set to blueprint, each file has one typed column per field instead (File formats and schema). Inferred columns are message_id, event_name and event_timestamp plus the event section’s top-level fields. The global block’s fields (identifiers, digests, UTM, consent) sit under @root and are not inferred as columns; declare them with an @root. source:

message_id
event_name
event_timestamp:timestamp
user_id:string:@root.user_id
email_sha256:string:@root.email_sha256
transaction_id
value:double
currency

which writes, for the same purchase:

message_id,event_name,event_timestamp,user_id,email_sha256,transaction_id,value,currency
msg-s1-amazon_s3-2,purchase,2026-09-23T09:30:02.000Z,crm-4711,86e0b9e56c17cc4d12387e1949b85053fbe73bc3ce5a1188713a9d300cc6133d,SIG-ORDER-1001,84.5,GBP

These files were produced by Signal’s Amazon S3 writer against LocalStack 4.4 on 2026-09-23 (batch S1 review), not written from memory.

Stored event context

Alongside the fields your blueprint maps (where email, phone, names and address are stored as normalised SHA-256 by default), each file row carries the event’s collected context exactly as Signal received it. The blueprint does not filter or hash the context: this is deliberate (decision 2026-09-23), so the context is complete for attribution, fraud and debugging queries, and it is listed here so you can decide what to keep.

Stored	What it contains
Visitor request	IP address, user agent, `Accept-Language`, locale and timezone
Page	URL including its query string, path, title and referrer
Campaign	UTM source, medium, campaign, term and content
Consent	The visitor’s consent state
Location	GeoIP country, region and city, and the network (ASN)
Device	Screen size; for app events the OS and version, manufacturer, model, advertising ID and IDFV when the app collected them
Vendor IDs	Cookie and click identifiers in `vendor_ids` (for example the `_fbp` browser ID, `gclid`, `ttclid`, the GA client ID)
Session and library	Session ID, number and engagement time; the collector’s name and version
Partner attributes	Profile attributes from partner enrichment (`partner_attrs`)

In each NDJSON line these are the context object, with anonymous_id, user_id, device_id, message_id, event_id, click_ids and collected_ids beside the mapped properties. In CSV and Parquet with the canonical schema they are the ip_address, user_agent, page_url, page_path and page_referrer columns and, in full, the context_json column. In blueprint schema mode only the columns you declare are written.

The IP address, device and advertising IDs, and cookie and click identifiers are personal data under UK and EU GDPR. To keep any of them out of storage, strip, mask or hash the field for every integration with the organisation data layer, apply URL redaction to page URLs, or remove it in the installed blueprint; and set the storage’s retention to match your privacy notice. See Personal data by integration.

The integration’s consent category in Signal decides which events reach Amazon S3; events that fail it are never written. Every row also carries the visitor’s consent state (context.consent / context_json, and consent_analytics / consent_marketing in @root) so downstream queries can apply their own rules.

Testing

Enable the integration in Signal and trigger a test event on your website.
Open the S3 console and navigate to your bucket.
Browse to the prefix path and verify that event files are appearing.
Download a file and inspect the contents to confirm the event data is correct.
In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

Problem	Solution
Events not appearing in S3	Verify the bucket name, region, and prefix are correct. Check that the bucket exists.
`AccessDenied`	The IAM user lacks `s3:PutObject` permission on the bucket. Update the IAM policy.
`NoSuchBucket`	The bucket does not exist. Verify the bucket name (names are case-sensitive and globally unique).
`InvalidAccessKeyId`	The access key ID is incorrect or the IAM user has been deleted. Verify credentials.
`SignatureDoesNotMatch`	The secret access key is incorrect. Regenerate the access key pair.
Files appearing but empty	Check the batch settings. Events are buffered before flushing to files.
Wrong region error	S3 bucket region must match the `region` config field. Check the bucket’s actual region in the S3 console.

Visit Amazon S3 documentation for full API reference, lifecycle policies, and cost optimisation guides.