File formats and schema

The four cloud storage destinations (Amazon S3, Google Cloud Storage, Azure Blob Storage and Azure Data Lake Storage Gen2) share one file writer. Everything on this page applies to all four.

Settings

Object layout

Objects are written under Hive-style partitions of the event timestamp (UTC), which Athena, BigQuery external tables, Synapse Serverless, Databricks and Trino read without an ETL pass:

<prefix>/yyyy=2026/mm=09/dd=15/hh=10/<pipeline_id>/<message_id>.parquet

With batching on, the file is named after the first event in it. Signal acknowledges each event only once the file containing it has been uploaded, so a restart mid-batch re-delivers rather than loses events. For Databricks, Spark and Auto Loader set batch_size (typically 1000) to avoid the small-files problem.

Parquet compression

Parquet files are compressed with Snappy unless you choose otherwise. Snappy is what Spark and Databricks write themselves; every Parquet reader decompresses it transparently, because the codec is recorded for each column chunk in the file footer. Choose zstd for smaller files at slightly higher CPU cost, or none if a downstream tool cannot read compressed Parquet.

Canonical schema (default)

CSV and Parquet files carry the same 16 columns as the SQL warehouse destinations, all as strings (timestamps as ISO-8601):

message_id, event_type, event_name, anonymous_id, user_id, ip_address, user_agent, page_url, page_path, page_referrer, org_id, pipeline_id, event_timestamp, received_at, properties_json, context_json

event_name is the original Signal event name. Blueprint output is inside properties_json. JSON files contain the full Signal event, one per line.

Blueprint schema

Set schema_mode to blueprint to write one typed column per field of your blueprint output instead. JSON, CSV and Parquet then carry the same columns: JSON as one flat object per line, CSV with those columns as the header.

Declared columns (recommended)

List the columns in schema_columns, one per line, as name[:type[:source]]:

message_id
event_name
source_event_name
order_id
value:double
quantity:int64
member:boolean:is_member
event_time:timestamp:event_timestamp

• Types: string (default), int64, double, boolean, timestamp. Parquet timestamps are TIMESTAMP(MICROS) in UTC. Every column is nullable.
• Source defaults to the column name. It can be any field your blueprint writes (dotted paths such as shipping.city reach into objects; arrays and objects are written as JSON text), or one of the built-in fields below.
• Values that do not fit the declared type are written as null. For example, 19.99 in an int64 column is null, not truncated.
• The list can also be a JSON array: [{"name":"value","type":"double","source":"order_total"}].

Declared columns give every file the same schema, which is what Auto Loader and external tables expect.

Inferred columns

With schema_columns empty, each file carries message_id, event_name and event_timestamp, followed by every field the blueprint mapped for the events in that file, in alphabetical order. Types come from the values: numbers become double, and a field seen with conflicting types in one file becomes string. Events with different fields share a file, and missing values are null. An event re-delivered after a failed upload keeps the record of which fields the blueprint mapped, so a retry does not add unmapped properties to its file. Because the column set can differ between files, prefer declared columns for production tables.

Built-in fields

A field your blueprint writes takes precedence over a built-in field of the same name, except event_name and source_event_name.