IntegrationsData Dictionary

Data Dictionary

An integration’s Data dictionary tab answers “what does this vendor receive?” and “what is in my table?”: every field, column or JSON key the integration sends or writes on a pipeline, for each event the pipeline receives or derives.

It is computed by the same code that delivers. Datafly Signal takes the shape of the events the pipeline receives from Schema Explorer, runs each one through the integration’s blueprint with the delivery engine, and lays the result out exactly as the destination writes it: the request body for a vendor, the row for a database or warehouse, the file layout for cloud storage. Nothing is stored, and the dictionary never shows a connection setting or a value from your events: it is built from field names and types, with invented values.

Open it

On an integration’s page, choose Data dictionary, then the pipeline. The dictionary is for the revision live on that pipeline; pick another revision to see what a draft will send before you publish it. In Schema Explorer, the integrations under Handled by open the dictionary on that pipeline.

What it shows

The summary says what the integration writes and how:

What it writesA request to the vendor, a database table row, a storage file, a stream or queue record, or a stored record, and the layout (for storage: the 16-column layout, declared columns, columns from the blueprint, or the whole event as JSON).
RevisionThe revision shown, and the one live on the pipeline.
Output contractStrict: only what the blueprint maps. Observe (vendors) or legacy (databases and storage installed before strict was the default): fields the blueprint does not map are also sent, without personal data.
ConsentThe consent category that governs this integration on the pipeline. Every field is sent only for visitors whose consent covers it.

Each event shows what the integration does with it: Sent, Sent when its filter holds (with the filter), Not sent (the blueprint does not handle it) or Dropped (a required field is missing). An event made by one of the pipeline’s event derivation rules says which event it is derived from.

For each event, every field shows:

ColumnMeaning
FieldThe payload path, the column, or a key inside a JSON column such as properties_json.
TypeText, number, true/false, object or list; for typed storage columns, the column type.
SourceWhere the value comes from: the event field a blueprint row maps, a template, a fixed value, a value map, the event’s other properties, a script, the event itself, or Added by the vendor spec for a field the vendor’s delivery spec writes on its own (an account ID from the connection, the event name).
Transform and conditionTransforms the blueprint applies (hashing, lowercasing and so on), hashing or masking by Data Governance, and the condition of a row that only writes for some events.
Personal dataPersonal data, approximate location, a pseudonymous identifier, hashed, or no: from your Data Governance flags, Datafly Signal’s standard fields, and the personal-data name rules (email, phone, name, address and so on). Hover for the reasons.
DescriptionYour Data Governance description of the field when there is one, then the description of an expected field in Schema Explorer, then Datafly Signal’s own description of its standard fields. A field copied from a standard field (directly, or as the first choice of a COALESCE) takes that field’s description; otherwise a standard name the blueprint writes, such as page_url or email, has one too.

Fields are marked Conditional when a blueprint row’s condition or an event filter decides whether they are written, and Blocked by Data Governance when a block rule names them: blocked fields are removed wherever they appear, before any integration receives the event, and delivery enforces it again. A field computed by a script is shown as Computed by script.

The dictionary lists the fields Schema Explorer has seen on the pipeline’s events. A field that has not arrived yet is not listed: add it to Schema Explorer’s expected fields to include it. A dictionary covers the 200 busiest events of a pipeline.

Destinations

DestinationWhat the dictionary lists
Vendors (HTTP)Every field of the request body for one event, including those the vendor’s delivery spec adds. Request headers and URL parameters are not listed.
SQL databases and warehouses (PostgreSQL, MySQL, Amazon Redshift, Snowflake, Azure SQL and Synapse, ClickHouse, Databricks SQL, Google Cloud Spanner)The 16 columns every row has, and the keys inside properties_json and context_json.
Cloud storage (Amazon S3, Azure Blob Storage, Azure Data Lake Storage, Google Cloud Storage)The file’s columns for CSV and Parquet (the 16-column layout, or the blueprint schema mode’s declared or inferred columns), or the keys of each NDJSON line. With columns from the blueprint (not declared), fields the blueprint’s global section writes have no column; declare the columns to write them.
Streams and queues (Kafka, Amazon Kinesis, Firehose and SQS, Google Pub/Sub, Azure Event Hubs, RabbitMQ, Azure Data Explorer)The keys of each record: the whole event as JSON, with the properties the blueprint writes. For Azure Data Explorer, the table’s columns come from your ingestion mapping.
Other stores (Google BigQuery, MongoDB, Amazon DynamoDB, Azure Cosmos DB, Elasticsearch, Google Firestore, Google Bigtable)The properties the blueprint writes. How the destination stores them alongside the event’s own fields is not shown.

Export

Export CSV downloads one row per event and field, with every column above. Export JSON Schema downloads the dictionary as JSON Schema, one $defs entry per event, in the same shape as Schema Explorer’s export.

The dictionary is also available from the Management API: GET /v1/admin/pipelines/{pid}/integrations/{id}/data-dictionary.