Elasticsearch

Datafly Signal indexes first-party events into Elasticsearch for full-text search, real-time analytics, log aggregation, and observability dashboards with Kibana.

Prerequisites

Before configuring Elasticsearch in Signal, you need an Elasticsearch cluster (self-hosted or Elastic Cloud), an index, and an API key for authentication.

Set Up an Elasticsearch Cluster

You have two options:

Option A: Elastic Cloud (Managed)

Sign up at cloud.elastic.co.
Click Create deployment.
Choose a cloud provider and region.
Select a deployment size based on your expected data volume.
Click Create deployment.
Note the Cloud ID from the deployment overview and the Elasticsearch endpoint URL.

Option B: Self-Hosted

Install Elasticsearch using the official guide.
Ensure the cluster is accessible from your Signal infrastructure.
Note the Endpoint URL (e.g. https://elasticsearch.example.com:9200).

Create an Index

Create an index with an appropriate mapping for event data:

curl -X PUT "https://your-cluster:9200/datafly-events" \
  -H "Content-Type: application/json" \
  -d '{
    "mappings": {
      "properties": {
        "event_id": { "type": "keyword" },
        "type": { "type": "keyword" },
        "event": { "type": "keyword" },
        "anonymous_id": { "type": "keyword" },
        "user_id": { "type": "keyword" },
        "timestamp": { "type": "date" },
        "received_at": { "type": "date" },
        "sent_at": { "type": "date" },
        "context": { "type": "object", "enabled": true },
        "properties": { "type": "object", "enabled": true },
        "traits": { "type": "object", "enabled": true },
        "source_id": { "type": "keyword" },
        "integration_id": { "type": "keyword" }
      }
    }
  }'

Using keyword type for ID fields and date type for timestamps enables efficient filtering and aggregations. The object type for context, properties, and traits allows flexible nested field querying.

Generate an API Key

In Kibana, go to Stack Management > API Keys > Create API key. Or use the API:

curl -X POST "https://your-cluster:9200/_security/api_key" \
  -H "Content-Type: application/json" \
  -u "elastic:your_password" \
  -d '{
    "name": "datafly-signal",
    "role_descriptors": {
      "datafly_writer": {
        "cluster": [],
        "index": [
          {
            "names": ["datafly-events"],
            "privileges": ["write", "create_index"]
          }
        ]
      }
    }
  }'

The response contains an encoded field — this is the Base64-encoded API key to use in Signal.

⚠️

Store the API key securely. Create keys with the minimum required privileges (write access to the specific index only).

Configuration

Field	Type	Required	Description
`addresses`	string	One of these	Comma-separated list of cluster URLs (e.g. `https://es-1:9243,https://es-2:9243`). Also accepts a single `endpoint` value.
`cloud_id`	string	One of these	The Elastic Cloud deployment ID. Alternative to `addresses` for Elastic Cloud deployments.
`index`	string	Yes	The target index name to write documents to.
`api_key`	secret	One of these	Base64-encoded API key (preferred).
`username`	string	One of these	Basic auth username — use only when API keys aren’t available.
`password`	secret	One of these	Basic auth password.

Signal Setup

Quick Setup

Navigate to Integrations in the sidebar.
Open the Integration Library tab.
Find Elasticsearch or filter by Database.
Click Install, select a variant if available, and fill in the required fields.
Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/elasticsearch/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Elasticsearch",
    "variant": "default",
    "config": {
      "addresses": "https://my-cluster.es.us-east-1.aws.elastic-cloud.com:9243",
      "index": "datafly-events",
      "api_key": "BASE64_ENCODED_API_KEY"
    },
    "delivery_mode": "server_side"
  }'

Schema

Each event becomes one indexed document. The recommended mapping (shown in Prerequisites) uses:

Field	Mapping type	Notes
`event_id`	`keyword`	Unique per event. Use as the document `_id`.
`type`	`keyword`	Event type.
`event`	`keyword`	Event name.
`anonymous_id`	`keyword`	First-party visitor identifier.
`user_id`	`keyword`	Logged-in user identifier (optional).
`timestamp`	`date`	Client event time — use as the index time field.
`received_at`	`date`	Time Signal received the event.
`sent_at`	`date`	Time the document was indexed.
`context`	`object`	Page, device, user agent, consent metadata.
`properties`	`object`	Custom event properties.
`traits`	`object`	User traits.
`source_id`	`keyword`	Pipeline source identifier.
`integration_id`	`keyword`	Signal integration identifier.

For high-volume deployments, use a data stream with ILM rollover instead of a single index.

Elasticsearch is a first-party destination under your control. The default blueprint forwards all events. Apply consent filtering via pipeline transforms, ingest pipelines, or filtered Kibana data views over context.consent if needed.

Testing

Enable the integration in Signal and trigger a test event on your website.
In Kibana, go to Discover and select the datafly-events index pattern.
Verify that event documents are appearing with the correct fields.
Or query directly:

curl "https://your-cluster:9200/datafly-events/_search?size=10&sort=timestamp:desc" \
  -H "Authorization: ApiKey YOUR_API_KEY"

In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

Problem	Solution
Events not appearing in the index	Verify the endpoint, index name, and API key are correct.
`Unauthorized` (401)	The API key is invalid or has been revoked. Generate a new API key.
`Forbidden` (403)	The API key lacks write permission on the index. Create a new key with `write` privileges on the target index.
`index_not_found_exception`	The index does not exist. Create it first, or enable `create_index` privilege on the API key so it can be auto-created.
Connection timeout	Verify the endpoint URL is correct and accessible from Signal’s network. Check firewall rules.
Mapping conflicts	If fields have different types than expected, check the index mapping. Consider using an index template for consistent mappings.
Cluster health yellow/red	Check cluster health with `GET _cluster/health`. Yellow means replicas are unassigned; red means primary shards are missing.

Visit Elasticsearch documentation for full API reference, index lifecycle management, and Kibana dashboard setup.