IntegrationsDatabasesElasticsearch

Elasticsearch

Datafly Signal delivers events to Elasticsearch for powerful full-text search, real-time analytics, log aggregation, and observability dashboards with Kibana.

Prerequisites

Before configuring Elasticsearch in Signal, you need an Elasticsearch cluster (self-hosted or Elastic Cloud), an index, and an API key for authentication.

Set Up an Elasticsearch Cluster

You have two options:

Option A: Elastic Cloud (Managed)

  1. Sign up at cloud.elastic.co.
  2. Click Create deployment.
  3. Choose a cloud provider and region.
  4. Select a deployment size based on your expected data volume.
  5. Click Create deployment.
  6. Note the Cloud ID from the deployment overview and the Elasticsearch endpoint URL.

Option B: Self-Hosted

  1. Install Elasticsearch using the official guide.
  2. Ensure the cluster is accessible from your Signal infrastructure.
  3. Note the Endpoint URL (e.g. https://elasticsearch.example.com:9200).

Create an Index

Create an index with an appropriate mapping for event data:

curl -X PUT "https://your-cluster:9200/datafly-events" \
  -H "Content-Type: application/json" \
  -d '{
    "mappings": {
      "properties": {
        "event_id": { "type": "keyword" },
        "type": { "type": "keyword" },
        "event": { "type": "keyword" },
        "anonymous_id": { "type": "keyword" },
        "user_id": { "type": "keyword" },
        "timestamp": { "type": "date" },
        "received_at": { "type": "date" },
        "sent_at": { "type": "date" },
        "context": { "type": "object", "enabled": true },
        "properties": { "type": "object", "enabled": true },
        "traits": { "type": "object", "enabled": true },
        "source_id": { "type": "keyword" },
        "integration_id": { "type": "keyword" }
      }
    }
  }'

Using keyword type for ID fields and date type for timestamps enables efficient filtering and aggregations. The object type for context, properties, and traits allows flexible nested field querying.

Generate an API Key

  1. In Kibana, go to Stack Management > API Keys > Create API key. Or use the API:
curl -X POST "https://your-cluster:9200/_security/api_key" \
  -H "Content-Type: application/json" \
  -u "elastic:your_password" \
  -d '{
    "name": "datafly-signal",
    "role_descriptors": {
      "datafly_writer": {
        "cluster": [],
        "index": [
          {
            "names": ["datafly-events"],
            "privileges": ["write", "create_index"]
          }
        ]
      }
    }
  }'
  1. The response contains an encoded field — this is the Base64-encoded API key to use in Signal.
⚠️

Store the API key securely. Create keys with the minimum required privileges (write access to the specific index only).

Configuration

FieldTypeRequiredDescription
endpointstringYesThe Elasticsearch cluster URL (e.g. https://my-cluster.es.us-east-1.aws.elastic-cloud.com:9243).
indexstringYesThe target index name to write documents to.
api_keysecretYesBase64-encoded API key for authentication.
cloud_idstringNoThe Elastic Cloud deployment ID. Alternative to endpoint for Elastic Cloud deployments.

Signal Setup

Quick Setup

  1. Navigate to Integrations in the sidebar.
  2. Open the Integration Library tab.
  3. Find Elasticsearch or filter by Cloud Storage.
  4. Click Install, select a variant if available, and fill in the required fields.
  5. Click Install Integration to create the integration with a ready-to-use default blueprint.

API Setup

curl -X POST http://localhost:8084/v1/admin/integration-catalog/elasticsearch/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Elasticsearch",
    "variant": "default",
    "config": {
      "endpoint": "https://my-cluster.es.us-east-1.aws.elastic-cloud.com:9243",
      "index": "datafly-events",
      "api_key": "BASE64_ENCODED_API_KEY"
    },
    "delivery_mode": "server_side"
  }'

Testing

  1. Enable the integration in Signal and trigger a test event on your website.
  2. In Kibana, go to Discover and select the datafly-events index pattern.
  3. Verify that event documents are appearing with the correct fields.
  4. Or query directly:
curl "https://your-cluster:9200/datafly-events/_search?size=10&sort=timestamp:desc" \
  -H "Authorization: ApiKey YOUR_API_KEY"
  1. In Signal, check the Live Events view to confirm delivery status shows as successful.

Troubleshooting

ProblemSolution
Events not appearing in the indexVerify the endpoint, index name, and API key are correct.
Unauthorized (401)The API key is invalid or has been revoked. Generate a new API key.
Forbidden (403)The API key lacks write permission on the index. Create a new key with write privileges on the target index.
index_not_found_exceptionThe index does not exist. Create it first, or enable create_index privilege on the API key so it can be auto-created.
Connection timeoutVerify the endpoint URL is correct and accessible from Signal’s network. Check firewall rules.
Mapping conflictsIf fields have different types than expected, check the index mapping. Consider using an index template for consistent mappings.
Cluster health yellow/redCheck cluster health with GET _cluster/health. Yellow means replicas are unassigned; red means primary shards are missing.

Visit Elasticsearch documentation for full API reference, index lifecycle management, and Kibana dashboard setup.

ClickHouseGoogle Bigtable