Bot Filtering

Bot filtering detects and removes non-human traffic before events reach your integrations. It runs as Step 3 of the Organisation Data Layer, scoring each event across multiple detection layers and either passing, flagging, or dropping it based on configurable thresholds.

All detection happens server-side — no changes to the Datafly.js collector are required.

How scoring works

Each event is scored from 0.0 (clean) to 1.0 (bot). The score is built by combining signals from three detection layers. Once the final score is computed, the event is classified into one of three actions:

Detection layers

Layer 1: IAB Pattern Matching

Matches the event’s user-agent string against 160+ known bot patterns sourced from the IAB/ABC International Spiders & Bots List. This includes:

• Search engine crawlers (Googlebot, Bingbot, etc.)
• SEO tools (Ahrefs, SEMrush, Moz, etc.)
• Monitoring services (Pingdom, UptimeRobot, etc.)
• Known scrapers and headless browsers

Scoring: A direct match against a known bot pattern scores 1.0 — the event is dropped immediately regardless of thresholds.

You can add your own custom patterns under Settings → Bot Filtering → Detection Layers → IAB Patterns. Custom patterns support wildcards (e.g. my-internal-bot*).

Layer 2: Signal Validation

Checks for the presence of browser signals that real users produce but bots typically lack:

A headless bot with no screen dimensions, no timezone, and no locale would score 0.5 — enough to be flagged but not dropped at default thresholds.

Each signal check can be individually enabled or disabled.

Layer 3: IP Classification

Classifies the source IP address against known infrastructure ranges:

Datacenter IPs are common for automated tools and scrapers. A datacenter IP combined with a missing browser signal would push the score above the default block threshold.

Configuration

Bot filtering is configured per organisation under Settings → Bot Filtering in the Management UI.

Master toggle

Enable or disable bot filtering entirely. When disabled, all events pass through unscored.

Thresholds

Both values must be between 0.0 and 1.0, and the flag threshold must be lower than the block threshold.

Detection layers

Each detection layer can be independently enabled or disabled:

• IAB Patterns — user-agent pattern matching against known bots
• Signal Validation — checks for screen dimensions, timezone, locale, and user-agent
• IP Classification — datacenter, Tor, and Spamhaus IP detection

Behaviour

Allowlists

You can exempt specific traffic from bot detection:

• Allowlisted User Agents — user-agent strings or wildcard patterns (e.g. Datadog Synthetics*) that should bypass bot filtering
• Allowlisted IP Ranges — CIDR notation (e.g. 10.0.0.0/8, 192.168.1.0/24) for trusted internal or monitoring IPs

API

Bot filtering can also be managed via the Management API:

# Get current config
curl -H "Authorization: Bearer $TOKEN" \
  https://your-api.example.com/v1/admin/org/bot-filtering
 
# Update config
curl -X PUT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "flag_threshold": 0.3,
    "block_threshold": 0.7,
    "log_blocked_events": false,
    "server_side_bypass": true,
    "layers": {
      "iab_patterns": { "enabled": true, "custom_patterns": [] },
      "signal_validation": {
        "enabled": true,
        "require_screen_dimensions": true,
        "require_timezone": true,
        "require_user_agent": true
      },
      "ip_classification": { "enabled": true }
    },
    "allowlisted_user_agents": [],
    "allowlisted_cidrs": []
  }' \
  https://your-api.example.com/v1/admin/org/bot-filtering

Monitoring bot traffic

Once bot filtering is enabled, you can monitor detected bot traffic in two places:

1. Observe → Bot Traffic — dedicated page showing bot events over time, bot rate percentage, and the top event names being filtered
2. Observe → Analytics — the main analytics page includes bot_filtered as a status in the event breakdown

Bot event logging

When Log blocked events is enabled, dropped bot events are published to a bot-events Kafka topic. This allows you to:

• Attach a data warehouse integration (e.g. BigQuery, Snowflake) to analyse bot patterns
• Build custom alerting on bot traffic spikes
• Audit what’s being filtered

The bot events retain all original event data plus the computed bot score and detection details.