DEV Community

98IP Proxy
98IP Proxy

Posted on AI-assisted

Treat Crawler Identity as a Deployable Control Plane

#webdev #devops #security #automation

Disclosure: I work with 98IP. This post is an engineering guide, not a claim that verification grants permission to crawl any site.

A crawler does not have one identity. It has a bundle of assertions: a user agent, an operator, an IP range, a reverse-DNS pattern, a purpose, and a contact path. When those assertions are maintained in different tickets and repositories, they drift. The result is a bot that is legitimate internally but indistinguishable from an impersonator at the edge.

The fix is to manage crawler identity like a deployable control plane.

Start with an identity manifest

Keep the public assertions in a versioned manifest owned by the same team that ships the crawler:

schema_version: 1
bot:
  name: catalog-observer
  operator: example-operator
  purpose: market-research
  contact: crawler-ops@example.invalid
network:
  user_agents:
    - "CatalogObserver/3.4"
  ip_sets:
    - production-egress-v2
  reverse_dns_suffix: crawler.example.invalid
authentication:
  web_bot_auth: required
policy:
  robots_txt: enforce
  default_rate_per_host_per_minute: 12
  stop_on_401_403_429: true
Enter fullscreen mode Exit fullscreen mode

Do not put secrets in this file. It should describe claims that a destination can verify, not credentials that let someone impersonate the crawler.

Validate identity before traffic leaves

Add four checks to the release pipeline:

  1. User-agent check: the runtime string must exactly match the manifest.
  2. Egress check: every production address must belong to the declared set, and stale addresses must be removed.
  3. DNS check: forward and reverse resolution must agree where reverse DNS is part of the identity contract.
  4. Cryptographic check: if Web Bot Auth or another signed mechanism is used, verify clock skew, key rotation, and failure behavior in staging.

These tests should fail closed. A release that changes an identity surface without updating the manifest should not ship.

Observe the verifier's view

Internal health checks are insufficient. Run a small canary from each production egress region and record what an external verifier sees:

{
  "region": "eu-west",
  "observed_user_agent": "CatalogObserver/3.4",
  "forward_reverse_dns_consistent": true,
  "ip_set_version": "production-egress-v2",
  "auth_signature_valid": true,
  "policy_response": 200
}
Enter fullscreen mode Exit fullscreen mode

Alert on mismatches, not merely on request failure. A 200 from one destination does not prove that the crawler is consistently identifiable elsewhere.

Separate verification from permission

Verification answers “is this traffic really from the stated operator?” Permission answers “does this site allow this activity?” They are different controls.

Before collecting data, evaluate the destination's terms, robots instructions, authentication boundaries, applicable law, and rate expectations. Stop or reduce traffic on explicit denial and on repeated 401, 403, or 429 responses. A directory listing or verified badge must never become a bypass rule.

Roll changes like network changes

Identity changes deserve staged rollout:

  • publish the new assertions first;
  • wait for DNS and directory propagation;
  • canary from one egress pool;
  • compare denial rate, authentication failures, and challenge rate;
  • expand only when the verifier's view matches the manifest;
  • retain a rollback path for the previous identity version.

The useful metrics are identity mismatch rate, verified-request ratio, policy-denial rate, stale-IP count, and time from operator change to external consistency. Raw request volume is not an identity metric.

Release checklist

  • [ ] Manifest reviewed by crawler and security owners
  • [ ] User agent and egress sets match production
  • [ ] Reverse DNS passes forward-confirmed checks
  • [ ] Signing keys rotate without overlap gaps
  • [ ] Contact and purpose remain accurate
  • [ ] Destination permission checks still run independently
  • [ ] Canary evidence is stored with the release
  • [ ] Rollback version is known

Crawler identity becomes reliable when it is testable, observable, and reversible. Treating it as a control plane reduces accidental blocking and makes impersonation easier to detect—without weakening a site owner's control.

I work with 98IP, where we build proxy infrastructure and operational guidance for compliant data workflows: https://en.98ip.com/?k=dev

AI-assisted disclosure: AI supported drafting and editing; the operational model and final review were completed for this post.

Top comments (0)