DEV Community

98IP Proxy
98IP Proxy

Posted on Fully Autonomous

Stop Rotating Proxies When a Crawl Fails a Content-Use Check

#webscraping #proxy #devops

Disclosure: I work with 98IP. This is a vendor-neutral operational pattern for authorized data collection.

Cloudflare's Browser Rendering /crawl endpoint now respects the Content Signals use directive. The caller can declare contentUse as reference or full; when the target's published ceiling is stricter, the request is rejected.

The most important engineering consequence is not the new parameter. It is the need to stop treating every rejected crawl as a bad proxy.

Add a policy result to your taxonomy

A useful classifier should separate at least:

proxy_auth_failed
dns_failed
tunnel_failed
transport_timeout
rate_limited
bot_challenge
content_use_denied
application_not_authorized
render_failed
success
Enter fullscreen mode Exit fullscreen mode

content_use_denied is deterministic policy feedback. Rotating the exit, changing the user agent or retrying at higher concurrency does not repair the mismatch.

Keep four layers in every observation 

{
  "identity": {
    "crawler": "catalog-reference-bot",
    "version": "2026.09.02"
  },
  "permission": {
    "approved_purpose": "reference",
    "declared_content_use": "reference",
    "observed_policy_at": "2026-09-02T03:00:00Z"
  },
  "transport": {
    "route": "us-east-residential-a",
    "family": "ipv4",
    "session_mode": "sticky"
  },
  "outcome": {
    "category": "content_use_denied",
    "retry": false,
    "bytes": 0
  }
}
Enter fullscreen mode Exit fullscreen mode

Do not put proxy passwords, cookies, tokens or signing keys in this record.

Identity tells you who called. Permission tells you what the caller was allowed to do. Transport tells you how the request traveled. Outcome tells you what happened. Combining them into proxy_failed=true destroys the evidence needed for a correct response.

Retry by cause, not by status code

A safe retry function can be boring:

function retryDecision(result) {
  if (result.category === 'content_use_denied') return 'stop';
  if (result.category === 'application_not_authorized') return 'stop';
  if (result.category === 'proxy_auth_failed') return 'inspect_config';
  if (result.category === 'rate_limited') return 'respect_retry_after';
  if (result.category === 'transport_timeout' && result.attempt < 2) {
    return 'bounded_retry';
  }
  return 'stop';
}
Enter fullscreen mode Exit fullscreen mode

The exact logic depends on your workload, but policy denial should never enter a proxy-rotation loop.

Test with an authorized policy canary

Before production rollout:

  1. Use a site you own or have explicit permission to test.
  2. Publish controlled content-use variants.
  3. Send bounded reference and full jobs.
  4. Confirm compatible jobs proceed.
  5. Confirm incompatible jobs produce content_use_denied.
  6. Verify no proxy rotation occurs after that result.
  7. Check that pages, sessions and timers are cleaned up.

This canary validates policy logic without generating unwanted traffic against third-party sites.

Keep provider scoring independent

Measure proxy quality only on permitted workloads and controlled targets:

  • connect and TLS success;
  • time to first byte;
  • p50 and p95 latency;
  • throughput;
  • unexpected rotation;
  • address-family behavior;
  • ASN and prefix concentration;
  • cost per useful authorized result.

A destination rejecting a declared purpose is not evidence that the exit is slow, unstable or incorrectly located.

Operational checklist

  • Every crawler has a named purpose and owner.
  • contentUse is set from approved workload metadata.
  • Policy evidence is timestamped.
  • Policy denials have a no-retry rule.
  • Provider metrics exclude deterministic policy denials.
  • Diagnostic records exclude credentials and personal data.
  • Changes are tested on an authorized canary.
  • Robots directives, terms, privacy and rate limits remain separate gates.

What not to do

Do not silently relabel a full workload as reference. Do not rotate residential exits to search for a policy inconsistency. Do not report policy denials as an IP-quality defect. And do not assume that one compatible machine-readable signal grants permission for every purpose.

For more vendor-neutral proxy operations guidance, visit 98IP.

Proxies are a transport tool for authorized work, not a method for overriding publisher decisions.

Top comments (0)