Our Legitimate Bot Was Blocked. The User-Agent Was Not the Problem.

When our authorized monitor started receiving blocks, the first suggestion was predictable:

“Change the User-Agent.”

It did not help, because the system was not evaluating one string. It was evaluating whether the request, connection, browser, session, and behavior described one believable client with an allowed purpose.

Anti-bot detection is a classification system, not a header checklist.

Understanding that model leads to a much safer response than cycling identities until something gets through.

What the detector can see

Modern anti-bot systems combine several signal families.

Network signals

IP reputation, network type, location, request concentration, and recent history can all affect a decision.

Transport consistency

TLS and HTTP behavior may reveal that headers claim one browser while the connection behaves like another client.

Browser execution

JavaScript APIs, feature support, timing, fonts, WebGL, and other environment properties help a site understand the runtime.

Session identity

Cookies, tokens, navigation order, and IP continuity describe whether requests belong to one coherent session.

Behavior and business rules

Request cadence, repeated paths, impossible navigation, and high-impact actions often matter more than a single fingerprint value.

The detector rarely needs certainty. It only needs enough risk to allow, challenge, slow, or deny the request.

Why legitimate automation gets caught

Our monitor was authorized, but its behavior still looked bad:

• many workers started at the same second;
• every worker hit the same endpoint;
• retries had no jitter;
• cookies persisted while IPs changed;
• and failed jobs retried more aggressively than successful jobs.

Nothing in that list requires malicious intent. It still creates an operational pattern that defensive systems are designed to stop.

Classify the response before retrying

def classify(status: int, body: str) -> tuple[str, bool]:
    text = body.lower()

    if "verify you are human" in text or "captcha" in text:
        return "challenge", False
    if status in (401, 403):
        return "denied", False
    if status == 429:
        return "rate_limited", True
    if 500 <= status < 600:
        return "server_error", True
    if 200 <= status < 300:
        return "candidate_success", False
    return "unexpected", False

The boolean is the key. Challenges and denials leave the automatic retry loop. Rate limits and transient server errors may receive bounded retries.

What authorized automation should do

Prefer the official interface

Use an API, export, feed, service account, or partner integration when one exists.

Get the scope in writing

Document allowed paths, volume, regions, credentials, retention, and escalation contacts. “The customer asked for it” is not an operating specification.

Keep identity coherent

Do not combine one session's cookies with constantly changing locations and browser profiles. Stability is often more important than novelty.

Pace the aggregate workload

Per-worker limits are misleading when hundreds of workers act together. Apply a shared budget and jittered scheduling.

Stop at explicit controls

A CAPTCHA, login wall, or denial is not an invitation to become more evasive. Route it to a human decision.

Where proxies help—and where they do not

Proxies can support authorized geographic testing, session routing, and distributed public-data collection. Nstdata Proxy Manager can centralize pools, routing, logs, and monitoring when several sources or policies are involved.

But proxy infrastructure does not create permission, and rotation does not repair incoherent behavior. Changing IPs while keeping the same broken retry policy simply distributes the mistake.

If you operate the protected site

Do not make one signal decisive. Test legitimate outliers such as assistive technology, privacy browsers, corporate networks, mobile carriers, and users with unusual locales.

Protect the business action, not only the page view. Add an appeal or allowlist path for approved automation. Roll out new rules in shadow mode before blocking production traffic.

Final takeaway

The useful question is not “How do I look less like a bot?” It is “Why does this authorized workflow look risky, and what approved route should it use?”

Stable identity, bounded traffic, explicit permission, semantic validation, and stop conditions solve more production problems than an endless fingerprint arms race.

Which signal would explain your last block if the User-Agent were removed from the investigation?