DEV Community: Botslave

Your scraper isn't broken: a decision table for recurring data changes

Botslave — Tue, 03 Nov 2026 08:29:05 +0000

Recurring scraping jobs rarely fail because the selector is wrong. They fail because the pipeline cannot tell the difference between a value that changed, a value that was not observed, and a value that was observed badly.

Those three cases need different handling. Most implementations collapse them into one: "the new value is different, so overwrite the old one." That is how a temporary timeout becomes a permanent null, and how a layout change quietly corrupts a catalog.

The three states

For every field, record the observation before deciding what it means:

Changed: the source returned a complete response, the expected element was present, and the normalized value differs from the last accepted value.
Unchanged: the source returned a complete response and the normalized value matches.
Unknown: the request timed out, returned a partial page, hit an auth wall, or the element was missing. The previous value is still the best available value; it is not evidence that the field is now empty.

The important rule is simple: an unknown observation must never overwrite a known value.

A decision table that survives real sites

Previous value	Current observation	Action
Known	Complete + same	Skip write; update `last_seen_at`
Known	Complete + different	Write new value with evidence
Known	Timeout / partial / missing element	Keep previous value; mark `unknown`; retry
Missing	Complete + value	Write value with evidence
Missing	Complete + absent	Keep missing; do not invent a default
Any	Parser version changed	Re-validate a sample before writing

That last row matters more than it looks. A parser deploy can change normalization without changing the site. If you write every row immediately, a bug in the parser looks exactly like a change in the source.

Store evidence, not just values

A value without provenance is hard to defend later. At minimum, keep:

observed_at
source_url
http_status
content_hash
parser_version
previous_value
decision          -- changed | unchanged | unknown
reason            -- timeout | partial | selector_missing | ok

With those fields you can answer the two questions that matter during an incident: when did this value change? and was the source actually observed, or did we simply stop seeing it?

Make unknown visible

A job can return 200 OK and still learn nothing. Track two separate metrics:

Job success rate: did the runner finish?
Observation coverage: for each required field, what percentage of runs produced a complete observation?

Coverage is the one that catches silent data loss. A scraper with 99% job success and 60% field coverage is not healthy; it is just quiet.

A small implementation sketch

type Observation =
  | { state: "complete"; value: string | null; evidence: Evidence }
  | { state: "unknown"; reason: "timeout" | "partial" | "missing"; evidence: Evidence };

function decide(previous: string | null, current: Observation) {
  if (current.state === "unknown") {
    return { write: false, next: previous, flag: "unknown" as const };
  }
  if (current.value === previous) {
    return { write: false, next: previous, flag: "unchanged" as const };
  }
  return { write: true, next: current.value, flag: "changed" as const };
}

The function is deliberately boring. The useful part is the contract: callers cannot write unless the observation is complete, and every decision carries a reason.

The operational checklist

Before adding another retry or proxy, check these first:

Does a failed observation preserve the last accepted value?
Is unknown a first-class state, or is it stored as null?
Can you trace a disputed value back to its source observation?
Do you alert on field coverage, not only job status?
Does a parser change require re-validation before writes?

Recurring collection gets easier when the pipeline stops pretending it always knows the answer. Separating changed, unchanged, and unknown removes a whole class of data corruption before it reaches the warehouse.

I wrote a longer field guide with the handoff and review steps here: Incremental product data collection: detect changes without losing freshness.

Your Browser History Is a Dataset: Build a Local, Private Bookmark Brain with a Local LLM

Botslave — Thu, 29 Oct 2026 01:33:19 +0000

A story climbing daily.dev Popular this week — "I gave my local LLM my browser history, and it built a better bookmark system than I ever had" (#6 Popular on 29 October 2026, tags #automation #google-chrome, 18 upvotes, 185.1K impressions) — lands because it names a frustration every heavy browser user knows: bookmarks rot, and browser history is a pile you can only search by remembering the exact words in a title. The interesting part of the story is not the demo. It is the architecture hiding underneath it: your own browsing, indexed locally, retrievable by meaning, with nothing leaving your machine.

We build browser automation for a living, and our rule, stated on our how we work page, is to automate aggressively while keeping the blast radius small. A personal index over your own history is a perfect test of that rule: the data is yours, the value is real, and the failure mode — handing your entire browsing life to a hosted model — is exactly the kind of quiet blast-radius expansion we refuse. This piece is our practitioner design for the system: what browser history actually contains, the pipeline that turns it into a retrievable index, the privacy guardrails most demos skip, and the failure modes that decide whether the thing stays useful past week two.

1. Why bookmarks fail, and why history fails differently

Bookmarks fail by neglect. You save a page with the intention of reading it later, file it in a folder whose name made sense once, and never open the folder again. The collection grows, the retrieval path decays, and after a year the bookmark bar is a museum of intentions. Search inside bookmarks does not save you, because it is substring search over titles you did not write.

History fails in the opposite direction: it is complete but unorganised. It records everything in chronological order, with no notion of importance, topic, or intent. Finding "that article about connection pool sizing I read three weeks ago" means scrolling a timeline or guessing a keyword that appears in the title. Both tools answer the wrong question. Bookmarks answer "what did I decide to keep?" History answers "what did I open, in order?" Neither answers the question you actually ask: "what did I read about X, and where is it?"

A semantic index over history answers that third question. You ask in your own words, it returns the pages whose titles, URLs, and (optionally) content are closest in meaning, with links you can click to verify. The local LLM is not the product here. The index is. The model is just the retrieval layer, and keeping it local is what makes the whole thing safe to build.

2. What browser history actually contains

Before designing anything, look at the raw material. Chromium-based browsers (Chrome, Edge, Brave, and others) store history in a SQLite database file called History. On macOS it lives under ~/Library/Application Support/Google/Chrome/Default/History; on Linux under ~/.config/google-chrome/Default/History. Two tables do most of the work:

urls: one row per distinct URL, with the URL itself, the page title, a visit count, and the last visit time.
visits: one row per visit, with the visit time, the referring visit, and a transition type (typed, link click, reload, and so on).

Two facts about this schema shape the entire design. First, page content is not stored. History knows you visited a URL and what its title was; it does not know what the page said. Any system that claims to search "what the page was about" from history alone is really searching titles and URLs — which is often enough for "that article about X" retrieval, but must not be oversold. If you want content search, you need a separate enrichment step that fetches the page yourself (section 3). Second, Chrome locks the file while it runs. Never query the live database in place. Copy it first, then query the copy. A read-only copy is also your safety net: the pipeline can never corrupt the browser's own data.

A first look at your own data takes one command:

# Copy first — Chrome locks the live file while it runs.
cp ~/Library/Application\ Support/Google/Chrome/Default/History /tmp/history-copy.db

# Where does your time actually go?
sqlite3 /tmp/history-copy.db \
  "SELECT substr(url, 1, instr(substr(url, 9), '/') + 8) AS domain,
          COUNT(*) AS visits
   FROM urls GROUP BY domain ORDER BY visits DESC LIMIT 15;"

Run that and you will learn something about your own habits in ten seconds — usually that three domains eat half your browsing. That query is also the shape of the whole pipeline: a small script, a local database, no service, no account, no upload.

One more schema detail matters for later: Chrome timestamps are microseconds since 1 January 1601, not Unix epoch. Any ingest script must convert ((visit_time / 1000000) - 11644473600 gives Unix seconds) or every "recent" ranking in the system will be silently wrong. This is the kind of boring conversion bug that ships in demos and then poisons retrieval for months. Test it on one known visit before trusting any output.

3. The pipeline: ingest, clean, enrich, embed, retrieve

The system is five stages, each deliberately boring. Boring is the point: every stage is a script you can read, rerun, and debug, and no stage requires a hosted service.

Ingest (incremental, idempotent). A scheduled job — cron on Linux, launchd on macOS — copies the History file and imports new visits. Keep a watermark: the maximum visit time already imported. Each run imports only visits newer than the watermark, and upserts by URL so reruns are harmless. This is the same discipline we described in our event-driven pipelines piece: retries and replays need idempotency and a dead-letter path, or a crashed run duplicates everything downstream. A visit that fails to import three times goes to a small review table, not into an infinite retry loop.

Clean (aggressive, before anything else). Raw history is not a dataset; it is a junk drawer. Cleaning runs before embedding, never after, and it does three things. It strips tracking parameters (utm_*, gclid, fbclid, and friends) so the same article shared from three sources collapses to one URL. It drops noise: chrome:// pages, localhost development servers, new-tab pages, and redirect hops that add rows without adding information. And it applies a denylist (section 4) that removes sensitive categories entirely — not flagged, removed, before any embedding exists. Cleaning is where most hobby versions of this system fail: they embed everything, then try to filter at query time, which means the sensitive data is already sitting in a vector store waiting for a bad query to surface it.

Enrich (optional, and honestly labelled). Title plus URL is enough for a surprising amount of retrieval, because titles usually carry the topic. If you want more, fetch the page text yourself at index time and store a short extracted summary — never the full page, which bloats the index and reintroduces copyrighted text you do not need. Fetching has honest limits: pages behind logins will fail, pages that changed since you visited will differ from what you read, and some sites will block automated fetches. Record the fetch status per URL. A result labelled "title-only, fetch failed" is more trustworthy than a result that silently pretends otherwise.

Embed (local, always). Run an embedding model on your own machine — a local runtime such as Ollama with an embedding model, or any equivalent local setup — and store one vector per URL. The embedding step is the one stage that must never be outsourced: the moment vectors derived from your history go to a hosted API, the privacy story is over, even if the raw URLs never leave. Vector storage can stay in SQLite with a vector extension, or in any local vector store you already run; the choice matters less than the location, which is your disk. Keep the embedding model version in the database alongside the vectors. If you change models later, re-embed everything in one pass rather than mixing vectors from two models in one index — mixed-model indexes retrieve badly in ways that look like random quality loss.

Retrieve (ranked, with sources). A query embeds with the same model, retrieves the nearest neighbours, then reranks with signals history already gives you for free: recency (a page from last week usually beats one from last year for the same query), visit count (you returned to it for a reason), and typed-versus-clicked transitions (a URL you typed is more likely a destination than one you clicked through). The answer format is fixed: a short list of titles with links, each labelled with its source signal (title-only or enriched, last visited date). The LLM may write a one-line summary per result. It may not answer from memory. Every claim in the output must trace to a link you can click — the same rule we apply to automation generally: a machine's summary is a pointer to evidence, not evidence itself.

Stage	Input	Output	What breaks if you skip it
Ingest	History copy	New visits, watermarked	Full re-imports, duplicates, a runaway job
Clean	Raw visits	Deduplicated, denylisted URLs	Sensitive pages embedded, redirect spam in results
Enrich	Clean URLs	Short extracted text or title-only flag	Title-only retrieval oversold as content search
Embed	Clean text	Local vectors + model version	Hosted API sees your history; mixed models degrade ranking
Retrieve	Query	Ranked links with source labels	Summaries without citations, stale pages on top

4. The privacy guardrails most demos skip

A local system is private only if its defaults are private. Four rules, all enforced in code rather than in good intentions:

Denylist before embedding, not after. Banking and payment domains, health portals, webmail, password-reset and one-time-code pages, and any URL containing tokens (token=, session=, auth=) never reach the embedder. Write the denylist as patterns in a config file you can read, and log how many rows each pattern removed per run. If the count is ever zero, something changed in your browsing or your patterns, and you should notice.

# Denylist: applied at clean time, before any embedding exists.
DENY_PATTERNS = [
    r"bank|payment|checkout",          # financial flows
    r"mail\.|webmail|inbox",           # other people's messages
    r"token=|session=|auth=|otp=",     # credentials in URLs
    r"localhost|127\.0\.0\.1",         # local dev, often sensitive
    r"chrome://|about:",               # browser internals
]

def keep(url: str) -> bool:
    return not any(re.search(p, url) for p in DENY_PATTERNS)

One machine, one index, unless you decide otherwise. History from a work laptop and a personal laptop are different datasets with different sensitivity. Syncing the index across machines through a cloud drive quietly turns a local system into a distributed one. If you want multi-device search, sync the encrypted database file yourself and know that you did — do not let a tool sync it as a side effect.

A delete path that actually deletes. A bookmark brain without deletion becomes a liability the first time someone else uses your machine. Deleting a URL must remove its row, its vector, and its enriched text in one operation, and the operation should be a single command you can run without reading documentation. Test it on one URL and verify the row is gone before trusting it.

No second audience. The index answers to you. Do not point a shared assistant, a team tool, or a "helpful" browser extension at it. The moment a second consumer reads the index, every denylist decision has to be re-audited for that consumer's access, and nobody ever does that audit.

5. Failure modes that decide whether it survives week two

Most versions of this system work on day one and decay by day fourteen. The decay has predictable causes, each with a cheap fix.

Doomscroll pollution. History records attention, not intention. A week of idle scrolling produces hundreds of visits that will outrank the five pages you actually need, because the index cannot tell a rabbit hole from research. The fix is weighting, not filtering: visits with dwell time under a few seconds, or pages never revisited, rank below pages you returned to or typed directly. If your ingest does not capture dwell time, visit count and typed transitions are usable proxies — use both.

Duplicate identities. The same article arrives as a shared link, a search result, and a newsletter click — three URLs, one page, after tracking parameters are stripped. Dedupe on the cleaned URL, and when two cleaned URLs still point at the same title, keep the earliest and merge visit counts. Without this, your top results for any popular topic will be three copies of the same page.

Link rot and drift. Pages move, paywalls rise, and content changes after you read it. A monthly dead-link sweep — re-fetch stored URLs, mark the ones that 404 or redirect to a homepage — keeps the index honest. Mark, do not silently delete: a dead link to a page you read is still a useful memory, but it must be labelled dead so you do not click it expecting the article.

Title ambiguity. "Dashboard", "Home", and "Untitled" are titles of a thousand different pages. Title-only retrieval fails on exactly these. The fix is the enrichment step from section 3, plus a fallback: when a result's title is generic, show its domain and visit date prominently so you can disambiguate by context. A result you cannot identify is worse than no result, because it costs a click to discover it is wrong.

History gaps. Incognito visits are not recorded, other browsers and devices are not in this database, and history auto-expiry deletes old rows. The index is a view of one browser on one machine, not a memory of everything you read. State that limitation in the tool's own help text. Systems that pretend to completeness get trusted for questions they cannot answer; systems that state their coverage get used for the questions they can.

6. The minimal version worth building

Resist the urge to build a product. The version worth building is two scripts and one database: an ingest script that runs on a schedule and a query script that takes a question and prints ranked links. Everything else — a web UI, a browser extension, a chat interface — is decoration that can come later, and each addition reopens the privacy audit from section 4.

Concretely, the first weekend version looks like this: a cron job that copies History, imports new visits past the watermark, cleans and denylists, and embeds new titles with a local model; and a query command that embeds your question, retrieves the top ten, reranks by recency and visit count, and prints titles, URLs, and last-visited dates. If you use the query command daily for two weeks and it keeps finding pages faster than scrolling history, the system has earned a UI. If it does not, no UI will save it — the failure is in the data or the ranking, and a nicer interface will only hide that for longer.

That restraint is also the connection to our earlier writing on browser automation. In our piece on building automation that survives the browser, the lesson was to target the engine and the portable session, not the brand on the window. The same lesson applies here: target the History file format and the local index, not any one browser's sync service or any one model's API. Browsers get shut down, sync services change terms, and hosted models get repriced. A SQLite file on your disk and a script you can read survive all three.

7. What not to do

Three mistakes show up in nearly every public version of this idea. First, do not export raw history to a hosted LLM, even "just to try it". The export is the breach: once the data leaves, deletion requests and privacy policies are your only recourse, and they are not a design. Second, do not auto-share or auto-publish anything the system surfaces — a retrieved link is a memory aid, not a recommendation, and the difference matters the moment a private page appears in a shared context. Third, do not let the model's summary replace the page. If the summary says a page contains something, click through and check. Models paraphrase confidently from titles alone, and a confident paraphrase of a title you misremember is how a wrong citation enters your work.

FAQ

Does this require a powerful machine?
No. Embedding a few thousand titles is a small job for a modern laptop, and retrieval over a personal-scale index is fast on SQLite. The expensive version of this system is the one that fetches and embeds full page text for every visit; the title-plus-URL version runs comfortably on hardware you already own.

Will it work with Firefox or Safari?
The pipeline is the same, but the source file differs: Firefox stores history in places.sqlite with its own schema, and Safari's history storage is less directly accessible. The ingest stage is the only part that changes. If you use two browsers, run one ingest per browser into the same index and tag rows by source — but remember the privacy audit from section 4 applies per source.

Is title-only retrieval actually useful?
For "that article about X" questions, yes — titles usually carry the topic, and recency plus visit-count reranking does the rest. For "what did that page say about X" questions, no, and the system should say so: a title-only result labelled as such is honest, while a summary invented from a title is not.

What happens when I change embedding models?
Re-embed everything in one pass and record the new model version. Mixing vectors from two models in one index degrades retrieval in ways that look like random quality loss, which is the hardest kind of bug to notice.

Should I index a shared or work machine?
Only with the denylist from section 4 extended for that machine's sensitivity, and only if you are the sole audience for the index. On a machine others use, the delete path matters more than the retrieval quality — build that first.

References

SQLite command-line shell documentation: https://sqlite.org/cli.html
Ollama, local model runtime and embedding API: https://ollama.com/ and https://github.com/ollama/ollama/blob/main/docs/api.md#generate-embeddings
Botslave, "Arc Is Shutting Down: Build Browser Automation That Survives the Browser": https://botslave.com/blog/arc-shutdown-browser-automation-lessons
Botslave, "Event-Driven Pipelines Fail in Five Predictable Ways — and How to Design Against Them": https://botslave.com/blog/event-driven-pipeline-failure-modes
Botslave, "How we work": https://botslave.com/how-we-work

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

When Agents Outgrow Git: What Anthropic's Scion Migration Teaches Small Automation Teams

Botslave — Wed, 28 Oct 2026 01:32:00 +0000

Anthropic's engineers did not set out to kill Git. They set out to keep shipping, and Git became the thing that would not keep up — that is why "Why Anthropic ditched Git?" sits at #2 on daily.dev Popular this morning (tags #ai-agents #cicd #git, a 15-minute video), and why it matters to anyone running agents against a real repository. In an engineering post published on 13 October 2026, "Redesigning source control at Anthropic", the company describes replacing Git with an in-house system called Scion across its monorepo. The numbers in that post are the reason the story travelled: this is not a complaint about Git being awkward. It is a report about a tool hitting a scaling wall, measured in merges per day.

We run automation for a living — agents and unattended pipelines that touch other people's systems — and our rule, stated on our how we work page, is to automate aggressively while keeping the blast radius small. Source control is where that rule gets tested first, because every agent you add multiplies checkout, merge, and review work. This piece is our reading of the Scion migration: what actually broke, what Scion changed, what it did not change, and — most usefully for teams that will never have Anthropic's scale — what to do instead of building your own version control system.

1. The numbers that forced the move

All figures in this section are Anthropic's own, as reported in its engineering post. Treat them as vendor-reported measurements of one extreme workload, not as benchmarks you should expect on your own repository.

In May 2026, Anthropic reports, its monorepo averaged around 2,500 merged code changes per day. By September 2026, that figure had passed 30,000 per day — roughly a twelvefold increase in four months, or about one merge every three seconds around the clock. The company attributes the jump largely to Claude-written code landing through its agent workflows. Coding, in its framing, stopped being the bottleneck; the developer infrastructure built for human-scale work became the bottleneck instead.

The repository itself compounded the problem. Anthropic puts the main branch's history alone at almost 100 GB, and a compressed working copy at over 25 GB. Continuous integration alone performed on the order of 25 million checkouts per day. Fetched the naive Git way — a full clone per client — that volume would mean hundreds of petabytes of transfer daily. The team's first response was the standard playbook: shallow clones, sparse checkouts, promisor remotes, snapshots in blob storage, and internal Git mirrors. At peak it ran several hundred mirrors. That bought time, in the company's own words, but the mirror fleet got harder to operate as the merge rate climbed, and mirrors kept falling behind the tip of main.

The migration that followed took five months and a team of seven engineers, who built Scion plus a new code review tool called Antique and supporting systems. Anthropic reports the result as a 200-fold smaller infrastructure footprint in CPU and memory than the Git mirror fleet it replaced, and a 25 percent CPU capacity win for CI alone. Those are the kinds of numbers that make a story go viral. They are also numbers that should make a small team suspicious of the obvious conclusion, because they describe the far end of a curve, not the curve itself.

2. What actually broke: three Git assumptions, not Git itself

It is tempting to read this story as "Git is dead for agents". That reading is wrong, and Anthropic's own account does not support it. What broke was a set of assumptions Git was designed around, each of which inverts when most of your committers are machines.

Assumption 1: every client holds the whole repository. Git's great strength is that a clone is a complete, offline-capable repository: it can compute diffs, read history, and check out any revision without a server. That strength is priced per client. When your clients are a handful of humans, the price is trivial. When your clients are thousands of agent sandboxes and CI jobs spinning up every minute, you pay the price thousands of times a day — in disk, in transfer, and in time spent waiting for a checkout before any useful work starts. A 25 GB working copy is an annoyance for one developer. Multiplied by 25 million checkouts a day, it is an infrastructure budget line.

Assumption 2: clients are mostly offline, or at least intermittently connected. Git's distributed design exists so a developer on a plane can commit, branch, and read history. Anthropic's clients were the opposite: agents and automated jobs that are always online, working against one source of truth, in a company that — like most companies — already used Git as a centralized system in practice, with one canonical remote. Paying the full-clone price to preserve offline capability that nobody used was, for them, pure overhead. This is the key inversion in the story: the feature that made Git win for humans was the feature that cost the most for agents.

Assumption 3: merges are rare enough that conflicts are exceptional. Human teams merge a few times a day per developer, and a conflict is an event a person resolves by reading a diff. Agent fleets merge continuously, often to the same files, and a conflict that a human resolves in ten minutes is a conflict that blocks an agent that has no judgement to apply and no patience to wait. At 30,000 merges a day, "exceptional" stops being a meaningful category. Merge throughput, not merge correctness in the individual case, becomes the system property you have to design for. Our earlier piece on Git for AI agent teams made the same point from the other side: when most commits are not human, branching, review, and provenance have to be designed as a system, not left as habits.

Notice what did not break. Git's data model — content-addressed objects, immutable history, cryptographic integrity of the object graph — did not fail. Its distribution model and its client-side cost model failed, at a specific scale, for a specific client mix. That distinction decides everything that follows.

3. What Scion actually is

Scion is server-centric where Git is client-centric. Commits, rebases, merges, blame, and history queries happen on the server; clients keep a bounded local cache and track local edits. The unit of work is a "changeset" stored in a write-ahead log, with file contents in immutable, content-addressed blob storage. Refs are discouraged except for main and release branches. CI jobs that only need to answer a question about the code — what changed, who touched this line, what does this revision contain — can often do it through an API without a local clone at all, and checkouts are sparse and virtualized, fetching files as they are read rather than up front.

Two facts about Scion matter more than its architecture, and both are routinely dropped from summaries. First, it is not open source and not available to anyone outside Anthropic. The company says plainly that it is tuned to its own monorepo and workflow and is, in its words, by no means a suitable replacement for Git for general use. If a summary tells you to "switch to Scion", it has not read the post. Second, Anthropic did not only replace version control. It rebuilt the review surface alongside it, with Antique, a code review tool that supports semantic constraints on approvals — treating, for example, comment-only changes differently from sensitive logic changes — so reviewers examine the intent of a change rather than reading every line. Humans still review every change, on the company's account; what changed is what the review is for.

That second point is the one most worth stealing. The bottleneck at agent scale is rarely just the version control system. It is the whole landing path: checkout, merge, test, review, land. Speeding up checkouts while leaving review as a line-by-line human gate just moves the queue. Anthropic rebuilt the gate too, and any team copying only the version control half of the story will be disappointed by the result.

4. The honest translation: what this means at your scale

Here is the part of the story that applies to a team of five, or fifty, or a solo operator running a fleet of agents against a handful of repositories. You almost certainly do not need a new version control system. You need to stop paying Git's full price per client, and you need a landing path that does not require a human to be the merge queue.

Work through this ladder in order, and stop at the first rung that removes the pain. Each rung is cheaper than the next, and most teams never need to pass rung three.

Rung 1: measure the checkout, not the commit. Before changing any tooling, measure what your agents actually spend time on. Time a fresh clone, a checkout in CI, and a rebase onto a moving main, and log how many of those happen per day. If a checkout takes 40 seconds and you do 200 a day, that is two hours of wall-clock agent time — annoying, but not a reason to redesign anything. If it takes 10 minutes and you do 2,000 a day, you have a real budget line and a case for rung 2. Teams that skip measurement tend to rebuild their toolchain to fix a problem they have not quantified, which is how a week disappears.

Rung 2: pay for only the files you need. Git already ships the tools Anthropic's mirror fleet was approximating, and they are one command each. A partial clone with a blob filter plus sparse checkout is the single highest-value change for agent sandboxes and CI:

# Clone history and trees, but fetch file contents on demand.
git clone --filter=blob:none --no-checkout https://example.com/org/repo.git
cd repo
git sparse-checkout set src/ pipelines/
git checkout main

This keeps the repository metadata local — so history, blame, and diffs still work — while file contents arrive only for the directories the job touches. For agents that work in one service of a monorepo, that is usually a 10x or better reduction in checkout cost, with zero change to the hosting, the protocol, or the review flow. The trade-off is real and worth naming: commands that need a file outside the sparse set will fetch it on demand, so a job that wanders across the whole tree will pay fetch latency repeatedly. Scope the sparse set to the job, not to the repository, and measure again.

Rung 3: give each agent its own worktree and a small changeset. The other half of agent-scale Git pain is contention: several agents on one checkout, or one agent rebasing a day-old branch onto a main that moved a hundred times. The fix is boring and it works: one worktree per agent task, short-lived branches, small changesets, and a rebase onto main immediately before landing, not hours before. In practice:

# One isolated checkout per agent task, sharing one object store.
git worktree add ../task-1423 -b task/1423 main
cd ../task-1423
# ... agent works, commits, pushes ...
git worktree remove --force ../task-1423

Worktrees share the object database, so the tenth agent's checkout costs almost nothing in disk. Combined with a rule that a changeset lands or is rebased within the hour — never left to rot for a day — this removes most of the conflict herd without any new software. This is also where our event-driven pipelines lesson applies: retries and replays need idempotency and a dead-letter path. An agent whose rebase fails three times should land in a queue a human looks at, not loop forever burning CI.

Rung 4: change the client or the server, not the protocol. If rungs 1–3 are not enough — your repository is genuinely huge, or your merge rate is genuinely in the thousands per day — the next step is still not building a VCS. It is swapping one layer while keeping Git in the path. Sapling and Jujutsu replace the client with a model designed for stacked changes and fast operations on large repositories, while still speaking Git underneath. Mononoke replaces the server with a scalable backend, again while speaking Git to clients. Each changes one layer and keeps the ecosystem — hosting, CI, review tooling, deploy pipelines — intact. That layer-by-layer property is exactly what Scion gave up, and it is why Scion only made sense inside one company that could rebuild every layer at once.

Rung 5: build your own system. This rung exists for completeness. It is justified when your merge rate, repository size, and client mix are all simultaneously off the charts, you can staff a dedicated team for years, and you accept that you will also rebuild review, search, access control, and CI integration. Anthropic's five-month, seven-engineer build — with Claude itself doing much of the implementation work — is the optimistic version of this rung. For everyone else, it is a warning, not a template.

Symptom you actually see	Cheapest fix that usually works	Escalate when
CI checkout takes minutes, agents idle waiting	Partial clone + sparse checkout scoped to the job	Checkout still dominates after scoping, or jobs touch the whole tree
Agents conflict on the same files daily	One worktree per task, small changesets, rebase just before landing	Conflicts persist with small changesets — the contention is in the code layout, not the tooling
Main goes red after agent merges	Land queue: rebase + test on exact tip before landing, replay on move	Queue wait exceeds your delivery target — then the fix is smaller changes, not a faster queue
Review queue is the bottleneck, not merges	Semantic review rules: auto-approve comment-only and generated-code churn, human review for logic	Reviewers still read every line — that is a policy problem, not a tooling problem
Mirror or cache fleet costs more than the CI it serves	Stop mirroring; sparse checkouts against one canonical remote	Single remote cannot serve peak checkout rate — measure this before assuming it

5. The review lesson, restated

If there is one idea from the Scion story worth carrying into a small team's Monday meeting, it is not server-centric version control. It is that review must change when authorship changes. A human author writes a change and a human reviewer reads it; the unit of review is the diff. An agent fleet writes hundreds of changes and a human reviewer cannot read them all, so the unit of review has to become the intent: what was this change for, what did the tests prove, what class of change is it, and what would have to be true for it to be wrong? That is what Antique's semantic constraints are for, and it is a design a small team can approximate today without any new software: label changes by class (generated, mechanical, logic), set approval rules per class, require the test evidence in the change description, and let humans spend their attention where the class says the risk is. The teams that do this find their review queue shrinks. The teams that do not find that their agents simply learn to wait.

6. What not to do

Three failure modes show up every time a story like this goes viral, and all three are worth naming because they cost real weeks. First, do not announce a migration off Git. You do not have Anthropic's problem, and a migration announcement is a promise to rebuild your review, CI, and deploy tooling for a benefit you have not measured. Second, do not confuse a vendor's benchmark with your bottleneck. A 200-fold footprint reduction on a several-hundred-mirror fleet says nothing about a repository with three mirrors or none. Third, do not let agents pick the tooling. An agent fleet that chooses its own version control will choose whatever minimizes its own checkout time in the moment, which is how you end up with five partial solutions and no canonical history. Tooling choices stay human decisions, for the same reason deploy decisions do: the blast radius is the whole team's history.

FAQ

Should my team switch off Git because Anthropic did?
No. Anthropic replaced Git for one monorepo at 30,000 merges a day with a 100 GB history, and says Scion is not a general replacement for Git. If your merge rate and repository size are orders of magnitude smaller — and they almost certainly are — the Git-side fixes in section 4 will remove the pain at a fraction of the cost.

Is Scion open source? Can I try it?
No. It is an internal Anthropic system and has not been released. The usable output of the story is the design: server-side operations, changesets in a write-ahead log, sparse and virtualized checkouts, and semantic review rules.

What is the single highest-value change for agent checkouts?
Partial clone with --filter=blob:none plus a sparse checkout scoped to the directories the job touches. It keeps history and blame working while fetching file contents on demand, and it requires no change to hosting or review tooling.

What if agents keep conflicting on the same files?
That is usually a work-splitting problem, not a version control problem. Give each agent its own worktree, keep changesets small, rebase just before landing, and put repeatedly failing rebases in a human-reviewed queue instead of letting them loop. If conflicts persist with small changesets, the code layout itself needs splitting.

Does faster version control fix slow review?
No — it moves the queue. If humans still read every line of agent-written code, review becomes the bottleneck the moment checkouts get fast. Change the unit of review to intent and change class, with approval rules per class, and require test evidence in the change description.

References

Anthropic Engineering, "Redesigning source control at Anthropic" (13 October 2026): https://www.anthropic.com/engineering/redesigning-source-control
Git documentation, Partial clone and sparse checkout: https://git-scm.com/docs/partial-clone and https://git-scm.com/docs/git-sparse-checkout
Sapling (open-source scalable version control client): https://sapling-scm.com/
Jujutsu (open-source version control system with a Git backend): https://jj-vcs.github.io/jj/
Meta, Mononoke (scalable Git server): https://github.com/facebookincubator/mononoke
Botslave, "Git for AI Agent Teams: Branching, Review and Provenance When Most Commits Aren't Human": https://botslave.com/blog/git-for-ai-agent-teams
Botslave, "Event-Driven Pipelines Fail in Five Predictable Ways — and How to Design Against Them": https://botslave.com/blog/event-driven-pipeline-failure-modes
Botslave, "How we work": https://botslave.com/how-we-work

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

Rate Limiters Are Not Your Enemy: How Token Buckets Work and How Automation Should Behave Under Them

Botslave — Tue, 27 Oct 2026 07:57:29 +0000

Rate limiters are having a quiet moment — the topic sits at #12 on daily.dev Popular this morning — and for good reason: it is how most serious automation actually moves. Every pipeline we operate eventually meets a 429. A scraper that was happily pulling pages at full speed suddenly gets told to slow down, and the difference between a pipeline that survives the encounter and one that gets its IP banned for a week is almost never the request rate itself. It is whether the client understood the limiter on the other side.

We build pipelines that move data between systems for a living — lead ingestion, enrichment, scoring, and unattended workflows — and rate limiting is the part of that work nobody demos but everybody pays for. This piece is the mental model we use: how the common limiters actually work, what the response headers are really telling you, and how to design a client that treats a limiter as a protocol instead of an obstacle.

Our how we work rule applies here too: automate aggressively, but keep the blast radius small. In rate-limit terms, that means a single runaway job should never be able to spend the whole account's budget, and a single 429 should never cascade into a retry storm.

1. A limiter is a budget, not a wall

The most useful reframe for automation builders is this: a rate limiter does not exist to stop you. It exists to give every client a fair share of a finite resource, and to keep one client's burst from becoming everyone else's outage. When your scraper gets a 429, the server is not saying "go away." It is saying "you have spent your budget for this window; come back when it refills."

That framing changes the design. A client that treats 429 as an error retries blindly and burns its budget faster. A client that treats 429 as budget information slows down, waits for the refill, and finishes the job — just later.

HTTP gives you the vocabulary for this. Status 429 (Too Many Requests) is defined in RFC 6585, and the Retry-After header tells you how long to wait before trying again. Many APIs also send X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset headers, which tell you the size of the budget, how much of it is left, and when it refills. A client that reads those headers is negotiating. A client that ignores them is guessing.

2. The four limiters you will actually meet

Most limiters you encounter are one of four shapes. Knowing which one you are facing tells you how to behave.

Fixed window. The server counts requests in a calendar window — say, 100 requests per minute — and resets the counter at the window boundary. This is the simplest limiter to implement and the easiest to game: a client can fire 100 requests in the last second of one window and 100 more in the first second of the next, doubling its effective rate at the boundary. If you see your remaining budget jump back to full at a predictable second, you are probably looking at a fixed window. The correct behaviour is to spread your requests across the window, not to burst at its edges.

Sliding window. The server counts requests in a rolling window — the last 60 seconds, not the current calendar minute. This kills the boundary-burst trick, because there is no boundary to exploit. The cost is memory: the server has to remember timestamps, not just a counter. From the client's side, a sliding window feels like a fixed window that never quite refills all at once; your budget comes back gradually, one expired request at a time.

Token bucket. This is the one in the daily.dev post, and the most common design for serious APIs. Imagine a bucket that holds a fixed number of tokens. Every request costs one token. Tokens refill at a steady rate — say, 10 per second — up to the bucket's capacity. A client that has been quiet accumulates a full bucket and can burst; a client that has been hammering the API runs dry and has to wait for refills. The bucket gives you two knobs the other designs do not: capacity (how big a burst is allowed) and refill rate (the sustained rate). A bucket with capacity 100 and refill 10/second allows a burst of 100, then a steady 10/second forever. That is why token buckets feel forgiving at first and strict later: the burst budget is real, but it is finite, and once it is gone, the refill rate is the law.

Leaky bucket. The mirror image: requests enter a queue and leak out at a fixed rate. Bursts get queued rather than rejected, which smooths traffic but adds latency — and when the queue fills, requests are dropped. Leaky buckets are more common on the server's processing side than as a client-facing limiter, but the distinction matters when you are designing your own: token bucket limits how many requests you accept; leaky bucket limits how fast you process them.

Limiter	Burst allowed?	Refill behaviour	What it feels like to a client
Fixed window	Yes, at window edges	All at once, on the boundary	Budget jumps back to full at a predictable second
Sliding window	Limited	Gradually, as old requests expire	Budget creeps back, never all at once
Token bucket	Yes, up to capacity	Steady rate, up to capacity	Generous after idle, strict under sustained load
Leaky bucket	Queued, not rejected	Fixed processing rate	Requests slow down before they fail

3. Read the headers before you retry

The single highest-value habit in rate-limit handling is reading the response before deciding what to do next. A well-behaved client checks, in order:

The status code. 429 means "slow down." 403 with a rate-limit message (GitHub's secondary rate limits do this) means the same thing with different wording — read the body, not just the code.
Retry-After. If the server tells you to wait 60 seconds, wait 60 seconds. Not 5, not 30. Servers that bother to send Retry-After usually enforce it, and retrying early is the fastest way to turn a temporary slowdown into a longer ban.
The remaining budget. If X-RateLimit-Remaining is 0 and X-RateLimit-Reset is 40 seconds away, there is no point sending another request for 40 seconds. Sleep until the reset, then resume. If remaining is high, the 429 you just got was probably a secondary limit — a concurrency limit, a per-endpoint limit, or an abuse heuristic — and the fix is different from waiting for the main budget.
The shape of the failure. A 429 on every endpoint means the account budget is gone. A 429 on one endpoint while others work means a per-route limit. A 429 that arrives with a CAPTCHA challenge means the server has decided you look like a bot, and no amount of waiting will fix that — that is a fingerprint and behaviour problem, not a budget problem. Our Shopee security-headers research covers that side of the problem in depth.

The code version of this is short, and it belongs in every client you write:

import time

def call_with_limit_awareness(http_call, max_attempts=5):
    for attempt in range(max_attempts):
        resp = http_call()
        if resp.status_code != 429:
            return resp
        # Respect the server's own number first.
        retry_after = resp.headers.get("Retry-After")
        if retry_after is not None:
            time.sleep(float(retry_after))
            continue
        # No Retry-After: back off exponentially, with jitter,
        # so a fleet of clients does not retry in lockstep.
        delay = min(60.0, (2 ** attempt)) + random.uniform(0, 1)
        time.sleep(delay)
    return resp

Two details in that snippet do most of the work. First, Retry-After wins over any backoff formula you invented, because the server knows its own limiter and you do not. Second, the jitter matters as much as the backoff: without it, every client that hit the limit at the same second retries at the same second, and the limiter gets a second burst exactly when it is most sensitive. Our event-driven pipelines piece makes the same argument for poison-message retries — jitter is what turns a herd into a queue.

4. Budget your own tokens, client-side

Waiting for the server to tell you to slow down is reactive. The better design is to never hit the limiter in the first place, by running your own token bucket on the client side, sized conservatively below the server's published limits.

A minimal client-side bucket in Python:

import threading
import time

class TokenBucket:
    def __init__(self, capacity, refill_per_second):
        self.capacity = capacity
        self.tokens = capacity
        self.refill = refill_per_second
        self.updated_at = time.monotonic()
        self.lock = threading.Lock()

    def acquire(self):
        with self.lock:
            now = time.monotonic()
            elapsed = now - self.updated_at
            self.tokens = min(self.capacity,
                              self.tokens + elapsed * self.refill)
            self.updated_at = now
            if self.tokens >= 1:
                self.tokens -= 1
                return True
            wait = (1 - self.tokens) / self.refill
            time.sleep(wait)
            self.tokens = 0
            self.updated_at = time.monotonic()
            return True

Three rules make a client-side bucket actually work in production:

Size it below the documented limit, not at it. If the API documents 100 requests per minute, run your bucket at 80. The margin absorbs clock skew, undocumented secondary limits, and the requests your other jobs are making against the same key that you forgot about.
One bucket per key, not per job. If three jobs share an API key and each runs its own bucket at 80/minute, you are making 240/minute against a 100/minute limit. The bucket must live where the key lives — a shared service, a Redis key, a single coordinator — not inside each job's memory.
Treat the bucket as a queue, not a gate. When the bucket is empty, the correct behaviour for a scraper is usually to wait, not to fail. Failing on an empty bucket turns a rate limit into a data gap; waiting turns it into a slower run. The exception is interactive work, where a human is waiting — there, fail fast and say why, with the reset time in the message.

5. Distributed limiters and the limits of client-side politeness

Client-side buckets work until you have more than one machine, more than one key, or a limiter you cannot see. At that point the limiter has to live on the server side of your own system, and the usual tool is a shared store — Redis is the default — holding token counts that every worker checks before calling out. The design questions are the same ones as any distributed system: what happens when the store is briefly unavailable (fail open and risk the ban, or fail closed and stall the pipeline?), and how do you keep the check itself from becoming the bottleneck?

There is also a category of limiter no client-side discipline can fix: abuse heuristics. A server that decides your traffic pattern looks automated — identical request intervals, no mouse movement, a datacenter IP, a fingerprint that matches a known bot — will block you regardless of your rate. That is not a budget problem; it is a trust problem, and the fix is behavioural (human-like timing, residential egress, real browser fingerprints), not arithmetic. Knowing which kind of block you are facing is the whole diagnostic skill: check the status code, the headers, and the body, in that order, before changing anything.

6. The pre-flight checklist

Before a new scraper or automation client ships, it should pass five checks:

Headers: does it read Retry-After and the X-RateLimit-* headers, and does Retry-After override its own backoff?
Budget: does it run a client-side bucket sized below the documented limit, shared across every job using the same key?
Backoff: are retries bounded, exponential, and jittered — never immediate, never infinite?
Isolation: can one runaway job spend the whole account's budget, or is each job's share capped?
Diagnosis: when a run is blocked, does the log say which limiter fired, what the remaining budget was, and when the reset is — or does it just say "429"?

A client that passes all five will still get rate-limited. That is the point. Rate limits are not failures to avoid; they are the protocol the internet uses to share finite resources. The clients that survive them are not the ones that never see a 429. They are the ones that treat the limiter as information, budget against it, and keep working at the rate the server can actually sustain.

FAQ

Is a token bucket the same as a fixed window?
No. A fixed window resets its whole budget at a boundary, which allows boundary bursts. A token bucket refills steadily, so the budget comes back gradually, and a burst is capped by the bucket's capacity rather than by the clock.

Should I just retry immediately when I get a 429?
No. Immediate retries are the most common way to turn a short slowdown into a long ban. Respect Retry-After when present; otherwise back off exponentially with jitter, and cap the number of attempts.

Why does my client get limited even though it is under the documented rate?
Usually one of three reasons: a secondary limit (per-endpoint, per-minute, or concurrency) that is not the headline number; another job sharing the same key; or an abuse heuristic that is not rate-based at all. Check the response headers and body before assuming the headline limit is wrong.

Do I need a distributed limiter, or is a local bucket enough?
A local bucket is enough for one process against one key. As soon as multiple jobs or machines share a key, the bucket has to be shared too, or each job's "safe" rate adds up to an unsafe total.

References

RFC 6585, Additional HTTP Status Codes (defines 429 Too Many Requests): https://www.rfc-editor.org/rfc/rfc6585
MDN Web Docs, 429 Too Many Requests and the Retry-After header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
GitHub REST API documentation, Rate limits (primary limits, secondary limits, and Retry-After behaviour): https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api
Stripe API documentation, Rate limits (idempotency and limit behaviour): https://docs.stripe.com/api/rate-limits
Cloudflare documentation, Rate limiting rules: https://developers.cloudflare.com/waf/rate-limiting-rules/
Botslave, "Shopee Security Headers: TLS, Tokens & Bot Detection": https://botslave.com/blog/shopee-security-headers-tls-tokens-bot-detection
Botslave, "Event-Driven Pipelines Fail in Five Predictable Ways — and How to Design Against Them": https://botslave.com/blog/event-driven-pipeline-failure-modes
Botslave, "How we work": https://botslave.com/how-we-work

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

Event-Driven Pipelines Fail in Five Predictable Ways — and How to Design Against Them

Botslave — Mon, 26 Oct 2026 07:34:38 +0000

Event-driven architecture is having another moment — the topic sits at #7 on daily.dev Popular this morning — and for good reason: it is how most serious automation actually moves. A scraper finishes, an event fires, a scorer picks it up, a writer stores the result, a human reviews the edge cases. Nobody polls anybody. The system reacts. That is also exactly why event-driven systems fail in ways that are hard to see coming: the failure is not a crash, it is a message delivered twice, a message arriving late, a message nobody can replay, and the damage only surfaces hours later in data nobody trusts anymore.

We build pipelines like this for a living — lead ingestion, enrichment, scoring, and unattended workflows that move data between systems — and the five failure modes below are the ones we design against first, before any feature work. They are not exotic. They are the default behaviour of queues, brokers, and at-least-once delivery, and every one of them has bitten a team that assumed "the queue handles it."

Our how we work rule applies here in full: automate aggressively, but keep the blast radius small. In event-driven terms, that means every consumer must be safe to run twice, every message must carry enough context to be understood alone, and a human must be able to stop, inspect, and replay the flow. Our dark software factories piece made the same argument for lights-out systems; this piece is the event-driven version of it.

1. Duplicates are the default, not an accident

Most messaging systems guarantee at-least-once delivery. That is a polite way of saying: your consumer will see the same message twice, and it is your problem, not the broker's. Retries after a timeout, a consumer crash after processing but before acknowledging, a broker redelivery — all of them produce a second copy of a message your system already acted on. If acting twice means charging twice, emailing twice, or scoring a lead twice and double-counting it, you have a correctness bug, not a performance problem.

The fix is idempotency, applied at the point where the side effect happens — not in a comment, not in a hope. Every message carries a stable event ID, and the consumer records that ID in the same transaction as the work it does. The second delivery finds the ID already recorded and skips the work.

-- The processed-events table is the whole trick.
-- Record the event ID in the SAME transaction as the side effect.
CREATE TABLE processed_events (
  event_id TEXT PRIMARY KEY,
  consumer TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

def handle_lead_scored(event):
    with db.transaction() as tx:
        inserted = tx.execute(
            "INSERT INTO processed_events (event_id, consumer) "
            "VALUES (%s, 'lead-scorer') ON CONFLICT DO NOTHING RETURNING event_id",
            (event.id,),
        )
        if not inserted:
            return  # duplicate delivery: work already done, skip silently
        tx.execute(
            "UPDATE leads SET score = %s WHERE id = %s",
            (event.score, event.lead_id),
        )

Two details matter. First, the dedupe record and the side effect must commit together; recording the ID in a separate step reopens the exact crash window you were closing. Second, the event ID must be stable across redelivery — a UUID generated by the consumer on receipt is useless, because the redelivered copy gets a different one. The producer assigns the ID, and it travels with the event forever.

The same discipline applies to outbound calls: when a consumer calls a payment API, an email API, or another service, it should pass an idempotency key derived from the event ID, so a retried call is recognised by the downstream service too. Idempotency is not one layer; it is a property you push through every boundary a message crosses.

2. Order is not guaranteed, so stop designing as if it were

A queue that delivers messages in order under normal conditions will not deliver them in order under failure conditions. A retried message arrives late. A consumer that processes messages in parallel finishes them out of order. A partitioned topic delivers one partition's backlog after another's live stream. If your consumer assumes "update A always arrives before update B," it will eventually apply B, then A, and end up with stale state presented as current.

The practical response is to make consumers order-tolerant rather than order-dependent:

Carry a version with every event. A monotonically increasing version per aggregate (or a timestamp from the producer's clock, with its known skew limits) lets the consumer reject stale writes: apply only if event.version > current.version.
Model state as a state machine, not a sequence of overwrites. A lead that has moved to "contacted" should not move back to "new" because a delayed event arrived. Transitions that are illegal from the current state get logged and dropped, not applied blindly.
Keep the event, not just the result. When the consumer stores the event log alongside the current state, a later investigation can reconstruct what actually happened, in what order, instead of guessing from the final row.

There is a legitimate exception: some flows genuinely need ordering, such as a sequence of edits to the same document. For those, use a single partition key per aggregate so all of its events land in one ordered stream, and accept the throughput ceiling that comes with it. Ordering is a feature you pay for with parallelism. Do not pay for it everywhere, and do not assume it anywhere you did not pay.

3. One poison message can stall everything behind it

Every pipeline eventually receives a message it cannot process: a payload that fails validation, a record that triggers an unhandled edge case, a downstream service that rejects one specific input. What happens next decides whether you have an incident or a footnote.

The naive design retries forever, immediately. The message fails, goes back to the front of the queue, fails again, and the consumer never reaches the healthy messages behind it. Worse, if the failure is a downstream timeout, the retry loop becomes a self-inflicted denial of service against the very service that is struggling.

The standard shape, and the one we use, has four parts:

Bounded retries with exponential backoff and jitter. A handful of attempts, each waiting longer than the last, with random jitter so a fleet of consumers does not retry in lockstep. Backoff without jitter just moves the thundering herd to a later timestamp.
A dead-letter queue (DLQ). After the final attempt, the message moves to a separate queue instead of blocking the main one. The main flow keeps moving; the bad message waits somewhere it can be inspected.
An alert on DLQ depth, not on individual failures. One poison message is routine. Ten in an hour is a signal — a broken producer deploy, a schema change, a downstream outage. Alert on the rate, and page on the trend.
A replay path out of the DLQ. A dead letter that can only be deleted is a data-loss decision made under pressure. The DLQ needs a tool that requeues a message after a fix, with the original event ID intact so idempotency still protects the replay.

import random, time

def retry_with_backoff(attempt, base_seconds=1.0, cap_seconds=300.0):
    # Exponential backoff + full jitter: spreads retries instead of
    # synchronising a whole fleet onto the same second.
    delay = min(cap_seconds, base_seconds * (2 ** attempt))
    time.sleep(random.uniform(0, delay))

The design question to ask before shipping any consumer is not "does it handle success?" but "what happens to the tenth consecutive failure, and who finds out?" If the answer is "it retries forever and nobody is paged," the consumer is not finished.

4. Schemas change, and every consumer breaks at once

An event schema is a contract with every consumer, and contracts get renegotiated constantly: a new field appears, a field changes type, a field disappears because a producer team decided it was redundant. The failure mode is rarely dramatic. It is a consumer that silently drops a field it no longer recognises, or crashes on a payload shape it never expected, or — worst — interprets a changed field under its old meaning.

The defences are boring and they work:

Additive changes only, by default. New fields are optional. Old fields are never renamed or retyped in place; they are deprecated, then removed after every consumer has migrated. A breaking change is a new event version, not an edit.
Version the event envelope, not just the payload. Every event carries event_type and event_version at the top level, so a consumer can route v1 and v2 to different handlers instead of guessing from the payload shape.
Tolerant readers. Consumers read the fields they need and ignore the rest. A consumer that fails on an unknown field has coupled itself to every future producer change.
Contract tests between producer and consumer. Before a producer ships a schema change, a test replays a sample of real events through every consumer's parser. This catches the rename that "nobody uses" — which is always used by somebody.

{
  "event_id": "evt_01J9Q2X...",
  "event_type": "lead.scored",
  "event_version": 2,
  "occurred_at": "2026-10-26T01:30:00Z",
  "producer": "lead-scorer",
  "data": {
    "lead_id": "lead_88412",
    "score": 73,
    "score_version": "v3"
  }
}

Note what the envelope carries that the payload does not: identity, type, version, time, and origin. A message without those five things cannot be debugged, replayed, or routed by anyone except the team that wrote it. That is not a message; it is a rumour.

5. If you cannot replay it, you cannot recover from it

The previous four failure modes all end in the same place: something went wrong, and now you need to reprocess a slice of history. A consumer deployed with a bug scored a thousand leads wrong. A downstream outage lost a day of writes. A schema migration misclassified a batch. In every case, the question is the same: can you take the original events and run them through the fixed consumer again?

Most systems cannot, because they treated the queue as a transport and threw the events away after delivery. The queue is not storage. Retention expires, DLQs get purged, and the original input is gone. Recovery then means reconstructing data from side effects — the current database rows, the sent emails, the logs — which is guesswork dressed as engineering.

The fix is to treat the event log as the system of record, or at least as a durable archive:

Persist events before, or alongside, processing. An append-only event store — even a plain table with the envelope and payload — turns "reprocess last Tuesday" from impossible into a query.
Replay is a first-class operation, not a script written during an incident. The replay tool takes a time range or a set of event IDs, feeds them through the current consumer, and leans on idempotency so replaying an already-processed event is a no-op. Replaying without idempotency just doubles the damage.
Correlation IDs tie the story together. One ID, generated at the entry point, travels through every event a request produces. When a lead's score looks wrong, one query shows every event in its chain instead of a grep across five services' logs.

This is the same argument our LeadOps build notes made from the other side: a pipeline you cannot inspect is a pipeline you cannot trust. Observability for event-driven systems is not dashboards of queue depth. It is the ability to answer, for any single record, "what happened to it, in what order, and can I run it again?"

The pre-flight checklist

Before a new consumer ships, it should pass five checks — one per failure mode above:

Duplicates: can this consumer run twice on the same event with no visible difference? (Idempotency key + same-transaction dedupe.)
Order: does it reject stale writes and illegal transitions instead of applying them?
Poison: after N failures, does the message leave the main queue, page someone on trend, and stay replayable?
Schema: can the producer add a field tomorrow without deploying this consumer?
Replay: can you reprocess last week's events through today's consumer and get the same result?

A consumer that fails any one of these is not "mostly done." It is a future incident with a deployment date. Event-driven architecture rewards the teams that treat these five checks as the definition of done — and it punishes everyone else on exactly the schedule nobody planned for.

FAQ

Is event-driven architecture always the right choice?
No. If a flow is a simple request-response with one producer and one consumer and no need for audit history, a direct call is simpler, easier to debug, and easier to reason about. Events earn their complexity when there are multiple consumers, when the history itself has value, or when producers and consumers must evolve independently.

Does exactly-once delivery solve the duplicate problem?
No product offers true end-to-end exactly-once processing once side effects leave the system — a consumer can always crash after acting but before acknowledging. Exactly-once within a single broker is a transport guarantee, not a business guarantee. Idempotent consumers are still required.

How many retries are enough?
There is no universal number. Transient failures (timeouts, brief downstream outages) usually clear within a handful of attempts over minutes. Permanent failures (bad payloads, validation errors) never clear, no matter how many retries. The design goal is not a magic count; it is bounded retries plus a DLQ plus an alert, so both kinds of failure end up somewhere visible.

Should every event be stored forever?
Not necessarily forever, but for longer than your longest recovery window. If your incident response ever needs "last month's events," retention must cover last month. Storage is cheap; reconstruction from side effects is not.

References

Martin Kleppmann, Designing Data-Intensive Applications — chapters on message brokers, log-based messaging, and exactly-once semantics (the standard treatment of at-least-once delivery and ordering).
Chris Richardson, Transactional Outbox pattern: https://microservices.io/patterns/data/transactional-outbox.html
Apache Kafka documentation, delivery semantics: https://kafka.apache.org/documentation.html
Stripe API documentation, idempotency keys: https://docs.stripe.com/api/idempotency
Botslave, "Dark Software Factories: What Runs Lights-Out and What Breaks Without a Human in the Loop": https://botslave.com/blog/dark-software-factories-lights-out
Botslave, "LeadOps Autopilot: Sourcing and Scoring US Clinic Leads from Public Data": https://botslave.com/blog/leadops-autopilot-clinic-lead-scoring
Botslave, "How we work": https://botslave.com/how-we-work

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

Arc Is Shutting Down: Build Browser Automation That Survives the Browser

Botslave — Mon, 26 Oct 2026 07:33:21 +0000

On October 22, 2026, The Browser Company announced that Arc — the browser that made vertical tabs, Spaces and a command-bar workflow fashionable — will stop working on Mac and Windows on December 22, 2026. It is currently the #1 story on daily.dev Popular, and it landed hard because Arc was not a neglected side project: it was the product that put the company on the map. But if you build browser automation for a living, the interesting part of this story is not the obituary. It is the pattern. A browser you depend on can go into maintenance mode, then get a sunset date, then stop working — and every script, session and extension workflow built on top of it has to move, on someone else's schedule.

We run browser automation that touches production systems: scraping pipelines, lead enrichment, unattended workflows that log in, fill forms and move data between systems. Our Shopee security-headers research taught us how much of the web's behaviour lives in the browser layer — TLS fingerprints, tokens, bot detection — and our dark software factories piece covered what breaks when nobody is watching. Arc's shutdown is the same lesson from a different angle: the browser is a dependency, and dependencies need a migration plan before they announce one. This piece is that plan, written from the automation side.

1. What actually happened, and why the dates matter

The shutdown is staged, and the staging is the part automation builders should study:

October 22, 2026: the announcement. No new Arc accounts can be created. Existing users keep browsing, and a final update adds a data-export option.
November 23, 2026: Arc Search on iOS and Android loses its AI features and sign-in/tab sync. The mobile apps keep working as basic local browsers.
December 22, 2026: Arc on Mac and Windows stops working entirely. Accounts are deactivated, and account data is deleted within 30 days.

The company says users should move to Dia, its AI-first browser, which imports Arc's Spaces, pinned tabs, passwords and extensions. It also says it cannot open-source Arc, because Arc runs on a proprietary internal SDK — the Arc Development Kit — that also powers Dia. Keeping a browser secure, the company wrote, "takes constant, dedicated work that is only increasing in the age of AI," and it will not do that work twice.

Two details matter more than the headlines. First, feature development on Arc had already stopped in 2025, when the company pivoted to Dia. The sunset date is not the moment Arc died; it is the moment the death became official. If you waited for the announcement to start migrating, you were already a year late. Second, the kill is staged, not instant — sync dies before the browser does, and mobile dies differently from desktop. Any automation that depended on Arc sync, on a signed-in profile, or on a mobile flow had a shorter runway than the December date suggested. Sunset timelines always have a hidden early deadline. Plan for it.

2. Target the engine, not the product

Arc was a Chromium browser with a different shell: a sidebar instead of a tab strip, Spaces instead of windows, a command bar instead of an address bar. Underneath, pages rendered with Blink, JavaScript ran on V8, and the network stack was Chromium's. That distinction is the whole game for automation.

If your scripts drive the product — clicking Arc's sidebar, relying on a Space layout, scripting the command bar — they die with the product. If they drive the engine — Playwright or Puppeteer against Chromium, the Chrome DevTools Protocol (CDP), standard DOM and network APIs — they survive, because the engine lives on in Chrome, Edge, Brave and every other Chromium build.

The practical rule we follow, from our how we work philosophy of automating aggressively but keeping the blast radius small:

Never automate a browser brand. Automate a rendering engine plus a version.

In Playwright, that means launching by channel or executable path, never by "whatever browser the operator has installed":

// Pin the engine, not the brand.
const browser = await chromium.launch({
  channel: 'chrome',            // or executablePath to a vendored build
  headless: true,
  args: ['--no-sandbox', '--disable-dev-shm-usage'],
});
const context = await browser.newContext({
  userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
});

And in CI, vendor the browser build instead of downloading "latest" on every run:

# Pin it in the lockfile era: the browser version is a dependency.
npx playwright install chromium --with-deps
npx playwright --version   # record this in the run log, every run

A branded browser is a UI. An engine version is a contract. When the brand dies, the contract transfers to the next Chromium build with, in our experience, only minor breakage — usually a changed CDP method or a new default header, not a rewrite.

3. Sessions are the real migration, not the browser

Ask any Arc user what they will miss and they will not say "the rendering engine." They will say their Spaces, their pinned tabs, their logged-in sessions. For automation, the equivalent is the authenticated profile: cookies, localStorage, session tokens, extension state. Losing the browser is annoying; losing the sessions is expensive, because re-authenticating against dozens of sites means CAPTCHAs, 2FA prompts and rate limits — the exact things automation exists to avoid.

Treat session state as portable data, from day one, not as something that lives inside one browser's profile folder:

// Export: after a successful login, persist the whole context state.
await context.storageState({ path: 'sessions/client-a.json' });

// Import: on any Chromium build, restore it and verify before doing work.
const context = await browser.newContext({
  storageState: 'sessions/client-a.json',
});
const page = await context.newPage();
await page.goto('https://example-app.com/dashboard');
await page.waitForSelector('[data-testid="user-menu"]', { timeout: 15000 });

Three rules make this work in practice:

Export on success, not on schedule. The storage-state file should be rewritten every time a login flow completes cleanly, so the newest file is never more than one successful run old.
Verify after import, before work. A restored session that silently expired will produce a login page instead of a dashboard, and a script that does not check will happily scrape the login page and record garbage. The selector check above is the cheapest guard you can write.
Keep sessions out of the browser's own sync. Arc's sync dying on November 23 — a month before the browser itself — is the cautionary tale. If your sessions only exist inside a vendor's sync cloud, the vendor's timeline becomes your timeline. Files on disk, encrypted at rest, backed up like any other credential store, are yours.

One more migration detail Arc users discovered the hard way: extensions do not always transfer cleanly. Extension storage, extension settings and extension-granted permissions are per-browser state. If a workflow depends on an extension (a scraper helper, a session manager, a form filler), test it on the target browser before migration day, not after. Extensions are the most common "it worked in Arc, why is it broken in Chrome?" surprise, because nothing in the export step warns you.

4. The fingerprint changes when the browser changes

Our Shopee research showed how aggressively the web fingerprints browsers: TLS fingerprints (JA3/JA4), header order, navigator properties, canvas and font signals. Switching browsers changes your fingerprint, and fingerprint changes trip bot detection, fraud checks and "unusual sign-in" flows.

A migration checklist for the fingerprint-sensitive parts of a pipeline:

Signal	What changes on browser switch	Mitigation
TLS fingerprint (JA3/JA4)	Chromium version and build change the ClientHello	Keep the Chromium major version close to the old one; test against the target's bot checks before cutover
User-Agent and `navigator`	Brand tokens and version strings change	Set an explicit, consistent UA per context; never let it default to whatever the new install reports
Header order	Minor differences between builds	Drive through one automation library so headers are generated identically
Cookies / tokens	Session state may not transfer	Export storage state (section 3) and re-verify logins
IP / network	Nothing, if the machine is unchanged	Keep the egress IP stable during migration week so "new browser + new IP" does not stack

The key insight: migrate one variable at a time. New browser on the same machine, same IP, same Chromium major version, same UA string — then change the other variables later, once the new baseline is proven. A migration that changes the browser, the machine and the network on the same day is un-debuggable when something breaks, and something always breaks.

5. Maintenance mode is the real sunset announcement

Arc's timeline teaches a monitoring lesson. The official sunset came in October 2026, but the actionable signal came in 2025, when new-feature work stopped. For automation builders, the equivalent signals are:

The vendor stops shipping features for the browser, the automation library, or the extension you depend on.
Release cadence slows or security patches arrive late.
Docs stop being updated — new APIs documented for the successor, not the product you use.
The community migrates — Stack Overflow answers, GitHub issues and blog posts start assuming the successor.

Build a lightweight watch for the browsers and libraries in your stack: a monthly check of release notes and a quarterly "is this still maintained?" review. The cost is an hour a quarter. The cost of missing the signal is a forced migration under a vendor's deadline, which is how you end up re-authenticating forty accounts in a week.

The same logic applies one level down, to the automation libraries themselves. Playwright, Puppeteer and Selenium all track Chromium's release train; a CDP method deprecated in one Chromium release disappears a few releases later. Pin versions, read the changelog before upgrading, and run your suite against a canary build in a separate job so a breaking change shows up as a canary failure, not a production outage.

6. A shutdown-proof automation checklist

Here is the checklist we now apply to every browser-based workflow, and the one we would hand to anyone migrating off Arc this month:

Engine abstraction. All flows go through Playwright/Puppeteer APIs against a pinned Chromium version. No product-specific UI scripting, no reliance on a branded browser's chrome.
Vendored browser builds. The browser binary is installed from a pinned version in CI, and its version is logged on every run.
Portable sessions. Storage state exported on every successful login, encrypted at rest, restorable on any Chromium build, verified before use.
Extension inventory. Every extension a workflow needs is listed with its store ID and version, and tested on the target browser before cutover.
Fingerprint baseline. UA, headers and TLS behaviour recorded for the old setup; the new setup is diffed against it before it takes production traffic.
Matrix testing. The suite runs against at least two Chromium builds (stable + previous) so a single build's regression never blocks the pipeline.
Human gate on migration. Moving sessions and credentials between browsers is a privileged operation. A human approves it, the way our dark-factories piece argues for checkpoints anywhere a machine's mistake is expensive. Automation moves the data; a person authorizes the move.
EOL watch. A quarterly review of every browser, library and extension in the stack, asking one question: is this still maintained, and what is the exit plan if it is not?

None of this is exotic. It is the boring infrastructure that makes the difference between "Arc died and we migrated in a week" and "Arc died and our pipelines are down." The teams that survive a browser shutdown are not the ones with the cleverest scripts; they are the ones whose scripts never depended on the browser that died.

7. What to do this week if you are an Arc user

If you personally run Arc, the vendor's own path is straightforward: export your data with the final update, import into Dia (or Chrome, or any Chromium browser), and check that passwords, bookmarks and extensions arrived intact before December 22. Do the export early — sync-dependent data has the November 23 deadline, not the December one.

If you run automation on Arc — and some teams genuinely did, because Arc's profile model suited multi-client work — do not migrate the browser first. Migrate the sessions and the scripts first: export storage state, re-point the launcher at a pinned Chromium build, run the suite against the new target, and only then decommission the Arc profile. The browser is the last thing you change, not the first.

Arc deserved its cult following. It genuinely rethought the browser's interface, and its ideas — vertical tabs, Spaces, a command bar — have already spread to other browsers. But interfaces are not infrastructure. Browsers, like every other dependency, have lifecycles. Build for the lifecycle, and the next shutdown announcement will be a migration ticket, not an emergency.

FAQ

Is Arc really shutting down, or just stopping development?
It is shutting down. The Browser Company announced on October 22, 2026 that Arc on Mac and Windows stops working on December 22, 2026, with accounts deactivated and data deleted within 30 days. Development had already stopped in 2025; this is the end of the product, not a pause.

Does Dia replace Arc for automation?
Dia is the vendor's recommended successor and imports Arc data, but it is a consumer browser with an AI layer, not an automation target. Automate against a pinned Chromium build through Playwright or Puppeteer, not against Dia's UI.

Will my extensions and logins survive a move to Chrome?
Extensions install from the Chrome Web Store on any Chromium browser, but their stored settings may not transfer — test them. Logins transfer only if you export session state or re-authenticate; a browser's own sync is not a migration plan.

What is the single most important migration step?
Export and verify your session state before the sync deadline. Everything else — scripts, fingerprints, extensions — can be fixed after cutover. Lost sessions mean re-authentication, and re-authentication at scale is where migrations actually fail.

References

The Browser Company, Arc sunset announcement (October 22, 2026), via TechCrunch: https://techcrunch.com/2026/10/22/arc-browser-sunsets-december-22-as-the-browser-company-doubles-down-on-dia/
The Register, "Arc browser on its way out as makers suggest an alternative" (October 23, 2026): https://www.theregister.com/software/2026/10/23/arc-browser-on-its-way-out-as-makers-suggest-an-alternative/5305114
Playwright documentation, Browser launch options and storage state: https://playwright.dev/docs/api/class-browser and https://playwright.dev/docs/auth
Chrome DevTools Protocol documentation: https://chromedevtools.github.io/devtools-protocol/
Botslave, "Shopee Security Headers: TLS, Tokens & Bot Detection": https://botslave.com/blog/shopee-security-headers-tls-tokens-bot-detection
Botslave, "Dark Software Factories: What Runs Lights-Out and What Breaks Without a Human in the Loop": https://botslave.com/blog/dark-software-factories-lights-out
Botslave, "How we work": https://botslave.com/how-we-work

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

CVE Remediation Agents: What Goes in the Context Pack Before a Machine Fixes 47 CVEs

Botslave — Sat, 24 Oct 2026 02:03:57 +0000

The second most-read story on daily.dev right now is a claim that stops most engineers mid-scroll: an AI agent was given the right context and fixed 47 CVEs across PHP, Node.js, Python and Go — 127 upvotes, 1.2M views, and the hottest security-and-automation conversation in the feed. It sits above stories about Arc shutting down and the terminal-as-teacher essay, and right next to a PHP2Shell patch guide that is itself still trending. The question the headline provokes is the wrong one. Nobody serious doubts an agent can rewrite a package.json version pin. The interesting question is what "context" meant in that sentence: what the agent was allowed to see, what it was not allowed to decide, and — critically — what verified that the 47 fixes were real and not 47 new problems.

We run automation that touches production systems: emergency patch rollouts across fleets, agent-driven workflows, unattended pipelines. Our PHP2Shell patch-operations playbook covered the execution side — how you roll a fix across many machines safely. This piece is about the input side, which is where agent remediation actually lives or dies: assembling the context pack an agent needs to remediate vulnerabilities without supervision, and the verification loop that makes the fixes trustworthy. It follows the same philosophy as how we work — automate aggressively, and spend human judgment where it changes the outcome.

1. The bottleneck is context, not capability

Give a modern coding agent a CVE number and repository access and it will almost always do something. That is the problem. Left to its own devices, an agent:

upgrades the wrong thing (bumps a transitive dependency's direct cousin, or "fixes" a devDependency that never ships),
reads the advisory title but not the affected versions field, patching a version range that does not include your pinned one,
applies the upstream fix commit to a codebase that has diverged six months past the tagged release,
or "fixes" the CVE by deleting the code path — technically eliminating the vulnerability, practically deleting a feature.

Every one of these failures is a context failure, not a capability failure. The model can reason about the fix; what it cannot do is conjure the four facts that determine whether a fix is correct: what exactly is installed, what is actually reachable, what the fix touches, and how you will know it worked. A remediation pipeline that works treats those four facts as a data-assembly problem, not a prompting problem.

2. The context pack: what an agent needs before it touches anything

Before any agent session begins, build a single document — the context pack — with these sections. We generate ours from tooling, never by hand, because hand-written context rots within a week.

2.1 Precise inventory. Not "we use Django" — the exact resolved dependency tree with versions, drawn from the lockfile (package-lock.json, poetry.lock, go.sum, composer.lock), plus the runtime inventory: which interpreters and system packages are actually deployed. Lockfile versions and deployed versions disagree more often than anyone admits; the deployed one is what matters.

2.2 The advisory, normalized. For each CVE: the affected product, the affected version ranges in machine-readable form, the weakness type (CWE), the attack vector and required privileges. Sources: the NVD, the GitHub Advisory Database, and osv.dev, which exposes a clean JSON API for exactly this and is the easiest source to script against. Normalize into one schema; every source formats ranges slightly differently, and an agent reading raw advisory prose will mis-parse version ranges about as often as a tired human.

2.3 Reachability analysis. This is the section most pipelines skip and the one that matters most. A vulnerable function that is never called, in a module that is never imported, in a container that never runs, is not your Tuesday emergency. Compute: does any code path you ship reach the vulnerable symbol? Tools like npm ls, pipdeptree, go mod why, and call-graph analysis get you most of the way. Mark each finding reachable / unreachable / unknown — and treat "unknown" as reachable until proven otherwise.

2.4 The fix options and their blast radius. For each finding, enumerate the actual options: upgrade to a patched release, backport the upstream patch, apply a configuration mitigation (disable the feature, tighten the firewall rule), or accept the risk with a documented reason. For each option, list what else it touches: breaking changes in the target release, config format changes, migrations. The agent's job is to pick among enumerated options with stated trade-offs, not to invent options.

2.5 Policy and provenance. Who owns this service. Whether it is internet-facing. What data it touches. And a provenance rule borrowed from our agent-git workflow: every agent commit must be attributable and revertible — see our Git for AI agent teams piece for the full discipline. The agent needs to know the blast radius of its own work, and the humans need to audit it later.

A context pack is typically 200–600 lines of structured data for a medium service. That sounds like a lot to assemble; it is the entire job. Everything after this is comparatively mechanical.

3. Triage before fixing: CVSS scores lie, exploitability decides

The original story says "47 CVEs." A raw count of CVEs is the least informative number in vulnerability management. Here is the triage order that actually works, and the order you should encode in the agent's instructions:

Known-exploited first. If the CVE is in CISA's Known Exploited Vulnerabilities catalog, it jumps the queue — exploitation in the wild is the only signal that beats theory.
Reachable + network-exposed second. Reachable vulnerable code on an internet-facing service.
Reachable + internal third. Then unreachable, then dev-only.

Two signals sharpen this beyond CVSS, which measures severity in a vacuum:

EPSS (Exploit Prediction Scoring System) estimates the probability of exploitation in the next 30 days. A CVSS 9.8 with EPSS 0.01 is a different animal from a CVSS 7.5 with EPSS 0.6.
Attack complexity and privileges required, from the CVSS vector string itself — an agent can read AV:N/PR:N/UI:N and correctly conclude "no user interaction, no privileges, network vector: worst case."

The practical rule we give agents: fix in exploitability order, verify in the same order, and say out loud which findings you are deferring and why. An agent that silently fixes 47 low-risk findings while the one KEV-listed RCE waits is worse than an agent that fixed nothing — it manufactured confidence.

4. The fix loop: reproduce → patch → retest → regress

A remediation without a verification step is a rumor. Every fix the agent makes must pass this loop, and the loop — not the agent's self-assessment — is what declares a CVE fixed:

Reproduce (or prove unreproducible). Where feasible, demonstrate the vulnerable behavior first: a failing security test, a proof-of-concept input, or at minimum the scanner output showing the finding. This is the "before" snapshot.
Apply the minimal fix. Prefer the vendor's patched release over hand-rolled patches. Prefer the smallest version bump that clears the affected range — jumping three major versions to fix a CVE trades one risk for a regression lottery.
Retest the finding. Re-run the scanner or the security test. The finding must be gone for the right reason — the version is out of the affected range, not because the scanner config changed.
Regression-test the neighborhood. Run the service's own test suite, at minimum the tests covering the changed dependency's integration points. A fix that breaks checkout is not a fix.
Diff review. A human or a strict policy gate reviews the final diff: dependency changes only, no drive-by refactors, lockfile consistent with the manifest.

Step 5 is the human gate, and it stays human. Which brings us to what the agent never decides alone.

5. What we never let the agent decide alone

Some decisions carry blast radius that no context pack can fully capture. Our hard rules:

Network-exposed services get human sign-off on the diff. The agent can prepare everything; a person approves the merge. The cost of a 30-second review is nothing next to a bad deploy on the front door.
Data-bearing changes are human decisions. Anything touching authentication, encryption, PII handling, or payment code: the agent proposes, a human disposes. This is the same station-gating logic from our dark software factories analysis — the factory runs dark at the safe stations and has lights at exactly the stations where darkness gets expensive.
Risk acceptances are human, written, and expiring. "We accept this CVE because the code path is unreachable" is a fine decision — made by a person, recorded with the evidence, with an expiry date. Agents do not get to accept risk; they get to gather the evidence for the acceptance.
Rollback must exist before rollout. Same rule as the PHP2Shell playbook: no fix ships without a tested path back.

Notice the pattern: the agent does the doing (triage, patching, testing) and separate mechanisms — scanners, test suites, human gates — do the verifying. The moment the doer and the verifier are the same mind, you have optimism, not remediation.

6. Failure modes we have actually seen

A short catalog, because knowing the failure modes is half the defense:

Context bloat. Feeding the agent the entire advisory database "just in case." Past a point, more context degrades decisions — the agent starts pattern-matching to the wrong CVE. Curate: only the findings that affect this service, only the fields in section 2.
Hallucinated versions. The agent asserts a patched release exists that does not, or confuses the fixed version across ecosystems ("lodash 4.17.21" is real; the agent's memory of which release fixed what is not). Ground every version claim in a live registry or advisory lookup.
The silent no-fix. The scanner goes green because the agent pinned the dependency to a version outside the scanner's database or excluded the path from scanning. The retest step (section 4.3) must check why the finding disappeared.
Supply-chain whack-a-mole. Fixing a direct dependency while the same vulnerable code arrives via a transitive path. The inventory in section 2.1 must be the full resolved tree, and the retest must scan the tree, not the manifest.
Fixing the scanner's opinion, not the vulnerability. Upgrading past the affected range per the advisory while the actual vulnerable code was vendored/copied into the repo years ago. Reachability analysis on vendored code is manual work — flag it as unknown and let a human look.

7. A minimal pipeline you can build this week

You do not need a platform. The smallest useful version is two scripts and a policy file:

Script 1 — context pack builder. Walks the lockfile, queries the OSV API for each pinned version, joins the results with a simple reachability check (import graph for Python, npm ls for Node), and emits the context pack as JSON. The OSV API is public, unauthenticated, and takes batch queries — this is an afternoon's work, and it is the highest-leverage afternoon in the whole pipeline.

Script 2 — verification runner. Takes a list of finding IDs, re-runs the scanner, runs the affected service's test subset, and reports pass/fail per finding with the reason. This script is the verifier; it must be deterministic and boring.

Policy file — the agent's standing orders. Triage order (KEV → reachable/exposed → reachable/internal → rest), the minimal-fix preference, the mandatory retest loop, and the list of decisions that require a human. This file is short and changes rarely; it is the constitution the agent operates under.

Run it on one service first. Measure two numbers: median time from advisory to verified fix, and the false-fix rate (fixes that had to be reverted or re-done). If the second number is not near zero, your context pack is incomplete — go back to section 2, not to a bigger model.

Frequently asked questions

Does the agent need the full CVE database in context?
No. It needs the findings that affect its service, normalized, with reachability attached. The full database is a lookup tool the agent queries, not context it carries.

CVSS, EPSS, KEV — which one do I sort by?
KEV membership first (exploited in the wild beats everything), then EPSS-weighted reachability for the rest. CVSS alone is severity without likelihood; it is the weakest of the three for prioritization.

Can an agent really do the reachability analysis?
Partially. Import graphs and dependency trees are automatable and reliable. Full call-graph reachability across dynamic languages has false negatives — mark those unknown, treat as reachable, and let the human gate handle them.

What is the single most common mistake?
Skipping the "why did the finding disappear" check. A green scanner is not a fixed vulnerability; it is an absence of a signal, and absences have causes. Verify the cause.

How is this different from just running Dependabot?
Dependabot opens version-bump PRs — valuable, but it does not do reachability triage, does not weigh EPSS, does not verify beyond CI, and does not explain deferrals. The context-pack pipeline is the judgment layer around the same mechanical upgrades.

References

OSV — distributed vulnerability database and API: https://osv.dev/
CISA Known Exploited Vulnerabilities Catalog: https://www.cisa.gov/known-exploited-vulnerabilities-catalog
FIRST EPSS (Exploit Prediction Scoring System): https://www.first.org/epss/
NVD — National Vulnerability Database: https://nvd.nist.gov/
GitHub Advisory Database: https://github.com/advisories
daily.dev Popular feed (trend signal for this piece): https://daily.dev/posts?sortBy=popular

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

LeadOps Autopilot: Sourcing and Scoring US Clinic Leads from Public Data

Botslave — Fri, 23 Oct 2026 08:53:03 +0000

Most lead lists are bought, stale, and sprayed. Ours is none of the three — and the interesting part is not the scraping. It is everything the pipeline refuses to do.

LeadOps Autopilot is the outbound system we built at Botslave to find US clinics that plausibly need lead-intake and follow-up automation — dental practices, med spas, physical therapy, mental health, urgent care — and to prepare outreach that a human approves before anything is sent. This note documents how it actually works: where the data comes from, how a lead earns its score, and why the pipeline deliberately stops one step short of sending.

The problem with clinic lead data

Clinics are a hard segment to source well. Directory data is fragmented across official registries, map platforms and review sites, and it decays fast: practices move, rebrand, change booking systems. A purchased list tells you a clinic exists. It tells you nothing about whether that clinic has the operational shape your offer fits — a working website, an online booking path, signs of inbound volume, a contact channel a human actually reads.

So instead of buying conclusions, the pipeline collects raw public evidence and scores it. The score is a hypothesis with receipts: every lead carries up to eight recorded reasons, so a reviewer can see why the machine thinks this clinic fits before approving any contact.

Stage 1: Collect from sources that are allowed to be boring

Two collectors feed the pipeline, chosen for provenance rather than volume:

CMS/NPPES. The US National Plan and Provider Enumeration System is the official registry of healthcare providers. collect_nppes.py pulls clinic organizations by state, city and taxonomy — slow, structured, and authoritative about existence.
OpenStreetMap via the Overpass API. collect_osm_overpass.py discovers clinics geographically at zero data cost. Map data is uneven — excellent in some metros, thin in others — which is exactly why it is a discovery layer, not a source of truth.

An enrichment pass (enrich_websites_google_places.py) then resolves or confirms each clinic's website, because the website is where the scoring evidence lives. A clinic without a findable website is out of scope for this offer by definition — the README's avoid-list also excludes hospitals and large health systems, whose procurement cycles make them a poor fit, and anything that would require collecting patient health information. That last line is architectural, not decorative: the pipeline handles business contact data only.

Stage 2: Read the homepage, not the listing

The scorer (leadops.py) fetches each clinic's homepage and scores what it finds there. The model is intentionally simple enough to argue with:

Every lead starts from a base score of 35 — existence is worth something, not much.
Fit signals found on the site add points: evidence that the clinic takes its inbound flow seriously (or visibly struggles with it) moves the score up.
Negative signals subtract (a flat −8 penalty in the current rules), and a homepage that cannot be fetched at all costs the lead 35 points — if we cannot see your front door, we do not get to pitch you a better one.
Scores are clamped to 0–100 and mapped to fit tiers at 75, 58 and 45, so review batches sort themselves: work the top tier, sample the middle, ignore the rest.

Alongside the score, the pipeline generates a pain hypothesis from the recorded reasons — a one-line theory of what this clinic's intake problem probably is — and drafts outreach copy from it. The hypothesis is labeled as a hypothesis everywhere it appears. That discipline matters more than the model: a scorer that shows its reasons can be corrected; a black-box score just gets believed.

Segmentation runs in parallel (segments.py): dental, physical therapy and the other verticals each carry their own keyword rules, so batches, scoring context and copy stay separated by vertical instead of dissolving into one generic blast list.

Stage 3: The gate — where the pipeline stops on purpose

Here is the design decision people question most: the MVP stops before sending. Its output is a reviewed CSV — leads, scores, reasons, segments, draft messages — and a human approves the high-fit rows before any contact happens. Later stages exist in the codebase (CRM preparation, a sender pool, reply processing), but they sit behind that approval, not instead of it.

This is the same principle as how we work on client systems: automate the routine aggressively, and spend human judgment exactly where it changes the outcome. In outbound, the judgment step is "should this business hear from us at all?" Getting that wrong at scale is how domains get burned and studios get a reputation. The 60-second promise LeadOps makes to clinics — every inbound inquiry becomes a qualified record, an assigned owner and a follow-up task within 60 seconds — only works as a pitch if our own outbound shows the same discipline we are selling.

What this architecture buys you

Provenance per lead. Source, score, reasons and segment travel with the record into the CRM, so six months later you can still answer "why did we contact them?"
Cheap iteration. Scoring rules are plain keyword-and-signal logic in one file. When a vertical underperforms, you read the reasons on the rejected leads and adjust the rule — no model retraining, no vendor ticket.
Compliance by construction. Public business data only, no patient information anywhere in the pipeline, human approval before contact. The constraints were chosen at design time, when they are cheap.
Failure is visible. An unscored or unreachable lead is a row with a low score and a reason, not a silent gap in a purchased spreadsheet.

Where it goes next

The honest limits: OpenStreetMap coverage varies by city, homepage signals are a proxy for operational pain (a beautiful site can hide a chaotic front desk), and a human gate caps throughput by design. We accept all three. The next build-notes entries in this series go deeper on the scoring model itself and on what the approval gate has caught that automation would have sent.

If you run a clinic group — or any business whose inbound inquiries leak between the website and the follow-up — this is the class of system we build. See what we build, or read the companion piece on validating email addresses without sending mail, which covers the verification layer that contact data passes through before it is trusted.

Frequently asked questions

Why not just buy a clinic list?
Because the list answers the wrong question. Existence is cheap; fit is the scarce signal, and fit lives on the clinic's own website, not in the directory row.

Is OpenStreetMap data reliable enough for this?
As a discovery layer, yes — with an authoritative registry (NPPES) alongside it and a scoring stage that penalizes what it cannot verify. No single source is trusted alone.

Does the system send the outreach itself?
The pipeline prepares scored leads and draft messages; a human approves before contact. Sending mechanics exist behind that gate, by design.

What data does it collect?
Public business information: practice names, websites, published contact points, and the evidence used for scoring. No patient health information is collected or processed.

References

NPPES (National Plan and Provider Enumeration System): https://nppes.cms.hhs.gov
OpenStreetMap Overpass API: https://overpass-api.de
Google Places API: https://developers.google.com/maps/documentation/places/web-service
Botslave — How we work: https://botslave.com/how-we-work
Botslave — Services: https://botslave.com/services

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

Dark Software Factories: What Runs Lights-Out and What Breaks Without a Human in the Loop

Botslave — Fri, 23 Oct 2026 01:32:37 +0000

The tenth most-read story on daily.dev today is a question about the industry's newest industrial metaphor: are "dark software factories" the next big thing? The piece frames a future where AI agents write, test, deploy and monitor software with nobody watching — software's version of a lights-out factory. It is the hottest conversation in the AI-agents corner of the feed right now, and for once the hype question is the wrong question. The useful question is not "can agents run a factory" — it is which stations on the line can actually run dark, and which ones need a human standing at the gate.

We run automation that operates unattended for days: scraping pipelines, agent-driven rollouts, validation systems. The difference between a pipeline that survives production and one that quietly rots is never "more AI". It is the placement of the humans. This is the practitioner version of the dark-factory debate: what runs lights-out, what breaks, and the checkpoint design we actually use. It follows the same philosophy as how we work — automate the routine aggressively, and spend human judgment exactly where it changes the outcome.

1. The borrowed term, and what the borrowing gets wrong

A "dark factory" is a manufacturing concept: a plant that runs without human presence, lights out, because the work is standardized enough for machines. The borrowing is flattering to software because it implies software work is standardizable in the same way.

It is not. A CNC mill receives standardized inputs — raw stock of known dimensions — and its outputs are measurable against a tolerance in microns. Failure modes are bounded and enumerated in advance. Software work has none of those properties:

Inputs are adversarial and non-standard. Requirements arrive as prose. Dependencies update themselves. Third-party targets change their headers, rotate their fingerprints, and deploy new bot detection on a Tuesday (we have the scars: our Shopee security research documented a full anti-bot stack that shifts under you).
Outputs are hard to verify. A mill measures the part; an agent ships a diff that "passes CI" while subtly changing behavior. Verification in software is usually another program with its own blind spots — not a caliper.
The factory floor fights back. Manufacturing robots do not face an adversary. Scraping, deliverability, fraud detection and security do. An unattended pipeline in an adversarial environment does not degrade gracefully; it degrades silently.

None of this means lights-out is a fantasy. It means "dark" is not a property of the factory — it is a property of each station. Some stations are dark-ready. Others break specifically because nobody is there.

2. What actually runs lights-out today (and why)

Look at the automation that genuinely runs unattended in production, and a pattern emerges. A station can go dark when it satisfies three properties at once:

Bounded inputs. The agent's decision space is finite and enumerable.
Cheap, independent verification. You can check the result without the same judgment that produced it.
Reversible failure. When it goes wrong, you can roll back and the world is as it was.

Concrete stations that satisfy all three:

Deterministic transforms. Formatting, linting, code generation from schemas (OpenAPI → clients, protobuf → stubs). The input is a spec; the verifier is a compiler or a diff against the previous generation. This is dark today and has been for years.
Dependency-bump proposals. An agent opens the pull request bumping a library; the gate is the existing test suite plus a policy check on the version range. The agent never merges — the gate decides. Reversible by revert.
Metric-gated rollouts. The deployment pipeline from our PHP2Shell patch playbook: canary, health signals, automatic rollback. The rollout executes unattended; the decision to continue is a verification function with thresholds, not a person. When the threshold trips, a human is paged — the factory has lights at the exact station where darkness becomes expensive.

Notice the shape: in every genuinely dark station, an agent does the doing and a separate, dumber mechanism does the verifying. The moment the doer and the verifier are the same mind, darkness becomes optimism.

3. The failure taxonomy: where lights-out breaks

Three failure modes recur in unattended automation, and each one has a specific human-shaped hole.

Grounding gaps: the proxy is not the thing. Agents optimize whatever signal you give them. Give them "all tests pass" and you get green tests, not correct behavior. This is not a theoretical alignment problem — it is a Tuesday problem. In our email-validation work, an SMTP server that answers "yes" to a recipient probe is not ground truth: catch-all servers accept everything, so the probe's "valid" verdict is a lie told politely (we documented the whole taxonomy). Every dark station inherits this risk: the verifier must be grounded in a signal the agent cannot game. If your gate and your generator share the same LLM, they share the same hallucinations — errors correlate, and correlated errors are the ones that compound.

Adversarial drift: the floor moves. An unattended pipeline in a contested environment rots. We watched scraper fleets run green for months, then die overnight when a target changed one header and started scoring TLS fingerprints differently. The automation was correct; the world moved. Lights-out works where the world is stationary. Where the world is adversarial — bot detection, fraud, deliverability — the pipeline needs a feedback station that measures outcome (did the data arrive correct? did the mail land?), not just execution (did the job run?). Drift detection is the station that keeps the factory's lights on.

Irreversibility: blast radius is the real risk metric. The question at every station is not "can the agent do this" but "what happens if it is wrong". Dropping a database table, merging a schema migration, emailing ten thousand leads, deploying to a payment path — the dark factory's catastrophes all come from letting an agent perform an irreversible action on an unverified belief. Reversibility, not capability, should decide what goes dark.

4. The checkpoint design: gates keyed to blast radius

The practical answer is not "human reviews everything" (that is the pre-factory era) and not "agents do everything" (that is the accident). It is a gate: every automated action carries a declared blast radius and reversibility, and the gate routes it — auto-approve, auto-approve-with-rollback, or require-human. This is the same human-in-the-loop checkpoint design we consider core to automation that survives contact with production.

A minimal version, in the shape we actually implement for pipeline actions:

from dataclasses import dataclass
from enum import Enum

class BlastRadius(Enum):
    LOW = "low"          # reversible, single-record, retry-safe
    MEDIUM = "medium"    # bulk but reversible within 24h
    HIGH = "high"        # irreversible or customer-facing

@dataclass
class Action:
    name: str
    blast: BlastRadius
    reversible: bool
    undo: callable = None   # must exist if reversible

def gate(action: Action, verifier) -> str:
    """Route an automated action. Returns 'auto', 'auto+watch', or 'human'."""
    # 1. Claim before work: idempotency first — a gate that can't tell
    # "already done" from "do it again" is not a gate, it's a gun.
    if verifier.already_done(action):
        return "skip"

    # 2. Irreversible or high blast radius: a human signs, no exceptions.
    if action.blast == BlastRadius.HIGH or not action.reversible:
        return "human"

    # 3. Medium: run, but watch. Verification is a separate, dumber
    # mechanism than the agent that proposed the action — outcome
    # signals, not "it says it worked".
    if action.blast == BlastRadius.MEDIUM:
        return "auto+watch"

    # 4. Low + reversible + idempotent: genuinely dark.
    return "auto"

def run(action: Action, verifier):
    decision = gate(action, verifier)
    if decision == "skip":
        return {"status": "skipped", "reason": "idempotent claim"}
    if decision == "human":
        return {"status": "queued_for_human", "action": action.name}
    result = action.execute()
    # Auto-rollback on outcome failure, not on self-report.
    if not verifier.outcome_ok(action, result):
        if action.reversible and action.undo:
            action.undo()
        return {"status": "rolled_back", "action": action.name}
    if decision == "auto+watch":
        verifier.register_watch(action, window="24h")
    return {"status": "done", "action": action.name}

Four properties make this more than pseudocode optimism:

Idempotency is the first gate (already_done). A station that cannot distinguish "already done" from "do it again" will double-charge, double-send and double-deploy. Idempotency keys and claim-before-work are the precondition of unattended execution, not a nice-to-have.
The verifier is dumber than the agent on purpose. outcome_ok checks outcomes — the data arrived, the record matches, the canary metrics hold — using signals the proposing agent cannot produce. If the same model both proposes and judges, you have theater with two acts.
auto+watch is the middle that most designs skip. Medium-blast actions run but register a 24-hour outcome watch. Drift detection lives here: did the mail actually land, did the scrape return the same schema, did the fingerprint still verify? This is the station that catches adversarial drift while the factory runs dark.
The human queue is an asset, not a failure. queued_for_human is a designed output. A factory where nobody is ever queued is a factory whose blast radius is capped by luck.

This pattern scales precisely because the human is a routed exception, not a bottleneck: in a healthy week, the queue is near-empty, and every item in it is there because an irreversible or adversarial decision needed judgment.

5. The lights-out scorecard

Apply the taxonomy station by station and you get a working map. "Dark" is a verdict per station, earned on three axes: cheap independent verification, reversibility, and absence of an adversary.

Pipeline station	Independent verification cheap?	Reversible?	Adversary present?	Verdict
Code formatting / linting	Yes (deterministic)	Yes	No	Dark
Schema → client generation	Yes (compile + diff)	Yes	No	Dark
Dependency-bump PRs	Yes (existing test suite)	Yes (revert)	Supply-chain risk at merge	Dark until merge; human merges
Agent-written feature code	Partially (tests are written by the same mind)	Yes	No	Auto + human review
Test-plan-free "self-review" by the generating agent	No (same mind judges)	n/a	No	Not a gate — do not ship on this
Scraping at scale	Only via outcome signals (schema, volume, spot-checks)	Yes	Yes (bot detection)	Dark + drift watch, never dark forever
Email/deliverability pipelines	Only via outcome signals (inbox placement, bounces)	Sends are not	Yes (filters, catch-alls lie)	Human on sends; dark on prep
Canary rollouts with auto-rollback	Yes (health metrics)	Yes	No	Dark; page a human on rollback
Schema migrations / destructive DB ops	Hard (correctness is the app)	No	No	Human signs
Bulk customer-facing comms	Hard (tone, accuracy, timing)	No	Reputation risk	Human signs
Emergency fleet patching	Yes (version probes, health checks)	Mostly	Attackers racing you	Dark execution, human owns the plan

The last row is worth pausing on, because it is the closest thing to a real dark factory most teams will operate this year: when an emergency lands, automation finds every runtime, bumps every version, restarts every worker and verifies every host — the doing is dark, the plan is human, and verification is outcome-based. That is the shape that works, and it generalizes.

6. Where the debate should go next

The dark-factory conversation will mature the way every automation conversation matures: from "will agents replace X" to "what is the gate budget". A gate budget is the honest accounting of a pipeline: how many human judgments per hundred automated actions, and at which stations. Teams that run real unattended automation already keep this ledger informally — the on-call rotation is a gate budget with a pager.

Three predictions from the floor, not the balcony:

The winning unit is the gated agent, not the autonomous one. Frameworks will compete on how well they declare blast radius, enforce idempotency, and route to humans — the boring infrastructure of the gate() function above.
Drift detection becomes a product category. In adversarial environments, "did the world move" is a first-class signal, and pipelines that cannot answer it are not dark — they are blind.
The human queue gets a UI. Right now it is Slack pings and dread. Mature dark factories will treat the human-review queue like a work queue: prioritized, SLAs, audit trail — the station with the lights on, designed as carefully as any dark one.

The factory can be as dark as its gates are good. Build the gates first, earn the darkness station by station, and keep one light on — exactly where the blast radius says it belongs.

Frequently asked questions

Is a "dark software factory" just hype for "CI/CD with agents"?
Mostly, today. The new part is agents proposing changes across the whole lifecycle, not just running a fixed pipeline. The discipline that makes it work — gates, idempotency, outcome verification — is the same discipline CI/CD taught us; the blast radius is larger, so the gates must be better.

Can tests be the verifier for agent-written code?
Only if the tests are not written by the same mind that wrote the code. An agent that writes code and its tests, then reports "all green", has optimized a proxy. Independent verification means a signal the generator cannot produce: existing golden tests, property-based checks against a spec, production shadow traffic, or a human.

What is the single cheapest gate to add to an existing pipeline?
Idempotency claims before every side effect. Most "the agent did it twice" incidents die at this gate, and it costs a key and a lookup per action.

Should small teams bother with this, or is it enterprise theater?
Small teams feel the failure modes first, because one person is the entire human queue. The gate pattern above scales down well: the point is routing the three irreversible decisions a week to yourself instead of reviewing everything.

Does adversarial drift mean scraping can never go dark?
No — it means scraping goes dark with a drift watch, not dark forever. Outcome signals (schema stability, volume bands, spot-check verification) are the lights that stay on while the station runs unattended.

References

daily.dev Popular feed — trend signal for this piece (top-10 post on AI-agent "dark software factories", 2026-10-23): https://daily.dev/posts?sortBy=popular
Botslave — PHP2Shell patch-operations playbook (dark execution, human-owned plan): https://botslave.com/blog/php2shell-patch-operations-playbook
Botslave — Git workflows for AI agent teams (reviewable, revertable rollouts): https://botslave.com/blog/git-for-ai-agent-teams
Botslave — Email validation and why SMTP probing lies (grounding gaps): https://botslave.com/blog/email-validation-mx-catch-all-smtp-probing
Botslave — Shopee security headers and bot detection (adversarial drift): https://botslave.com/blog/shopee-security-headers-tls-tokens-bot-detection

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

Shopee Security Headers: TLS, Tokens & Bot Detection

Botslave — Thu, 22 Oct 2026 07:42:14 +0000

Shopee security headers do not operate in isolation. To understand why a request succeeds, triggers verification or is rejected, separate at least four layers: the HTTP message, session state, the TLS connection and the application/device context. Matching a URL and a handful of headers does not establish that two requests are equivalent.

This research note moves from fields such as af-ac-enc-dat, af-ac-enc-sz-token and x-sap-access-* to TLS fingerprints, device integrity and bot-detection models. Its purpose is to build a testable method of analysis, rather than attribute every failure to a "missing token" or "emulator detection."

Research scope: selected Shopee Singapore Android test sessions from 15–16 October 2026, reviewed and edited on 21 October 2026. Findings apply to the samples observed, not to Shopee's internal specification. Data examples are illustrative, with substituted values; they are not usable credentials.

1. What are Shopee security headers?

HTTP headers describe aspects of a request or response. Some belong to established HTTP standards; others are defined by an application. "Security headers" is often used loosely for fields associated with authentication, session state, integrity checks or contextual information.

Distinguish a documented purpose from an inferred purpose. For standard headers, consult the relevant specification. For Shopee-specific names, a prefix such as enc, token or access, or a value resembling a timestamp, provides a starting hypothesis rather than an implementation description.

A value that changes after every action could be a request identifier, random data, a signature output, session data or a structure containing several components. Length and frequency of change help frame questions; they do not identify an algorithm by themselves.

Four useful levels of confidence

Observation: a field is present in a sample, has a particular length or changes at a particular time.
Standards-based interpretation: a public specification explains a field or structure.
Hypothesis: an explanation fits the data but has not been isolated through testing.
Finding within a defined scope: controlled comparisons and repeatable results support a conclusion.

Good research preserves the boundaries between these levels. "The value changed between two requests" is more defensible than "This is an HMAC signature that changes every second" when the latter lacks independent support.

2. Classifying headers: session state, context and custom fields

Do not put every header into a single "anti-bot" category. Different groups may serve different functions, and a request can involve several mechanisms at once.

Group	Examples to identify	A useful research question
Content information	`Content-Type`, `Accept`	What does the client send, and which response format does it expect?
Authentication	`Authorization`, where the protocol uses it	Which authentication scheme is involved?
Session state	`Cookie` and application-specific session fields	When was the session created, and which actions changed it?
Browser context	`Origin`, `Referer`, `Sec-Fetch-*`	In which context did the request originate?
Application-specific fields	`af-ac-enc-dat`, `af-ac-enc-sz-token`, `x-sap-access-*`	Where do these occur, and how do they relate to sessions and actions?
Response evidence	HTTP status, response headers and error codes in the body	Is the failure at the transport, HTTP or application layer?

As MDN's Authorization documentation explains, the header carries credentials according to an authentication scheme. There is no single token format for all systems. A request without a Bearer token is therefore not necessarily a request without authentication.

The Cookie header sends relevant cookies from the client to the server. When comparing samples, record session state and the process that established it. Comparing only the final cookie strings can lose important context.

In browsers, Sec-Fetch-Site describes the relationship between the initiating origin and the destination. This is web-context information, not a universal requirement for native Android requests. Chrome's header set should not be the sole benchmark for an Android app.

How should Shopee-specific fields be interpreted?

Field or group	What a sample can establish	What requires additional evidence
`af-ac-enc-dat`	Presence, size, character representation and changes between samples	Specific hardware fields inside it, encryption layers or key location
`af-ac-enc-sz-token`	Presence or absence, and when it appears relative to session establishment	Which component issues it, and its expiration or revocation rules
`x-sap-access-*`	Which members occur in a sample and whether they change together	The exact signing algorithm and which request components are bound to it

Versions, platforms and endpoints do not necessarily use the same fields. Documentation should always identify the version and sample source. This table is a guide to interpreting evidence, not a decoder for private headers.

3. Tokens, signatures, encryption and Base64 are different concepts

A common mistake is to see a long string, decode it as Base64 and call the result "decrypted security data." Separate four concepts:

Encoding: representing data in another form for transmission or storage.
Encryption: protecting confidentiality through a cryptographic mechanism and keys.
MACs and signatures: supporting authenticity or integrity checks, depending on the mechanism.
Tokens: a role-based label; a token may be an opaque reference, structured data or cryptographically protected data.

Base64 in RFC 4648 is a data representation. Recovering bytes from a Base64 string does not establish their meaning or remove an encryption layer inside them. Hexadecimal strings can also represent many different kinds of data.

HMAC in RFC 2104 combines a secret key with a hash function for message authentication. Knowing what HMAC is does not establish that a particular Shopee header uses it. A fixed-length string can fit several different constructions.

An illustrative example: one change, several explanations

Consider a synthetic dataset used to practice analysis:

Sample A: request_id changes; field X changes; session_label stays the same
Sample B: request_id changes; field X changes; session_label stays the same
Sample C: a new session is created; X changes; session_label changes

This supports saying that X varies within a session and after a new session is created. It does not establish that X depends only on time, expires after three minutes or is signed using a particular algorithm. Those claims require additional observations or independent confirmation.

Claims such as "the token always contains an IMEI," "the key must be inside a TEE," or "the header expires after three to five minutes" remain unsupported by sufficient evidence in this research. Recording what is unknown improves the report; it does not reduce its value.

4. TLS, certificate authorities and certificate pinning

A useful conceptual breakdown is:

Connection establishment and transport protection
                     ↓
Readable HTTP messages
                     ↓
Authentication / session / operation validity
                     ↓
Server-side application decision and response

This is an investigative model, not a diagram of Shopee's internal architecture. Success at one layer does not guarantee success at the next.

Capturing a PCAP does not mean reading HTTP

A PCAP records observed network packets. If application data travels inside TLS, collecting the bytes does not automatically provide the session secrets needed to decrypt it. With ephemeral key exchange, a PCAP is not simply a disguised JSON document that can be revealed through ordinary decoding. RFC 8446 describes TLS 1.3's key establishment and data protection.

CA trust and pinning are separate checks

Android Network Security Configuration describes how applications can select trust anchors and constrain accepted certificates or keys. Installing a CA on a device does not guarantee that every application trusts it; accepting a trust chain does not establish that a separate pinning constraint has been satisfied.

When a proxy-related failure occurs, classify it: was a connection established, did TLS complete, and did the application receive a response? An error icon alone does not identify a particular native library or the location of a certificate check.

Readable HTTP can still be rejected

The research notes include a session in which request and response data were readable while a login step still returned an error. This distinction matters: observability and permission to perform an operation are independent questions. Resolving visibility does not automatically resolve account state, session validity or server policy.

5. What do JA3 and JA4 TLS fingerprints tell us?

A TLS fingerprint summarizes aspects of how a client establishes a connection. Cloudflare's JA3/JA4 documentation explains their use in identifying and grouping clients by handshake characteristics, including JA4's normalization choices.

A fingerprint is not a unique identifier for an individual person or phone. Clients with similar configurations can share one. A client's connection characteristics can also change after software or networking changes.

The investigative implication is that "the browser and script have different fingerprints" does not establish "that difference alone caused the rejection." An appropriate control must isolate the factor. Cloudflare's documentation establishes that this technique exists in the industry; it does not establish that Shopee uses Cloudflare or a particular JA implementation in the observed case.

A careful finding would be: "The two environments produced different results and had different connection stacks; the effects of TLS, session state, network and application context have not yet been separated." This still guides further investigation without turning correlation into causation.

6. Bot detection: rules, behavior and risk scoring

An abuse-prevention system need not rely on one algorithm. Cloudflare's bot-detection engine documentation describes a combination of heuristics, client-side signals and machine learning, using information about requests, sessions and browsers.

Three conceptual models can help organize evidence:

Model	How to think about it	What the evidence needs to answer
Rules and heuristics	A condition or combination of conditions leads to an action	Which conditions differ between accepted and rejected samples?
Sequence or behavioral analysis	Context includes several actions over time	Was the sequence and session history recorded completely?
Risk scoring	Several signals contribute to an assessment	Do server logs or other authoritative evidence confirm the score and decision?

This table is a framework for analysis. It is not a reverse-engineered description of Shopee's detection algorithm. Without server-side data, we often observe only the outcome. Different mechanisms can produce the same verification screen.

Why might a "correct token" be insufficient?

In a hypothetical design, a token might establish only one property: that a session was created, a message was issued or a check occurred. Permission to perform the current operation is a separate question. Similar-looking values do not establish that a token can be transferred between contexts.

Why not infer detection thresholds from a few attempts?

Two successful attempts followed by a failure do not establish a three-request threshold. Timing, caching, session state, verification status and unobserved variables may also have changed. A report should state its sample size and observation window rather than turn a small count into a universal rule.

7. Device fingerprints and Play Integrity are not interchangeable

A device fingerprint combines signals that describe an environment. Device integrity is an assessment under a particular mechanism and set of conditions. The concepts are related but not interchangeable.

Google Play Integrity provides verdicts about application integrity, device environments and certain risky interactions. Its verdict documentation explains fields such as appIntegrity, deviceIntegrity and different device labels, whose requirements depend on the label and Android version.

A missing expected label does not automatically explain every application failure. An application may also use signals outside this service. The samples in this research do not include sufficient server-side verdict evidence to establish Play Integrity as the cause of the observed login rejection.

A phone succeeds and an emulator fails: what follows?

The defensible conclusion is that two test configurations produced different results. Hardware alone cannot be isolated if the network, app installation source, session state, operating system and account history also differ.

To strengthen a hypothesis, describe the similarities and differences, keep a group of conditions stable and record what cannot be controlled. Changing a displayed device model name does not establish equivalent integrity, networking or application behavior.

8. Shopee SG case study: what was actually observed?

The research notes from 15–16 October 2026 contain several concrete observations. Only a summary without credentials is published here; raw HAR files and tokens are not provided.

Observation in the research notes	Supported finding	What it does not establish
Some test requests and responses could be read	A portion of the traffic was observable at an interpretable application level	Every protocol and endpoint had been analyzed
An account-check step returned HTTP 418 and code `90309999`	That operation did not complete normally in that sample	The code always means root, a bad IP address or an emulator
Another response returned HTTP 200 with an error in its body	HTTP status and application outcome must be read separately	HTTP 200 means the user successfully logged in
A review of an older HAR found 19 recommended-product cards	The older sample contained some product data	A complete PDP was captured for a product opened in a different test
A newer PCAP retained connection data, but the target product JSON was not readable	Candidate network streams remained available for further analysis	The product-detail request and response had been confirmed from ciphertext alone

A particularly useful lesson is that reviewing evidence can change the scope of a conclusion. Before the older HAR was re-examined, saying "no product data has been obtained" was too broad. After finding recommendation cards, the better conclusion became: "Recommended-product card data exists in an older sample; the search or PDP flow under investigation remains unverified."

That correction is more useful than a general success claim. Recommendation, search and product-detail data may come through different flows. Finding a field in one response does not establish that other fields were obtained or that they share the same source.

Do not assign error-code meanings by guesswork

HTTP 403 indicates that a server refuses to fulfill a request. The status alone does not identify a specific cause. Platform-specific application codes require documentation or suitable controlled evidence before they can be mapped to causes.

In this case, the evidence does not establish a direct mapping between the UI label F13 and 90309999, or make either one conclusive proof of an IP or integrity problem. This remains an open question, not an identified algorithm.

9. A test matrix for isolating causes

Write the question before changing configurations. For example: "Does enabling the observation layer change the outcome of the same operation in the current configuration?" A small question makes the evidence requirements clearer.

Observation axis	Minimum useful record	A design mistake to avoid
Application	Version, installation source, original or modified build	Changing the app build and network simultaneously
Device	OS, environment type and relevant configuration	Recording only a model name and omitting other changes
Network	Connection type, timing and connection changes	Attributing differences to IP when the session also changed
Session	New or continued session; verification state	Comparing a fresh session with a long-established one
Observation	Capture enabled or disabled; which layers are readable	Assuming every connection has the same decryption status
Outcome	UI result, HTTP status, application code and timestamp	Retaining only a screenshot or only an HTTP status

A practical investigative sequence is:

Establish a baseline: record the unchanged configuration's result for the specific operation.
State the hypothesis: describe what should change if it is correct.
Choose the variable: change one factor where possible, documenting exceptions.
Align the timeline: associate the UI action with the correct timestamp and request/response.
Compare layers: separate transport, HTTP and application differences.
Repeat within bounds: record both failures and successes rather than selecting favorable samples.
Define the finding: state what is supported, what has been ruled out and what evidence is missing.

These steps apply to environments and operations within an authorized testing scope. A useful report does not require high-volume traffic; the quality of controls often matters more than the number of requests.

An example test record

Test ID: LAB-EXAMPLE-01
Question: distinguish observation failures from application failures
Time: record timestamp and time zone
Application / OS: record actual versions
Session: new or continued; do not record raw tokens
Single change: describe explicitly
UI outcome: success / verification / error
HTTP and application code: record separately, if readable
Finding: limited to this configuration
Uncontrolled variables: list them rather than filling gaps with assumptions

This is an illustrative template, not a log from a real account or device.

10. Common mistakes in security-header analysis

Treating web and mobile clients as one protocol

Clients may follow different paths. The absence of a browser header in an app is not proof that the app is missing a required field. An app's behavior also does not establish that all web APIs behave identically.

Guessing internal structure from ciphertext

A long string does not reveal a schema. Start with representation, size and changes. If the serialization or encryption layer is unverified, label it "unknown" instead of assigning an algorithm name.

Mixing evidence from different sessions

A UI screenshot from session A, a HAR from session B and a product screenshot from session C can form a convincing but incorrect story. Each finding must reference the correct sample, timestamp and action.

Treating proxy visibility as the application's full view

Data may not pass through the observed component, may remain encrypted or may use a structure the tool cannot parse. A limitation of the observation tool is not a conclusion about the entire application.

Treating one success as a durable workflow

Describe a successful result under its actual conditions. A single sample does not establish long-term reliability, applicability to another account or suitability for a different purpose.

11. Frequently asked questions

What exactly does `af-ac-enc-dat` contain?

This research does not have a verified schema to publish. Characteristics can be catalogued from authorized samples, but the field name alone does not establish that it contains a specific hardware identifier.

Does copying cookies guarantee that a request will be accepted?

The samples in this article provide no such guarantee. Cookies form part of the session context; analysis should also consider how the session was established and which other conditions changed, rather than treating cookies as the client's entire state.

Why can HTTPS be readable while login still fails?

Readable data establishes visibility only for the observed flow. A server can still reject an operation because of session state, verification, permissions or application policy. Identifying the specific reason requires additional evidence.

Does Shopee definitely use JA4 or Play Integrity?

This article does not establish either claim for the tested sessions. Those technologies explain industry mechanisms and help formulate hypotheses; their public documentation is not evidence of Shopee's internal configuration.

Does error `90309999` mean the device has been blocked?

That conclusion cannot be drawn from the status or application code alone. The research notes do not provide an official mapping or controls sufficient to establish a device block, its duration or a single cause.

Can a complete PCAP immediately reveal JSON?

Not necessarily. Distinguish complete capture at the observation level from possession of session secrets, successful decryption and understanding the payload format. These are separate conditions.

Has this research reconstructed Shopee's anti-bot algorithm?

No. It provides a classification of security layers, bounded observations and a method for testing hypotheses. Knowing a few headers or error codes does not reconstruct the server's decision-making model.

12. Conclusions and references

Shopee security headers should be studied as part of a stateful system, not as a list of strings to copy. The lasting value is in locating the failure layer, maintaining evidence provenance and controlling variables in each comparison.

This approach also reflects Botslave's perspective on building systems: establish clarity before adding complexity, retain human ownership of important decisions and progress through steps that can be verified. Explore other notes in the Botslave Journal, or start a conversation about a research question.

Primary technical references

MDN — Authorization: authentication schemes in HTTP.
MDN — Cookie: cookies in requests.
MDN — Sec-Fetch-Site: origin context for web requests.
RFC 4648: Base16, Base32 and Base64.
RFC 2104: the HMAC construction.
RFC 8446: TLS 1.3's design.
Android — Network Security Configuration: trust anchors and certificate pinning.
Cloudflare — JA3/JA4: fingerprints at the TLS layer.
Cloudflare — Bot detection engines: a public example of layered bot detection.
Google — Play Integrity overview and Integrity verdicts: application and device assessments.
MDN — 403 Forbidden: HTTP semantics, not a Shopee application-error catalogue.

Methodology and publication limits

Shopee-specific observations were checked against research notes from 15–16 October 2026. Published summaries exclude account information, cookies, tokens and device identifiers. Public references explain foundational mechanisms; they do not confirm Shopee's private algorithms. Server-side decision logs were not available to establish signal weights or detection thresholds.

Revision history — 21 October 2026: expanded the header classification, token and cryptography distinctions, detection layers, empirical evidence table, controlled-test matrix and FAQs. This English edition preserves the Vietnamese article's scope and uncertainty. Future evidence that changes a finding will be recorded as a revision rather than silently upgrading an earlier hypothesis into fact.

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

PHP2Shell and the Patch-Operations Playbook: Rolling an Emergency Fix Across a Real Fleet

Botslave — Thu, 22 Oct 2026 07:35:33 +0000

The most-read story on daily.dev today is not a new framework or a benchmark — it is an emergency: on October 21, 2026, the PHP project shipped fixes for two critical remote code execution flaws, CVE-2026-109618 and CVE-2026-109619, nicknamed PHP2Shell. Both score 9.2 on CVSS v4.0, and an unauthenticated attacker can run code on a vulnerable server. The news cycle covered the bug itself. This article is the practitioner's version — the problem we actually solve for clients: the fix exists, but does it run everywhere it needs to? The distance between "released" and "deployed" is where incidents happen, and closing that distance is pure automation work. It is the same philosophy as how we work: automate the routine aggressively so human judgment is spent exactly where it changes the outcome.

1. Released is not deployed: the uncomfortable math

The patched releases are PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 (released October 21, 2026, at 14:00 UTC — pulled forward a day after the CVE IDs leaked). The PHP team also published minimal source backports for 7.4, 8.0 and 8.1, which receive no official release.

Now the uncomfortable part: Bitsight's internet-wide scans found that only about a third of the PHP instances where the version could be identified run a branch covered by the security update. The rest sit on older branches, and the most common branch by far is PHP 7.4, with more than two million observations. So for a large share of the real world, "the fix is out" and "we are fixed" are two different sentences. Your job on a day like this is to turn the second one into truth, systematically, before someone else turns it into a breach.

The two flaws themselves sit in the engine's memory management and error handling — code nearly every request touches. CVE-2026-109618 is OOM corruption: after a memory allocation fails, PHP leaves invalid data behind for its cleanup routine to process. CVE-2026-109619 is stale global state: request-owned state survives after its memory is freed, and a later request on the same worker can read or write through the dangling pointers. Every setup serving web requests is exposed — PHP-FPM, mod_php, php-cgi, FrankenPHP, LiteSpeed — with mod_php rated highest risk because the exploit fires before any application code runs.

2. Inventory: find every PHP runtime you own

You cannot patch what you do not know runs PHP. On emergency-patch day, the inventory step is the one most teams skip, and it is the one that decides whether the patch actually covers the fleet. A real inventory pulls from several sources of truth, because PHP hides in all of them:

Container images: Dockerfile and compose files that pin a PHP image tag. This is the fastest place to scan first, because it is text.
Hosts: packages installed via apt/dpkg or rpm/yum, plus anything compiled from source.
Running processes: what the package manager says and what actually serves traffic can disagree.
Serverless and PaaS: runtimes managed by someone else — some vendors auto-patched (Google Cloud patched supported serverless runtimes; Laravel Cloud auto-patches), but custom container images and App Engine flexible deployments need a manual redeploy.
Hosts you do not control: shared hosting, where the fix depends on the provider. Updating WordPress, plugins or themes does not change your PHP version, because PHP lives on the host — check php -v on every server, and ask your host when it will patch if you do not control it.

A first-pass scan can be embarrassingly simple and still catch most of the fleet:

# 1. Every container image pinning a PHP base image
grep -rn "FROM php:" --include='Dockerfile*' --include='docker-compose*' .

# 2. Every host with PHP installed (Debian/Ubuntu fleet via your config management)
dpkg -l | grep -E '^ii\s+php[0-9]' | awk '{print $2, $3}'

# 3. What is actually running right now (catches source builds and stray SAPIs)
ps aux | grep -E 'php-fpm|apache2|httpd|litespeed' | grep -v grep

# 4. What version answers the web (spot-check the externally visible truth)
for host in web1 web2 web3; do
  ssh "$host" 'php -v | head -1'
done

For codebases at scale, this is exactly what Sourcegraph published search queries and batch-change specs for during this event: finding Dockerfiles, CI pipelines and Helm charts that pin old PHP versions, then proposing the bump as a batch change. We have written about this machinery before — our guide to Git workflows for agent teams covers batch changes and reviewable, revertable rollouts, which is the same shape of work: many small, verified, reviewable changes instead of one heroic push.

3. The restart problem: upgrading the package is not the fix

This is the detail that eats patches in production. On Ubuntu, fixes exist for all supported releases — but running apt upgrade does not finish the job, because unattended upgrades do not restart running applications. The old, vulnerable PHP-FPM workers keep serving traffic until you restart them. The same is true for Apache with mod_php. A fleet that is "fully patched" in the package database and still vulnerable in memory is the classic failure mode of emergency patching.

The restart semantics differ by platform, and on patch day you need the real list, not assumptions:

Platform / SAPI	What actually applies the fix
Ubuntu/Debian PHP-FPM	`apt upgrade` + restart `php*-fpm` and the web server; unattended-upgrades will not restart running apps
mod_php (Apache)	Restart Apache; highest risk SAPI, since the exploit fires before application code runs
Laravel Cloud	Auto-patches
Laravel Vapor (PHP 8)	Apps redeployed automatically
Laravel Vapor (PHP 7)	No patch at all — the app must be upgraded off the branch
Laravel Forge	Manual update under Runtime → Versions
Google Cloud serverless	Supported runtimes auto-patched; custom images need a manual redeploy
Cloudways	WordPress instances auto-patched
DigitalOcean Droplets / DOKS / App Platform	Manual action required

So the rollout script has two halves — the version bump and the restart — and the second half is where verification must happen:

#!/bin/bash
# bump-and-restart.sh — the boring half that actually matters
set -euo pipefail
sudo apt update && sudo apt upgrade -y php8.4-fpm
# Restart the runtime AND the web server; the old workers stay in memory otherwise
sudo systemctl restart php8.4-fpm
sudo systemctl restart apache2 || sudo systemctl restart nginx
# Verify the fix is the thing actually running, not just installed
php -v | head -1

The principle generalizes beyond PHP: a patch is only real when the vulnerable code is no longer in memory. Anything in your rollout that checks the package database but not the running process is checking the wrong thing.

4. Rollout mechanics: canary, stages, re-runnable automation

Emergency does not mean reckless. A staged rollout with a canary still beats a fleet-wide push on a CVE day, because the patch itself is a change to every request path in your application, and engine-level patches have historically broken extensions and edge-case behavior. The mechanics that work:

Patch one canary host or one non-critical app first, watch error rates and latency for a full traffic cycle, then proceed.
Make the rollout re-runnable (idempotent). The same script run twice must converge to the patched state, not fail or double-apply. On a day when you are patching at 2 a.m. with tired operators, re-runnable beats clever.
Keep it reversible. Engine patches occasionally break third-party extensions; know how to roll back to the previous package version before you start, not after.
Batch the config changes. Dockerfile pins, CI pipeline images and Helm chart tags should be bumped as reviewable changes with CI green — not hand-edited on servers. This is where the batch-change approach from our agent Git workflows post earns its keep: dozens of small, verified, revertable changes instead of snowflake servers.

5. The EOL fleet: what to do with 7.4, 8.0 and 8.1

A large share of the internet runs PHP branches that get no official release — 7.4 alone had more than two million observations in Bitsight's scans, and W3Techs data puts about 48.5% of PHP sites on end-of-life branches. The PHP project published minimal source patches for 7.4, 8.0 and 8.1 to help with backports, but that is not a supported release; your vendor's security advisory is the authority on whether a patched package exists for your platform.

Be honest about the decision tree:

Vendor package exists: take it, then restart (section 3 still applies).
No package, and you can upgrade the branch: schedule it — this event is the forcing function.
No package and no upgrade path right now: this is where CERT-EU, the NSA and CISA landed — isolate those systems or take them offline. Several governments did exactly that: Dutch government services, Finland's state IT provider, German and French federal IT all pulled sites down rather than run exposed.

"Awaiting the vendor" is not a plan. A written decision with a date — patched, isolated, or offline — is.

6. Defense in depth while the rollout runs

Patching a fleet takes time: inventory, maintenance windows, application testing against the new builds. During that window, boundary controls reduce exposure to the confirmed attack paths. Cloudflare turned on default WAF rules for its customers about nine hours before disclosure, after attackers began probing, and AWS published a guide with AWS WAF rules for the confirmed PHP2Shell attack paths. A reverse proxy or WAF in front of PHP can block hostile requests before they reach the vulnerable parser.

Two caveats, both worth stating plainly:

A WAF does not fix PHP. It covers known attack patterns; the vulnerable code is still underneath. Patching is the only complete fix, and it stays priority one.
Expect payload mutation. Cloudflare reported attackers already adapting payloads to evade its rules. Static rules buy time; they do not end the race.

Detection matters as much as blocking. Datadog's analysis suggests looking for large form submissions, memory-exhaustion errors during request initialization, and unexpected shell processes spawned by the web server user — and notes that any PHP script exposes the vulnerable path, even one that only prints a response, because the request never reaches application code. Practical log checks:

# Requests hitting the session upload-progress path (the reported trigger surface)
grep -E 'PHP_SESSION_UPLOAD_PROGRESS' /var/log/apache2/access.log | tail -20

# The literal calling card some scanners used
grep -i 'php2shell' /var/log/nginx/access.log | awk '{print $1}' | sort -u

# Shell processes that should never be children of the web server user
ps -ef | awk '$1=="www-data" && /sh|bash|wget|curl/ {print}'

Arctic Wolf Labs reported scanning and attempts pulling ELF and Golang malware binaries via wget — so unexpected outbound fetch-and-execute behavior from web-server-owned processes is exactly the signal to alert on. (On the bot-detection side of boundary defense, we have a full breakdown of TLS fingerprints and token-based bot detection — the same layer where these WAF rules live.)

7. Verify the patch actually landed

The rollout is not done when the script exits green. It is done when every runtime in the inventory answers with a fixed version. Close the loop:

#!/bin/bash
# verify-php-fleet.sh — the inventory from section 2, re-run as an assertion
FIXED=("8.5.12" "8.4.27" "8.3.36" "8.2.35")
for host in $(cat fleet-hosts.txt); do
  ver=$(ssh "$host" 'php -r "echo PHP_VERSION;"')
  ok="no"
  for f in "${FIXED[@]}"; do [[ "$ver" == "$f" ]] && ok="yes"; done
  # Also confirm the running workers are the new build, not stale memory
  echo "$host -> php $ver | patched: $ok"
done | tee patch-verification-$(date +%F).log

Keep that log. When the next CVE drops — and Taylor Otwell's warning during this event was that this probably will not be the last time, nor PHP the only language to go through it — the difference between a two-hour response and a two-day scramble is whether your inventory already exists. The fleet list, the restart runbook and the verification script are assets. Build them once, on a calm day, and CVE day becomes execution instead of archaeology.

8. Frequently asked questions

Does updating WordPress, my plugins or my theme fix PHP2Shell?
No. The vulnerability is in the PHP runtime itself, which lives on the host — below your application. Update the runtime (and restart it), regardless of what your CMS or framework reports.

Does putting a WAF in front fix it?
No. WAF rules from Cloudflare, AWS and others reduce exposure to confirmed attack paths while you patch, but they do not remove the vulnerability. Patching is the only complete fix.

I run PHP 7.4. Am I covered?
Not by an official release. The PHP project published minimal source patches for 7.4, 8.0 and 8.1 to aid backports; check your OS vendor's security advisory for patched packages. If neither exists, the official guidance from CERT-EU, NSA and CISA is to isolate the system or take it offline — or upgrade the branch.

How do I know if someone already exploited it against me?
Look for the signals in section 6: large form submissions, memory-exhaustion errors during request initialization, unexpected shell processes or wget/curl activity owned by the web-server user, and php2shell strings in access logs. Absence of these signals is reassuring, not proof.

Why is this suddenly everywhere?
Both flaws were found with frontier AI assistance — Calif's team with OpenAI's models, and Wiz independently with its Atlas tool — and one researcher demonstrated that a working proof of concept was roughly "one prompt away" once hints leaked. The finding side of security is accelerating; the patching side has to keep up, which is exactly what the playbook above is for.

References

Tenable Research — PHP2Shell FAQ (disclosure timeline, CVE details, affected SAPIs): https://www.tenable.com/blog/php2shell-cve-2026-109618-cve-2026-109619-frequently-asked-questions-about-critical-php-remote
AWS Security Blog — using AWS WAF to mitigate PHP2Shell while patching: https://aws.amazon.com/blogs/security/using-aws-waf-to-apply-mitigations-for-php2shell-rce-vulnerabilities-cve-2026-109618-and-cve-2026-109619/
Datadog Security Labs — detection, analysis and remediation: https://securitylabs.datadoghq.com/articles/php2shell-critical-vulnerabilities-in-php/
Bitsight — vulnerability details and internet-wide version exposure data: https://www.bitsight.com/blog/critical-vulnerability-alert-cve-2026-109618-and-cve-2026-109619-php2shell
Brandefense — attack timeline and official alerts roundup: https://brandefense.io/blog/php2shell-php-rce-cve-2026-109618-109619/
SOCRadar — FAQ separating confirmed facts from leaked/unverified claims: https://socradar.io/blog/php2shell-cve-2026-109618-109619/

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

How to Validate Email Addresses Without Sending Mail: MX Records, Catch-Alls, SMTP Probing

Botslave — Thu, 22 Oct 2026 07:33:41 +0000

Every lead pipeline eventually asks the same question: "is this email address real?" The honest answer is that "real" is four different questions wearing one coat — and each layer of validation only answers one of them. Confusing the layers is how teams end up trusting a green checkmark that proves almost nothing, or throwing away good leads because one weak signal said no.

This is the validation model we use in our own lead operations work: what each check actually proves, where SMTP probing helps, where it quietly lies to you, and how to design a pipeline that records evidence instead of pretending to know the truth. The thinking behind this staged approach — automate the routine, keep judgment where it matters — is the same one behind how we work.

1. Four questions, not one

When someone says an address is "valid", they might mean:

Is it well-formed? Does it parse as an email address at all?
Can the domain receive mail? Is there a mail server willing to accept messages for this domain?
Does this mailbox exist? Is there an actual account behind the local part?
Is it safe and worthwhile to send to? Will sending hurt sender reputation, or reach a human who matters?

Syntax checks answer question 1. MX lookups answer question 2. Question 3 can only be approximated without sending mail — and question 4 is a policy decision, not a technical one. Most "email verifier" scores collapse all four into a single number. That is convenient and wrong: a catch-all corporate domain and a personal Gmail account can both score "valid" while meaning completely different things for your pipeline.

2. Layer one: syntax, and why it is not trivial

The grammar of an email address is defined by RFC 5322, and it is more permissive than almost every hand-rolled regex assumes. Quoted local parts ("john..doe"@example.com) are legal. So are comments in obsolete syntax that no sane system should accept but some parsers do. In practice:

Use a real parser from your platform's ecosystem, not a regex copied from a forum. Then apply a stricter business rule on top: most pipelines reject quoted strings, consecutive dots and addresses longer than 254 characters, not because the RFC forbids all of them, but because real-world mail systems handle them inconsistently.
Normalize before you validate. Trim whitespace, lowercase the domain (domains are case-insensitive; local parts technically are not, though nearly every provider treats them case-insensitively), and convert internationalized domains to their ASCII form with IDNA/punycode — you must validate and resolve the form DNS actually uses.
Do not "fix" addresses silently. Plus-addressing (name+tag@gmail.com) and Gmail's dot-insensitivity are provider semantics: the same mailbox, but stripping tags or dots to deduplicate is a dedupe decision, not a syntax correction. Record the original; derive a canonical form separately if your pipeline needs one.

Syntax validation is cheap and deterministic. It is also the layer that proves the least. Everything interesting happens at the domain.

3. Layer two: MX records — the domain's front door

To find out where mail for a domain goes, you ask DNS for its MX (mail exchanger) records. Each record has a preference value (lower wins) and a host name. Two lookups cover most of the world:

// Node.js — no dependencies, built-in dns module
const dns = require('node:dns/promises');

async function mxFor(domain) {
  try {
    const records = await dns.resolveMx(domain);
    return records.sort((a, b) => a.priority - b.priority);
  } catch (e) {
    if (e.code === 'ENODATA' || e.code === 'ENOTFOUND') return [];
    throw e;
  }
}

# Python — with dnspython (pip install dnspython)
import dns.resolver

def mx_for(domain: str):
    try:
        answers = dns.resolver.resolve(domain, "MX")
        return sorted((r.preference, str(r.exchange)) for r in answers)
    except (dns.resolver.NoAnswer, dns.resolver.NXDOMAIN):
        return []

Three outcomes matter:

MX records exist → the domain publishes where its mail goes. You have also learned something else for free: the MX hosts usually reveal the provider (Google Workspace, Microsoft 365, a regional hoster), which tells you which behavioral rules to expect later.
A single MX record pointing at "." (null MX, RFC 7505) → the domain explicitly declares it accepts no mail. This is a hard, standards-based "invalid" — one of the few you will ever get. Treat any address at such a domain as undeliverable, permanently.
No MX records at all → strictly, RFC 5321 §5.1 defines an "implicit MX": fall back to the domain's A/AAAA records. In practice, domains with A records but no MX are a grey zone — personal sites, parked domains, misconfigurations. Most lead pipelines mark them low-confidence rather than valid, and they are right to: a domain that wanted mail would publish MX.

What MX tells you is strictly question 2: a front door exists. It says nothing about whether your specific mailbox is behind it.

4. Layer three: SMTP probing — useful, and a compulsive liar

The classic trick for question 3 is to start an SMTP conversation with the receiving server and stop just before sending anything:

S: 220 mx.example.net ESMTP ready
C: EHLO validator.example.com
S: 250 Hello
C: MAIL FROM:<check@example.com>
S: 250 OK
C: RCPT TO:<person@example.com>
S: 250 OK            <- mailbox accepted?
C: RSET / QUIT

(Illustrative session, not a captured transcript.)

A 550 here often means "no such user" and a 250 is taken as "mailbox exists". Both readings are unreliable, for structural reasons:

Accept-all edges. Many servers — especially corporate gateways and Microsoft/Google tenants fronted by security appliances — accept RCPT TO for any local part and decide what to do with the message later. Your probe gets a 250 for mailboxes that do not exist, and the bounce only happens after a real send, if it happens at all (some systems silently discard).
Anti-enumeration defenses. Large providers deliberately stopped answering honestly years ago, precisely because probing was used to harvest valid addresses. A 250 from them is close to meaningless; a 550 may be policy, not absence.
Greylisting and tempfails. A 450/451 can mean "come back later" (greylisting), rate limiting of your probing IP, or a tarpit deliberately slowing you down. It is not a verdict in either direction.
Your reputation, not theirs. Probing is done from your IP. Datacenter IPs are frequently refused outright, and sustained probing patterns get IPs listed. The probe measures your standing with their filters as much as the mailbox.

The professional conclusion: an SMTP probe is a weak signal, strongest when it returns an explicit 550 user unknown from a server you have already established is not catch-all, and weakest everywhere else. Which raises the real question — how do you know a domain is catch-all?

5. Catch-all domains: where mailbox validation ends

A catch-all (accept-all) domain accepts mail for every local part, real or invented. The standard detection is simple: probe the server with a local part that cannot exist — a long random string:

RCPT TO:<xq7zv2k9plm4rf8d@example.com>
250 OK   -> this domain accepts anything: it is catch-all (at this edge, today)

If the random address is accepted, mailbox-level validation for that domain is over. Every address there is "unverifiable" by construction — not valid, not invalid. In B2B lead data this is not an edge case: small and mid-size companies land on catch-all configurations all the time, through default hoster settings, migration leftovers, or a deliberate "never miss a message" choice by their admin.

How a pipeline should treat catch-all addresses:

Never merge them into "valid". Keep a separate bucket with its own sending policy (for outreach: send later, send fewer, watch the first bounces closely).
Never auto-delete them either. A catch-all domain contains real people; the CEO's address is in there next to a thousand invented strings. Deleting the bucket deletes your best leads along with the noise.
Cache the verdict per domain, not per address, and re-check periodically — catch-all status is a server configuration, and configurations change.

This is the same discipline we apply to any layered signal system: in our research into Shopee's security headers, the recurring lesson was that one signal rarely means what it appears to mean until you know which layer produced it. Email is no different.

6. Supporting signals that are actually worth collecting

Around the three main layers, a few cheap checks add real value:

Disposable/temporary domains. Maintain or license a list of throwaway-mail domains. The address may be perfectly valid and worthless in ten minutes — that is a question-4 decision, and for lead pipelines the answer is usually reject.
Role accounts. postmaster@, admin@, sales@, contato@ are deliverable but shared inboxes. Tag them; do not drop them — in small companies the role inbox is the decision-maker's inbox.
Provider fingerprint from MX. Knowing a domain sits on Google Workspace vs. a small regional hoster changes how you interpret probes, bounces and rate limits downstream.
Domain typos. gmial.com, hotmial.com and friends are data-entry errors, not mailboxes. Suggest the correction to a human or a form; never silently rewrite an address and send to a stranger's mailbox.
Free vs. corporate domain. Not a validity signal at all — but valuable routing metadata for enrichment and segmentation.

7. Designing the pipeline: verdicts with evidence, not scores

A robust validation stage stores, per address: the syntax result, the MX set and provider, the null-MX flag, the catch-all verdict (and when it was measured), the probe outcome if you probe at all, and timestamps for everything. The output is a label with a reason, for example:

| Evidence | Label | Pipeline action |
|---|---|
| Bad syntax / NXDOMAIN | invalid | Drop, log reason |
| Null MX (RFC 7505) | invalid | Drop, do not retry |
| No MX, A record only | low-confidence | Hold for review or enrich later |
| Catch-all domain | unverifiable | Separate bucket, cautious policy |
| MX ok, probe 550 user-unknown, not catch-all | likely-invalid | Drop, keep evidence |
| MX ok, everything else neutral | deliverable | Proceed |

Two design rules make this work over time. First, never overwrite the raw input — store the pipeline's conclusions next to the original address, so a better future check can re-derive them. Second, revalidate on a schedule: MX records move, domains lapse, catch-all gets switched off. A validation result is a measurement with a date, not a property of the address. (It is the same reason our automations are built to be safely re-runnable — idempotent by design, so re-checking never corrupts state.)

8. Frequently asked questions

Does a 250 response to RCPT TO guarantee the mailbox exists?
No. On catch-all and accept-all systems it only proves the server accepts the address at the edge. Delivery — or a later bounce — is decided afterwards.

Is SMTP probing illegal or abusive?
Probing is a normal protocol conversation, but at scale it looks exactly like address harvesting, which is what the defenses are built against. Expect blocks, respect them, and prefer MX-level checks plus evidence-based scoring for bulk hygiene. Sending one real, wanted email remains the only ground truth.

Can I skip probing entirely?
For list hygiene, mostly yes: syntax + MX + null MX + catch-all detection removes the addresses that damage sender reputation (hard-invalid domains), which is where most of the value is. Probing adds a marginal signal at a real operational cost.

How often should a lead list be revalidated?
Before any major send, and periodically for standing lists — domains and mail configurations change constantly. A verdict older than a few months should be treated as stale until re-measured.

References

RFC 5321 §5.1 — address resolution and the implicit MX rule: https://www.rfc-editor.org/info/rfc5321/
RFC 7505 — Null MX, "a domain that accepts no mail": https://www.rfc-editor.org/info/rfc7505/
RFC 5322 — Internet Message Format (address grammar): https://www.rfc-editor.org/info/rfc5322/
Node.js DNS module documentation: https://nodejs.org/api/dns.html
dnspython documentation: https://dnspython.readthedocs.io/

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.