DEV Community: Nguyen Tu

LeadOps Autopilot: Sourcing and Scoring US Clinic Leads from Public Data

Nguyen Tu — Fri, 23 Oct 2026 08:53:03 +0000

Most lead lists are bought, stale, and sprayed. Ours is none of the three — and the interesting part is not the scraping. It is everything the pipeline refuses to do.

LeadOps Autopilot is the outbound system we built at Botslave to find US clinics that plausibly need lead-intake and follow-up automation — dental practices, med spas, physical therapy, mental health, urgent care — and to prepare outreach that a human approves before anything is sent. This note documents how it actually works: where the data comes from, how a lead earns its score, and why the pipeline deliberately stops one step short of sending.

The problem with clinic lead data

Clinics are a hard segment to source well. Directory data is fragmented across official registries, map platforms and review sites, and it decays fast: practices move, rebrand, change booking systems. A purchased list tells you a clinic exists. It tells you nothing about whether that clinic has the operational shape your offer fits — a working website, an online booking path, signs of inbound volume, a contact channel a human actually reads.

So instead of buying conclusions, the pipeline collects raw public evidence and scores it. The score is a hypothesis with receipts: every lead carries up to eight recorded reasons, so a reviewer can see why the machine thinks this clinic fits before approving any contact.

Stage 1: Collect from sources that are allowed to be boring

Two collectors feed the pipeline, chosen for provenance rather than volume:

CMS/NPPES. The US National Plan and Provider Enumeration System is the official registry of healthcare providers. collect_nppes.py pulls clinic organizations by state, city and taxonomy — slow, structured, and authoritative about existence.
OpenStreetMap via the Overpass API. collect_osm_overpass.py discovers clinics geographically at zero data cost. Map data is uneven — excellent in some metros, thin in others — which is exactly why it is a discovery layer, not a source of truth.

An enrichment pass (enrich_websites_google_places.py) then resolves or confirms each clinic's website, because the website is where the scoring evidence lives. A clinic without a findable website is out of scope for this offer by definition — the README's avoid-list also excludes hospitals and large health systems, whose procurement cycles make them a poor fit, and anything that would require collecting patient health information. That last line is architectural, not decorative: the pipeline handles business contact data only.

Stage 2: Read the homepage, not the listing

The scorer (leadops.py) fetches each clinic's homepage and scores what it finds there. The model is intentionally simple enough to argue with:

Every lead starts from a base score of 35 — existence is worth something, not much.
Fit signals found on the site add points: evidence that the clinic takes its inbound flow seriously (or visibly struggles with it) moves the score up.
Negative signals subtract (a flat −8 penalty in the current rules), and a homepage that cannot be fetched at all costs the lead 35 points — if we cannot see your front door, we do not get to pitch you a better one.
Scores are clamped to 0–100 and mapped to fit tiers at 75, 58 and 45, so review batches sort themselves: work the top tier, sample the middle, ignore the rest.

Alongside the score, the pipeline generates a pain hypothesis from the recorded reasons — a one-line theory of what this clinic's intake problem probably is — and drafts outreach copy from it. The hypothesis is labeled as a hypothesis everywhere it appears. That discipline matters more than the model: a scorer that shows its reasons can be corrected; a black-box score just gets believed.

Segmentation runs in parallel (segments.py): dental, physical therapy and the other verticals each carry their own keyword rules, so batches, scoring context and copy stay separated by vertical instead of dissolving into one generic blast list.

Stage 3: The gate — where the pipeline stops on purpose

Here is the design decision people question most: the MVP stops before sending. Its output is a reviewed CSV — leads, scores, reasons, segments, draft messages — and a human approves the high-fit rows before any contact happens. Later stages exist in the codebase (CRM preparation, a sender pool, reply processing), but they sit behind that approval, not instead of it.

This is the same principle as how we work on client systems: automate the routine aggressively, and spend human judgment exactly where it changes the outcome. In outbound, the judgment step is "should this business hear from us at all?" Getting that wrong at scale is how domains get burned and studios get a reputation. The 60-second promise LeadOps makes to clinics — every inbound inquiry becomes a qualified record, an assigned owner and a follow-up task within 60 seconds — only works as a pitch if our own outbound shows the same discipline we are selling.

What this architecture buys you

Provenance per lead. Source, score, reasons and segment travel with the record into the CRM, so six months later you can still answer "why did we contact them?"
Cheap iteration. Scoring rules are plain keyword-and-signal logic in one file. When a vertical underperforms, you read the reasons on the rejected leads and adjust the rule — no model retraining, no vendor ticket.
Compliance by construction. Public business data only, no patient information anywhere in the pipeline, human approval before contact. The constraints were chosen at design time, when they are cheap.
Failure is visible. An unscored or unreachable lead is a row with a low score and a reason, not a silent gap in a purchased spreadsheet.

Where it goes next

The honest limits: OpenStreetMap coverage varies by city, homepage signals are a proxy for operational pain (a beautiful site can hide a chaotic front desk), and a human gate caps throughput by design. We accept all three. The next build-notes entries in this series go deeper on the scoring model itself and on what the approval gate has caught that automation would have sent.

If you run a clinic group — or any business whose inbound inquiries leak between the website and the follow-up — this is the class of system we build. See what we build, or read the companion piece on validating email addresses without sending mail, which covers the verification layer that contact data passes through before it is trusted.

Frequently asked questions

Why not just buy a clinic list?
Because the list answers the wrong question. Existence is cheap; fit is the scarce signal, and fit lives on the clinic's own website, not in the directory row.

Is OpenStreetMap data reliable enough for this?
As a discovery layer, yes — with an authoritative registry (NPPES) alongside it and a scoring stage that penalizes what it cannot verify. No single source is trusted alone.

Does the system send the outreach itself?
The pipeline prepares scored leads and draft messages; a human approves before contact. Sending mechanics exist behind that gate, by design.

What data does it collect?
Public business information: practice names, websites, published contact points, and the evidence used for scoring. No patient health information is collected or processed.

References

NPPES (National Plan and Provider Enumeration System): https://nppes.cms.hhs.gov
OpenStreetMap Overpass API: https://overpass-api.de
Google Places API: https://developers.google.com/maps/documentation/places/web-service
Botslave — How we work: https://botslave.com/how-we-work
Botslave — Services: https://botslave.com/services

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

Dark Software Factories: What Runs Lights-Out and What Breaks Without a Human in the Loop

Nguyen Tu — Fri, 23 Oct 2026 01:32:37 +0000

The tenth most-read story on daily.dev today is a question about the industry's newest industrial metaphor: are "dark software factories" the next big thing? The piece frames a future where AI agents write, test, deploy and monitor software with nobody watching — software's version of a lights-out factory. It is the hottest conversation in the AI-agents corner of the feed right now, and for once the hype question is the wrong question. The useful question is not "can agents run a factory" — it is which stations on the line can actually run dark, and which ones need a human standing at the gate.

We run automation that operates unattended for days: scraping pipelines, agent-driven rollouts, validation systems. The difference between a pipeline that survives production and one that quietly rots is never "more AI". It is the placement of the humans. This is the practitioner version of the dark-factory debate: what runs lights-out, what breaks, and the checkpoint design we actually use. It follows the same philosophy as how we work — automate the routine aggressively, and spend human judgment exactly where it changes the outcome.

1. The borrowed term, and what the borrowing gets wrong

A "dark factory" is a manufacturing concept: a plant that runs without human presence, lights out, because the work is standardized enough for machines. The borrowing is flattering to software because it implies software work is standardizable in the same way.

It is not. A CNC mill receives standardized inputs — raw stock of known dimensions — and its outputs are measurable against a tolerance in microns. Failure modes are bounded and enumerated in advance. Software work has none of those properties:

Inputs are adversarial and non-standard. Requirements arrive as prose. Dependencies update themselves. Third-party targets change their headers, rotate their fingerprints, and deploy new bot detection on a Tuesday (we have the scars: our Shopee security research documented a full anti-bot stack that shifts under you).
Outputs are hard to verify. A mill measures the part; an agent ships a diff that "passes CI" while subtly changing behavior. Verification in software is usually another program with its own blind spots — not a caliper.
The factory floor fights back. Manufacturing robots do not face an adversary. Scraping, deliverability, fraud detection and security do. An unattended pipeline in an adversarial environment does not degrade gracefully; it degrades silently.

None of this means lights-out is a fantasy. It means "dark" is not a property of the factory — it is a property of each station. Some stations are dark-ready. Others break specifically because nobody is there.

2. What actually runs lights-out today (and why)

Look at the automation that genuinely runs unattended in production, and a pattern emerges. A station can go dark when it satisfies three properties at once:

Bounded inputs. The agent's decision space is finite and enumerable.
Cheap, independent verification. You can check the result without the same judgment that produced it.
Reversible failure. When it goes wrong, you can roll back and the world is as it was.

Concrete stations that satisfy all three:

Deterministic transforms. Formatting, linting, code generation from schemas (OpenAPI → clients, protobuf → stubs). The input is a spec; the verifier is a compiler or a diff against the previous generation. This is dark today and has been for years.
Dependency-bump proposals. An agent opens the pull request bumping a library; the gate is the existing test suite plus a policy check on the version range. The agent never merges — the gate decides. Reversible by revert.
Metric-gated rollouts. The deployment pipeline from our PHP2Shell patch playbook: canary, health signals, automatic rollback. The rollout executes unattended; the decision to continue is a verification function with thresholds, not a person. When the threshold trips, a human is paged — the factory has lights at the exact station where darkness becomes expensive.

Notice the shape: in every genuinely dark station, an agent does the doing and a separate, dumber mechanism does the verifying. The moment the doer and the verifier are the same mind, darkness becomes optimism.

3. The failure taxonomy: where lights-out breaks

Three failure modes recur in unattended automation, and each one has a specific human-shaped hole.

Grounding gaps: the proxy is not the thing. Agents optimize whatever signal you give them. Give them "all tests pass" and you get green tests, not correct behavior. This is not a theoretical alignment problem — it is a Tuesday problem. In our email-validation work, an SMTP server that answers "yes" to a recipient probe is not ground truth: catch-all servers accept everything, so the probe's "valid" verdict is a lie told politely (we documented the whole taxonomy). Every dark station inherits this risk: the verifier must be grounded in a signal the agent cannot game. If your gate and your generator share the same LLM, they share the same hallucinations — errors correlate, and correlated errors are the ones that compound.

Adversarial drift: the floor moves. An unattended pipeline in a contested environment rots. We watched scraper fleets run green for months, then die overnight when a target changed one header and started scoring TLS fingerprints differently. The automation was correct; the world moved. Lights-out works where the world is stationary. Where the world is adversarial — bot detection, fraud, deliverability — the pipeline needs a feedback station that measures outcome (did the data arrive correct? did the mail land?), not just execution (did the job run?). Drift detection is the station that keeps the factory's lights on.

Irreversibility: blast radius is the real risk metric. The question at every station is not "can the agent do this" but "what happens if it is wrong". Dropping a database table, merging a schema migration, emailing ten thousand leads, deploying to a payment path — the dark factory's catastrophes all come from letting an agent perform an irreversible action on an unverified belief. Reversibility, not capability, should decide what goes dark.

4. The checkpoint design: gates keyed to blast radius

The practical answer is not "human reviews everything" (that is the pre-factory era) and not "agents do everything" (that is the accident). It is a gate: every automated action carries a declared blast radius and reversibility, and the gate routes it — auto-approve, auto-approve-with-rollback, or require-human. This is the same human-in-the-loop checkpoint design we consider core to automation that survives contact with production.

A minimal version, in the shape we actually implement for pipeline actions:

from dataclasses import dataclass
from enum import Enum

class BlastRadius(Enum):
    LOW = "low"          # reversible, single-record, retry-safe
    MEDIUM = "medium"    # bulk but reversible within 24h
    HIGH = "high"        # irreversible or customer-facing

@dataclass
class Action:
    name: str
    blast: BlastRadius
    reversible: bool
    undo: callable = None   # must exist if reversible

def gate(action: Action, verifier) -> str:
    """Route an automated action. Returns 'auto', 'auto+watch', or 'human'."""
    # 1. Claim before work: idempotency first — a gate that can't tell
    # "already done" from "do it again" is not a gate, it's a gun.
    if verifier.already_done(action):
        return "skip"

    # 2. Irreversible or high blast radius: a human signs, no exceptions.
    if action.blast == BlastRadius.HIGH or not action.reversible:
        return "human"

    # 3. Medium: run, but watch. Verification is a separate, dumber
    # mechanism than the agent that proposed the action — outcome
    # signals, not "it says it worked".
    if action.blast == BlastRadius.MEDIUM:
        return "auto+watch"

    # 4. Low + reversible + idempotent: genuinely dark.
    return "auto"

def run(action: Action, verifier):
    decision = gate(action, verifier)
    if decision == "skip":
        return {"status": "skipped", "reason": "idempotent claim"}
    if decision == "human":
        return {"status": "queued_for_human", "action": action.name}
    result = action.execute()
    # Auto-rollback on outcome failure, not on self-report.
    if not verifier.outcome_ok(action, result):
        if action.reversible and action.undo:
            action.undo()
        return {"status": "rolled_back", "action": action.name}
    if decision == "auto+watch":
        verifier.register_watch(action, window="24h")
    return {"status": "done", "action": action.name}

Four properties make this more than pseudocode optimism:

Idempotency is the first gate (already_done). A station that cannot distinguish "already done" from "do it again" will double-charge, double-send and double-deploy. Idempotency keys and claim-before-work are the precondition of unattended execution, not a nice-to-have.
The verifier is dumber than the agent on purpose. outcome_ok checks outcomes — the data arrived, the record matches, the canary metrics hold — using signals the proposing agent cannot produce. If the same model both proposes and judges, you have theater with two acts.
auto+watch is the middle that most designs skip. Medium-blast actions run but register a 24-hour outcome watch. Drift detection lives here: did the mail actually land, did the scrape return the same schema, did the fingerprint still verify? This is the station that catches adversarial drift while the factory runs dark.
The human queue is an asset, not a failure. queued_for_human is a designed output. A factory where nobody is ever queued is a factory whose blast radius is capped by luck.

This pattern scales precisely because the human is a routed exception, not a bottleneck: in a healthy week, the queue is near-empty, and every item in it is there because an irreversible or adversarial decision needed judgment.

5. The lights-out scorecard

Apply the taxonomy station by station and you get a working map. "Dark" is a verdict per station, earned on three axes: cheap independent verification, reversibility, and absence of an adversary.

Pipeline station	Independent verification cheap?	Reversible?	Adversary present?	Verdict
Code formatting / linting	Yes (deterministic)	Yes	No	Dark
Schema → client generation	Yes (compile + diff)	Yes	No	Dark
Dependency-bump PRs	Yes (existing test suite)	Yes (revert)	Supply-chain risk at merge	Dark until merge; human merges
Agent-written feature code	Partially (tests are written by the same mind)	Yes	No	Auto + human review
Test-plan-free "self-review" by the generating agent	No (same mind judges)	n/a	No	Not a gate — do not ship on this
Scraping at scale	Only via outcome signals (schema, volume, spot-checks)	Yes	Yes (bot detection)	Dark + drift watch, never dark forever
Email/deliverability pipelines	Only via outcome signals (inbox placement, bounces)	Sends are not	Yes (filters, catch-alls lie)	Human on sends; dark on prep
Canary rollouts with auto-rollback	Yes (health metrics)	Yes	No	Dark; page a human on rollback
Schema migrations / destructive DB ops	Hard (correctness is the app)	No	No	Human signs
Bulk customer-facing comms	Hard (tone, accuracy, timing)	No	Reputation risk	Human signs
Emergency fleet patching	Yes (version probes, health checks)	Mostly	Attackers racing you	Dark execution, human owns the plan

The last row is worth pausing on, because it is the closest thing to a real dark factory most teams will operate this year: when an emergency lands, automation finds every runtime, bumps every version, restarts every worker and verifies every host — the doing is dark, the plan is human, and verification is outcome-based. That is the shape that works, and it generalizes.

6. Where the debate should go next

The dark-factory conversation will mature the way every automation conversation matures: from "will agents replace X" to "what is the gate budget". A gate budget is the honest accounting of a pipeline: how many human judgments per hundred automated actions, and at which stations. Teams that run real unattended automation already keep this ledger informally — the on-call rotation is a gate budget with a pager.

Three predictions from the floor, not the balcony:

The winning unit is the gated agent, not the autonomous one. Frameworks will compete on how well they declare blast radius, enforce idempotency, and route to humans — the boring infrastructure of the gate() function above.
Drift detection becomes a product category. In adversarial environments, "did the world move" is a first-class signal, and pipelines that cannot answer it are not dark — they are blind.
The human queue gets a UI. Right now it is Slack pings and dread. Mature dark factories will treat the human-review queue like a work queue: prioritized, SLAs, audit trail — the station with the lights on, designed as carefully as any dark one.

The factory can be as dark as its gates are good. Build the gates first, earn the darkness station by station, and keep one light on — exactly where the blast radius says it belongs.

Frequently asked questions

Is a "dark software factory" just hype for "CI/CD with agents"?
Mostly, today. The new part is agents proposing changes across the whole lifecycle, not just running a fixed pipeline. The discipline that makes it work — gates, idempotency, outcome verification — is the same discipline CI/CD taught us; the blast radius is larger, so the gates must be better.

Can tests be the verifier for agent-written code?
Only if the tests are not written by the same mind that wrote the code. An agent that writes code and its tests, then reports "all green", has optimized a proxy. Independent verification means a signal the generator cannot produce: existing golden tests, property-based checks against a spec, production shadow traffic, or a human.

What is the single cheapest gate to add to an existing pipeline?
Idempotency claims before every side effect. Most "the agent did it twice" incidents die at this gate, and it costs a key and a lookup per action.

Should small teams bother with this, or is it enterprise theater?
Small teams feel the failure modes first, because one person is the entire human queue. The gate pattern above scales down well: the point is routing the three irreversible decisions a week to yourself instead of reviewing everything.

Does adversarial drift mean scraping can never go dark?
No — it means scraping goes dark with a drift watch, not dark forever. Outcome signals (schema stability, volume bands, spot-check verification) are the lights that stay on while the station runs unattended.

References

daily.dev Popular feed — trend signal for this piece (top-10 post on AI-agent "dark software factories", 2026-10-23): https://daily.dev/posts?sortBy=popular
Botslave — PHP2Shell patch-operations playbook (dark execution, human-owned plan): https://botslave.com/blog/php2shell-patch-operations-playbook
Botslave — Git workflows for AI agent teams (reviewable, revertable rollouts): https://botslave.com/blog/git-for-ai-agent-teams
Botslave — Email validation and why SMTP probing lies (grounding gaps): https://botslave.com/blog/email-validation-mx-catch-all-smtp-probing
Botslave — Shopee security headers and bot detection (adversarial drift): https://botslave.com/blog/shopee-security-headers-tls-tokens-bot-detection

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

Shopee Security Headers: TLS, Tokens & Bot Detection

Nguyen Tu — Thu, 22 Oct 2026 07:42:14 +0000

Shopee security headers do not operate in isolation. To understand why a request succeeds, triggers verification or is rejected, separate at least four layers: the HTTP message, session state, the TLS connection and the application/device context. Matching a URL and a handful of headers does not establish that two requests are equivalent.

This research note moves from fields such as af-ac-enc-dat, af-ac-enc-sz-token and x-sap-access-* to TLS fingerprints, device integrity and bot-detection models. Its purpose is to build a testable method of analysis, rather than attribute every failure to a "missing token" or "emulator detection."

Research scope: selected Shopee Singapore Android test sessions from 15–16 October 2026, reviewed and edited on 21 October 2026. Findings apply to the samples observed, not to Shopee's internal specification. Data examples are illustrative, with substituted values; they are not usable credentials.

1. What are Shopee security headers?

HTTP headers describe aspects of a request or response. Some belong to established HTTP standards; others are defined by an application. "Security headers" is often used loosely for fields associated with authentication, session state, integrity checks or contextual information.

Distinguish a documented purpose from an inferred purpose. For standard headers, consult the relevant specification. For Shopee-specific names, a prefix such as enc, token or access, or a value resembling a timestamp, provides a starting hypothesis rather than an implementation description.

A value that changes after every action could be a request identifier, random data, a signature output, session data or a structure containing several components. Length and frequency of change help frame questions; they do not identify an algorithm by themselves.

Four useful levels of confidence

Observation: a field is present in a sample, has a particular length or changes at a particular time.
Standards-based interpretation: a public specification explains a field or structure.
Hypothesis: an explanation fits the data but has not been isolated through testing.
Finding within a defined scope: controlled comparisons and repeatable results support a conclusion.

Good research preserves the boundaries between these levels. "The value changed between two requests" is more defensible than "This is an HMAC signature that changes every second" when the latter lacks independent support.

2. Classifying headers: session state, context and custom fields

Do not put every header into a single "anti-bot" category. Different groups may serve different functions, and a request can involve several mechanisms at once.

Group	Examples to identify	A useful research question
Content information	`Content-Type`, `Accept`	What does the client send, and which response format does it expect?
Authentication	`Authorization`, where the protocol uses it	Which authentication scheme is involved?
Session state	`Cookie` and application-specific session fields	When was the session created, and which actions changed it?
Browser context	`Origin`, `Referer`, `Sec-Fetch-*`	In which context did the request originate?
Application-specific fields	`af-ac-enc-dat`, `af-ac-enc-sz-token`, `x-sap-access-*`	Where do these occur, and how do they relate to sessions and actions?
Response evidence	HTTP status, response headers and error codes in the body	Is the failure at the transport, HTTP or application layer?

As MDN's Authorization documentation explains, the header carries credentials according to an authentication scheme. There is no single token format for all systems. A request without a Bearer token is therefore not necessarily a request without authentication.

The Cookie header sends relevant cookies from the client to the server. When comparing samples, record session state and the process that established it. Comparing only the final cookie strings can lose important context.

In browsers, Sec-Fetch-Site describes the relationship between the initiating origin and the destination. This is web-context information, not a universal requirement for native Android requests. Chrome's header set should not be the sole benchmark for an Android app.

How should Shopee-specific fields be interpreted?

Field or group	What a sample can establish	What requires additional evidence
`af-ac-enc-dat`	Presence, size, character representation and changes between samples	Specific hardware fields inside it, encryption layers or key location
`af-ac-enc-sz-token`	Presence or absence, and when it appears relative to session establishment	Which component issues it, and its expiration or revocation rules
`x-sap-access-*`	Which members occur in a sample and whether they change together	The exact signing algorithm and which request components are bound to it

Versions, platforms and endpoints do not necessarily use the same fields. Documentation should always identify the version and sample source. This table is a guide to interpreting evidence, not a decoder for private headers.

3. Tokens, signatures, encryption and Base64 are different concepts

A common mistake is to see a long string, decode it as Base64 and call the result "decrypted security data." Separate four concepts:

Encoding: representing data in another form for transmission or storage.
Encryption: protecting confidentiality through a cryptographic mechanism and keys.
MACs and signatures: supporting authenticity or integrity checks, depending on the mechanism.
Tokens: a role-based label; a token may be an opaque reference, structured data or cryptographically protected data.

Base64 in RFC 4648 is a data representation. Recovering bytes from a Base64 string does not establish their meaning or remove an encryption layer inside them. Hexadecimal strings can also represent many different kinds of data.

HMAC in RFC 2104 combines a secret key with a hash function for message authentication. Knowing what HMAC is does not establish that a particular Shopee header uses it. A fixed-length string can fit several different constructions.

An illustrative example: one change, several explanations

Consider a synthetic dataset used to practice analysis:

Sample A: request_id changes; field X changes; session_label stays the same
Sample B: request_id changes; field X changes; session_label stays the same
Sample C: a new session is created; X changes; session_label changes

This supports saying that X varies within a session and after a new session is created. It does not establish that X depends only on time, expires after three minutes or is signed using a particular algorithm. Those claims require additional observations or independent confirmation.

Claims such as "the token always contains an IMEI," "the key must be inside a TEE," or "the header expires after three to five minutes" remain unsupported by sufficient evidence in this research. Recording what is unknown improves the report; it does not reduce its value.

4. TLS, certificate authorities and certificate pinning

A useful conceptual breakdown is:

Connection establishment and transport protection
                     ↓
Readable HTTP messages
                     ↓
Authentication / session / operation validity
                     ↓
Server-side application decision and response

This is an investigative model, not a diagram of Shopee's internal architecture. Success at one layer does not guarantee success at the next.

Capturing a PCAP does not mean reading HTTP

A PCAP records observed network packets. If application data travels inside TLS, collecting the bytes does not automatically provide the session secrets needed to decrypt it. With ephemeral key exchange, a PCAP is not simply a disguised JSON document that can be revealed through ordinary decoding. RFC 8446 describes TLS 1.3's key establishment and data protection.

CA trust and pinning are separate checks

Android Network Security Configuration describes how applications can select trust anchors and constrain accepted certificates or keys. Installing a CA on a device does not guarantee that every application trusts it; accepting a trust chain does not establish that a separate pinning constraint has been satisfied.

When a proxy-related failure occurs, classify it: was a connection established, did TLS complete, and did the application receive a response? An error icon alone does not identify a particular native library or the location of a certificate check.

Readable HTTP can still be rejected

The research notes include a session in which request and response data were readable while a login step still returned an error. This distinction matters: observability and permission to perform an operation are independent questions. Resolving visibility does not automatically resolve account state, session validity or server policy.

5. What do JA3 and JA4 TLS fingerprints tell us?

A TLS fingerprint summarizes aspects of how a client establishes a connection. Cloudflare's JA3/JA4 documentation explains their use in identifying and grouping clients by handshake characteristics, including JA4's normalization choices.

A fingerprint is not a unique identifier for an individual person or phone. Clients with similar configurations can share one. A client's connection characteristics can also change after software or networking changes.

The investigative implication is that "the browser and script have different fingerprints" does not establish "that difference alone caused the rejection." An appropriate control must isolate the factor. Cloudflare's documentation establishes that this technique exists in the industry; it does not establish that Shopee uses Cloudflare or a particular JA implementation in the observed case.

A careful finding would be: "The two environments produced different results and had different connection stacks; the effects of TLS, session state, network and application context have not yet been separated." This still guides further investigation without turning correlation into causation.

6. Bot detection: rules, behavior and risk scoring

An abuse-prevention system need not rely on one algorithm. Cloudflare's bot-detection engine documentation describes a combination of heuristics, client-side signals and machine learning, using information about requests, sessions and browsers.

Three conceptual models can help organize evidence:

Model	How to think about it	What the evidence needs to answer
Rules and heuristics	A condition or combination of conditions leads to an action	Which conditions differ between accepted and rejected samples?
Sequence or behavioral analysis	Context includes several actions over time	Was the sequence and session history recorded completely?
Risk scoring	Several signals contribute to an assessment	Do server logs or other authoritative evidence confirm the score and decision?

This table is a framework for analysis. It is not a reverse-engineered description of Shopee's detection algorithm. Without server-side data, we often observe only the outcome. Different mechanisms can produce the same verification screen.

Why might a "correct token" be insufficient?

In a hypothetical design, a token might establish only one property: that a session was created, a message was issued or a check occurred. Permission to perform the current operation is a separate question. Similar-looking values do not establish that a token can be transferred between contexts.

Why not infer detection thresholds from a few attempts?

Two successful attempts followed by a failure do not establish a three-request threshold. Timing, caching, session state, verification status and unobserved variables may also have changed. A report should state its sample size and observation window rather than turn a small count into a universal rule.

7. Device fingerprints and Play Integrity are not interchangeable

A device fingerprint combines signals that describe an environment. Device integrity is an assessment under a particular mechanism and set of conditions. The concepts are related but not interchangeable.

Google Play Integrity provides verdicts about application integrity, device environments and certain risky interactions. Its verdict documentation explains fields such as appIntegrity, deviceIntegrity and different device labels, whose requirements depend on the label and Android version.

A missing expected label does not automatically explain every application failure. An application may also use signals outside this service. The samples in this research do not include sufficient server-side verdict evidence to establish Play Integrity as the cause of the observed login rejection.

A phone succeeds and an emulator fails: what follows?

The defensible conclusion is that two test configurations produced different results. Hardware alone cannot be isolated if the network, app installation source, session state, operating system and account history also differ.

To strengthen a hypothesis, describe the similarities and differences, keep a group of conditions stable and record what cannot be controlled. Changing a displayed device model name does not establish equivalent integrity, networking or application behavior.

8. Shopee SG case study: what was actually observed?

The research notes from 15–16 October 2026 contain several concrete observations. Only a summary without credentials is published here; raw HAR files and tokens are not provided.

Observation in the research notes	Supported finding	What it does not establish
Some test requests and responses could be read	A portion of the traffic was observable at an interpretable application level	Every protocol and endpoint had been analyzed
An account-check step returned HTTP 418 and code `90309999`	That operation did not complete normally in that sample	The code always means root, a bad IP address or an emulator
Another response returned HTTP 200 with an error in its body	HTTP status and application outcome must be read separately	HTTP 200 means the user successfully logged in
A review of an older HAR found 19 recommended-product cards	The older sample contained some product data	A complete PDP was captured for a product opened in a different test
A newer PCAP retained connection data, but the target product JSON was not readable	Candidate network streams remained available for further analysis	The product-detail request and response had been confirmed from ciphertext alone

A particularly useful lesson is that reviewing evidence can change the scope of a conclusion. Before the older HAR was re-examined, saying "no product data has been obtained" was too broad. After finding recommendation cards, the better conclusion became: "Recommended-product card data exists in an older sample; the search or PDP flow under investigation remains unverified."

That correction is more useful than a general success claim. Recommendation, search and product-detail data may come through different flows. Finding a field in one response does not establish that other fields were obtained or that they share the same source.

Do not assign error-code meanings by guesswork

HTTP 403 indicates that a server refuses to fulfill a request. The status alone does not identify a specific cause. Platform-specific application codes require documentation or suitable controlled evidence before they can be mapped to causes.

In this case, the evidence does not establish a direct mapping between the UI label F13 and 90309999, or make either one conclusive proof of an IP or integrity problem. This remains an open question, not an identified algorithm.

9. A test matrix for isolating causes

Write the question before changing configurations. For example: "Does enabling the observation layer change the outcome of the same operation in the current configuration?" A small question makes the evidence requirements clearer.

Observation axis	Minimum useful record	A design mistake to avoid
Application	Version, installation source, original or modified build	Changing the app build and network simultaneously
Device	OS, environment type and relevant configuration	Recording only a model name and omitting other changes
Network	Connection type, timing and connection changes	Attributing differences to IP when the session also changed
Session	New or continued session; verification state	Comparing a fresh session with a long-established one
Observation	Capture enabled or disabled; which layers are readable	Assuming every connection has the same decryption status
Outcome	UI result, HTTP status, application code and timestamp	Retaining only a screenshot or only an HTTP status

A practical investigative sequence is:

Establish a baseline: record the unchanged configuration's result for the specific operation.
State the hypothesis: describe what should change if it is correct.
Choose the variable: change one factor where possible, documenting exceptions.
Align the timeline: associate the UI action with the correct timestamp and request/response.
Compare layers: separate transport, HTTP and application differences.
Repeat within bounds: record both failures and successes rather than selecting favorable samples.
Define the finding: state what is supported, what has been ruled out and what evidence is missing.

These steps apply to environments and operations within an authorized testing scope. A useful report does not require high-volume traffic; the quality of controls often matters more than the number of requests.

An example test record

Test ID: LAB-EXAMPLE-01
Question: distinguish observation failures from application failures
Time: record timestamp and time zone
Application / OS: record actual versions
Session: new or continued; do not record raw tokens
Single change: describe explicitly
UI outcome: success / verification / error
HTTP and application code: record separately, if readable
Finding: limited to this configuration
Uncontrolled variables: list them rather than filling gaps with assumptions

This is an illustrative template, not a log from a real account or device.

10. Common mistakes in security-header analysis

Treating web and mobile clients as one protocol

Clients may follow different paths. The absence of a browser header in an app is not proof that the app is missing a required field. An app's behavior also does not establish that all web APIs behave identically.

Guessing internal structure from ciphertext

A long string does not reveal a schema. Start with representation, size and changes. If the serialization or encryption layer is unverified, label it "unknown" instead of assigning an algorithm name.

Mixing evidence from different sessions

A UI screenshot from session A, a HAR from session B and a product screenshot from session C can form a convincing but incorrect story. Each finding must reference the correct sample, timestamp and action.

Treating proxy visibility as the application's full view

Data may not pass through the observed component, may remain encrypted or may use a structure the tool cannot parse. A limitation of the observation tool is not a conclusion about the entire application.

Treating one success as a durable workflow

Describe a successful result under its actual conditions. A single sample does not establish long-term reliability, applicability to another account or suitability for a different purpose.

11. Frequently asked questions

What exactly does `af-ac-enc-dat` contain?

This research does not have a verified schema to publish. Characteristics can be catalogued from authorized samples, but the field name alone does not establish that it contains a specific hardware identifier.

Does copying cookies guarantee that a request will be accepted?

The samples in this article provide no such guarantee. Cookies form part of the session context; analysis should also consider how the session was established and which other conditions changed, rather than treating cookies as the client's entire state.

Why can HTTPS be readable while login still fails?

Readable data establishes visibility only for the observed flow. A server can still reject an operation because of session state, verification, permissions or application policy. Identifying the specific reason requires additional evidence.

Does Shopee definitely use JA4 or Play Integrity?

This article does not establish either claim for the tested sessions. Those technologies explain industry mechanisms and help formulate hypotheses; their public documentation is not evidence of Shopee's internal configuration.

Does error `90309999` mean the device has been blocked?

That conclusion cannot be drawn from the status or application code alone. The research notes do not provide an official mapping or controls sufficient to establish a device block, its duration or a single cause.

Can a complete PCAP immediately reveal JSON?

Not necessarily. Distinguish complete capture at the observation level from possession of session secrets, successful decryption and understanding the payload format. These are separate conditions.

Has this research reconstructed Shopee's anti-bot algorithm?

No. It provides a classification of security layers, bounded observations and a method for testing hypotheses. Knowing a few headers or error codes does not reconstruct the server's decision-making model.

12. Conclusions and references

Shopee security headers should be studied as part of a stateful system, not as a list of strings to copy. The lasting value is in locating the failure layer, maintaining evidence provenance and controlling variables in each comparison.

This approach also reflects Botslave's perspective on building systems: establish clarity before adding complexity, retain human ownership of important decisions and progress through steps that can be verified. Explore other notes in the Botslave Journal, or start a conversation about a research question.

Primary technical references

MDN — Authorization: authentication schemes in HTTP.
MDN — Cookie: cookies in requests.
MDN — Sec-Fetch-Site: origin context for web requests.
RFC 4648: Base16, Base32 and Base64.
RFC 2104: the HMAC construction.
RFC 8446: TLS 1.3's design.
Android — Network Security Configuration: trust anchors and certificate pinning.
Cloudflare — JA3/JA4: fingerprints at the TLS layer.
Cloudflare — Bot detection engines: a public example of layered bot detection.
Google — Play Integrity overview and Integrity verdicts: application and device assessments.
MDN — 403 Forbidden: HTTP semantics, not a Shopee application-error catalogue.

Methodology and publication limits

Shopee-specific observations were checked against research notes from 15–16 October 2026. Published summaries exclude account information, cookies, tokens and device identifiers. Public references explain foundational mechanisms; they do not confirm Shopee's private algorithms. Server-side decision logs were not available to establish signal weights or detection thresholds.

Revision history — 21 October 2026: expanded the header classification, token and cryptography distinctions, detection layers, empirical evidence table, controlled-test matrix and FAQs. This English edition preserves the Vietnamese article's scope and uncertainty. Future evidence that changes a finding will be recorded as a revision rather than silently upgrading an earlier hypothesis into fact.

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

PHP2Shell and the Patch-Operations Playbook: Rolling an Emergency Fix Across a Real Fleet

Nguyen Tu — Thu, 22 Oct 2026 07:35:33 +0000

The most-read story on daily.dev today is not a new framework or a benchmark — it is an emergency: on October 21, 2026, the PHP project shipped fixes for two critical remote code execution flaws, CVE-2026-109618 and CVE-2026-109619, nicknamed PHP2Shell. Both score 9.2 on CVSS v4.0, and an unauthenticated attacker can run code on a vulnerable server. The news cycle covered the bug itself. This article is the practitioner's version — the problem we actually solve for clients: the fix exists, but does it run everywhere it needs to? The distance between "released" and "deployed" is where incidents happen, and closing that distance is pure automation work. It is the same philosophy as how we work: automate the routine aggressively so human judgment is spent exactly where it changes the outcome.

1. Released is not deployed: the uncomfortable math

The patched releases are PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 (released October 21, 2026, at 14:00 UTC — pulled forward a day after the CVE IDs leaked). The PHP team also published minimal source backports for 7.4, 8.0 and 8.1, which receive no official release.

Now the uncomfortable part: Bitsight's internet-wide scans found that only about a third of the PHP instances where the version could be identified run a branch covered by the security update. The rest sit on older branches, and the most common branch by far is PHP 7.4, with more than two million observations. So for a large share of the real world, "the fix is out" and "we are fixed" are two different sentences. Your job on a day like this is to turn the second one into truth, systematically, before someone else turns it into a breach.

The two flaws themselves sit in the engine's memory management and error handling — code nearly every request touches. CVE-2026-109618 is OOM corruption: after a memory allocation fails, PHP leaves invalid data behind for its cleanup routine to process. CVE-2026-109619 is stale global state: request-owned state survives after its memory is freed, and a later request on the same worker can read or write through the dangling pointers. Every setup serving web requests is exposed — PHP-FPM, mod_php, php-cgi, FrankenPHP, LiteSpeed — with mod_php rated highest risk because the exploit fires before any application code runs.

2. Inventory: find every PHP runtime you own

You cannot patch what you do not know runs PHP. On emergency-patch day, the inventory step is the one most teams skip, and it is the one that decides whether the patch actually covers the fleet. A real inventory pulls from several sources of truth, because PHP hides in all of them:

Container images: Dockerfile and compose files that pin a PHP image tag. This is the fastest place to scan first, because it is text.
Hosts: packages installed via apt/dpkg or rpm/yum, plus anything compiled from source.
Running processes: what the package manager says and what actually serves traffic can disagree.
Serverless and PaaS: runtimes managed by someone else — some vendors auto-patched (Google Cloud patched supported serverless runtimes; Laravel Cloud auto-patches), but custom container images and App Engine flexible deployments need a manual redeploy.
Hosts you do not control: shared hosting, where the fix depends on the provider. Updating WordPress, plugins or themes does not change your PHP version, because PHP lives on the host — check php -v on every server, and ask your host when it will patch if you do not control it.

A first-pass scan can be embarrassingly simple and still catch most of the fleet:

# 1. Every container image pinning a PHP base image
grep -rn "FROM php:" --include='Dockerfile*' --include='docker-compose*' .

# 2. Every host with PHP installed (Debian/Ubuntu fleet via your config management)
dpkg -l | grep -E '^ii\s+php[0-9]' | awk '{print $2, $3}'

# 3. What is actually running right now (catches source builds and stray SAPIs)
ps aux | grep -E 'php-fpm|apache2|httpd|litespeed' | grep -v grep

# 4. What version answers the web (spot-check the externally visible truth)
for host in web1 web2 web3; do
  ssh "$host" 'php -v | head -1'
done

For codebases at scale, this is exactly what Sourcegraph published search queries and batch-change specs for during this event: finding Dockerfiles, CI pipelines and Helm charts that pin old PHP versions, then proposing the bump as a batch change. We have written about this machinery before — our guide to Git workflows for agent teams covers batch changes and reviewable, revertable rollouts, which is the same shape of work: many small, verified, reviewable changes instead of one heroic push.

3. The restart problem: upgrading the package is not the fix

This is the detail that eats patches in production. On Ubuntu, fixes exist for all supported releases — but running apt upgrade does not finish the job, because unattended upgrades do not restart running applications. The old, vulnerable PHP-FPM workers keep serving traffic until you restart them. The same is true for Apache with mod_php. A fleet that is "fully patched" in the package database and still vulnerable in memory is the classic failure mode of emergency patching.

The restart semantics differ by platform, and on patch day you need the real list, not assumptions:

Platform / SAPI	What actually applies the fix
Ubuntu/Debian PHP-FPM	`apt upgrade` + restart `php*-fpm` and the web server; unattended-upgrades will not restart running apps
mod_php (Apache)	Restart Apache; highest risk SAPI, since the exploit fires before application code runs
Laravel Cloud	Auto-patches
Laravel Vapor (PHP 8)	Apps redeployed automatically
Laravel Vapor (PHP 7)	No patch at all — the app must be upgraded off the branch
Laravel Forge	Manual update under Runtime → Versions
Google Cloud serverless	Supported runtimes auto-patched; custom images need a manual redeploy
Cloudways	WordPress instances auto-patched
DigitalOcean Droplets / DOKS / App Platform	Manual action required

So the rollout script has two halves — the version bump and the restart — and the second half is where verification must happen:

#!/bin/bash
# bump-and-restart.sh — the boring half that actually matters
set -euo pipefail
sudo apt update && sudo apt upgrade -y php8.4-fpm
# Restart the runtime AND the web server; the old workers stay in memory otherwise
sudo systemctl restart php8.4-fpm
sudo systemctl restart apache2 || sudo systemctl restart nginx
# Verify the fix is the thing actually running, not just installed
php -v | head -1

The principle generalizes beyond PHP: a patch is only real when the vulnerable code is no longer in memory. Anything in your rollout that checks the package database but not the running process is checking the wrong thing.

4. Rollout mechanics: canary, stages, re-runnable automation

Emergency does not mean reckless. A staged rollout with a canary still beats a fleet-wide push on a CVE day, because the patch itself is a change to every request path in your application, and engine-level patches have historically broken extensions and edge-case behavior. The mechanics that work:

Patch one canary host or one non-critical app first, watch error rates and latency for a full traffic cycle, then proceed.
Make the rollout re-runnable (idempotent). The same script run twice must converge to the patched state, not fail or double-apply. On a day when you are patching at 2 a.m. with tired operators, re-runnable beats clever.
Keep it reversible. Engine patches occasionally break third-party extensions; know how to roll back to the previous package version before you start, not after.
Batch the config changes. Dockerfile pins, CI pipeline images and Helm chart tags should be bumped as reviewable changes with CI green — not hand-edited on servers. This is where the batch-change approach from our agent Git workflows post earns its keep: dozens of small, verified, revertable changes instead of snowflake servers.

5. The EOL fleet: what to do with 7.4, 8.0 and 8.1

A large share of the internet runs PHP branches that get no official release — 7.4 alone had more than two million observations in Bitsight's scans, and W3Techs data puts about 48.5% of PHP sites on end-of-life branches. The PHP project published minimal source patches for 7.4, 8.0 and 8.1 to help with backports, but that is not a supported release; your vendor's security advisory is the authority on whether a patched package exists for your platform.

Be honest about the decision tree:

Vendor package exists: take it, then restart (section 3 still applies).
No package, and you can upgrade the branch: schedule it — this event is the forcing function.
No package and no upgrade path right now: this is where CERT-EU, the NSA and CISA landed — isolate those systems or take them offline. Several governments did exactly that: Dutch government services, Finland's state IT provider, German and French federal IT all pulled sites down rather than run exposed.

"Awaiting the vendor" is not a plan. A written decision with a date — patched, isolated, or offline — is.

6. Defense in depth while the rollout runs

Patching a fleet takes time: inventory, maintenance windows, application testing against the new builds. During that window, boundary controls reduce exposure to the confirmed attack paths. Cloudflare turned on default WAF rules for its customers about nine hours before disclosure, after attackers began probing, and AWS published a guide with AWS WAF rules for the confirmed PHP2Shell attack paths. A reverse proxy or WAF in front of PHP can block hostile requests before they reach the vulnerable parser.

Two caveats, both worth stating plainly:

A WAF does not fix PHP. It covers known attack patterns; the vulnerable code is still underneath. Patching is the only complete fix, and it stays priority one.
Expect payload mutation. Cloudflare reported attackers already adapting payloads to evade its rules. Static rules buy time; they do not end the race.

Detection matters as much as blocking. Datadog's analysis suggests looking for large form submissions, memory-exhaustion errors during request initialization, and unexpected shell processes spawned by the web server user — and notes that any PHP script exposes the vulnerable path, even one that only prints a response, because the request never reaches application code. Practical log checks:

# Requests hitting the session upload-progress path (the reported trigger surface)
grep -E 'PHP_SESSION_UPLOAD_PROGRESS' /var/log/apache2/access.log | tail -20

# The literal calling card some scanners used
grep -i 'php2shell' /var/log/nginx/access.log | awk '{print $1}' | sort -u

# Shell processes that should never be children of the web server user
ps -ef | awk '$1=="www-data" && /sh|bash|wget|curl/ {print}'

Arctic Wolf Labs reported scanning and attempts pulling ELF and Golang malware binaries via wget — so unexpected outbound fetch-and-execute behavior from web-server-owned processes is exactly the signal to alert on. (On the bot-detection side of boundary defense, we have a full breakdown of TLS fingerprints and token-based bot detection — the same layer where these WAF rules live.)

7. Verify the patch actually landed

The rollout is not done when the script exits green. It is done when every runtime in the inventory answers with a fixed version. Close the loop:

#!/bin/bash
# verify-php-fleet.sh — the inventory from section 2, re-run as an assertion
FIXED=("8.5.12" "8.4.27" "8.3.36" "8.2.35")
for host in $(cat fleet-hosts.txt); do
  ver=$(ssh "$host" 'php -r "echo PHP_VERSION;"')
  ok="no"
  for f in "${FIXED[@]}"; do [[ "$ver" == "$f" ]] && ok="yes"; done
  # Also confirm the running workers are the new build, not stale memory
  echo "$host -> php $ver | patched: $ok"
done | tee patch-verification-$(date +%F).log

Keep that log. When the next CVE drops — and Taylor Otwell's warning during this event was that this probably will not be the last time, nor PHP the only language to go through it — the difference between a two-hour response and a two-day scramble is whether your inventory already exists. The fleet list, the restart runbook and the verification script are assets. Build them once, on a calm day, and CVE day becomes execution instead of archaeology.

8. Frequently asked questions

Does updating WordPress, my plugins or my theme fix PHP2Shell?
No. The vulnerability is in the PHP runtime itself, which lives on the host — below your application. Update the runtime (and restart it), regardless of what your CMS or framework reports.

Does putting a WAF in front fix it?
No. WAF rules from Cloudflare, AWS and others reduce exposure to confirmed attack paths while you patch, but they do not remove the vulnerability. Patching is the only complete fix.

I run PHP 7.4. Am I covered?
Not by an official release. The PHP project published minimal source patches for 7.4, 8.0 and 8.1 to aid backports; check your OS vendor's security advisory for patched packages. If neither exists, the official guidance from CERT-EU, NSA and CISA is to isolate the system or take it offline — or upgrade the branch.

How do I know if someone already exploited it against me?
Look for the signals in section 6: large form submissions, memory-exhaustion errors during request initialization, unexpected shell processes or wget/curl activity owned by the web-server user, and php2shell strings in access logs. Absence of these signals is reassuring, not proof.

Why is this suddenly everywhere?
Both flaws were found with frontier AI assistance — Calif's team with OpenAI's models, and Wiz independently with its Atlas tool — and one researcher demonstrated that a working proof of concept was roughly "one prompt away" once hints leaked. The finding side of security is accelerating; the patching side has to keep up, which is exactly what the playbook above is for.

References

Tenable Research — PHP2Shell FAQ (disclosure timeline, CVE details, affected SAPIs): https://www.tenable.com/blog/php2shell-cve-2026-109618-cve-2026-109619-frequently-asked-questions-about-critical-php-remote
AWS Security Blog — using AWS WAF to mitigate PHP2Shell while patching: https://aws.amazon.com/blogs/security/using-aws-waf-to-apply-mitigations-for-php2shell-rce-vulnerabilities-cve-2026-109618-and-cve-2026-109619/
Datadog Security Labs — detection, analysis and remediation: https://securitylabs.datadoghq.com/articles/php2shell-critical-vulnerabilities-in-php/
Bitsight — vulnerability details and internet-wide version exposure data: https://www.bitsight.com/blog/critical-vulnerability-alert-cve-2026-109618-and-cve-2026-109619-php2shell
Brandefense — attack timeline and official alerts roundup: https://brandefense.io/blog/php2shell-php-rce-cve-2026-109618-109619/
SOCRadar — FAQ separating confirmed facts from leaked/unverified claims: https://socradar.io/blog/php2shell-cve-2026-109618-109619/

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

How to Validate Email Addresses Without Sending Mail: MX Records, Catch-Alls, SMTP Probing

Nguyen Tu — Thu, 22 Oct 2026 07:33:41 +0000

Every lead pipeline eventually asks the same question: "is this email address real?" The honest answer is that "real" is four different questions wearing one coat — and each layer of validation only answers one of them. Confusing the layers is how teams end up trusting a green checkmark that proves almost nothing, or throwing away good leads because one weak signal said no.

This is the validation model we use in our own lead operations work: what each check actually proves, where SMTP probing helps, where it quietly lies to you, and how to design a pipeline that records evidence instead of pretending to know the truth. The thinking behind this staged approach — automate the routine, keep judgment where it matters — is the same one behind how we work.

1. Four questions, not one

When someone says an address is "valid", they might mean:

Is it well-formed? Does it parse as an email address at all?
Can the domain receive mail? Is there a mail server willing to accept messages for this domain?
Does this mailbox exist? Is there an actual account behind the local part?
Is it safe and worthwhile to send to? Will sending hurt sender reputation, or reach a human who matters?

Syntax checks answer question 1. MX lookups answer question 2. Question 3 can only be approximated without sending mail — and question 4 is a policy decision, not a technical one. Most "email verifier" scores collapse all four into a single number. That is convenient and wrong: a catch-all corporate domain and a personal Gmail account can both score "valid" while meaning completely different things for your pipeline.

2. Layer one: syntax, and why it is not trivial

The grammar of an email address is defined by RFC 5322, and it is more permissive than almost every hand-rolled regex assumes. Quoted local parts ("john..doe"@example.com) are legal. So are comments in obsolete syntax that no sane system should accept but some parsers do. In practice:

Use a real parser from your platform's ecosystem, not a regex copied from a forum. Then apply a stricter business rule on top: most pipelines reject quoted strings, consecutive dots and addresses longer than 254 characters, not because the RFC forbids all of them, but because real-world mail systems handle them inconsistently.
Normalize before you validate. Trim whitespace, lowercase the domain (domains are case-insensitive; local parts technically are not, though nearly every provider treats them case-insensitively), and convert internationalized domains to their ASCII form with IDNA/punycode — you must validate and resolve the form DNS actually uses.
Do not "fix" addresses silently. Plus-addressing (name+tag@gmail.com) and Gmail's dot-insensitivity are provider semantics: the same mailbox, but stripping tags or dots to deduplicate is a dedupe decision, not a syntax correction. Record the original; derive a canonical form separately if your pipeline needs one.

Syntax validation is cheap and deterministic. It is also the layer that proves the least. Everything interesting happens at the domain.

3. Layer two: MX records — the domain's front door

To find out where mail for a domain goes, you ask DNS for its MX (mail exchanger) records. Each record has a preference value (lower wins) and a host name. Two lookups cover most of the world:

// Node.js — no dependencies, built-in dns module
const dns = require('node:dns/promises');

async function mxFor(domain) {
  try {
    const records = await dns.resolveMx(domain);
    return records.sort((a, b) => a.priority - b.priority);
  } catch (e) {
    if (e.code === 'ENODATA' || e.code === 'ENOTFOUND') return [];
    throw e;
  }
}

# Python — with dnspython (pip install dnspython)
import dns.resolver

def mx_for(domain: str):
    try:
        answers = dns.resolver.resolve(domain, "MX")
        return sorted((r.preference, str(r.exchange)) for r in answers)
    except (dns.resolver.NoAnswer, dns.resolver.NXDOMAIN):
        return []

Three outcomes matter:

MX records exist → the domain publishes where its mail goes. You have also learned something else for free: the MX hosts usually reveal the provider (Google Workspace, Microsoft 365, a regional hoster), which tells you which behavioral rules to expect later.
A single MX record pointing at "." (null MX, RFC 7505) → the domain explicitly declares it accepts no mail. This is a hard, standards-based "invalid" — one of the few you will ever get. Treat any address at such a domain as undeliverable, permanently.
No MX records at all → strictly, RFC 5321 §5.1 defines an "implicit MX": fall back to the domain's A/AAAA records. In practice, domains with A records but no MX are a grey zone — personal sites, parked domains, misconfigurations. Most lead pipelines mark them low-confidence rather than valid, and they are right to: a domain that wanted mail would publish MX.

What MX tells you is strictly question 2: a front door exists. It says nothing about whether your specific mailbox is behind it.

4. Layer three: SMTP probing — useful, and a compulsive liar

The classic trick for question 3 is to start an SMTP conversation with the receiving server and stop just before sending anything:

S: 220 mx.example.net ESMTP ready
C: EHLO validator.example.com
S: 250 Hello
C: MAIL FROM:<check@example.com>
S: 250 OK
C: RCPT TO:<person@example.com>
S: 250 OK            <- mailbox accepted?
C: RSET / QUIT

(Illustrative session, not a captured transcript.)

A 550 here often means "no such user" and a 250 is taken as "mailbox exists". Both readings are unreliable, for structural reasons:

Accept-all edges. Many servers — especially corporate gateways and Microsoft/Google tenants fronted by security appliances — accept RCPT TO for any local part and decide what to do with the message later. Your probe gets a 250 for mailboxes that do not exist, and the bounce only happens after a real send, if it happens at all (some systems silently discard).
Anti-enumeration defenses. Large providers deliberately stopped answering honestly years ago, precisely because probing was used to harvest valid addresses. A 250 from them is close to meaningless; a 550 may be policy, not absence.
Greylisting and tempfails. A 450/451 can mean "come back later" (greylisting), rate limiting of your probing IP, or a tarpit deliberately slowing you down. It is not a verdict in either direction.
Your reputation, not theirs. Probing is done from your IP. Datacenter IPs are frequently refused outright, and sustained probing patterns get IPs listed. The probe measures your standing with their filters as much as the mailbox.

The professional conclusion: an SMTP probe is a weak signal, strongest when it returns an explicit 550 user unknown from a server you have already established is not catch-all, and weakest everywhere else. Which raises the real question — how do you know a domain is catch-all?

5. Catch-all domains: where mailbox validation ends

A catch-all (accept-all) domain accepts mail for every local part, real or invented. The standard detection is simple: probe the server with a local part that cannot exist — a long random string:

RCPT TO:<xq7zv2k9plm4rf8d@example.com>
250 OK   -> this domain accepts anything: it is catch-all (at this edge, today)

If the random address is accepted, mailbox-level validation for that domain is over. Every address there is "unverifiable" by construction — not valid, not invalid. In B2B lead data this is not an edge case: small and mid-size companies land on catch-all configurations all the time, through default hoster settings, migration leftovers, or a deliberate "never miss a message" choice by their admin.

How a pipeline should treat catch-all addresses:

Never merge them into "valid". Keep a separate bucket with its own sending policy (for outreach: send later, send fewer, watch the first bounces closely).
Never auto-delete them either. A catch-all domain contains real people; the CEO's address is in there next to a thousand invented strings. Deleting the bucket deletes your best leads along with the noise.
Cache the verdict per domain, not per address, and re-check periodically — catch-all status is a server configuration, and configurations change.

This is the same discipline we apply to any layered signal system: in our research into Shopee's security headers, the recurring lesson was that one signal rarely means what it appears to mean until you know which layer produced it. Email is no different.

6. Supporting signals that are actually worth collecting

Around the three main layers, a few cheap checks add real value:

Disposable/temporary domains. Maintain or license a list of throwaway-mail domains. The address may be perfectly valid and worthless in ten minutes — that is a question-4 decision, and for lead pipelines the answer is usually reject.
Role accounts. postmaster@, admin@, sales@, contato@ are deliverable but shared inboxes. Tag them; do not drop them — in small companies the role inbox is the decision-maker's inbox.
Provider fingerprint from MX. Knowing a domain sits on Google Workspace vs. a small regional hoster changes how you interpret probes, bounces and rate limits downstream.
Domain typos. gmial.com, hotmial.com and friends are data-entry errors, not mailboxes. Suggest the correction to a human or a form; never silently rewrite an address and send to a stranger's mailbox.
Free vs. corporate domain. Not a validity signal at all — but valuable routing metadata for enrichment and segmentation.

7. Designing the pipeline: verdicts with evidence, not scores

A robust validation stage stores, per address: the syntax result, the MX set and provider, the null-MX flag, the catch-all verdict (and when it was measured), the probe outcome if you probe at all, and timestamps for everything. The output is a label with a reason, for example:

| Evidence | Label | Pipeline action |
|---|---|
| Bad syntax / NXDOMAIN | invalid | Drop, log reason |
| Null MX (RFC 7505) | invalid | Drop, do not retry |
| No MX, A record only | low-confidence | Hold for review or enrich later |
| Catch-all domain | unverifiable | Separate bucket, cautious policy |
| MX ok, probe 550 user-unknown, not catch-all | likely-invalid | Drop, keep evidence |
| MX ok, everything else neutral | deliverable | Proceed |

Two design rules make this work over time. First, never overwrite the raw input — store the pipeline's conclusions next to the original address, so a better future check can re-derive them. Second, revalidate on a schedule: MX records move, domains lapse, catch-all gets switched off. A validation result is a measurement with a date, not a property of the address. (It is the same reason our automations are built to be safely re-runnable — idempotent by design, so re-checking never corrupts state.)

8. Frequently asked questions

Does a 250 response to RCPT TO guarantee the mailbox exists?
No. On catch-all and accept-all systems it only proves the server accepts the address at the edge. Delivery — or a later bounce — is decided afterwards.

Is SMTP probing illegal or abusive?
Probing is a normal protocol conversation, but at scale it looks exactly like address harvesting, which is what the defenses are built against. Expect blocks, respect them, and prefer MX-level checks plus evidence-based scoring for bulk hygiene. Sending one real, wanted email remains the only ground truth.

Can I skip probing entirely?
For list hygiene, mostly yes: syntax + MX + null MX + catch-all detection removes the addresses that damage sender reputation (hard-invalid domains), which is where most of the value is. Probing adds a marginal signal at a real operational cost.

How often should a lead list be revalidated?
Before any major send, and periodically for standing lists — domains and mail configurations change constantly. A verdict older than a few months should be treated as stale until re-measured.

References

RFC 5321 §5.1 — address resolution and the implicit MX rule: https://www.rfc-editor.org/info/rfc5321/
RFC 7505 — Null MX, "a domain that accepts no mail": https://www.rfc-editor.org/info/rfc7505/
RFC 5322 — Internet Message Format (address grammar): https://www.rfc-editor.org/info/rfc5322/
Node.js DNS module documentation: https://nodejs.org/api/dns.html
dnspython documentation: https://dnspython.readthedocs.io/

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

Git for AI Agent Teams: Branching, Review and Provenance When Most Commits Aren't Human

Nguyen Tu — Thu, 22 Oct 2026 07:26:14 +0000

One of the most-discussed posts on daily.dev this week asks a question that sounds philosophical and turns out to be purely operational: Git was built for humans — what happens when your dev team is mostly agents? AI coding agents now open pull requests, fix failing CI, refactor modules and update dependencies. They produce changes faster than any human team can read them. And they do it inside a version-control system whose every assumption — authorship, commit messages, code review, blame — was designed around people.

We run multi-agent workflows in our own automation practice, and the lesson arrived quickly: the first thing that breaks is not Git itself. It is the workflow around Git. This article is the practical version of the trend: what actually breaks when agents join the team, and the concrete Git practices — isolation, commit discipline, review tiers, provenance — that keep a human–agent codebase healthy. It is the same philosophy as how we work: automate the routine aggressively, and spend human judgment exactly where it changes the outcome.

1. The assumptions Git makes about you

Git was designed in 2005 for a very specific social structure: a network of trusted human maintainers exchanging patches. Its workflow layer (the part your team adds on top: branches, PRs, reviews) quietly assumes:

Authors are identifiable and accountable. A commit has a person behind it who can answer "why?"
Commit messages carry intent. Humans write messages for other humans, however imperfectly.
Review bandwidth scales with authors. If ten people write code, roughly ten people can review it.
History is written at human speed. A busy team produces dozens of commits a day, not hundreds.

Agents violate all four. An agent's "intent" lives in a task specification and a conversation, not in its head. Its output scales independently of your review capacity. And left alone, two agents will happily edit the same files in the same working tree, producing a merge situation no human would create. The fix is not a new tool — agents work fine with plain Git. The fix is tightening the workflow at exactly the points where the assumptions crack.

2. What breaks first (and it is not merge conflicts)

Teams new to agent workflows expect conflict chaos. In practice, the failures arrive in a different order:

Review throughput. Agents generate plausible-looking diffs at machine speed. Human review — the actual quality gate — becomes the bottleneck within the first week. Everything else in this article is downstream of this fact.
Diff size. An agent asked to "fix the bug" may refactor five files on the way. Large diffs get rubber-stamped, which is worse than no review because it creates false confidence.
Commit messages without a "why". Agents summarize what changed fluently. The reasoning lives in the task prompt, which is not in your repository — six months later, git log cannot tell you why a change exists.
Shared working trees. Two agents (or an agent and a human) in one checkout will stage, stash and commit over each other. This is the one genuinely Git-level failure, and it is entirely preventable.
Attribution mush. If agents commit under a human's identity, blame, audit and access control all degrade at once — you can no longer answer "who or what touched the payment code?"

3. Isolation: one worktree per agent, never a shared checkout

The single highest-leverage practice: every agent works in its own Git worktree on its own branch. A worktree is a separate working directory backed by the same repository — cheap to create, fully isolated files, shared object store:

# Give each task its own workspace + branch
git worktree add ../wt-fix-checkout-timeout -b agent/fix-checkout-timeout
git worktree add ../wt-bump-dependencies   -b agent/bump-dependencies

# List and clean up when tasks finish
git worktree list
git worktree remove ../wt-fix-checkout-timeout

Rules that make this work:

Branch per task, named for the task: agent/<task-slug>. The branch name is the first piece of provenance.
No shared checkouts, ever. Not between two agents, not between agent and human. Most "agent corrupted my work" stories are shared-working-tree stories.
Disposable by default. If an agent's branch goes sideways, delete the worktree and restart the task from a clean base. Restarting is cheap; untangling is not.
Partition tasks to avoid collisions. Two agents refactoring the same module in parallel will conflict no matter how good your process is. Split work by module or by layer (schema / API / UI), the same way you would for humans who cannot talk to each other.

4. Commit discipline for machines

Agents should commit more strictly than humans, not less, because their history will be read during incident response:

Small, atomic commits. One logical change per commit, enforced by task design ("implement, test, commit" as separate steps). Atomic commits are what make git bisect and clean reverts possible later.
Structured messages with machine-readable trailers. The subject describes the change; the body links the task; trailers carry provenance:

Fix checkout timeout by retrying token refresh once

Retry logic added to the payment session client; falls back to
a fresh token on 401 instead of failing the checkout.

Task: checkout-timeout-2026-10-21
Agent: builder-2
Verified: unit tests 42/42, staging smoke check

Git trailers (git commit --trailer "Agent: builder-2") are parseable with git interpret-trailers, so provenance stays queryable instead of becoming folklore.

A distinct identity per agent (or per agent role). Agents commit as a bot identity, never as a human team member. On GitHub this means a machine user or GitHub App with scoped permissions — least privilege per repository, no admin, no force-push rights.
No force-push to shared branches. Agents rebase their own task branches freely; shared branches (main, release) are append-only and protected.

5. Review is the bottleneck: tier it, gate it, shrink it

Since review capacity is the constraint, design for it explicitly:

Shift verification left, into machines. Tests, type checks, lint, security scans and build must pass before a human ever sees the diff. Treat the test suite as the contract the agent works against — the same evidence-over-opinion discipline we apply in data pipelines, where a validation stage records what was actually checked rather than a vibes-based score (see our breakdown of email validation evidence layers).
Cap diff size. A PR above an agreed size (say, a few hundred changed lines) gets split or escalated — never silently merged. Size caps are crude and extremely effective.
Tier by risk, not by author. Documentation, tests and dependency bumps with a green pipeline can auto-merge. Anything touching auth, payments, data migrations or infrastructure gets mandatory human review — enforced mechanically with CODEOWNERS on sensitive paths, not by convention.
Review the task spec, not just the diff. For agent work, the highest-value human read is often the instruction and the verification evidence ("what did you run, what passed?") rather than every generated line.

6. Provenance and the revert-first culture

When something breaks at 2 a.m., agent-era Git hygiene pays off in three ways:

Blame still works — if commits are atomic and honestly authored, git blame and git log -L point at the exact change and, via trailers, at the task that produced it.
Bisect still works — git bisect run ./test.sh can binary-search a regression across hundreds of agent commits only if each commit builds and is one logical change. This is the concrete return on commit discipline.
Reverts are cheap and shame-free. With agents, the fastest fix is often git revert <sha> and re-dispatch the task with a better spec, instead of hand-patching generated code. Design for this: feature flags for agent-shipped behavior, reversible migrations, and no manual edits on top of agent branches that a revert would erase.

Keep the task specification itself with the code: a short spec file committed alongside the change (or linked from the commit body) means the "why" survives even when the chat that produced it is long gone.

7. A pragmatic workflow, end to end

Putting it together, a sane human–agent Git workflow looks like this:

Human writes a task spec with explicit acceptance checks.
System creates a worktree + agent/<task> branch from a clean main.
Agent works in isolation: small commits, structured messages, bot identity.
Agent runs the full verification suite; evidence is attached to the PR.
Automation enforces the gates: green CI, diff-size cap, CODEOWNERS for sensitive paths.
Human reviews at the risk tier the change earned — skim the auto-verified, read the critical.
Merge to a protected main; branch and worktree are deleted.
Anything that misbehaves in production is reverted first and re-dispatched with a better spec.

None of this requires new tooling. It requires admitting that the scarce resource moved: code production is now abundant, and verified, reviewable, reversible change is the thing your workflow must manufacture.

8. Frequently asked questions

Do agents need their own GitHub accounts?
Effectively, yes — a bot identity (machine user or GitHub App) per team or per role. It keeps permissions scoped, audit logs honest, and human blame meaningful.

Should agent commits be signed?
Signing proves which key produced a commit, which is genuinely useful once multiple agents and automations write to the same repo. Bot identities with managed keys (e.g., GitHub App–generated commits are marked verified) give you most of the value with little ceremony.

How do you review fifty agent PRs a day?
You don't — that is the point of tiering. Most agent PRs should be small, mechanically verified and auto-mergeable; human attention concentrates on the risky minority. If everything needs deep human review, the agents are producing the wrong size of change.

What about two agents editing the same file?
Prevent it structurally: separate worktrees, task partitioning by module, and short-lived branches. Conflict resolution heroics are a symptom of missing isolation, not a workflow.

References

Git worktree documentation: https://git-scm.com/docs/git-worktree
Git trailers (git interpret-trailers): https://git-scm.com/docs/git-interpret-trailers
Pro Git, "Git Branching — Branches in a Nutshell": https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell
GitHub Docs — protected branches: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches
GitHub Docs — CODEOWNERS: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.

DEV Community: Nguyen Tu

LeadOps Autopilot: Sourcing and Scoring US Clinic Leads from Public Data

The problem with clinic lead data

Stage 1: Collect from sources that are allowed to be boring

Stage 2: Read the homepage, not the listing

Stage 3: The gate — where the pipeline stops on purpose

What this architecture buys you

Where it goes next

Frequently asked questions

References

Dark Software Factories: What Runs Lights-Out and What Breaks Without a Human in the Loop

1. The borrowed term, and what the borrowing gets wrong

2. What actually runs lights-out today (and why)

3. The failure taxonomy: where lights-out breaks

4. The checkpoint design: gates keyed to blast radius

5. The lights-out scorecard

6. Where the debate should go next

Frequently asked questions

References

Shopee Security Headers: TLS, Tokens & Bot Detection

1. What are Shopee security headers?

Four useful levels of confidence

2. Classifying headers: session state, context and custom fields

How should Shopee-specific fields be interpreted?

3. Tokens, signatures, encryption and Base64 are different concepts

An illustrative example: one change, several explanations

4. TLS, certificate authorities and certificate pinning

Capturing a PCAP does not mean reading HTTP

CA trust and pinning are separate checks

Readable HTTP can still be rejected

5. What do JA3 and JA4 TLS fingerprints tell us?

6. Bot detection: rules, behavior and risk scoring

Why might a "correct token" be insufficient?

Why not infer detection thresholds from a few attempts?

7. Device fingerprints and Play Integrity are not interchangeable

A phone succeeds and an emulator fails: what follows?

8. Shopee SG case study: what was actually observed?

Do not assign error-code meanings by guesswork

9. A test matrix for isolating causes

An example test record

10. Common mistakes in security-header analysis

Treating web and mobile clients as one protocol

Guessing internal structure from ciphertext

Mixing evidence from different sessions

Treating proxy visibility as the application's full view

Treating one success as a durable workflow

11. Frequently asked questions

What exactly does af-ac-enc-dat contain?

Does copying cookies guarantee that a request will be accepted?

Why can HTTPS be readable while login still fails?

Does Shopee definitely use JA4 or Play Integrity?

Does error 90309999 mean the device has been blocked?

Can a complete PCAP immediately reveal JSON?

Has this research reconstructed Shopee's anti-bot algorithm?

12. Conclusions and references

Primary technical references

Methodology and publication limits

PHP2Shell and the Patch-Operations Playbook: Rolling an Emergency Fix Across a Real Fleet

1. Released is not deployed: the uncomfortable math

2. Inventory: find every PHP runtime you own

3. The restart problem: upgrading the package is not the fix

4. Rollout mechanics: canary, stages, re-runnable automation

5. The EOL fleet: what to do with 7.4, 8.0 and 8.1

6. Defense in depth while the rollout runs

7. Verify the patch actually landed

8. Frequently asked questions

References

How to Validate Email Addresses Without Sending Mail: MX Records, Catch-Alls, SMTP Probing

1. Four questions, not one

2. Layer one: syntax, and why it is not trivial

3. Layer two: MX records — the domain's front door

4. Layer three: SMTP probing — useful, and a compulsive liar

5. Catch-all domains: where mailbox validation ends

6. Supporting signals that are actually worth collecting

7. Designing the pipeline: verdicts with evidence, not scores

8. Frequently asked questions

References

Git for AI Agent Teams: Branching, Review and Provenance When Most Commits Aren't Human

1. The assumptions Git makes about you

2. What breaks first (and it is not merge conflicts)

What exactly does `af-ac-enc-dat` contain?

Does error `90309999` mean the device has been blocked?