Every scheduled agent we run has eventually hit the same wall: the data is on a page, and there is no API behind the page. A freelance marketplace renders filtered listings only after a JS filter panel settles. A public procurement portal builds its results table from an XHR that needs a session cookie minted by the landing page. A newsletter site hides its archive behind infinite scroll.
The reflex is to launch a headless browser. Sometimes that is right. More often it is the most expensive correct-looking decision on the table, and the bill arrives three weeks later at 4am when the cron job OOMs the runner.
What the protocol actually buys you
The Chrome DevTools Protocol is a WebSocket JSON-RPC interface into a running Chrome or Chromium. You connect, you enable domains — Page, Network, Runtime, DOM, Fetch — and you get events and commands for each. Playwright and Puppeteer are wrappers over this; you can also speak it directly, and for small agents that is often less code than the wrapper.
The part worth internalizing: you usually do not want the DOM. Enabling Network and listening for Network.responseReceived, then calling Network.getResponseBody with the request ID, hands you the exact JSON the site's own frontend consumed. The browser is doing the work you actually needed — executing the auth handshake, running the JS that constructs the request, holding the cookies — and you are reading the clean payload instead of parsing rendered markup.
We have automated 11 no-API targets this way. Eight of them resolved to intercepting a single JSON XHR. Only three genuinely needed DOM reads, and those three are the ones that break. Across roughly nine months of nightly runs, the DOM-reading jobs broke six times from markup churn — a renamed utility class, a wrapper div, a lazily-hydrated section. The XHR-intercepting jobs broke twice, both times because a response field was renamed, and both times the Zod schema at the boundary failed loudly instead of silently writing nulls.
The cost side is not subtle. In our runner, a Chromium instance idles around 180 MB RSS and peaks between 400 and 500 MB on a heavy page. Browser cold start is about 1.1 seconds; the target pages take another 2–6 seconds to reach a usable state. The equivalent fetch against a real API returns in roughly 40 ms. The container image goes from about 90 MB to around 400 MB once the Chrome shared libraries are in it. You are paying two orders of magnitude in wall clock and a full order in memory for the privilege of running someone else's JavaScript.
Spend two hours in DevTools before you write any browser code. Open the Network tab, filter to XHR/Fetch, and click through the flow by hand. If the page fetches its own data as JSON, you may be able to replay that request with a plain HTTP client plus a cookie, and skip the browser entirely. Three of our targets collapsed from "needs Chrome" to "needs one POST and a session cookie" during exactly this exercise.
The four failure modes that only show up on a schedule
Browser automation that works on your laptop and dies on a cron is not a mystery. It fails in four specific ways, and each has a boring fix.
Leaked browsers. If your process dies between launching Chrome and closing it, the Chrome stays. Do this nightly for two weeks on a small box and you get an OOM kill that has nothing to do with the run that triggered it. The fix is a preflight step that kills orphaned browser processes belonging to your job before launching a new one, plus a hard per-run timeout that escalates to SIGKILL. Track the browser PID in a file you own; do not trust the library to clean up after a hard crash.
Nondeterministic waits. networkidle is the most attractive wrong answer available. It never fires on pages with analytics beacons, polling, or a websocket, so your job hangs until the timeout instead of failing fast. Wait on a predicate you actually care about — the specific network response, or a selector that only exists once the real content is there.
Silent partial success. The page loads its shell, the list stays empty, your extractor returns zero rows, and the pipeline records a successful run. This is the worst one because nothing alerts. Assert cardinality: if a page that has returned 40–60 items every night for a month returns 3, that run failed. Pick a floor and fail below it.
State drift. Sessions expire, login flows get redesigned, an interstitial appears. Keep auth refresh in a separate step from extraction and give it its own exit code, so "we could not log in" pages someone and "the markup changed" opens a ticket.
Our incident log for that nine-month stretch: 23 failed runs total — 9 memory or zombie-process related, 6 wait-condition timeouts, 5 auth expiry, 3 markup changes. The first two categories are 65% of the failures and neither has anything to do with the site you are reading. They are operational bugs in your own harness.
When it is the wrong call
Driving a browser is the wrong call more often than the tooling ecosystem suggests. Concretely:
| Situation | Do this instead |
|---|---|
| An RSS feed, sitemap, or public JSON endpoint exists | Use it — check /sitemap.xml, /feed, and the Network tab first |
| The terms of service prohibit automated access | Stop. The technical question is downstream of the permission question |
| You need thousands of pages per hour | Renegotiate for data access, or narrow the scope |
| The data is needed inside a user request | Never. Move it to a queue with a cached result |
| The site runs commercial anti-bot | Stop. Evading it is a different activity than reading a public page |
That last row deserves being explicit. Once a site has deployed a bot-detection product, the remaining engineering is evasion, and evasion is an arms race you will lose on a schedule — your job breaks on their release cadence, not yours. It is also a clear signal about consent. Read it as one.
Before you automate any third-party site: read the terms of service, honor
robots.txt, rate-limit yourself well below what the server can absorb, and identify your agent. We cap at one request every 3 seconds with a single concurrent session per host, and our User-Agent carries a contact URL. If a site asks us to stop, we stop. None of this is optional politeness — it is the difference between an automation practice you can defend and one you cannot.
The honest heuristic we use now: browser automation is justified when the target is small (tens of pages, not thousands), the cadence is slow (daily, not per-minute), the access is permitted, and the value of the data clears the roughly 100x cost multiplier over a plain HTTP call. Three of our 11 targets have since been retired because they stopped clearing that bar.
Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.
Top comments (0)