Nobody ships flaky browser automation because their timeout is too short. They ship it because there is nothing useful on the other side of the timeout firing.
I learned this maintaining a Medium mirror that runs on a shared machine. The job opens a headless Chrome via CDP every hour, renders posts, and syncs them. It was reliable until it was not. Adding content generation and publishing jobs to the same cron schedule meant the machine was often busy when the mirror fired. Cold Chrome launches that normally took 2 to 3 seconds were now exceeding 15 seconds. The session connect timed out. The mirror aborted at the session() call. No content mirrored.
The obvious fix is to extend the timeout, and yes, I did that: 15 seconds to 45 seconds. On a loaded machine Chrome needs real CPU time to fork subprocesses, bind a debugging port, and signal readiness. 45 seconds is enough headroom without being irresponsible.
But a longer timeout only buys you more patience. It does not help the second failure mode, which is the dangerous one: Chrome launches, binds the port, partially initializes, and dies. Now you have a zombie process squatting the socket. The next connection attempt gets an instant ConnectionRefused regardless of your timeout setting. You waited 45 seconds and then failed immediately anyway.
The real fix is the recovery path. On RuntimeError at session startup, kill any process holding the debugging port and the Chrome profile directory, then retry once. This handles the zombie case cleanly. One retry is the right ceiling. If Chrome fails to start twice in a row on the same machine, the problem is the machine, not the timeout value. An infinite retry loop here is how you turn a flaky job into a runaway process.
The tradeoff is worth naming. 45 seconds is a long blocking window in an hourly job. I considered a shorter initial timeout plus an immediate retry to reduce the worst case wait, but that approach obscures a pattern worth seeing. If launches are consistently slow, I want the logs to show consistent slowness, not a stream of recovered retries hiding an underlying load problem. One longer attempt plus one explicit recovery gives cleaner signal.
What I would do differently: the retry logic currently lives inside the function that needs a CDP session, which means it gets duplicated the next time I add an entrypoint to the mirror pipeline. The right abstraction is a dedicated acquire_browser(deadline, retries) helper that owns the process lifecycle entirely. Each caller gets a ready browser or an exception; none of them deal with port cleanup or zombie detection. The policy becomes testable in isolation without running the full pipeline.
The deeper point: browser automation reliability is a process lifecycle problem, not a protocol problem. CDPConnectionError and RuntimeError at session open are almost never caused by CDP internals. They are caused by Chrome being a heavy application that spawns many processes, each needing to initialize, and the whole chain taking longer when the machine is under load. Most automation code assumes a warm idle machine. The fix is to stop assuming that and build for what actually happens in production.
Top comments (0)