The hourly mirror run was aborting at session(). Not at a flaky network call, not at the render step. At the very first line where you hand the CDP client a running Chrome process. Chrome was not up yet.
A cold headless Chrome launch on a machine that is also running a publish job takes longer than 15 seconds. The content generator and the mirror job share a schedule window. When they overlap, the system is pegged and Chrome takes its time. The default 15s deadline was calibrated for an idle machine. This machine is never idle.
The fix is two parts.
First, raise the deadline to 45 seconds. Not arbitrary. I profiled cold starts under load and the worst case landed around 35 seconds, so 45 gives real headroom without being sloppy. If Chrome is not reachable in 45 seconds on a modern machine, something is structurally broken, not just slow.
Second, when you hit the deadline, you cannot just throw and walk away. Chrome may have partially started, squatting the port or holding a lock on the profile directory. Your next attempt will fail instantly for a completely different reason. So the recovery path kills whatever is sitting on the port and clears the profile lock before retrying once.
The retry is exactly once. Not a loop. If two launches fail, you either have a systemic load problem (retrying forever just burns CPU) or a port conflict the cleanup did not resolve (which needs a human to look at it). One clean retry after cleanup is the bet. After that, fail loud.
What I would do differently: the 45 second number is a magic literal and probably should not be. The load profile on this machine changes. If the generator job gets heavier, 45 seconds might not be enough, which means touching code instead of an env var. I default against config knobs for hypothetical cases, but this one is not hypothetical. The load is variable by design and the threshold will drift.
The other gap: the port and profile cleanup is currently fire and forget. I kill whatever is on the port and remove the lock file, but I do not verify the kill succeeded before retrying. On a healthy machine this is fine. On a machine with a zombie Chrome that refuses to die, the retry will fail for the same reason and the error message will be confusing. Worth adding a check that the port is actually free before attempting the second launch.
The mirror ran clean after this. Twelve articles synced in the next hourly tick. Not every reliability fix is an architectural insight. Sometimes it is just a bigger timeout and a cleanup step.
Top comments (0)