Never run Puppeteer inside a web request: scheduled captures with WP-Cron + a background worker

#wordpress #php #node #webdev

We let people put a URL on a schedule: re-capture this page daily, weekly, or monthly, and email me only when it actually changes. Simple feature, one sharp edge — and we found it in production.

The naive version runs the capture where the request is: user hits "check now", PHP shells out to headless Chrome, waits for the PDF, responds. It works on your machine. Then a handful of schedules come due in the same window, each capture holds a PHP worker for the 10–40 seconds Chrome takes, and the FPM pool is exhausted. On 2026-08-01 that's exactly what happened to us: synchronous captures inside web requests starved the pool and the whole site stopped answering.

The rule that came out of it: no PHP process — web request or cron — ever waits on headless Chrome. Here's the architecture that enforces it.

The shape

Two phases, deliberately split across cron ticks:

Dispatch — a cron tick finds due schedules and hands each one to a background worker (the same one the interactive tool already uses). It records a job_id and returns immediately.
Collect — a later tick picks up finished jobs, diffs the content against the previous run, keeps or discards the archive, and emails the owner.

Nothing blocks. The capture runs detached; cron just bookkeeps.

The cron tick

WordPress cron isn't a real cron — it fires on traffic — but for daily-or-slower schedules that's plenty. We register a 15-minute tick (the interval is about collect latency, not schedule precision: a dispatched job needs a later tick to be noticed, so 15 min bounds how long "Run now" takes to report).

function webtools_watches_run_due() {
    if ( get_transient( 'webtools_watches_lock' ) ) return; // a previous tick still running
    set_transient( 'webtools_watches_lock', 1, 10 * MINUTE_IN_SECONDS );

    // Phase 1: collect anything that finished since last tick.
    webtools_watches_collect_jobs();

    // Phase 2: dispatch what's due — bounded, so a burst can't stall anything.
    $now = time();
    foreach ( webtools_watches_all() as $id => $w ) {
        if ( ! empty( $w['job_id'] ) ) continue;             // already in flight
        if ( ( $w['status'] ?? '' ) === 'active' && (int) $w['next_run'] <= $now ) {
            $due[] = $id;
        }
    }
    // oldest-due first, capped at WEBTOOLS_WATCHES_PER_TICK; the rest wait for the next tick
    // ...
    delete_transient( 'webtools_watches_lock' );
}

Three things that matter more than they look:

A transient lock so overlapping ticks don't double-dispatch.
A per-tick cap (WEBTOOLS_WATCHES_PER_TICK) plus a ceiling on simultaneous Chrome jobs (WEBTOOLS_WATCH_JOBS_MAX) — each capture is its own headless browser, so unbounded concurrency is just the original outage with extra steps.
job_id as a mutex. A schedule with a job in flight is skipped, so it can never dispatch twice.

Dispatch: advance the schedule now, not later

The subtle bug we designed out: if you advance next_run when the job finishes, a job that dies never advances, and the schedule fires it again every tick forever. So next_run is bumped at dispatch, and the in-flight job_id is what prevents a re-dispatch. A dead job costs one missed run, not an infinite loop.

function webtools_watch_dispatch( $id ) {
    $watch = webtools_watch_get( $id );
    if ( ! $watch || ! empty( $watch['job_id'] ) ) return;

    $job_id = webtools_launch_site_job( [ /* url, format, device, selectors… */
        'watch_id' => $id,
    ] );

    $patch = [ 'next_run' => webtools_watch_next_run( $watch['frequency'], time() ) ];
    if ( $job_id ) { $patch['job_id'] = $job_id; $patch['job_started'] = time(); }
    webtools_watch_update( $id, $patch );
}

A single-page schedule is just a one-URL job, so the same path serves "watch this page" and "watch this whole site" — the site variant captures a fixed page list chosen up front, so later runs stay comparable and the cost can't quietly balloon.

Collect: diff the text, not the pixels

When a job reports done, we compare it to the previous run. The comparison is on a text hash per page, not the image — the worker records a textHash while it has the DOM open, and we diff those:

$prev  = (array) ( $watch['page_hashes'] ?? [] );
$fresh = [];
foreach ( $job['pages'] as $p ) {
    if ( $p['status'] === 'done' && ! empty( $p['textHash'] ) ) {
        $fresh[ $p['url'] ] = $p['textHash'];
    }
}

$first   = empty( $prev );
$changed_pages = [];
foreach ( $fresh as $url => $hash ) {
    if ( isset( $prev[ $url ] ) && $prev[ $url ] !== $hash ) $changed_pages[] = $url;
}
$changed = $first ? true : ! empty( $changed_pages );

Text hashing sidesteps the classic monitoring false positive: a pixel diff screams every time an ad rotates or a carousel advances. Hashing the extracted text ignores all of that and fires on content that actually changed. The first run is a silent baseline — there's nothing to compare to yet — so you don't get an email announcing that a page you just added "changed."

Then it's just policy: notify every run, only on change, or off (still archive, never email). In change-only mode an unchanged run's archive is deleted rather than kept as a duplicate.

What this bought us

The site can't be taken down by its own scheduled work — captures are detached, cron only bookkeeps.
A dead or stuck job self-heals: next_run already moved, and a timeout closes out the job_id so the schedule resumes.
Change detection is cheap and quiet, because it compares meaning (text) not appearance (pixels).

The general lesson travels beyond WordPress: anything that can take tens of seconds — headless browsers, video transcode, big PDF assembly — does not belong in the request that a user (or cron) is waiting on. Dispatch it, record a handle, collect it later.

This runs on every scheduled capture at Site2PDF. If you want the non-engineering version — how to actually use scheduled snapshots to track a page over time — we wrote how to document website changes.

DEV Community