DEV Community

VolkanGunay
VolkanGunay

Posted on

Our nightly measurement run reads 800 keyword country pairs before it stops

Every ranked position in Storelift comes from a scheduled run, not an on-demand scrape. The run starts at 02:10 UTC and reads up to 800 keyword×country pairs before its own gate stops it.

Why a hard ceiling on the collector itself

800 isn't a marketing number, it's an engineering constraint we set on purpose: the collector has its own internal time budget per run, and past a certain unit count within that budget, results start arriving from a slower, less fresh part of the window than the ones at the front. Rather than letting the run silently degrade past that point, it stops at 800 and the remainder rotates in on the following run.

The alternative — no ceiling, let it run as long as it takes — sounds more thorough and is actually worse: a run with no bound eventually produces "fresh" data that's hours stale by the time it's written, with no visible signal that it happened.

What "rotation" means in practice

Above the 800-unit line, keyword×country pairs aren't dropped, they're queued for the next scheduled pass instead of squeezed into a run that's already at capacity. If your portfolio's total pairs exceed 800, you see this directly as a rotation cadence rather than a promise of "everything, every night" that quietly isn't true past a certain portfolio size.

Why 02:10 UTC specifically

Fixed, published, boring on purpose — a scheduled time you can point at is falsifiable in a way "continuously updated" isn't. If a number looks stale, there's an exact timestamp to check it against, not a vague claim about real-time tracking that never quite gets audited.

We think a tool that states its own collection limits plainly is more trustworthy than one that implies unlimited capacity and hopes nobody stress-tests it. The schedule lives in our own infrastructure code, not just in copy — it's the actual cron. We build Storelift.

Top comments (0)