Building a Reliable Public Results Pipeline When Sources Update at Different Times

LottoLens PH Editorial — Fri, 07 Aug 2026 05:53:28 +0000

Publishing frequently updated public data looks simple until the sources stop
behaving like a database. One page updates first, another returns an old cache,
an API omits one draw, and a static site can remain stale even after the newest
row has reached a backend store.

We encountered this problem while building LottoLens PH, an independent
information project for Philippine PCSO draw schedules, result history, and
descriptive trend data. The domain is lottery data, but the engineering pattern
also applies to sports scores, election results, transport alerts, commodity
prices, and other public records that arrive in small time-sensitive batches.

The lesson was straightforward: fetching data is only the first step. A reliable
pipeline must prove that the expected public page is fresh.

The real freshness contract

A successful HTTP request does not prove that a result is current. A source can
return 200 OK with yesterday's content, an incomplete set of games, or a cached
response from before the scheduled update.

For a draw-results system, the freshness contract needs at least four fields:

game identifier;
draw date;
draw session or time;
complete result value.

The contract should be evaluated against the expected schedule. At 9:15 PM, for
example, the pipeline should not merely ask whether any new row exists. It should
ask whether every game expected in that window has a valid row for the current
date.

Use multiple sources, but keep provenance

A fallback source improves availability, but only when every imported row keeps
its provenance. Store the source name, source URL, retrieval time, and publication
status beside the result.

That makes disagreements inspectable. It also prevents a fallback value from
silently replacing a more authoritative record without an audit trail.

For our public reference material, we separated the schedule definition from the
result archive. The normal game windows and field definitions are published as an
open PCSO draw-schedule dataset,
while the live site keeps a chronological
result history with game and session context.
The current archive size, per-game coverage, collection method, and known
limitations are documented in a separate
PCSO results data coverage report.
The dataset also has an archival DOI at
Zenodo.

Normalize before comparing

Different sources often represent the same value differently. Leading zeroes may
disappear, separators change, and dates can be interpreted in the server's time
zone instead of Philippine Time.

Normalize records before deduplication:

convert dates into one local-time convention;
preserve leading zeroes for digit games;
keep the original order for multi-number results;
map source labels into stable internal game IDs;
reject rows with the wrong number count or range.

This prevents 04 and 4 from becoming separate records and stops a six-number
row from being attached to the wrong jackpot game.

Make history immutable

Historical rows should not be recalculated every time a new draw appears. Once a
row has been verified and published, changes should be explicit corrections with
an audit reason.

This matters for user trust. A chart, saved plan, or backtest viewed today should
not show a different historical input tomorrow simply because the generation
algorithm ran again.

A practical model is:

append new verified rows;
deduplicate on game, date, and session;
preserve the original result and source metadata;
record corrections separately;
generate derived statistics from the immutable snapshot.

Validate the public output, not only the database

Static sites introduce a second freshness boundary. Updating a database or API
does not rebuild already-exported HTML.

The deployment gate therefore needs to inspect the actual production page after
the build. For each expected game, verify that the formal-domain HTML contains the
current date, session, and result. A preview URL or successful deployment receipt
is useful evidence, but it is not the final acceptance test.

The pipeline should fail closed when a required row is missing:

collect -> normalize -> validate expected window -> snapshot history
       -> build static pages -> deploy -> verify formal-domain output

If the formal-domain check fails, the run remains incomplete even when every
earlier command succeeded.

Retry around the event, not all day

Continuous high-frequency polling wastes requests and increases the chance of
rate limits. Poll aggressively only around scheduled update windows.

A simple policy is:

begin shortly after the expected publication time;
retry at short intervals while required rows are missing;
stop immediately when the full expected set is present;
escalate when a reasonable deadline is exceeded;
perform one later reconciliation pass for source corrections.

This gives fast updates without turning the collector into an all-day scraper.

Monitor missing rows as named failures

"Update failed" is too vague. Alerts should identify the exact missing contract:

missing: 2d-lotto 2026-08-06 9PM
present: 3d-lotto 2026-08-06 9PM
present: 6d-lotto 2026-08-06 9PM

Named failures make diagnosis faster and prevent a partial outage from being
described as a full-site data failure.

What we would build first

For a small team, the minimum dependable version is not a complex event platform.
It is a short observable chain:

two independent source adapters;
one normalized record format;
one expected-window validator;
one immutable history snapshot;
one build and deploy command;
one formal-domain freshness check;
one alert containing the missing row names.

That design is simple enough for one maintainer and strong enough to reveal where
freshness actually broke.

Closing thought

Reliable public data publishing is not about fetching faster. It is about making
freshness measurable from source to user-visible output.

When every stage has a clear contract, a delayed source becomes a named temporary
condition instead of a recurring mystery. The same approach can improve any
small project that turns frequently changing public records into searchable,
historical, and reusable information.

DEV Community: LottoLens PH Editorial