DEV Community

Muhammad Abdullah
Muhammad Abdullah

Posted on Originally published at stackendbox.com Fully Autonomous

A future revision should not change yesterday's data selection

An as-of join can run successfully and still answer the wrong historical question. The important timestamp is when a particular version became available, not just the period the value describes.

Suppose a synthetic September fact is published on October 3 with value 100, then revised on October 7 to 80. For an October 4 decision, the eligible value is 100. A pipeline that joins today's latest revision to every earlier date silently returns 80 instead.

There is a useful regression property here: adding a future revision must not change a past selection.

from datetime import datetime

def available(value):
    result = datetime.fromisoformat(value)
    if result.tzinfo is None:
        raise ValueError("Explicit timezone required")
    return result

def select(rows, entity, period, decision):
    cutoff = available(decision)
    eligible = [r for r in rows
                if r["entity"] == entity and r["period"] == period
                and available(r["available_at"]) <= cutoff]
    if not eligible:
        return None
    stamps = [available(r["available_at"]) for r in eligible]
    if len(stamps) != len(set(stamps)):
        raise ValueError("Ambiguous revision order")
    return max(eligible, key=lambda r: available(r["available_at"]))["value"]

original = [{"entity": "DEMO", "period": "2026-09-30",
             "available_at": "2026-10-03T09:00:00+00:00", "value": 100}]
revised = original + [{"entity": "DEMO", "period": "2026-09-30",
                      "available_at": "2026-10-07T09:00:00+00:00", "value": 80}]
decision = "2026-10-04T12:00:00+00:00"

assert select(original, "DEMO", "2026-09-30", decision) == 100
assert select(revised, "DEMO", "2026-09-30", decision) == 100
assert select(revised, "DEMO", "2026-09-30", "2026-10-08T12:00:00+00:00") == 80
assert select(revised, "DEMO", "2026-09-30", "2026-10-01T12:00:00+00:00") is None
print("All four synthetic selection checks passed")
Enter fullscreen mode Exit fullscreen mode

This is a deliberately small selection example. It isn't a trading strategy, a performance result or a complete financial data model. Equal-time revisions are rejected because file order isn't an evidence-based tie-breaker.

The availability contract still needs review. Replaying what a live application knew requires actual receipt history as well as publication time. A hypothetical research archive can use declared delivery-lag assumptions, but that does not reconstruct your own system's history.

In a larger pipeline, also check the entity and fact keys, timezone precision, missing observations and revision identifiers. A backward join does not repair unreliable timestamps, retrospective universe membership, leaked feature transforms or a model trained after the historical cutoff.

The useful engineering question is whether you can preserve and test these boundaries, rather than whether a chart looks convincing.

I’m affiliated with StackEndBox. This adaptation uses synthetic examples from our tested point-in-time data guide. It contains no client data or claimed customer outcome. Prepared with AI assistance, with the displayed code executed and checked.

Top comments (0)