DEV Community: Russlan Ramdowar

Your Backtest Can Pass Every Unit Test and Still Know the Future

Russlan Ramdowar — Mon, 05 Oct 2026 19:34:43 +0000

A two-timestamp schema and a future-corruption test can catch silent look-ahead leakage in financial feature pipelines.

A backtest does not need an obvious bug to cheat.

The feature calculations can be correct. The train/test split can be chronological. Every unit test can pass. Yet the model may still be reading financial data that was revised, restated, or simply unavailable when the historical decision was supposed to occur.

This is one of the nastier properties of look-ahead leakage: the pipeline can be internally consistent while the experiment is historically impossible.

The fix starts with a distinction that deserves to be explicit in every time-sensitive data model:

When was this fact true, and when could the system actually know it?

Those are different timestamps.

Event time is not knowledge time

Suppose a company reports revenue for a quarter ending June 30.

At least four dates may matter:

period_end: the economic period the fact describes
published_at: when the filing became publicly available
ingested_at: when your system received and processed it
decision_at: when the model supposedly made its prediction

A naïve feature pipeline often joins only on period_end:

SELECT
  d.symbol,
  d.decision_at,
  f.revenue
FROM decisions AS d
JOIN fundamentals AS f
  ON d.symbol = f.symbol
 AND f.period_end <= DATE(d.decision_at);

This looks chronological. It is not necessarily point-in-time correct.

A quarter can end weeks before its filing becomes public. A vendor can backfill a corrected value months later. A pipeline outage can delay ingestion. Joining only on the period being described says nothing about whether the value was available to the model.

The minimum eligibility rule is:

max(published_at, ingested_at) <= decision_at

The later of publication and ingestion becomes the earliest moment the system could legitimately use the record.

Give every fact an availability timestamp

A useful schema for versioned financial facts might look like this:

symbol
metric
period_end
value
published_at
ingested_at
source_id

Then derive:

available_at = max(published_at, ingested_at)

Do not overwrite an earlier observation when a new filing, amendment, or vendor revision arrives. Append another version.

That provides two useful properties:

The current application can still select the newest accepted value.
A historical experiment can reconstruct only the values available before its cutoff.

The SEC company-facts API illustrates why this distinction matters. Its XBRL records include reporting-period dates, values, accession numbers, forms and filing dates. The API is updated as submissions are disseminated, while the SEC also documents post-acceptance corrections and removals.

A production pipeline should capture the highest-precision dissemination timestamp available from its source. A filing date alone may be too coarse for intraday decisions.

A single “latest value” table cannot reliably reconstruct every earlier information state.

Build features with an as-of boundary

The following BigQuery-style SQL is illustrative. It selects, for each decision and metric, the latest reporting period that had become available by the decision timestamp.

It assumes published_at and ingested_at are non-null timestamps. If either may be missing, define the fallback policy explicitly instead of letting null behavior decide it silently.

WITH eligible_facts AS (
  SELECT
    d.decision_id,
    d.symbol,
    d.decision_at,
    f.metric,
    f.period_end,
    f.value,
    f.published_at,
    f.ingested_at,
    f.source_id,
    GREATEST(f.published_at, f.ingested_at) AS available_at
  FROM decisions AS d
  JOIN fundamental_fact_versions AS f
    ON d.symbol = f.symbol
  WHERE GREATEST(f.published_at, f.ingested_at) <= d.decision_at
),
ranked_facts AS (
  SELECT
    *,
    ROW_NUMBER() OVER (
      PARTITION BY decision_id, metric
      ORDER BY period_end DESC, available_at DESC, source_id DESC
    ) AS version_rank
  FROM eligible_facts
)
SELECT * EXCEPT (version_rank)
FROM ranked_facts
WHERE version_rank = 1;

The exact ranking rule depends on the metric.

For some features, you want the newest fiscal period. For others, you need a fixed period, trailing window or exact vintage. The important invariant is not the ranking expression. It is that no candidate row crosses the information boundary.

Pandas provides merge_asof for backward-looking time joins, but the same warning applies: the join key should represent availability, not merely the date the observation describes.

Add a test that poisons the future

Timestamp filters are necessary, but they are easy to break during later refactoring.

A stronger test treats the pipeline as a black box:

Calculate features at a historical cutoff.
Corrupt every source value that became available after that cutoff.
Calculate the features again.
Assert that the output is identical.

If changing the future alters a historical feature, some path is leaking information backward.

Here is a compact Python version. It assumes the timestamp fields are timezone-aware datetime objects.

from copy import deepcopy


def available_at(row):
    return max(row["published_at"], row["ingested_at"])


def build_features(rows, symbol, decision_at):
    eligible = [
        row
        for row in rows
        if row["symbol"] == symbol
        and available_at(row) <= decision_at
    ]

    latest = {}

    for row in eligible:
        metric = row["metric"]
        rank = (
            row["period_end"],
            available_at(row),
            row["source_id"],
        )

        if metric not in latest or rank > latest[metric]["rank"]:
            latest[metric] = {
                "rank": rank,
                "value": row["value"],
            }

    return {
        metric: selected["value"]
        for metric, selected in latest.items()
    }


def assert_no_future_dependency(rows, symbol, decision_at):
    baseline = build_features(rows, symbol, decision_at)
    poisoned = deepcopy(rows)

    for row in poisoned:
        if available_at(row) > decision_at:
            row["value"] = "__FUTURE_SENTINEL__"

    repeated = build_features(poisoned, symbol, decision_at)

    assert repeated == baseline, (
        "Historical features changed after future data "
        "was corrupted. Possible look-ahead leakage."
    )

This is a form of metamorphic testing. You may not know the one perfect feature vector for every historical date, but you do know a relationship that must hold:

Data unavailable at the cutoff must not influence the result.

Run the test against intermediate transformations as well as the final model matrix. Otherwise, a leaky aggregate can be materialized upstream and appear harmless downstream.

Warehouse time travel is not enough

Database time travel is useful, but it solves a different problem.

For example, BigQuery time travel retains changed or deleted table data within a configurable two-to-seven-day window, with seven days as the default. That helps with recovery and recent table reconstruction.

It does not automatically tell you when an external fact became public, when your pipeline first ingested it, or which vendor revision a year-old model would have seen.

System-version history and domain-level availability timestamps complement each other:

Warehouse history answers: “What did this table contain?”
available_at answers: “What was this experiment allowed to know?”

For durable research, preserve explicit fact versions rather than depending entirely on a short infrastructure-retention window.

Where the test can still miss leakage

A passing future-corruption test is not proof that the whole backtest is clean.

It will not automatically detect:

labels calculated with future-adjusted prices;
a security universe reconstructed from today’s constituents;
delisted companies missing from the dataset;
hyperparameters selected after repeatedly inspecting the holdout;
revised macroeconomic series stored without vintage information;
documents whose publication timestamps are recorded too coarsely;
an upstream vendor feed that already replaced historical values.

The test is best treated as one invariant in a broader evaluation system.

You still need time-ordered validation, immutable experiment records, versioned datasets, explicit universe construction, and a final holdout that is not repeatedly used for model selection.

The practical rule

If a feature table has only one timestamp, ask what that timestamp means.

If the answer is “the date this fact describes,” the table is probably missing the date that matters most to a backtest: when the fact became knowable.

Store both. Filter on knowledge time. Then poison the future and prove that the past does not move.

A backtest should not merely reproduce its calculations.

It should reproduce the information constraints of the decision it claims to simulate.

Sources

AI-assistance disclosure: This article was developed with AI assistance for structure, drafting and code review. The author reviewed and is responsible for the technical claims, examples, code and sources.

Audit your AI forecast dataset before calling it a benchmark

Russlan Ramdowar — Sun, 04 Oct 2026 10:12:41 +0000

An AI consensus dataset is easy to mistake for a benchmark. It has scores, multiple advisor perspectives, forecast horizons, and enough rows to make a chart look convincing.

Before asking which forecast performed best, I ask a more basic engineering question: what does one row represent, and what evidence does it actually contain?

Here is a small audit of a public release from iPulse AI, the Open Agentic Investment Research Platform we build at Future Edge Group. You can reproduce it without a production account or private market data.

Start with the unit of observation

On October 4, 2026, I audited all 746 records in our historical consensus snapshot dataset. Each row describes an asset's stored consensus output for a particular snapshot and forecast horizon. It is not an individual advisor forecast, a realized return, or an independent trading experiment.

The audit found:

Check	Result
Records / distinct record IDs	746 / 746
One-year / five-year records	373 / 373
Missing values in the six required metadata fields checked below	0
Records with `model_count = 1`	746
Records marked `not_applicable_not_evaluated`	746

The last two rows matter more than a polished leaderboard. The release records forecasts, but it explicitly does not supply a completed performance evaluation. And its model count tells us to be careful about treating different advisor perspectives as independent models.

This is a bounded metadata audit of one public release. It does not validate its source data, forecast quality, or every field in its schema.

Run the audit before plotting the scores

This Python example uses only the standard library and the public Hugging Face viewer API. It reads the snapshots configuration's train split in pages. Here, train is the dataset split name; it does not establish that a model was trained on these records.

import json
from collections import Counter
from urllib.parse import urlencode
from urllib.request import urlopen

query = {
    "dataset": "future-edge-group/ipulse-ai-historical-consensus-snapshots",
    "config": "snapshots",
    "split": "train",
}
rows, offset = [], 0

while True:
    params = {**query, "offset": offset, "length": 100}
    url = "https://datasets-server.huggingface.co/rows?" + urlencode(params)
    with urlopen(url, timeout=30) as response:
        page = json.load(response)
    rows.extend(item["row"] for item in page["rows"])
    offset += len(page["rows"])
    if offset >= page["num_rows_total"]:
        break
    if not page["rows"]:
        raise RuntimeError("Incomplete pagination")

required = [
    "record_id", "snapshot_id", "forecast_horizon",
    "scoring_completed_at_utc", "source_algorithm_version",
    "record_checksum",
]
print("rows:", len(rows))
print("distinct IDs:", len({r["record_id"] for r in rows}))
print("missing metadata:", {
    field: sum(r.get(field) in (None, "") for r in rows)
    for field in required
})
print("horizons:", Counter(r["forecast_horizon"] for r in rows))
print("model counts:", Counter(r["model_count"] for r in rows))
print("evaluation status:", Counter(
    r["evaluation_methodology_version"] for r in rows
))

The API serves the current public release. Record your execution time, repository revision, and downloaded file digest when doing a formal study; today's row count is not a promise about the next release. This example checks that checksum fields exist. It does not recompute or verify them.

Distinct voices do not establish independent evidence

An advisor persona can ask a useful different question: one perspective might emphasize business quality while another emphasizes downside risks. That diversity can improve the review process.

It does not establish statistical independence. Shared models, evidence, prompts, and aggregation rules can create correlated errors. Counting perspectives is therefore a different operation from measuring error correlation against realized outcomes.

For a future comparison, I would declare the experimental unit first: model configuration, advisor configuration, asset-horizon pair, or publication snapshot. Then I would preserve those identities through the evaluation rather than treating every displayed opinion as another independent observation.

A consensus score also needs its own interpretation. Agreement is useful metadata; it is not automatically a calibrated probability that a forecast will be correct.

Keep three layers separate

My preferred design has three explicit layers:

Recorded output: the forecast, its identity, horizon, versions, and publication provenance.
Evaluation inputs: observed outcomes, data source, observation window, and adjustment rules.
Evaluation result: the metric, baseline, eligible population, exclusions, and evaluation version.

For example, a one-year forecast that has not reached its declared endpoint should not silently become a failed forecast or disappear from a table. Record it as unresolved under a stated policy. A shorter interim comparison can be useful, but it answers a different question and needs a different label.

Likewise, a financial-health field that does not apply to an asset category should not be converted to zero merely to simplify a chart. Missing, inapplicable, and measured zero are different states.

Before reporting performance, specify the horizon, available-information cutoff, outcome definition, baseline, duplicate policy, and treatment of unresolved records. Those choices belong in the evaluation contract before seeing the result.

Preserve provenance without overstating it

We also work on Forecast Library and the public Open Forecast Receipt specification and verifier. These address the record-preservation side of the problem: making forecast records portable and inspectable.

A matching digest can help detect a changed artifact. It cannot establish that the forecast was insightful, the inputs were correct, or the record was published before the event. A retrospective receipt needs an explicit retrospective label; stronger timing claims require separate evidence.

That separation is deliberate. A trustworthy research workflow should let a reader verify a recorded claim, inspect its origin, and evaluate its outcome as distinct questions.

For developers working on forecasting or agent evaluation: what do you use as your experimental unit when several advisor personas share one underlying model? I'd be interested in approaches that preserve useful perspectives without inflating the apparent sample size.

Designing AI research reports that people can actually review

Russlan Ramdowar — Sat, 03 Oct 2026 10:06:36 +0000

An AI research report can be fluent and still be difficult to review. A reader sees a conclusion, but cannot tell which facts support it, which assumptions drive it, or what would make it wrong.

I am Russlan Ramdowar, founder of Future Edge Group and builder of iPulse AI, an Open Agentic Investment Research Platform. This is a design note about making research inspectable. The examples below are illustrative design patterns, not a claim that every field is implemented in our production system.

Start with a review contract

Before choosing agents or writing prompts, define what a reviewer needs to see. A compact research view should distinguish:

Observations: what a source actually says, with its timestamp.
Interpretations: what the system infers from those observations.
Assumptions: conditions required for the interpretation to hold.
Scenarios: plausible paths, rather than one unqualified prediction.
Limitations: missing data, stale evidence and unresolved disagreement.

These distinctions matter because prose tends to blend them. “Demand may rise” is an inference. A reported order figure is an observation. Keeping them separate makes it easier to challenge an argument without losing the underlying evidence.

Give evidence an identity

A URL alone is often insufficient. The page may change, and its publication date may differ from the period described. A proposed evidence record could look like this:

{
  "evidence_id": "example-001",
  "source_url": "https://example.org/report",
  "published_at": "2026-09-30",
  "observed_at": "2026-10-03",
  "claim_type": "observation",
  "summary": "A reported fact, with its scope made explicit",
  "limitations": ["Illustrative record; not real market evidence"]
}

The useful property is traceability: a claim points to a specific record, and that record exposes the context needed to assess it. In a real implementation, the schema should also reflect the permissions and retention rules of the data source.

Preserve disagreement before summarizing it

Independent advisor perspectives are valuable only if their differences remain visible. Several agents agreeing on a conclusion does not establish that the conclusion is correct; they may share the same missing input or assumption.

A review interface can display the conclusion alongside the strongest opposing argument, the evidence each view relies on, and the assumptions that explain the difference. Consensus then becomes a summary of recorded perspectives, rather than a substitute for evidence.

This is also a useful failure mode to test: if one advisor raises a material limitation, can the final summary accidentally hide it? A reviewer should be able to recover that limitation from the report.

Treat forecasts as dated artifacts

A forecast is easier to evaluate when its original horizon, inputs and assumptions remain inspectable. Updating a view should create a new dated version while preserving what the earlier version said. Otherwise, a history of forecasts can quietly become a history of rewritten explanations.

Evaluation should also show what failed. Selecting only successful examples gives readers little basis for understanding the system’s limits. The relevant question is what a person could have known at the time the view was produced.

Put openness in the user experience

For iPulse AI, “open” refers to making research, methodology, evidence, past forecasts, evaluation, limitations and lessons inspectable. It does not automatically mean all source code, proprietary data or production infrastructure is open source.

The platform brings together ranked Top Picks, asset-level forecast paths, independent AI advisor reports, risk and driver summaries, and transparent consensus scoring. The purpose is to support research and human decision-making, without guaranteeing returns or carrying out autonomous trading.

For anyone building an AI research product, a practical starting point is one report with a visible claim, its evidence, its assumptions, a competing view and a dated limitation. If a reader cannot inspect those pieces, adding more agents will not solve the review problem.

AI assistance disclosure: This article was drafted with AI assistance from approved iPulse AI positioning and design principles. The schema is an illustrative proposal, not production documentation.