DEV Community: Russlan Ramdowar

Audit your AI forecast dataset before calling it a benchmark

Russlan Ramdowar — Sun, 04 Oct 2026 10:12:41 +0000

An AI consensus dataset is easy to mistake for a benchmark. It has scores, multiple advisor perspectives, forecast horizons, and enough rows to make a chart look convincing.

Before asking which forecast performed best, I ask a more basic engineering question: what does one row represent, and what evidence does it actually contain?

Here is a small audit of a public release from iPulse AI, the Open Agentic Investment Research Platform we build at Future Edge Group. You can reproduce it without a production account or private market data.

Start with the unit of observation

On October 4, 2026, I audited all 746 records in our historical consensus snapshot dataset. Each row describes an asset's stored consensus output for a particular snapshot and forecast horizon. It is not an individual advisor forecast, a realized return, or an independent trading experiment.

The audit found:

Check	Result
Records / distinct record IDs	746 / 746
One-year / five-year records	373 / 373
Missing values in the six required metadata fields checked below	0
Records with `model_count = 1`	746
Records marked `not_applicable_not_evaluated`	746

The last two rows matter more than a polished leaderboard. The release records forecasts, but it explicitly does not supply a completed performance evaluation. And its model count tells us to be careful about treating different advisor perspectives as independent models.

This is a bounded metadata audit of one public release. It does not validate its source data, forecast quality, or every field in its schema.

Run the audit before plotting the scores

This Python example uses only the standard library and the public Hugging Face viewer API. It reads the snapshots configuration's train split in pages. Here, train is the dataset split name; it does not establish that a model was trained on these records.

import json
from collections import Counter
from urllib.parse import urlencode
from urllib.request import urlopen

query = {
    "dataset": "future-edge-group/ipulse-ai-historical-consensus-snapshots",
    "config": "snapshots",
    "split": "train",
}
rows, offset = [], 0

while True:
    params = {**query, "offset": offset, "length": 100}
    url = "https://datasets-server.huggingface.co/rows?" + urlencode(params)
    with urlopen(url, timeout=30) as response:
        page = json.load(response)
    rows.extend(item["row"] for item in page["rows"])
    offset += len(page["rows"])
    if offset >= page["num_rows_total"]:
        break
    if not page["rows"]:
        raise RuntimeError("Incomplete pagination")

required = [
    "record_id", "snapshot_id", "forecast_horizon",
    "scoring_completed_at_utc", "source_algorithm_version",
    "record_checksum",
]
print("rows:", len(rows))
print("distinct IDs:", len({r["record_id"] for r in rows}))
print("missing metadata:", {
    field: sum(r.get(field) in (None, "") for r in rows)
    for field in required
})
print("horizons:", Counter(r["forecast_horizon"] for r in rows))
print("model counts:", Counter(r["model_count"] for r in rows))
print("evaluation status:", Counter(
    r["evaluation_methodology_version"] for r in rows
))

The API serves the current public release. Record your execution time, repository revision, and downloaded file digest when doing a formal study; today's row count is not a promise about the next release. This example checks that checksum fields exist. It does not recompute or verify them.

Distinct voices do not establish independent evidence

An advisor persona can ask a useful different question: one perspective might emphasize business quality while another emphasizes downside risks. That diversity can improve the review process.

It does not establish statistical independence. Shared models, evidence, prompts, and aggregation rules can create correlated errors. Counting perspectives is therefore a different operation from measuring error correlation against realized outcomes.

For a future comparison, I would declare the experimental unit first: model configuration, advisor configuration, asset-horizon pair, or publication snapshot. Then I would preserve those identities through the evaluation rather than treating every displayed opinion as another independent observation.

A consensus score also needs its own interpretation. Agreement is useful metadata; it is not automatically a calibrated probability that a forecast will be correct.

Keep three layers separate

My preferred design has three explicit layers:

Recorded output: the forecast, its identity, horizon, versions, and publication provenance.
Evaluation inputs: observed outcomes, data source, observation window, and adjustment rules.
Evaluation result: the metric, baseline, eligible population, exclusions, and evaluation version.

For example, a one-year forecast that has not reached its declared endpoint should not silently become a failed forecast or disappear from a table. Record it as unresolved under a stated policy. A shorter interim comparison can be useful, but it answers a different question and needs a different label.

Likewise, a financial-health field that does not apply to an asset category should not be converted to zero merely to simplify a chart. Missing, inapplicable, and measured zero are different states.

Before reporting performance, specify the horizon, available-information cutoff, outcome definition, baseline, duplicate policy, and treatment of unresolved records. Those choices belong in the evaluation contract before seeing the result.

Preserve provenance without overstating it

We also work on Forecast Library and the public Open Forecast Receipt specification and verifier. These address the record-preservation side of the problem: making forecast records portable and inspectable.

A matching digest can help detect a changed artifact. It cannot establish that the forecast was insightful, the inputs were correct, or the record was published before the event. A retrospective receipt needs an explicit retrospective label; stronger timing claims require separate evidence.

That separation is deliberate. A trustworthy research workflow should let a reader verify a recorded claim, inspect its origin, and evaluate its outcome as distinct questions.

For developers working on forecasting or agent evaluation: what do you use as your experimental unit when several advisor personas share one underlying model? I'd be interested in approaches that preserve useful perspectives without inflating the apparent sample size.

Designing AI research reports that people can actually review

Russlan Ramdowar — Sat, 03 Oct 2026 10:06:36 +0000

An AI research report can be fluent and still be difficult to review. A reader sees a conclusion, but cannot tell which facts support it, which assumptions drive it, or what would make it wrong.

I am Russlan Ramdowar, founder of Future Edge Group and builder of iPulse AI, an Open Agentic Investment Research Platform. This is a design note about making research inspectable. The examples below are illustrative design patterns, not a claim that every field is implemented in our production system.

Start with a review contract

Before choosing agents or writing prompts, define what a reviewer needs to see. A compact research view should distinguish:

Observations: what a source actually says, with its timestamp.
Interpretations: what the system infers from those observations.
Assumptions: conditions required for the interpretation to hold.
Scenarios: plausible paths, rather than one unqualified prediction.
Limitations: missing data, stale evidence and unresolved disagreement.

These distinctions matter because prose tends to blend them. “Demand may rise” is an inference. A reported order figure is an observation. Keeping them separate makes it easier to challenge an argument without losing the underlying evidence.

Give evidence an identity

A URL alone is often insufficient. The page may change, and its publication date may differ from the period described. A proposed evidence record could look like this:

{
  "evidence_id": "example-001",
  "source_url": "https://example.org/report",
  "published_at": "2026-09-30",
  "observed_at": "2026-10-03",
  "claim_type": "observation",
  "summary": "A reported fact, with its scope made explicit",
  "limitations": ["Illustrative record; not real market evidence"]
}

The useful property is traceability: a claim points to a specific record, and that record exposes the context needed to assess it. In a real implementation, the schema should also reflect the permissions and retention rules of the data source.

Preserve disagreement before summarizing it

Independent advisor perspectives are valuable only if their differences remain visible. Several agents agreeing on a conclusion does not establish that the conclusion is correct; they may share the same missing input or assumption.

A review interface can display the conclusion alongside the strongest opposing argument, the evidence each view relies on, and the assumptions that explain the difference. Consensus then becomes a summary of recorded perspectives, rather than a substitute for evidence.

This is also a useful failure mode to test: if one advisor raises a material limitation, can the final summary accidentally hide it? A reviewer should be able to recover that limitation from the report.

Treat forecasts as dated artifacts

A forecast is easier to evaluate when its original horizon, inputs and assumptions remain inspectable. Updating a view should create a new dated version while preserving what the earlier version said. Otherwise, a history of forecasts can quietly become a history of rewritten explanations.

Evaluation should also show what failed. Selecting only successful examples gives readers little basis for understanding the system’s limits. The relevant question is what a person could have known at the time the view was produced.

Put openness in the user experience

For iPulse AI, “open” refers to making research, methodology, evidence, past forecasts, evaluation, limitations and lessons inspectable. It does not automatically mean all source code, proprietary data or production infrastructure is open source.

The platform brings together ranked Top Picks, asset-level forecast paths, independent AI advisor reports, risk and driver summaries, and transparent consensus scoring. The purpose is to support research and human decision-making, without guaranteeing returns or carrying out autonomous trading.

For anyone building an AI research product, a practical starting point is one report with a visible claim, its evidence, its assumptions, a competing view and a dated limitation. If a reader cannot inspect those pieces, adding more agents will not solve the review problem.

AI assistance disclosure: This article was drafted with AI assistance from approved iPulse AI positioning and design principles. The schema is an illustrative proposal, not production documentation.