A practical example of how small data-quality checks can prevent misleading conclusions in financial market research.
Two market-data feeds can report different values for the same instrument without either feed necessarily being broken.
The difference might come from timestamps, trading sessions, missing observations, historical adjustments, or source-specific coverage.
Before comparing charts or drawing conclusions, we should establish whether the underlying observations are actually comparable.
In this tutorial, we'll build a small Python validation workflow to identify several common issues:
Missing or inconsistent timestamps
Duplicate market observations
Records outside a defined trading session
Invalid price values
Gaps in a one-minute data series
The goal is not to build a production market-data engine. It's to demonstrate how a few explicit validation rules can make research more reproducible.
- Start With the Observation, Not the Chart
Imagine receiving these observations for a hypothetical instrument called DEMO.
Timestamp (UTC)
Price
Source
13:29
101.20
A
13:30
101.25
A
13:30
101.25
A
13:31
-1.00
A
13:32
101.40
A
13:35
101.60
A
At first glance, we have six rows of market data.
But a quick review raises several questions.
Why does 13:30 appear twice? Is a negative price valid for this particular data type? Are all observations from the intended trading session? And what happened between 13:32 and 13:35?
A chart alone might not make these problems obvious.
Before visualization, we need context and validation.
- Define the Research Context
For this example, we'll use the regular trading-session hours commonly associated with US equity markets:
9:30 a.m. to 4:00 p.m. America/New_York time.
This is an illustrative session definition, not a universal market-data rule. It does not account for market holidays, early closures, auctions, or instrument-specific trading schedules.
Python's zoneinfo module helps us work with the appropriate time zone instead of assuming a permanent UTC offset.
from datetime import datetime, time, timezone
from zoneinfo import ZoneInfo
EASTERN = ZoneInfo("America/New_York")
def parse_utc(text):
dt = datetime.fromisoformat(
text.replace("Z", "+00:00")
)
if dt.tzinfo is None or dt.utcoffset() is None:
raise ValueError(
"Timestamp must include timezone"
)
return dt.astimezone(timezone.utc)
def session_label(dt):
local = dt.astimezone(EASTERN)
if local.weekday() >= 5:
return "outside_regular_session"
if time(9, 30) <= local.time() < time(16, 0):
return "regular_session"
return "outside_regular_session"
This gives us two useful properties.
First, timestamps must carry explicit timezone information.
Second, session classification takes the relevant local market time into account, including daylight-saving transitions.
For production systems, use an appropriate exchange calendar rather than relying on weekday and clock-time checks alone.
- Build a Small Sample Dataset
We'll use entirely hypothetical data.
records = [
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:29:00Z",
"price": 101.20,
"source": "A"
},
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:30:00Z",
"price": 101.25,
"source": "A"
},
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:30:00Z",
"price": 101.25,
"source": "A"
},
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:31:00Z",
"price": -1.00,
"source": "A"
},
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:32:00Z",
"price": 101.40,
"source": "A"
},
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:35:00Z",
"price": 101.60,
"source": "A"
}
]
Notice that every record includes a source identifier.
That matters because two providers can legitimately deliver observations at the same timestamp. A duplicate should be defined within the appropriate source, instrument, and data-series context rather than across all providers indiscriminately.
- Validate Each Observation
Let's check three conditions:
Whether the source and instrument already have an observation at the same timestamp.
Whether the price is positive, as required by our simplified sample data contract.
Whether the record belongs to the regular session we defined.
seen = set()
valid = []
for row in records:
dt = parse_utc(row["timestamp"])
key = (
row["source"],
row["symbol"],
dt
)
issues = []
if key in seen:
issues.append(
"duplicate timestamp for source/symbol"
)
seen.add(key)
if row["price"] <= 0:
issues.append("nonpositive price")
if session_label(dt) != "regular_session":
issues.append("outside regular session")
if issues:
print(
row["timestamp"],
"->",
"; ".join(issues)
)
else:
valid.append((dt, row["price"]))
Expected output:
2026-10-22T13:29:00Z -> outside regular session
2026-10-22T13:30:00Z -> duplicate timestamp for source/symbol
2026-10-22T13:31:00Z -> nonpositive price
These results don't necessarily mean every flagged observation should be deleted.
An extended-session observation may be perfectly valid for research that includes premarket activity.
Likewise, duplicate timestamps could indicate a data-ingestion problem, an update to an earlier observation, or a feed that allows multiple events at the same time.
Validation flags issues. The research context determines how to resolve them.
- Detect Missing Minute Observations
After excluding the flagged rows in this simplified example, we can inspect the remaining chronological sequence.
valid.sort()
for (prev_dt, _), (current_dt, _) in zip(
valid,
valid[1:]
):
gap = int(
(current_dt - prev_dt).total_seconds() // 60
)
if gap > 1:
print(
f"Gap: {gap - 1} missing minute(s) "
f"after {prev_dt.isoformat()}"
)
Expected output:
Gap: 1 missing minute(s) after 2026-10-22T13:30:00+00:00
Gap: 2 missing minute(s) after 2026-10-22T13:32:00+00:00
The first gap arises because the invalid 13:31 record was excluded.
The second reflects missing minute observations at 13:33 and 13:34.
However, a missing timestamp is not automatically proof of missing source data.
An event-driven trade feed may contain no trades during a quiet minute. A complete one-minute bar feed, by contrast, might be expected to provide a bar for every scheduled minute.
Before interpreting a gap, we need to know the data contract.
- Why Historical Data Needs More Than a Price Column
The example identifies structural problems, but several important questions remain.
Adjusted versus unadjusted prices
A historical series adjusted for stock splits or other corporate actions may differ significantly from an unadjusted series.
Timestamp semantics
Does the timestamp represent the beginning of an interval, the end of a bar, an individual market event, or the time the provider processed the record?
Trading-session coverage
Does the feed include only regular hours, or also premarket and after-hours observations?
Provider revisions
Can a data source update previously published observations? If so, how are versions tracked?
These details can affect comparisons even when all price values are technically valid.
A reproducible research workflow should preserve them alongside the numerical observations.
- Build a Reproducible Research Record
Instead of recording only a chart interpretation, save the context needed to investigate the result later.
For example:
research_record = {
"instrument": "DEMO",
"source": "A",
"data_type": "one_minute_bars",
"timezone": "UTC",
"session_timezone": "America/New_York",
"session_scope": "regular_session",
"adjustment_policy": "unadjusted",
"validation_rules": [
"timestamp_timezone_required",
"duplicate_timestamp_check",
"positive_price_check",
"session_classification",
"one_minute_gap_check"
],
"research_note": (
"Sample observations require validation "
"before chart-based comparison."
)
}
A production implementation should also preserve the instrument identifier, source version, retrieval time, expected market calendar, adjustment methodology, and the validation results themselves.
The essential principle is simple:
A research observation becomes more useful when another analyst can reconstruct the conditions under which it was made.
- Connecting Validation to Research Workflows
Market-data quality is not just an engineering concern. It directly affects analytical interpretation.
A practical sequence is:
Screen → Add context → Review the chart → Verify the source → Record the observation.
This approach also helps explain the research-oriented positioning described in public materials about Ositrustees.
Ositrustees is described as a financial data and analytical technology platform focused on market screeners, sector monitoring, chart-based research, and structured market-data workflows.
The Python example above is an independent educational demonstration of data-validation concepts. It is not Ositrustees production code, an API example, or evidence of the platform's internal implementation.
The wider lesson applies to any market-data research environment: validation and provenance should be treated as part of the research workflow, not as optional cleanup after the analysis.
Conclusion
Reliable market research starts before a chart is rendered.
A small validation routine can expose duplicate records, invalid prices, mismatched session boundaries, and missing observations.
But the most important step is deciding what each warning means in the context of the instrument, source, and research question.
When we preserve timestamps, source identity, session assumptions, adjustment rules, and validation results, the research becomes easier to review and reproduce.
Research. Compare. Verify.
Technical note: This tutorial uses hypothetical observations and the Python standard library. Its weekday/session logic is intentionally simplified; use an exchange calendar and instrument-specific data contracts for production applications.
Top comments (0)