tomasz dobrowolski

Posted on Sep 15 Originally published at flashalpha.com

Every API Response Should Prove Its Own Freshness. Here Is How We Do It for Options Data

#api #datascience #finance #devops

TL;DR: Options data fails silently. A stale chain, misattributed OI, a one-sided quote solved into a fake IV, a replay that returns the nearest row: none throw an error, all produce a confident number. FlashAlpha's answer is to verify against independent sources several times a day, refuse with a named reason when a check fails, and stamp every response with data_as_of so freshness is a field you read rather than a claim you accept. Keyless verification at the bottom.

Three properties, six layers

Trustworthy data is checked against something independent, fails closed with a machine-readable reason, and carries its own provenance in the response. Six layers deliver that.

1. Ingest with per-feed liveness

Exchange-grade feeds, instrument mapping resolved before storage so a contract's identity survives symbol reuse and rolls. Every physical feed (equity options, index options, futures, futures options, spot) is tracked on its own row with throughput and session health evaluated every minute. Settled OI confirmed before every open, written idempotently so re-runs cannot double-count. Universe of 6,000+ option roots refreshed as roots list and delist.

2. Cross-source reconciliation, several times a day

Check	Compared	Independent reference
Chain reconciliation	Every contract's coverage, OI and price, scored per symbol	Second market-data source
Index spot re-derivation	Spot implied by the chain via put-call parity, per expiry, outlier-trimmed	The chain itself (arbitrage identity)
Futures-to-cash parity	ES, NQ, RTY vs SPX, NDX, RUT cash	Cost-of-carry basis band
Session volume, two layers	Live tape vs vendor totals; per-contract accumulator vs its own tape	Vendor totals, conservation identity
Served-layer conservation	OI actually served by exposure endpoints, summed per strike	Truth source, per strike
Earnings dates	Rotating sample of upcoming reports	Second calendar provider

Recent full sweep: 54,705 contracts across 41 symbols at 99.6% OI agreement on comparable contracts.

The served-layer row is the one most vendors skip. Reconciling what the feed delivered is table stakes. We also reconcile what the API serves: the per-strike OI behind the GEX profile in your response is compared with the reference per strike.

A reconciliation whose reference returns nothing comparable is itself a failure. The check cannot quietly stop checking.

3. Serving guards that fail closed

Verification is theatre if degraded data is served anyway. Every refusal is machine-readable.

409 stale_data instead of a frozen price on the realised-vol and forecast endpoints:

HTTP/1.1 409 Conflict
{
  "error": "stale_data",
  "message": "The historical price archive's most recent bar is 2026-07-28 (5 trading days old). Data is stale; refusing to serve a frozen price or fit models on it. Re-hydrate the symbol's history.",
  "stale_since": "2026-07-28",
  "stale_trading_days": 5
}

Ghost quotes excluded. One-sided or crossed quotes are dropped from IV solving, Greeks and exposure aggregates rather than solved into a fictitious surface.

Gamma flip certified before publication. gamma_flip_status is available or names the failed check. No regime label is ever derived from an uncertified level.

Isolated computation per request. Each request computes on its own snapshot of the store, so numbers in one response are internally consistent and data_as_of describes exactly the feed deliveries used.

Coverage and availability gates. Strategy scorers return insufficient_data on thin history. Settled expiries never contaminate live profiles. A fleet-wide control can withdraw serving for a class or symbol in seconds with the standard no_data envelope.

4. Thirty pipelines, one health registry

Every pipeline (feed sessions, ingest, nightly writers, reconciliation monitors, alert delivery, serving loops, across every host) reports status, last-run, staleness and consecutive-failure count into one registry. Dedicated monitors on top: feed liveness every minute, served spot checked for movement every five minutes in session, OI publication confirmed before open, external processors tracked by heartbeat.

Alerts are edge-triggered and trading-calendar aware. A digest reaches humans twice per trading day whether or not anything is wrong. Silence is never assumed to mean health.

5. Point-in-time archive with archive_as_of

Stored inputs are never restated. Corrections go in the changelog, never silently under a backtest. A look-ahead guard in the replay engine and a freshness guard on the archive enforce this by construction. Every historical response carries archive_as_of in the same nine-feed shape as the live stamp. Same engine serves live and replay; parity pinned by test and sampled continuously against production.

6. data_as_of on every response

Two fields, added in one place so no endpoint can omit them: endpoint_version (the deployed build) and data_as_of (delivery timestamp of each upstream feed the responding node uses, plus node name).

{
  "symbol": "SPY",
  "underlying_price": 764.23,
  "as_of": "2026-08-24T18:49:00.697Z",
  "endpoint_version": "2026.08.25",
  "data_as_of": {
    "node": "fa2",
    "equity_feed":          "2026-08-24T18:49:00.512Z",
    "equity_options_feed":  "2026-08-24T18:48:58.204Z",
    "index_feed":           null,
    "index_options_feed":   null,
    "futures_feed":         null,
    "futures_options_feed": null,
    "flow_feed":            "2026-08-24T18:49:00.220Z",
    "oi_feed":              "2026-08-21T20:00:00.000Z",
    "macro_feed":           "2026-08-24T18:42:11.000Z"
  }
}

Feeds are stamped by physical pipe, not asset class, because spot and options arrive over different connections. Read each slot against its own cadence:

Key	Feed	Expected cadence
`equity_feed`	Equity/ETF spot	Seconds, market hours
`equity_options_feed`	Equity/ETF options	Seconds, market hours
`index_feed`	Index spot (SPX, NDX, RUT, VIX)	Seconds, market hours
`index_options_feed`	Index options	Seconds, market hours
`futures_feed`	Futures prices	Seconds, futures session
`futures_options_feed`	Futures options	Seconds, futures session
`flow_feed`	Classified trade tape	Seconds, market hours
`oi_feed`	Settled open interest	Daily, prior 16:00 ET close
`macro_feed`	VIX, VVIX, SKEW, MOVE, SPX, Fear and Greed	Minutes; reports its oldest field

The example is Monday midday and oi_feed shows Friday's close. Correct: Monday's settled OI does not exist until Monday settles. Slots a request did not read are null. macro_feed deliberately reports its oldest constituent so one lagging series cannot hide.

Top-level array responses carry the same object in X-Data-As-Of and X-Endpoint-Version headers. The Python SDK types it as DataAsOf.

Keyless provenance. /health, /livez and /readyz on lab.flashalpha.com return the same data_as_of object with no API key. Point your own monitoring at it.

Verified from outside

An external monitor probes every 30 seconds over the public customer path: a real analytics endpoint with a real key (well-formed but wrong counts as failure), the ingest with per-feed liveness, and the historical service. It reads data_as_of itself and fails any feed behind inside its own session. It also checks via put-call parity that the served book prices the served spot. Results drive flashalpha.com/status with per-component 90-day uptime, above 99.9% on every component at time of writing. The monitoring stack has 5,000+ automated tests.

Verify it in sixty seconds

# 1. Feed freshness, no key
curl -s https://lab.flashalpha.com/health

# 2. Public availability, per component, per day, 90-day window
curl -s https://lab.flashalpha.com/v1/status

# 3. Any data call: read data_as_of beside the numbers
curl -s -H "X-Api-Key: $FLASHALPHA_API_KEY" \
     https://lab.flashalpha.com/v1/exposure/gex/SPY

Gating in a pipeline, against the feeds your query actually depends on:

from datetime import datetime, timedelta, timezone
from flashalpha import FlashAlpha

fa = FlashAlpha(api_key="...")
gex = fa.gex("SPY")

feeds = gex["data_as_of"]
now = datetime.now(timezone.utc)

def age(key):
    stamp = feeds.get(key)
    if not isinstance(stamp, str):
        return None
    try:
        return now - datetime.fromisoformat(stamp.replace("Z", "+00:00"))
    except (ValueError, TypeError):
        return None

# A SPY GEX call depends on equity spot and equity options, both of which
# should be seconds old in session. Settled OI follows its own daily cadence.
in_session = [age("equity_feed"), age("equity_options_feed")]
fresh = all(a is not None and timedelta(0) <= a < timedelta(minutes=5) for a in in_session)

if fresh:
    print(f"GEX {gex['net_gex']:.3e} from node {feeds['node']}, build {gex['endpoint_version']}")
else:
    print("Feeds behind cadence - hold the signal until they catch up.")

Pair it with the status fields already on the analytics: gamma_flip_status before gamma_flip, data_quality_score on the 0DTE path, archive_as_of on every replay. Your pipeline is then gating on the same signals our monitors gate on.

FAQ

data_as_of vs as_of? data_as_of describes the inputs (when each feed last delivered). as_of describes the output (when the response was generated or the newest contract ticked).

Why is oi_feed dated to yesterday? Settled OI is published once per session, dated to the prior close. That is the newest settled OI that exists.

Historical too? Yes. archive_as_of in the same shape, same engine, same guards, archive never restated.

Due diligence access? Everything verifiable from outside is public. Tolerances, schedules, source pairings and SLA riders are shared under NDA via flashalpha.com/institutional.

DEV Community