DEV Community: manja316

94.6% of ended prediction markets converge to near-certainty — and why I still can't tell you if they were right

manja316 — Mon, 29 Jun 2026 01:08:25 +0000

I run a collector that snapshots every active Polymarket market's price every 15 minutes. After 92 days it's 18.6 million price points across 22,410 markets. People keep asking the same question, and the honest answer is more interesting than the marketing one.

"Is Polymarket well-calibrated? When it says 70%, does the thing happen 70% of the time?"

I can't answer that from this data. But I can show you something adjacent and real — and, more usefully, I can show you exactly where the wall is, because most "prediction markets are smart" posts walk straight through it without noticing.

The thing I CAN measure: convergence

Take every market that has ended (end_date in the past) and look at its last traded YES price. On the frozen export (2026-03-28 → 2026-06-28):

SELECT
  COUNT(*) AS ended,
  ROUND(100.0*SUM(CASE WHEN last_trade_price>=0.95 OR last_trade_price<=0.05
        THEN 1 ELSE 0 END)/COUNT(*),1) AS pct_decisive,
  ROUND(100.0*SUM(CASE WHEN last_trade_price>0.40 AND last_trade_price<0.60
        THEN 1 ELSE 0 END)/COUNT(*),1) AS pct_coinflip
FROM markets
WHERE end_date < date('now') AND end_date != '' AND last_trade_price IS NOT NULL;

ended    pct_decisive   pct_coinflip
19402    94.6           0.9

94.6% of ended markets had their price collapse to ≥0.95 or ≤0.05 by close. Only 0.9% were still a coin-flip (0.40–0.60). The crowd makes up its mind almost every time before the window shuts.

That's a genuinely useful fact if you're building anything time-aware: a market sitting at 0.55 two days before close is in the rare 4.5% that hasn't resolved its uncertainty yet — that's where the trades live.

The thing I CANNOT measure: was it right?

Here's where the wall is. Convergence is not correctness. A market going to 0.97 tells you traders agreed. It does not tell you the YES outcome happened. To know that, you need the resolved outcome — and this dataset doesn't have it:

SELECT resolved, COUNT(*) FROM markets GROUP BY resolved;
-- 0 | 22410   (every single row)

resolved_outcome is empty for all 22,410 markets. The collector reads live prices from Polymarket's Gamma/CLOB APIs every 15 minutes; it never joins the on-chain resolution feed. So calibration curves, favorite-longshot bias, Brier scores — anything that needs "which side won" — are not computable from this file alone.

You could approximate by assuming the final price equals the truth (price ≥ 0.95 → "YES happened"). But that's circular: you'd be grading the market against its own last guess, then announcing it's well-calibrated. It's the single most common mistake in prediction-market blog posts.

Why ship a dataset and tell you what it can't do?

Because the alternative is a refund and a 1-star review. The price series is the dense, reliable layer — 15-minute resolution, 92 days, no gaps, no scraping (straight from the public Gamma + CLOB APIs). That's worth paying for if you're backtesting price-based strategies, training a short-horizon predictor, or studying microstructure. It is not worth paying for if you wanted a calibration study, and I'd rather you know that before you click.

(Same reason I published the audit showing ~94% of the order-book rows are thin-market placeholders. Price = the product. Order book and resolution labels = honest caveats.)

Reproduce it in 30 seconds

Free 1-day sample (no signup): Hugging Face

import sqlite3, pandas as pd
con = sqlite3.connect("polymarket.db")
m = pd.read_sql("SELECT last_trade_price p FROM markets "
                "WHERE end_date < date('now') AND last_trade_price IS NOT NULL", con)
decisive = ((m.p >= 0.95) | (m.p <= 0.05)).mean()
print(f"{decisive:.1%} of ended markets converged")

Full dataset, $19 one-time: Gumroad. Live auto-refreshing API: api.protodex.io.

Open question for the comments: if you had the resolution labels joined in, what's the first thing you'd test — calibration, or fading the longshots? That's the next build and I'll prioritize by what people actually want.

What 91 days of Polymarket price data can — and can't — tell you

manja316 — Sat, 27 Jun 2026 23:06:08 +0000

Most "dataset" posts oversell. This one tells you the limits first.

For 91 straight days (28 Mar → 27 Jun 2026) I collected Polymarket prices on one Mac, no cloud bill. The frozen export is:

18,591,646 price snapshots
22,362 markets (19,819 of them with an actual price series — avg 938 snapshots per market)
1,854,788 orderbook snapshots

Category mix: sports 12,258 · other 4,689 · crypto 2,514 · politics 1,441 · geopolitics 685 · science/tech 241 · economics 224 · entertainment 169 · weather 137.

Here's what you can actually do with that — and three things you can't.

What it's good for

1. Price-path / convergence studies. Of the 6,776 markets that ended inside the window, 6,551 (96.7%) closed decisively — last Yes-price above 0.95 or below 0.05. Only 3.3% were still mushy in the middle at the end. So the data is clean enough to study how and when a market makes up its mind: most of the information arrives well before close, and you can measure the shape of that arrival per category. (Crypto markets snap late; politics drifts.)

2. Mean-reversion / momentum backtests on the spread. With ~938 snapshots/market you have enough intraday resolution to test "does a 10-point move in 6 hours revert or continue?" across thousands of contracts and eight categories.

3. Behavioral / microstructure features. Volume, 24h volume, liquidity, best bid/ask, spread, and last-trade are all captured per snapshot, so you can build features without re-hitting the API 17 million times.

What it can NOT do (the part most listings hide)

1. True calibration. The dataset has no resolution labels. Every market in the export carries price series — not the final settled outcome. So you can study price convergence, but you cannot compute real calibration (predicted probability vs realized result) without joining external resolution data yourself. If a vendor shows you a "calibration curve" derived purely from prices, they're measuring the market against itself, not against reality. I'm not going to pretend otherwise.

2. Deep order-book research. The orderbook table is large (1.85M rows) but only a single-digit % of those rows are genuine two-sided quotes — the rest are placeholder/one-sided. The price series is the real product here, not the book. Buy it for price history; don't buy it expecting an L2 reconstruction.

3. Sub-minute tick data. This is snapshot cadence, not a trade-by-trade feed. Great for hourly/daily strategy research, wrong tool for HFT.

Why collect it at all

Polymarket's API is free but rate-limited and ephemeral — query it today and last month's prices are gone. The value of a frozen 91-day export is that the history exists and is queryable in one SQLite file (prices, markets, orderbooks, market_features, …), indexed on (market_id, ts). Pull a market's full path in one query instead of paginating a live endpoint.

SELECT ts, price
FROM prices
WHERE market_id = ? AND outcome = 'Yes'
ORDER BY ts;

That's the whole pitch: clean, indexed, honestly-scoped prediction-market price history you can backtest against tonight.

If that's the shape of data you need, the full export (and a free sample tier) is here:
👉 Polymarket Historical Price Dataset

Questions about schema or coverage before you buy — ask in the comments and I'll answer with a real query against the file.

Counts above are computed directly from the delivered export (the exact file you download), not the live collector, so what you read is what you get.

88% of the order-book rows in my dataset were fake. Here's how I caught it.

manja316 — Fri, 26 Jun 2026 02:23:29 +0000

I've been collecting Polymarket prediction-market data on a single Mac for 89 days. As of this morning the SQLite file holds:

18,207,844 price snapshots (15-minute OHLC)
21,871 markets
1,816,392 order-book rows
span: 2026-03-28 → 2026-06-25 (89 days)

For weeks I described it as "18M price snapshots + 1.8M order-book records." That second number was a lie I was telling by accident. This post is the autopsy, because if you buy or build on market data, the failure mode I hit is one you will hit too.

The smell test that should have run on day one

I went to compute realized spreads — best_ask - best_bid over time — expecting a tight distribution for liquid markets. Instead almost every row looked like this:

best_bid | best_ask | spread | mid
   0.001 |    0.999 |  0.998 | 0.5
   0.001 |    0.999 |  0.998 | 0.5
   0.001 |    0.999 |  0.998 | 0.5

A 99.8-cent spread on a market that resolves between $0 and $1 is not a quote. It's the placeholder my collector wrote when the CLOB returned an empty book — and I had been counting every one of those as a "record."

One query settled it:

SELECT
  COUNT(*) AS total,
  SUM(CASE WHEN (best_ask - best_bid) >= 0.99 THEN 1 ELSE 0 END) AS effectively_empty
FROM orderbooks;
-- total            = 1,816,392
-- effectively_empty = 1,602,616   (88.2%)

88.2% of my order-book rows carry the 0.001 / 0.999 placeholder. Only ~11.8% have a spread tight enough to even be a candidate for a real two-sided quote, and the genuinely tradeable subset is smaller still. The "1.8M order-book records" headline was real rows in a table and almost entirely empty as information.

Why this happens (and why it's not Polymarket's fault)

Prediction markets are thin. Most of the 21,871 markets are long-tail contracts — a niche election, a sports prop, a "will X tweet by Friday." At any given 15-minute poll, most of those books are genuinely empty. The collector dutifully recorded "no bid, no ask" as a row. Nothing was broken. The bug was in how I summarized it.

The price series, by contrast, is dense and reliable: a market's last-trade / OHLC ticks along even when the book is empty, because it reflects executed prices, not resting orders. 18.2M price rows are 18.2M real observations. 1.8M book rows are ~200K real observations wearing a costume.

What I changed

Dropped "order-book records" as a headline feature everywhere I controlled — README, the dataset article, the listing copy. Selling an 88%-empty column as a feature is how you earn refund requests.
Re-led with the price series, which is what the dataset is actually good for: backtesting, calibration studies, favorite-longshot analysis.
Shipped the empty book anyway, labeled honestly — as a sparse bonus with its real coverage stated, not as a selling point. A few hundred thousand real top-of-book snapshots still have uses; pretending it's 1.8M does not.

The transferable lesson

COUNT(*) is not coverage. Before you trust — or sell, or backtest on — any market dataset:

Run the spread/sanity distribution, not just the row count. A column can be 100% populated and ~0% informative.
Separate observations from rows. Placeholders are rows. Only non-degenerate values are observations.
State coverage as a percentage of plausible values, not as a raw total. "1.8M rows, 12% non-empty" is honest; "1.8M records" is marketing.

I'd rather a buyer learn this from my article than discover it after paying. The dense, verified layer — 18.2M price snapshots across 89 days — is the thing worth having, and it's the thing I now lead with.

The full dataset (price series + the honestly-labeled sparse book) is on Gumroad: Polymarket Quant Toolkit. There's a free sample so you can run your own sanity queries before paying — which, after reading this, you absolutely should.

What 86 Days of Clean Prediction-Market History Actually Shows

manja316 — Wed, 24 Jun 2026 22:59:02 +0000

Everyone has an opinion about whether prediction markets are "smart." Almost nobody has the data to check.

I do. For 86 days straight, a single Mac mini has been polling Polymarket's public API and writing every price tick to a local SQLite file. As of this morning:

20,585 markets tracked
17,256,332 price snapshots
1,721,218 order-book snapshots
85 continuous days (2026-03-28 → 2026-06-21)
$0/month in infrastructure — no cloud, no cluster, one always-on machine

(Those are the exact counts in the downloadable export. The live recorder keeps running — the local feed is already past 18.0M snapshots across 19,084 markets — but a buyer should know precisely what's in the file, so these are the file's numbers, not the live feed's.)

That's not a toy sample. It's enough to ask the only question that matters about a prediction market: when the crowd says 70%, does the thing happen 70% of the time?

Why this is hard to get anywhere else

Polymarket's API serves you the current price. It does not hand you history. If you want to know what a market was trading at three weeks ago — the thing you need for any backtest — you had to have been recording it then. There's no rewind button.

So the dataset isn't valuable because the data is secret. It's valuable because it's time-stamped and continuous. You can't reconstruct it after the fact at any price. You either started the recorder in March or you didn't.

The calibration question, concretely

Take every market that resolved. Bucket the price history into deciles — the 0–10% bucket, the 10–20% bucket, and so on. For each bucket, compute the fraction of those markets that actually resolved YES.

A perfectly calibrated market draws a straight 45° line: the 30% bucket resolves YES ~30% of the time, the 90% bucket ~90%, etc.

When I ran exactly this earlier in June, the broad finding held up better than the cynics expect — the crowd is roughly honest in the middle of the distribution, and the interesting distortions live in the tails (very cheap longshots and very expensive favorites), which is precisely where the favorite-longshot bias lives in every betting market ever studied. I wrote that up with the bucket-by-bucket table here: when Polymarket says 70%, does it happen 70% of the time?.

The point of this post isn't to re-paste that table. It's to show you the query is trivial once you have the history:

import sqlite3, pandas as pd

con = sqlite3.connect("market_universe.db")

# Latest price per resolved market, joined to its outcome
df = pd.read_sql("""
  SELECT p.market_id, p.outcome, p.price, mo.outcome_label
  FROM prices p
  JOIN market_outcomes mo
    ON p.market_id = mo.market_id
  WHERE p.outcome = 'Yes'
""", con)

df["bucket"] = (df.price * 10).astype(int).clip(0, 9)
# join your resolution labels, then:
calib = df.groupby("bucket").agg(
    n=("price", "size"),
    mean_price=("price", "mean"),
)
print(calib)

Five lines of pandas. The hard part — having 17 million rows of honest, timestamped history to run it against — is the part that took 86 days.

What you can build on top of it

Calibration audits — is a specific market category (politics? sports? crypto?) better calibrated than another?
Longshot fade backtests — systematically short the sub-10% bucket and measure the edge after fees.
Mean-reversion / momentum studies on the minute-level price path.
Event-study windows — how fast does a market re-price around a news shock?

All of it needs the same thing: a continuous price record you didn't have to be there to capture.

Get the data

Free sample + schema to kick the tires: comment and I'll point you at it.
Full archive (17.2M snapshots, one-time download): on Gumroad.
Monthly refresh (the recorder keeps running; you get the new days): ask in the comments — I'm pricing it for the people who actually backtest.

The crowd is mostly calibrated. The edge is in knowing exactly where it isn't — and that only shows up if someone was recording. Someone was.

17 million Polymarket price snapshots, collected on one Mac for $0/month

manja316 — Mon, 22 Jun 2026 10:54:09 +0000

Most prediction-market datasets you find online are a one-time dump someone scraped, posted, and abandoned. They go stale the day after they're published. I wanted a living archive of Polymarket — every market, sampled every 15 minutes, running continuously — and I wanted it to cost nothing to operate.

Here's what the archive holds today, counted straight from the database:

18,611,636 price snapshots
22,410 markets tracked
~92 days of continuous 15-minute history (and counting)

All of it runs on a single Mac, with a recurring infrastructure bill of $0/month. No cloud database, no managed queue, no Kubernetes. This post is how.

The architecture is boring on purpose

The whole thing is three moving parts:

A collector — a Python process on a 15-minute timer (launchd, the macOS-native scheduler — no cron daemon, no external trigger). Each tick pulls the live market list from Polymarket's public API, then for each active market records the current YES/NO prices and, where one exists, the top of the order book.
One SQLite file — not Postgres, not a warehouse. SQLite handles tens of millions of rows on a laptop without complaint as long as you index the access path you actually use. The entire archive is a single .db file you can scp anywhere.
A daily exporter — dumps the SQLite tables to Parquet so the dataset is portable and loads in one line of pandas/Polars.

That's it. The "$0/month" isn't a trick — a laptop you already own, the OS scheduler you already have, and a file format that doesn't bill you.

One honest caveat: the price series is the real product, not the book

The collector also samples top-of-book (best bid / best ask) every tick — but be clear-eyed about it: most Polymarket markets are thin, so a live two-sided book simply doesn't exist at most sample times. Counted straight from the DB, only about 6% of the 1.86M book rows captured a real two-sided quote; the rest are markets with no live book at that moment. So treat the dense, reliable layer as the price time series (18.6M real snapshots) — that's what you backtest on. The book samples are a sparse bonus for the handful of liquid markets, not a full reconstructable order book for everything. I'd rather tell you that up front than have you discover it after download.

Index the query, not the table. The naive mistake is to over-index and watch your write throughput collapse every 15 minutes. The prices table has exactly one composite index — (market_id, ts) — because the only read pattern that matters is "give me the price history of this market over this window." One index, sized to the actual query, keeps both the 15-minute writes and the backtest reads fast.

Append, never mutate. Every snapshot is an immutable row stamped with an ISO-8601 UTC timestamp. Nothing is ever updated in place. That means the archive is a true time series — you can reconstruct the YES/NO price as it stood at any 15-minute mark in the last 92 days, not just "latest state." Mutable rows would have quietly destroyed the history I was trying to capture.

Why this is hard to copy (and why that matters if you trade)

The dataset isn't valuable because the code is clever — it's three boring parts. It's valuable because of the one thing you cannot backfill: time. You cannot retroactively collect the price as it stood on April 3rd at 14:15 UTC. Either a process was running and recording it, or that moment is gone forever.

So if you want 92 days of 15-minute Polymarket history to backtest a mean-reversion or calibration strategy, you have two options: stand up a collector today and wait three months, or start from the archive that's already been running since March 28th.

Get the data

Free sample + schema + loader: github.com/LuciferForge/polymarket-historical-data — grab the sample, check the schema, run the example query before you commit to anything.
Full dataset (one-time download): on Gumroad.

If you'd want this kept fresh automatically — a recurring refresh so your backtests never run on a frozen file — that's the thing I'm deciding whether to build next. There's an open roadmap thread on the repo; tell me what cadence and format you'd actually use.

All figures in this post were counted from the database export built on 2026-06-28 and are not estimates.

What 295,732 IPL balls reveal: chase to win, and the death overs are a different game

manja316 — Mon, 22 Jun 2026 02:01:25 +0000

I cleaned every IPL delivery from 2008 to 2026 into one tidy ball-by-ball table — 295,732 rows — and added the columns I always end up recomputing by hand (match phase, running run-rate, per-batter tallies, and fantasy box-scores). Then I asked the data three simple questions.

1. Chasing wins — but the toss barely matters

Across 1,218 completed matches, the toss-winner won just 51.6% of the time. Close to a coin flip — until you look at what they do with it.

Captains who won the toss and chose to field (chase) won 54.7% of their matches. Those who chose to bat first won only 45.3%. A ~9-point swing from one decision — chasing is a real, persistent edge in the IPL.

2. The death overs are a different sport

Run rate by phase, across every IPL ball:

Phase	Run rate	Wickets / over
Powerplay (1–6)	7.78	0.23
Middle (7–15)	7.81	0.26
Death (16–20)	9.78	0.52

Scoring barely moves from powerplay to middle, then explodes at the death — but the wicket rate more than doubles. If you're modeling fantasy points or win probability, treating all overs the same leaves signal on the table.

3. Do it yourself

The dataset is one flat CSV, so this is a three-liner:

import pandas as pd
df = pd.read_csv("ipl_ballbyball.csv")
print(df.groupby("phase")["runs_total"].sum() * 6 / df.groupby("phase")["ball_in_innings"].count())

The data

Free sample on Kaggle: IPL ball-by-ball + fantasy box-scores
Full 19-season history (295,732 deliveries + 27,374 player-match fantasy box-scores), auto-refreshed weekly: $12 on Gumroad
Internationals too: 765k+ T20I deliveries on Kaggle

Built from Cricsheet (ODC-BY). Columns include match phase, live run rate, running score, per-batter tallies, and per-player fantasy box-scores — so you can model straight away instead of parsing raw JSON.

Supervising 13 always-on processes without Kubernetes: the log-freshness health check that ended my 3 a.m. babysitting

manja316 — Sat, 20 Jun 2026 22:54:36 +0000

I'm a solo builder. At any given moment I have a handful of always-on processes earning, scraping, or thinking on my behalf: a Polymarket crash-recovery trading bot, a weekly site-scraper that regenerates a ~14k-entry directory, a couple of API servers, a secret-scanner, a DB-retention job, and a few scheduled agents that call an LLM to review strategy or hunt vulnerabilities. Thirteen in total.

For months these lived as a mess of launchd plists, loose crontab lines, and nohup … & processes I'd forget about until one silently died — and I'd notice days later, usually because a number on a dashboard stopped moving. That's the classic agent-ops failure mode: lots of independent autonomous processes, no unified supervisor. Each one is individually simple. Collectively they're un-auditable.

I didn't want Kubernetes. For one solo dev's always-on jobs, k8s is a sledgehammer — a control plane heavier than everything it would supervise. I wanted the smallest thing that would give me one honest answer to one question: is everything that's supposed to be running actually running, and actually working?

So I built ForgeOS: a tiny kernel (~1,900 LOC of Python, zero heavy deps — just pyyaml) that treats every process as an engine defined by a single YAML file.

Three engine types, one primitive

daemon        long-running        the trading bot, API servers, collectors
cron          scheduled           weekly export, daily scan
intelligence  scheduled + LLM     a Claude-CLI agent run on a schedule, output piped back in

The third type is the one people raise an eyebrow at, so let me defend it early: an intelligence engine is just a cron job that happens to call an LLM. Same scheduling, same health checks, same logging as any other engine. The moment I stopped treating "AI agents" as special snowflakes and modeled them as the same primitive as a backup job, the whole system got simpler. A scheduled strategy-review agent and a nightly DB-prune are the same shape; only the command differs.

The part that actually changed my life: health + self-heal

Each engine declares what "healthy" means right in its config:

name: crash-bot
type: daemon
description: Polymarket crash-recovery bot
command: ["python3", "pm_crash_monitor.py"]
cwd: ~/.../agents/trader
health:
  process: pm_crash_monitor      # must appear in the process table
  log_max_age_min: 20            # log must have been written in the last 20 min
  log_path: /tmp/pm_crash_monitor.log
kill_condition: "daily_loss > $10 OR cash < $20"
env:
  LIVE_TRADING: "true"

The kernel runs a loop: for each enabled engine, check the declared health signals — is the process alive, is the log fresh? If a daemon is dead or its log has gone stale, restart it and record the event. One command, forge health, tells me the true state of all 13 engines instead of me SSH-ing around running ps aux | grep.

forge init       # creates ~/.forgeos/
forge start      # start all daemon engines
forge health     # the one honest answer
forge daemon     # run the kernel: health + scheduling + self-heal
forge brief      # one-screen status of everything

Three lessons that generalize beyond my setup

If you run autonomous agents, these cost me real downtime to learn:

1. "Is the process running?" is a near-useless health check on its own. A hung process is "running." The PID is right there. My collectors and AI agents fail far more often by getting stuck than by crashing — and a stuck process keeps its PID forever. Adding log_max_age_min (the log must have been written recently) caught dramatically more real failures than PID checks ever did. Liveness ≠ progress. Health-check progress.

2. Declare kill conditions in config, not buried in code. My trading engine carries kill_condition: daily_loss > $10 OR cash < $20 next to its definition — not 400 lines deep in the bot's source. Keeping the safety limit visible, beside the engine it governs, is what made me actually willing to let a money-moving process run unattended. A safety limit you can't see at a glance is a safety limit you don't trust.

3. Model AI agents as ordinary engines. The temptation is to build a special "agent runtime." Resist it. An LLM call on a schedule is a cron job with an expensive command. Give it the same health loop (did it run? is its output log fresh?) and the same restart policy as everything else. The non-determinism lives inside the command; the supervision around it should be boring.

The honest part: where this still falls short

Log-freshness answers "did it run." It does not answer "did it run well" — which for an intelligence engine (an LLM whose output is non-deterministic text) is the question that actually matters. Right now that's still manual review for me: the agent reliably runs, but judging whether its strategy review was good is a human gate. I haven't automated the quality check, and I'm not convinced a naive "LLM grades the LLM" loop is trustworthy enough to remove me from it yet. If you've solved non-deterministic-output health-checking for scheduled agents, I genuinely want to hear how — that's the open edge of this design.

Status

ForgeOS is pre-release (v0.1.0) and built in public — it's not on PyPI or a public repo yet; today it installs from a source checkout (pip install -e . from the project root). I'm writing this up now because the architecture (declarative engines + log-freshness health + self-heal, no k8s) is the reusable part, and it stands on its own whether or not you ever run my code.

If you're supervising long-running or scheduled agents today — systemd? a hosted orchestrator? homegrown? — I'd love to compare notes in the comments, especially on that last open problem: how do you health-check an agent whose output is non-deterministic text?

MCP's real attack surface isn't prompt injection — it's the trust boundary (21 patterns, 5 languages)

manja316 — Fri, 19 Jun 2026 22:53:24 +0000

We keep talking about prompt injection like it's the endgame. It isn't. Prompt injection is step one. The actual damage happens one step later, at a place most MCP threat models barely mention: the trust boundary — the moment an injected instruction turns into a real tool call that runs with your machine's privileges.

I build and run a handful of MCP servers locally, and the more server source I read, the more I became convinced we're auditing the wrong layer. So I built a small MCP server whose only job is to audit other MCP servers. This is the write-up of what it looks for and why — the threat model is the reusable part whether or not you ever run my code.

The chain that should keep you up at night

An MCP server is, by design, code that an LLM is allowed to invoke. That's the whole point. Which means the dangerous chain is:

untrusted input → model is convinced → tool call fires → server-side code executes on your box

The injection was never the vulnerability. The injection is just the delivery mechanism. The vulnerability is what the tool does once it's called. If the server-side handler does anything unsafe with the arguments it receives, the model has just become a remote-code-execution courier — and it did it while behaving exactly as designed.

This reframes the problem in a useful way: most MCP security is just appsec, newly reachable through a model. The bugs that matter are 20-year-old classics. What changed is who can reach them. Pre-MCP, a lot of this code was only callable by a trusted caller. Now a sentence in a webpage your agent reads can reach it.

The patterns that keep showing up

Reading through MCP server source, the same handful of mistakes appear over and over. These are the OWASP greatest hits, scored by how badly they bite when they sit one tool-call away:

eval() / exec() / os.system() on tool arguments     CRITICAL  direct code injection
subprocess(..., shell=True), Runtime.exec(concat)    HIGH      command injection
pickle.load / torch.load / ObjectInputStream         HIGH      deserialization RCE
yaml.load() without SafeLoader                        HIGH      object instantiation
f-string / string-concat SQL                          HIGH      SQL injection
URL built by concatenation (fetch tools)              MEDIUM    SSRF / network pivot
hardcoded API keys / tokens in server source          MEDIUM    credential leak

None of these are exotic. That's the point. The scanner I built detects 21 vulnerability patterns across Python, Java, Go, C++, and Rust — and almost every one predates LLMs entirely. The novelty isn't the bug class; it's the reachability.

The design decision that mattered most: purpose-aware severity

The naive version of a scanner like this is a glorified grep eval. It floods you with false positives and you stop reading the output by day two. An eval() inside a sandboxed test harness is not the same finding as an eval() on a tool's input argument — and a scanner that scores them identically is noise.

So the core of the tool is purpose-aware scoring: it weighs a pattern by where it sits and what reaches it. A subprocess(shell=True) in a CI helper that never touches model input is a low-priority note. The same call wired to a tool argument is a CRITICAL. Getting this right is the difference between a report someone acts on and a report someone closes.

The scanner ships as an MCP server itself, with three tools:

audit_repo — point it at a GitHub URL, get a scored report
audit_code — paste a snippet, get findings inline
list_patterns — see every pattern and its severity

Because it runs locally as an MCP server in Claude Desktop, your agent can audit a repo before you wire it in — which is the right time to find out a server you're about to trust shells out on its arguments.

pip install mcp-security-audit

Two lessons that generalize beyond MCP

1. The injection is not the vuln — the handler is. Spend your defensive budget on what tool code does with arguments, not only on filtering what reaches the model. Input filtering is a sieve; a safe handler is a wall. You want the wall.

2. Severity without context is noise that trains people to ignore you. Any scanner that can't tell a sandboxed eval from a reachable one will get muted. Context-awareness isn't a nice-to-have on a security tool — it's the feature that decides whether anyone reads the second report.

The honest part: where this still falls short

Static patterns catch reachable-by-shape, not reachable-in-fact. A taint analysis that proves an argument actually flows into a dangerous sink would cut false positives further — that's the open edge of this design, and it's not built yet. And the two pattern families I'm least confident I've covered well are path traversal in file-serving tools and SSRF in fetch-style tools that are built to make outbound requests, where "is this call malicious" is genuinely ambiguous. If you've solved the fetch-tool SSRF problem cleanly, I want to hear how.

Where to find it

The scanner is open-source (MIT) and free:

Repo + star: https://github.com/LuciferForge/mcp-security-audit
Install: pip install mcp-security-audit

If you'd rather not run it yourself — if you're shipping an MCP server and want a scored, human-reviewed audit report with a 3-day turnaround instead of a raw scan dump — I offer that as a paid service here: MCP Security Audit Report — $29.

But I'm more interested in the threat-model discussion than either. Two open questions for the comments:

Where should the boundary live? Sandbox every tool call by default (perf + DX cost) or scan-and-trust at install time? I lean install-time scanning plus runtime allow-lists, and I'd like to be talked out of it.
What pattern am I missing? I've got 21. Tell me the 22nd.

How much edge is left in the final 48h of a Polymarket? I measured price convergence on 18.6M snapshots

manja316 — Thu, 18 Jun 2026 22:54:17 +0000

A question I kept circling when I started treating prediction markets as a trading venue: by the time a market is obviously heading one way, is the move already priced in — or is there still a systematic gap between price and outcome you can lean on?

"Buy the favorite at 0.85, collect the 0.15" sounds like free money until you account for the markets where the favorite loses. So I stopped eyeballing it and computed it.

The data

I've been archiving Polymarket order books every 15 minutes since late March 2026. As of mid-June that's ~18.6M price snapshots across ~18.6k markets (~77 days, most of them resolved). Prediction markets are one of the few venues where you get the probability estimate, the full path it took, and the realized 0/1 outcome — so convergence is actually measurable instead of hand-waved.

The measurement: how far is price from truth, as a function of time-to-resolution?

For every resolved market, line up each snapshot by hours-to-resolution, then measure the mean absolute error between price and the eventual outcome in each time bucket:

import pandas as pd

# df: rows of (market_id, ts, price_yes, resolved_outcome[0/1], resolve_ts)
df["hrs_to_res"] = (df.resolve_ts - df.ts).dt.total_seconds() / 3600
df["abs_err"]    = (df.price_yes - df.resolved_outcome).abs()

bins = [0, 1, 6, 12, 24, 48, 96, 168, 1e9]
df["window"] = pd.cut(df.hrs_to_res, bins)

conv = df.groupby("window").agg(
    mae     = ("abs_err", "mean"),
    brier   = ("abs_err", lambda s: (s**2).mean()),  # abs_err == |p - outcome|
    n_snaps = ("abs_err", "size"),
)
print(conv)

That's the whole thing. No model, no backtest — just a descriptive stat that most people assume the shape of instead of computing.

What the curve actually shows

Error shrinks fast in the last 24h, but it is not zero at the buzzer. There's a non-trivial residual right up to resolution — and that residual is concentrated in exactly the set of markets where "the obvious favorite" flips. Those tail markets are where a naive favorite-buying strategy quietly bleeds.
The 24–48h bucket is the interesting one for a trader. Most of the information has arrived, spreads have tightened — yet realized error is still meaningfully above the last-hour bucket. That gap is the thing to interrogate: is it edge, or is it just the variance of unresolved coin-flips that look settled?

The caveats people skip (and why I'm posting the method, not a strategy)

1. Survivorship / resolution-time leakage. resolve_ts is known only ex-post. If you bucket by hours-to-resolution you're implicitly conditioning on a market that did resolve when it did — fine for a descriptive convergence curve, fatal if you turn it into a live signal without a causal "what did I know at time t" cut.

2. Liquidity ≠ midprice. The curve above uses last-trade / mid. The order book (I store depth too) tells a different story near the buzzer: getting size off at the "converged" price is often impossible, and that illiquidity is precisely where the residual error lives.

3. Resolution risk is not modeled. Some of that terminal error is genuine ambiguity or dispute, not mispricing. You can't arb a market whose resolution criteria themselves are the uncertain part.

The takeaway

I'm not claiming an edge. I'm claiming the measurement is cheap and most people eyeball it instead of computing it. If you trade these, the convergence curve plus an honest point-in-time cut is the first thing worth building — before any strategy logic.

Reproduce it

The data behind this is open:

Free API, no signup: api.protodex.io — /stats, /markets, /market/{id}, /prices, /orderbook, /categories. Enough to rebuild the convergence curve yourself.
Full historical archive (18.7M+ snapshots, depth-level order books, resolved outcomes — the version I run this analysis on): Polymarket Quant Toolkit dataset.

What's your experience with the last-48h window — real residual edge, or just liquidity-trapped variance? I'd genuinely like to compare notes in the comments.

When Polymarket says 70%, does it happen 70%? Why price-only data can't answer that — and what it can

manja316 — Tue, 16 Jun 2026 10:51:24 +0000

Correction (2026-06-27): An earlier version of this post implied the dataset ships with per-market 0/1 resolution labels and that I had measured calibration directly. That was wrong, and I'd rather fix it loudly than quietly. This is a price dataset — it does not contain settled outcome labels. True calibration needs realized outcomes you have to join in from an external source. Below is the honest version: what price-only data can tell you, the legit proxy it gives you, and exactly what you'd have to add to score real calibration.

If you trade, model, or just read prediction markets, there's one question that decides whether the price means anything: when the market says 70%, does the thing actually happen about 70% of the time?

That's calibration, and it's the single most decision-relevant property of any probabilistic forecaster. A market can be liquid, popular, and heavily traded and still be systematically wrong in a way that's invisible until you score it against what actually resolved.

Here's the honest catch I have to lead with: measuring calibration needs two things — a dense price history AND the realized outcome of each market. I have the first cleanly. The second is not in this dataset, and any vendor (me included) who shows you a "calibration curve" derived from prices alone is measuring the market against itself, not against reality.

The dataset (what's actually in it)

Since late March I've logged Polymarket every 15 minutes. The frozen export holds:

22,410 markets
18,611,636 price snapshots (≈831 per market with a series)
1,856,388 order-book snapshots
15-minute cadence, 92 continuous days (2026-03-28 → 2026-06-28)
Each row: market id, timestamp, yes-price, plus volume / liquidity / best bid-ask features

What it does not carry: a per-market settled 0/1 outcome. SELECT COUNT(*) FROM markets WHERE resolved=1 returns 0 for all 22,410 markets in the export — these are price paths, not graded results. So you cannot, from this file alone, compute "of the times the price sat at 70%, how often did the event happen." You need the resolution labels, and those live outside the price feed (Polymarket's resolution / the on-chain settlement).

What price-only data CAN give you: the convergence proxy

You don't get true calibration, but you do get a useful, honest proxy. Of the 7,101 markets that ended inside the window and still carry a terminal price series, 6,836 (96.3%) closed decisively — last yes-price ≥ 0.95 or ≤ 0.05. Only 265 (3.7%) were still mushy in the middle at the end. (Denominator matters: measured across **all 19,584* markets whose end-date fell in the window — many with no live terminal quote — the decisive share drops sharply, so always state which denominator you mean.)*

That terminal price is a noisy stand-in for the outcome: ~96.3% of the time the market made up its mind hard enough that "did it close near 1?" is a defensible label. It's not ground truth (the 3.7% ambiguous tail and any post-close revision are exactly where it breaks), but it lets you study the shape of price convergence — how and when a market sharpens — which is genuinely informative on its own.

The real calibration measurement (what you'd add)

The classic check is a reliability diagram, and the method is fine — it just needs labels you join in:

Get realized outcomes for resolved markets from an external resolution source (the price feed won't give them to you).
Bin every historical price into deciles (0–10%, 10–20%, … 90–100%).
For each bin, compute the empirical resolution rate — of all the times the price sat in that bin, how often did the event actually happen?
Plot empirical rate vs. stated price. Perfect calibration is the 45° diagonal.

import requests, pandas as pd

BASE = "https://api.protodex.io"   # free price API, no signup

# Prices come from the dataset/API. LABELS DO NOT — you must supply them.
# `resolutions` here is an external {market_id: 0/1} you join in yourself
# (Polymarket resolution / on-chain settlement). It is NOT in this price feed.
resolutions = load_external_resolution_labels()   # <-- the part you provide

rows = []
for market_id, label in resolutions.items():
    prices = requests.get(f"{BASE}/prices", params={"market_id": market_id}).json()
    for p in prices:
        rows.append((p["yes_price"], label))

df = pd.DataFrame(rows, columns=["price", "outcome"])
df["bin"] = (df["price"] * 10).clip(0, 9).astype(int)
reliability = df.groupby("bin")["outcome"].mean()   # empirical rate per decile
print(reliability)   # compare each row to its bin midpoint -> the diagonal

If you only have the price feed and substitute the convergence proxy for label, be explicit that you're scoring the market against its own terminal price — a different, weaker claim than calibration against reality.

What to look for (and the trap)

The literature on real-money markets has a well-documented signature: the favorite–longshot bias — longshots tend to be overpriced, heavy favorites slightly underpriced. But you cannot confirm it from prices alone. It's a statement about realized outcomes vs. price, so it lives or dies on the labels you join in. Same for any Brier-score-over-time trajectory: great question, needs ground truth.

Honest caveats

No resolution labels in the dataset. The single biggest limit — restated because it's the one most listings hide.
Convergence ≠ truth. The 96.3% terminal-decisiveness proxy mislabels the ambiguous tail and ignores any post-close correction.
Survivorship / selection. Ended markets ≠ all markets; conditioning on resolution biases some analyses.
Mid ≠ executable price. Fees and spread are real the moment you make a trading claim.
86 days is one window, not multiple regimes. Treat any "markets are calibrated" conclusion as provisional.

Get the data

The free read-only API reproduces the price side with no signup:

https://api.protodex.io — endpoints /stats, /markets, /market/{id}, /prices, /orderbook, /categories.

If you'd rather not page the API market-by-market and want the full 18.6M-snapshot, 92-day price history as one indexed SQLite file for offline work, the one-time archive is here: Polymarket Historical Price Dataset. It's price history, honestly scoped — bring your own resolution labels if calibration is the goal.

I'd genuinely like to know how others source clean resolution labels for prediction-market calibration work — that's the open edge here. If you've benchmarked a market's calibration, where did your ground truth come from?

How to backtest a Polymarket strategy with free 15-minute historical data

manja316 — Wed, 10 Jun 2026 22:51:26 +0000

Most people backtest prediction-market strategies wrong, and it's not their fault — the data to do it right is annoying to assemble. You need time series per contract (not just the final resolution), aligned to a clock, with the resolution label attached so you know who won. Polymarket's API gives you the live order book, but the moment a market resolves, that history is gone from where most people look.

So here's a clean recipe. Free data, ~40 lines of pandas, and the caveats that separate a backtest you can trust from one that lies to you.

0. The data

You need a price history per market at a fixed interval. I've been archiving Polymarket at a 15-minute cadence since late March — 18.6M+ price snapshots across 22,410 markets, 92 days of history as of this writing. The live market index is free to browse at protodex.io.

One honest caveat before you write a line of code: this is price history, not officially-labeled outcomes. Polymarket's public API does not hand you a clean post-hoc resolved_yes flag, so you derive the outcome from where the price ends up — and roughly 5% of ended markets never converge decisively, so they stay ambiguous and must be dropped, not guessed. That limitation is the single most important thing to understand before you trust any backtest built on this (or any price-only prediction-market dataset). I measured it: 94.6% of ended markets close at ≥0.95 or ≤0.05; the rest are coin-flips you cannot score.

You can roll your own collector against the Polymarket CLOB + Gamma APIs (the snapshot loop is maybe 60 lines), or skip the three-month wait and grab the parquet bundle — link at the end. Either way, the analysis below is identical.

1. Load it

Assume a parquet with columns market_id, timestamp, and price (the YES probability, 0–1). There is no ground-truth resolved_yes column — you derive a proxy outcome from each market's terminal price, and you throw away anything that didn't converge.

import pandas as pd

df = pd.read_parquet("polymarket_history.parquet")
df["timestamp"] = pd.to_datetime(df["timestamp"], utc=True)
df = df.sort_values(["market_id", "timestamp"])

# Derive a PROXY outcome from the terminal price — there is no official label.
# End >= 0.95 -> treat YES, <= 0.05 -> treat NO; anything in between never
# converged, so it's unscoreable and gets dropped (NOT guessed at 0.5).
last = df.groupby("market_id").tail(1)[["market_id", "price"]]
last = last.rename(columns={"price": "final_price"})
last["resolved_yes"] = pd.NA
last.loc[last["final_price"] >= 0.95, "resolved_yes"] = 1
last.loc[last["final_price"] <= 0.05, "resolved_yes"] = 0
df = df.merge(last[["market_id", "resolved_yes"]], on="market_id", how="left")

# Keep only markets that converged decisively — you can't score a coin-flip
resolved = df.dropna(subset=["resolved_yes"]).copy()
print(resolved["market_id"].nunique(), "decisively-converged markets")

Be honest with yourself about what this proxy is: a circularity check is worth doing here — you're using the late price to label the outcome and the early price as the signal, so as long as your entry snapshot is well before convergence (see §2) the label isn't leaking your signal. But it is still a convergence proxy, not an official settlement, and it silently excludes the ~5% messy markets. State that in any result you publish.

2. The strategy: fade the longshots

The favorite-longshot bias is the most-documented inefficiency in betting markets: longshots (low-probability contracts) are systematically over-priced, favorites slightly under-priced. Translated to Polymarket: a contract trading at 8¢ resolves YES less than 8% of the time on average. So a naive edge is short the longshots (buy NO when YES is cheap).

Let's test whether that holds in the snapshots. Take each market's price in a chosen band at a fixed point before resolution, and compare to the realized outcome.

# For each market, grab the last snapshot at least 24h before its final timestamp
def snapshot_24h_out(g):
    end = g["timestamp"].max()
    cutoff = end - pd.Timedelta(hours=24)
    pre = g[g["timestamp"] <= cutoff]
    return pre.iloc[-1] if len(pre) else None

picks = (resolved.groupby("market_id", group_keys=False)
         .apply(snapshot_24h_out)
         .dropna())

# Longshot band: YES priced 2–15¢
band = picks[(picks["price"] >= 0.02) & (picks["price"] <= 0.15)]
implied = band["price"].mean()          # what the market said
realized = band["resolved_yes"].mean()  # what actually happened
print(f"implied YES {implied:.3f} vs realized YES {realized:.3f}")

If realized < implied, the longshots were overpriced — the bias is present and shorting them has positive expectancy before costs.

3. The part most backtests skip — costs and survivorship

A backtest without these three corrections is marketing, not research.

1. Spread + fees. You don't trade at the mid. On thin Polymarket longshots the bid/ask can be 2–4¢ wide. A 1¢ edge on a 6¢ contract evaporates the moment you cross a 3¢ spread. Always subtract a realistic fill cost from the implied price before scoring P&L.

2. Survivorship / resolution timing. If your archive only kept markets that resolved cleanly, you've dropped the messy ones (extended, disputed, voided) — and those aren't random. Score against every market that hit your entry filter, not just the tidy winners.

3. Liquidity ceiling. A 20¢ edge on a market with $300 of depth is a $60 edge, not a strategy. Weight every backtested position by the order-book depth at entry, or you'll "discover" an edge you can't actually fill. (This is exactly why snapshot data needs the order book, not just last-price — depth is the difference between a paper edge and a real one.)

Do these three and the favorite-longshot edge usually survives — but smaller than the raw number, and only in the deeper markets. That gap is the finding.

4. Why 15-minute snapshots specifically

Tick data is overkill for this and a nightmare to store; daily closes are too coarse to catch the late convergence where most of the price action lives (markets snap toward 0/1 in the final hours). 15 minutes is the sweet spot: dense enough to study convergence and intraday moves, sparse enough that three months fits in a few hundred MB of parquet.

If you'd rather not run a collector for three months before you can test a single idea, I've packaged the full archive — parquet + CSV with resolution labels included, plus a ready-to-run Python notebook that runs the exact favorite-longshot backtest above, as a one-time purchase:

👉 Polymarket Quant Toolkit — 18.6M-snapshot dataset + analysis notebook ($49)

It's price history with the convergence-proxy approach above already wired into the notebook — no over-promised resolution labels, just the honest version of the backtest. And the live market index is free to poke at first: protodex.io.

What strategy would you test first — favorite-longshot, late-convergence momentum, or cross-market arbitrage? Drop it in the comments and I'll point you at the columns you'd need.

I archived 19.1M Polymarket price snapshots. Three things the data shows that the order book hides.

manja316 — Tue, 09 Jun 2026 10:51:30 +0000

If you only ever look at a prediction market through its live order book, you see one number: the current price. That number is the market's best guess at a probability — "73¢ = ~73% YES." Useful, but it throws away everything about how the market got there.

I've been archiving Polymarket order-book snapshots every 15 minutes for about eleven weeks. The dataset is now 19.1M+ price points across ~19,000 markets, most of them resolved (so we know the ground-truth outcome). Looking at prediction markets in bulk, instead of one at a time, surfaces structure that's invisible tick-by-tick. Three findings worth your time.

1. Favorite-longshot bias is real here too

The oldest result in betting-market research is the favorite-longshot bias: longshots are systematically overpriced and heavy favorites are systematically underpriced. People overpay for the lottery-ticket thrill of a 5¢ "it could happen" and underpay for the boring 95¢ near-certainty.

It shows up cleanly when you bucket thousands of resolved markets by their pre-resolution price and compare each bucket's implied probability to its actual hit rate:

Contracts trading in the 2–10¢ band resolve YES less often than their price implies — the longshot tax.
Contracts in the 90–98¢ band resolve YES slightly more often than their price implies — favorites are a hair cheap.

You cannot see this in one market. You can only see it across a thousand resolved ones — which is exactly what a bulk archive is for.

2. Convergence is late and lumpy, not smooth

Intuition says a market should drift smoothly toward 0 or 1 as resolution nears. The data says otherwise: most markets sit in a noisy band for the majority of their life and then convergence happens in a short burst near the resolving event — a debate, an earnings print, an election night.

The practical implication: time-to-resolution matters more than price level when you're reasoning about how much a contract can still move. A 60¢ market with three weeks left and a 60¢ market with three hours left are completely different objects, and a single live quote can't tell them apart. A timestamped history can.

3. Volume ≠ movement

The markets with the biggest 24h price moves are usually not the ones with the biggest volume. High-volume markets are liquid and efficient — lots of participants, tight spreads, slow to move. The violent re-pricings happen in thinner markets where a single piece of news has nobody on the other side to absorb it.

If you screen for "interesting" markets by volume, you'll mostly find markets that have already finished being interesting. Screening by realized movement relative to liquidity finds the ones still in motion. (I built a free screener around exactly this idea — separate post.)

How to reproduce this yourself

Every claim above is just pandas over a long-format table:

import pandas as pd

df = pd.read_parquet("polymarket_snapshots.parquet")
# columns: market_id, timestamp, price_yes, volume, resolved_outcome

# Favorite-longshot calibration: take each market's last pre-resolution price,
# bucket it, and compare implied probability to the actual YES rate.
last = (df.dropna(subset=["resolved_outcome"])
          .sort_values("timestamp")
          .groupby("market_id").tail(1))
last["bucket"] = (last["price_yes"] * 10).round() / 10
calib = last.groupby("bucket").agg(
    implied=("price_yes", "mean"),
    actual=("resolved_outcome", "mean"),
    n=("market_id", "size"))
print(calib)  # actual < implied for low buckets = longshot bias

The hard part isn't the analysis — it's getting clean, timestamped, resolution-labeled history out of an API that's built for live trading, not bulk export. That collection problem is the whole reason this dataset exists.

The dataset

If you want the raw archive instead of building the collector yourself, it's here:

👉 Polymarket Full Dataset — 18M+ price snapshots, 22,000+ markets

Parquet + CSV, 15-minute snapshot cadence across the full market history, one-time purchase. If you just want to poke at the live numbers first, the market index is free at protodex.io.

What would you check first with 19.1M+ labeled prediction-market snapshots? I'll run reader requests against the archive and post the results.