DEV Community: Pavel Kazantsev 💎

I Went Looking for the Dead Internet in My RAG. I Found a Bad Experiment.

Pavel Kazantsev 💎 — Tue, 03 Nov 2026 08:36:33 +0000

I removed duplicate sources from a RAG benchmark and watched held-out accuracy fall from 10 of 20 tasks to 5 of 20. The tempting conclusion was that source deduplication made retrieval worse.

The audit found another change hiding in the comparison. Lexical top-k supplied roughly five documents; the collapse policy often supplied one or two. The run changed source diversity and the evidence budget together, so the accuracy drop could not identify a provenance effect.

With a fixed-budget control, root-aware retrieval recovered the primary record and accuracy returned to 10 of 20. It did not improve further. That correction became the point of this investigation: retrieval correctness and answer correctness are different measurements, and an apparent result can disappear when the control is fair.

TL;DR

Removing duplicate sources appeared to halve accuracy, but the comparison also reduced the evidence budget.

Under the corrected control, root-aware retrieval recovered primary records without improving answer accuracy.

A 15-case real-world pilot validated the provenance pipeline, not retrieval effectiveness. V2 will test adversarial cases.

Part 1 corrects a misleading synthetic result and validates a provenance-capture and evaluation pipeline on real sources. Part 2 will test whether provenance-aware retrieval changes answers when the evidence actually conflicts.

The part of the "dead internet" idea worth testing

The phrase "dead internet" is associated with a 2021 forum post, Dead Internet Theory: Most of the Internet Is Fake. Contemporary reporting traces the popular term to that thread. Its broad version bundles together bots, manufactured engagement, algorithms, and a claim that real human activity has become marginal.

That claim is too broad for this experiment. A traffic statistic cannot establish who wrote a page, whether a claim is true, or whether people have stopped participating online.

For RAG, a systems problem remains. Imperva's 2025 report estimated that automation represented 51% of the web traffic it observed in 2024. The category includes useful crawlers and malicious bots, so it cannot count a "fake internet."

For retrieval, a smaller question is enough: how many independent source roots support the evidence in context?

Repeated documents can reach an index through a documented syndication chain, a press release, a scrape, a summary, or a generated rewrite. The causes differ. The failure is the same when repetition reaches the model as apparent corroboration.

The wider theory also tends to mix four questions:

Did a human or an automated client make the request?
Was the page written by a person, copied, or generated?
Where did the claim originate?
Does the claim have enough support to warrant an answer?

They require different evidence. This experiment addresses the third question, provenance. It does not diagnose the web as a whole.

What I mean by a source root

A source root is an information lineage, not a hostname. One root can include a public primary record and several documented syndications. One publisher can contain several independent roots.

I assign a root only when the relationship has evidence: a canonical link, a syndication notice, an explicit citation, or a documented editorial relationship. Near-identical phrasing is useful for discovery, but it does not prove copying. Treating a similarity score as provenance would create a confident-looking field from an unverified assumption.

The system

The implementation keeps evidence capture, retrieval, generation, and scoring separate. Otherwise one provenance_aware switch can quietly change the candidate pool, context length, answer rule, and source metadata at once.

Evidence capture saves the public and canonical URLs, timestamp, HTTP status, extracted text, hash, and documented relation. Retrieval selects documents under a defined budget. Generation receives document text, IDs, roots, and roles, but never an expected stance or fixture label. Evaluation checks the returned stance, citation IDs, and retrieval trace.

candidates = lexical_retrieve(question, pool=50)
context = root_aware_fill(candidates, k=5)
response = generate(question, context, fixed_prompt, seed)
audit(response, context, expected_stance)

The pool=50 is important. Collapsing sources only after a top-five cutoff cannot recover a primary record that was ranked sixth. Candidate grouping has to happen before the context budget is spent.

The result that fooled me

I built a 60-task synthetic fixture with known source roots. The first comparison looked clean: lexical top-k got 10 of 20 held-out tasks right; a root-collapse policy got 5 of 20.

The comparison was not clean. Root collapse sometimes returned one or two documents where lexical top-k returned five. Both variables changed, so the first run cannot attribute its regression to source grouping alone.

Held-out synthetic control, k = 5	Correct	Mean documents	Primary-record recall
Lexical top-k	10 / 20	4.25	20%
Post-top-k collapse	5 / 20	1.15	20%
Root-aware fill	10 / 20	4.25	100%
Conflict-preserving fill	10 / 20	4.25	100%

The original collapse policy spent fewer evidence slots. Root-aware fill chooses roots from a larger pool, then restores the intended document budget.

With the budget fixed, root-aware fill recovered the primary record and retained the deterministic accuracy of lexical top-k. It did not improve accuracy. The remaining slots still let the answer rule count repeated stances.

The original drop measured a policy that also shrank the prompt. It did not isolate the effect of source grouping.

Why keep the synthetic fixture?

The fixture isolates a retrieval mechanism that is hard to control on public pages. It contains 60 tasks across duplicate dilution, true repetition, conflicting roots, and satire stripped of its context. Each category has ten development tasks and five held-out tasks.

The publishers and documents are fictional because the fixture needs declared roots and known answers without claiming to measure the open web. The first experiment uses a deterministic stance aggregator, making the intervention inspectable. It also exposes a limitation: recovering a primary record does not automatically improve the final answer if the aggregator still counts every retained document equally.

A concrete held-out fixture trace. It demonstrates a retrieval change, not a real-world effectiveness result.

Fixtures cannot show whether the pipeline works on public pages, so I collected a separate real-data pilot.

The 15-case real pilot

The pilot contains 15 NASA-to-Phys.org syndication chains. Every accepted case has:

a public NASA record;
a Phys.org page whose snapshot explicitly says Provided by NASA;
timestamped HTML, extracted text, a canonical URL, HTTP status, and a SHA-256 content hash;
a graph edge supported by the documented syndication marker.

Similar language never counted as proof of dependence. If the source relationship was undocumented, the case did not enter the slice.

Gemma 3 4B and Qwen 2.5 7B independently labelled document roles. They disagreed enough that I did not use their votes as ground truth. Final roles came from the explicit record type and syndication marker; model annotations remain in the corpus as machine annotations. This is a machine-adjudicated pilot, not a human-verified dataset.

I ran both models on the same 15 questions under four conditions:

primary-only;
primary plus the documented dependent copy;
root-aware;
conflict-preserving root-aware.

The prompt, seed, decoding settings, evidence ordering, retrieval trace, and raw responses were saved for every run. Neither model received the expected answer or a stance label in its prompt.

What V1 actually found

15 source chains · 2 local models · 120 retained responses

Gemma 3 4B and Qwen 2.5 7B each returned the correct stance for all 15 cases in every condition. Citation-ID validity changed when the dependent copy was present. The slice has one root and positive support in every case, so it validates provenance capture and evaluation plumbing. It does not test whether root-aware retrieval prevents a real answer error.

Both models answered all 15 questions correctly in every condition.

Every accepted chain has one source root and supports one positive claim, so root-aware retrieval reduces to primary-only. The slice has no false assertion or contradictory root to resolve.

The citation trace moved:

Model	Primary-only valid citation IDs	With dependent copy
Gemma 3 4B	3 / 15	6 / 15
Qwen 2.5 7B	12 / 15	10 / 15

Those are exploratory counts, not a model ranking. Citation-ID validity only says whether the returned ID was present in supplied context. It does not establish that the cited document supports the answer.

The V1 confound, stated plainly

There is a second limit on the V1 comparison. Adding the dependent page changes the context from one document to two, and Ollama's recorded prompt-token count changes in every model-case pair.

The citation results are descriptive because the comparison cannot isolate source dependence from the extra context. V2 will block an evaluation unless each paired case has equal document and prompt-token budgets.

The real corpus also has a ceiling effect. Primary-reference recall is 100% by construction in the one-root conditions, and root concentration is 1.0 because every selected context belongs to one lineage. These measurements confirm that the policy did what it was asked to do. They also explain why the pilot cannot support an accuracy claim.

Metrics worth keeping separate

Correctness is useful, but it can hide the failure that changed. A model may preserve a correct stance while losing the primary record, naming a citation ID that was never in context, or becoming impossible to audit after a page changes.

I would report at least these outcomes for a larger corpus:

answer stance;
primary-reference recall;
root concentration;
citation validity;
contradiction coverage;
false-claim adoption.

A single combined score would hide which part failed. The unit for a future comparison is the source case, not the generated response. Paired accuracy deltas, bootstrap intervals, and an exact sign test make sense only after enough independent held-out cases survive the admission gate.

For production, I would retain this trace beside every retrieved chunk:

document_id · canonical_url · publisher · publication_date
retrieved_at · content_hash · source_root · source_role
claim_relation · dependency_evidence

Candidate selection, source weighting, and conflict handling should stay separate. A source root can come from a documented origin or explicit citation evidence. It cannot come from cosine similarity alone.

What a reader can verify today

The public release supports artifact verification, not a byte-for-byte end-to-end rerun. It contains public URLs, hashes, annotations, the run manifest, the paired-budget audit, and 120 raw model responses. It deliberately excludes third-party HTML and extracted page text.

git clone https://github.com/icesurf666/rag-provenance-lab.git
cd rag-provenance-lab
python3 -m venv .venv && . .venv/bin/activate
pip install -e '.[test]' && PYTHONPATH=src pytest -q

The reproducibility guide explains that boundary. The paired-budget audit records all 30 unequal model-case pairs; the raw responses, run manifest, and recorded results are public.

A real retrieval failure outside this pilot

In May 2024, Google AI Overviews advised users to add glue to pizza sauce. Associated Press reported that the answer traced back to a satirical Reddit comment, and Google later published its response.

The incident shows a retrieval failure: the words reached the system while the sentence that explained their status did not. It cannot estimate a dead web or the prevalence of AI-generated content.

What comes next: V2 — when provenance changes the answer

V1 is complete as a pipeline validation. V2 will test the effectiveness question it leaves open: can source provenance prevent a model from accepting a repeated false claim when a correct primary record is available?

V1 and V2 answer different questions. This diagram describes a planned V2 protocol, not results.

The next corpus will include corrections alongside persistent stale copies, satire retold as fact, contradictory evidence, genuinely independent corroboration, and conflicts within one source lineage. The target is 40 to 60 quality-gated real cases with documented provenance, snapshots, hashes, independent machine annotations, and adjudication.

Every case that passes the gate will stay in the evaluation set. V2 will include cases where both policies are correct, both are wrong, lexical retrieval wins, and root-aware retrieval wins. Selecting only later success stories would turn the corpus into a demonstration instead of an experiment.

Gemma and Qwen will run pre-specified paired conditions with equal document and prompt-token budgets. Document-order permutations will be assigned before outcomes are inspected. The result may show an improvement, a regression, or no material effect.

Until V2 exists, a claim that root-aware retrieval improves real-world answer accuracy would be premature.

The RAG Provenance Lab repository contains the public V1 code, artifact-verification materials, model outputs, run manifests, and the protocol. It distinguishes artifact verification from end-to-end reproduction: third-party page snapshots and extracted text are not distributed in the public release.

I Tested 7 Crypto Models. The Best Signal Failed the Cost Test.

Pavel Kazantsev 💎 — Fri, 30 Oct 2026 09:17:47 +0000

Huber regression had the strongest predictive ranking in my crypto model comparison: mean out-of-sample Spearman IC +0.055. Its selected trades nevertheless averaged −7.74 basis points after modeled costs.

The tree model with the least negative result was an XGBoost configuration at −1.73 bps. Its uncertainty interval crossed zero. LightGBM was close behind at −1.93 bps. Neither result establishes a profitable strategy.

Those numbers came only after an uncomfortable discovery: the first version of the backtest had a boundary leak and treated gaps in price history as consecutive bars. We fixed both and reran the comparison.

Research status: this is exploratory, retrospective research on history already examined in earlier work. Prices are OHLC proxies, costs are assumed, the universe was selected retrospectively, and there is no executable portfolio P&L. It is not an independent holdout or a live-trading result.

Signal is not a strategy

The question began with three features: 1-, 5-, and 20-bar log returns on 15-minute Bybit perpetual candles. Could a model use them to rank the next return? Could a trading rule survive costs?

An information coefficient, or IC, describes rank correlation between a score and the future return. To assess a trading rule, we also need to know how often it trades, what each trade earns and what execution costs.

Here IC is one Spearman correlation pooled across available symbol-bar observations per fold, then averaged equally across folds. It is not a daily cross-sectional IC; pooled correlations can reflect differences between symbols as well as within-symbol ranking. We did not estimate an IC confidence interval in this run.

Why can the rankings differ? IC uses all scored observations to ask about ordering. The trading result uses only observations beyond a model-specific threshold, assigns a direction and subtracts costs. A model can rank the whole panel well yet select a less rewarding set of extreme scores. The two statistics also weight observations differently, so this experiment does not isolate a single cause of the gap.

An earlier OLS study reported 4,218 trades and +5.45 bps mean net using a fixed 12 bps round-trip assumption and a fixed ±12 bps signal threshold. Its bootstrap implementation also had a bug: repeated sampled days were discarded. The reported interval could not be relied on. A positive number did not answer whether another model, evaluated under the same rules, would do better.

Seven candidates, one historical protocol

The original comparison used reversal, OLS, Ridge, Huber and constrained LightGBM. I later added XGBoost and CatBoost as a post-hoc extension. The seven configurations share the same time splits, universe, features available to learned models, signal-selection rule and modeled costs. The reversal baseline uses only the one-bar feature.

Data: 16 Bybit USDT perpetuals; 15-minute bars; symbols enter as data becomes available. Selecting this universe retrospectively creates survivorship risk.
Evaluation: six expanding walk-forward folds, 2022 H1 through 2024 H2; training starts in 2021.
Target: next-bar close-to-close return in bps. The model and scaler are fitted on training data only.
Signal: a long or short when the score exceeds the 99th percentile of its own training score magnitude in either direction.
Economics: next-bar open-to-close return, less modeled round-trip costs of 13, 15 or 19 bps by liquidity tier.

The cost assumption is 5.5 bps taker fee per side plus 1, 2 or 4 bps assumed slippage per side. These are inputs to the historical proxy, not measured historical fees or fills.

The threshold helps make differently scaled scores comparable, but it does not force models to trade equally often in the evaluation period. Models may select different symbols and dates, so their mean per-trade returns are not returns on an equally allocated portfolio.

The tree configurations are fixed and untuned here; they use different algorithms and regularization. This is a comparison of seven configurations, not a universal ranking of model families.

The later boosters each use 200 trees, depth 4 and learning rate 0.03. XGBoost uses min_child_weight=2000, L1=1 and L2=10; CatBoost uses L2=10. Ridge uses a strong fixed alpha=1,000,000 on standardized features, so its effective regularization changes relative to the growing training sample. A different tuning budget could change the ranking.

Five equations and one timing problem

For bar close C[t] and open O[t]:

features:    x[t, k] = log(C[t] / C[t-k]), k ∈ {1, 5, 20}
train label: y[t]    = 10,000 × (C[t+1] / C[t] − 1)
threshold:   q       = 99th percentile of |training score|
signal:      s[t]    = +1 if score > q; −1 if score < −q; else 0
trade net:   p[t]    = s[t] × 10,000 × (C[t+1] / O[t+1] − 1) − cost

The last equation applies to selected trades only. A basis point is 0.01% of position notional.

The model learns close-to-close returns, while the economic proxy measures the next candle's open-to-close move. The next open can occur before a system reacts to a confirmed close, so the OHLC price does not establish that a trade could have filled there.

The timing diagram is illustrative, not a measured latency distribution. A later bid/ask observation does not establish that an order filled.

The corrected results

Model	Mean fold IC	Trade-weighted net / trade	95% block interval, bps	Trades
XGBoost*	+0.012	−1.73 bps	[−7.41, +3.63]	10,885
LightGBM	+0.012	−1.93 bps	[−7.64, +3.65]	11,006
CatBoost*	+0.011	−5.73 bps	[−11.54, +0.83]	11,341
Ridge	+0.044	−7.14 bps	[−14.34, −0.76]	12,685
Huber	+0.055	−7.74 bps	[−15.34, −0.88]	12,487
OLS	+0.036	−9.82 bps	[−16.00, −4.42]	12,461
One-bar reversal	+0.040	−12.55 bps	[−18.39, −6.48]	11,497

XGBoost and CatBoost were added after the first run. “Out-of-sample” means out of each fold's fitting window, not a clean final holdout. IC values are descriptive means, not claims of statistical significance.

Net is weighted by the number of trades. Averaging the six fold-level means instead gives LightGBM −3.09 bps: another valid summary, but a different weighting. Neither measure is portfolio return.

Intervals use 2,000 resamples of 10-calendar-day blocks. All trades within a sampled day stay together; the same days are sampled for each model. Fitted models and selected trades stay fixed during resampling; the models are not refitted. The intervals do not account for model selection or multiple comparisons. The XGBoost and LightGBM intervals crossing zero neither prove profit nor rule it out. The 0.20 bps difference between their point estimates is not a basis for declaring a winner.

Look inside the average

LightGBM's fold means range from −12.49 to +4.29 bps. Its three positive half-years are small beside its two largest negative ones. All seven configurations lose in 2022 H2 under these assumptions. This is regime sensitivity, not a clean prediction of what happens next.

The day-block bootstrap gives an exploratory LightGBM-minus-OLS interval of [+3.71, +12.90] bps per trade. The models select different trades, the research is post-selection, and this interval is not adjusted for testing multiple candidates. It is not proof that LightGBM generally beats OLS.

What happens when positions share capital?

The table weights each trade equally. A model that selects many simultaneous markets may look different when those positions have to share capital. In a separate post-hoc portfolio proxy, we divide 1× gross notional equally among selected symbols on each active bar. The proxy enters at the next open, exits at the following close and pays the same modeled round-trip costs. It holds cash when there is no valid selected signal.

With the original in-sample threshold, the mean daily additive proxy is −53.68 bps for LightGBM and −54.62 bps for XGBoost. LightGBM has the less negative number under this aggregation, reversing their per-trade ordering. The paired 10-day block-bootstrap interval for XGBoost minus LightGBM is [−14.95, +11.59] bps per day: it includes zero.

Does the threshold matter?

The original 99th percentile comes from predictions on the same observations used to fit each model. For a sensitivity check, we fit another model on the first 80% of each outer training window, purge labels crossing the boundary, and calculate the percentile on the remaining 20%. We then refit on the full outer training window before evaluating the next fold. Refitting can change score scale; this is one post-hoc chronological split, not a tuned or independently validated rule.

With the training-holdout threshold, the mean daily additive proxy changes to −80.98 bps for LightGBM and −79.67 bps for XGBoost. Their ordering switches again. The paired interval for XGBoost minus LightGBM is [−12.15, +13.46] bps per day, again including zero. Both intervals use 2,000 resamples and common 10-day blocks. Models are not refitted, the historical period was already examined, and the intervals are not adjusted for multiple comparisons.

These numbers sum bar-level bps by next-bar entry day and divide by all calendar days, including days in cash. Notional resets to 1× whenever a signal is active; returns are not compounded. The proxy omits funding, impact, order-book depth, margin, liquidation and changing equity. Missing or invalid symbol-bars are treated as unavailable signals, and equal gross notional is not equal risk. This is not an executable portfolio backtest.

The cost cliff

For the original five candidates, I held signals fixed and scaled the assumed fee plus slippage:

LightGBM's selected trades averaged +16.15 bps gross proxy and 18.08 bps modeled cost. At half the assumed cost, net is +7.11 bps; at baseline, −1.93; at twice the cost, −20.01. The break-even multiplier is about 16.15 / 18.08 = 0.89.

That number is an arithmetic result for this historical trade set; it does not establish an obtainable fee quote. The sensitivity scales both fees and slippage, keeps the trades fixed, and does not model queueing, market impact or latency. XGBoost and CatBoost are not included in this cost chart.

To put the units in context: on a hypothetical $10,000 position, one basis point is $1. The LightGBM averages correspond to roughly $16.15 gross proxy, $18.08 assumed costs and −$1.93 net per selected trade. These are scaled averages, not cash flows from a funded account; actual position sizes, simultaneous trades and fills are not modeled in that calculation.

With 5- and 20-day bootstrap blocks, LightGBM's intervals still cross zero: approximately [−7.86, +3.83] and [−7.88, +3.80] bps. Other dependence assumptions could produce different intervals.

Two bugs that changed the experiment

The train/test boundary. The first script selected training rows by timestamp but constructed their targets using the next row. A final training row could therefore use a price from the evaluation period. The corrected split requires each training label to finish by the evaluation boundary. A regression test changes future prices and verifies that training data does not change.

# Bar timestamps denote bar opens; label_end is the next bar's close.
train = panel[(panel.ts >= train_start) & (panel.label_end <= eval_start)]
eval_ = panel[(panel.ts >= eval_start) & (panel.label_end <= eval_end)]

Compare the end of the label with the evaluation boundary, not just the timestamp on the feature row. This rule matches this next-bar target; longer holding periods or overlapping labels need their own purge.

Missing bars. A row shift is not necessarily a 15-minute shift. The corrected run requires 20 consecutive historical intervals and one consecutive future interval. Across loaded history it detected 1,631 gaps and excluded 34,587 rows, including warm-up and incomplete outcomes. The downloadable result lists exclusions and full-file hashes by symbol.

The old +5.45 bps OLS estimate and this rerun changed both the threshold and cost convention. We cannot attribute their difference to costs alone. We can say that the earlier positive result did not carry over to the new decision and cost contract.

What would change my mind?

An executable-price simulation with point-in-time universe rules, realistic order sizing and a portfolio risk model could change this assessment. So could a genuinely new period.

If you are building a similar comparison, keep three questions separate:

Can the score rank returns? Compare models on the same untouched time windows, and state whether the IC is pooled or cross-sectional.
Does the decision rule survive costs? Freeze the threshold, timing, sizing and fee assumptions; report gross and net results together.
Would the portfolio have been executable? Check simultaneous exposure, missing data, available liquidity and actual decision-to-order timing. An OHLC open is not a fill.

This experiment answers the first two only under its stated historical assumptions. It does not establish the third.

We have frozen seven fitted candidates for a prospective shadow period from 1 November 2026 to 1 February 2027 UTC, with LightGBM versus OLS as the primary comparison. As of 30 October 2026, that period had not started. It uses the same exchange and local infrastructure, so it is independent in time, not an external audit. A separate observer will record post-decision bid/ask references with at least 500 ms delay and mark missed windows; the quotes are not guaranteed fills.

Before the prospective test, we checked public Bybit top-of-book snapshots during a short window on 29 October 2026, 14:30–18:00 UTC. The collector recorded 224 confirmed symbol-bars across 16 markets; 222 had a retained bid/ask snapshot. Median quoted spread across the snapshots was 1.06 bps. On our host clock, the median confirmed-candle receipt was 724 ms after the scheduled boundary. This is a small pre-forward diagnostic: quotes can precede a decision, local clock error affects the delay, and no order was sent. It does not measure actual execution. Diagnostic summary.

Separately, between 18:57 and 18:59 UTC on 29 October, we submitted five 0.001 BTCUSDT market-order round trips to Bybit's demo environment. In the first, simulated fills were 80,851.8 USDT on entry and 80,845.9 on exit; fees totalled 0.08893374 USDT. All five closed with a loss of between 0.08730960 and 0.10012708 USDT including fees and the price move while holding each position. The anonymized records include pre-entry quotes and execution timestamps. This short, one-symbol demo sample contains no model signals: it is not a representative cost estimate, evidence of live fills, or validation of the historical backtest.

A separate demo-only pilot is configured to attempt at most one 0.001 BTCUSDT round trip per primary model on eligible forward signals, sequentially on a shared account. It can stop for manual review, and connectivity failures can leave a demo position open. Signals can be missed, and the two trades are neither paired nor a statistically useful comparison. Demo fills are simulated; they cannot establish live execution. The pilot does not change the primary shadow protocol.

Download the research code and results · Reproduction instructions. We reran all seven models from the extracted bundle using the original files and reproduced the reported aggregates. The bundle includes a public-API downloader and a semantic fingerprint of the original candle slice, including missing timestamps. We separately verified all 140,255 BTC rows and 1 January 2024 for all 16 markets (96 bars each); the other 15 markets have not yet been independently downloaded in full. The downloader stops if the data available from Bybit do not match.

References: LightGBM paper, XGBoost paper, CatBoost paper, and scikit-learn's time-aware cross-validation example.

My test suite had 216 passing tests. It was lying to me.

Pavel Kazantsev 💎 — Mon, 26 Oct 2026 14:58:44 +0000

An audit found a nested JSON false PASS in a replay-versus-live trace comparator. The local fix is recursive and type-aware; CSV equality remains deliberately unverified without provenance.

[1] and [true] are different JSON values. My Python comparator said they were equal.

It returned:

{
  "state": "PASS",
  "deterministic_parity_verified": true,
  "findings": []
}

That result came from a tool I built to compare a historical replay with a live-style decision pipeline. The point was to find the first recorded difference in features, model scores, decisions, configuration, or input evidence. A strict PASS was supposed to be the strongest possible result.

It was wrong.

The false PASS, reduced to five lines

The audit input was deliberately boring:

baseline = {"f": [1]}
candidate = {"f": [True]}

Both values were valid JSON-like input. Both traces had matching metadata and input hashes. The public recorder wrote them, the parser read them, and the old CLI certified them.

The underlying Python behavior is legal:

[1] == [True]           # True
{"x": 1} == {"x": 1.0}  # True

My comparator was asking the wrong question. Scalars went through type-aware logic, but nested lists and dictionaries eventually reached ordinary Python equality. For application code, that can be convenient. For a parity checker, it erases a recorded type difference.

Before / after: the same trace pair

I reran the preserved pre-fix source snapshot against this trace shape. It returned a verified strict PASS:

{
  "state": "PASS",
  "deterministic_parity_verified": true,
  "findings": []
}

The public synthetic fixture now runs through the current CLI. This is the actual output:

$ hfm-parity diff --baseline examples/nested_values/baseline.jsonl \
    --candidate examples/nested_values/candidate.jsonl \
    --output parity-demo.html --json-output parity-demo.json
FAIL: mode=strict; not-verified; aligned=1; findings=1

The JSON report identifies the leaf, its classification, and the recorded values:

{
  "state": "FAIL",
  "deterministic_parity_verified": false,
  "first_divergence": {
    "event_id": "evt-001",
    "stage": "features",
    "field": "payload[0]",
    "classification": "FEATURE_TYPE_MISMATCH",
    "baseline": 1,
    "candidate": true
  }
}

The fixture, source code, and regression tests are public in HFM Parity Lab. It is synthetic by design: it proves a regression boundary, not a historical market measurement.

The pre-fix PASS was reproduced from the preserved source snapshot; the current FAIL is generated by the public synthetic fixture. The fixture demonstrates a regression boundary, not a historical market measurement.

Why 216 tests did not catch it

The candidate had 216 Python tests when the second audit supplied the nested counterexample. The scalar integer defect was fixed. Clean wheel and source installs worked. The CLI, MCP demo, Ruff, mypy, and frontend build passed.

The gap was structural: the tests covered scalar numeric semantics, but not the same semantics at every JSON leaf. More happy-path tests would not have made that contract explicit.

What the tool is intended to compare

Backtests and live pipelines often disagree for reasons that are annoyingly ordinary. A feature was defined differently. A model version changed. A timestamp uses a different clock. An event came from a different source. The final BUY/HOLD decision is usually too late in the chain to explain much.

I built a trace comparator called HFM Parity Lab around a narrower question:

Given two recorded executions, what is the first observable difference, and do we have enough evidence to say the executions are comparable?

Each JSONL trace records run metadata plus event-level fields: an event ID, stage, features, timestamps, input hash, model score, decision, and optional state hash. The engine aligns event.id + stage, then returns PASS, FAIL, NOT_COMPARABLE, or UNVERIFIABLE.

Strict mode is for two runs that claim to reproduce the same thing. Regression mode is for deliberate old-versus-new comparisons. Regression mode can report agreement, but it never certifies strict parity.

The first audit found other failures too

The first independent audit began with 87 passing tests and found nine problems. These three are the easiest to reproduce.

1. Large integers were converted to floats

The first comparison helper turned numbers into float values before comparison.

9007199254740993 != 9007199254740992

Past 2**53, IEEE-754 doubles cannot represent every integer. The initial engine could call those values equal and return a verified strict PASS.

The fix compares integers exactly. An integer and a float are a type mismatch even when their values look equal; tolerance applies only when both values are floats. The repaired scalar tests cover values far beyond the floating-point range.

2. Matching event IDs could hide a different market

Two traces might both include evt-17, while one came from BTC and the other from ETH. The initial strict comparison did not treat symbol, venue, or data_source as part of the identity contract.

That makes event alignment look more meaningful than it is.

Strict mode now returns NOT_COMPARABLE for declared identity mismatches. It also needs a data source, input hashes on aligned events, an actual output to compare, and a model version when model scores exist. Missing evidence returns UNVERIFIABLE.

There is a practical difference between "I did not see a mismatch" and "the traces contain enough evidence to verify parity." A useful tool needs both answers.

3. The MCP output limit was per value, not per response

The first MCP formatter shortened nested values one at a time, then built the response. That did not constrain the total payload. The remediation report recorded a roughly 10 MB input yielding about 17.7 MB from one comparison call.

For a local agent tool, that is enough to flood a context window with repeated evidence.

The current implementation limits the whole response, preserves totals, labels truncation, and drops detail in a deterministic order. Its tests measure the actual structured plus text payload received by an MCP client as well as the formatting helper.

The audit also found recorder/parser disagreement, event reordering that could pass, missing provenance that looked like agreement, and a report viewer that could not open the external JSON it documented.

The P0 repair

The fix walks JSON-like values recursively and keeps the type explicit at every path: null, boolean, integer, float, string, array, object, or missing.

That makes the rules identical at the root and inside a container:

[1] versus [true] is a mismatch;
{ "x": 1 } versus { "x": 1.0 } is a mismatch;
null and a missing key are distinct;
tolerance applies only to finite float-versus-float values.

In compact form, the contract is:

same(path) = same_json_type(a, b) ∧ compare_by_type(a, b)

integers: exact
floats:   finite(a, b) ∧ |a - b| ≤ ε
arrays:   same length ∧ same(index) for every index
objects:  same keys ∧ same(key) for every key

The comparison checks type before value at every nested path. Float tolerance applies only when both recorded values are floats.

This is a specification. Its purpose is to make the important bit reviewable: a convenient host-language equality operator never gets to silently redefine the verification contract.

The regression coverage takes those cases through the public recorder, JSONL reader, CLI, and a real stdio MCP process. The local suite is now 242 Python tests; Ruff, strict mypy, package builds, clean-wheel installation, frontend tests, and the frontend build also pass.

That is a precise claim about the known failure: the local release candidate no longer certifies those nested type differences as strict parity.

CSV onboarding and verdict strength

The same candidate adds a small CSV Quick Compare path. A developer maps identity columns, stages, features, types, and units from two CSV logs; the existing trace engine then aligns and compares them. It does not import raw exchange data or introduce a second diff engine.

The important limitation is deliberate: equal CSV rows normally cannot establish matching code, configuration, model, input snapshot, or clock domain. An equal comparison therefore stays UNVERIFIABLE unless that provenance is supplied. A mismatch remains useful evidence of a difference.

A useful tool does not turn absent evidence into agreement. It says what the supplied records can and cannot support.

A test technique worth reusing

For any validator that can return a strong success result, write inputs that contradict the result by construction. For a JSON comparator, that means testing each rule at the root and inside objects and arrays:

True versus 1;
1 versus 1.0;
a missing key versus null;
a value just inside and just outside float tolerance;
identical event IDs with different declared provenance.

This makes the guarantee concrete enough to attack. The repository is public; the reported 242 local tests and quality gates are useful evidence, not an independent correctness proof.

Try it

The repository includes only synthetic examples; it does not contain HFM market data or a trading system.

git clone https://github.com/icesurf666/hfm-parity-lab.git
cd hfm-parity-lab && python -m venv .venv && source .venv/bin/activate
pip install . && hfm-parity demo

That writes a synthetic HTML and JSON report. The nested-value regression tests cover the false-PASS class described here; the README has the trace format and CSV workflow.

I Let an LLM Agent Search for Trading Strategies. A Simple Heuristic Beat It.

Pavel Kazantsev 💎 — Tue, 20 Oct 2026 18:22:17 +0000

I gave DeepSeek V3 forty chances to find a profitable trading strategy.

A simple heuristic found +52% net. The LLM's early proposals lost money. After fixing three silent bugs the agent surfaced, it improved — then optimized its way to +123.7%, a number that didn't survive scrutiny. Here's what happened across four rounds.

The setup

In earlier posts in this series I tested six dead-simple strategies and three more sophisticated edges — all killed by fees, adverse selection, or data leakage. What survived was a backtest harness: point-in-time universe, walk-forward, trading costs, out-of-sample scoring only. Same rules for every strategy, no exceptions. I put an agent in front of it.

A proposal is a typed JSON object:

@dataclass
class Proposal:
    family: str        # "reversal" | "factor_residual" | "momentum"
    bar_size: str      # "15m" | "1h" | "1d"
    lookback: int      # days
    holding_period: int
    entry_threshold: float

# The agent loop
def agent_loop(n_rounds: int = 40) -> list[ScoredProposal]:
    history: list[ScoredProposal] = []
    for _ in range(n_rounds):
        prompt = build_prompt(history)          # what failed, what survived
        json_str = llm(prompt)                  # DeepSeek V3 via OpenRouter
        proposal = parse_proposal(json_str)
        verdict = harness.score(proposal)       # walk-forward over 451 symbols, 76 windows
        history.append(ScoredProposal(proposal, verdict))
    return history

The agent never touches the scoring. It proposes; the same judge that every hand-built strategy went through decides. There is no way for it to talk its way to a good number.

I also kept a heuristic proposer alongside it: samples the space randomly, occasionally perturbs the current best. It is not purely random — the perturbation step is local optimization — but it needs no model and costs nothing. Both the heuristic and the agent ran for the same 40 evaluation rounds. That baseline turned out to matter a lot.

Round one: the model lost to the heuristic

The first LLM run was worse than the heuristic search.

DeepSeek had the right instinct — it leaned toward factor-residual reversion on daily bars (buy coins whose recent return lagged the cross-sectional average, hold for ~30 days), which was the only corner that remained promising in this run. But it picked short lookbacks and fast rebalances:

factor_residual/1d/lb30/reb15   -23.7%   reject
factor_residual/1d/lb20/reb10   -44.4%   reject
reversal/15m/lb5/reb2           -99.1%   reject

Meanwhile the heuristic had stumbled into factor_residual/1d/lb60/reb30: about +52% net, positive in most out-of-sample windows, tiny turnover. The model landed one step away from the edge and picked the wrong side of it. Right family, wrong calibration. The heuristic beat it.

The bugs the harness caught

Before I wrote the model off, something more useful turned up.

Several proposals came back with zero turnover and zero return. Not negative — flat. The strategy had never opened a position. The three bugs had the same symptom: a flat book, zero turnover, no crash, and no error message. Just silence.

Bug 1: Momentum's volatility window was scaled wrong.
The signal normalized by a 96-bar trailing volatility. On 15-minute data that is one day — fine. On daily bars it is 96 days, longer than the formation window, so the signal never became valid.

# Bug: hardcoded window ignores bar size
vol_window = 96

# Fix: scale to timeframe; floor of 5 bars avoids a 1-bar window on daily
BAR_MINUTES = {"15m": 15, "1h": 60, "1d": 1440}
vol_window = max(5, round(96 * 15 / BAR_MINUTES[bar_size]))
# 15m → 96 bars (one day)  |  1h → 24 bars  |  1d → max(5,1) = 5 days

Bug 2: Lookback could exceed the formation window.
The agent proposed a 100-day lookback on a 90-day formation window. Fix: clamp the lookback to what is actually available.

lookback = min(proposal.lookback, formation_window - 1)

Bug 3: The rebalance grid starved itself.
The wrapper refreshed positions on a fixed grid from bar zero: bar 0, bar 40, bar 80. If the signal only came alive around bar 90, every refresh point landed in the warmup period. A holding period of 40 was dead while 30 and 45 traded — non-monotonic results that mean "bug", not "no edge".

# Bug: grid anchored at bar 0
rebalance_bars = range(0, n_bars, holding_period)

# Fix: anchor at first live bar
first_live = warmup_length
rebalance_bars = range(first_live, n_bars, holding_period)

None of these were crashes. Each returned a flat book with no error. That is exactly why a harness should fail loudly when a strategy that is meant to trade has zero turnover. Treating "no position" as a valid zero-return result would have buried all three.

Round two: it maps the edge

With the space cleaned up, I reran the LLM agent. This time it clustered on factor-residual daily and swept the lookback itself:

factor_residual/1d/lb70/reb35   +67.1%
factor_residual/1d/lb50/reb25   +65.9%
factor_residual/1d/lb40/reb20    +6.5%
factor_residual/1d/lb30/reb15   -26.6%

The model drew the line itself. Long lookback positive, short lookback negative, and it found the boundary from feedback alone — no knowledge of what the heuristic had found. Right family, right direction, just needed a clean space to work in.

So is the edge real? Not established.

Across lookbacks from 40 to 80 days with monthly rebalance it stays net positive, roughly +46% to +84% over the backtest period. (The edge map above shows lb40 at only +6.5% because that sweep used a faster reb20 rebalance — different configuration, same lookback.) Positive in four of five years, down hard in 2023. Survives a pessimistic 30 bps per side. Annualized Sharpe around 0.9. A label-permutation null — which scrambles which coin gets which cross-sectional residual, testing whether the ranking actually carries signal — puts it at p = 0.048, right at the edge of significant.

A promising but unconfirmed lead. The parameters and family were selected on these same results, so the p-value doesn't account for the full search process. Not a confirmed strategy — the backtest window is used up.

Round three: a better optimizer, a bigger number

I wrapped the model in a hybrid proposer — the LLM picks the family and timeframe, but on alternating turns it sweeps the lookback and rebalance around the current best.

It found a peak at a 50-day lookback with monthly rebalance: +123.7% net.

(Round 2 showed lb50 at +65.9% with reb25. Same lookback, different rebalance, and now optimizing both jointly — that's where the extra 60 points came from. Also the best of 35 configs, not the average.)

This is a trap, and one column gives it away:

	Round 3 (argmax)	Round 4 (ensemble)
Net return	+123.7%	+75.8%
% positive OOS windows	58%	62%
Bootstrap worst case	−55%	−9%
Bootstrap positive (20 runs)*	16/20	18/20

* 20 bootstrap runs is preliminary; treat the positive-run counts as directional, not precise.

Magnitude doubled. Breadth barely moved.

You can inflate magnitude by picking better parameters. Breadth doesn't work that way. It didn't move.

Here's the problem with reporting the best of 35 configs: even if none of them had any real edge, random variation across 35 tries would still produce one winner. A deflated Sharpe helps, but only for the search you tell it about. The reported 0.70 used this local 7×5 Round 3 grid: N = 35. It did not include the earlier LLM rounds, the heuristic search, the family/timeframe choice, or the debugging runs that led to the grid. Nor are these 35 independent tests: nearby lookbacks produce highly correlated variants of the same signal.

So 0.70 is not a fully search-corrected significance claim. It is a local-grid diagnostic — and even under that friendlier accounting it missed the project's 0.95 qualification threshold. The honest conclusion is simpler: the best configuration did not establish significance. A broader accounting of the search would make the evidence weaker, not stronger.

Round four: I hadn't used my own tools

Overfitting has known cures. I hadn't reached for any of them.

Stop picking a winner. Average the whole plateau instead.

I took all factor-residual configs on daily bars with lookback between 40 and 80 — the range where the sign stayed consistent — and averaged them without picking a winner.

The plateau range itself came from inspecting these same results, so it doesn't fully eliminate selection bias. But it does reduce sensitivity to any single lucky configuration. The +75.8% is less sensitive to one lucky setting, but it is still an in-sample research result, not independent confirmation of alpha.

# Plateau: family=factor_residual, bar=1d, lookback in [40..80]
# Range chosen where sign was consistent — but still data-derived, not pre-specified.
plateau = [r for r in results
           if r.family == "factor_residual"
           and r.bar_size == "1d"
           and 40 <= r.lookback <= 80]

ensemble_signal = np.mean([r.signal for r in plateau], axis=0)

I equal-weighted the factor-residual signal across lookbacks from 40 to 80 days, one combined strategy, monthly rebalance.

Result: +75.8% net, annualized Sharpe 0.61, positive in 62% of windows. That is not a significance claim. A rough calculation using five annual observations puts the t-statistic at about 1.4, before any search correction. The 30-day holds also make adjacent walk-forward windows overlap. So 62% positive windows is a useful breadth check, not 76 independent confirmations.

Both configurations target the same hypothesized signal. The ensemble is less exposed to one lucky (lb, reb) combination; that is all the bootstrap difference shows.

The point

I did not build an AI that finds alpha.

I built a cheap agent with the right instinct and the wrong calibration. The harness exposed three silent bugs because a flat book looked flat. The optimizer then used the cleaned-up space to add 60 percentage points to a backtest number. The ensemble made the remaining signal less tied to one parameter choice, but did not turn it into proof.

Three things I'd take from this:

Run a heuristic baseline first. Until the search space was clean, it beat the model. Don't skip it just because the LLM reasons better.

The agent is only as honest as the thing that scores it. The judge comes first. The proposer is easy to swap out.

Once the proposer can optimize, it will. Watch breadth, not magnitude. And when you can average the plateau instead of picking the best config, do it — the gap between +124% and +76% was entirely parameter sensitivity.

Give a language model a truthful referee and a constrained enough space, and it can find a corner worth inspecting. It will not necessarily beat a simple heuristic, especially if the search space has silent bugs. Once it can optimize, expect a number larger than the evidence warrants unless you correct for the search.

The referee is the part that matters.

14,399 Backtest Events Gave Me t = 8.20. Equal-Weighting Days Gave 1.66

Pavel Kazantsev 💎 — Fri, 16 Oct 2026 16:40:42 +0000

A crypto-perpetual breakout simulation produced 14,399 events across 15 symbols from September 2025 to September 2026. Mean net return was +25.98 bps per event, with a naive event-level t-statistic of +8.20.

But the result depended heavily on how observations were weighted. Returns were concentrated in a small number of market sessions, something the aggregate statistic did not reveal.

Pooled over events, the mean is positive. Per day, it is negative. The gap is entirely Oct 10.

Same events, different unit of analysis

I grouped events by calendar day, computed one mean event return per day, then ran a one-sample t-test on those 365 daily means:

event-level:  t = mean(event returns) / (sd / √N)
               = +25.98 / (380.2 / √14,399)  =  +8.20

day-level:    t = mean(daily means)   / (sd / √N)
               = −4.67  / (53.82 / √365)     =  −1.66

These are different estimands. The first weights every event equally; the second weights every trading day equally. The sign reversal does not prove the naive SE was wrong — it shows that the result is sensitive to the weighting choice, which is reason enough to report both. The second is a diagnostic, not a replacement for clustered standard errors or a block bootstrap. For any strategy that fires multiple times a day on correlated moves, it's a useful sanity check.

The day behind the result

October 10, 2025 accounted for most of the positive event-level return. DOTUSDT traded between $0.94 and $4.29 during a market-wide flash crash. ATOMUSDT also experienced extreme price dislocations. Whether the simulated trades could have been executed at those prices is unclear. The simulation's 434 events that day averaged +981 bps. Remove that one day and the event-level mean changes from +25.98 to −3.69 bps.

The median event lost −16.2 bps. Win rate was 34%. The maximum event return was +8,756 bps. The top 1% of events were enough to flip the mean. Only one day made it positive.

October 10, 2025 contributed 434 of 14,399 events, but those events averaged +981 bps. Removing that one day changes the event-level mean from +25.98 to −3.69 bps.

Statistic	Unit / N	Mean net	SD	t-stat	Note
Naive t-test	14,399 events	+25.98 bps	380.2 bps	+8.20	Assumes independent events
Daily diagnostic	365 days	−4.67 bps	53.8 bps	−1.66	Two-sided p ≈ 0.098
Sensitivity: without Oct 10	364 days	−7.37 bps	14.9 bps	−9.42	Post-hoc; exposes concentration

Event-level SD is 380 bps; day-level SD is 54 bps — a 7× collapse. Both t-stats use the same formula (mean / SE), but they're measuring different dispersions. The event-level SE is tiny because N=14,399; the day-level SE is larger because the daily means are themselves variable, and one day (Oct 10, mean +981 bps) inflates both the event-level mean and the day-level SD in opposite directions.

The sensitivity row is not a confirmatory result. Removing an outlier and calling the rerun clean is p-hacking. It is there to show how much of the claim rests on one day.

The diagnostic

import numpy as np
import pandas as pd
from scipy import stats

def day_block_t(returns_bps: np.ndarray, timestamps: pd.Series) -> dict:
    """Equal-weighted daily diagnostic for event returns.

    Groups events by calendar day, computes the mean return per day,
    and tests whether the distribution of daily means differs from zero.

    This changes the unit of analysis from event to trading day.
    It is NOT a cluster-robust standard error for the event-level mean.
    """
    returns_bps = np.asarray(returns_bps, dtype=float)
    days = pd.to_datetime(pd.Series(timestamps), utc=True).dt.floor("D").to_numpy()
    if len(returns_bps) != len(days):
        raise ValueError("returns_bps and timestamps must have equal length")
    if not np.isfinite(returns_bps).all():
        raise ValueError("returns_bps must be finite")
    frame = pd.DataFrame({"return_bps": returns_bps, "day": days})
    daily = frame.groupby("day")["return_bps"].mean()
    if len(daily) < 2:
        raise ValueError("at least two trading days are required")
    t_stat, p_value = stats.ttest_1samp(daily, popmean=0)

    return {
        "n_events": len(frame),
        "n_days": len(daily),
        "daily_mean_bps": round(daily.mean(), 2),  # input already in bps
        "t_stat": round(float(t_stat), 4),
        "p_value": round(float(p_value), 4),
        "daily": daily,   # keep the series — inspect it, don't trust the summary
    }

# Usage
result = day_block_t(event_returns_bps, event_timestamps)
print(f"Event-level t = {event_returns_bps.mean() / (event_returns_bps.std(ddof=1) / len(event_returns_bps)**0.5):.2f}")
print(f"Day-level   t = {result['t_stat']}  (N={result['n_days']} days)")
# Event-level t = +8.20
# Day-level   t = -1.6579  (N=365 days)

Keep the daily series in the result. A t-statistic is one number; the series shows where the result is coming from.

Inspecting the concentration

import matplotlib.pyplot as plt

daily = result["daily"]  # already in bps

fig, ax = plt.subplots(figsize=(10, 3))
colors = ["#ef4444" if v > 200 else ("#4ade80" if v > 0 else "#f87171")
          for v in daily]
ax.bar(range(len(daily)), daily.values, color=colors, alpha=0.7, width=1)
ax.axhline(0, color="#6b7280", linewidth=1)
ax.set_xlabel("Trading day")
ax.set_ylabel("Daily mean return (bps)")
ax.set_title(f"Daily mean return — {result['n_days']} days, t = {result['t_stat']:.2f}")

# Flag days > 3σ
mean, std = daily.mean(), daily.std()
outliers = daily[(daily - mean).abs() > 3 * std]
for idx, val in outliers.items():
    ax.annotate(f"{val:.0f} bps", xy=(daily.index.get_loc(idx), val),
                xytext=(0, 8), textcoords="offset points",
                ha="center", fontsize=8, color="#ef4444")
plt.tight_layout()

Leave-one-day-out

Removing October 10 explicitly is post-hoc. The full diagnostic runs the same removal for all 365 days and plots t-stat vs dropped day.

LOO diagnostic across all 365 trading days. Dropping any day other than Oct 10 barely moves the t-stat (−1.3 to −1.8 range). Dropping Oct 10 alone moves it to −9.42. One spike, standing alone — concentration, not fat tails.

If several days had moved the t-stat materially, the process would be fat-tailed and the right test would be a week-block bootstrap. That's not what happened here: one day, one event cluster, and it's gone when you drop Oct 10.

Limits

There is no single correct return series. If capital is allocated to a portfolio, construct portfolio returns. If the estimand is per-event alpha, preserve that estimand and use an estimator that accounts for dependence: clustered standard errors, a block bootstrap, or HAC/Newey–West where it applies.

Nor is deleting October 10 a valid way to obtain a clean p-value. The primary result keeps every day. The removal is explicitly a post-hoc sensitivity analysis: it tells us how fragile the claim is, not what a pre-specified experiment would have concluded.

The daily t-test still assumes independence across days. It does not automatically account for serial correlation, which is another reason it is a diagnostic rather than the final inference procedure.

The 14,399 events weren't 14,399 independent observations. Most of the positive return came from 434 events on a single day. The naive t-stat didn't reveal that concentration. The daily breakdown did.

Part of the HFM Research methodology series — six things we found when we audited our own backtest results.

I Built an AI That Checks Whether Digital Products Deliver What They Promise

Pavel Kazantsev 💎 — Tue, 13 Oct 2026 20:17:51 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

I Built an AI That Checks Whether Digital Products Deliver What They Promise

A digital-product listing can promise “48 pages”, “A4 and US Letter editions”, and “debt repayment worksheets.” Then the customer opens the ZIP and finds something else.

For a small seller, that turns into refunds, support messages, rough reviews, and the awful feeling that you missed something obvious. PromiseDiff is a local-first pre-publish check for Etsy, Gumroad, template, and educational-PDF sellers.

Paste in the listing. Upload the actual PDF or ZIP. PromiseDiff shows what the files can back up, what they contradict, and what they simply cannot prove.

The rule I cared about most: missing text is not a failure. A measured mismatch is a contradiction; a claim the files cannot prove is just unverified.

What I built

Every claim gets one of four plain outcomes:

Result	Meaning
`SUPPORTED`	The supplied evidence supports this exact promise.
`CONTRADICTED`	A reproducible check established a conflicting fact.
`UNVERIFIED`	The files do not establish the claim. Missing evidence is not a contradiction.
`OUT_OF_SCOPE`	The claim needs a capability the MVP cannot test.

Every result keeps the method and the evidence that led to it.

For example, a page-count claim is deterministic:

Listing: “The main PDF contains exactly 48 pages.”
File:    budget_planner_A4.pdf → 48 pages
Result:  SUPPORTED — DETERMINISTIC

“Includes debt repayment worksheets” is different. A PDF might call that page a “Loan Balance Reduction Tracker.” Here Gemma looks at the extracted text, but PromiseDiff only accepts the result after checking the suggested filename, page, and quote against the actual PDF.

Demo

The demo uses a real 48-page A4 Budget Planner PDF. Nothing is mocked.

I used ProgressCut to cut the screen recording down to a short walkthrough.

The listing promises:

The main PDF contains exactly 48 pages.
Includes both A4 and US Letter PDF editions.
Contains monthly budget review worksheets.
Includes savings goal trackers.

The upload only has the A4 edition. PromiseDiff confirms the page count, flags the missing US Letter version, and lets the seller open the evidence or export the report as Markdown.

How I built it

Why a hybrid system instead of “ask the model”?

Some questions have a simple answer; others need interpretation.

Question	Correct tool
How many PDF pages exist?	PyMuPDF metadata
Is this page actually A4?	PDF geometry with a 3-point tolerance
Does a listed file exist?	Complete file inventory
Does “Loan Balance Reduction Tracker” plausibly satisfy a debt-repayment claim?	Gemma plus validated quotation

Asking a language model to count pages is slower and worse than reading the PDF metadata. But keyword search alone will not reliably connect “debt repayment” and “loan balance reduction.” So each tool only gets the jobs it can support with evidence.

The rule is:

missing evidence != contradiction

If the document is scanned, the claim scope is ambiguous, a citation points to a nonexistent page, or a quote does not occur in extracted text, the answer is UNVERIFIED.

Architecture and safety

It is a small React + FastAPI app. The verification-flow diagram above shows how the pieces connect.

Uploads are untrusted, so the backend puts tight boundaries around them: size, PDF count, total pages, extracted text, ZIP ratio, file types, and expanded ZIP size. It rejects traversal paths, duplicate entries, encrypted PDFs, nested archives, and corrupt files. The original upload is never saved.

The model work is deliberately kept small:

Gemma extracts structured promises from the listing.
Python validates source quotations and normalizes measurable claims.
Local retrieval selects compact, potentially relevant PDF excerpts.
Gemma evaluates semantic correspondence in batch.
Evidence validation checks filename, page, and exact quote before serialization.

Why does open innovation matter?

Many sellers are working on unpublished material: course notes, templates, client downloads, and draft listings. With the local setup, Ollama keeps the listing text and extracted PDF text on the seller’s machine.

Open-weight Gemma also makes this affordable for a small operation. A local run has no per-audit third-party API bill, although someone still pays for the hardware or hosting. Sellers can repeat the same check before every release without handing their draft material to another service.

Reproducible benchmark, including a failure

I made ten independently labelled synthetic PDF cases and ran them through the same local API the UI uses. The runner saves the raw responses and timings in benchmark/results.json.

exact-label accuracy = correct labels / total cases = 9 / 10 = 90%

xychart-beta
  title "Exact classification by category (10 synthetic cases)"
  x-axis [Supported, Contradicted, Unverified, OutOfScope]
  y-axis "Correct cases" 0 --> 4
  bar [3, 2, 3, 1]

Metric	Result	Caveat
Exact-label accuracy	90% (9/10)	Tiny synthetic evaluation, not a generalization claim.
Contradiction precision	100% (2/2)	Both measured failures were page/dimension facts.
Contradiction recall	100% (2/2)	Same small set.
Supported recall	75% (3/4)	One semantic paraphrase was conservatively left unverified.
Median API latency	2.11 s	Local Gemma on this machine.
Mean API latency	3.50 s	Prompt-injection case took 13.81 s.

The miss is the useful bit. The listing said debt repayment worksheet and the PDF said Loan Balance Reduction Tracker. The model did not have strong enough evidence to connect them, so PromiseDiff returned UNVERIFIED instead of pretending it knew. That is a limitation to fix, not a result to hide.

Known limitations

The benchmark is small and synthetic. It is not a claim about every kind of seller PDF.
Scanned documents without extractable text can still receive metadata checks, but semantic claims remain unverified.
The current retrieval can miss paraphrases. In the benchmark, “debt repayment worksheet” and “Loan Balance Reduction Tracker” stayed unverified rather than becoming a false positive.
Public deployments should isolate PDF processing with strict resource limits.

What I would build next

Better paraphrase retrieval before semantic evaluation.
Seller feedback on which promise types produce the most costly mistakes.
Batch release audits and version-to-version diffs.
Isolated PDF processing for public deployments.

Code

PromiseDiff source code

The repository has the local setup, real example PDF, test commands, security notes, and benchmark runner.

Prize categories

Best Use of Gemma

I Tested 58 Trading Hypotheses. 48 Failed. Here’s the Framework That Killed Them.

Pavel Kazantsev 💎 — Fri, 09 Oct 2026 14:53:19 +0000

58 hypotheses went in. 48 were rejected, 9 remained promising, and 1 could not be evaluated because the required data was unavailable.

None of the 9 earned the label "edge." They earned the right to be tested further.

Some ideas should die before a backtest is ever run. The rest should face the cheapest falsification test first. A hypothesis that cannot answer "who pays us?" and "why does this persist?" is not a hypothesis. It is a guess with a Python file attached.

The kill pipeline that processed all 58, in the order most ideas die:

hypothesis idea
    ↓
cost filter        (does it survive fees and turnover at all?)
    ↓
robustness check   (is the result a plateau or a lucky parameter?)
    ↓
null benchmark     (does it beat a structure-preserving random version?)
    ↓
untouched holdout  (does it hold on a year it never saw?)
    ↓
contamination audit (is the result real or did the code lie to me?)
    ↓
survived research → forward / shadow validation → capital

Each gate catches a different type of false alpha. A strategy can clear the first four and die on the fifth. Passing all five is not a green light. It is permission to start collecting forward data.

Gate 1: costs

The most common way a backtest looks good: skip costs. Cross-sectional reversal on 15-minute Bybit bars had a gross Sharpe of 7 to 13, positive in 80%+ of out-of-sample windows. The gross effect was clearly present before costs. At taker fees, rebalancing every bar, trading costs ran 5 to 10 times the size of the raw edge. After costs, zero out of 76 out-of-sample windows remained positive.

I tried entry/exit thresholds, hysteresis, coarser timeframes, slower rebalancing. No plateau anywhere near break-even. Adjacent parameter settings swung from +45% to −60%.

Run the cost test before you believe the signal, not after.

Gate 2: robustness

Daily momentum had one configuration that returned +97% net at taker fees. The lookback sweep: l=7 gave +33%, l=10 gave −23%, l=14 gave +97%, l=21 gave +35%. No plateau. The +97% was a lucky draw from a jagged surface.

Half the return came from the 2021 bull alone. 2022 was negative. A strategy that only makes money in bull markets and loses in bear is more consistent with market beta than neutral alpha.

One good result on a swept parameter is not evidence. The whole surface needs to hold up.

Gate 3: null benchmark

Funding carry looked strong in development: +48.6% net over 4.5 years, Sharpe 1.17. Before trusting it, I ran a null test: scramble which coin's funding the strategy reads, keep everything else identical. The scrambled version returned +28%.

More than half the return was the structural funding premium: exposure to perpetual market mechanics rather than signal-driven selection. The selection component showed evidence beyond this null in development (empirical permutation p=0.024), but it was much smaller than the headline return implied.

Before believing the result, ask whether a random version of the signal would also make money.

Gate 4: untouched holdout

The lockbox is a date range fixed in advance, never touched until a single opening event. No research, no tuning, no plotting inside that range until development is complete.

Funding carry passed development and validation. The lockbox was a 12-month period (2025-09 to 2026-09), sealed from day one. The frozen strategy made −0.5% on it. The premium that paid from 2021 to 2024 was absent in the lockbox year. One year of absence is not a permanent verdict. It is exactly the kind of regime dependence the lockbox is designed to expose early.

Once a lockbox is opened, it is burned. Any further confirmation requires a new, still-unseen date range. Development results are hypotheses, and only sealed, untouched data can confirm them.

Gate 5: contamination audit

One BTC funding-trend hypothesis had a development t-stat of 2.6. The 1,248 events had 72-hour holding periods, so observations were heavily overlapping. The Newey-West corrected t-stat fell from 2.6 to 1.07, removing the apparent statistical evidence from that test. A real-looking result from a real-looking test, broken by one statistical assumption.

On a separate project, a football prediction model passed closing-line value (CLV) validation with +152bps, 95% CI excluded zero. A five-check forensic audit found two contaminated features. Four of five checks failed. The metric that reported the good result was the last place to look for the error.

Where the 58 actually died

Each hypothesis is assigned to the first decisive failure that stopped further evaluation. Some failed more than one check.

Kill gate	Count	Example
Costs	~20	Price MR: gross Sharpe 7–13, net negative
Robustness	~10	Daily momentum: +97% was one lucky lookback
Null benchmark	~8	Funding carry null still returned +28%
Holdout / fresh OOS	~7	Funding carry −0.5% on sealed year; agent ensemble −35% OOS
No detectable signal / other rejection	~3	New-listing cohort: corr≈0, null p=0.94
Total rejected	48
Data unavailable	1	Required data unavailable
Remaining PROMISING	9	Passed development and validation gates
Benjamini–Hochberg (BH) significant at FDR=0.05	8
Independent mechanisms	4–5	roughly 4–5 distinct economic bets

9 survivors does not mean 9 strategies. Several share the same economic mechanism with different entry filters. 8 Benjamini–Hochberg (BH) significant hypotheses at FDR=0.05 collapse to roughly 4–5 independent bets on market structure by economic mechanism.

The uncomfortable failures

Two cases that took longer than they should have.

An LLM-generated factor-residual ensemble showed +76% in development, Sharpe 0.6, tuned over multiple iterations. On a fresh holdout it reversed to −35%. The development result held inside the data it was tuned on. The fresh holdout showed what the agent had actually learned: the development period.

In a funding carry reproduction audit a month after the original publication, the holdout result changed from −0.5% to +1.3%. Same code, same data cache, fingerprints matched. I still do not have a clean explanation. I published the discrepancy table rather than quietly resolving it. I scrutinized the original negative result far less than I would have scrutinized a positive one. Negative results get a lower evidentiary standard by default. They should not.

What survived

A handful of hypotheses passed enough gates to justify forward validation. They are currently shadow-deployed: signals recorded in real time, parameters locked, no capital.

None of them earned the right to be trusted based on the backtest. The backtest is permission to collect more data, not permission to trade.

The specific signals are not published here. Forward validation is ongoing and the gate has not been reached.

Steal this framework

Before trusting any strategy result:

[ ] Realistic costs included from the start, not added after
[ ] Parameter robustness: result holds over adjacent settings (plateau, not spike)
[ ] Null benchmark: structure-preserving random version loses
[ ] Untouched holdout: a date range sealed before any research on that data
[ ] Contamination audit: no look-ahead, no survivorship, no feature leakage
[ ] Forward validation: signals collected in real time, parameters locked

Costs and robustness killed roughly half of the 58 before a holdout was needed. Run the cheap gates first.

The real lesson

A high kill rate is not evidence that the pipeline is good. But a research process that rarely kills its own ideas is one I would distrust.

48 of 58 died. If the kill rate were much lower, I would not trust the process.

More experiments, engineering notes and quantitative research at pkazantsev.com

Green Tests, Broken Architecture: Four Models Review TypeScript Diffs

Pavel Kazantsev 💎 — Tue, 06 Oct 2026 10:20:38 +0000

This is a submission for the Kaggle Benchmarking Challenge.

In ProgressCut, it is possible to write a change that works, passes a narrow test, and still violates a boundary I care about. A renderer can reach for Electron directly. A recovery record can be overwritten halfway through a write. An exported filename can escape its session folder.

Those are review problems. I made a small benchmark to see how four models review short TypeScript diffs against explicit architecture rules.

Three models got exactly 0.949. Each failed for a completely different reason.

The public Kaggle task contains 13 compact synthetic diffs. Each has an expected APPROVE or BLOCK verdict, a violated invariant where applicable, and a requirement to cite evidence from the diff.

What I Benchmarked

This is a deliberately narrow review exercise:

Can a model reject a functionally plausible change when it violates an explicit repository invariant?

The diffs are synthetic and small. They are based on rules from ProgressCut, a local-first Electron application, but expose no private screenshots, sessions or unreleased code.

The six rules under test:

engine-no-node: algorithm code cannot import Node or platform APIs.
domain-no-runtime-deps: domain types have no runtime-library dependency.
renderer-no-electron: UI reaches native capabilities only through a narrow bridge.
atomic-recovery-write: recovery metadata is written to a temporary file, then renamed.
exact-render-duration: FFmpeg output is capped at the selected story duration.
safe-output-path: an export cannot escape the configured session directory.

Seven changes should be blocked and six should be approved. The approval cases sit close to the violations: a renderer may call a typed bridge, but may not import Electron; resolve() plus a path-boundary check is acceptable, while join(outputDir, userName) is not.

How the score works

Each case is worth one point, split across three deterministic checks:

case score = (correct verdict + correct violated rule + grounded evidence) / 3
benchmark score = mean(case scores)

For an approved change, rule_id must be none. A reviewer has to recognize when a change satisfies a rule, not merely spot its vocabulary.

I ran the unchanged task once per model through Kaggle's benchmark runner. Every model received the same rules and diff. The scorer is deterministic; no LLM judges another LLM.

Models Tested

I used a small, deliberately mixed set from four providers, including the coding-focused Qwen3 Coder. This is not a claim of market coverage; it is a way to compare different review failure modes under the same prompt and scorer.

Model	Score	Server runtime
Gemini 3.7 Flash	1.000	10m 48s
Claude Sonnet 5	0.949	32.9s
GPT-5.4 mini	0.949	14.2s
Qwen3 Coder 480B	0.949	3m 29s

Findings

I had no baseline for whether 0.95 was good or bad, so the number alone told me little. With 13 cases, one third-point miss accounts for the whole gap. The failures were more informative than the scores.

The three 0.949 scores have the same arithmetic: twelve full points and one third-point case. They do not mean the same thing in practice.

Claude

For a direct import { shell } from "electron" inside the renderer, Claude returned:

{"decision":"BLOCK","rule_id":"placeholder","evidence":"placeholder"}

The verdict was right, but the response was unusable for automated enforcement. I cannot tell from this run whether the placeholders came from the model or its structured-output path. Either way, an agent pipeline needs schema validation and a retry or fallback path.

GPT-5.4 mini

The safe version resolves the destination and verifies a separator-bounded prefix:

const root = resolve(outputDir);
const destination = resolve(root, requestedName);
if (!destination.startsWith(`${root}${sep}`)) {
  throw new Error("Invalid export path");
}
await writeFile(destination, data);

GPT-5.4 mini blocked it anyway, arguing that the guard was unsafe. The benchmark expects approval: the guard also rejects the directory itself, which is conservative but not an escape route. This is a real review-agent failure mode: treating a stricter policy as a vulnerability.

Qwen

Qwen blocked a renderer function that takes a DesktopBridge and calls bridge.openFile(path). Its explanation said the bridge ultimately uses Electron APIs.

The bridge may eventually call Electron APIs, but the renderer does not receive Electron itself. Judging the rule from the call graph rather than the declared import boundary quietly changes its meaning.

What the perfect score does not mean

Gemini's 1.000 is a result on 13 hand-written examples. It does not establish that Gemini is the best code reviewer, will enforce arbitrary policy, or is safe to run without tests and human review.

The task has obvious limits:

one repository style and six named invariants;
synthetic diffs rather than a history of production pull requests;
no multi-file reasoning, tool use or test execution;
one run per model, so no variance estimate;
no analysis of cost or token usage.

This is a seed benchmark. Every failure can be inspected and reproduced.

What I would measure next

The next version needs held-out fixtures and harder near-misses: indirect imports, unrelated noisy edits, an atomic write with bad cleanup, and path guards that look safe but accept sibling prefixes. I would also run repeated trials and report schema compliance separately from architecture reasoning.

I would not reduce an AI review to one confidence score. In a real merge gate, I want separate signals for the policy verdict, the structured response, the cited evidence and the deterministic check.

If the response is malformed, its policy decision should not be accepted. When a rule has a deterministic check, that check wins. The model reviews; deterministic logic makes the final decision.

My Benchmark

The public leaderboard is Architecture-Aware TypeScript Code Review on Kaggle. The underlying Safe TypeScript Change Review task and the task source and report generator are public too.

I Killed My Electron App Twice to Test Crash Recovery

Pavel Kazantsev 💎 — Mon, 05 Oct 2026 18:36:40 +0000

"We have crash recovery" is an easy thing to claim. I wanted to know what happens if the process disappears right now.

The question is not specific to Electron. It applies to a media exporter, downloader, migration runner or AI workflow that writes files locally. Restarting an app is easy. Recovering enough state to finish the user's work is the part worth testing.

I used an Electron recorder as the example below. The mechanics are simple: persist something, kill the process, restart, find the leftovers, rebuild, then inspect the result.

For ProgressCut, a local-first desktop app that turns a work session into a short visual story, "right now" has two uncomfortable meanings:

while screenshots are being captured;
while FFmpeg is assembling the final MP4.

In both cases, raw frames can exist while the output is partial or absent.

I started isolated Electron processes, created real session folders, sent SIGKILL, then started fresh Electron processes and asked them to find and rebuild what remained. SIGKILL ends a process immediately, so shutdown handlers cannot quietly clean up the state first.

The test found a rendering bug too: a story budgeted for 3.00 seconds encoded to 3.46 seconds.

What had to survive

If there are captured frames, a later app process must be able to offer a rebuild without recording again.

The implementation is a per-session manifest stored under the user's local application-support directory. It contains the output folder, raw-frame folder, target duration, output format and timestamps. It does not contain screenshot data.

The record has to be per session. A single active-session.json works until two incomplete sessions exist and the second one overwrites the first.

session A → recovery/<hash(A)>.json
session B → recovery/<hash(B)>.json

Every manifest is written as a temporary file and then renamed into place. A successful recovery clears only the manifest for the recovered session.

What I actually killed

The smoke test uses synthetic PNGs instead of recording a real screen. Everything after that is production code: Electron startup, session orchestration, manifest storage, story selection and FFmpeg.

The harness runs two scenarios:

Scenario	Kill point	What a restart must prove
Capture interruption	after the third PNG is persisted	the three frames are discovered and remain byte-for-byte unchanged
Encoding interruption	after the real FFmpeg process starts	a fresh process finds the session and can render a valid MP4

The parent launches each Electron app in its own process group and sends SIGKILL to that group. This matters: killing the Electron parent while leaving FFmpeg behind does not simulate an interruption cleanly.

After each kill, a fresh Electron process reads local recovery manifests. The harness rebuilds every discovered session, runs ffprobe (FFmpeg's inspection tool) on the output and verifies:

H.264 video;
yuv420p pixel format;
encoded duration within 150 ms of the requested three seconds;
raw PNG bytes unchanged;
recovery metadata removed only for the session that completed.

The command is part of the repository:

pnpm test:crash-recovery

If you are testing a different app

Replace "frames" with whatever your app leaves behind: uploaded chunks, edited documents, downloaded files, generated reports or queued jobs.

Choose the smallest irreversible user value. In my case, it is a persisted screenshot; in your app it might be a database checkpoint or a downloaded part file.
Write a recoverable record before expensive work begins. Keep enough information to resume, but avoid storing sensitive content in that record.
Interrupt the real process at a deterministic checkpoint. Do not mock the thing whose behaviour you are claiming to test.
Start a fresh process with the same storage. A retry in the same memory space is not recovery.
Verify the final artefact, then scope cleanup. Check bytes, schema, duration, checksum or domain result, not only that a file exists.

In pseudocode it looks like this:

await runUntil("checkpoint:persisted");
killProcessGroup();

const restartedApp = await launchWithSameDataDirectory();
const pending = await restartedApp.discoverRecoverableWork();
const result = await restartedApp.resume(pending[0]);

expect(await inspect(result)).toMatchObject(expectedArtefact);
expect(await sourceBytes()).toEqual(bytesBeforeCrash);
expect(await pendingRecords()).toEqual([]);

I check cleanup separately because it is easy to get wrong: stale records cause repeated recovery prompts, while broad cleanup can erase another unfinished job.

The failure I wasn't looking for

The first recovery run failed on a stricter check than "the file exists."

The story target was three seconds. ffprobe measured 3.458333 seconds.

That was annoying, because the video looked fine. It played, the codec was right, nothing seemed broken. A test that only checked for a file would have passed it, and I would have shipped the bug.

The renderer uses FFmpeg's concat demuxer because story moments have variable durations. The usual workaround repeats the last file so FFmpeg honours that last still frame. The side effect was extra tail time in the encoded video.

I capped the encoded output at the story's calculated duration and kept a regression test that creates weighted moments and measures the resulting file with ffprobe.

await runFfmpeg([
  // concat input and video filter omitted
  "-movflags", "+faststart",
  "-t", (story.totalDurationMs / 1000).toString(),
  "-y", outputPath,
]);

That is the whole reason to inspect the output rather than assert that a file exists. A report can exist with missing rows; a video can play and still have the wrong length.

What passed

On 2026-10-05, the smoke ran on macOS 26.3, an arm64 Mac17,2 and Node.js 22.14.0:

capture interruption  → 3 frames discovered → recovered
encoding interruption → 3 frames discovered → recovered
source PNGs           → unchanged
output                → H.264 / yuv420p / 3.00 s

I also tested the unsigned packaged arm64 .app rather than only the development process. The app launched with a temporary profile, loaded Sharp from inside app.asar, accepted a production IPC re-render request and generated a three-second MP4.

CSC_IDENTITY_AUTO_DISCOVERY=false pnpm run pack
pnpm test:packaged

That catches a different class of failure: code that works in a workspace because it accidentally resolves a native dependency from the developer machine.

What this does not prove

This covers process crashes and nothing more.

It does not test power loss or filesystem durability beyond atomic rename.
It does not test a multi-hour recording, sleep/wake, monitor changes or a full disk.
It does not test Screen Recording permission UX.
It does not test Developer ID signing, notarization, Gatekeeper or a clean-machine install.
It does not evaluate whether the story selector makes better summaries. That requires real held-out sessions and blind review.

Why I added this before release

For a local-first tool, the useful data can already be on disk when the UI is gone. In this app, throwing away an unfinished session because FFmpeg died would be a bad default.

Each run of the harness goes through the same loop:

Next I need a real two-to-five-hour session with resource measurements, sleep/wake and an intentional interruption. This smoke test narrows the gap; it does not replace that run.

Source: ProgressCut on GitHub, MIT licensed.

I Recorded 10,847 Frames While Coding. Only 30 Were Worth Keeping.

Pavel Kazantsev 💎 — Fri, 02 Oct 2026 19:38:56 +0000

I recorded a two-hour coding session and got 10,847 frames.

The final timelapse needed only 30.

So I built a pipeline to figure out which 30 were actually worth keeping.

Source session: youtube.com/watch?v=GznmPACXBlY

Most of those 10,847 frames were noise: the cursor blinked, the system clock ticked, while nothing meaningful in the code changed.

The first step isn't clever. It's just getting rid of those.

ProgressCut is a local macOS app that turns a coding session into a short visual story. No talking, no editing. You hit stop and it hands you a GIF. Here's how the algorithm actually works.

The compression pipeline

Each stage removes a different kind of noise:

Capture → dHash dedupe → SSIM novelty → Segments → 30 moments
10,847  →    1,203     →     284      →    48     →     30

Real numbers from a 2-hour session.

Stage 1 — dHash: kill the identical frames

89% gone in one pass. At 2 fps, consecutive screenshots are often nearly identical: the cursor blinks, the clock changes, or a single character appears while the rest of the screen stays untouched.

dHash resizes each frame to 9×8 pixels, converts to greyscale, then compares each pixel to the one on its right (1 if brighter, 0 otherwise). 8 rows × 8 comparisons = 64 bits, stored as a BigInt. XOR two hashes and count the differing bits. Distance ≤ 8: duplicate.

Kernighan's bit trick — O(set bits) not O(64)

Counting those differing bits naively loops 64 times. Kernighan's trick loops once per set bit, so for near-duplicates that's typically 2–6 iterations:

export function hammingDistance(a: DHash, b: DHash): number {
  let diff  = a ^ b;      // XOR: 1 where bits differ
  let count = 0;
  while (diff > 0n) {
    diff &= diff - 1n;    // clears the lowest set bit
    count++;
  }
  return count;
}

Why BigInt? JS numbers are IEEE 754 doubles: they top out at 53 safe integer bits. A dHash is 64 bits, so BigInt is the only way to get exact XOR. The n suffix (0n, 1n) is the literal syntax.

Result: 10,847 → 1,203 frames (−89%)

Stage 2 — SSIM: kill the boring frames

Unique isn't the same as interesting. After dHash you still have ~1,200 frames that each look slightly different, but most of that difference is one more character on a line. You need a score that reflects how much the visual state actually changed, not just whether the pixels moved.

SSIM (Structural Similarity Index) compares luminance, contrast, and spatial structure between consecutive frames. Score close to 1: nothing happened. Score close to 0: something worth keeping.

         (2μₓμᵧ + C₁)(2σₓᵧ + C₂)
SSIM = ─────────────────────────────────
       (μₓ² + μᵧ² + C₁)(σₓ² + σᵧ² + C₂)

Full-frame SSIM would average out a big change in one corner against stagnation everywhere else. Instead the frame is tiled into 16 regions, SSIM computed per region, then averaged. A file-save that changes the status bar doesn't steal credit from a tab switch that changed everything.

Single-pass variance — one loop, not two

The naive path does two loops: one for the mean, one for the variance. Both collapse into one using Var(X) = E[X²] − E[X]²: accumulate Σx, Σx², Σy, Σy², Σxy in a single pass and derive everything after:

let sumA=0, sumB=0, sumAA=0, sumBB=0, sumAB=0;
const n = (x1-x0) * (y1-y0);

for (let y=y0; y<y1; y++) {
  for (let x=x0; x<x1; x++) {
    const pa = a.pixels[y*W+x] ?? 0;
    const pb = b.pixels[y*W+x] ?? 0;
    sumA  += pa;    sumB  += pb;
    sumAA += pa*pa; sumBB += pb*pb; sumAB += pa*pb;
  }
}

const muA  = sumA/n,  muB  = sumB/n;
const varA  = sumAA/n - muA*muA;   // Var(X) = E[X²] − E[X]²
const varB  = sumBB/n - muB*muB;
const covAB = sumAB/n - muA*muB;

Result: 1,203 → 284 frames (−77%)

Stage 3 — Segmentation: budget the moments fairly

Sessions aren't uniformly active. There's 20 minutes of flow where you're actually building something, then 10 minutes reading docs, then another sprint. If you spread the 30-moment budget evenly across the timeline, the reading gaps eat slots that should go to the interesting parts.

The pipeline splits the frame sequence into activity windows by novelty density. Each window's share of the budget scales with how much happened inside it:

warm-up │ ███████████ active  │ ░ idle │ ████████ active │ ████ wrap-up
 2 mom  │   12 moments        │ 1 mom  │  9 moments      │ 6 mom

A 20-minute sprint might take 8 moments; a 10-minute idle stretch gets 1.

Result: 284 frames → 48 segments → 30 moments

What comes out

30 moments → a GIF or MP4, under 30 seconds, no input from you. Each moment holds for a duration proportional to the activity level of its segment.

Input:  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  10,847 frames
Output: █  ██   █████  █  ████  █  ██   ████      30 moments

No frames leave the machine. The app calls no model. Run it twice on the same input and you get the same output. That last part matters more than it sounds: a non-deterministic story compressor is harder to trust and impossible to debug.

What's next

Better novelty scoring with embeddings

SSIM can measure structural change, but it has no idea what that change means. Opening a new file and typing a few characters may affect a similar amount of screen area even though one is a much more meaningful transition.

An embedding-based scorer might separate those cases better by measuring semantic change rather than raw pixel structure. That's the next hypothesis to test.

The port in packages/engine is already the swap point: implement a new scorer behind the same interface, run the same 187 tests, and leave the rest of the pipeline untouched.

But swapping scorers without a way to evaluate the result is just guessing. The feedback loop comes first: let users remove moments they don't think belong in the final story, then use that data to compare scoring methods.

SSIM stays the default until an embedding-based scorer demonstrates a measurable improvement.

Longer term: adaptive budget

The 30-moment budget is fixed right now. A focused 20-minute session and a messy two-hour session both get 30 moments.

The segmentation step already estimates how much activity happened in each window, so the next step is to let the total story length depend on the session itself.

A short session with lots of meaningful changes might need fewer, faster moments. A long session with large idle stretches should compress harder instead of wasting slots on inactivity.

If you were building this, would you keep SSIM or move to embeddings?

Code

MIT, pnpm monorepo, 187 tests, CI on macos-latest. The algorithm lives in packages/engine/src/ with no Electron dependency, runs in plain Node. If you want to poke at the frame selection logic without building the whole app, that's the entry point.

→ github.com/icesurf666/progress-cut

Three Crypto Edges. What Survived the Audit?

Pavel Kazantsev 💎 — Tue, 29 Sep 2026 15:13:08 +0000

A research platform should not make it easy to stop at a good-looking number. That applies to negative results too. Transaction-cost checks, held-out evaluation, and parameter sweeps are useful only when their results can be traced back to their runs.

Candidate	Historical report	Current audit status
Price mean reversion	Gross Sharpe 7–13; rejected after costs	Historical revision range not independently reproduced here
Funding carry	+48.6% development; −0.5% holdout	Rerun: +45.62% development; +1.31% holdout; discrepancy unresolved
Daily momentum	+97% best case; sensitive to lookback	Matching run/configuration not located; sweep remains unverified

Edge 1: Mean reversion and the cost problem

The historical summary reports cross-sectional reversal on 15-minute bars with gross Sharpe between 7 and 13, positive in over 80% of out-of-sample windows across 76 test periods. These figures span research revisions; they are not measurements from one controlled run.

That summary also reports friction at 5–10 times the available edge when rebalancing every bar, and neighboring settings ranging from +45% to −60% net. Those revision-specific ratios and parameter results have not been independently reproduced in this audit. They should not be treated as verified evidence that every implementation must fail.

The reported decision was to reject the tested implementations because a robust net result had not been established. The engineering question is whether a pre-cost pattern survives a specified execution model, not whether mean reversion is universally untradeable.

A separate deep-dive describes a controlled evaluation. Its metrics should not be mixed with this historical revision summary as if they came from one run.

Edge 2: Funding carry and a failed reproduction

The tested carry construction combines long spot with short perpetual exposure and funding payments. Lower turnover reduces transaction costs; it does not eliminate them or other risks.

The original report gives +48.6% net development return, Sharpe 1.17, and −0.5% net holdout return, Sharpe −0.02. On September 30, I reran the current implementation against the current local cache. Both the original evaluation script and a separate audit runner produced different results:

Metric	Historical report	Current rerun
Development net cumulative return	+48.6%	+45.62%
Development Sharpe	1.17	1.079
Holdout net cumulative return	−0.5%	+1.31%
Holdout Sharpe	−0.02	0.208

Historical report: −0.5%. Current rerun: +1.31%. These are different run versions, not a controlled experiment measuring an improvement. The cause of the discrepancy is not established.

The current configuration uses FundingCarry(384), rebalances every 1344 15-minute bars (14 days), normalizes to fixed gross exposure, and applies modelled fees of 4 bps plus slippage of 2 bps. It selects 45 symbols and scores 76 development windows and 17 holdout windows.

The 17 complete 21-day holdout windows cover September 1, 2025 through August 24, 2026 (exclusive): 357 days, not the entire configured interval ending September 1. The original saved window list was not found, so historical coverage cannot be independently certified.

In the current rerun, gross holdout return is +2.14% and net return is +1.31% on the same windows. Costs reduce the result under this model. This comparison does not explain the historical −0.5% return. Development returns are positive in each represented calendar year, including +3.44% in 2022; that does not recover the original annual figures.

The audit runner fingerprints source code and cached input files before and after execution, saves scenario returns, and independently recalculates cumulative return and Sharpe from those returns. Those checks passed, as did 14 relevant engine tests. They check these calculations, not every modelling assumption or the historical result.

The current universe filter uses funding-record counts from the cache before date restriction. Applying the threshold using only pre-holdout records yielded the same 45-symbol set in this cache. That does not establish point-in-time eligibility throughout development. No original run manifest or committed HFM code history was available to recover the historical code, universe and dataset. Nor does this audit prove the reserved interval was originally untouched.

The earlier development null comparison, reported at about +28%, was not reproduced in this audit. It is not used here to infer a market-wide premium or selection skill.

Null diagnostics on the current rerun

Two structure-preserving null tests were run post-hoc on the 17 holdout windows of the current reproduction result (net +1.31%, Sharpe +0.208). These diagnostics do not apply to the historical result and do not resolve the discrepancy. They ask two separable questions: did the temporal alignment between position decisions and outcomes contribute, and did the specific coin-selection rule add anything beyond random assignment?

CircularShiftNull — temporal alignment. For each coin i, a shift τᵢ is drawn uniformly from [0, T). The weight path becomes:

w̃ᵢ(t) = wᵢ((t + τᵢ) mod T)

What this preserves: per-asset weight distribution, autocorrelation structure, holding period, and per-asset turnover Σₜ |wᵢ(t) − wᵢ(t−1)|. What it destroys: the alignment between a position decision and the specific return interval it faced. H₀: the observed Sharpe is no better than an arbitrarily time-shifted version of the same weight paths. N = 500 seeds.

RebalanceSelectionNull — coin selection. Rebalance events are detected as:

T_R = {t : ‖w(t) − w(t−1)‖₁ > ε}

At each t ∈ T_R, the weight vector is permuted across universe symbols. What this preserves: rebalance schedule, gross exposure Σᵢ|wᵢ|, number of active positions, weight magnitudes. What it destroys: which symbol receives which weight at each rebalance. N = 500 seeds.

Both use the Davison-Hinkley finite-sample p-value to avoid the p = 0 artifact from small N:

p = (1 + #{θ_null ≥ θ_obs}) / (N + 1)

Turnover inflation. CircularShiftNull wraps each coin's weight path at the window boundary. When τᵢ > 0, the series at time T − τᵢ jumps discontinuously to the value at time 0, creating large simultaneous |Δw| spikes at the seam. Mean null turnover reached 1.82× frozen. RebalanceSelectionNull avoids the wrap-around artifact but permutes across the full universe rather than the positive-funding eligible set, changing the effective cost profile. Mean null turnover: 1.34× frozen. In both cases the frozen strategy received a structural cost advantage independent of signal quality. The CircularShift p-value (0.010) reflects a mix of genuine temporal misalignment loss and artificial cost inflation that cannot be cleanly separated. These p-values are reported for transparency, not as evidence.

The right next step is a funding-matched null: at each rebalance, permute weight assignments only within the positive-funding eligible set. This would hold funding-exposure loading constant and isolate the coin-ranking question directly. It was not implemented because the execution API does not expose the contemporaneous panel signal needed to define the eligible set.

Sharpe uncertainty via block bootstrap. The 17 holdout windows are serially dependent: consecutive windows share overlapping formation data, and 21-day non-overlapping hold periods are far too short for serial correlation to decay. The naive standard error 1/√17 ≈ 0.24 treats windows as independent and understates uncertainty by a material factor.

A moving-block bootstrap was run on the 15-minute holdout return series. Block length L = 4032 bars corresponds to approximately 3 full rebalance cycles (~63 days), chosen to span the dominant autocorrelation horizon of the strategy. N = 2000 resamples; annualization factor √(365 × 24 × 4).

point estimate:      Sharpe +0.208
95% CI (L = 4032):  [−0.21, +0.44]
p(Sharpe > 0):       78.2%

The 95% interval spans zero. With 17 non-overlapping windows, wide CIs are the honest result, not a flaw in the method.

The historical REJECT decision is part of the research record, but a negative holdout is no longer a verified rationale in this article. The candidate remains on hold pending reconciliation. The positive rerun alone does not establish a tradeable edge. Analysis of this already-opened interval is exploratory, not a new out-of-sample confirmation.

Edge 3: Momentum and an unverified parameter sweep

The historical summary describes daily cross-sectional momentum with weekly rebalancing and a best net return of approximately +97%, including taker fees. It gives four lookback results: 7 days +33%, 10 days −23%, 14 days +97%, and 21 days +35%.

Values transcribed from the frozen narrative report. A matching saved run/configuration was not located in this check. These are not four independently validated results.

If reproduced, that sweep would raise a parameter-sensitivity concern: the best setting is not surrounded by comparably strong results. It would not, by itself, prove the peak was luck. The historical summary also attributes about half the return to 2021 and reports a negative 2022. Those regime figures remain unverified here; the report did not estimate a formal beta decomposition.

The reported WEAK verdict is a historical assessment, not a newly verified finding from this audit.

What the audit actually established

The current carry calculation is reproducible locally with recorded code and input hashes. Its headline values do not match the historical report. For the mean-reversion revision range and momentum sweep, narrative evidence is available but numerical reproduction remains incomplete.

That is a less tidy conclusion than three precise diagnoses. It is also the result the evidence supports. A negative backtest deserves the same provenance checks as a profitable one.

No candidate is being advanced to live trading on the basis of this article. The next step is to recover the historical run artifacts or explicitly retire unsupported historical numbers, not silently substitute new results into the old story.

The research package preserves the original narrative evidence. This is not a complete independent reproduction of the private data pipeline. SHA-256 checks establish file identity, not economic validity. The frozen source retains stronger claims that this corrected article does not endorse.

Evidence source SHA-256: cf9ce73542af24be3e5c2872864b2b41c6ce864e3b2f07b760e5434a1bdb23ba

Research system: HFM · Historical reports and a post-publication rerun · Bybit spot and perpetuals · Python

My Football Model Passed Validation. A Five-Check Audit Killed It.

Pavel Kazantsev 💎 — Sun, 27 Sep 2026 07:08:32 +0000

My football prediction model passed validation. Val away CLV was +152 bps, confidence interval excluded zero. Then I ran five forensic checks. Four failed, one passed, and the pass made the four failures harder to argue away.

The experiment

The hypothesis: EPL bookmakers systematically underprice high-odds away teams. A LightGBM classifier trained on seven seasons of opening-line data would find those games and generate positive closing line value.

CLV is mean(model_prob - devigged_closing_prob). Devigging removes the bookmaker's margin from decimal odds to recover fair probabilities that sum to 1. Positive CLV means your model put higher probability on an outcome than the final Pinnacle price did, after sharp money finished moving it. It is the standard measure of model skill in sports prediction research.

14 features, all opening-line: devigged Pinnacle probabilities, B365, market consensus, Pinnacle-vs-consensus disagreement, raw decimal odds. No temporal index, no threshold from the full dataset. LightGBM with n_estimators=300, learning_rate=0.03, num_leaves=15, hyperparameters locked before touching val. Logistic regression as a same-feature baseline. 380 val games (EPL 2022-23). One pre-registered hypothesis: AWAY outcome.

The surface result:

Split	Outcome	CLV	CI	Status
Val 2223	Away	+152 bps	[+55, +251]	CLV_ONLY_BRIER_FAIL
Val 2223	Home	−39 bps	[−132, +52]	NULL_RESULT
Test 2324	Away	+11 bps	[−79, +100]	NULL_RESULT
Test 2324	Home	+176 bps	[+95, +263]	CLV_ONLY_BRIER_FAIL

Val away CI excludes zero, which is what the hypothesis required. The verdict column reads CLV_ONLY_BRIER_FAIL rather than EDGE_CANDIDATE because the protocol requires positive CLV and Brier improvement over the closing line. Five forensic checks determined why.

Check 1: What does the market itself drift?

Any model trained on opening odds will inherit whatever directional drift the market already has between open and close. Measure that drift first, with no model, so you know what the baseline is.

Val away drift: +3.9 bps (CI: −19.5 to +26.8, crosses zero). No systematic directional movement in 2022-23 for away outcomes.

The model showed +152 bps on the same split, so +148 bps was model-generated with no structural drift to explain it.

Check 2: Brier skill score

A model that systematically overestimates probabilities in the right direction will show positive CLV while being less accurate than the market. Brier score separates the two. To claim predictive value, the model needs a lower Brier score than the closing line on the same games.

Model Brier vs closing line, all 6 split-outcome combinations. Lower is better. The model is worse on every single one.

The model lost on Brier in all six combinations. Delta ranged from +0.004 to +0.019, consistently worse than the closing line.

A model can beat the closing line on CLV while losing to it on Brier. It is overestimating probabilities in the right direction at the wrong magnitude. That is systematic bias.

Check 3: Feature cleanliness

The first version of this experiment had two contaminated features. season_num was a global index 0-8 derived from the full nine-season dataset, so val season got index 7 and the model extrapolated outside its training range. is_high_odds_away was a binary flag using a threshold of 0.62 set by a calibration run on all nine seasons, including val and test.

I removed both and reran from scratch with 14 features: opening odds from Pinnacle, B365, and market consensus; bookmaker disagreement; raw decimal odds. All from pre-match data only. Check 3: PASS. Zero violations.

Removing the contaminated features dropped val away CLV from +216 bps to +152 bps, so the contamination was inflating the result by about 64 bps. With a clean feature set, the remaining failures have nothing to blame.

Check 4: Incremental CLV

Standard CLV is model_prob - close_prob. To separate the model's contribution from the market's own drift, compute model_prob - open_prob instead. The closing line cancels algebraically:

(model - close) - (open - close) = model - open

Val away CLV decomposed. Market drifted +3.9 bps. Model added +147.7 bps. The model contributes 38x the underlying signal.

The model pushed away probabilities 147.7 bps above the opening line (CI: [+52.5, +244.4]). True market drift was +3.9 bps. The model overshot by 38x.

On test, the same model produced +11 bps CLV for away (CI: [−79, +100], crosses zero). The model was overestimating away probabilities in both splits. In val that overshoot happened to align with a small positive drift. In test it produced noise.

Check 5: Season stability

If removing one training season flips the sign of the result, the result depends on that season, not on a real underlying relationship. Leave-one-out validation removes one training season at a time and re-measures val CLV.

Outcome	Mean CLV	Std	Range	Sign stable?
Home	−39.7 bps	85.6	[−200, +114]	No
Draw	−134.2 bps	44.2	[−213, −81]	Yes (consistently negative)
Away	+106.3 bps	126.6	[−151, +314]	No (sign flips)

Away CLV standard deviation: 126.6 bps. Removing season 2020-21 gives +314 bps, removing season 2021-22 gives −151 bps. The val result is entirely a function of which seasons happened to land in the training set.

Four failures. One pass. Verdict unchanged.

Fixing the contamination improved the experiment, not the result. The core failures are independent of the bad features.

Check	Finding	Status
Opening-line baseline	True drift +3.9 bps; model added 38x more	FAIL
Brier skill score	Model worse than market on all 6 combinations	FAIL
Feature cleanliness	14 features, zero violations, clean rerun	PASS
Incremental CLV	Model overshoots by 148 bps; collapses to noise on test	FAIL
Season stability	Std 126.6 bps, sign flips on leave-one-out	FAIL

Logistic regression on the same features: val away CLV +87 bps (CI: [+57, +117]), Brier 0.185 vs closing 0.182, test away −102 bps (CI: [−135, −71]). Same pattern. The calibration failure is not specific to LightGBM.

All four failing checks trace to the same thing: the model overestimates away probabilities. In 2022-23 that overshoot happened to align with a small and statistically insignificant positive drift. In 2023-24 it produced +11 bps with a CI that crosses zero. REJECT.

The contamination fix that changed nothing

Val away CLV dropped from 216 to 152 bps when the contaminated features were removed. The contamination inflated the number. But 152 bps still excludes zero, so one could argue the signal survived.

Brier closes that argument. With clean features, on both val and test, the model is less accurate than the closing line on all six outcome-split combinations. A model that cannot beat the market's accuracy is overfit, not predictive. The contamination was real. Removing it changed the CLV. The Brier failure is independent and survived the cleanup.

Finding contamination and fixing it is not enough. The clean rerun has to pass all gates. This one does not.

What a genuine signal would require

To restart this experiment honestly, the protocol needs harder gates. Brier improvement over closing as a required gate, not a diagnostic. Sign-stable LOO results before the test set is touched at all. Multi-league or multi-season test data, because one EPL season (380 games) is too small for confirmation at this effect size. At 150 bps, 380 games gives borderline statistical power and cannot separate a real edge from model miscalibration of the same magnitude. That is a different experiment from what this was.

Research system: HFM · LightGBM · 9 EPL seasons · Walk-forward OOS · Clean features v2