DEV Community: Pavel Kazantsev 💎

I Built an AI That Checks Whether Digital Products Deliver What They Promise

Pavel Kazantsev 💎 — Tue, 13 Oct 2026 20:17:51 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

I Built an AI That Checks Whether Digital Products Deliver What They Promise

A digital-product listing can promise “48 pages”, “A4 and US Letter editions”, and “debt repayment worksheets.” Then the customer opens the ZIP and finds something else.

For a small seller, that turns into refunds, support messages, rough reviews, and the awful feeling that you missed something obvious. PromiseDiff is a local-first pre-publish check for Etsy, Gumroad, template, and educational-PDF sellers.

Paste in the listing. Upload the actual PDF or ZIP. PromiseDiff shows what the files can back up, what they contradict, and what they simply cannot prove.

The rule I cared about most: missing text is not a failure. A measured mismatch is a contradiction; a claim the files cannot prove is just unverified.

What I built

Every claim gets one of four plain outcomes:

Result	Meaning
`SUPPORTED`	The supplied evidence supports this exact promise.
`CONTRADICTED`	A reproducible check established a conflicting fact.
`UNVERIFIED`	The files do not establish the claim. Missing evidence is not a contradiction.
`OUT_OF_SCOPE`	The claim needs a capability the MVP cannot test.

Every result keeps the method and the evidence that led to it.

For example, a page-count claim is deterministic:

Listing: “The main PDF contains exactly 48 pages.”
File:    budget_planner_A4.pdf → 48 pages
Result:  SUPPORTED — DETERMINISTIC

“Includes debt repayment worksheets” is different. A PDF might call that page a “Loan Balance Reduction Tracker.” Here Gemma looks at the extracted text, but PromiseDiff only accepts the result after checking the suggested filename, page, and quote against the actual PDF.

Demo

The demo uses a real 48-page A4 Budget Planner PDF. Nothing is mocked.

I used ProgressCut to cut the screen recording down to a short walkthrough.

The listing promises:

The main PDF contains exactly 48 pages.
Includes both A4 and US Letter PDF editions.
Contains monthly budget review worksheets.
Includes savings goal trackers.

The upload only has the A4 edition. PromiseDiff confirms the page count, flags the missing US Letter version, and lets the seller open the evidence or export the report as Markdown.

How I built it

Why a hybrid system instead of “ask the model”?

Some questions have a simple answer; others need interpretation.

Question	Correct tool
How many PDF pages exist?	PyMuPDF metadata
Is this page actually A4?	PDF geometry with a 3-point tolerance
Does a listed file exist?	Complete file inventory
Does “Loan Balance Reduction Tracker” plausibly satisfy a debt-repayment claim?	Gemma plus validated quotation

Asking a language model to count pages is slower and worse than reading the PDF metadata. But keyword search alone will not reliably connect “debt repayment” and “loan balance reduction.” So each tool only gets the jobs it can support with evidence.

The rule is:

missing evidence != contradiction

If the document is scanned, the claim scope is ambiguous, a citation points to a nonexistent page, or a quote does not occur in extracted text, the answer is UNVERIFIED.

Architecture and safety

It is a small React + FastAPI app. The verification-flow diagram above shows how the pieces connect.

Uploads are untrusted, so the backend puts tight boundaries around them: size, PDF count, total pages, extracted text, ZIP ratio, file types, and expanded ZIP size. It rejects traversal paths, duplicate entries, encrypted PDFs, nested archives, and corrupt files. The original upload is never saved.

The model work is deliberately kept small:

Gemma extracts structured promises from the listing.
Python validates source quotations and normalizes measurable claims.
Local retrieval selects compact, potentially relevant PDF excerpts.
Gemma evaluates semantic correspondence in batch.
Evidence validation checks filename, page, and exact quote before serialization.

Why does open innovation matter?

Many sellers are working on unpublished material: course notes, templates, client downloads, and draft listings. With the local setup, Ollama keeps the listing text and extracted PDF text on the seller’s machine.

Open-weight Gemma also makes this affordable for a small operation. A local run has no per-audit third-party API bill, although someone still pays for the hardware or hosting. Sellers can repeat the same check before every release without handing their draft material to another service.

Reproducible benchmark, including a failure

I made ten independently labelled synthetic PDF cases and ran them through the same local API the UI uses. The runner saves the raw responses and timings in benchmark/results.json.

exact-label accuracy = correct labels / total cases = 9 / 10 = 90%

xychart-beta
  title "Exact classification by category (10 synthetic cases)"
  x-axis [Supported, Contradicted, Unverified, OutOfScope]
  y-axis "Correct cases" 0 --> 4
  bar [3, 2, 3, 1]

Metric	Result	Caveat
Exact-label accuracy	90% (9/10)	Tiny synthetic evaluation, not a generalization claim.
Contradiction precision	100% (2/2)	Both measured failures were page/dimension facts.
Contradiction recall	100% (2/2)	Same small set.
Supported recall	75% (3/4)	One semantic paraphrase was conservatively left unverified.
Median API latency	2.11 s	Local Gemma on this machine.
Mean API latency	3.50 s	Prompt-injection case took 13.81 s.

The miss is the useful bit. The listing said debt repayment worksheet and the PDF said Loan Balance Reduction Tracker. The model did not have strong enough evidence to connect them, so PromiseDiff returned UNVERIFIED instead of pretending it knew. That is a limitation to fix, not a result to hide.

Known limitations

The benchmark is small and synthetic. It is not a claim about every kind of seller PDF.
Scanned documents without extractable text can still receive metadata checks, but semantic claims remain unverified.
The current retrieval can miss paraphrases. In the benchmark, “debt repayment worksheet” and “Loan Balance Reduction Tracker” stayed unverified rather than becoming a false positive.
Public deployments should isolate PDF processing with strict resource limits.

What I would build next

Better paraphrase retrieval before semantic evaluation.
Seller feedback on which promise types produce the most costly mistakes.
Batch release audits and version-to-version diffs.
Isolated PDF processing for public deployments.

Code

PromiseDiff source code

The repository has the local setup, real example PDF, test commands, security notes, and benchmark runner.

Prize categories

Best Use of Gemma

I Tested 58 Trading Hypotheses. 48 Failed. Here’s the Framework That Killed Them.

Pavel Kazantsev 💎 — Fri, 09 Oct 2026 14:53:19 +0000

58 hypotheses went in. 48 were rejected, 9 remained promising, and 1 could not be evaluated because the required data was unavailable.

None of the 9 earned the label "edge." They earned the right to be tested further.

Some ideas should die before a backtest is ever run. The rest should face the cheapest falsification test first. A hypothesis that cannot answer "who pays us?" and "why does this persist?" is not a hypothesis. It is a guess with a Python file attached.

The kill pipeline that processed all 58, in the order most ideas die:

hypothesis idea
    ↓
cost filter        (does it survive fees and turnover at all?)
    ↓
robustness check   (is the result a plateau or a lucky parameter?)
    ↓
null benchmark     (does it beat a structure-preserving random version?)
    ↓
untouched holdout  (does it hold on a year it never saw?)
    ↓
contamination audit (is the result real or did the code lie to me?)
    ↓
survived research → forward / shadow validation → capital

Each gate catches a different type of false alpha. A strategy can clear the first four and die on the fifth. Passing all five is not a green light. It is permission to start collecting forward data.

Gate 1: costs

The most common way a backtest looks good: skip costs. Cross-sectional reversal on 15-minute Bybit bars had a gross Sharpe of 7 to 13, positive in 80%+ of out-of-sample windows. The gross effect was clearly present before costs. At taker fees, rebalancing every bar, trading costs ran 5 to 10 times the size of the raw edge. After costs, zero out of 76 out-of-sample windows remained positive.

I tried entry/exit thresholds, hysteresis, coarser timeframes, slower rebalancing. No plateau anywhere near break-even. Adjacent parameter settings swung from +45% to −60%.

Run the cost test before you believe the signal, not after.

Gate 2: robustness

Daily momentum had one configuration that returned +97% net at taker fees. The lookback sweep: l=7 gave +33%, l=10 gave −23%, l=14 gave +97%, l=21 gave +35%. No plateau. The +97% was a lucky draw from a jagged surface.

Half the return came from the 2021 bull alone. 2022 was negative. A strategy that only makes money in bull markets and loses in bear is more consistent with market beta than neutral alpha.

One good result on a swept parameter is not evidence. The whole surface needs to hold up.

Gate 3: null benchmark

Funding carry looked strong in development: +48.6% net over 4.5 years, Sharpe 1.17. Before trusting it, I ran a null test: scramble which coin's funding the strategy reads, keep everything else identical. The scrambled version returned +28%.

More than half the return was the structural funding premium: exposure to perpetual market mechanics rather than signal-driven selection. The selection component showed evidence beyond this null in development (empirical permutation p=0.024), but it was much smaller than the headline return implied.

Before believing the result, ask whether a random version of the signal would also make money.

Gate 4: untouched holdout

The lockbox is a date range fixed in advance, never touched until a single opening event. No research, no tuning, no plotting inside that range until development is complete.

Funding carry passed development and validation. The lockbox was a 12-month period (2025-09 to 2026-09), sealed from day one. The frozen strategy made −0.5% on it. The premium that paid from 2021 to 2024 was absent in the lockbox year. One year of absence is not a permanent verdict. It is exactly the kind of regime dependence the lockbox is designed to expose early.

Once a lockbox is opened, it is burned. Any further confirmation requires a new, still-unseen date range. Development results are hypotheses, and only sealed, untouched data can confirm them.

Gate 5: contamination audit

One BTC funding-trend hypothesis had a development t-stat of 2.6. The 1,248 events had 72-hour holding periods, so observations were heavily overlapping. The Newey-West corrected t-stat fell from 2.6 to 1.07, removing the apparent statistical evidence from that test. A real-looking result from a real-looking test, broken by one statistical assumption.

On a separate project, a football prediction model passed closing-line value (CLV) validation with +152bps, 95% CI excluded zero. A five-check forensic audit found two contaminated features. Four of five checks failed. The metric that reported the good result was the last place to look for the error.

Where the 58 actually died

Each hypothesis is assigned to the first decisive failure that stopped further evaluation. Some failed more than one check.

Kill gate	Count	Example
Costs	~20	Price MR: gross Sharpe 7–13, net negative
Robustness	~10	Daily momentum: +97% was one lucky lookback
Null benchmark	~8	Funding carry null still returned +28%
Holdout / fresh OOS	~7	Funding carry −0.5% on sealed year; agent ensemble −35% OOS
No detectable signal / other rejection	~3	New-listing cohort: corr≈0, null p=0.94
Total rejected	48
Data unavailable	1	Required data unavailable
Remaining PROMISING	9	Passed development and validation gates
Benjamini–Hochberg (BH) significant at FDR=0.05	8
Independent mechanisms	4–5	roughly 4–5 distinct economic bets

9 survivors does not mean 9 strategies. Several share the same economic mechanism with different entry filters. 8 Benjamini–Hochberg (BH) significant hypotheses at FDR=0.05 collapse to roughly 4–5 independent bets on market structure by economic mechanism.

The uncomfortable failures

Two cases that took longer than they should have.

An LLM-generated factor-residual ensemble showed +76% in development, Sharpe 0.6, tuned over multiple iterations. On a fresh holdout it reversed to −35%. The development result held inside the data it was tuned on. The fresh holdout showed what the agent had actually learned: the development period.

In a funding carry reproduction audit a month after the original publication, the holdout result changed from −0.5% to +1.3%. Same code, same data cache, fingerprints matched. I still do not have a clean explanation. I published the discrepancy table rather than quietly resolving it. I scrutinized the original negative result far less than I would have scrutinized a positive one. Negative results get a lower evidentiary standard by default. They should not.

What survived

A handful of hypotheses passed enough gates to justify forward validation. They are currently shadow-deployed: signals recorded in real time, parameters locked, no capital.

None of them earned the right to be trusted based on the backtest. The backtest is permission to collect more data, not permission to trade.

The specific signals are not published here. Forward validation is ongoing and the gate has not been reached.

Steal this framework

Before trusting any strategy result:

[ ] Realistic costs included from the start, not added after
[ ] Parameter robustness: result holds over adjacent settings (plateau, not spike)
[ ] Null benchmark: structure-preserving random version loses
[ ] Untouched holdout: a date range sealed before any research on that data
[ ] Contamination audit: no look-ahead, no survivorship, no feature leakage
[ ] Forward validation: signals collected in real time, parameters locked

Costs and robustness killed roughly half of the 58 before a holdout was needed. Run the cheap gates first.

The real lesson

A high kill rate is not evidence that the pipeline is good. But a research process that rarely kills its own ideas is one I would distrust.

48 of 58 died. If the kill rate were much lower, I would not trust the process.

More experiments, engineering notes and quantitative research at pkazantsev.com

Green Tests, Broken Architecture: Four Models Review TypeScript Diffs

Pavel Kazantsev 💎 — Tue, 06 Oct 2026 10:20:38 +0000

This is a submission for the Kaggle Benchmarking Challenge.

In ProgressCut, it is possible to write a change that works, passes a narrow test, and still violates a boundary I care about. A renderer can reach for Electron directly. A recovery record can be overwritten halfway through a write. An exported filename can escape its session folder.

Those are review problems. I made a small benchmark to see how four models review short TypeScript diffs against explicit architecture rules.

Three models got exactly 0.949. Each failed for a completely different reason.

The public Kaggle task contains 13 compact synthetic diffs. Each has an expected APPROVE or BLOCK verdict, a violated invariant where applicable, and a requirement to cite evidence from the diff.

What I Benchmarked

This is a deliberately narrow review exercise:

Can a model reject a functionally plausible change when it violates an explicit repository invariant?

The diffs are synthetic and small. They are based on rules from ProgressCut, a local-first Electron application, but expose no private screenshots, sessions or unreleased code.

The six rules under test:

engine-no-node: algorithm code cannot import Node or platform APIs.
domain-no-runtime-deps: domain types have no runtime-library dependency.
renderer-no-electron: UI reaches native capabilities only through a narrow bridge.
atomic-recovery-write: recovery metadata is written to a temporary file, then renamed.
exact-render-duration: FFmpeg output is capped at the selected story duration.
safe-output-path: an export cannot escape the configured session directory.

Seven changes should be blocked and six should be approved. The approval cases sit close to the violations: a renderer may call a typed bridge, but may not import Electron; resolve() plus a path-boundary check is acceptable, while join(outputDir, userName) is not.

How the score works

Each case is worth one point, split across three deterministic checks:

case score = (correct verdict + correct violated rule + grounded evidence) / 3
benchmark score = mean(case scores)

For an approved change, rule_id must be none. A reviewer has to recognize when a change satisfies a rule, not merely spot its vocabulary.

I ran the unchanged task once per model through Kaggle's benchmark runner. Every model received the same rules and diff. The scorer is deterministic; no LLM judges another LLM.

Models Tested

I used a small, deliberately mixed set from four providers, including the coding-focused Qwen3 Coder. This is not a claim of market coverage; it is a way to compare different review failure modes under the same prompt and scorer.

Model	Score	Server runtime
Gemini 3.7 Flash	1.000	10m 48s
Claude Sonnet 5	0.949	32.9s
GPT-5.4 mini	0.949	14.2s
Qwen3 Coder 480B	0.949	3m 29s

Findings

I had no baseline for whether 0.95 was good or bad, so the number alone told me little. With 13 cases, one third-point miss accounts for the whole gap. The failures were more informative than the scores.

The three 0.949 scores have the same arithmetic: twelve full points and one third-point case. They do not mean the same thing in practice.

Claude

For a direct import { shell } from "electron" inside the renderer, Claude returned:

{"decision":"BLOCK","rule_id":"placeholder","evidence":"placeholder"}

The verdict was right, but the response was unusable for automated enforcement. I cannot tell from this run whether the placeholders came from the model or its structured-output path. Either way, an agent pipeline needs schema validation and a retry or fallback path.

GPT-5.4 mini

The safe version resolves the destination and verifies a separator-bounded prefix:

const root = resolve(outputDir);
const destination = resolve(root, requestedName);
if (!destination.startsWith(`${root}${sep}`)) {
  throw new Error("Invalid export path");
}
await writeFile(destination, data);

GPT-5.4 mini blocked it anyway, arguing that the guard was unsafe. The benchmark expects approval: the guard also rejects the directory itself, which is conservative but not an escape route. This is a real review-agent failure mode: treating a stricter policy as a vulnerability.

Qwen

Qwen blocked a renderer function that takes a DesktopBridge and calls bridge.openFile(path). Its explanation said the bridge ultimately uses Electron APIs.

The bridge may eventually call Electron APIs, but the renderer does not receive Electron itself. Judging the rule from the call graph rather than the declared import boundary quietly changes its meaning.

What the perfect score does not mean

Gemini's 1.000 is a result on 13 hand-written examples. It does not establish that Gemini is the best code reviewer, will enforce arbitrary policy, or is safe to run without tests and human review.

The task has obvious limits:

one repository style and six named invariants;
synthetic diffs rather than a history of production pull requests;
no multi-file reasoning, tool use or test execution;
one run per model, so no variance estimate;
no analysis of cost or token usage.

This is a seed benchmark. Every failure can be inspected and reproduced.

What I would measure next

The next version needs held-out fixtures and harder near-misses: indirect imports, unrelated noisy edits, an atomic write with bad cleanup, and path guards that look safe but accept sibling prefixes. I would also run repeated trials and report schema compliance separately from architecture reasoning.

I would not reduce an AI review to one confidence score. In a real merge gate, I want separate signals for the policy verdict, the structured response, the cited evidence and the deterministic check.

If the response is malformed, its policy decision should not be accepted. When a rule has a deterministic check, that check wins. The model reviews; deterministic logic makes the final decision.

My Benchmark

The public leaderboard is Architecture-Aware TypeScript Code Review on Kaggle. The underlying Safe TypeScript Change Review task and the task source and report generator are public too.

I Killed My Electron App Twice to Test Crash Recovery

Pavel Kazantsev 💎 — Mon, 05 Oct 2026 18:36:40 +0000

"We have crash recovery" is an easy thing to claim. I wanted to know what happens if the process disappears right now.

The question is not specific to Electron. It applies to a media exporter, downloader, migration runner or AI workflow that writes files locally. Restarting an app is easy. Recovering enough state to finish the user's work is the part worth testing.

I used an Electron recorder as the example below. The mechanics are simple: persist something, kill the process, restart, find the leftovers, rebuild, then inspect the result.

For ProgressCut, a local-first desktop app that turns a work session into a short visual story, "right now" has two uncomfortable meanings:

while screenshots are being captured;
while FFmpeg is assembling the final MP4.

In both cases, raw frames can exist while the output is partial or absent.

I started isolated Electron processes, created real session folders, sent SIGKILL, then started fresh Electron processes and asked them to find and rebuild what remained. SIGKILL ends a process immediately, so shutdown handlers cannot quietly clean up the state first.

The test found a rendering bug too: a story budgeted for 3.00 seconds encoded to 3.46 seconds.

What had to survive

If there are captured frames, a later app process must be able to offer a rebuild without recording again.

The implementation is a per-session manifest stored under the user's local application-support directory. It contains the output folder, raw-frame folder, target duration, output format and timestamps. It does not contain screenshot data.

The record has to be per session. A single active-session.json works until two incomplete sessions exist and the second one overwrites the first.

session A → recovery/<hash(A)>.json
session B → recovery/<hash(B)>.json

Every manifest is written as a temporary file and then renamed into place. A successful recovery clears only the manifest for the recovered session.

What I actually killed

The smoke test uses synthetic PNGs instead of recording a real screen. Everything after that is production code: Electron startup, session orchestration, manifest storage, story selection and FFmpeg.

The harness runs two scenarios:

Scenario	Kill point	What a restart must prove
Capture interruption	after the third PNG is persisted	the three frames are discovered and remain byte-for-byte unchanged
Encoding interruption	after the real FFmpeg process starts	a fresh process finds the session and can render a valid MP4

The parent launches each Electron app in its own process group and sends SIGKILL to that group. This matters: killing the Electron parent while leaving FFmpeg behind does not simulate an interruption cleanly.

After each kill, a fresh Electron process reads local recovery manifests. The harness rebuilds every discovered session, runs ffprobe (FFmpeg's inspection tool) on the output and verifies:

H.264 video;
yuv420p pixel format;
encoded duration within 150 ms of the requested three seconds;
raw PNG bytes unchanged;
recovery metadata removed only for the session that completed.

The command is part of the repository:

pnpm test:crash-recovery

If you are testing a different app

Replace "frames" with whatever your app leaves behind: uploaded chunks, edited documents, downloaded files, generated reports or queued jobs.

Choose the smallest irreversible user value. In my case, it is a persisted screenshot; in your app it might be a database checkpoint or a downloaded part file.
Write a recoverable record before expensive work begins. Keep enough information to resume, but avoid storing sensitive content in that record.
Interrupt the real process at a deterministic checkpoint. Do not mock the thing whose behaviour you are claiming to test.
Start a fresh process with the same storage. A retry in the same memory space is not recovery.
Verify the final artefact, then scope cleanup. Check bytes, schema, duration, checksum or domain result, not only that a file exists.

In pseudocode it looks like this:

await runUntil("checkpoint:persisted");
killProcessGroup();

const restartedApp = await launchWithSameDataDirectory();
const pending = await restartedApp.discoverRecoverableWork();
const result = await restartedApp.resume(pending[0]);

expect(await inspect(result)).toMatchObject(expectedArtefact);
expect(await sourceBytes()).toEqual(bytesBeforeCrash);
expect(await pendingRecords()).toEqual([]);

I check cleanup separately because it is easy to get wrong: stale records cause repeated recovery prompts, while broad cleanup can erase another unfinished job.

The failure I wasn't looking for

The first recovery run failed on a stricter check than "the file exists."

The story target was three seconds. ffprobe measured 3.458333 seconds.

That was annoying, because the video looked fine. It played, the codec was right, nothing seemed broken. A test that only checked for a file would have passed it, and I would have shipped the bug.

The renderer uses FFmpeg's concat demuxer because story moments have variable durations. The usual workaround repeats the last file so FFmpeg honours that last still frame. The side effect was extra tail time in the encoded video.

I capped the encoded output at the story's calculated duration and kept a regression test that creates weighted moments and measures the resulting file with ffprobe.

await runFfmpeg([
  // concat input and video filter omitted
  "-movflags", "+faststart",
  "-t", (story.totalDurationMs / 1000).toString(),
  "-y", outputPath,
]);

That is the whole reason to inspect the output rather than assert that a file exists. A report can exist with missing rows; a video can play and still have the wrong length.

What passed

On 2026-10-05, the smoke ran on macOS 26.3, an arm64 Mac17,2 and Node.js 22.14.0:

capture interruption  → 3 frames discovered → recovered
encoding interruption → 3 frames discovered → recovered
source PNGs           → unchanged
output                → H.264 / yuv420p / 3.00 s

I also tested the unsigned packaged arm64 .app rather than only the development process. The app launched with a temporary profile, loaded Sharp from inside app.asar, accepted a production IPC re-render request and generated a three-second MP4.

CSC_IDENTITY_AUTO_DISCOVERY=false pnpm run pack
pnpm test:packaged

That catches a different class of failure: code that works in a workspace because it accidentally resolves a native dependency from the developer machine.

What this does not prove

This covers process crashes and nothing more.

It does not test power loss or filesystem durability beyond atomic rename.
It does not test a multi-hour recording, sleep/wake, monitor changes or a full disk.
It does not test Screen Recording permission UX.
It does not test Developer ID signing, notarization, Gatekeeper or a clean-machine install.
It does not evaluate whether the story selector makes better summaries. That requires real held-out sessions and blind review.

Why I added this before release

For a local-first tool, the useful data can already be on disk when the UI is gone. In this app, throwing away an unfinished session because FFmpeg died would be a bad default.

Each run of the harness goes through the same loop:

Next I need a real two-to-five-hour session with resource measurements, sleep/wake and an intentional interruption. This smoke test narrows the gap; it does not replace that run.

Source: ProgressCut on GitHub, MIT licensed.

I Recorded 10,847 Frames While Coding. Only 30 Were Worth Keeping.

Pavel Kazantsev 💎 — Fri, 02 Oct 2026 19:38:56 +0000

I recorded a two-hour coding session and got 10,847 frames.

The final timelapse needed only 30.

So I built a pipeline to figure out which 30 were actually worth keeping.

Source session: youtube.com/watch?v=GznmPACXBlY

Most of those 10,847 frames were noise: the cursor blinked, the system clock ticked, while nothing meaningful in the code changed.

The first step isn't clever. It's just getting rid of those.

ProgressCut is a local macOS app that turns a coding session into a short visual story. No talking, no editing. You hit stop and it hands you a GIF. Here's how the algorithm actually works.

The compression pipeline

Each stage removes a different kind of noise:

Capture → dHash dedupe → SSIM novelty → Segments → 30 moments
10,847  →    1,203     →     284      →    48     →     30

Real numbers from a 2-hour session.

Stage 1 — dHash: kill the identical frames

89% gone in one pass. At 2 fps, consecutive screenshots are often nearly identical: the cursor blinks, the clock changes, or a single character appears while the rest of the screen stays untouched.

dHash resizes each frame to 9×8 pixels, converts to greyscale, then compares each pixel to the one on its right (1 if brighter, 0 otherwise). 8 rows × 8 comparisons = 64 bits, stored as a BigInt. XOR two hashes and count the differing bits. Distance ≤ 8: duplicate.

Kernighan's bit trick — O(set bits) not O(64)

Counting those differing bits naively loops 64 times. Kernighan's trick loops once per set bit, so for near-duplicates that's typically 2–6 iterations:

export function hammingDistance(a: DHash, b: DHash): number {
  let diff  = a ^ b;      // XOR: 1 where bits differ
  let count = 0;
  while (diff > 0n) {
    diff &= diff - 1n;    // clears the lowest set bit
    count++;
  }
  return count;
}

Why BigInt? JS numbers are IEEE 754 doubles: they top out at 53 safe integer bits. A dHash is 64 bits, so BigInt is the only way to get exact XOR. The n suffix (0n, 1n) is the literal syntax.

Result: 10,847 → 1,203 frames (−89%)

Stage 2 — SSIM: kill the boring frames

Unique isn't the same as interesting. After dHash you still have ~1,200 frames that each look slightly different, but most of that difference is one more character on a line. You need a score that reflects how much the visual state actually changed, not just whether the pixels moved.

SSIM (Structural Similarity Index) compares luminance, contrast, and spatial structure between consecutive frames. Score close to 1: nothing happened. Score close to 0: something worth keeping.

         (2μₓμᵧ + C₁)(2σₓᵧ + C₂)
SSIM = ─────────────────────────────────
       (μₓ² + μᵧ² + C₁)(σₓ² + σᵧ² + C₂)

Full-frame SSIM would average out a big change in one corner against stagnation everywhere else. Instead the frame is tiled into 16 regions, SSIM computed per region, then averaged. A file-save that changes the status bar doesn't steal credit from a tab switch that changed everything.

Single-pass variance — one loop, not two

The naive path does two loops: one for the mean, one for the variance. Both collapse into one using Var(X) = E[X²] − E[X]²: accumulate Σx, Σx², Σy, Σy², Σxy in a single pass and derive everything after:

let sumA=0, sumB=0, sumAA=0, sumBB=0, sumAB=0;
const n = (x1-x0) * (y1-y0);

for (let y=y0; y<y1; y++) {
  for (let x=x0; x<x1; x++) {
    const pa = a.pixels[y*W+x] ?? 0;
    const pb = b.pixels[y*W+x] ?? 0;
    sumA  += pa;    sumB  += pb;
    sumAA += pa*pa; sumBB += pb*pb; sumAB += pa*pb;
  }
}

const muA  = sumA/n,  muB  = sumB/n;
const varA  = sumAA/n - muA*muA;   // Var(X) = E[X²] − E[X]²
const varB  = sumBB/n - muB*muB;
const covAB = sumAB/n - muA*muB;

Result: 1,203 → 284 frames (−77%)

Stage 3 — Segmentation: budget the moments fairly

Sessions aren't uniformly active. There's 20 minutes of flow where you're actually building something, then 10 minutes reading docs, then another sprint. If you spread the 30-moment budget evenly across the timeline, the reading gaps eat slots that should go to the interesting parts.

The pipeline splits the frame sequence into activity windows by novelty density. Each window's share of the budget scales with how much happened inside it:

warm-up │ ███████████ active  │ ░ idle │ ████████ active │ ████ wrap-up
 2 mom  │   12 moments        │ 1 mom  │  9 moments      │ 6 mom

A 20-minute sprint might take 8 moments; a 10-minute idle stretch gets 1.

Result: 284 frames → 48 segments → 30 moments

What comes out

30 moments → a GIF or MP4, under 30 seconds, no input from you. Each moment holds for a duration proportional to the activity level of its segment.

Input:  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  10,847 frames
Output: █  ██   █████  █  ████  █  ██   ████      30 moments

No frames leave the machine. The app calls no model. Run it twice on the same input and you get the same output. That last part matters more than it sounds: a non-deterministic story compressor is harder to trust and impossible to debug.

What's next

Better novelty scoring with embeddings

SSIM can measure structural change, but it has no idea what that change means. Opening a new file and typing a few characters may affect a similar amount of screen area even though one is a much more meaningful transition.

An embedding-based scorer might separate those cases better by measuring semantic change rather than raw pixel structure. That's the next hypothesis to test.

The port in packages/engine is already the swap point: implement a new scorer behind the same interface, run the same 187 tests, and leave the rest of the pipeline untouched.

But swapping scorers without a way to evaluate the result is just guessing. The feedback loop comes first: let users remove moments they don't think belong in the final story, then use that data to compare scoring methods.

SSIM stays the default until an embedding-based scorer demonstrates a measurable improvement.

Longer term: adaptive budget

The 30-moment budget is fixed right now. A focused 20-minute session and a messy two-hour session both get 30 moments.

The segmentation step already estimates how much activity happened in each window, so the next step is to let the total story length depend on the session itself.

A short session with lots of meaningful changes might need fewer, faster moments. A long session with large idle stretches should compress harder instead of wasting slots on inactivity.

If you were building this, would you keep SSIM or move to embeddings?

Code

MIT, pnpm monorepo, 187 tests, CI on macos-latest. The algorithm lives in packages/engine/src/ with no Electron dependency, runs in plain Node. If you want to poke at the frame selection logic without building the whole app, that's the entry point.

→ github.com/icesurf666/progress-cut

Three Crypto Edges. What Survived the Audit?

Pavel Kazantsev 💎 — Tue, 29 Sep 2026 15:13:08 +0000

A research platform should not make it easy to stop at a good-looking number. That applies to negative results too. Transaction-cost checks, held-out evaluation, and parameter sweeps are useful only when their results can be traced back to their runs.

Candidate	Historical report	Current audit status
Price mean reversion	Gross Sharpe 7–13; rejected after costs	Historical revision range not independently reproduced here
Funding carry	+48.6% development; −0.5% holdout	Rerun: +45.62% development; +1.31% holdout; discrepancy unresolved
Daily momentum	+97% best case; sensitive to lookback	Matching run/configuration not located; sweep remains unverified

Edge 1: Mean reversion and the cost problem

The historical summary reports cross-sectional reversal on 15-minute bars with gross Sharpe between 7 and 13, positive in over 80% of out-of-sample windows across 76 test periods. These figures span research revisions; they are not measurements from one controlled run.

That summary also reports friction at 5–10 times the available edge when rebalancing every bar, and neighboring settings ranging from +45% to −60% net. Those revision-specific ratios and parameter results have not been independently reproduced in this audit. They should not be treated as verified evidence that every implementation must fail.

The reported decision was to reject the tested implementations because a robust net result had not been established. The engineering question is whether a pre-cost pattern survives a specified execution model, not whether mean reversion is universally untradeable.

A separate deep-dive describes a controlled evaluation. Its metrics should not be mixed with this historical revision summary as if they came from one run.

Edge 2: Funding carry and a failed reproduction

The tested carry construction combines long spot with short perpetual exposure and funding payments. Lower turnover reduces transaction costs; it does not eliminate them or other risks.

The original report gives +48.6% net development return, Sharpe 1.17, and −0.5% net holdout return, Sharpe −0.02. On September 30, I reran the current implementation against the current local cache. Both the original evaluation script and a separate audit runner produced different results:

Metric	Historical report	Current rerun
Development net cumulative return	+48.6%	+45.62%
Development Sharpe	1.17	1.079
Holdout net cumulative return	−0.5%	+1.31%
Holdout Sharpe	−0.02	0.208

Historical report: −0.5%. Current rerun: +1.31%. These are different run versions, not a controlled experiment measuring an improvement. The cause of the discrepancy is not established.

The current configuration uses FundingCarry(384), rebalances every 1344 15-minute bars (14 days), normalizes to fixed gross exposure, and applies modelled fees of 4 bps plus slippage of 2 bps. It selects 45 symbols and scores 76 development windows and 17 holdout windows.

The 17 complete 21-day holdout windows cover September 1, 2025 through August 24, 2026 (exclusive): 357 days, not the entire configured interval ending September 1. The original saved window list was not found, so historical coverage cannot be independently certified.

In the current rerun, gross holdout return is +2.14% and net return is +1.31% on the same windows. Costs reduce the result under this model. This comparison does not explain the historical −0.5% return. Development returns are positive in each represented calendar year, including +3.44% in 2022; that does not recover the original annual figures.

The audit runner fingerprints source code and cached input files before and after execution, saves scenario returns, and independently recalculates cumulative return and Sharpe from those returns. Those checks passed, as did 14 relevant engine tests. They check these calculations, not every modelling assumption or the historical result.

The current universe filter uses funding-record counts from the cache before date restriction. Applying the threshold using only pre-holdout records yielded the same 45-symbol set in this cache. That does not establish point-in-time eligibility throughout development. No original run manifest or committed HFM code history was available to recover the historical code, universe and dataset. Nor does this audit prove the reserved interval was originally untouched.

The earlier development null comparison, reported at about +28%, was not reproduced in this audit. It is not used here to infer a market-wide premium or selection skill.

Null diagnostics on the current rerun

Two structure-preserving null tests were run post-hoc on the 17 holdout windows of the current reproduction result (net +1.31%, Sharpe +0.208). These diagnostics do not apply to the historical result and do not resolve the discrepancy. They ask two separable questions: did the temporal alignment between position decisions and outcomes contribute, and did the specific coin-selection rule add anything beyond random assignment?

CircularShiftNull — temporal alignment. For each coin i, a shift τᵢ is drawn uniformly from [0, T). The weight path becomes:

w̃ᵢ(t) = wᵢ((t + τᵢ) mod T)

What this preserves: per-asset weight distribution, autocorrelation structure, holding period, and per-asset turnover Σₜ |wᵢ(t) − wᵢ(t−1)|. What it destroys: the alignment between a position decision and the specific return interval it faced. H₀: the observed Sharpe is no better than an arbitrarily time-shifted version of the same weight paths. N = 500 seeds.

RebalanceSelectionNull — coin selection. Rebalance events are detected as:

T_R = {t : ‖w(t) − w(t−1)‖₁ > ε}

At each t ∈ T_R, the weight vector is permuted across universe symbols. What this preserves: rebalance schedule, gross exposure Σᵢ|wᵢ|, number of active positions, weight magnitudes. What it destroys: which symbol receives which weight at each rebalance. N = 500 seeds.

Both use the Davison-Hinkley finite-sample p-value to avoid the p = 0 artifact from small N:

p = (1 + #{θ_null ≥ θ_obs}) / (N + 1)

Turnover inflation. CircularShiftNull wraps each coin's weight path at the window boundary. When τᵢ > 0, the series at time T − τᵢ jumps discontinuously to the value at time 0, creating large simultaneous |Δw| spikes at the seam. Mean null turnover reached 1.82× frozen. RebalanceSelectionNull avoids the wrap-around artifact but permutes across the full universe rather than the positive-funding eligible set, changing the effective cost profile. Mean null turnover: 1.34× frozen. In both cases the frozen strategy received a structural cost advantage independent of signal quality. The CircularShift p-value (0.010) reflects a mix of genuine temporal misalignment loss and artificial cost inflation that cannot be cleanly separated. These p-values are reported for transparency, not as evidence.

The right next step is a funding-matched null: at each rebalance, permute weight assignments only within the positive-funding eligible set. This would hold funding-exposure loading constant and isolate the coin-ranking question directly. It was not implemented because the execution API does not expose the contemporaneous panel signal needed to define the eligible set.

Sharpe uncertainty via block bootstrap. The 17 holdout windows are serially dependent: consecutive windows share overlapping formation data, and 21-day non-overlapping hold periods are far too short for serial correlation to decay. The naive standard error 1/√17 ≈ 0.24 treats windows as independent and understates uncertainty by a material factor.

A moving-block bootstrap was run on the 15-minute holdout return series. Block length L = 4032 bars corresponds to approximately 3 full rebalance cycles (~63 days), chosen to span the dominant autocorrelation horizon of the strategy. N = 2000 resamples; annualization factor √(365 × 24 × 4).

point estimate:      Sharpe +0.208
95% CI (L = 4032):  [−0.21, +0.44]
p(Sharpe > 0):       78.2%

The 95% interval spans zero. With 17 non-overlapping windows, wide CIs are the honest result, not a flaw in the method.

The historical REJECT decision is part of the research record, but a negative holdout is no longer a verified rationale in this article. The candidate remains on hold pending reconciliation. The positive rerun alone does not establish a tradeable edge. Analysis of this already-opened interval is exploratory, not a new out-of-sample confirmation.

Edge 3: Momentum and an unverified parameter sweep

The historical summary describes daily cross-sectional momentum with weekly rebalancing and a best net return of approximately +97%, including taker fees. It gives four lookback results: 7 days +33%, 10 days −23%, 14 days +97%, and 21 days +35%.

Values transcribed from the frozen narrative report. A matching saved run/configuration was not located in this check. These are not four independently validated results.

If reproduced, that sweep would raise a parameter-sensitivity concern: the best setting is not surrounded by comparably strong results. It would not, by itself, prove the peak was luck. The historical summary also attributes about half the return to 2021 and reports a negative 2022. Those regime figures remain unverified here; the report did not estimate a formal beta decomposition.

The reported WEAK verdict is a historical assessment, not a newly verified finding from this audit.

What the audit actually established

The current carry calculation is reproducible locally with recorded code and input hashes. Its headline values do not match the historical report. For the mean-reversion revision range and momentum sweep, narrative evidence is available but numerical reproduction remains incomplete.

That is a less tidy conclusion than three precise diagnoses. It is also the result the evidence supports. A negative backtest deserves the same provenance checks as a profitable one.

No candidate is being advanced to live trading on the basis of this article. The next step is to recover the historical run artifacts or explicitly retire unsupported historical numbers, not silently substitute new results into the old story.

The research package preserves the original narrative evidence. This is not a complete independent reproduction of the private data pipeline. SHA-256 checks establish file identity, not economic validity. The frozen source retains stronger claims that this corrected article does not endorse.

Evidence source SHA-256: cf9ce73542af24be3e5c2872864b2b41c6ce864e3b2f07b760e5434a1bdb23ba

Research system: HFM · Historical reports and a post-publication rerun · Bybit spot and perpetuals · Python

My Football Model Passed Validation. A Five-Check Audit Killed It.

Pavel Kazantsev 💎 — Sun, 27 Sep 2026 07:08:32 +0000

My football prediction model passed validation. Val away CLV was +152 bps, confidence interval excluded zero. Then I ran five forensic checks. Four failed, one passed, and the pass made the four failures harder to argue away.

The experiment

The hypothesis: EPL bookmakers systematically underprice high-odds away teams. A LightGBM classifier trained on seven seasons of opening-line data would find those games and generate positive closing line value.

CLV is mean(model_prob - devigged_closing_prob). Devigging removes the bookmaker's margin from decimal odds to recover fair probabilities that sum to 1. Positive CLV means your model put higher probability on an outcome than the final Pinnacle price did, after sharp money finished moving it. It is the standard measure of model skill in sports prediction research.

14 features, all opening-line: devigged Pinnacle probabilities, B365, market consensus, Pinnacle-vs-consensus disagreement, raw decimal odds. No temporal index, no threshold from the full dataset. LightGBM with n_estimators=300, learning_rate=0.03, num_leaves=15, hyperparameters locked before touching val. Logistic regression as a same-feature baseline. 380 val games (EPL 2022-23). One pre-registered hypothesis: AWAY outcome.

The surface result:

Split	Outcome	CLV	CI	Status
Val 2223	Away	+152 bps	[+55, +251]	CLV_ONLY_BRIER_FAIL
Val 2223	Home	−39 bps	[−132, +52]	NULL_RESULT
Test 2324	Away	+11 bps	[−79, +100]	NULL_RESULT
Test 2324	Home	+176 bps	[+95, +263]	CLV_ONLY_BRIER_FAIL

Val away CI excludes zero, which is what the hypothesis required. The verdict column reads CLV_ONLY_BRIER_FAIL rather than EDGE_CANDIDATE because the protocol requires positive CLV and Brier improvement over the closing line. Five forensic checks determined why.

Check 1: What does the market itself drift?

Any model trained on opening odds will inherit whatever directional drift the market already has between open and close. Measure that drift first, with no model, so you know what the baseline is.

Val away drift: +3.9 bps (CI: −19.5 to +26.8, crosses zero). No systematic directional movement in 2022-23 for away outcomes.

The model showed +152 bps on the same split, so +148 bps was model-generated with no structural drift to explain it.

Check 2: Brier skill score

A model that systematically overestimates probabilities in the right direction will show positive CLV while being less accurate than the market. Brier score separates the two. To claim predictive value, the model needs a lower Brier score than the closing line on the same games.

Model Brier vs closing line, all 6 split-outcome combinations. Lower is better. The model is worse on every single one.

The model lost on Brier in all six combinations. Delta ranged from +0.004 to +0.019, consistently worse than the closing line.

A model can beat the closing line on CLV while losing to it on Brier. It is overestimating probabilities in the right direction at the wrong magnitude. That is systematic bias.

Check 3: Feature cleanliness

The first version of this experiment had two contaminated features. season_num was a global index 0-8 derived from the full nine-season dataset, so val season got index 7 and the model extrapolated outside its training range. is_high_odds_away was a binary flag using a threshold of 0.62 set by a calibration run on all nine seasons, including val and test.

I removed both and reran from scratch with 14 features: opening odds from Pinnacle, B365, and market consensus; bookmaker disagreement; raw decimal odds. All from pre-match data only. Check 3: PASS. Zero violations.

Removing the contaminated features dropped val away CLV from +216 bps to +152 bps, so the contamination was inflating the result by about 64 bps. With a clean feature set, the remaining failures have nothing to blame.

Check 4: Incremental CLV

Standard CLV is model_prob - close_prob. To separate the model's contribution from the market's own drift, compute model_prob - open_prob instead. The closing line cancels algebraically:

(model - close) - (open - close) = model - open

Val away CLV decomposed. Market drifted +3.9 bps. Model added +147.7 bps. The model contributes 38x the underlying signal.

The model pushed away probabilities 147.7 bps above the opening line (CI: [+52.5, +244.4]). True market drift was +3.9 bps. The model overshot by 38x.

On test, the same model produced +11 bps CLV for away (CI: [−79, +100], crosses zero). The model was overestimating away probabilities in both splits. In val that overshoot happened to align with a small positive drift. In test it produced noise.

Check 5: Season stability

If removing one training season flips the sign of the result, the result depends on that season, not on a real underlying relationship. Leave-one-out validation removes one training season at a time and re-measures val CLV.

Outcome	Mean CLV	Std	Range	Sign stable?
Home	−39.7 bps	85.6	[−200, +114]	No
Draw	−134.2 bps	44.2	[−213, −81]	Yes (consistently negative)
Away	+106.3 bps	126.6	[−151, +314]	No (sign flips)

Away CLV standard deviation: 126.6 bps. Removing season 2020-21 gives +314 bps, removing season 2021-22 gives −151 bps. The val result is entirely a function of which seasons happened to land in the training set.

Four failures. One pass. Verdict unchanged.

Fixing the contamination improved the experiment, not the result. The core failures are independent of the bad features.

Check	Finding	Status
Opening-line baseline	True drift +3.9 bps; model added 38x more	FAIL
Brier skill score	Model worse than market on all 6 combinations	FAIL
Feature cleanliness	14 features, zero violations, clean rerun	PASS
Incremental CLV	Model overshoots by 148 bps; collapses to noise on test	FAIL
Season stability	Std 126.6 bps, sign flips on leave-one-out	FAIL

Logistic regression on the same features: val away CLV +87 bps (CI: [+57, +117]), Brier 0.185 vs closing 0.182, test away −102 bps (CI: [−135, −71]). Same pattern. The calibration failure is not specific to LightGBM.

All four failing checks trace to the same thing: the model overestimates away probabilities. In 2022-23 that overshoot happened to align with a small and statistically insignificant positive drift. In 2023-24 it produced +11 bps with a CI that crosses zero. REJECT.

The contamination fix that changed nothing

Val away CLV dropped from 216 to 152 bps when the contaminated features were removed. The contamination inflated the number. But 152 bps still excludes zero, so one could argue the signal survived.

Brier closes that argument. With clean features, on both val and test, the model is less accurate than the closing line on all six outcome-split combinations. A model that cannot beat the market's accuracy is overfit, not predictive. The contamination was real. Removing it changed the CLV. The Brier failure is independent and survived the cleanup.

Finding contamination and fixing it is not enough. The clean rerun has to pass all gates. This one does not.

What a genuine signal would require

To restart this experiment honestly, the protocol needs harder gates. Brier improvement over closing as a required gate, not a diagnostic. Sign-stable LOO results before the test set is touched at all. Multi-league or multi-season test data, because one EPL season (380 games) is too small for confirmation at this effect size. At 150 bps, 380 games gives borderline statistical power and cannot separate a real edge from model miscalibration of the same magnitude. That is a different experiment from what this was.

Research system: HFM · LightGBM · 9 EPL seasons · Walk-forward OOS · Clean features v2

My crypto backtest had a sharpe of 13.4. Then I added trading costs

Pavel Kazantsev 💎 — Wed, 23 Sep 2026 19:53:44 +0000

I was not trying to prove a grand theory about crypto. I was testing simple, falsifiable baselines and looking for one result worth a second experiment.

Cross-sectional reversal looked almost too clean: Sharpe 13.44 before transaction costs, with positive returns in 67 / 76 reported out-of-sample windows. Then I charged the same signal for the trading it required. At 10 bps per side, net Sharpe fell to -40.13, and not one of those 76 windows remained positive.

That is not a typo. I genuinely did not see it coming — the pre-cost numbers looked clean enough that I spent twenty minutes checking whether I had made an error somewhere before I accepted the result. It is what can happen when a short-lived pattern asks for a new portfolio every 15 minutes.

The less glamorous part of the experiment: execution delay, turnover, fees, slippage, and funding inside the same evaluation path.

What I actually tested

This article covers one intentionally narrow reproduction of H3, a cross-sectional reversal baseline, on cached Bybit USDT perpetual data:

15-minute bars from January 2021 to the pre-lockbox boundary in September 2025;
an eight-bar, or two-hour, return signal;
90-day formation periods followed by 21-day OOS periods;
76 non-overlapping OOS windows, rolled every 21 days;
causal volatility targeting at 10% annualized volatility, using a 96-bar trailing estimate and a 3× leverage cap;
a sealed lockbox beginning September 1, 2025, which was not opened for this article.

A window counts as positive when the sum of its per-bar returns is greater than zero. That is the exact definition used by the evaluator for the window fractions below.

The current cache contains 451 symbol directories. That is not the same thing as a properly reconstructed point-in-time liquid universe. The panel uses the histories available in the cache, but the current reproduction does not apply a historical liquidity screen. Survivorship and availability bias therefore remain possible. This result is a development diagnostic, not evidence of a deployable alpha.

The signal was deliberately boring

At each timestamp, H3 measures every available asset's return over the previous eight bars, standardizes those returns across the cross-section, and reverses the sign.

Recent relative losers receive positive weights. Recent relative winners receive negative weights. The vector is scaled to unit gross exposure before portfolio-level volatility targeting.

The core signal is short enough to inspect:

recent = prices.select(
    "ts", (pl.col(symbols) / pl.col(symbols).shift(config.lookback_bars) - 1.0)
)
long = recent.unpivot(
    index="ts", on=symbols, variable_name="symbol", value_name="return"
).drop_nulls("return")
stats = long.group_by("ts").agg(
    mean=pl.col("return").mean(),
    standard_deviation=pl.col("return").std(),
)
signal = long.join(stats, on="ts").with_columns(
    signal=pl.when(pl.col("standard_deviation") > 0)
    .then(-(pl.col("return") - pl.col("mean")) / pl.col("standard_deviation"))
    .otherwise(0.0)
)
gross = signal.group_by("ts").agg(
    gross_signal=pl.col("signal").abs().sum()
)
normalized = signal.join(gross, on="ts").with_columns(
    weight=pl.when(pl.col("gross_signal") > 0)
    .then(pl.col("signal") / pl.col("gross_signal"))
    .otherwise(0.0)
)

The production implementation also handles an empty cross-section and zero variance. No model is fitted here. There is no feature search hidden behind the result.

A weight computed at time t becomes live at t+1. The evaluator enforces that delay centrally:

def live(column: str) -> pl.Expr:
    return pl.col(column).shift(1).fill_null(0.0)

Without the shift, the backtest would earn the same closing-bar return used to calculate the signal.

The number that made me stop

With fees and slippage set to zero, the current run produced the following.

I am not a professional quant. I run these experiments on my own time, publish the results as I go, and try to document what actually happens rather than what I hoped would happen. When I saw 13.44 I let myself get briefly excited. That was a mistake.

The zero-transaction-cost baseline still includes funding PnL and is not a pure gross-price-return series. Funding belongs to holding a perpetual position rather than transaction execution, so 13.44 is the baseline Sharpe before fees and slippage.

Metric	Zero-transaction-cost baseline
Annualized Sharpe	`13.44`
Positive OOS windows	`67 / 76` (`88.2%`)
Maximum drawdown	`-11.62%`
Summed absolute turnover	`29,451`

That last row is the warning. A huge pre-cost Sharpe and huge turnover can be two descriptions of the same fragile result.

The comparison above is controlled: both bars come from the same code revision, data fingerprint, walk-forward windows, signal weights, and normalization. Only the cost model changes.

Making the backtest pay its bill

Gross PnL uses execution-shifted weights. Turnover is the sum of absolute changes in live positions. Fees and slippage are fixed basis-point charges on turnover; funding is aligned separately with the long/short sign.

The accounting identity is explicit:

fees = turnover * fee_bps / 10_000
slippage = turnover * slippage_bps / 10_000
net = gross - fees - slippage + funding

This is a rejection model, not an execution simulator. It does not model spread variation, order-book depth, queue position, partial fills, market impact, or adverse selection.

Using 6 bps fees plus 4 bps slippage per side, the same every-bar signal produced:

Metric	Net result
Annualized Sharpe	`-40.13`
Positive OOS windows	`0 / 76`
Cost / gross alpha	`4.00×`
Total return	approximately `-100%`

I was hoping the cost model would hurt but leave something survivable. It did not. At this point I considered just not publishing — a Sharpe of -40.13 is not a great headline. But the result is real, the methodology is documented, and a clean failure is more useful to me than a buried one.

The extreme Sharpe is less mysterious than it looks. A continuously refreshed cross-sectional portfolio creates persistent turnover. A relatively stable negative cost stream, annualized from 15-minute observations, can generate an absurdly negative ratio.

The useful result is not the theatrical -40.13. It is that the modeled trading bill was four times the gross PnL.

Slowing the strategy down

The obvious response was to hold weights longer. Under a deliberately pessimistic 15 bps per-side scenario, net Sharpe improved as turnover fell:

Rebalance interval	Net Sharpe	Positive OOS windows
Every bar / 15 minutes	`-66.00`	`0 / 76`
Every 16 bars / 4 hours	`-16.24`	`0 / 76`
Every 96 bars / 1 day	`-4.45`	`10 / 76`

The friendliest tested corner used daily rebalancing and 3 bps per side. It reached a Sharpe of -0.01, with 39 / 76 positive windows. A 7.5 bps daily scenario produced -1.70, with 27 / 76 positive windows.

Slower trading removed much of the damage, but it did not reveal a robust net edge. The signal decayed while it waited. And the maker-style scenario is especially optimistic because this vectorized test says nothing about queue position or whether a passive order would actually fill.

What failed, and what did not

The experiment does not establish that “crypto mean reverts.” It shows a strong pre-transaction-cost short-horizon reversal pattern inside this particular development sample and cache. It also shows that the naive implementation cannot capture that pattern after even simple modeled friction.

Those are different conclusions:

Research lead: relative two-hour moves contain a repeatable pre-cost pattern in this sample.
Failed strategy: resizing the book continuously consumes more than the pattern earns.
Open engineering problem: reduce turnover without waiting so long that the signal disappears.

That leaves specific follow-ups: threshold entries, hysteresis bands, cost-aware sizing, historical universe reconstruction, and a fill model for passive execution. None is assumed to work. They are simply better questions than adding more indicators to the same high-turnover baseline.

Reproduction boundary

The internal HFM repository reproduces the article slice with two commands once the market-data cache exists:

PYTHONPATH=src python scripts/reproduce_round1_h3_article.py
PYTHONPATH=src python scripts/make_article_h3_charts.py

The first command rebuilds the panel and 76 walk-forward windows, evaluates all seven cost/rebalance scenarios, and writes machine-readable results. It also records the SHA-256 fingerprint of 1,442 cached files (880,708,217 bytes), the hashes of every relevant source module, and the Python and Polars versions. The second command creates both figures directly from that result file.

The public repository contains the article, result snapshot, figures, minimal signal/cost code, deterministic tests, and a manifest that fails CI if any frozen artifact drifts. It does not distribute the market-data cache, so a reader can reproduce the mechanics and verify the published evidence package, but cannot independently regenerate the historical metrics without sourcing the data.

Before treating the numbers as anything stronger than development evidence, keep four limits attached:

The cached-symbol universe is not a point-in-time liquidity universe and may contain survivorship or availability bias.
Trading costs are fixed-bps assumptions, not observed live fills; maker scenarios omit queue and fill risk.
The sealed lockbox was not evaluated, and these are backtest results rather than live returns.
The 76 OOS windows do not overlap, but they are not 76 independent market regimes.

The result

I started the experiment hoping to find an edge. What survived was a better research rule: turnover belongs inside the hypothesis, not in the cleanup after a backtest looks good.

A pre-cost Sharpe is not a strategy. It is a claim before execution. The strategy begins with what remains after the position delay, turnover, fees, slippage, funding, universe construction, and untouched data have all had their turn.

In this run, almost nothing remained.

If you have watched a promising backtest collapse once execution entered the model, what did the damage: turnover, spread, fill probability, market impact, or something less obvious?

Results file SHA-256: fd1f506549b8f43ea33029acb3da760dbc88478f12f0122cf980bc725e6e85a9