DEV Community: Jeet Dhandha

An ML search loop that can't overfit its own eval — the gate is in code, not a prompt

Jeet Dhandha — Sat, 25 Jul 2026 11:55:36 +0000

I kept seeing agent and eval demos where the honesty — held-out discipline, "no metric gaming" — lives in a prompt, or in a paper's methodology section. So I tried to build the opposite: a search loop where the anti-overfitting rules are enforced in executable code, then ran it against a real external grader (MLE-bench) to see whether that discipline actually costs you anything.

The result is heldout — MIT, zero runtime deps, works as a Claude Code skill: https://github.com/jeet-dhandha/heldout

What the engine enforces — in code, not prompts

every node is bound to a real measured score — a shell command that must print SCORE=<n>; it refuses to bind a metric it can't execute
a dev-slice win cannot be kept unless an untouched held-out slice confirms it by a pre-set margin — dev-up / held-out-flat is a hard kill
the baseline only ratchets up on a fresh re-measure — no keeping by vibes
every killed branch must leave a lesson, injected into future code-gen prompts, so the loop never re-proposes a dead direction
no LLM ever grades its own work — scores come from a grader the generated code can't read or influence
the dollar budget is a wall inside the tool, not a guideline

Harness, not a Kaggle agent

AIDE / AutoKaggle / AIRA bundle the search policy (what to try next) and the ML operator (write + run the code) into one system, and enforce eval honesty by convention. heldout splits them: the engine owns only the search policy and its invariants, domain-agnostically; the operator (LLM draft → sandboxed run → parse score) is a replaceable module. MLE-bench is the proof surface, not the product.

The honest results — including where it did NOT win

All numbers are from the official MLE-bench grader on nomad2018-predict-transparent-conductors (private test, n=240). Never self-reported.

Phase 1: held-out RMSLE 0.06117 → silver medal, for $3.13 (9 LLM calls, each in a network-less container).
Phase 2 + 2b: I pre-registered an equal-dollar A/B against a dumb linear-retry loop (same model, same budget) to isolate whether the search policy itself earns its cost. Two nulls — the held-out gap stayed under my pre-set 0.005 decision bar both times (0.00479, then 0.00216). So "search beats retries" is NOT established — and I report that, because the whole point is a harness that can tell you when it didn't work.

The sharpest moment

In Phase 2b, a Random Forest node scored 0.06227 on held-out — silver-tier, and better than what the retry loop shipped. The gate killed it: its edge over the baseline was 0.00288, below the noise-calibrated 0.0036 margin, and its dev→held-out gap (0.0055) flagged overfit. A naive "ship the best held-out score among all nodes" rule would have declared a win — by 0.00004 over the bar. The honest last-kept rule refused it → null.

That refusal is the product.

Honest limitations

n=1 run per arm, one task — a controlled A/B, not a statistical claim; the Phase 2b direction flip is inside single-draw noise (the retry arm drew weak, 6 of 10 attempts empty-drafted). This is a harness plus a methodology, not a SOTA result.

Repo, pre-registrations, full trees, and grader ledgers: https://github.com/jeet-dhandha/heldout

I'd love to be told the comparison is still too generous to the search arm, or where the enforcement has a hole.

qs-fast: a drop-in faster qs (2.3 parse, 3.8 stringify, identical output)

Jeet Dhandha — Fri, 17 Jul 2026 13:30:35 +0000

qs is the query-string parser behind Express's extended query parsing and a huge amount of HTTP-handling code. It's careful, spec-faithful, and security-hardened — it has to be, because it parses untrusted input. That carefulness also means it does a lot of work on every call, even for the boring, common case of a=1&b=2&c=3.

qs-fast is a drop-in replacement that fast-paths that common case — 2.3× faster parse, 3.8× faster stringify — with byte-for-byte identical output, and no loss of qs's safety behavior.

Fast-path the easy case, fall back for everything else

The core idea: qs-fast handles the default-options path with a hand-rolled parser and stringifier, and transparently bails to real qs the moment anything non-trivial shows up:

non-default options (custom delimiter, allowDots, custom charset, …)
deep nesting (depth > 5)
__proto__ / prototype-pollution vectors
array index overflow
cyclic object input
anything else outside the fast lane

So you never trade away qs's correctness or its security hardening — you just skip its overhead when it's provably safe to. qs is a runtime dependency, and encoding goes through qs's own qs/lib/utils so the bytes match exactly.

// before
import qs from "qs";

// after
import qs from "@libs-jd/qs-fast";

npm i @libs-jd/qs-fast

The benchmark

Measured on Node v24 / V8, best-of-7, against qs@6 — every case checked for identical output first:

op	qs	qs-fast	speedup
`stringify` (typical objects)	0.0277 ms	0.0072 ms	3.8×
`parse` (typical query strings)	0.0634 ms	0.0275 ms	2.3×

In the correctness gate, the fast path covered 96% of parse and 100% of stringify workloads; the rest fell through to real qs and still returned identical results.

The part that matters: identical output

Same rule as any honest optimization — prove it before you trust it. qs-fast ships a 142-case gate (nested objects, arrays, encoded characters, edge cases, and the fallback triggers themselves) asserting the output is identical to real qs. 0 mismatches, run in CI on every change. When in doubt, it defers to qs, so the failure mode is "as fast as qs," never "wrong."

diff-fast: a drop-in faster jsdiff (2.6–4.8 , identical output)

Jeet Dhandha — Fri, 17 Jul 2026 13:29:29 +0000

If you render diffs, do three-way merges, or build review tooling in JavaScript, there's a good chance jsdiff (the diff package) is already in your dependency tree — it's one of the most-depended-on packages on npm. Its Myers-diff core is correct and battle-tested. But the hot loop leaves real speed on the table.

diff-fast is a drop-in replacement that reclaims it — 2.6×–4.8× faster on the common diffChars / diffLines paths, with byte-for-byte identical output.

Drop-in means drop-in

It re-exports the entire diff API unchanged and only shadows diffChars and diffLines with a faster implementation. Same function signatures, same options, same return objects. The whole migration is one line:

// before
import { diffChars, diffLines } from "diff";

// after
import { diffChars, diffLines } from "@libs-jd/diff-fast";

Everything else in the API (diffWords, applyPatch, structuredPatch, …) is passed straight through to the real diff, which is a runtime dependency — so you're never running an incomplete fork.

npm i @libs-jd/diff-fast

The benchmark

Measured on Node v24 / V8, best-of-7, against diff@9 — comparing element-for-element identical output on every case:

workload	jsdiff	diff-fast	speedup
`diffChars`, 600 chars, ~15% edits	3.06 ms	0.63 ms	4.8×
`diffChars`, 1.2k chars, ~8% edits	3.55 ms	0.97 ms	3.7×
`diffLines`, 400 lines, ~10% edits	1.08 ms	0.34 ms	3.2×
`diffLines`, 150 lines, ~20% edits	0.51 ms	0.19 ms	2.6×

What actually changed

No new algorithm — same Myers diff, same result. The wins are all in how the inner loop touches memory:

Dense diagonal in a typed array. jsdiff keys its furthest-reaching paths in a plain array indexed by a signed diagonal, so negative indices push it into dictionary (hash) mode. diff-fast offsets into an Int32Array instead, keeping the whole search in packed, unboxed elements.
A flat component pool instead of allocating a fresh {count, value, added, removed} object on every edit step, which turns O(d²) object churn into index bumps.
Dropped per-iteration bookkeeping (including a Date.now() call jsdiff makes on the default path).

The part that matters: identical output

A speedup you can't trust is worthless. diff-fast ships with a 200-case gate (character and line diffs, random edit rates, identical inputs, insert/delete/replace mixes) that asserts the fast output is element-for-element identical to jsdiff via JSON.stringify equality — including the exact key order of the component objects. 0 mismatches. That gate runs in CI on every change.

One honest caveat: on tiny or identical inputs the diff early-exits, so there diff-fast is roughly neutral (~1×). The win shows up on real-sized diffs, which is where it counts.

Links

MIT licensed. If you diff big things, swap the import and let me know how it goes.

Stop re-parsing your font 100,000 times: faster bulk PDF generation with pdf-lib

Jeet Dhandha — Thu, 16 Jul 2026 21:23:37 +0000

The one line hiding in every pdf-lib batch job

If you generate PDFs at scale with pdf-lib — invoices, tickets, certificates, statements — you have almost certainly written a loop like this:

for (const invoice of invoices) {
  const doc = await PDFDocument.create();
  doc.registerFontkit(fontkit);
  const font = await doc.embedFont(fontBytes, { subset: true });
  const page = doc.addPage();
  page.drawText(`Invoice #${invoice.id}`, { x: 50, y: 700, size: 24, font });
  await doc.save();
}

It looks fine. It is not fine. embedFont() calls fontkit.create(bytes) every single time it runs. Parsing a TrueType font is not free — it walks the glyph tables, the cmap, the hmtx metrics, the whole thing. So if you render 100,000 invoices from the same TTF, you parse that identical font file 100,000 times. You are doing the exact same expensive work, throwing the result away, and doing it again on the next iteration.

I hit this while generating a few hundred thousand documents from one corporate font, watched the profiler light up inside fontkit, and realized the fix is almost embarrassingly small. So I packaged it: pdf-lib-bulk — a tiny, zero-dependency helper that parses your font once and reuses it across the whole batch.

npm install @libs-jd/pdf-lib-bulk pdf-lib @pdf-lib/fontkit

The numbers

100 one-page PDFs, one custom TTF, Apple M1. Reproduce with bun run bench/bench.ts in the repo.

Workload (100 PDFs, custom TTF)	pdf-lib naive	pdf-lib-bulk	Speedup
Variable text per document (`bulkGenerate`)	1,399 ms	347 ms	4.0×
Identical documents (`makeTemplateCloner`)	1,515 ms	48 ms	31×

Two very different wins, because there are two very different situations:

Every document is different (invoices, letters, tickets — same font, different text). The font parse is the redundant part. Cache it and you get ~4×.
Every document is identical (blank forms, certificates you fill in later, a fixed cover page). Here the entire render is redundant, not just the font. Build the document once and clone the bytes, and you get ~31× — and it climbs higher the simpler each document is.

The variable-content number is stable; the template-clone number scales with how much work you're skipping, so on very simple documents I've seen it well past 31×. The table above uses the conservative published figure.

Three tools, pick by your situation

1. Variable content per document — `bulkGenerate`

This is the common case: a batch where each PDF has different text but shares fonts. bulkGenerate is a generation loop with the font cache pre-wired. You provide your fonts and a draw callback; it hands you a fresh document and the already-embedded fonts for each item.

import fontkit from "@pdf-lib/fontkit";
import { bulkGenerate } from "@libs-jd/pdf-lib-bulk";

const fontBytes = await fs.promises.readFile("Inter-Regular.ttf");

const pdfs = await bulkGenerate(invoices, {
  fontkit,                       // parsed once, reused for every document
  fonts: { body: fontBytes },
  draw({ doc, fonts, item }) {
    const page = doc.addPage();
    page.drawText(`Invoice #${item.id} — ${item.total}`, {
      x: 50, y: 700, size: 24, font: fonts.body,
    });
  },
});
// pdfs: Uint8Array[] — one standalone PDF per invoice

Every output is a complete, standalone PDF with the font properly embedded and subset. Nothing is shared between the files themselves — only the parse step is shared, in memory, during generation.

2. Identical documents — `makeTemplateCloner`

When every output is the same bytes — a blank form, a certificate template, a fixed handout — you shouldn't be re-running layout, font embedding, or drawing at all. Build the document once, however you like, then clone it:

import { makeTemplateCloner } from "@libs-jd/pdf-lib-bulk";

// build the document once (fonts embedded once, layout once)
const templateBytes = await buildCertificateTemplate();

const cloner = await makeTemplateCloner(templateBytes);
const pdfs = await cloner.cloneMany(10_000);

Under the hood this uses copyPages to duplicate the finished pages, skipping layout, font parsing, and font embedding entirely. That's why it's an order of magnitude faster than the naive rebuild-per-document loop.

3. Just the font cache — `cachingFontkit`

Maybe you have your own carefully tuned generation loop and don't want to hand it over. Fine — take just the cache and drop it in. It's the same fontkit API, so nothing else in your code changes:

import fontkit from "@pdf-lib/fontkit";
import { cachingFontkit } from "@libs-jd/pdf-lib-bulk";

const fk = cachingFontkit(fontkit);
// ...
doc.registerFontkit(fk);                     // same API, parse happens once
const font = await doc.embedFont(fontBytes, { subset: true });

How it works (it really is this simple)

cachingFontkit memoizes fontkit.create() per font buffer. It's a WeakMap keyed by the Uint8Array you pass in, so the expensive TTF/OTF parse happens once per font instead of once per document. Because it's a WeakMap, there's no leak to manage — when your font buffer is gone, so is the cache entry. Fonts are still embedded and subset per document, so every output PDF stays standalone.
bulkGenerate is the loop with that cache pre-wired: create doc → embed cached fonts → your draw callback → save.
makeTemplateCloner loads your finished template once and produces copies via copyPages, skipping the redundant work entirely.

No magic, no clever tricks — just not throwing away the font parse on every iteration.

Why a separate package instead of a PR to pdf-lib?

Caching inside embedFont by default would change pdf-lib's memory profile for everyone, including people who embed a font exactly once. The redundant-parse problem only exists in bulk loops, so the right place to fix it is a thin bulk-oriented layer on top — opt-in, no behavior change for single-document users. pdf-lib stays exactly as it is; you reach for this only when you're generating many documents.

The details that matter

Zero runtime dependencies. It wraps pdf-lib and fontkit, which you already have. Nothing else comes along for the ride.
Drop-in. cachingFontkit is the same fontkit interface. bulkGenerate/makeTemplateCloner are small, focused functions.
TypeScript, fully typed.
MIT licensed.

If your service renders PDFs in batches, this is close to free performance — install it, swap your loop for bulkGenerate (or take just cachingFontkit), and stop paying for the same font parse a hundred thousand times.

Repo (benchmark + source): https://github.com/jeet-dhandha/pdf-lib-bulk
npm: npm install @libs-jd/pdf-lib-bulk

If it saves your batch job some hours, a ⭐ on the repo helps other people find it. Issues and PRs welcome.

Revwy is live: paste a Google Maps link, get a business website

Jeet Dhandha — Thu, 04 Jun 2026 06:31:06 +0000

Revwy is now live 🚀

Paste any Google Maps business listing and Revwy auto-builds a unique, themed website from its existing photos, reviews, and info — in seconds.

Free to start — hosted on a revwy.com address
Basic / Plus / Pro tiers as the business grows
Built from real Google Maps data, not generic templates

Try it 👉 https://revwy.com

How I stopped Claude from cloning entire GitHub repos for a 10-line snippet

Jeet Dhandha — Tue, 12 May 2026 16:43:35 +0000

TL;DR — I wrote a Claude Code skill that prevents Claude from cloning or npm install-ing a repo when I only wanted one function or one idea from it. Raw files only, into /tmp, smallest useful unit, adapted to my project's style. MIT, 70 lines of markdown. Part of jd-skills.

The pattern that finally annoyed me enough

You've done this. I've done this. You're in a Claude Code session, you paste a GitHub URL, and you say something like:

"Look at how this repo handles agent handoffs — can we do something similar?"

And then Claude goes: git clone https://github.com/..., reads 47 files, asks you which __init__.py is interesting, and 90 seconds later you're three levels deep in someone else's repo scaffolding for what should have been a 12-line concept.

Or worse — it adds the whole library to your package.json as a dependency. For one function. You now own its transitive deps, its CVE notifications, and a version pin you'll never upgrade.

The problem isn't Claude being lazy or sloppy. The problem is that "use this library" and "borrow an idea from this library" deserve completely different workflows, and there was no rule telling Claude which one I meant.

The rule

The fix is dumb-simple as a rule and surprisingly effective in practice:

When the user references a GitHub repo for inspiration (not as a dependency), never clone it and never install it. Read the README first. If the README answers the question, stop there. If code is needed, fetch raw files via raw.githubusercontent.com into the OS temp dir, lift the minimum useful unit, adapt it to the user's style, and cite the source commit SHA.

I wrote this up as a Claude Code skill — a SKILL.md with a description that triggers auto-invocation when a GitHub URL is dropped in as inspiration. Claude reads the skill on session start, and it just… does the right thing now.

Walkthrough — the concept-only case

I was working on a YC job-applier and someone pointed me at TauricResearch/TradingAgents. "Can we use this multi-agent pattern?"

Without the skill: Claude would have cloned the whole trading repo, then I'd have spent 20 minutes pruning irrelevant files. The repo is for trading. Almost none of it transfers to job applications.

With the skill: Claude started with raw.githubusercontent.com/TauricResearch/TradingAgents/main/README.md to map the rough shape — specialised role agents (analysts → bull/bear researchers debating → judge → trader → risk team) coordinated by a graph. Then it pulled a handful of source files via raw URLs to see how the pattern was actually wired: the agent prompt templates, the JSON schema each role hands back, and the graph node that routes between them. Four files into /tmp, not a clone.

That was enough to propose an analogue for my use case: JobFitAnalyst + RecruiterPersonaWriter + Critic (a role that argues against applying), orchestrated by my existing extractor pipeline — with the prompt and schema shapes borrowed from the trading agents but rewritten for jobs. We discussed the design before any code got written.

Result: the pattern was useful. The framework around it wasn't. Cloning would have been pure friction.

Walkthrough — the surgical lift case

Different prompt: "There's a clean exponential backoff in litl/backoff — can we just steal it?"

With the skill, Claude:

Pinned the SHA so the source is reproducible:

   gh api repos/litl/backoff/commits/master --jq '.sha' | cut -c1-7
   # → abc1234

Listed the tree via the API instead of cloning:

   gh api repos/litl/backoff/contents/backoff --jq '.[].path'

Identified _wait_gen.py as the target — ~40 lines of generator logic.
Fetched just that file to /tmp/sge-backoff-abc1234/:

   curl -fsSL "https://raw.githubusercontent.com/litl/backoff/abc1234/backoff/_wait_gen.py" \
     -o /tmp/sge-backoff-abc1234/_wait_gen.py

Extracted the 8-line expo generator, rewrote it as a plain function matching my scraper's style (no Pythonic generators, just a list).
Pasted it into utils/retry.py with a provenance comment:

   # adapted from github.com/litl/backoff@abc1234:backoff/_wait_gen.py
   def exponential_backoff(...):
       ...

No pip install backoff. No git submodule. Eight lines, owned by me, traceable to the source.

Why a Skill and not an MCP

Quick aside in case you're considering writing your own. People sometimes ask: should this be a Skill or an MCP server?

MCP ships new tools to Claude. You'd build a github_raw_fetch server, host it, version it, wire it into clients.
Skill ships instructions that shape how Claude uses tools it already has — WebFetch, curl via Bash, gh, Read.

This is purely a discipline layered on existing tools. Skill. Anthropic's own mcp-builder is itself a Skill, which is a hint.

Rule of thumb: if you can write your idea as a paragraph of instructions, it's a Skill. If you need to add a new verb to Claude's vocabulary, it's an MCP.

Install

mkdir -p ~/.claude/skills/surgical-github-extraction
curl -fsSL https://raw.githubusercontent.com/jeet-dhandha/jd-skills/main/skills/surgical-github-extraction/SKILL.md \
  -o ~/.claude/skills/surgical-github-extraction/SKILL.md

Or project-scoped under .claude/skills/ in your repo.

Repo: https://github.com/jeet-dhandha/jd-skills

It's part of jd-skills, a small collection of Claude Code skills I'm building. The sibling skill, code-graft, handles the case where a one-off snippet isn't enough but a runtime dep is too much — vendor only the slice of a library you actually use into your project, trim the rest, re-sync selectively from upstream. Useful for things like "I want one tokenizer out of HuggingFace transformers without the 2GB."

Issues, prompts that misfired, and "this should also handle X" reports very welcome — skills only get sharper with concrete failure cases.

If you write Claude Code skills, I'd love to see them. Drop them in the comments.

DEV Community: Jeet Dhandha

An ML search loop that can't overfit its own eval — the gate is in code, not a prompt

What the engine enforces — in code, not prompts

Harness, not a Kaggle agent

The honest results — including where it did NOT win

The sharpest moment

Honest limitations

qs-fast: a drop-in faster qs (2.3 parse, 3.8 stringify, identical output)

Fast-path the easy case, fall back for everything else

The benchmark

The part that matters: identical output

Links

diff-fast: a drop-in faster jsdiff (2.6–4.8 , identical output)

Drop-in means drop-in

The benchmark

What actually changed

The part that matters: identical output

Links

Stop re-parsing your font 100,000 times: faster bulk PDF generation with pdf-lib

The one line hiding in every pdf-lib batch job

The numbers

Three tools, pick by your situation

1. Variable content per document — bulkGenerate

2. Identical documents — makeTemplateCloner

3. Just the font cache — cachingFontkit

How it works (it really is this simple)

Why a separate package instead of a PR to pdf-lib?

The details that matter

Revwy is live: paste a Google Maps link, get a business website

How I stopped Claude from cloning entire GitHub repos for a 10-line snippet

The pattern that finally annoyed me enough

The rule

Walkthrough — the concept-only case

Walkthrough — the surgical lift case

Why a Skill and not an MCP

Install

1. Variable content per document — `bulkGenerate`

2. Identical documents — `makeTemplateCloner`

3. Just the font cache — `cachingFontkit`