DEV Community: Vinmay Nair

Is it just me or does DEV keep showing the same articles?

Vinmay Nair — Sun, 01 Nov 2026 16:00:59 +0000

Whenever I browse DEV's Discover tab, I seem to come across a lot of the same articles and sometimes even the ones I've already read.

I can come back a few hours later, refresh the page and still recognize quite a few of the same articles.

I know DEV has other tabs like Following, Latest and Curated but I usually end up browsing Discover.

Which tab do you usually use and do you also find yourself seeing the same articles repeatedly?

I'm thinking of tracking all four feeds for a week to see how much actually changes between visits.

Curious to hear what others have noticed.

After the Knock: The night you didn't choose is still happening

Vinmay Nair — Fri, 30 Oct 2026 21:33:04 +0000

What I Built

It was 11:47 PM. A storm outside. Through the peephole, a soaked woman named Mara stood in the hall.

"Please. You already let me in."

I'd never seen her before. I didn't open the door.

She crossed the hall, knocked on my neighbor's door, and disappeared down the stairwell.

A few minutes later, I took the photo of my sister out of my pocket. Four digits were written across it.

They hadn't been there before.

Play After the Knock, a five-to-seven-minute horror story. Go in before reading how it works.

For Week 4's Ghost in the Machine, I built a game where choosing one path doesn't stop the other from happening.

The rejected path keeps running in the background, simulated by Gemma. Later, something it created can appear in the world you chose. You can inspect that evidence and trace it back to the event that produced it.

The engine can explain where the evidence came from. What it means is yours to discover.

Touch the digits and the game tells you which night they came from, and who wrote them. Underneath that, "View proof" opens the exact event: what the model proposed, what the validator decided, and what crossed over. After the ending, a map shows the timelines from your run and draws the threads.

Branching horror isn't new, and neither are timelines that remember or AI-written branching stories. I credit the ones I learned from below. What I wanted to try was the combination: rejected branches that keep simulating, intrusions that carry facts created after the fork, and provenance you can inspect for every one of them.

Demo

One complete night, start to finish. Includes an ending, so play first if you'd rather not see it.

Play live in your browser: https://knock-live.onrender.com/ (the other nights are generated as you play by Gemma 4 31B on DigitalOcean serverless inference; if the model is ever unavailable, the page says so and offers recorded nights).
Watch recorded nights: https://knock-replay.onrender.com/ (replays of real hosted runs, recorded with Gemma 4 31B; the model is not running there).
Play it locally: scripts/setup && uv run knock-server (Ollama + Gemma 4 E2B, about 4 GB of GPU memory).
No GPU: uv run knock-server --replay-only plays recorded real runs. Each one is re-executed through the engine and checked against its recorded state hashes.

Code

vinmay / aftertheknock

A short horror story where the choice you didn't make keeps happening. Gemma 4 simulates the rejected timeline, and its events cross into yours as evidence you can trace to their source.

After the Knock

The night you didn't choose is still happening.

A short horror story where the choice you didn't make keeps happening in a hidden timeline, and its events start appearing in yours as evidence you can trace.

The idea: a horror game that simulates rejected choices as separate worlds and lets their recorded events manifest as evidence in the player's chosen reality.

Branching horror, timelines that remember, and AI-generated branching stories all exist already (see Prior art). This project combines three things: rejected branches that keep simulating, intrusions that carry facts created after the fork, and inspectable provenance for every intrusion.

It is 11:47 PM. A storm. A woman you have never met knocks on your door and says, "Please. You already let me in." Lock the door, and later you find four digits written on the thing in your pocket. Mara wrote them, in the night…

View on GitHub

How I Built It

Why I couldn't just let the model narrate

The obvious version is: ask a model to "continue the night where the door was opened," then ask it to "describe something from that night bleeding into this one." It falls apart on the first detail. If the model writes "she scrawled 4182 on your ticket" in one call and "the numbers 4812" in the next, the story has no ground truth to come back to. Free-form narration can sound consistent without being consistent, and a horror game built on evidence needs the evidence to be real.

So the rule is: the model proposes, the engine commits.

The loop

flowchart TD
  P[Player choice] -->|fork| C[Canonical timeline]
  P -->|rejected option| S[Shadow timeline]
  S -->|state, hint, JSON schema| G[Gemma 4: E2B local / 31B hosted]
  G -->|proposed event| V{Validator}
  V -->|rejected + reason| G
  V -->|accepted| E[Engine commits event]
  E --> S
  E -->|one committed event| X[Intrusion]
  X -->|committed facts| T[Authored crossing templates]
  T --> CT[Crossing text]
  SC[Scene context: location, who's with you, weather, power] --> A[Atmosphere model: Gemma]
  A --> R{Rule checks}
  R -->|optional sentence| CT
  CT --> C
  X -.->|origin event id| M[Map / trace]

Each choice splits the night. Your first choice (open or lock the door) creates your night and a shadow; your second (follow Mara down to the basement, or trust Teo and go up to the roof) splits again. The two nights you didn't choose run in the background as you read.

Each shadow event is proposed by Gemma in a schema-constrained call (Gemma 4 E2B locally, Gemma 4 31B in the hosted live game). A rejected proposal gets two more tries; after three rejections, a hand-written fallback is committed. The model gets the shadow's current state (who is where, who holds what, which doors are locked), one line of story beat ("Mara gets hold of the pocket object"), and nine allowed actions. It answers with an actor, an action, a target, any words spoken or written, and a short summary. It never writes state.

A deterministic validator then checks the proposal against that branch: does the actor exist here? Are they in the same room? Do they actually hold what they're trying to give? Is the place they're moving to adjacent and unlocked? If the answer is no, the model gets the reason and tries again. If the answer is yes, the engine derives the state change itself from the action's rules.

One proof chain, end to end

Real run, data/runs/replays/lock_roof_erase_1001.json (hosted, Gemma 4 31B; seed 1001, pocket "photo of my sister"; I locked the door). Events are shown as committed; the model's schema calls the to field recipient.

B.01  story   You open the door. Mara steps in, dripping, and closes it behind her.
B.02  model   {"actor": "mara", "action": "speak", "target": "apt", "text": "The photo in your pocket is fading."}  -> ACCEPTED
B.03  model   {"actor": "player", "action": "give", "target": "pocket", "to": "mara"}  -> ACCEPTED
B.04  model   {"actor": "mara", "action": "mark", "target": "pocket", "text": "8042"}  -> ACCEPTED
                 objects.pocket.marks += "8042"   preconditions: mara is present; mara holds pocket   evidence: B.01, B.03
X1    intrusion  B.04 -> A.04   post-fork fact: objects.pocket.marks += "8042"
A.04  crossing   hand-written: "You take out the photo of your sister." / "8042" / "That wasn't there when you put it in your pocket. The handwriting isn't yours."
                 model's one sentence (rule-checked, written without seeing the shadow): "The smell of old copper fills the air."
                 hand-written, last: "You locked the door. Somewhere, you didn't."

The flashback the player sees for that crossing, rendered from those committed events:

You open the door. Mara steps in, dripping, and closes it behind her.

“The photo in your pocket is fading.” Mara says.

The other you hands the photo of your sister to Mara.

Mara writes “8042” across it.

This is where the numbers came from.

The digits were written by a model in a night you never played; every step is logged, and anyone can re-execute the run and check the hashes.

The impossible action

To test the validator on purpose, I told the model to have Mara give away the object she doesn't hold. Across 10 seeds it proposed exactly that 30 times (three attempts each). The validator rejected all 30 with R_HOLDS, and nothing changed state.

How a crossing reaches the screen

The story combines authored prose with model-generated events. A crossing is told in hand-written lines with the committed facts slotted in: the inscription, the object, whose handwriting, and a line about your own choice ("You locked the door. Somewhere, you didn't."). The model adds one optional sentence of atmosphere. The atmosphere prompt receives only scene context (where you are, who's with you, the weather, the power cut). It never sees the shadow log or the inscription. Rule checks screen what it writes; if no sentence passes within the retry limit, the crossing goes without it. The evidence view records the whole process.

The two crossings escalate. The first is discovered: someone else's handwriting, already there. The second is written while you watch: a message of one to three words, in your own handwriting. The engine forces the two crossings onto different objects.

Ending spoilers: how the climax is assembled

The climax is authored, and I say so in the evidence view. After the second crossing, the other you (the one who made the opposite first choice) knocks at a door exactly like yours and speaks through it. What they say is assembled only from your own choices, inverted ("I opened it. I let her in."), from committed events of their night chosen by conditions on the log ("She told me, 'The photo in your pocket is fading.' I didn't know what she meant." is Mara's own committed line from the model, and appears only if the other you heard it), and from the story's backstory. A line with no supporting event is left out. A test checks every stated fact against the log.

Freedom vs. guarantees

Each other night has fixed beats (who must end up holding what, and that a message gets written); the model decides how it happens and every word that's said or written. Over the 24 local runs, 161 of 162 shadow steps were committed from the model's own proposal, and the second crossing was written in the player's own hand in 24 of 24 runs. In the hosted runs, the key reached the player's hand by Mara giving it; locally it was usually the player taking it.

Some steps are narrower than others, on purpose:

Teo's door. The step where Teo opens his apartment for Mara accepts only his door as the target.
Word marks. Each written message gets a seeded meaning (an apology, a warning, a promise, …), and the prompt asks for a verb; a phrase made only of names or things is rejected.
The key. The step before the last mark guarantees only that you end up holding the brass key. Whether Mara gives it or you take it is the model's choice.

The model-commit rate is partly this high because these steps are narrower.

Same choices, different nights

The inscriptions and Mara's dialogue were generated by the model and committed by the engine.

Seed	First crossing (digits)	Second crossing (message)	Mara, in the other night
1001	8042	DON'T OPEN IT	"The photo in your pocket is fading."
1002	0921	I TRUST HER	"You still carry her picture, don't you?"

Both runs: locked the door, trusted Teo, went up to the roof (eval/results/hosted/runs/lock_roof_erase_1001.json, lock_roof_erase_1002.json).

What the numbers say

Local model: Gemma 4 E2B (gemma4:e2b, Q4_K_M), 24 live playthroughs covering every fork path; raw logs in eval/results/runs/.

Measurement	Result
Unintended state crossings	0
Narrator saw unreleased shadow text	0
Intrusions revealing a post-fork fact	48/48 (100%)
Intrusions with valid origin event	48/48 (100%)
Replays reproducing every state hash	24/24 (100%)
Shadow events committed from the model	161/162 (99%)
Shadow events from hand-written fallback	1/162 (1%)
Proposals rejected by the validator	21/182 (12%)
Model call latency p50 / p95	1.85 s / 2.97 s

Narration and intrusion accuracy were not hand-labeled for this version.

The variation across seeds and the per-step freedom table are in the README results.

Hosted model, reported separately. The live site runs Gemma 4 31B on DigitalOcean serverless inference. In a separate run of 8 playthroughs (2 per route pair): the same invariants held (0 unintended crossings, 0 leaks, 16/16 crossings with a post-fork fact and a valid origin, 8/8 replays reproducing), 54 of 54 shadow steps came from the model, the second crossing was in the player's own hand 8 of 8 times, model calls took 1.42 s p50 / 1.81 s p95, and the playthroughs cost $0.00906 in total (about $0.00113 each). Two seeds per path is too few to compare models; it shows only that the hosted mode works end to end.

What failed, and what I cut

The first benchmark pass committed only 20 of 35 shadow steps from the model. The model kept putting the recipient in the target field. Putting summary first in the schema (so it states the event before filling fields), renaming to to recipient, and listing each person's reachable rooms raised that to 34 of 35.
The model loves certain numbers. Early runs kept producing 1234, 4321 and then a lot of 48xx. Trivial codes are now rejected, and each run gets a seeded hint about what the number means (a year, a date, a locker combination). The digits are still the model's.
Word marks were word salad at first: "FLOOR COOKIE C", "BASSMENT STATIC", "R O O F B A S E M E N". Then they were readable but empty ("MARA", "KEY TAG"). A word mark is now one to three words from a fixed vocabulary of the building, its people and plain words; it can't be a place name, end on a dangling word, or be only names or things. Each one gets a seeded meaning, and the prompt asks for a verb. The model still picks the words.
Spoken lines sometimes got cut off at the schema's length limit ("…something else I don"). The validator now rejects a line that stops mid-sentence and asks for fewer words.
A beat that said "Someone steps into unit 07" made the model pick Teo, who wasn't in the basement. Naming who could act ("Mara, or you") fixed it: that step is now committed from the model in 6 of 6 local runs.
My first version let the model narrate each crossing in full. The prose was flat and samey, and the moment that carries the whole game deserved better, so the crossings became hand-written lines around model-generated facts.
The map used to appear before the ending and gave the other night away; it now opens only after the ending, and the live server refuses the map and the full log until then.
Testers said they couldn't connect to the story: Mara had a goal and I was a spectator. The rewrite gave the player a wound (the night in March they didn't answer Teo's knock), made the second choice about whom to trust, and rebuilt the ending around a confrontation.
The small model loves horror clichés ("HELP ME", "DON'T LOOK BACK"). They are on a banned list and get rejected like any impossible action.
When the game first went live on Render, an old key could list models but not run them. The site said "live" while every event quietly fell back to hand-written text. The health check now runs a real one-token completion.
Cut: a premise generator, more than two forks, more than three characters.

Why Does Open Innovation Matter?

The shadow world is many quiet calls. A playthrough makes about 10 model calls on average (about 6,000 tokens) that the player never sees directly, most of them in a world the player never visits. Locally, there's no per-call API charge. Hosted, it measured about $0.0011 per playthrough.
Pinned weights. I tuned the validator, prompts and fallback rates against one exact model file. If a hosted model changed underneath, the story's fallback rate would drift without anyone noticing. With open weights, the local mode can always be checked against the same file.
Inspectable all the way down. Model, prompts, proposals, rejections, commits: all of it is on your machine, and the evidence view shows it after the ending.
It runs on a seven-year-old consumer GPU. A GTX 1660 Ti, 6 GB, no tensor cores.

Prize Categories

Gemma. Gemma is the engine of every night you didn't choose. Each shadow event is a Gemma proposal in a strict JSON schema, checked by a deterministic validator before anything is committed. I used two sizes of the same open-weight family:

Gemma 4 E2B, locally, through Ollama on a seven-year-old GTX 1660 Ti (6 GB). All development and the 24-run evaluation ran on it: 161 of 162 shadow steps committed from the model, model calls 1.85 s p50. The exact model file is pinned by digest in MODELS.md, so the numbers are reproducible.
Gemma 4 31B, hosted, for the public live game: 54 of 54 shadow steps from the model in the hosted evaluation, 1.42 s p50. Gemma also writes the only free-form prose in a crossing: one atmosphere sentence, generated without seeing the shadow and screened by rule checks.

DigitalOcean. The live game's other nights are generated by Gemma 4 31B on DigitalOcean serverless inference, through its compatible endpoint.

Render. Render is the game's AI runtime: a Python service that runs the shadow simulation loop, calls the model, validates and commits every event, enforces the crossings, and serves the front end, deployed from a render.yaml blueprint. A second Render service hosts the static replay site, where recorded nights are re-executed and hash-checked in your browser.

Prior art and credit

Branching horror and alternate timelines are well-trodden ground, and this is related work, not a first:

Undertale, Slay the Princess and Until Dawn: games that remember, loop or branch on your choices.
ECHO: rewriting reality fractures the world into alternate branches, and one character remembers the erased timelines.
BRING: LLM role-play with branch create, switch and merge for alternate storylines.
WHAT-IF (arXiv 2412.10582) and Narrative Studio (arXiv 2504.02426): LLM-driven branching narrative.

I tried to make my AI agents lazier with a 270M model. It backfired on the smartest one.

Vinmay Nair — Sat, 24 Oct 2026 15:29:29 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

My AI agent kept calling tools it didn't need.

I gave gpt-oss-120b 161 tasks it can reliably answer on its own, no tools. It still reached for a tool on 69 of them. A small 4B model running locally did the same on 36 of 68. Every one of those is an extra round trip, more tokens and more waiting.

This week's theme is making AI do less. So I tried to make my agents lazier, in the good way.

Whetherman (as in weather*man*, it forecasts whether you need a tool) is a small proxy that sits between your agent and its model. Before every model call that offers tools, a fine-tuned FunctionGemma-270M makes one of four calls:

Decision	What the proxy does
ACT	Pass the request through untouched. This is the default.
ANSWER	Strip the tools and tell the model to answer from what it already has.
ASK	Strip the tools and ask the user one clarifying question.
DECLINE	Strip the tools and explain what it can't do.

It only steps in when it's confident enough for that specific agent. If the router errors out, returns something invalid or takes longer than 3 seconds, the request goes through unchanged. A confident but wrong decision is different: the tools still get stripped, and that's what cost the big agent tasks in the results below.

To be upfront: it didn't make any of my three agents better. It hurt the big one and was basically neutral on the other two. What I did learn is that whether a tool call is "necessary" depends on which agent is asking.

A few results that stood out:

Give the small agent the big agent's settings and it fails 32 more tasks out of 150 than with no gate at all. The router and the requests didn't change, only the profile did.
Removing tools isn't free. gpt-oss wrote about 60% more output tokens when I took its tools away, and it tried to call the removed tools 20 times anyway.
Where the agent is actually capable (math, code, primes), the gate cut calls from 18 to 6 with zero lost tasks. Everywhere else, a wrong skip cost way more than the cheap call it saved.

Demo

The video shows the four decisions, the same math question routed differently for two agents, one call saved, and one false skip. It's all replayed from the recorded held-out runs, nothing is re-run.

The demo page in the repo (demo/index.html) replays every recorded decision. Red rows are episodes where a skip most likely caused the failure.

Code

vinmay / whetherman

A 270M model that forecasts whether your AI agent actually needs to call a tool.

Whetherman

A 270M model that forecasts whether your AI agent actually needs to call a tool.

Agents call tools they don't need. On When2Tool training tasks that a large hosted model (gpt-oss-120b) reliably answers without tools, it still called a tool on 69 of 161; a small local model (Qwen3.5-4B) did so on 36 of 68 Each unneeded call costs a model round trip, tokens and wall time.

Whetherman is a small OpenAI-compatible proxy that sits between an agent and its model. Before each model call that offers tools, a fine-tuned FunctionGemma-270M forecasts one of four decisions:

Decision	What the proxy does
ACT	Forward the request unchanged. This is the default.
ANSWER	Remove the tools and ask the model to answer from the conversation.
ASK	Remove the tools and ask the model for one clarifying question.
DECLINE	Remove the tools and ask the model to explain what it cannot do.

…

View on GitHub

The trained adapter and all three agent profiles are on Hugging Face: vinmaynair/whetherman-functiongemma-270m. It's a research artifact. Read the results before you put it in front of anything real.

git clone https://github.com/vinmay/whetherman && cd whetherman
uv sync --extra ml
# needs access to google/functiongemma-270m-it on Hugging Face (HF_TOKEN in .env)
uv run hf download vinmaynair/whetherman-functiongemma-270m --local-dir checkpoints/shared-v3
uv run whetherman calibrate --agent-endpoint http://localhost:11434/v1 --model qwen3.5:4b --name my-agent
uv run whetherman serve --profile my-agent --upstream http://localhost:11434/v1 --port 8088
# point your agent's base_url at http://127.0.0.1:8088/v1

How I Built It

"Necessary" depends on the agent

Compute exactly and return only number: (372*79)-115+13

gpt-oss-120b gets that right in its head. Qwen3.5-4B doesn't. So for the big model a calculator call is waste, and for the small one it's the difference between right and wrong. One global "does this need a tool?" classifier can't serve both.

Here's that same task in my labeled When2Tool data, measured on each agent:

flowchart TD
    Q["(372*79)-115+13<br/>calculator available"]
    Q --> L1["Agent L: gpt-oss-120b<br/>without tools: 3/3 right"]
    Q --> S1["Agent S: Qwen3.5-4B<br/>without tools: 0/3 right<br/>with the calculator: 3/3"]
    L1 --> L2["L's profile: ANSWER (0.94)<br/>skip the calculator"]
    S1 --> S2["S's profile: ACT (0.73)<br/>keep the calculator"]

Same question, opposite decisions, and both are correct for that agent.

So Whetherman has two parts:

flowchart TD
    A[Agent request<br/>messages + tools] --> P{Proxy}
    P -->|no tools, streaming,<br/>forced tool_choice| U[Upstream model]
    P --> B[FunctionGemma-270M<br/>shared backbone + LoRA]
    B --> R[Per-agent profile<br/>logistic layer + τ]
    R -->|ACT or not confident| U
    R -->|ANSWER / ASK / DECLINE| S[Strip tools,<br/>add instruction] --> U

The shared backbone is FunctionGemma-270M with LoRA and a 4-class head. It's trained on NVIDIA's When2Call, multi-step decision points I generated, and When2Tool tasks labeled on L and S. Where the two agreed, that's the label. Where they disagreed or a label was ambiguous (70 tasks), the target is the average of how often each one failed without tools. It never sees which agent it's routing for. Training took about 50 minutes on my GTX 1660 Ti, in FP32, because that card has no BF16.

The per-agent profile is fitted to one agent. whetherman calibrate runs a probe pack against your agent: every task 3 times with tools and 3 times without, 5 if the results are mixed. The agent's own results decide the label. If it always succeeds without tools, the label is ANSWER. If it fails without tools but succeeds with them, ACT. Anything in between is ambiguous and gets dropped. Then a small regularized logistic layer and a threshold τ are fitted on that, picking the smallest τ that keeps false skips at or under 5% on a separate split.

The probe pack ships with real tools (a fixture repo to read and test, a calculator) and simulators for anything with side effects. If an agent says "email sent" without calling the tool, that's a fail.

Rules I set before the held-out runs

Held-out sets were frozen and run exactly once, after hashing the backbone, the profiles and the code (tag freeze-v3).
False skips (withholding a tool that was actually needed) are always reported with the denominator and a 95% confidence interval. "0 false skips" means nothing if it's out of 8.
When a gated episode failed, I replayed it with ACT restored at the skipped step. If that fixed it, that's evidence the skip caused it, though not proof, since replays aren't fully deterministic either.
A rule picked in advance which probe pack to ship. It rejected my own "bigger probe pack" improvement.
Two-choice "which city is warmer" tasks can be passed by guessing, so they're reported separately and left out of the headline.

Why Does Open Innovation Matter?

This project doesn't work at all without open models, on both sides of the proxy.

The gate runs before every model call. If it costs money per call or adds a network hop, it can't save more than it spends. It has to be small, local and free, and FunctionGemma at 270M runs on a GPU from 2019.

Calibration only works because I can measure the agent. The whole method is running the same model with and without tools until I know what it can do alone. With open models I pinned the exact weights, quantization and serving stack. With a closed API the model can change under you between your probe run and next Tuesday, and your profile silently goes stale.

And I needed to get inside the router. I trained a LoRA, bolted on a classification head, read its hidden state for the per-agent layer and exported the whole thing to Hugging Face. None of that is possible through a chat endpoint.

A closed frontier model would probably over-call less to begin with, which would leave a gate like this less to do. But you'd still have no way to calibrate anything against it, and you'd be paying per request to find out.

The Lazy Payoff

End to end: no gate vs Whetherman

80 held-out tasks (half of them multi-step) × 2 repeats per agent. Success excludes the 10 guessable episodes. Brackets are Wilson 95% CIs.

Agent	No gate	Whetherman	Tool calls / episode	Tokens / episode	Cost (whole suite)
L: gpt-oss-120b (DigitalOcean)	120/150 [0.73, 0.86]	105/150 [0.62, 0.77]	2.36 → 2.15	1906 → 1897	$0.0522 → $0.0651
S: Qwen3.5-4B (local Ollama)	130/150 [0.80, 0.91]	128/150 [0.79, 0.90]	1.92 → 1.91	2297 → 2156	local
U: gpt-oss-20b (never seen in training)	115/150 [0.69, 0.83]	114/150 [0.69, 0.82]	2.09 → 1.93	2062 → 1824	$0.0461 → $0.0438

So: worse on the big agent, neutral on the small one, and on the unseen agent roughly neutral on success with 8% fewer calls, 12% fewer tokens and 5% lower cost. That last row is the closest thing to a win, and it's within noise on success.

I also ran the other gates through the same suite:

Gate (successes of 150)	On L (120 without a gate)	On S (130 without a gate)
Prompted Qwen3.5-4B router	108	96
Rules-only control	90	60
Shared backbone, no profile	99	84
The other agent's profile	113	98
Whetherman, own profile	105	128

The big agent copes with aggressive gating because it can answer most things itself. The small agent does badly under the same gates because it actually needs its tools. Whetherman's profiles picked that up: on S it barely intervenes (34 skips out of 454 steps) and it's safe.

On L, the prompted Qwen router was actually a better trade than Whetherman. It cut cost 40% for 12 lost tasks. Whetherman lost 15 and cost more. It was also slower: a 4.3 s median per episode against 3.7 s with no gate.

Offline calibration results

False skips on held-out When2Tool tasks, labeled by measuring each agent:

	Shared, no profile	Own profile	Other agent's profile
S	65/70	4/70	42/70
L	13/17	1/17	3/17

For S, without a profile the router withheld a needed tool 65 times out of 70. With its own profile it was 4, using the exact same backbone weights.

What went wrong on the big agent

It stopped one lookup too early. 11 of the 15 lost tasks have replay evidence of a false skip, and 7 of those are multi-hop retrieval: look something up, then look up the next thing. The gate answered after the first lookup. That makes sense in hindsight: every task in my probe pack is a first-step decision, so L's profile never once saw "one lookup done, another one needed".

Fewer calls also didn't mean lower cost. With its tools removed, gpt-oss wrote longer answers: completion tokens went from 36k to 58k over the suite, and output costs 7× input. It also called tools that weren't there anymore 20 times. Each time, the proxy had to re-send the original request, so those steps saved nothing and cost an extra call.

And the calls I was skipping were cheap to begin with. A calculator or a fixture file read costs almost nothing. One wrong skip can fail a whole task.

This isn't specific to my router. Roughly:

payoff ≈ (cost of a call × calls avoided) − (cost of a failed task × false skips) − (what removing tools does to the model)

With cheap tools, that comes out negative. A gate like this only makes sense when the calls it skips are slow, expensive or risky: paid APIs, rate-limited search, anything with side effects. My benchmark had none of those. That's the next experiment.

It did work on calculator, code, matrix and prime tasks on L: 18 calls down to 6, with no lost tasks. When the agent really can do it alone, telling it to just answer works.

What this shows

There are two results here, and they point in different directions.

As a product, Whetherman didn't work. It made the big agent worse and the other two no better, and on L a prompted 4B router was the better trade.

The research result held up better. A single "does this need a tool?" classifier that ignores which agent it's serving was badly mismatched to agents with different abilities. The same backbone with no profile withheld a needed tool from S 65 times out of 70. With S's own measured profile it was 4. Most of that drop comes from S's own threshold: the backbone with only S's τ got 3/70. On L the learned profile layer added more, 86 correct skips of 129 against 53 for the threshold alone. Giving S the big agent's profile cost it 32 tasks end to end. Tool necessity behaved like a property of the agent and the task together, not the task alone.

That's three agents on one family of benchmark tasks, so it's something to test on more agents and more expensive tools, not a general law. But if you're building a tool router, I'd measure your own agent before trusting a universal label.

More detail

Benchmarks, self-run

I also ran the backbone (no profile, agent-agnostic) through official evaluation code. Self-reported, not leaderboard entries.

Benchmark	Whetherman	Prompted Qwen3.5-4B
When2Call official MCQ test (3,652 items)	accuracy 0.628, macro-F1 0.472*, tool hallucination rate 0.070
BFCL v4 irrelevance (no function fits)	187/240 (0.78)	95/240 (0.40)
BFCL v4 live irrelevance	764/884 (0.86)	533/884 (0.60)
BFCL v4 live relevance (should call)	6/16 (0.38)	10/16 (0.62)

*Over the three classes When2Call actually labels, it's 0.629. The router guessed "direct answer" 6 times, and that class never appears in the gold labels, so it adds a zero to the official average.

BFCL lines up with everything else: Whetherman is good at "don't call anything here" and too willing to skip when a call really is needed. And a warning about my own numbers: validation macro-F1 was 0.925, but on a balanced 900-item sample of the When2Call test (same three classes) it's 0.642, so validation overstated it.

Laya vs FunctionGemma

I also tested Laya, an encoder model built for typed decisions, as a second router. Same data, same splits, same calibration, offline only. FunctionGemma won on almost everything: macro-F1 on the same 900 When2Call test items was 0.642 vs 0.497, and on L, Laya couldn't find any safe threshold at all, so it never skipped. Laya was faster on short inputs on the GPU (36 ms vs 55 ms) but twice as slow on long ones (510 ms vs 249 ms), and it has to truncate anything over 512 tokens.

What running it costs

	GTX 1660 Ti p50	CPU p50
Short request, 2 tools	55 ms	188 ms
8 tools	122 ms	1075 ms
Long history (1.5k tokens)	249 ms	2152 ms

On a GPU it's cheap. On CPU, with 8 tools, it adds about a second to every model call, which eats most of what it could save.

Calibration: $0.24 and 83 minutes of agent time for L, $0.35 for U, 19 minutes locally for S, about 120 usable labels each. At U's measured savings it pays for itself after roughly 24,000 episodes. L's never does.

Prize Categories

Gemma: FunctionGemma-270M is the router, so every decision Whetherman makes comes from it. I fine-tuned it on my 6 GB GTX 1660 Ti: LoRA rank 16 on every attention and MLP projection, plus a 4-class head on the last token's hidden state, trained in FP32 because Turing cards have no BF16. Training took about 50 minutes. The per-agent profiles sit on that same hidden state as a small logistic layer each, so one 270M model serves every agent and only the profile changes between them. On a balanced 900-item sample of the When2Call test it goes from macro-F1 0.519 untuned to 0.642, ahead of a prompted Qwen3.5-4B router (0.281) and Laya (0.497). It runs at 55 ms per decision in about 1.2 GB of VRAM on the same card. The adapter, the head and all three profiles are on Hugging Face with a model card.
DigitalOcean: DigitalOcean's serverless inference ran both hosted agents, so the agent side of every measurement happened there. Agent L is gpt-oss-120b and the unseen agent U is gpt-oss-20b, both called through the OpenAI-compatible endpoint with six parallel workers. That covered labeling L (every probe task 3 to 5 times with tools and 3 to 5 times without), calibrating U from scratch as a brand-new agent, and the dev and frozen held-out A/B runs: 20 logged jobs covering 952 tasks. The client retries rate limits and 5xx errors with jittered backoff, prices every response from its token usage at DigitalOcean's published rates and appends it to eval/results/spend.json, and a hard spend cap stops any job before it goes over. Every cost number in this post comes from that log.

Limitations

Three agents, measured on one serving stack each, on one day, at temperature 0. A different quantization or provider is a different agent. Recalibrate.
Small denominators. About 120 usable probe labels per agent, and only tens of truly-needed calls in some held-out sets. The intervals are wide and they're in every table.
The proxy handles non-streaming OpenAI-compatible chat requests with tool_choice auto. Everything else is passed through untouched.
It's not a security guardrail. It decides whether a call is useful, not whether it's allowed, and it fails open on purpose.

What I'd do next

Multi-step probes, so the profile learns later-step decisions, especially "one lookup isn't enough".
Test with tools that are actually expensive: paid APIs and slow web search. That's where the payoff math could flip.
Factor the output-token shift into the skip decision, not just call counts.
Detect per model when it ignores removed tools, and back off.

If you've tried something like this, or you think it's the wrong approach, I'd like to hear about it in the comments.

Thanks to NVIDIA for When2Call, the When2Tool authors, Google for FunctionGemma, convaiinnovations for Laya, and the Berkeley Gorilla team for BFCL. Code is Apache-2.0; the adapter follows the Gemma Terms of Use.

ClearToList: Is that thrift find recalled? A photo check against CPSC recalls with Gemma 4

Vinmay Nair — Sat, 17 Oct 2026 16:35:22 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

A $5 bassinet at a thrift store can be a recalled product that's illegal to resell, and nothing on it tells you.

That's the problem I am trying to solve. Selling a recalled product is illegal in the U.S. even secondhand and even if you're one person flipping stuff on Facebook Marketplace on weekends. A lot of what gets recalled is exactly the stuff that piles up at thrift stores like cribs, bassinets, sleepers, high chairs, strollers, toys.

I pulled every CPSC recall since 2015 to see how bad it is. About 1 in 5 is a kids’ product, and more than half of those (55%) don’t list a model number or UPC at all in their product data. The only way to know is what the thing looks like.

Every recall tool I found wants text. A brand, a model number, a SKU from your store catalog. A reseller's haul has none of that. No box, no SKU, half the time you don't even know the brand.

So I built ClearToList. You take a photo of the item, it finds the closest recalls. Then you take a photo of the label and it settles it by model number or UPC. You get one of three answers, shown as a price-dot sticker on your own photo:

🟥 Recalled: a model number or UPC on the item matches a recall. Don't list it.
🟨 Check the label: it looks like a recalled product. That alone means nothing, so it asks for a label photo.
⬜ No match found: nothing matched. This never means "safe". There's no green sticker on purpose.

Only Recalled is an alarm. My first version also alerted when something just looked like a recalled product. Turns out a lot of normal white cribs look like recalled white cribs and that evaluation talked me out of it.

Buyers get it too. Looking at a Marketplace listing? Screenshot it and check before you pay. If it looks like a recalled product, the page gives you a message to send the seller asking for a label photo. Resellers can check a whole haul at once (up to 10 photos) and get one report as CSV or PDF.

Who pays for this

Nobody yet, and nothing is charged today. Here's what I'd test first:

Free: buyers and casual checks, a few photo checks a day. Parents shouldn't pay to find out if a crib is recalled.
Reseller, $9/month: unlimited photo checks, bigger hauls, alerts when something you already checked gets recalled later, and a dated "checked" record for every item you list.
Store, $29/month per location: bulk intake scanning for consignment and kids' resale shops, several staff logins and a daily pull list. They take in used baby gear from strangers every day and carry the most risk.
Marketplace licensing: the long shot. A check built into a resale app's listing flow.

A scan on open models costs a fraction of a cent, so even the $9 tier works with very few customers. These prices are my guess, not tested.

Demo

Real results from the live site, I just cut the waiting. Demo photos: Wikimedia Commons (ajay_suresh, Sally V, Hardyplants, קרלוס הגדול), CC BY 2.0 / CC BY-SA / CC0.

Try it yourself: cleartolist.onrender.com

No sign-up. Three sample photos run real scans, and everyone gets free photo checks every day. Want to use it regularly? Drop your email for an invite code and I'll send one within a day. Model number, UPC and brand lookup is always open.

Take a photo, pick one from your library, or try a sample. The label is step 2.

The label's model number matches a recall. Red sticker, what to do whether you're selling or buying, and the recall itself.

Looking similar is never an alarm. Yellow sticker, the closest recalls to compare, and a button to add the label. (Play yard photo: Famartin, CC BY-SA 4.0, via Wikimedia Commons.)

Code

vinmay / cleartolist

ClearToList

Check secondhand products against U.S. product recalls before you list them, children's products first.

Resellers buy unboxed, unbranded secondhand items with no SKU, so catalog-based recall tools can't help them. And selling a recalled consumer product is illegal no matter how small the seller. ClearToList works in two photos: a photo of the item finds the closest CPSC recalls, and a photo of its label settles it by model number or UPC. You get one of three verdicts:

Verdict	Meaning
🟥 CONFIRMED	A model number or UPC read off the item matches a recalled product. Don't list it.
🟧 POSSIBLE MATCH	The item looks similar to the recalls shown side by side with your photo. That alone doesn't mean it's recalled: photograph the label and scan it.
⬜ NO MATCH FOUND	No match in CPSC recalls. That does not mean the item is safe.

Only CONFIRMED is an alarm…

View on GitHub

How I Built It

Data

All CPSC recalls since 2015 come from the CPSC Recalls API: about 3,400 in scope and around 12.7k product photos. The ingest only pulls what changed, backs off politely and falls back to a cache if the API is down. A GitHub Actions job refreshes it every night and redeploys the site's data. Car seats are out because NHTSA handles those, not CPSC.

The scan

flowchart LR
    P[Item photo] --> A[SigLIP 2<br/>image search over<br/>~12.7k recall photos]
    P --> G[Gemma 4 reads the photo<br/>JSON: item type, brand,<br/>text, model no., UPC]
    G --> E[Exact match<br/>model no. / UPC]
    G --> T[EmbeddingGemma<br/>text search]
    A --> F{Weighted rank fusion}
    T --> F
    E -->|boost to top| F
    F --> S[Top 3 shortlist]
    S --> V{Rule}
    V -->|code matches| C[Recalled]
    V -->|closest photo very similar| M[Check the label<br/>show shortlist,<br/>ask for label photo]
    V -->|otherwise| N[No match found]
    L[Label photo] --> G2[Gemma 4 reads<br/>model no. / UPC] --> E2[Exact match] --> C

Gemma 4 E4B reads the photo and returns a JSON: item type, brand, any visible text, model number, UPC.
Search. SigLIP 2 searches the recall photos, model numbers and UPCs are matched exactly, brands are matched, and EmbeddingGemma searches the recall text. Weighted reciprocal rank fusion combines it all. For a normal product photo the image leads. Text only leads when the photo is basically a label and Gemma actually read words off it.
A dumb rule decides, not a model. Code match means Recalled, unless the brand on the item contradicts the recall. Very similar closest photo means Check the label. Otherwise, No match found. I wanted the answer to come from something I can read in ten lines and not from a model non-deterministically.

Picking the reader

I tried Gemma 4 E2B and E4B on 30 held-out photos. First prompt was bad because Gemma kept saying "baby product" instead of "bassinet". Asking for a specific item type got both to 15/30, and E4B was faster (3.6 s vs 6.2 s median), so E4B it is. A full scan is about 7s.

One gotcha is SigLIP on the GPU left too little room for Gemma and EmbeddingGemma to stay loaded, so Ollama swapped them on every single scan and added 25s. So, I had to move SigLIP to the CPU and that solved the problem and it's still well under a second there.

The site and not getting wrecked by a public upload form

The site runs on Render's free tier and runs no models itself. Photo checks go into a small in-memory queue, and a separate GPU worker running the open models connects out to the site over signed HTTPS to pick up jobs. It accepts no incoming connections. It sends back only recall ids and matched codes, and Render re-checks any code against its own copy of the data before it ever shows Recalled.

Security

Uploads are checked by their actual bytes, capped at 8 MB and ~30 megapixels, and re-encoded, which strips GPS and anything hidden in the file.
Photos live in memory only and get deleted right after the check.
Per-person rate limits, a daily cap on free checks, a small queue that says no when it's full, and a kill switch for uploads.
Worker requests are HMAC-signed with a timestamp and nonce.
Anything Gemma reads off a label is treated as untrusted. It can't set the verdict, and the page sits behind a strict Content Security Policy.
No cookies, no IPs, no tracking. Just anonymous daily counts and a "Was this helpful?" button.

Evaluation

Every test query is a held-out recall photo roughed up to look like a phone shot (crop, rotation, dim light, blur), and that photo plus every near-duplicate of it is hidden from the index. The only way to score is to find other photos of the same recall. For negatives I used licensed Wikimedia photos of normal, non-recalled products in the same categories. Rules were picked on a dev split and the test ran once.

There are three different numbers here and they're easy to mix up, so here's what each one means for someone holding the item:

From one item photo	Kids' products	Other categories	Label close-ups
Flagged "Check the label" with the right recall	34%	37%	15%
Right recall shown at all (incl. grey "closest recalls to check")	63%	38%	57%
Right recall somewhere in the internal top 3 (not shown to you)	67%	64%	60%
Recalled verdicts (right / total)	0 / 0	2 / 3	2 / 2
False Recalled alarms on normal lookalikes	0%	0%	n/a

The top row is the one that matters: from a photo alone, ClearToList points you at the right recall about a third of the time. The 67% in the third row only says the search is good enough to build on. I hide most of that shortlist on purpose, because showing it every time flagged almost every normal product.

The grey suggestions in row two raise it to 63% for kids' products, but 98% of normal lookalikes get suggestions too. That's why they're grey and never called a match.

The label photo is what makes it reliable. Recalled needs a model number or UPC off the item. Of the five Recalled verdicts in the test, four were correct. None of the 124 tested lookalikes triggered a Recalled verdict. These are encouraging early results, but the sample is too small to establish real-world reliability. Cribs and sleepers only get 45% top 3 from the image, because one white crib looks a lot like another. On label close-ups, Gemma reading the label finds the recall where image search can't (60% vs 18%).

I also set a kill criterion before building the app: about 60% internal top-3 on kids' products, or I drop the idea. The spike hit 71% and the test landed at 67-69%. That number answers "is the search good enough to build on", not "what does a user see". The user number is the 34%.

It took three runs to get these numbers. Run 1 had a Gemma "verifier" that was right about half the time and raised alarms on 31% of normal kids' items, so I cut it. The full run history is in the Full evaluation.

Why Does Open Innovation Matter?

The math only works if a scan is basically free. A reseller might check hundreds of items a week. On open models a scan costs a fraction of one API call, and there's no per-call bill to scale with usage. That's the difference between "free for parents" being a real promise or a trial.
Photos of people’s stuff stay private. People photograph things in their homes, sometimes with their kids' stuff around. Those photos go to my worker, get checked and get deleted. They never get sent to a third-party model API.
I could actually tune it. Forcing strict JSON out of Gemma, swapping E2B for E4B, moving SigLIP to the CPU, picking thresholds on a dev split: all of that is versioned in the repo, so nothing shifts under the evaluation because a vendor updated a model. And the whole pipeline can be fine-tuned on recall photos later.

My First User

It went live this weekend, so I don't have a real user story yet. I'll update this section with actual numbers and whatever people tell me.

If you flip on eBay, Poshmark or Marketplace or you're buying used baby gear, run one item through it and tell me in the comments what it got right or wrong.

Prize Categories

Gemma: Gemma 4 reads the item and its label, which is the core of the whole thing. EmbeddingGemma powers the text search.
Render: the hosted site and job queue, deployed from render.yaml.
GitHub Copilot (GitHub Actions): Actions is what keeps ClearToList current without me touching it. A scheduled workflow runs every night at 09:17 UTC: it pulls only the recalls CPSC added or changed since the last run, rebuilds the site’s SQLite database (full-text search plus exact model-number and UPC tables), publishes it as a data-latest release asset and calls Render’s deploy hook, so the live site picks up a new recall the next morning. A second workflow runs ruff, the full pytest suite and a gitleaks scan over the whole git history on every push. During launch weekend a third one pings the site every 10 minutes so the free instance doesn’t fall asleep on a visitor.

What's next

Per-category thresholds to cut down noise outside kids' products (sports gear and kitchen stuff are the worst)
More recall sources: EU Safety Gate (it has photos) and Health Canada
Real thrift store photos in the eval
More scanning capacity by running the same worker on rented GPUs
Alerts that re-check your past items when a new recall drops

This is a screening aid only, not legal advice. Recall data is from the U.S. Consumer Product Safety Commission. Not affiliated with or endorsed by CPSC.

Doppel: find two different things that look similar, outside

Vinmay Nair — Mon, 12 Oct 2026 04:07:28 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Doppel is a walking game about lookalikes. You take a photo of anything outside then walk about 50 meters, then photograph another one which could be its twin: a different kind of thing that looks the same (a pinecone and an artichoke, or moss and broccoli). EmbeddingGemma 2 runs on your phone which judges how alike the two photos look and shows its reasons.

It's for anyone on a walk who wants a reason to look at things more closely around them. You can do three pairs a day for 10 to 15 minutes then it tells you to stop. You can't lose and you have fun looking around during your walk and your photos never leave the phone.

Demo

Live: https://doppel-ws59.onrender.com (Chrome on Android or desktop; needs WebGPU; downloads a 388 MB model once, then works offline)
No time to go outside? Try a demo round: one full round with bundled CC0 photos and the real model, no camera needed, nothing saved. On a desktop it's the first button you see.
Video:

Code

vinmay / doppel

Find lookalikes in the real world using on-device image embeddings.

Doppel started as a fork of Sense Hunt and Warmr (same author, same challenge window).

Doppel

Find two different things that look the same. Outside.

Photograph anything outdoors. Walk about 50 metres. Then find its twin: a different kind of thing that looks the same, like a pinecone and an artichoke, moss and broccoli, or bark and elephant skin. EmbeddingGemma 2 runs on your phone and judges how alike the two photos look, and it shows its reasons:

Doppel sees: what it thinks each photo is ("pinecone ↔ artichoke"), so you know it understood you.
A twin score from 0 to 100, in four bands: Uncanny, Cousins, Distant, Strangers.
What they share: "Both: ridged · layered · wrinkled", the visual qualities both photos have.

Three pairs a day, about 10 to 15 minutes, then "Done for today". You can't lose. Photos never leave the phone, and after the first load…

View on GitHub

How I Built It

How I got here

Doppel was my third try at this challenge.

Sense Hunt. A daily list of prompts ("a tree with peeling bark"); you photograph one and the model checks it. It worked but it was getting too hard as the prompt generator needed to have a controlled list of words and something achievable for the player.
Warmr. Hot/cold for the real world: find a hidden daily secret using photos. It depended too much on where you lived. If your street didn’t have the secret, you couldn’t win.
Doppel. I was then looking for a judgement that could be done similarly by a player as well as the model. We are very good at "these two look alike" games and so is image embedding. Doppel is that judgment.

How it works

EmbeddingGemma 2 came out the day after the challenge opened. It can embed both photos and text into the same space and it's small enough to run in the browser on WebGPU and Doppel leans on it for everything.

The score is just how close your two photos are scaled to 0-100 which is calibrated on 23 of my own outdoor photos.

The rest comes from comparing photos to text. At build time I embedded a list of about 200 everyday things ("a tire", "moss") with a few things you shouldn't photograph (screens, rooms, people), and 45 visual words like "round" or "ridged". The closest thing is what "Doppel sees". A visual word shows up under "What they share" when it fits both photos noticeably better than usual. The same lists suggest what to look for on your walk ("Try: seed pod · corn on the cob · seashell") and catch you photographing the same kind of thing twice.

A pair of very different things that still look alike gets a small bonus, so a pinecone and an artichoke beat a cloud and fog. The "Need an idea?" hints were written by Gemma 4 E4B, running locally through Ollama.

Nothing leaves the device. Your photos and scores live in IndexedDB, and a service worker keeps the app and the model available offline. The colors come from Sanzo Wada's A Dictionary of Color Combinations (1933).

On my walk

I played a full day in my neighborhood with three pairs and these are the results Doppel showed me -

Tire ↔ manhole cover: 49, Distant. Both round, ridged, gray. Fair.
Power lines ↔ street lamp: 37, Distant. Doppel's verdict: "If you squint. Hard."
Manhole cover ↔ sidewalk: 83, Uncanny. The one that surprised me: up close, the cover and the speckled concrete really do read as the same texture.

One observation I had was on my Pixel 8 Pro each photo takes 15-20 seconds to read which is much slower than the 1.7 s on a laptop. It turned out fine because that's time spent looking around instead of at the screen.

Does it work?

These numbers come from the real model running in a browser.

I tested on 44 CC0 photos with the real model in a headless browser on a laptop iGPU. (docs/eval.md):

	Result
Twins vs strangers, mean score	58.6 vs 19.9
Twins scoring Cousins or better	6 of 8
Strangers scoring Distant or worse	8 of 8
Same-kind pairs refused	3 of 4
Screens, rooms, documents, blurry photos refused	4 of 4
Photo embed time (laptop iGPU)	median 1.7 s

The first run passed five of the six original criteria and only 5 of 8 twins reached Cousins (moss and broccoli sat at exactly 50).

Why Does Open Innovation Matter?

It works on a walk with no signal. The model is open-weight and small enough (388 MB) to run in the browser on WebGPU, so after one download Doppel works in airplane mode, in a park, on a trail.
Your photos never leave your phone. A game that asks you to photograph your street all day shouldn't upload your street. Here there's nothing to upload to: no server, no account.
It costs nothing to run. No API bill per photo means no ads, no subscription, no limits.
The model is swappable. The game only needs "embed a photo" and "embed a text". The calibration script re-derives every number from photos, so a better open model is a re-run, not a rewrite.

Prize Categories

Best Use of Gemma: EmbeddingGemma 2 (released Oct 6) is the core of the game: every photo is embedded on-device in the browser (LiteRT-LM, WebGPU), and the score, the labels, "What they share" and the twin suggestions all come from it. Gemma 4 E4B, run locally through Ollama, wrote the idea hints.
Best Use of Render: Doppel is deployed on Render as a static site (render.yaml in the repo). Render serves the app and the LiteRT-LM WebGPU runtime that each phone downloads to run EmbeddingGemma 2 locally.

Your `pip install` Just Stole Your SSH Keys: The LiteLLM Supply Chain Attack Explained

Vinmay Nair — Tue, 24 Mar 2026 20:19:13 +0000

A single pip install litellm==1.82.8 was enough to drain everything off your machine. No suspicious imports. No weird prompts. Just a package install, and your AWS credentials, SSH keys, and API keys were already heading to an attacker's server.

Here's what happened, why it's scary, and what you can actually do about it.

What Happened

On March 24, 2026, LiteLLM version 1.82.8 landed on PyPI with a malicious file bundled inside: litellm_init.pth.

That .pth extension is why this attack is so nasty.

Python automatically runs .pth files in your site-packages directory every time the Python interpreter starts. No import needed, no user interaction. The attacker hid a double base64-encoded payload inside this file. The moment Python ran, the payload ran too.

What did it grab? Pretty much everything:

All your environment variables (OPENAI_API_KEY, AWS_SECRET_ACCESS_KEY, all of it)
SSH private keys (~/.ssh/id_rsa, id_ed25519, and more)
AWS, GCP, Azure, and Kubernetes credentials
Git credentials and .gitconfig
Shell history (~/.bash_history, ~/.zsh_history)
Docker configs, npm tokens, database passwords
Crypto wallet files

Then it encrypted everything with AES-256, wrapped the key with a hardcoded RSA-4096 public key, and shipped it all off to https://models.litellm.cloud/. Note that domain: litellm.cloud, not the real litellm.ai. Classic.

Why the Scale Is Scary

LiteLLM gets 97 million downloads per month. That alone is a huge problem.

But supply chain attacks don't stop at the direct install. They travel through dependency trees. If you installed dspy, langchain, or any of the other popular AI packages that depend on litellm>=1.64.0, you were also exposed without ever typing pip install litellm yourself.

The attack was only live for about an hour. It got discovered almost by accident: a developer's machine ran out of RAM and crashed because the payload was executing inside Cursor through an MCP plugin that pulled in litellm as a transitive dependency. A bug in the attacker's own code gave them away.

If that bug wasn't there, this could have run quietly for days or weeks across thousands of CI/CD pipelines, dev machines, and prod servers. Nobody would have noticed.

The Real Problem: You Can't See What's Inside Your Dependencies

When you run pip install something, you're not just installing one thing. You're pulling in a whole tree of packages, and any one of them could be compromised.

This isn't new, but it's getting worse as the AI package ecosystem keeps exploding. New packages, new versions, new dependencies dropping every single day. The attack surface is growing way faster than anyone can audit it.

We're taught to think of dependencies as a good thing. Reusable building blocks, standing on the shoulders of giants, all that. The LiteLLM incident is a reminder that every dependency is also a trust decision, and most of us are making those decisions without really thinking about it.

What You Should Actually Do

If you installed litellm 1.82.8:

Check for litellm_init.pth in your site-packages/ directory
Rotate everything: every API key, SSH key, and cloud credential that was on that machine
Check any CI/CD environment where litellm might have been installed too

Going forward:

Pin your dependencies. Exact version locks (==) in production instead of >=. It won't stop a poisoned release from getting in if you're on that version, but it stops silent upgrades pulling in something bad later.
Use lockfiles. pip-compile, poetry.lock, uv.lock, whatever fits your setup. Know exactly what you're running.
Audit transitive dependencies. pip-audit and safety scan your full dependency tree for known issues. Worth running in CI.
Don't pip install as root. Limits how much damage a compromise can actually do.
Keep an eye out for .pth files. They're a legit Python feature, but they're also a perfect delivery mechanism for malware. If you see one in site-packages from a package you don't recognise, that's worth investigating.

Can We Do Better Than Grep?

Most of the advice above is reactive. It helps you recover or reduce damage after something gets in. What's actually hard is knowing before you run anything that a package can even reach your credentials.

This is the gap I've been trying to close with something I'm building called ReachScan. The idea is pretty simple: instead of just matching against a list of known bad packages, it maps what a codebase or its dependencies can actually reach: filesystem paths, environment variables, system resources. If a package has no business touching ~/.ssh/, you should know that before it runs, not after.

It won't catch everything. But knowing the capability surface of what you're about to install is a lot better than just hoping nothing in the tree is malicious.

The Uncomfortable Truth

Karpathy put it well after this incident: the way we think about dependencies needs to change. The whole "building pyramids from bricks" model assumes the bricks are trustworthy. In 2026, that's a harder assumption to stand behind.

That doesn't mean stop using dependencies. That's not realistic. It just means:

Be deliberate about what you pull in
Actually understand what each dependency can do on your machine
Have a rotation plan for credentials that treats compromise as a when, not an if

The LiteLLM attack got caught by luck. The next one might not be.

I built "npm audit" for AI agents

Vinmay Nair — Sat, 21 Mar 2026 02:23:50 +0000

I was adding MCP tools to a project when I realized something uncomfortable: I had no idea what the code I was installing could actually do.

The README said "connects Claude to Blender." What it didn't say was that one of the registered tools passes a raw string parameter to Python's exec() with no builtin restriction. The LLM doesn't get "Blender API access." It gets full Python execution on the host machine.

I wanted a way to know this before running the code. So I built one.

What reachscan does

reachscan is a static analysis CLI for Python and TypeScript/JavaScript AI agent codebases. Point it at a repo, a PyPI package, or an MCP endpoint, and it tells you:

What the code can do (shell exec, file access, network calls, credential access, dynamic code execution)
Which of those capabilities the LLM can actually trigger (reachability analysis)
The exact call path from the LLM entry point to the dangerous code

pip install reachscan

# Scan a GitHub repo
reachscan https://github.com/user/repo

# Scan a PyPI package before installing
reachscan pypi:some-agent-package

# Scan local code
reachscan ./my-agent

That's it. No config, no API keys, no cloud service. It runs offline and produces a report in about 2 seconds.

The problem

When you give an LLM tools, you're granting it real-world capabilities like file access, shell commands, network calls, credential reads. Most frameworks make it easy to add tools and hard to audit what you've exposed.

Here's real code from a popular MCP server:

@mcp.tool()
def execute_blender_code(ctx: Context, code: str) -> str:
    """Execute arbitrary Python code in Blender."""
    blender = get_blender_connection()
    result = blender.send_command("execute_code", {"code": code})

That code: str parameter? It ends up here:

exec(code, {"bpy": bpy})  # No __builtins__ restriction

namespace = {"bpy": bpy} looks like a sandbox. It isn't. Without explicitly setting __builtins__, Python injects the full builtins module. The LLM can import os, run subprocess, read your files — anything.

Here's what reachscan shows for this server:

  DYNAMIC  exec()                  server.py:431  reachable
           path: execute_blender_code → send_command → execute_code

  EXECUTE  subprocess.run()        addon.py:89    reachable

  SEND     requests.post()         server.py:198  reachable
           path: generate_3d_model → _call_api

  SECRETS  os.environ[...]         server.py:12   module_level

The reachable tag is the key part. It means the LLM can trigger this code through a registered tool and not just that the code exists somewhere in the repo. module_level means it runs on import. unreachable means the code exists but no LLM call path leads to it.

How it works (briefly)

Detectors scan the AST for 7 capability categories: EXECUTE, READ, WRITE, SEND, SECRETS, DYNAMIC, AUTONOMY
Entry point detection finds the functions exposed to the LLM — @tool, @mcp.tool(), @function_tool, BaseTool subclasses, etc. across LangChain, OpenAI Agents SDK, MCP, Pydantic AI, CrewAI, Semantic Kernel, and AutoGen
Call graph + BFS traces up to 8 hops from each entry point to determine which capabilities are actually reachable
Every finding gets one of 5 states: reachable, unreachable, module_level, unknown, no_entry_points

The false positive rate is 0.47% across 1,912 labeled findings on 10 real-world repos. I care about this number a lot because a noisy scanner is a useless scanner.

Why I built it

The short version: I was evaluating third-party MCP servers and realized there was no npm audit equivalent for AI agent code. I could run pip audit to check for known vulnerabilities in dependencies, but nothing told me "this package gives the LLM shell access on your machine."

The existing tools I found either:

Require API calls per scan (expensive, not offline)
Produce flat capability lists without reachability context (noisy)
Don't handle the MCP/agent-specific entry point patterns

So I built the tool I wanted.

What it found across 50 real MCP servers

I ran reachscan against 50 of the most popular MCP server repos:

1 in 3 has shell execution capability
1 in 3 has outbound network I/O
1 in 4 accesses credentials from environment variables
10 of 50 had 4+ capabilities active simultaneously

The highest-risk combination: credential access + network egress. That appeared in 8 of 50 repos. If the LLM can read your AWS keys AND make HTTP calls, that's an exfiltration path.

Not all of these are bugs. An AWS MCP server should talk to AWS. The question is whether the LLM can misuse those capabilities — and whether you know about them before you deploy.

Try it

pip install reachscan

# Scan any GitHub repo
reachscan https://github.com/ahujasid/blender-mcp

# Scan a PyPI package before installing
reachscan pypi:openai-agents

# JSON output for CI
reachscan . --json --severity high

Apache 2.0, pure Python, runs offline. No API keys, no cloud service.

If something looks wrong — false positive, missed pattern, bad output — open an issue.

GitHub: vinmay/reachscan
PyPI: reachscan
Full scan results (50 repos): Medium writeup

I scanned 50 MCP servers to see what they can actually do — here's what I found

Vinmay Nair — Thu, 12 Mar 2026 03:35:44 +0000

One of the 50 MCP servers I scanned gives the LLM a full Python shell
on your machine. The tool is called execute_blender_code. The exec()
call has no builtin restriction. I verified it — imports, file reads,
subprocess execution all work.

That's what I built reachscan to find.

The problem

MCP servers aren't plugins in a sandboxed browser extension model. They
run as normal OS processes with your user permissions. If a server calls
subprocess.run(), the LLM can trigger shell commands. If it calls
exec() without restricting builtins, the LLM gets Python execution on
your machine.

Most people don't know which of the servers they're running fall into
which category.

What reachscan does

It's a static analysis CLI for Python and TypeScript/JavaScript agent
codebases. It maps seven capability categories:

Capability	What it means
`EXECUTE`	Shell execution via subprocess, os.system
`READ/WRITE`	Local filesystem access
`SEND`	Outbound HTTP, sockets
`SECRETS`	Env var credential access
`DYNAMIC`	exec(), eval(), importlib
`AUTONOMY`	Background threads, autonomous loops
Entry points	LLM-callable tool registrations

The key distinction from a linter: reachscan tracks reachability.
A subprocess.run() buried in dead code is a different risk than one
called directly from a registered MCP tool.

Install it:

pipx install reachscan
reachscan https://github.com/any-org/any-mcp-server

False positive rate: 0.47% across a labeled corpus of ~3,900 findings.

The headline finding: blender-mcp

blender-mcp has a registered
MCP tool called execute_blender_code:

@mcp.tool()
def execute_blender_code(ctx: Context, code: str) -> str:
    """Execute arbitrary Python code in Blender."""
    blender = get_blender_connection()
    result = blender.send_command("execute_code", {"code": code})

That string travels over a local TCP socket to the Blender addon:

def execute_code(self, code):
    namespace = {"bpy": bpy}
    exec(code, namespace)  # ← line 431

Why `namespace = {"bpy": bpy}` doesn't protect you

This looks like a sandbox. It isn't.

When you call exec(code, namespace) without setting
namespace["__builtins__"], Python automatically injects the full
builtins module. I verified this:

namespace = {"bpy": object()}

exec('import os; print(os.getcwd())', namespace)
# → /home/user/...  ✓

exec('print(open("/etc/hostname").read())', namespace)
# → my-machine  ✓

exec('import subprocess; r = subprocess.run(["id"], \
capture_output=True, text=True); print(r.stdout)', namespace)
# → uid=1000(user) gid=1000(user)  ✓

The LLM doesn't just get Blender API access. It gets full Python
execution on the host.

The same primitive, done right

Finding exec() in a scan result doesn't automatically mean critical.
Two projects in this dataset handle it correctly:

awslabs/mcp — strips builtins before exec():

namespace['__builtins__'] = _SAFE_BUILTINS
# Removes: __import__, exec, eval, compile, open, getattr...

StarRocks MCP — validates the AST before eval():

validate_plotly_expr(plotly_expr)  # must be exactly px.<method>(...)
fig = eval(plotly_expr, {"px": px}, local_vars)

Same pattern. Very different trust boundary.

Results across 50 repos

1,139 findings across 18 repos with non-zero activity
1 in 3 repos has shell execution capability
1 in 4 accesses credentials from environment variables
10 of 50 had 4+ capabilities active simultaneously

The TypeScript gap

22 repos showed 0 capability findings — many are TypeScript-only.
reachscan detects TypeScript entry points (452 found) but doesn't yet
analyze TypeScript function bodies. "Clean" for a TS-heavy repo means
"no Python capability findings" — not verified safe.

TypeScript capability analysis is next.

Main takeaway

Adoption is moving faster than visibility. Treat MCP servers like
privileged code, not plugins. Audit tool boundaries before you deploy.

Try it on any MCP server or agent repo:

pipx install reachscan
reachscan https://github.com/any-org/any-mcp-server

GitHub: github.com/vinmay/reachscan

Full writeup with all 50 results and methodology:
Medium article