DEV Community: Bhoomika Choudhury

The Fool: a local AI that plays dumb in your git repo, then catches your real mistakes

Bhoomika Choudhury — Sat, 31 Oct 2026 22:16:28 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

The Fool is an AI that lives in your git repo and looks broken on purpose. Its brain loads to 12% and stays there. Ask it something easy and it sometimes answers with obvious nonsense. Then, out of nowhere, it interrupts, jumps your code view to a file and line, and tells you about a real mistake you made: the TODO you wrote 40 days ago, the file you edited and never committed, the commit you made at 3 AM.

The scary part is meant to be the gap between how stupid it acts and how much it actually knows. I wanted a fool that is not a fool.

It runs entirely on your machine with Gemma (through Ollama). It is read-only: it never changes your repo and never runs your code.

Demo

Code

BhoomikaC15 / the-fool

The Fool

My entry for Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

The Fool is an AI that lives in your git repo. It looks broken and unfinished on purpose: it stutters, it loads its brain to 12% and stays there, and when you ask it something it sometimes says something obviously stupid. But it has read every file in your project, and every now and then, out of nowhere, it interrupts to point out a mistake you made. A real one, with the file, the line and the date.

It runs entirely on your machine with a small open model (Gemma through Ollama). It is read-only: it never changes your repo and never runs your code.

How it works

Two things are kept strictly apart:

Code finds the mistakes. scan.py reads the working tree and the git history and reports: a secret in a file, a…

View on GitHub

How I Built It

The whole design rests on one split: the model suggests, code verifies.

Code finds the mistakes. scan.py reads the files and the git history and reports secrets (including ones you deleted but git remembers), tracked .env files, syntax errors, leftover debuggers and print("here"), bare except:, old TODOs, empty files, stub functions, unused imports, files edited and never committed, and commits after midnight. Every finding has an exact file, line, time and quote. A small language model is bad at counting days or noticing line 214, so I did not ask it to.

The model only talks. When you chat, Gemma answers from the three most relevant snippets of the repo and has to cite file and line. Code then checks every citation against the real files and drops the ones that do not exist. Haunts are worded by templates by default. With --phrase model, Gemma can reword them, but only if its line still contains the file name and every number from the finding.

The "stupid" answers never come from the model. They are fixed lines, and the interface labels them "glitch, not an answer". A real wrong answer from a model is a bug. A staged one is a character, and I did not want the two to be confusable.

Finding the right file was the hard part. My first search was plain word matching. On 20 questions I wrote about my own project, in my own words, it put the right file in the top 3 snippets 65% of the time. Asking "where is the average duration calculated" does not match a function called avg_dur. I added synonyms, prefix and typo matching, a boost for definition lines and a penalty for README files, which took it to 70%. An optional step where Gemma first guesses likely code names took it to 80% (16 of 20), at the cost of one extra model call per question.

What the numbers do and do not show

Scanner on planted mistakes: 67 of 67 found, no false alarms, on 12 sample repos. This is a sanity test. I wrote the mistakes and the scanner, so it proves the scanner does what I meant and nothing more.
Scanner on real code. I ran it on click, requests and my own unfinished project. The first run flagged test certificates as secrets, so I excluded test folders and cut noisy stub and import rules. I tuned those rules after seeing real output, so they are heuristics, not a benchmark. On my own project it found 15 things (5 uncommitted files, one untouched for 138 days; 2 after-midnight commits; 8 unused imports). All were real.
Q&A on my 20 questions: retrieval 65% → 70% → 80% as above. That is 20 questions, so each one is 5 points, and I tuned the default rules after seeing it fail on these same questions, so 70% is not a clean result. It still misses 4 of 20.
Citations prove a line exists, not that the answer is right. 19 of 20 answers had a real file:line receipt, but some of those pointed at the wrong file.
Most checks are Python-specific. On my laptop (8 GB RAM, no GPU) Gemma 3 4B takes about 60 seconds per answer.

Why Does Open Innovation Matter?

A tool that reads your private code and your git history should not need to send either to a server. Because Gemma is open and small enough to run locally, the haunting stays on your machine, and you can read every rule in scan.py to see exactly what it will accuse you of. Being open also let me limit what the model is allowed to do, which is the point of the project.

My Agent Session

I built The Fool in conversation with Claude. The idea and the character are mine: an AI that looks stupid and glitchy but secretly knows everything, and haunts you with a real mistake. Claude helped turn that into a design and wrote most of the code. I directed it, ran everything on my own machine and repo, and decided what went in.

How the session actually went:

Design pushback. I started with a vague “spooky AI”. Claude proposed a design where code finds the mistakes and the model only talks. I kept that design because it made the haunts trustworthy.
Real-world testing found problems. I ran the scanner on my own unfinished project. It found 15 things, and I felt called out. Early Gemma answers took 100 to 160 seconds, and one answer looped. We fixed these by shrinking the context and cutting repeated text.
My own evaluation exposed a weakness. I wrote 20 questions about my own repo in natural wording. The first search scored 65%. That drove the search rules and the optional --expand step, which reached 80%.
My feedback changed the interface. I disliked the first chat-only UI, so we rebuilt it as a code viewer where haunts fly in over the exact line and fade out.
Claude made mistakes too. It got the deadline time and the weekdays wrong, and I corrected it. Its sample tests were circular, which is why the post calls them sanity tests.

What I did myself: the concept, running and testing on real repos, writing the 20 questions, choosing what to keep, recording the demo and writing the final post.

The Spirit of It

I did not want an AI that is merely creepy. I wanted one that is wrong in a way you can see and right in a way you can check. The fool's nonsense is labeled and staged. The haunts come with receipts. When I ran it on my own half-finished project and it told me a file had sat uncommitted for 138 days, it felt like being called out, and I could not argue, because it was right.

Prize Categories

Best Use of Gemma: Gemma 3 4B runs locally through Ollama. It answers questions about the repo and can reword haunts. Its output is never trusted without a code check.
Best Use of ElevenLabs: the narration in the demo video is generated with ElevenLabs. The app itself does not use ElevenLabs: its optional "speak aloud" uses the browser's built-in local voice, so nothing from your repo is sent to a cloud service.

inbox-triage: Out of 100 emails, only a few matter. I built a tool that finds them.

Bhoomika Choudhury — Sun, 25 Oct 2026 19:01:20 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
Out of 100 emails in my inbox, a few matter: an internship offer, a hackathon, a reply from a repo owner. The rest is noise. I built a tool that sorts every email into four labels (deadline, needs reply, might interest you, skippable) and shows me the few that matter first. It never deletes, moves or sends anything.

What I Built

I am a college student. Travel to and from college, assignments and exams take most of my day, and open-source work takes what is left. A few important emails still reach my Gmail: an internship offer, a hackathon I care about, a reply from a repo owner. Everything else is noise, and students get a lot of it, especially fake promotions. I do not want to know that someone I barely know commented on a LinkedIn post. I want to know whether my pull request was merged. Out of 100 emails, usually only a few matter, and reading all 100 costs time I do not have.

So I built a triage tool for people like me. It gives every email one label:

Label	What it means	Example
Deadline	You must do something by a date. The tool shows the date and weekday.	"Submit your lab record by 27 October"
Needs reply	Someone is waiting for your answer, and no date is attached.	"Which logo do you prefer, A or B?"
Might interest you	Optional, but possibly worth a look.	A hackathon, a contest, a talk
Skippable	Ads, receipts, notifications, and other people's deadlines.	"FLASH SALE ends tonight!"

The report is built to feel like Gmail so it is not scary. The first screen shows only the counts. A fifth folder, OTP codes, is filled by a simple rule instead of the model: if an email names a one-time code and contains one, it goes there with the code shown on its row. You open one label at a time, and each email shows the exact line the label was based on.

Spam and promotions go to Skippable. Skippable means you can ignore the email. It does not mean the email is safe, because the tool does not check for phishing. It also never deletes, moves or sends anything.

Demo

Video: https://youtu.be/7rRF_DzIY7Q

The demo page runs on the 96 synthetic test emails. Every sender, name and address in it is made up. In the video, my own emails appear only as blurred counts.

Screenshot 1: the overview with the counts

Screenshot 2: the Deadlines list, sorted by date

Screenshot 3: one email opened, showing the quoted line the label came from

Code

Repository: https://github.com/BhoomikaC15/inbox-triage

Everything is plain Python with the standard library, except the Tinker fine-tune. To try the local pipeline on a mailbox export (.mbox or a folder of .eml files):

python triage.py inbox.mbox --model gemma3:4b --limit 10 --out report.html

To reproduce the results below:

python evaluate.py test.jsonl preds_baseline.jsonl
python evaluate.py test.jsonl preds_gemma.jsonl
python evaluate.py test.jsonl preds_qwen_base.jsonl
python evaluate.py test.jsonl preds_tinker.jsonl

The repo includes the test set, the training set, the generator for the training set, and every prediction file. My Tinker API key is not in the repo.

How I Built It

I wanted numbers, not a demo that only works on three emails. So I built the test first.

The test set

I used 96 fictional emails: 25 deadline, 25 needs reply, 19 might interest, 27 skippable. Claude drafted them, and I reviewed every label myself. About a third are hard on purpose: dates like "by Friday" or "within 3 days", two dates in one email, someone else's deadline, fake urgency in ads, quoted old replies, and optional events that carry a date. Nothing in the test set was ever used for training.

Four methods, same test

Regex baseline: keyword rules and a date parser. This is what any model has to beat.
Gemma 3 4B, run locally through Ollama on my laptop (8 GB RAM, no GPU).
Qwen3.5-4B, untuned, through Tinker, with the same long instruction prompt Gemma got.
Qwen3.5-4B, fine-tuned on Tinker (LoRA, rank 16) on 320 synthetic training emails, 3 epochs, 60 steps, about 170,000 tokens in total. A script generated the training emails and computed their labels and dates, and any email too similar to a test email was dropped. ### Code checks the model

The model only does the fuzzy part. For each email it returns a label, a deadline date and one quoted line. Plain code then checks the answer against the email:

A date is trusted only if it actually appears in the email. "By Friday" is resolved against the sent date.
The quoted line must really be in the email.
If the model's answer cannot be read, the email is shown as needs reply instead of being hidden.
If a date was dropped by the check, code fills it in only when the email holds exactly one candidate date or says "within N days". With several candidates it leaves the date empty. ### Results on the 96 test emails

Method	Accuracy (95% interval)	Important emails marked skippable	Exact deadline dates
Regex baseline	68.8% (59–78%)	11 of 50	14 of 25
Gemma 3 4B, local	76.0% (68–84%)	0 of 50	21 of 25
Qwen3.5-4B, untuned	94.8% (90–99%)	1 of 50	20 of 25
Qwen3.5-4B, fine-tuned	97.9% (95–100%)	1 of 50	23 of 25

The fine-tuned model is clearly better than the baseline and Gemma. Against untuned Qwen it fixed 5 emails and broke 2, so the gap is within noise (p about 0.45). I do not claim the fine-tune is more accurate. What it changes is the prompt: it needs only a one-line instruction. Measured with the tokenizer on the 96 test emails, an average email costs 133 input tokens instead of 368 (2.8 times fewer). At Tinker's list prices that is about $0.09 instead of $0.16 per 1,000 emails. If the long prompt's shared prefix is cached, untuned Qwen costs about $0.08, so the saving disappears. The fine-tuned model also writes slightly more output (42 tokens against 35). These are estimates from the price table, not from my bill, and I did not time latency.

With the code fallback, the fine-tuned model gets 24 of 25 exact dates. I added that rule after seeing the test errors, so treat that figure as post-hoc. The cleaner evidence: on the 320 training emails, the rule alone resolved 84 of 96 deadline dates exactly, got 2 wrong and left 10 empty.

What went wrong

Gemma invents deadlines. It labeled 16 emails as deadlines that were not, and only 14 of 27 skippable emails were correctly skipped. It errs toward showing you too much. It also never hid an important email (0 of 50), which matters most for an inbox cleaner.
Small models get weekday arithmetic wrong. Both Gemma and untuned Qwen lost the same four deadline dates. For three of them the correct date was easy to find in the text with plain code.
The fine-tuned model still made three mistakes. It marked a real deadline (a library membership expiry) as skippable. It called an optional contest a deadline. And it could not resolve "within 3 days" until the code fallback did.
Gemma is slow on my laptop: about 36 seconds per email, so one hour for 96 emails.

Limits
The test and training emails are all synthetic. Claude drafted the test emails, which could make them easier for another AI to read than real mail. My only real-inbox check is a small spot check (below), not a benchmark.
My training emails are more templated than real mail, and the fine-tune's loss fell to almost zero. The test set is the check, and it held up, but real mail will be messier.
My best result ran on Tinker's cloud. Only the Gemma path keeps emails entirely on my machine, and it is the weaker one.
I did not test a closed model, so I cannot say how much better one would do. The OTP rule was checked on only a handful of OTP emails, and I added it after realising I had forgotten that label, so it is outside the measured results above.
My labels have no slot for an important notification, like "your pull request was merged". That email would probably land in Skippable or Might interest you. That is exactly the kind of email I care about, so it is the first thing I would fix.

Why Does Open Innovation Matter?

Email is the most private data most people have: offers, bank notices, messages from family. A tool that sorts it should not need to send it to a company's server. Open-weight models make that possible. Gemma 3 4B and Qwen3.5-4B are small enough to run on an ordinary student laptop. Mine has 8 GB of RAM and no GPU, and Gemma ran there, slowly but without sending a single email anywhere.

Open models also made the fine-tune affordable. I trained Qwen3.5-4B on Tinker with a $10 credit, and the training run used about 170,000 tokens. I could afford to try it, compare it to the untuned model, and report that the gain is small, instead of just trusting a bigger model.

My Agent Session

I built this with Claude as a coding agent over about three days, working from a plan I set at the start: measure first, and report whatever the numbers say.

What Claude did: drafted the 96 test emails and the generator for the 320 training emails, wrote the baseline, the scorer, the Gemma and Tinker scripts, and ran its own checks before handing files over.

What I did: chose the idea and the labels, reviewed every test label, ran everything on my own laptop (Ollama, Tinker login, the full scoring runs), debugged my setup, and decided which results to claim.

Where the session changed the project:

It started as a group-chat summarizer. We switched to email because email is more private and because I actually hate it.
Claude pushed me to build the test set and baseline before any model. That is why I can say Gemma only scored 76%.
When untuned Qwen already scored 94.8%, we did not hide it. The post says the fine-tune's gain is within noise.
When I ran my own emails, I found problems the synthetic set never showed: long promo emails that overflowed the model, and I had forgotten the OTP label. Both are fixed. Session link or transcript: not available. I built this before I knew about DevRelay, so I could not save the session there. The account above is my own summary of what Claude did and what I did.

The Lazy Payoff

The work I stop doing is opening emails to find out whether they matter. I look at the counts, open Deadlines and Needs reply, and leave the rest. Every opened email shows the line the label came from, so I can trust a label without reading the whole message.

My test set is balanced on purpose, so about half of it needs action. My real inbox is not like that: far fewer than half of my emails matter. I also ran the tool locally with Gemma on my own emails. On a first run of 20 I agreed with 17 labels. I then fixed what I found (long promo emails, no interests list, no OTP folder) and reran on 25, where I agreed with 23 of 25. That is a small spot check, not a benchmark, and since I changed the tool after the first run it is not independent evidence. I have not timed myself on a real inbox, so I will not claim a number of minutes saved.

Prize Categories

I am entering the two categories this project honestly uses.

Best Use of Tinker. I fine-tuned Qwen3.5-4B on Tinker for one task: sorting email into four labels. Against the regex baseline, accuracy went from 68.8% to 97.9% on 96 held-out emails. Against the untuned Qwen3.5-4B (94.8%) the accuracy gain is within noise, so what I can show is a smaller prompt: 133 instead of 368 input tokens per email, about $0.09 instead of $0.16 per 1,000 emails at list prices, with no saving if the long prompt's prefix is cached. Training cost about 170,000 tokens on a $10 credit.
Best Use of Gemma. Gemma 3 4B runs locally through Ollama and is the private path in the tool: it labels emails on my laptop without sending them anywhere. It scored 76.0%, below Qwen, and the post reports where it fails. I am not entering other categories, because I did not use those partners' tools.

DEV Community: Bhoomika Choudhury

The Fool: a local AI that plays dumb in your git repo, then catches your real mistakes

What I Built

Demo

Code

BhoomikaC15 / the-fool

The Fool

How it works

How I Built It

What the numbers do and do not show

Why Does Open Innovation Matter?

My Agent Session

The Spirit of It

Prize Categories

inbox-triage: Out of 100 emails, only a few matter. I built a tool that finds them.

What I Built

Demo

Code

How I Built It

The test set

Four methods, same test

What went wrong

Limits

Why Does Open Innovation Matter?

My Agent Session

The Lazy Payoff

Prize Categories