DEV Community: Saharsh Tibrewala

Before Gemma touches my mom's phone, I raced it on mine: 35 minutes deleting Good Morning forwards by hand vs one night

Saharsh Tibrewala — Mon, 26 Oct 2026 05:59:25 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

On Sunday night I sat with my phone and deleted WhatsApp forwards by hand. Thirty-five minutes later I had thrown away 1,641 pictures, videos and screenshots, 1.89 GB of them. My phone also holds a lot of my mom's and dad's photos, so every swipe was a small bet that I was not binning one of theirs.

Then I plugged the phone into my laptop and let an open model go through the same phone, one picture at a time: the ones I had kept, and the ones I had just thrown away. This post is what it found, and how it compares with my 35 minutes.

The narrator is Amazon Polly (the "Arjun" voice), not me, and the pictures in the video are samples, not my phone's.

What I Built

Good Morning Sweep clears junk off a phone without uploading anything. You plug the phone into a laptop. The laptop copies the WhatsApp pictures, WhatsApp videos, screenshots and camera photos, and then Gemma 4, an open AI model running on the laptop itself, looks at them one by one. When it is done, you get one page that shows what it would delete and why. You tap anything you want to keep, then press "Delete from phone".

I built it for my mom's phone, which is where the Good Morning flood really lands. The first real run was on mine, which holds plenty of her photos and my dad's too. When I showed them what it does, their verdict was: "This is great, it will save so much time."

For each picture, Gemma says what it is (a greeting, a quote, a meme, forwarded news, an ad, a real photo, a document, a screenshot or an accidental shot) and copies out the words printed on it. Then plain rules, written by me, decide what to suggest:

What the page suggests	How it decides	Ticked to delete?
WhatsApp forwards	Gemma reads the words; the rules look for greeting and forward words in Hindi, English and seven other Indian scripts	yes
Exact copies	same file twice; one copy always stays	yes
Blurry or black photos	the code measures blur and brightness, and Gemma must agree it was a mistake with nobody in it	yes
Almost the same photo	the sharper one stays	no
Old screenshots of one-time codes, payments, deliveries	older than 30 days	no
Your call	Gemma thinks it is a forward, but real people may be in it	no

Photos of people, documents such as bills, prescriptions and IDs, and anything Gemma is unsure about are never suggested. Nothing leaves the laptop. Nothing is deleted until you have scrolled past it on the page, and just before each deletion the phone itself checks that the file is exactly the one backed up on the laptop.

The rule I kept coming back to: it reads before it deletes. Every red tile says why, in words you can check against the picture: Forwarded greeting. It reads: "शुभ प्रभात आपका दिन मंगलमय हो".

From the public demo, with sample cards I made: Gemma reads Hindi word for word.

Demo

Try the review page yourself: good-morning-sweep.vercel.app. It is the real page with sample pictures (stock photos and cards I made), because the real run's pictures stay on my laptop. Tap a picture to keep it. The "Your call" group at the bottom is the trap described below.

The trap: Gemma calls this a greeting, but says real people may be in it, so it is my call, and it is not ticked.

Code

itssaharsh / good-morning-sweep

Frees a parent's phone of WhatsApp Good Morning forwards, copies and mistakes, with Gemma 4 running on the family laptop. It reads before it deletes.

Good Morning Sweep

Frees a parent's phone from WhatsApp "Good Morning" forwards, duplicate copies and accidental shots, using an open model (Gemma 4) that runs on the family laptop. It reads every picture before it suggests deleting one, and nothing is deleted until a person has looked at the pile.

Built for the Hacktoberfest Open-Source AI Challenge, Week 3: Slow and Steady The repository was started on 24 October 2026, inside the challenge window.

What it does

Plug the phone into a laptop over USB. sweep run copies the phone's WhatsApp Images, WhatsApp Video, Screenshots and Camera folders to the laptop (that copy is the backup), then works through them slowly, one picture at a time, with Gemma 4 E2B running locally through llama.cpp. When it is done, sweep review opens a page that groups what it would remove and says why on every picture:

Group	Who decides	Ticked

…

View on GitHub

The rules are in sweep/guard.py, each with a test. The question Gemma is asked is in sweep/model.py. The comparison with my 35 minutes is eval/compare.py, and its results, counts only, are in eval/results.md.

How I Built It

The pieces are small on purpose. Gemma 4 E2B is the smallest Gemma 4 that can see pictures, and it runs on the laptop's processor through llama.cpp, with no graphics card, no cloud and no account. Everything else is Python: adb to talk to the phone, a small database to remember every answer, one web page for the review, and Temporal to keep the overnight run going.

Gemma gets the same instructions for every picture and answers with one line:

{"kind": "greeting", "words": "शुभ प्रभात आपका दिन मंगलमय हो", "people": false, "screen": ""}

I give each picture the smallest size Gemma 4 accepts (70 tokens), which is still enough to read a greeting card word for word. That brought one answer from about 14 seconds down to 5.0 seconds on average, over 1,947 real pictures on my laptop's i7.

Slow is the point. A phone with a few thousand WhatsApp pictures is a night's work for a laptop. So every answer is saved the moment it arrives, the biggest files go first (forwarded videos are where the gigabytes are), and identical files are asked about once. The scan runs as a Temporal workflow: if the model crashes, each picture is retried until it gets an answer, and if the laptop sleeps, running the command again carries on from the next picture.

The scan of my phone, 31 minutes in. It finished with 1,137 pictures looked at.

My 35 minutes against Gemma

The gallery app does not really delete anything for 30 days. It hides the file and keeps it. So after my 35 minutes the phone still held every picture I had thrown away, and I could copy those too and ask Gemma about them, without it ever seeing my choices. The model was the same for both sets, and each set was scanned once.

	Pictures	Gemma alone would delete	Gemma + the rules, ticked	Shown on the page at all
What I kept	1,124	207	5	50
What I had deleted by hand (random sample)	823	94	6	21

The first row matters most. Of the pictures I had decided to keep, Gemma on its own would have deleted 207, including an electricity receipt and an Aadhaar notice. With the rules, five are ticked, and I looked at every one.

Then I labelled a random 100 of those 207 myself, as junk or keep. I called all 100 junk. So on my phone, Gemma alone was right more often than my rules gave it credit for: they ticked 3 of those 100 and showed 17. I am keeping the rules that careful anyway, because the phone this is for is my mom's, and she keeps every receipt.

The second row surprised me. Of the 823 pictures I had deleted by hand, Gemma called 554 "photo", 88 screenshots and 57 documents. Most of my 35 minutes was not forwards at all. It was me deciding about photos, which no tool should decide for me. On my phone the sweep is a second pair of eyes after the clean-up. On my mom's phone, where the forwards are the flood, it is the clean-up.

Things that went wrong

On my own phone, it wanted to delete my electricity recharge receipt. My test pictures were greeting cards and stock photos. My real phone was not. Among the pictures I had kept, the first version ticked 134 for deletion, including an electricity recharge receipt, a food order, an Aadhaar notice, a maths problem I had sent, a Bubble Sort quiz, a letter addressed to me that began "Congratulations! On behalf of the entire team", and birthday posts I had sent. Gemma had read every one of them correctly. It had just called them "ad" or "quote", and my rule trusted "an ad with words on it" to be a forward.

I changed three rules. A forward now needs a forward word (sale, offer, free, good morning, share and so on). Anything that reads like a bill, receipt, order, ID, ticket, bank card, letter or exam question is kept, whatever Gemma called it. Anything in WhatsApp's Sent folder is never ticked, because I chose to send it. Ticked went from 134 to 10, and I looked at all ten: a gambling ad, a meme, two forwarded quotes, a photo of the floor and five black frames from my pocket. These rule changes came after I had seen my kept pictures, so the numbers for that set are after fixes found on it. The pictures I had deleted by hand were scored after that, once.

Gemma started thinking out loud. Once the instructions were long enough to be useful, every answer came back as paragraphs that began "The user wants me to analyze an image…", with no answer in sight. One setting in the request, "enable_thinking": false, turned 14 seconds of rambling into an answer in about 3.

It nearly deleted a family photo because it looked like a greeting card. To save time, I first asked Gemma about one picture from each group of look-alikes and copied the answer to the rest. On my test set, a plain photo got the answer meant for the same photo with "Happy Diwali from the Sharmas" written on it, and landed in the forwards pile. Now an answer is only shared between files that are byte-for-byte identical, and that case is a test.

The review page could delete pictures nobody saw. A second AI agent reviewed the code before it touched a phone. It found that the page showed only the first 600 pictures of a group but would delete every ticked one, and that deletion checked only file sizes. Now a picture counts as seen only once it has loaded on screen, and the phone checks each file's fingerprint (its SHA-1 hash) against the backup before removing it.

Android 14 said "Permission denied" at 3 a.m. The check that the phone can compute those fingerprints read a system file that Android 14 does not allow. It would have refused every deletion. It now hashes a known word instead, and I found out on my own phone, not my mom's.

The faster answer format was the wrong one. A shorter answer format saved about a second per picture. On my test set it also called landscapes "people" and a chat screenshot a "greeting". I went back. Over a night, a second per picture is two hours, but one wrong deletion is a photo you don't get back.

What one run costs. Both sets together were 2.7 hours of laptop time. At a rough 45 W and ₹8 a unit, that is about ₹1 of electricity (an estimate, not a meter reading), and ₹0 in API fees.

Why Does Open Innovation Matter?

Because the pictures are a family's. My WhatsApp folder, and my mom's, holds the last few years: people, prescriptions, a photo of the electricity bill, a cousin's wedding card. I was not going to upload 1,947 of them to someone's server to find out which ones are roses. With an open model, the whole job runs on a laptop I already own.

It is not about the money, and I want to be straight about that. I priced it: sending about 7,000 pictures to a cheap closed vision model costs somewhere between about ₹81 (OpenAI's gpt-6-luna, by my estimate) and ₹345 (Google's gemini-3.5-flash-lite at its default image size) per run. That is less than one month of the ₹130 Google One 100 GB plan. What it really costs is the upload. On the laptop it costs about ₹1 of electricity and a night, nothing leaves the house, and next month it is the same model, because Gemma 4 is a 3 GB file under the Apache 2.0 licence.

And it is fixable. When something went wrong, I could see exactly why, change a few lines and add a test. The model, the rules and the page are all open, so anyone with a phone and a laptop can run it, read why each picture was picked, and change the rules for their own family's groups.

My Agent Session

I built this with Claude Code as my pair, over one long weekend. It wrote most of the code and tests from a plan we agreed on, and a second agent reviewed the delete path before it touched a phone. I chose what the product is for, decided that nothing is deleted without a review, and ran it on my phone overnight.

The Lazy Payoff

By hand: 35 minutes for 1,641 pictures, videos and screenshots (1.89 GB), about 47 a minute, and I had to look at every one.
With the sweep: the laptop read 1,947 pictures in 2.7 hours while I slept, 5 seconds each. My part was labelling 100 pictures with two keys and one scroll through the review page. Among the pictures I had kept, it found ten I had missed: a gambling ad, a meme, two forwarded quotes, a photo of the floor and five black pocket shots (24 MB). The three I scrolled past were deleted after the phone confirmed each file matched the laptop's backup.
What it did not do: decide about photos. That part of my 35 minutes is still mine.
My parents: "This is great, it will save so much time." Their phones are next.

On my phone, 35 minutes of tapping and second-guessing became a few minutes of looking, while the laptop did the reading overnight. On my mom's phone, where nobody is going to sit through 35 minutes, it is the difference between "phone full" and a phone she can take photos with again.

Prize Categories

Best Use of Gemma: Gemma 4 E2B reads every WhatsApp picture, video frame and screenshot on the laptop, and its answer (what it is, the words on it, whether real people may be in it) is what every suggestion is built on.
Best Use of Temporal: the overnight scan of my phone ran as a Temporal workflow, so a crashed model is retried and a closed laptop picks up where it left off.

SetLedger: PRS fingerprints DJ sets in 46 UK clubs. This does it for every other gig

Saharsh Tibrewala — Sun, 18 Oct 2026 16:44:08 +0000

Clubs and festivals pay licence fees for the music played in them, and PRS, the UK's
performing-rights society, shares that money out according to what was actually played. In 46
UK venues it knows exactly, because it has installed music-recognition devices that fingerprint
every DJ set (PRS).
Everywhere else it relies on setlists, and dance music is where setlists go missing. PRS has put
reporting at roughly 27% of sets at dance events, against about 95% at guitar-led festivals,
and at Creamfields in 2018 only 10 setlists came back from around 230 DJs
(DIY).
An unreported set gets allocated from proxy data, which is unlikely to include the unreleased
edit a producer played at 2 a.m. (India has IPRS, and the problem travels.)

This week I built SetLedger, which is the same fingerprinting idea run by the DJ, on their own
recording, for every gig outside those 46 rooms. Give it a set recording and your crate of
tracks, and it hands back a setlist report with evidence attached to every line. My bet is that
a DJ-producer would pay a few pounds a set for that, because it recovers money they aren't
claiming now, and PRS gives members three years to claim live royalties, so last year's
recordings are still worth scanning.

SetLedger never claims a play it can't prove. Every claim traces back to fingerprint evidence,
and anything it can't prove gets flagged for a human instead. Most of this post is about what
went wrong along the way, including the afternoon my own benchmark lied to me.

What I Built

SetLedger is a scanner for DJ-producers. You give it two things already sitting on your laptop,
the recording of your set and the folder of tracks you own, and you get back three things.

A claims timeline shows which of your tracks played, when, for how long, with a confidence
score and the method that proved it. Review spans cover every stretch it couldn't prove,
labeled "review, not claimed". And a setlist report (CSV plus PDF) carries what PRS's
live-performance report asks for, which is the date, the venue, and each work with its duration,
plus whether each work is registered, so you can see which lines you can report today and which
tracks you need to register first.

Behind that sits a ledger. Every confirmed play from every gig goes into Postgres on Tiger Data,
and before each of PRS's four payout rounds SetLedger reads off the statement you'd actually
use: per track, how many plays, how many minutes, and the weakest piece of evidence behind them.

Demo

Live demo: setledger.risp.tech, seeded, no sign-in. Scans run
for real on a small Render server, with nomination going through pgvector on Tiger. If that
server doesn't answer within six seconds, the page falls back to a real scan I recorded from
the same pipeline, replayed on the timeline it actually ran, and the header says which one
you're watching. The first screen tells you how to judge it in 90 seconds: press Scan, watch claims land with confidence scores, click a blue ≈
chip (that play is tempo-shifted, caught by embeddings and confirmed by fingerprints), notice
the amber span that stays unclaimed, then download the report.

Code

github.com/itssaharsh/setledger. The demo corpus is 25 procedurally generated tracks I own outright.
scripts/fetch_corpus.py rebuilds the audio byte for byte, scripts/demo_run.py reproduces the
demo-set benchmark, and scripts/ledger_demo.py generates and scans the five extra gigs and
prints the quarterly statements (it falls back to an embedded Postgres if you don't have a Tiger
account).

The evaluation outputs are in the repo if you'd rather read than run: the
five-gig benchmark and quarterly statements,
the demo-set benchmark, the
Gemma before/after table, and both failure
write-ups in docs/memory/failures/. The README has a
Deploy to Render button that builds the same live scanner from render.yaml.

How I Built It

The bug that taught me what a match is

My first exact matcher picked the alignment with the lowest bit-error rate. Textbook, and
wrong. Techno is loop-based, so a track matches itself at every bar multiple. The scanner kept
locking onto a short, self-similar alignment eight bars off the real one, and my very first
benchmark run rejected a play that was sitting right there in the set.

The fix was to stop asking which offset has the lowest error and start asking which offset gives
the longest run I'd actually be willing to claim. Every offset within a tolerance band gets its
error profile trimmed to the longest stretch of low error, and the longest stretch wins. I wrote
the failure up in the repo because I suspect every beginner at audio matching hits some version
of it.

Embeddings nominate, fingerprints confirm

Chromaprint is unforgiving about tempo. A play sped up 6% (DJs do this constantly) matched the
right track for about 30% of its length, and the rest fell into a review gap. The obvious move
is an audio-embedding layer. The obvious risk is that cosine similarity starts making claims no
fingerprint ever checked, and on a crate of 25 synthetic techno tracks that risk is real: CLAP
scored a track against itself at 0.992 and against a different track at 0.964. Thin.

So the embedding layer is not allowed to claim anything. For each unmatched segment, CLAP
vectors stored in pgvector nominate two candidate tracks. The segment then gets tempo-corrected
across a 0.90 to 1.10 sweep with ffmpeg, and the same exact matcher that guards the rest of the
set has to pass before a claim exists. The claim gets labeled "embedding" with the fingerprint
run's confidence.

In one leftover gap the nominator proposed two tracks and both were wrong. The confirmation step
refused both, the gap stayed amber, and the benchmark logged zero false claims.

My benchmark lied to me

With all that working, the demo set scored perfectly, tempo-shifted plays included. I almost
wrote that number into this post. Instead I generated five more gigs from different seeds and
scanned those too, and the tempo-shifted score fell to 4 out of 10.

It took one diagnostic script to see why, and the answer was worse than a recall problem. Recall
was fine: every shifted play was covered 74% to 97%. But a shifted play gets proven in pieces
(an exact run in the middle, recall-confirmed runs either side, a couple of seconds apart), and
the scanner was reporting each piece as its own play. My scorer only credited the best single
piece, which hid the recall. The ledger counted every piece, so one track showed five plays in a
quarter where it had really been played three times, and in one gig a single play had become
three claims. On a royalty report, that's one play filed three times.

The fix is a dozen lines: join same-track claims that sit within five seconds of each other into
one play, label it "embedding" if any piece needed recall, and give it the confidence of its
weakest piece. Across all six sets the numbers are now:

tier	plays	found	rate
clean	30	30	100%
tempo-shifted (+6% / −5%)	12	12	100%
foreign track, not in the crate	6	0 claimed	no false claims

Each gig now records exactly 7 plays in the ledger, one per real play. The lesson I'm keeping:
a benchmark on your demo data is a smoke test, not a result.

Gemma may reformat. It may not invent, and it may not lose

Crate tags are a mess: MIDNIGHT_HARBOUR_FINAL_v2, artists filed as "unknown", feat. credits
jammed into filenames. PRS's report form matches what you type against registered works, so the
tags need a canonical artist and title. Gemma 3 1B runs locally through ollama on CPU to clean
them up, behind a guard with two rules. Every word in Gemma's suggestion must already exist in
the raw tag, and every meaningful word in the raw tag must survive into the suggestion. On top
of that, placeholder artists like "unknown" are blanked before the model sees the tag, so
echoing one back counts as inventing it.

The guard earned its keep on the first live run, when Gemma 1B turned "Low Orbit Club" into
"Low Orbit" and echoed "Unknown" as an artist. The second lesson came from checking every row
against the true names instead of eyeballing the table: two suggestions kept "Final v2" in the
title. No invented words, so the guard passed them, but PRS's search wouldn't match "Paper Motor
Final v2" to "Paper Motor". So the model now does the language and plain code finishes the
facts, stripping a trailing junk run like "Final v2" and fixing ALL-CAPS names. On the last run,
checked row by row: 14 of 25 tags fixed correctly, 4 already-clean tags left alone, 0 wrong, and
7 sent to passthrough by the guard. All 7 were feat.-credit tags, which is exactly where the 1B
model struggles.

My favorite failure wasn't the model's fault. My synthetic mess generator wrote "Title - Artist"
filenames and Gemma kept "swapping" them back, because real filenames are "Artist - Title" and
its prior knew the convention better than my test data did.

A cold read fixed my UI

Before shipping the interface I gave three screenshots to a fresh agent with no context and
asked it to judge like someone in a hurry. It failed my first screen on the five-second test,
and it was right. Nothing on the page said "DJ" or "royalty", the green/blue/amber evidence
colors had no legend, and the product was, in its words, renting its clarity from the how-to
card. Now the tagline names the user and the job, a legend explains each color, and
tempo-shifted claims carry track names. If you build with an agent, have a second agent grade
the result cold. Mine over-praises its own work about as much as I do.

What each partner does

Tiger Data: the ledger and the vectors. One Postgres holds both jobs. CLAP window embeddings
live in a pgvector table with an HNSW index, and every nomination in the five-gig benchmark is a
cosine top-k query against it (the store also has a tested hybrid query that puts relational
filters such as "registered works only" under the cosine ordering, for crates big enough to
need it). The ledger is a TimescaleDB
hypertable, plays, partitioned on played_at, with a continuous aggregate, plays_monthly,
that rolls plays up by track and month. The quarterly statement reads those pre-rolled buckets
instead of rescanning every play ever recorded, and each row keeps the weakest confidence behind
it and whether recall was involved. All of it runs on a free Tiger Cloud service (Postgres 18,
TimescaleDB 2.30): the hypertable and continuous-aggregate tests pass there, and the five-gig
benchmark above nominated every candidate through pgvector on it and wrote its plays into the
hypertable (the resulting statement is also one
click from the demo's crate panel).

Gemma: tag cleanup with a guard. Covered above. It runs locally on the DJ's own machine, so
tags never leave it, and the before/after table is one click from the demo's crate panel.

.tech: the address. The demo lives at setledger.risp.tech, a subdomain of the .tech domain
I'm keeping for what comes after this.

Render: the live scanner, squeezed into 512 MB. The FastAPI app runs on Render from the repo's
render.yaml and Dockerfile: a Starter instance in Virginia, next to the Tiger database in
us-east-1, using the lightweight spectral recall driver. Fitting it took one fix I'd never have
found on my laptop. Drawing the waveform meant decoding the whole 7-minute set at startup, and
when I measured it, that alone took the process from 51 MB to 407 MB, most of the box before
the app had done anything else. Now the Docker build computes the waveform once and caches it,
and the same container setup peaked around 190 MB when I ran it locally. The .tech page sends
its scans to Render over CORS, and a full scan there takes about 25 seconds. While Render was
still bringing up the first deploy it answered with 502s, which is exactly the case the page's
six-second fallback to the recorded replay is for.

DigitalOcean: built for it, blocked from it. A droplet deploy script (scripts/deploy_do.sh)
and a client for DigitalOcean's Serverless Inference are in the repo, checked against doctl
2.0's own help and a fake server speaking the same wire format. Neither has run on DigitalOcean:
support fixed a team-slot glitch on my account within a day, and then my bank declined the card
with card_declined: transaction_not_allowed. Judge this part as code, not as a deployment.

Why Does Open Innovation Matter?

Everything except the hosted ledger runs on a laptop with no API key, and pointing it at a local
Postgres takes the ledger local too. Chromaprint is open source, CLAP's weights are open,
Gemma's weights are open, and Postgres and pgvector are open source. A royalty side hustle only
works if each scan costs close to nothing, and a per-call API fee eats exactly the small,
specific money this is meant to recover. Open weights also stay put. The model that cleaned this
quarter's tags is the same file next quarter, so a suggestion can be reproduced when someone
asks where it came from, while a hosted model can change underneath you.

My Agent Session (optional)

I built this with Claude Code under an evidence-ledger workflow: nothing counts as done without a
recorded passing run, and every failure becomes a searchable record (the plan is in the repo
under .prod-build/). The ledger once refused to close a task because I'd edited code after its
last passing run. Sending fresh sub-agents after my own work paid off twice: a cold-read judge
caught that my first screen never said "DJ" or "royalty", and a writing reviewer caught that I'd
called my export "PRS-format" without checking what PRS asks for. It's date, venue, works and
durations, and the post now says only that.

My First User (optional)

I showed the demo to a DJ friend who posts his sets on Instagram (he'd rather stay anonymous).
His reply, word for word:

Dang its good Might use sometimes if it also helps me save time dang bro

"Might use sometimes" is the honest part, and it's the most useful thing anyone told me this week.
He didn't mention the money. He mentioned time. Right now SetLedger asks him to find the
recording, point it at a crate folder and press Scan, and for someone who plays every weekend
that's one more chore after the gig. To earn a place in his routine it has to cost zero minutes:
watch the folder his recordings land in, scan each new set on its own, and only ping him when
something lands in review. That's the next milestone. I haven't scanned one of his real
recordings yet either; that's the test I want to run next, and I'll publish the numbers either way.

Limits, so you don't have to find them yourself

Every number here comes from synthetic audio I generated, with tempo shifts I chose. It shows the architecture works, not that club recordings behave. Real sets bring crowd noise, EQ, keylock and mashups, and more of a real set will land in review, which is the designed failure mode.
The tempo sweep covers ±10%, but I only tested +6% and −5%. Key shifts, loops and mashups land in review by design.
Filing stays manual. SetLedger produces the setlist and the evidence; it doesn't submit anything to PRS on your behalf.
One venue, one server, scan state in memory for the demo. The ledger is the persistent part.

Prize categories

Best Use of Tiger Data · Best Use of Render · Best Use of Gemma · Best Use of .tech ·
Best Use of DigitalOcean (code only, see above)

I planted typos to catch AI models editing quotes. 0 of 962 got "fixed" the real failure is sentence boundaries

Saharsh Tibrewala — Sun, 11 Oct 2026 22:18:22 +0000

`
This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

There's a piece of folk wisdom about language models that I believed until yesterday: if you ask one to quote text verbatim, it will quietly clean the text up. Fix your typos. Americanize your spellings. Repair your grammar. The RLHF polish instinct, overriding the instruction to copy.

If that's true, it matters. A journalist who gets back a "quote" with a silently corrected word has a correction to print, maybe a retraction. A lawyer misquoting a contract clause is in sanctions territory. A developer piping "cite the exact line from the log" into an incident report is lying about their own system without knowing it.

There's a famous number floating around this topic: the Tow Center at Columbia found 153 of 200 publisher quotes misrepresented by ChatGPT. But that study was about retrieval: the model had to find the text first. I wanted the harder, cleaner question: what happens when the text is right there in the context window and the only job is to copy one sentence?

So I set a trap. And I'm writing this up honestly: the trap caught nobody. What I found instead is, I think, more interesting.

The benchmark

I wrote 10 short original documents (200 to 250 words each): an internal memo, release notes, an incident postmortem, hosting terms, a travel letter, meeting minutes, a hardware spec, an email thread, a README, and a press release. Ordinary workplace prose, written fresh for this benchmark, so there's no copyright issue and no training-data contamination.

Into them I planted 18 irregularities, on a temptation gradient:

Class	Planted in the corpus
typo	"customers ... currently recieve two separate case numbers"
grammar	"This library have zero dependencies"; "an approach the company argues is more simpler"
British spelling	"Whilst the platform is loved by the core team..."
wrong fact	"DNS, which resolves IP addresses to domain names" (it's the reverse); "the Pacific, the world's second-largest ocean"; "1.2 kg (2.2 lb)"
near-duplicate	two sentences identical except phone/tablet and 4.6/5.1 seconds

Each dirty sentence is paired with clean control sentences from the same documents: 37 items, 18 dirty, 19 clean. Every item asks one thing:

Quote verbatim the sentence that [uniquely identifies one sentence]. Output only that sentence, exactly as it appears in the document, and nothing else.

Each item runs twice: once with the plain instruction above, and once with an explicit warning bolted on: "Reproduce it character-for-character, preserving any typos, spelling choices, grammatical errors, or unusual wording exactly as written - do not correct, complete, or clean up anything." If models fix text under the plain instruction but stop when warned, that's an instruction-following gap with a one-line mitigation. If they fix text even when warned, the cleanup happens somewhere prompts don't reach.

Scoring is deterministic. No LLM judge anywhere. A quote is faithful if it matches the planted sentence exactly (I track raw byte-equality and a conservative normalized match that only forgives curly quotes, unicode dashes, and whitespace). Everything else is classified: corrected (the planted error got fixed), truncated (a partial quote), wrong_span (real text, wrong boundaries), or rewritten (text that exists nowhere in the document). The scorer is about 60 lines of Python with unit tests, and you can audit any single verdict with Ctrl+F.

My Benchmark

https://www.kaggle.com/benchmarks/saharshtibrewala/exact-words

Everything is public in the task notebook: all 10 documents, all 37 items, the scorer, and every one of the 962 model responses.

The Models I Tested and Why

13 models, 74 prompts each, all run on Kaggle Benchmarks: open-weight models because it's Hacktoberfest and self-hosters deserve to know (Gemma 4 31B and 26B-A4B, GLM-5, Qwen3-Next-80B, DeepSeek-R1, GPT-OSS-20B and 120B), and closed models as reference points (Claude Haiku 5.5 and Sonnet 5.5, GPT-5.4-mini and GPT-6-sol, Gemini 3.5 Flash-Lite and 3.7 Flash). I deliberately included small/large pairs within three families, because my prior was that more polish means more silent correction, and family pairs isolate that.

What I Found (And What Surprised Me)

Finding 1: the trap caught nobody. 0 silent corrections in 962 quotes.

My headline hypothesis is dead, and I have receipts. Across 13 models, two instruction arms, and 962 scored responses, not one model fixed a planted error inside a quote. "Recieve" survived 26 encounters. So did "more simpler", "This library have", the reversed DNS definition, and "the Pacific, the world's second-largest ocean", which I was certain someone would quietly promote back to first place.

It's stronger than that. Of 911 faithful quotes, every single one was byte-identical to the source. Not one needed even my gentle normalization. When a 2026 model copies, it copies exactly.

And the twelve responses that produced text existing nowhere in the document? Five of them contained a planted error, and all five preserved it. Even while mangling a quote's boundaries, no model touched the errors. Whatever typo-fixing machinery lives inside these models (there's published mechanistic work on dedicated typo-correction structure in LLMs), the quoting context switches it off. Folk wisdom: falsified, at least for short documents in 2026.

Finding 2: what actually fails is the sentence boundary.

51 of 962 quotes failed, and the taxonomy is lopsided: 26 truncated, 13 wrong-span, 12 rewritten, 0 corrected. Nearly every failure is a disagreement about where a sentence starts and ends:

The colon problem. The hardest item in the whole benchmark (13 failures out of 26 attempts) was: "But a part of it is our own doing: customers who contact us twice about the same order currently recieve two separate case numbers, and about a tenth of those end up double-refunded." Models split on the colon. Gemini 3.5 Flash-Lite started the "sentence" at "customers" and capitalized it to "Customers", producing a sentence that exists nowhere in the memo (typo intact!). Claude Sonnet 5.5 went the other way and glued on the previous sentence. GPT-OSS-120B stopped mid-clause. One sentence, three different theories of its edges.
The neighbor-sentence problem. Asked for "the final sentence of the letter", models failed 9 times out of 26: Sonnet 5.5 and Qwen3 returned the final two sentences ("Some trips end with a landmark. This one ended with..."), and Flash-Lite once returned only "Some trips end with a landmark.", which is the second-to-last sentence, alone. The instruction said one sentence; they knew where the text was and disagreed about which one it was.
Section labels. GPT-OSS-20B prepended "4.3" (and elsewhere "4.2") to terms-of-service quotes, faithfully copying labels that aren't part of the sentence.
Fabricated punctuation. DeepSeek-R1 cut a long sentence after its first clause and appended a period that isn't in the source, with the reasoning trace visibly talking itself into it: "The period after 'water' is part of the original sentence structure." It isn't; the source continues with a comma. The one true paraphrase of the whole run also lives here: Flash-Lite, in the strict arm, restructured "The extended battery delivers a full two-shift day, defined as..." into "A full two-shift day is defined as..." — the only failure in 962 that reads like the folk-wisdom rewrite, and it's 1 case, not a pattern.

Finding 3: telling it not to correct does nothing (because there's nothing to prevent).

Plain arm: 95.0% faithful. Strict "do not correct anything" arm: 94.4%. A null result, but a clean one: the explicit warning neither rescued the boundary failures nor prevented corrections that were never going to happen. If you've been adding "preserve all typos exactly" to your prompts, on this evidence it's a lucky charm.

Finding 4: bigger is worse, twice out of three.

GPT-OSS-120B: 82.4%. GPT-OSS-20B: 89.2%. The 6x larger sibling is 7 points worse at copying, mostly through truncation. Both siblings share a family tic: swapping plain hyphens for U+2011 non-breaking hyphens inside otherwise perfect quotes ("day‑old", "hot‑swappable"), a single invisible character that makes the quote no longer exist in the source.
Claude Sonnet 5.5: 93.2%. Claude Haiku 5.5: 94.6%. The smaller model wins again.
Gemma 4 31B and 26B-A4B: both a perfect 100%, every byte, both arms, alongside Gemini 3.7 Flash, GPT-6-sol, and GLM-5.

Five models walked through my trap untouched, and two of them are open-weight models you can run yourself. The scaling intuition ("the big model is better, use it for precision work") points exactly the wrong way on this task.

The grader is always the first suspect

Two confessions, because this genre earns its keep through self-audit.

First: my initial leaderboard showed DeepSeek-R1 at 0.0%, with 100% of its quotes "not in the source". Alarming headline, completely false. R1 wraps its answers in <think>...</think> reasoning, my scorer was grading the thinking instead of the answer, and the real answer was sitting right after the closing tag. One line of stripping later, R1 scores 91.9%. If I had shipped the first number, the first commenter who knows R1 would have ended this post. (Its remaining failures are real: truncating clauses and inventing terminal periods, reasoning aloud about punctuation the whole time.)

Second: this isn't even the benchmark I started building yesterday. The first one asked models to compute cron firing times across daylight-saving transitions. I killed it after discovering that cron implementations themselves disagree about DST semantics (Vixie cron skips the nonexistent hour, Quartz handles it differently), which means my "ground truth" would have been one dialect's opinion. And while building that grader I inverted PEP 495 fold semantics and spent twenty minutes confidently marking correct answers wrong. Both experiences set this benchmark's design rule: the correct answer must be a string I planted myself, findable with Ctrl+F, with no dialects and no judge model.

Limitations (read before quoting my numbers)

37 items, 13 models. Big enough that 0/962 corrections is a real signal; small enough that I hand-verified every item. Per-class splits (4 typos, 3 wrong facts) are directional.
Short documents (~250 words). Copy fidelity at 200 words deep tells you nothing about 50,000 tokens deep. I built a combined-document variant (same questions, all ten documents fused into one 2,225-word distractor field) that has not run yet; it is my first follow-up.
Run-to-run variance is real. I ran the full suite more than once at provider-default sampling; per-model scores wobble by 1–4 points between runs, while the headline findings held every time: zero corrections, the same five models at 100%, byte-identical faithful quotes. Numbers in this post are from one complete 962-call run; per-item CSVs are public in the notebook output.
English workplace prose only. Code, poetry, non-Latin scripts: open questions.
Exact-match is strict on purpose. Several failures differ from the source by a single character, like a non-breaking hyphen or an invented terminal period. I publish the full taxonomy so you can draw the line elsewhere; the CSVs have every response. A handful of "rewritten" verdicts are models adding a lead-in like "The sentence is:" despite "output only that sentence", which I count as failures of the task as instructed.
My sentence boundaries follow period-to-period convention; the colon item shows models hold other theories. I'd call that a finding, but you're free to call it item ambiguity, and the item-level data lets you recompute without it.

What I'd Measure Next

Depth stress: the combined-document variant above, then real long-context distances.
Code: quote the exact line from a stack trace or diff, where an invisible "fix" is maximally dangerous.
The colon census: a benchmark made entirely of sentences with mid-sentence colons, semicolons, and conjunctions, since that's where models' sentence-boundary theories diverge.
Trailing-punctuation fidelity as its own tracked class; it may explain a chunk of "near-miss" exact-match failures in other benchmarks.
Reasoning-model output hygiene: R1's failures all happened inside otherwise-correct answers; how much benchmark noise across the industry is really answer-extraction noise?

Honesty section

The documents, planted errors, benchmark design, and analysis are mine; I used AI coding tools (Claude Code) to accelerate the harness, scorer tests, and charts, as the challenge rules permit. Every item was hand-verified against its document, and every number in this post comes from the public run on Kaggle. The project was built during the contest window.

Benchmark: https://www.kaggle.com/benchmarks/saharshtibrewala/exact-words. All code, documents, and per-item results are public. If you find a scoring bug, comment; as established above, the grader is always the first suspect.
`