inbox-triage: Out of 100 emails, only a few matter. I built a tool that finds them.

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
Out of 100 emails in my inbox, a few matter: an internship offer, a hackathon, a reply from a repo owner. The rest is noise. I built a tool that sorts every email into four labels (deadline, needs reply, might interest you, skippable) and shows me the few that matter first. It never deletes, moves or sends anything.

What I Built

I am a college student. Travel to and from college, assignments and exams take most of my day, and open-source work takes what is left. A few important emails still reach my Gmail: an internship offer, a hackathon I care about, a reply from a repo owner. Everything else is noise, and students get a lot of it, especially fake promotions. I do not want to know that someone I barely know commented on a LinkedIn post. I want to know whether my pull request was merged. Out of 100 emails, usually only a few matter, and reading all 100 costs time I do not have.

So I built a triage tool for people like me. It gives every email one label:

Label	What it means	Example
Deadline	You must do something by a date. The tool shows the date and weekday.	"Submit your lab record by 27 October"
Needs reply	Someone is waiting for your answer, and no date is attached.	"Which logo do you prefer, A or B?"
Might interest you	Optional, but possibly worth a look.	A hackathon, a contest, a talk
Skippable	Ads, receipts, notifications, and other people's deadlines.	"FLASH SALE ends tonight!"

The report is built to feel like Gmail so it is not scary. The first screen shows only the counts. A fifth folder, OTP codes, is filled by a simple rule instead of the model: if an email names a one-time code and contains one, it goes there with the code shown on its row. You open one label at a time, and each email shows the exact line the label was based on.

Spam and promotions go to Skippable. Skippable means you can ignore the email. It does not mean the email is safe, because the tool does not check for phishing. It also never deletes, moves or sends anything.

Demo

Video: https://youtu.be/7rRF_DzIY7Q

The demo page runs on the 96 synthetic test emails. Every sender, name and address in it is made up. In the video, my own emails appear only as blurred counts.

• Screenshot 1: the overview with the counts

• Screenshot 2: the Deadlines list, sorted by date

• Screenshot 3: one email opened, showing the quoted line the label came from

Code

Repository: https://github.com/BhoomikaC15/inbox-triage

Everything is plain Python with the standard library, except the Tinker fine-tune. To try the local pipeline on a mailbox export (.mbox or a folder of .eml files):

python triage.py inbox.mbox --model gemma3:4b --limit 10 --out report.html

To reproduce the results below:

python evaluate.py test.jsonl preds_baseline.jsonl
python evaluate.py test.jsonl preds_gemma.jsonl
python evaluate.py test.jsonl preds_qwen_base.jsonl
python evaluate.py test.jsonl preds_tinker.jsonl

The repo includes the test set, the training set, the generator for the training set, and every prediction file. My Tinker API key is not in the repo.

How I Built It

I wanted numbers, not a demo that only works on three emails. So I built the test first.

The test set

I used 96 fictional emails: 25 deadline, 25 needs reply, 19 might interest, 27 skippable. Claude drafted them, and I reviewed every label myself. About a third are hard on purpose: dates like "by Friday" or "within 3 days", two dates in one email, someone else's deadline, fake urgency in ads, quoted old replies, and optional events that carry a date. Nothing in the test set was ever used for training.

Four methods, same test

• Regex baseline: keyword rules and a date parser. This is what any model has to beat.
• Gemma 3 4B, run locally through Ollama on my laptop (8 GB RAM, no GPU).
• Qwen3.5-4B, untuned, through Tinker, with the same long instruction prompt Gemma got.
• Qwen3.5-4B, fine-tuned on Tinker (LoRA, rank 16) on 320 synthetic training emails, 3 epochs, 60 steps, about 170,000 tokens in total. A script generated the training emails and computed their labels and dates, and any email too similar to a test email was dropped. ### Code checks the model

The model only does the fuzzy part. For each email it returns a label, a deadline date and one quoted line. Plain code then checks the answer against the email:

• A date is trusted only if it actually appears in the email. "By Friday" is resolved against the sent date.
• The quoted line must really be in the email.
• If the model's answer cannot be read, the email is shown as needs reply instead of being hidden.
• If a date was dropped by the check, code fills it in only when the email holds exactly one candidate date or says "within N days". With several candidates it leaves the date empty. ### Results on the 96 test emails

Method	Accuracy (95% interval)	Important emails marked skippable	Exact deadline dates
Regex baseline	68.8% (59–78%)	11 of 50	14 of 25
Gemma 3 4B, local	76.0% (68–84%)	0 of 50	21 of 25
Qwen3.5-4B, untuned	94.8% (90–99%)	1 of 50	20 of 25
Qwen3.5-4B, fine-tuned	97.9% (95–100%)	1 of 50	23 of 25

The fine-tuned model is clearly better than the baseline and Gemma. Against untuned Qwen it fixed 5 emails and broke 2, so the gap is within noise (p about 0.45). I do not claim the fine-tune is more accurate. What it changes is the prompt: it needs only a one-line instruction. Measured with the tokenizer on the 96 test emails, an average email costs 133 input tokens instead of 368 (2.8 times fewer). At Tinker's list prices that is about $0.09 instead of $0.16 per 1,000 emails. If the long prompt's shared prefix is cached, untuned Qwen costs about $0.08, so the saving disappears. The fine-tuned model also writes slightly more output (42 tokens against 35). These are estimates from the price table, not from my bill, and I did not time latency.

With the code fallback, the fine-tuned model gets 24 of 25 exact dates. I added that rule after seeing the test errors, so treat that figure as post-hoc. The cleaner evidence: on the 320 training emails, the rule alone resolved 84 of 96 deadline dates exactly, got 2 wrong and left 10 empty.

What went wrong

• Gemma invents deadlines. It labeled 16 emails as deadlines that were not, and only 14 of 27 skippable emails were correctly skipped. It errs toward showing you too much. It also never hid an important email (0 of 50), which matters most for an inbox cleaner.
• Small models get weekday arithmetic wrong. Both Gemma and untuned Qwen lost the same four deadline dates. For three of them the correct date was easy to find in the text with plain code.
• The fine-tuned model still made three mistakes. It marked a real deadline (a library membership expiry) as skippable. It called an optional contest a deadline. And it could not resolve "within 3 days" until the code fallback did.

• Gemma is slow on my laptop: about 36 seconds per email, so one hour for 96 emails.

  Limits

• The test and training emails are all synthetic. Claude drafted the test emails, which could make them easier for another AI to read than real mail. My only real-inbox check is a small spot check (below), not a benchmark.

• My training emails are more templated than real mail, and the fine-tune's loss fell to almost zero. The test set is the check, and it held up, but real mail will be messier.

• My best result ran on Tinker's cloud. Only the Gemma path keeps emails entirely on my machine, and it is the weaker one.

• I did not test a closed model, so I cannot say how much better one would do. The OTP rule was checked on only a handful of OTP emails, and I added it after realising I had forgotten that label, so it is outside the measured results above.

• My labels have no slot for an important notification, like "your pull request was merged". That email would probably land in Skippable or Might interest you. That is exactly the kind of email I care about, so it is the first thing I would fix.

Why Does Open Innovation Matter?

Email is the most private data most people have: offers, bank notices, messages from family. A tool that sorts it should not need to send it to a company's server. Open-weight models make that possible. Gemma 3 4B and Qwen3.5-4B are small enough to run on an ordinary student laptop. Mine has 8 GB of RAM and no GPU, and Gemma ran there, slowly but without sending a single email anywhere.

Open models also made the fine-tune affordable. I trained Qwen3.5-4B on Tinker with a $10 credit, and the training run used about 170,000 tokens. I could afford to try it, compare it to the untuned model, and report that the gain is small, instead of just trusting a bigger model.

My Agent Session

I built this with Claude as a coding agent over about three days, working from a plan I set at the start: measure first, and report whatever the numbers say.

What Claude did: drafted the 96 test emails and the generator for the 320 training emails, wrote the baseline, the scorer, the Gemma and Tinker scripts, and ran its own checks before handing files over.

What I did: chose the idea and the labels, reviewed every test label, ran everything on my own laptop (Ollama, Tinker login, the full scoring runs), debugged my setup, and decided which results to claim.

Where the session changed the project:

• It started as a group-chat summarizer. We switched to email because email is more private and because I actually hate it.
• Claude pushed me to build the test set and baseline before any model. That is why I can say Gemma only scored 76%.
• When untuned Qwen already scored 94.8%, we did not hide it. The post says the fine-tune's gain is within noise.
• When I ran my own emails, I found problems the synthetic set never showed: long promo emails that overflowed the model, and I had forgotten the OTP label. Both are fixed. Session link or transcript: not available. I built this before I knew about DevRelay, so I could not save the session there. The account above is my own summary of what Claude did and what I did.

The Lazy Payoff

The work I stop doing is opening emails to find out whether they matter. I look at the counts, open Deadlines and Needs reply, and leave the rest. Every opened email shows the line the label came from, so I can trust a label without reading the whole message.

My test set is balanced on purpose, so about half of it needs action. My real inbox is not like that: far fewer than half of my emails matter. I also ran the tool locally with Gemma on my own emails. On a first run of 20 I agreed with 17 labels. I then fixed what I found (long promo emails, no interests list, no OTP folder) and reran on 25, where I agreed with 23 of 25. That is a small spot check, not a benchmark, and since I changed the tool after the first run it is not independent evidence. I have not timed myself on a real inbox, so I will not claim a number of minutes saved.

Prize Categories

I am entering the two categories this project honestly uses.

• Best Use of Tinker. I fine-tuned Qwen3.5-4B on Tinker for one task: sorting email into four labels. Against the regex baseline, accuracy went from 68.8% to 97.9% on 96 held-out emails. Against the untuned Qwen3.5-4B (94.8%) the accuracy gain is within noise, so what I can show is a smaller prompt: 133 instead of 368 input tokens per email, about $0.09 instead of $0.16 per 1,000 emails at list prices, with no saving if the long prompt's prefix is cached. Training cost about 170,000 tokens on a $10 credit.
• Best Use of Gemma. Gemma 3 4B runs locally through Ollama and is the private path in the tool: it labels emails on my laptop without sending them anywhere. It scored 76.0%, below Qwen, and the post reports where it fails. I am not entering other categories, because I did not use those partners' tools.