This is my submission for the Hacktoberfest Open-Source AI Challenge: Week 3, "Slow and Steady".
"0 flagged" tells you nothing
I built a tool that reads a group-chat export and writes a short digest: what was decided and what remains an open question. A local model writes the digest. Then a second pass re-reads the chat and checks every line the first pass wrote.
On my first test run, the second pass flagged nothing on two of the three days. That sounds good. It is also exactly what a checker that cannot find anything would report. A quiet checker is not evidence of a correct digest.
So I did what I should have done first: I wrote wrong items on purpose and watched whether the checker caught them.
What I Built
nightshift reads a LINE chat export, hides private details, and works through the chat one day at a time with a local open model (Gemma 3 12B through Ollama). It leaves a Markdown digest of decisions, open questions and news, grouped by day.
It is built to be slow and careful, not fast:
- Hide first. Before the model sees anything, emails, links, phone numbers, long numbers and voucher-style codes become placeholders, and a list of names I choose is blanked out.
- Resumable. Each finished day is saved straight away. Stop the run, let the laptop sleep, and run the same command again; it carries on.
- No quote, no item. ** Every line must carry a few words copied from one message. If it's not in that day's chat, drop the line.
- A second look. Each surviving line goes back to the model with the whole day and one question: is this really what the messages say? Any number in the line must also appear in the chat, which needs no model at all.
- You decide. ** A small review page shows each line beside its quote and what the second pass said. I mark it keep, edit or wrong.
Everything runs on my own machine. Everything shown here comes from a fabricated Thai group chat about a trip to Chiang Mai, written for the repo. Nobody in it is real.
I built this in a pair-programming session with Claude Code. It wrote most of the code, the tests and the 22 probe items below; I ran the demo and the probe on my own MacBook, and the demo and probe numbers are what my machine printed.
Demo
All three come from the fabricated chat. The names in them are made up.
Code
nightshift: an overnight digest of your LINE chat
Drop a LINE chat export in before bed. A local open model (Gemma 3 12B through Ollama) works through it one day at a time while you sleep, and leaves a Markdown digest of decisions, open questions and news. Nothing leaves your machine.
Built for the Hacktoberfest Open-Source AI Challenge: Week 3, "Slow and Steady".
Status: work in progress. The code, the review page and the tests are in place; the real night run and its evaluation (see EVAL.md) have not been done yet, so there are no accuracy numbers here.
How it works
LINE .txt export ─► parse ─► one chunk per day ─► Gemma 3 12B (Ollama, local)
─► items: decision / question / point, each with a quote
─► quote check (drop any item whose quote is not in the chat)
─► second pass: numbersMIT licensed. To try it:
ollama pull gemma3:12b
uv sync
uv run python run.py demo/demo_chat.txt
uv run python app.py # review page
uv run python probe.py # the planted-error test
How I Built It
Parsing a real export. I wrote the parser for the format I expected, with tabs between fields. On a real LINE export, it found zero messages: the real file separates time, name and text with plain spaces, and names can contain spaces and emoji. The fix infers each name from the file itself: a name is the longest run of leading words that all of that person's lines share. After that, the real export parsed correctly.
Redaction, and a leak the tests caught. One rule treats the Thai word for "password" as a keyword and redacts the value after it. My first version let that keyword swallow part of a glued code, leaving a few characters visible. A unit test caught it, and I limited the keyword to Thai letters. A second gap: an email-like string without a .com ending slipped through until I made the ending optional. Redaction is a safety net, not a guarantee. It matches patterns, and a Thai nickname is not a pattern, so names go in a list I write myself, and a --preview option shows exactly what the model will read.
A review page that lost my clicks. Each click on keep, edit or wrong rewrites one small file. Gradio runs every click in its own thread, so two quick clicks could both read the old file and the later write would erase the earlier verdict. A unit test that clicked from many threads at once failed every time; a lock around the change and an atomic file write fixed it.
Resumable by design. Each day becomes one small JSON file, written under a temporary name and then renamed, so a crash cannot leave a half-written result that looks finished. A result is reused only if it came from the same text, model and settings. Changing the redaction list, or turning the second pass off, quietly redoes the affected days.
The second pass. Two checks, cheapest first. First, every number in an item must appear in that day's messages. Second, the model gets the whole day plus the item and its quote, and must answer supported, unsupported or unclear with a reason. I told it to be strict: an item is unsupported if it adds a name, date, number or who-did-what the messages do not state; if a question it calls open is answered later that day; or if a decision it reports was never agreed. An unsupported item stays out of the digest unless I put it back, so nothing is deleted behind my back.
The obvious weakness is that the same model is grading its own work. I did not want to assume the check helps, so I measured it.
What happened
The demo run. On the fabricated chat (20 messages over three days), the model wrote 14 items in total. The quote check dropped none, and the second pass flagged one on the third day. The three days took 39.4, 14.3, and 22.1 seconds, about 76 seconds total on a 16 GB M1 Pro; the first day includes loading the model. The second pass took 41.3 of those 75.8 seconds, more than half. I ran the first day alone, stopped, and ran the command again. It reported done 2, already done 1: it skipped the finished day and did only the other two.
Judging the 14 items. I read every item against the chat on the review page: 14 keep, 0 edit, 0 wrong. ** The one item the second pass flagged, I judged correct, so that was a false alarm: 1 of 14. It missed no wrong item, but there were none to miss, so this run says nothing about what the second pass misses. That is the gap the next test fills. Reading the finished digest again, two lines on the first day sit uneasily together: one lists "who will drive" as unclear, and another refers to "the driver". In the chat, one person says right away that he can drive. I judged both keep and I stand by that, but a reader looking at the digest will see the tension, so I am pointing it out. A stricter reader may call the first line misleading. Two more caveats: the chat is 20 messages I wrote, so it is easy, and the digest items came out mostly in English, although I asked for the chat's language (the quotes stay Thai).
Trying to trick it. probe.py feeds the second pass 22 items whose answers I know: 7 correct, and 15 wrong on purpose in seven ways. All of them are made from the fabricated chat. A test checks that every quote really is in the chat, and that each wrong item can only be caught the way its label says.
| Result | |
|---|---|
| Wrong items flagged | 13 of 15 |
| Caught by the number check alone (no model) | 4 of 4 |
| Caught by the model's re-read | 9 of 11 |
| Correct items wrongly flagged | 0 of 7 |
Set that next to the demo run, where the second pass flagged 1 of 14 correct items. "Zero false alarms" on the probe was a little rosy; both samples are tiny.
It caught every example I tried of: a number that is not in the chat, a number from the chat used in the wrong place (a deposit and a per-person amount swapped), the wrong person, an invented detail, an overclaim ("everyone paid" when one person had), and a question reported as open that was answered a few lines later. For some of these kinds I only had one example, so that is a hint, not a rate.
What it missed. The same kind of error, twice. One item said everyone agreed to split the fuel cost; in the chat, one person asked, and nobody replied. Another said the group agreed to leave at 7 a.m.; in the chat it was a request nobody had answered yet. The second pass called both supported. Its reason for the second one: the quote matches a message in the chat. It checked that the sentence was said. It did not check that anyone agreed. Of three items of this "never actually agreed" kind, it caught one, so it is not that it cannot do it; it does it unreliably. This is also the kind of mistake that makes a digest dangerous, because "we decided X" reads as settled.
What I did not do. I planned to judge at least 20 digest items from a real chat against a bar I wrote down first. I did not run that. The only real chat I have is a work conversation with other people's details, and I wasn't willing to publish results from it. So the accuracy bar in the repo's EVAL.md (at least 70% correct, at most 10% wrong) is untested on real chats. On the fabricated chat, the same bar would be met: 14 of 14, but that says nothing about real chats, and nothing here is an accuracy rate for them. What I can stand behind is narrower: on 22 planted items, 13 of the 15 wrong ones were flagged, and none of the 7 right ones was.
Why Open Innovation Matters
A chat export is full of other people's messages, which is exactly what you do not send to someone else's server. An open-weight model let me run the whole pipeline on my own laptop with no account and no per-token cost. It also made the careful parts affordable. A second model call for every item would bill on a hosted service; here it is only time, and time is what "slow and steady" is about.
Local is not an excuse to be careless, though, so the chat is redacted before even my own model reads it.
Honest limits
- Every result above comes from a fabricated chat of 20 messages and 14 items, and from 22 probes written by hand. Neither is a benchmark. The chat produced no wrong items, so it couldn't show what the checks miss; only the probes could.
- The digest items came out mostly in English even though the prompt asked for the language of the chat. I have not fixed that.
- The probes were written after the tool existed, so they test the mistakes we thought of, not what real chats contain.
- The second pass is the same model checking its own work. It missed "agreed" claims twice, and I have not tuned the prompt. Tuning it against these same probes would make the numbers meaningless, so the next step needs new probes it has never seen.
- The number check is strict: it flags a figure the model worked out itself even when the arithmetic is right.
- Redaction catches patterns, not people.
- I have not run this overnight. A small chat finishes in minutes.
What's next
Write fresh probes for the "agreed" errors, measure the current prompt on them, then tighten the prompt and measure again, reporting both numbers. Try a second, different model as the checker so it does not share the first one's blind spots. And run the tool on a real chat with the participants' consent, judge at least 20 items, and report against the bar I set before looking.
Top comments (0)