This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
What I Built
A buyer opens a dispute on a $29 set of linen kitchen towels: "The box was crushed and the item inside is damaged." You packed it in a double-walled box with bubble wrap. USPS shows it delivered four days after it shipped. The platform gives you until the end of the month to answer.
Now you have to dig up the order, the tracking, the message thread and the policy, and write something calm. It's late, and the whole thing is worth $29. Plenty of people just refund it.
I made this case up. It's case_002 in my test set, and every case in this project is synthetic. It is the shape of problem I wanted to attack: the evidence is scattered, the deadline is short, and the amount is too small to feel worth the effort. That is my premise, not a statistic.
Dispute Defender is a local AI that drafts the seller's reply, and every clause has to cite numbered evidence. You give it a case folder (dispute notice, order CSV, messages, photos). It gives back:
- numbered evidence (E1, E2, ...) and a list of what's missing,
- a fight-or-fold verdict (for case_002: FIGHT, score 0.45, expected recovery $13.05, from a simple heuristic of mine, not a calibrated probability),
- a letter where every clause ends in a citation like [E5],
- a REPAIRS list of anything the code had to change.
Who would pay? The person I built it for is a small marketplace seller who handles disputes alone, in the gaps between other work. I haven't asked anyone whether they'd pay, so I'm not putting a price on it. [Optional: your price idea, labeled untested.]
Order 9814174 for Linen kitchen towel set ($29.00) was placed on 2026-01-13. [E2]
The carrier shows the package delivered on 2026-01-18. [E4]
On 2026-01-23, the buyer wrote: "The box was crushed and the item inside is broken." [E5]
Code
Dispute Defender
A local AI that drafts a seller's reply to a marketplace dispute, where every clause must cite numbered evidence.
A buyer says the item never arrived, was not as described, or arrived damaged. The seller has a short deadline, the facts are scattered across an order record, tracking, a message thread and a policy, and the amount is often too small to feel worth the effort. Dispute Defender builds the evidence list, drafts the letter with a local Gemma model, then checks each clause against only the evidence it cites. Code repairs what it can without adding a word and drops the rest.
- Model: Gemma 3 4B, open-weight, running locally through Ollama. Customer data stays on the seller's machine.
- Code: Python 3.10+, standard library only (nothing to install).
- Built for the DEV Hacktoberfest Open-Source AI Challenge, Week 2 ("Side Hustle"). The project and this repository were…
Python standard library only (nothing to install), 163 tests, MIT license. The README has the commands to reproduce every number below.
How I Built It
The open-source AI: Gemma 3 4B, an open-weight model, running locally through Ollama. The project is built around one rule: a language model never decides what counts as true.
case folder -> numbered evidence -> triage -> Gemma drafts -> verify each clause -> retry once -> repair or drop
Each clause must cite numbered evidence. A verifier checks each clause against only the evidence it cites. Code then repairs what it can without adding a word, and drops the rest. Everything after the model is plain, deterministic code.
That rule caught real failures:
It described photos it never saw. My photo evidence is only a file name. Gemma wrote that the photo "shows the scarf packed in its original condition," and kept doing it. A stricter prompt cut photo claims from 7 of 15 runs to 1, but it created new failures (evidence ids in prose, uncited lines) and made the model slower. So I took the decision away from the model. Code now writes the photo sentences and the model never sees them.
It cited a true fact to the wrong evidence item. In one case, all 3 runs wrote "Tracking shows it was delivered on April 22 [E6]." The fact is in E7. The repair added E7 without adding a word. In another case, the verifier dropped "The buyer opened a dispute on 2026-06-06 [E6]." That date appears in two evidence items, and the repair only re-cites when exactly one other item contains the fact. With two, it refuses to guess and drops the sentence. That can lose a sentence that was true. I chose to fail closed.
It restated the buyer's claim in its own words. "The buyer says I have not received my order [E4]." The verifier flagged received, which isn't in E4, and dropped the sentence. A paraphrase that's fair to a human got cut. That is the cost of a word-level check.
With the rules removed, it turned into a customer-service bot. On a minimal prompt, Gemma wrote "Thank you for reaching out" and "I understand your concern that the package never arrived, despite the tracking." The verifier flagged never and the uncited line after it. I also added a check that rejects apologies and admissions unless the evidence itself contains them.
The numbers. "First draft clean" means the verifier found nothing wrong in the model's first attempt. All rows are Gemma 3 4B on one ASUS laptop (about 46% of the model on the GPU), on 30 generated cases (26 scored; 4 were fold verdicts).
| Setup | Runs scored | First draft clean | Content dropped | Attempts | Seconds/case |
|---|---|---|---|---|---|
| Full prompt, 1 run per case | 26 | 92% (24 of 26) | 4% | 1.04 | 25.2 |
| Full prompt, 3 runs per case | 77 | 94% (72 of 77) | 3% | 1.03 | 25.5 |
| Minimal prompt, no fine-tune | 26 | 0% (0 of 26) | 50% | 2.0 | 42.1 |
The three-run row isn't three independent confirmations: three of its five non-clean runs are the same case. Put differently, 23 of 26 cases were clean in every run. One run failed with an Ollama server error and is not counted.
Cutting the prompt to save time made the model worse and slower, because every case needed a second attempt. In fairness, the minimal prompt is designed for a fine-tuned model, and I ran it on the untuned one. It shows what happens when the rules are left out.
Earlier, on 15 runs of 3 hand-built cases, 40% of first drafts were clean when the model wrote the photo sentences, and 60% with the stricter prompt. Letting code write them gave 100% clean on those same cases. That 100% is on cases I tuned against, so it is not a held-out number. On 20 unseen generated cases an earlier build scored 95%, and on 12 scored runs of a harder generator on a CPU-only laptop it scored 67%. Those came from different builds and machines, so I'm not claiming a trend. On the same 6 cases on the ASUS, the full prompt gave 4 of 6 clean and the minimal prompt 1 of 6.
Limits
- The Etsy policy and checklist are placeholders. No real policy is encoded.
- The verifier checks words, not meaning. A misleading rearrangement of true facts can pass.
- Every case is synthetic, written by my own generator, and the samples are small.
- The win score is a heuristic I made up.
- On a CPU-only laptop it took about 136 seconds per case on my harder cases.
Why Does Open Innovation Matter?
Dispute data is customer data: names, addresses, messages. Because Gemma's weights are open, the whole thing runs on the seller's own machine, offline, and none of it leaves. There's no per-token bill either, only electricity.
Open weights also let me do the work this project needed. I could read every raw draft, find that the model invents photo details, and build a verifier around its mistakes. I could pin one model file and rerun the same 30 cases to get the numbers above. With a closed hosted model I couldn't fix which version answered or inspect what it did between releases. The price is speed: a 4B model on a laptop is slow, and I've measured exactly how slow.
Prize Categories
- Best Use of Gemma: Gemma 3 4B, running locally through Ollama, writes every letter, and all the measurements above are Gemma runs.
Top comments (0)