This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
What I Built
ReplyDesk writes replies to Google reviews for small shops: garages, salons, dentists, cafes, plumbers, small stores.
Every shop owner I know has the same pile of unanswered reviews. Replying matters, because people read the replies before they pick a shop, but writing a good one takes ten minutes, and the replies a chat assistant writes all sound the same: "Thank you for taking the time to share your valuable feedback, we strive to..." Customers can tell.
So ReplyDesk does one thing. You get an inbox of reviews. Click one and you get:
- a draft reply written in an owner's voice by a small open model I fine-tuned on real owner replies
- how you, and owners like you, answered similar reviews before
- a check that flags any promise in the draft that your shop facts don't back up ("we'll refund you", "we'll call you tomorrow")
You edit it, copy it, and post it yourself in Google Business Profile. It never posts on its own.
Who pays and why it could work as a side hustle: one reply costs about $0.0001 on the fine-tuned model. A shop with 60 reviews a month costs less than a cent to serve, so even $3 to $5 a month per shop would be nearly all margin. That only works because the model is small and open. On a large closed model the per-reply cost is many times higher, and the price is someone else's to change.
The fair question: isn't this just pasting a review into ChatGPT? Partly, yes. A chat assistant with a good prompt writes a perfectly fine reply, and in my tests people often liked it more (see the results below). What ReplyDesk adds around the model is the shop context: your facts, your past replies, and a check for promises you can't keep. I don't think the model alone is worth paying for. The workflow around it might be, and I haven't proved that.
Demo
Live: https://replydesk-shop.vercel.app (the demo shop right now is my first user's cafe)
Pages:
- Inbox: reviews, the draft, the promise check, similar replies
- Setup: shop type, facts, past replies
- Is it better?: the full eval, read live from the database
- Blind test: two replies to one review, you don't know which model wrote which, pick the one you'd send
Code
Sarcastic-Soul
/
replydesk
Google review replies for small shops, written by a Qwen3.5-4B fine-tuned on real owner replies
ReplyDesk
Short, warm replies to Google reviews, written in the shop owner's voice by a small open model (Qwen3.5-4B) fine-tuned on real owner replies. Built for the Hacktoberfest 2026 DEV challenge ("Side Hustle").
Live demo: https://replydesk-shop.vercel.app
Small shops get reviews every week and most never answer them, because writing a good reply takes time and the replies that chat assistants write sound like a bank. ReplyDesk gives the owner an inbox of reviews, a draft reply for each one, the shop facts that matter for that review, and how they (and owners like them) answered similar reviews before. The owner edits, copies and posts it. Nothing is posted for them.
What's inside
flowchart LR
B[Browser] -- review text --> E[bge-small in a Web Worker]
B -- server action --> S[Next.js on Vercel]
S -- draft --> T[Tinker: fine-tuned Qwen3.5-4B]
S -- facts for this review --> M[Backboard memory]
S --…How I Built It
The data
The Google Local Reviews dataset from the McAuley lab at UCSD has reviews together with the owner's reply. I took four states (Wyoming, Vermont, Montana, Idaho), which gave me 590,054 review and reply pairs.
Most owner replies are not worth learning from. Lots of them are the same pasted text under every review, some are machine-translated, some include phone numbers, and some are from chains with a social media team. My filter (build_dataset.py) kept a pair only if the reply was in English, came within 60 days, had no contact details, was from a non-chain business in one of six shop types, wasn't a template, and actually shared a word with the review. That left 50,954 pairs.
I split train and test by business, so the test reviews come from shops the model never saw.
Fine-tuning on Tinker (three tries)
Model: Qwen3.5-4B, LoRA rank 32, using the tinker-cookbook supervised recipe, loss on the reply only, thinking turned off.
v1 trained on everything that passed the filter. It learned to write like real owners, including the bad ones. One test reply opened with "Cameron, you are obviously confused. We did not see you in the store." Real owners write that. My users shouldn't.
v2 dropped replies that argue, call the review false, or are pure stock phrases. Now it was polite but bland: a lot of "Thanks for the review, see you next time" under reviews that described a specific problem.
v3 kept only replies of 20+ words that talk about at least two things the reviewer said, and for 1 to 3 star reviews, own the problem in the opening lines. 6,000 training pairs, one epoch, 1.28M tokens, about $0.95. All three runs together cost about $4 of the $10 Tinker credit.
The app calls the fine-tuned model through Tinker's OpenAI-compatible endpoint. One gotcha worth sharing: on Qwen3.5 you turn thinking off with extra_body={"reasoning_effort": False} (a boolean). The strings "none" and "low" are rejected, and if you leave thinking on, a small max_tokens gets used up by the thinking and you get an empty reply.
Did it work? The eval
I wrote replies to 300 held-out reviews (half of them 1 to 3 stars) with four writers: the same 4B with a careful prompt, my fine-tuned 4B, Qwen3.8-27B with the same prompt, and the real owner's reply. A judge from a different model family (gpt-oss-120b, also on Tinker) saw all four in random order with no labels.
| 4B prompted | ReplyDesk (4B tuned) | 27B prompted | Real owner | |
|---|---|---|---|---|
| Median words | 60.5 | 34 | 56 | 34 |
| Stock phrases | 19.3% | 2.0% | 9.7% | 7.3% |
| Specific promises | 18.0% | 11.3% | 20.3% | 12.7% |
| Mentions a detail | 98.3% | 56.3% | 98.3% | 55.3% |
| Judge's first pick | 40.3% | 5.0% | 44.0% | 10.7% |
| Cost per reply | $0.00013 | $0.00010 |
Here is the honest reading.
Fine-tuning did exactly what it was trained to do. The replies moved from where the prompted model sits to where real owners sit: the same length as real replies, far fewer stock phrases than even real owners, and fewer made-up promises (the thing that gets a shop in trouble).
The judge hates it. It also ranks real owners' replies near the bottom. The prompted models write longer replies that repeat a detail back, and LLM judges reward that. So I suspected the judge was measuring polish, not "would a shop owner post this". I didn't bend the eval until my model won, so the table stays as it came out.
That is why the app has a blind test page: two replies to one review, one from each 4B model, sides shuffled, pick the one you'd send. Votes go into the database and show on the "Is it better?" page. I gave it to a real shop owner. The result is in the next section, and it did not go my way.
Tiger Data: one query for "how did we answer this before?"
The Tiger Cloud free service runs Postgres 18 with pg_textsearch (BM25) and vectorscale (DiskANN). It holds the shop, reviews, sent replies, the eval results, the blind-test votes, and 11.5k example replies from the training businesses.
When you open a review, one SQL statement takes the top 20 by BM25 and the top 20 by vector distance within your shop type, and merges them with reciprocal rank fusion. Your own past replies come first. BM25 catches the exact words ("brake pads", "root canal"), vectors catch the meaning ("they kept me waiting" vs "took forever").
I ran out of Hugging Face inference credit halfway through, so the query embedding runs in your browser (@huggingface/transformers with bge-small-en-v1.5 in a Web Worker). Until it has loaded, the search runs on BM25 alone. No embedding server at all.
These examples are shown next to the draft. They are not put in the prompt. The model was trained on the review alone, so the app sends exactly the training prompt.
Backboard: memory for shop facts
Each shop gets a Backboard assistant, and every shop fact is stored as a memory. When you open a review, memory search returns only the facts that match it. For a garage, a review about a car that took too long pulls "Same-day WhatsApp update when a job is delayed", not "Ask for Ramesh at the front desk". If Backboard is down, the app falls back to all facts.
The promise check
Groq (gpt-oss-20b) lists every promise in the draft and checks it against the shop facts. Anything unbacked flips the stamp to "Check" and shows the line. A keyword check takes over if Groq fails.
Stack
Next.js 16, React 19, Tailwind 4, drizzle-orm, Phosphor icons, bun, deployed on Vercel. Python with uv and polars for the data and training side.
Why Does Open Innovation Matter?
- I own the model. The weights are mine. If Tinker disappeared tomorrow, I could run a 4B model on a laptop or a cheap GPU. A shop's reply tool shouldn't break because an API changed.
- The price is mine to set. At $0.0001 a reply a $3 subscription works. On a big closed model the margin shrinks to nothing and depends on someone else's price list.
- I could change what the model does, not just what I ask it. Fine-tuning moved the replies to the owner-like length and cut the stock phrases and made-up promises. That is something a prompt on a closed API couldn't do for me, even though it didn't make people like the replies more.
- Open data made it possible. None of this works without researchers publishing half a million real owner replies.
My First User
My first user is Pratik, who runs HOP-In Cafe near IIIT Naya Raipur. He set up his shop in ReplyDesk (name, area, three facts about the cafe), pasted in a real review from his Google page, and sent a reply. Then he did the blind test: 20 rounds, and 19 votes were saved.
Result of the 19 saved votes:
| Pick | Votes |
|---|---|
| Prompted 4B | 15 |
| ReplyDesk (tuned 4B) | 4 |
| About the same | 0 |
So the human test agrees with the judge model. The tuned model sounds more like a real owner, but a real owner still picked the longer, more detailed prompted reply about four times out of five. It is one person and 19 votes, so I wouldn't call it a general result. It does tell me that my main bet, that matching owner style beats a good prompt, did not hold up in this test.
There is no billing in the app, so nobody paid. I'm not going to claim a price I haven't tested.
What I'd Do Next
- Test the prompted model plus the shop facts and promise check against the tuned one, since that is probably the real product.
- Pull reviews in automatically instead of copy and paste. Posting to Google needs Google's Business Profile API, which I haven't tried.
- Train a version that keeps the "mention what the reviewer said" habit that people seem to like.
Prize Categories
- Best Use of Tinker: fine-tuned Qwen3.5-4B (three runs, about $4), serving it through Tinker's OpenAI-compatible endpoint, and running the whole eval on Tinker (900 generated replies and 300 judge calls). The gains I can show against the baselines are cost, length and fewer stock phrases and promises. The preference tests did not favor the tuned model, and the post says so.
-
Best Use of Tiger Data: hybrid BM25 + DiskANN search with RRF in one SQL statement on
pg_textsearchandvectorscale, plus all app data, eval results and blind-test votes. - Best Use of Backboard: per-shop memory for shop facts, with memory search choosing the facts for each review.
Dataset credit: Google Local Reviews, McAuley lab, UCSD (Li, Shang, McAuley, UCTopic, ACL 2022; Yan et al., Personalized Showcases, SIGIR 2023).
Top comments (0)