This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
TL;DR. Small hotels in Pakistan take bookings on WhatsApp, in Roman Urdu, at midnight. Munshi reads each message with a Qwen3.5-4B I fine-tuned on Tinker, prices the advance by how likely the guest is to cancel (TabPFN), drafts the reply with Gemma 4, and holds the room in a Temporal workflow until the advance arrives or the clock runs out. The fine-tune alone got 75% of booking messages exactly right. Moving the calendar arithmetic and the counting into plain Python took it to 30 of 30 on a test set I wrote after the rules were frozen, a tie with its teacher Gemma 4 (29 of 30) at an eighth of the prompt. Models read. Code decides.
What I Built
If you've booked a guest house in Murree or Naran, you've probably done it like this:
"AoA, kal 2 log ke liye 1 room chahiye 2 raat ke liye. rate kya hai? advance kitni?"
That one line is a booking: tomorrow ("kal"), two people, one room, two nights, and two questions. On the other side of it is an owner running a twelve-room hotel from a phone. The message arrives at 11:40 pm. The rate gets quoted from memory, the advance is "jo munasib ho" (whatever suits you), and nobody chases it. Booking.com fills rooms, but it doesn't answer that message.
The villain here is the promise with no deadline. Not the price. Not the competition. A quote nobody holds and an advance nobody chases.
So I built the clerk who holds the clock.
Munshi (the man who keeps the register at a small hotel's front desk) does four jobs for every message:
- It reads the message. A 4B model fine-tuned on Tinker turns Roman Urdu, English, a mix of both, or Urdu script into strict JSON: check-in date, nights, guests, rooms, room type, budget. If the date or the head count is missing, it asks, then waits for the answer.
- It prices the advance by risk. TabPFN scores how likely the booking is to cancel. Low risk pays 20% and the room is held for 12 hours. High risk pays 50% and the hold is 2 hours.
- It drafts the reply. Gemma 4 writes it in the guest's language, from the parsed facts only, and code checks every number before the owner sees it.
- It holds the room. A Temporal workflow waits for the advance. Partial JazzCash transfers add up. If the money doesn't come, the room is released and the guest gets a polite note.
The product in one picture. Booking.com fills the room. Munshi makes sure the room it held was worth holding.
Demo
57 seconds, no narration: the 11:40 pm message, the fine-tune reading it, the risk-priced advance, Gemma's reply with every number checked, the hold with two partial deposits, and the server kill. The file is also in the repo: docs/demo.mp4.
This is a real run, with the fine-tuned parser, TabPFN-3.5 and Gemma 4 all live:
The owner's desk, not a chatbot. The text is Gemma's; every number on the screen is Python's.
Code
ArqamWaheed
/
munshi
Munshi: a WhatsApp front desk for small Pakistani hotels (HF26 Week 2)
Munshi: a WhatsApp front desk for small hotels
One inquiry: WhatsApp text → fine-tuned Qwen3.5-4B on Tinker parses it to strict JSON → TabPFN-3.5 scores
P(cancel/no-show) → (optional) SerpApi Google Hotels nearby nightly rates set the price, otherwise your base rate → Gemma 4 drafts the reply
from parsed fields only → a Temporal RoomHoldWorkflow holds the room for a time that depends on the risk score
waits for a deposit_received signal, then confirms or releases. A Temporal Schedule writes the owner's morning
brief. Sentry traces every model and tool call.
Rule of the codebase: models read and write, code decides. Dates, counts, the advance, the hold time and every
number in a reply are computed or checked in plain Python (munshi/dates.py, parser.apply_rules, policy.py
reply.check_draft). Demo (57 s, no narration): docs/demo.mp4. Brand kit: DESIGN.md.
Results (every number is a file in results/ or
…MIT licensed. Every number below is in a file under results/ or evidence/, and the scripts in eval/ regenerate them.
Interesting files:
-
munshi/dates.py: the calendar the 4B model kept getting wrong, as 135 lines of rules. -
munshi/temporal/workflows.py: the room hold. One workflow per inquiry, signals for deposits and guest replies, a durable timer whose length comes from the risk score. -
munshi/reply.py:check_draft(), which stops a reply from quoting a number nobody computed. -
eval/temporal_drills.py: kills the worker, then the whole Temporal server, mid-hold, and prints the timeline from Temporal's own event history.
temporal server start-dev --db-filename munshi.db # terminal 1
python -m munshi.temporal.worker # terminal 2
uvicorn app.server:app --port 8000 # terminal 3 -> http://localhost:8000
python -m eval.temporal_drills # the durability numbers below
How I Built It
The rule behind all of it: models read and write, code decides.
Three models, three red boxes. Every red box is plain Python that can overrule the model before it.
Tinker: teaching a 4B model to read a booking message
Why fine-tune, not prompt. Untuned Qwen3.5-4B, given a one-line instruction and "parso 5 bande family room 3 din", invented its own schema and read "parso" (the day after tomorrow) as a hotel name. Gemma 4 reads it fine, but only with a 532-token rulebook in every request. The fine-tune gets a 68-token prompt and answers with the right date, 5 guests, 2 rooms and a family room.
The teacher writes the homework, code marks it. Gemma 4 (free tier) wrote 1,000 synthetic WhatsApp inquiries and labelled every one with the long rulebook. A check of 30 random labels against the rulebook found 7 with a wrong booking field, and 6 of those were the same habit (four guests became two rooms). Code fixed that before training, along with two bigger problems I'll come back to. 678 rows survived.
The fine-tune: LoRA rank 32 on Qwen3.5-4B, 3 epochs, batch 16, 126 steps. Loss only on the assistant's JSON, chat template rendered with thinking off, identically for training and serving. Dev loss went from 1.00 to 0.048. Tinker billed 387,659 training tokens per run, $0.29 a run, and I ran it twice. Everything this project did on Tinker (both training runs, every eval, the checkpoint storage) came to $0.66 on tinker billing usage.
The test: 80 messages I labelled by hand, none of them in the training data: 61 Roman Urdu or mixed, 4 in Urdu script, 15 that need a clarifying question, and 3 raw WhatsApp exports with names and phone numbers in them.
The first result was not the one I wanted.
| Parser, first run (no code rules) | Booking fields exactly right (95% CI) | Prompt | $ per 1k messages | p50 latency |
|---|---|---|---|---|
| Gemma 4 + 532-token rulebook (the teacher) | 87.5% (80.0–93.8) | 532 tok | $0.075 list ($0 on free tier) | 3.1 s |
| Qwen3.5-4B, untuned, short prompt | 0.0% | 68 tok | n/a | n/a |
| Qwen3.5-4B, untuned + the rulebook | 68.8% (58.8–78.8) | 532 tok | $0.27 | 2.4 s |
| Qwen3.5-4B + Munshi LoRA | 75.0% (63.7–82.5) | 68 tok | $0.084 | 1.0 s |
The student lost to its teacher. 11 messages only Gemma got right, 1 only the fine-tune got right, p = 0.006.
It did not lose at reading. I went through every miss. The fine-tune found the dates, the people and the rooms in the text. It got the arithmetic wrong: which Saturday "hafte ko" means, how many nights "itwaar se peer tak" is, whether "chhe log" is six. A 4B model doing calendar maths in its head is the wrong job, not a bad model.
So the calendar moved into code. munshi/dates.py applies the rulebook: aaj, kal, parso (in Roman Urdu, English and Urdu script), weekday names, "is weekend", "agle hafte", bare day numbers, explicit dates and ranges. It only answers when the message is unambiguous. "Next friday", "aaj kal" ("these days") and a weekday said on that same weekday all go back to the model. I built it against the training set: it agrees with the teacher on 542 dates, abstains on 451, and the 7 disagreements are all the teacher getting the rulebook wrong ("25 ko" on 28 October means 25 November).
Then the counting. A head count or a room count written once in the message ("chhe log", "do kamray", "a double room") is read by code, in three scripts.
| Booking fields exactly right | Run 1: no rules | Run 2: + dates | Run 3: + counts |
|---|---|---|---|
| Qwen3.5-4B, untuned + the rulebook | 68.8% | 81.2% | 87.5% |
| Qwen3.5-4B + Munshi LoRA | 75.0% | 86.2% | 95.0% |
| Gemma 4 + the rulebook | 87.5% | n/a | 98.8% |
Code helped the teacher too: Gemma went from 87.5% to 98.8%. On this set it still beats the fine-tune, 3 messages to 0 (p = 0.25, not significant). Gemma has no run-2 number because its first run didn't store raw outputs, so it got one new run with all the rules.
Here is the honest part. I found the counting rules by reading the fine-tune's misses on this test set. That makes run 3 a tuned number, not a held-out one. So I froze the rules, wrote 30 new messages with labels from the rulebook, committed them to the repo, and only then ran the models, once:
| Fresh set, run once after the rules froze | Booking fields exactly right | All fields | Roman Urdu |
|---|---|---|---|
| Qwen3.5-4B, untuned + the rulebook | 27 / 30 | 43.3% | 87.5% |
| Qwen3.5-4B + Munshi LoRA | 30 / 30 | 90.0% | 100% |
| Gemma 4 + the rulebook | 29 / 30 | 86.7% | 95.8% |
30 of 30 sounds better than it is. With 30 messages, the exact 95% lower bound is 88%. On all fields the fine-tune beats the rulebook-prompted base model 16 messages to 2 (p = 0.001). On the booking fields alone it's 3 to 0, which isn't significant. Against its teacher it's 30 to 29: Gemma got one room type wrong. That's a tie, not a win (p = 1). A 4B model with a 68-token prompt now reads these messages as well as Gemma 4 26B with a 532-token rulebook, and the guest's words never leave for Google.
Every red cell is the same model with code doing the arithmetic.
The fine-tune learned to read. Code learned to count. Neither was enough alone.
Read the tables with these in mind:
- I wrote every test label, with an AI coding assistant, checked against the rulebook. Re-deriving them caught three of my own mistakes (e03, m13, e15: "a family room" and "a double room" are one room). Real messages from a real hotel would be a better test.
- "Booking fields" leaves out one field: language. The line between "Roman Urdu" and "mixed" was never in the teacher's rulebook and only picks the reply's style. I chose this before running any model. The all-fields number is in every table.
- Latency through Tinker's hosted sampler moves around. The same fine-tune measured 1.0 s p50 in one run and 5.9 s in another twenty minutes later. I'm not claiming a speed win. The fixed wins are the 68-token prompt, a third of the cost of the prompted base model, and that the raw message goes to a model I own.
- Budget is still the model's call. "8 hazar mein 2 din" is a total; "budget 6 hazar" is per night. I tried a rule. It got the first kind wrong on the training data, so I dropped it.
- Urdu script is thin. One training row and 4 test messages. Code reads the dates and counts in Urdu script, but the model has barely seen it. More Urdu-script data is the first thing I'd add.
Gemma 4: writes every reply, decides no number
Gemma sees facts, not the guest. The raw message never goes to Google. Gemma gets a small JSON of parsed fields and the quote, and writes one short reply in the guest's language.
Then code reads the reply. check_draft() rejects a draft if it contains any number that isn't in the facts (Latin or Urdu digits, "16,000", "3.5k", "8 hazar"), if a quote leaves out the total, the advance or the hold time, or if it runs past 90 words. A rejected draft gets one retry with the complaint as a hint. If that fails too, the owner gets a template that can't invent anything.
Over 70 quote drafts, it never fired. 70 of 70 passed first time, in Roman Urdu, mixed, English and Urdu script. Gemma, given only the facts, never invented a price. The checker is insurance, not a fix.
It also showed me what the checker can't see. Every single draft says the room "available hai". Munshi doesn't know the hotel's inventory. That's not a number, so check_draft() lets it through. The owner's screen now says, next to the copy button, "Munshi cannot see your room inventory: check the room is free before sending."
Asked, not guessed. The workflow waited for the guest's answer, then re-read the whole conversation.
TabPFN: is the risk score honest, and is it worth money?
A small hotel has a small register. I gave each model 300 past bookings, about a year for a 12-room hotel, and tested on 2,000 it had never seen (public hotel-bookings dataset, CC BY 4.0). Ten different 300-row samples:
| Model (300 training rows) | ROC-AUC mean ± std | Seeds won | Brier |
|---|---|---|---|
| TabPFN-3.5 | 0.830 ± 0.007 | 10 / 10 | 0.148 |
| Logistic regression | 0.806 ± 0.007 | 0 / 10 | 0.157 |
| Gradient boosting | 0.789 ± 0.018 | 0 / 10 |
A win, not a thrashing. Then the question a hold policy actually asks: when TabPFN says "high risk", is it?
Calibrated, not just accurate. The bands are where the money is.
The advance is the product. When a guest cancels, the hotel keeps the advance. Charging everyone 30% keeps 30% of the value of cancelled bookings and asks 30% of every honest guest. Pricing the advance by TabPFN's band keeps 37.7% and asks only 26.9%. More from the no-shows, less from everyone else.
The concession: logistic regression's bands do almost exactly as well on that policy (37.8% and 27.3%). TabPFN wins on ranking and calibration. The tiers are what pay. And the data is Portuguese hotels from 2015 to 2017, not Murree.
Temporal: a room hold is a promise with a deadline
One workflow per inquiry. Parse, score, draft, then wait for the advance with a timeout set by the risk band. Deposits and guest replies are signals, so a guest can pay in two JazzCash transfers and the hold confirms only when they add up to the advance.
Partial, not confirmed. A short payment never confirms a room, and a released hold shows the refund owed.
I didn't trust it until I tried to break it. eval/temporal_drills.py starts its own Temporal server and runs three drills with the real models:
| Drill | What I did | What happened |
|---|---|---|
| 1 | SIGKILLed the worker mid-hold, then sent the deposit while no worker was running | Signal recorded at once; a fresh worker confirmed the booking. 6 activities scheduled, 6 completed, 0 run twice |
| 2 | SIGKILLed the worker and the Temporal server, and kept everything down 15 s past the hold deadline | The server came back at t = 127.3 s and fired the timer at 127.4 s: late by exactly the outage, not lost. Released, 0 run twice |
| 3 | Chaos: 50% of first attempts fail on purpose, 5 holds | 5 / 5 completed; 11 of 25 activities failed once and succeeded on retry |
The orange bar is the promise. Nothing was running for most of it, and it still ended on time.
The drill caught a real bug on its first run. The first time I killed a worker, the guest's quote took 107 seconds. Temporal's history showed why: score_risk was cancelled at its 90-second timeout. tabpfn_client's fit() is lazy, so the first predict_proba() after a worker starts does all the work, and it sat in the API for over a minute and a half. Now the worker makes one throwaway prediction before it takes any work, and a guest-facing TabPFN call gets two 30-second tries before falling back to logistic regression. Same drill afterwards: 1.5 seconds.
Sentry: where the time goes, and where code overruled the model
One inquiry is one trace, carried through Temporal by hand (there's no Sentry integration for Temporal, so every activity's input carries the trace headers and the worker resumes them):
The parser's chat span carries 121 tokens and $0.00008. Gemma's carries 286 tokens and $0 on the free tier. reply_check is the 0.08 ms where code reads the reply before the owner does.
The first real batch found a bug. I sent three WhatsApp messages at once. Sentry showed the parser's chat spans at 7 to 22 seconds, against about one second in the eval, plus two short chat spans with no tokens at all. The worker log said why: three parse_inquiry activities on the worker's thread pool all did the first from transformers import AutoTokenizer at the same moment. That import isn't thread-safe, so two of them failed and Temporal retried them. The worker now loads the tokenizer once before it takes work. The same three messages at once: 2.7, 2.7 and 7.1 seconds, no failed attempts.
And it showed what my error reporting missed. Those ImportErrors never became Sentry issues, because my activity wrapper only reported the failures I expected (rate limits, bad model output, provider errors). Anything else now gets reported before Temporal retries it. The one issue Sentry did catch, MUNSHI-1, was mine too: asking for a hold that doesn't exist was an unhandled 500. It's a 404 now.
Sentry never sees the guest. Spans carry model names, token counts, cost, timings and the checker's verdict. Never the message, the reply or a phone number. tests/test_tracing.py runs a real activity through a capturing transport and fails if the reply text shows up anywhere in what would be sent. Writing that test taught me something too: sentry-sdk 2.72 ships gen_ai spans as separate span items, not inside the transaction, so my first version found nothing.
Things I refused to do:
- Do not let a model choose a price, an advance or a hold length.
- Do not send the guest's raw message to a free-tier API. The fine-tune is the only model that sees it, after names, phone numbers and emails are stripped.
- Do not fall back to Gemma when the parser fails. The owner replies by hand instead.
- Do not change a test label because a model disagreed with it. Change it only if it broke a rule written down before the run, and say so.
What went wrong
All real, all fixed, most with a test or an evidence file in the repo.
- My fine-tune lost to its teacher, 75% to 87.5%. It could read; it couldn't count. The fix was code, not more training.
- TabPFN's first prediction blew the activity timeout. The 107-second quote above. The durability drill found it, not me.
- I asked Gemma for "about 15%" incomplete messages and got about 60%. That would have taught the parser to answer "I don't know". Each generated batch is now told explicitly whether its messages are complete.
- The teacher asked a question whenever the number of nights was missing. 136 of the first 650 training rows. The rulebook says ask only when the date or the head count is missing, so the rule moved into code.
- My own rule read "19 ko room" as nineteen rooms. Caught by a test that checks every hand label is unchanged by the rules, before it reached production.
-
My drill made Temporal look slow. After a full restart the server took 0.5 s to accept connections, but my readiness check called
asyncio.run()inside a running event loop, swallowed the error, and waited out its whole retry budget. The first timeline said 60 seconds. - My drills wrote fake bookings into the hotel's register. The next morning brief listed five arrivals who didn't exist.
- Three messages at once broke the parser's first attempt. A thread-unsafe lazy import, found in the first real Sentry traces. The worker now loads the tokenizer before it takes work.
- Pydantic called the untuned model's invented schema valid JSON. Every field came out empty, and the eval counted it as a valid answer.
Why Does Open Innovation Matter?
The guest's message is the hotel's customer data. Names, phone numbers, family size, travel dates. With an open 4B model fine-tuned for the job, the raw text goes to a model the owner controls. Gemma's free tier, where prompts may be used to improve Google's products, only ever sees parsed fields.
The weights are mine. The adapter is a 278 MB rank-32 LoRA on Qwen3.5-4B, kept in my Tinker account with no expiry. tinker checkpoint download exports it, and tinker checkpoint push-hf puts it on the Hugging Face Hub as a PEFT adapter. Tonight the export timed out three times while Tinker built the archive, so for now it lives there. Once it's out, it runs anywhere that serves Qwen3.5-4B.
Small and open is cheap enough for a 12-room hotel. The fine-tune costs $0.084 per 1,000 messages at Tinker's list price, a third of the untuned model carrying the rulebook ($0.27). Gemma 4 on a paid API would be cheaper still ($0.075), so cost alone isn't the argument. Keeping the guest's words off someone else's free tier is.
I could fix the model's habits, not just prompt around them. The teacher's bad habits were corrected in the data before training, and the arithmetic moved into code the owner can read. A closed API would leave me writing a longer prompt and paying for it on every message.
Where closed would win, honestly: a frontier API would probably read these messages with no training at all, and with code doing the counting the gap would be small. A hotel with no developer also can't run a Temporal server yet.
My First User
I don't have one yet, and I'm not going to invent one.
I haven't shown Munshi to a hotel owner. This weekend went into building it and measuring it, so every number here comes from messages I wrote, and the hotel-bookings data under TabPFN is Portuguese, not Pakistani. Whether an owner in Murree would trust a model with the advance, or pay for it, is still the open question. Finding out is the next step, and I'd rather report that honestly than fill this section in.
What I Learned
- Put the rule in code, not in the prompt. Every serious bug in this build was a model doing something reasonable the business couldn't accept: asking about nights, dividing a nightly budget, counting the wrong Saturday. Each fix was a few lines of Python with a test.
- A small model should read, not reckon. The fine-tune went from 75% to 95% without a single extra training step.
- The advance is the product, not the model. TabPFN is the better model. The risk-tiered advance is what makes the money.
- Kill it before you trust it. The durability drill was supposed to produce a screenshot. It found the slowest bug in the project.
- Write the test set after you stop changing the rules. I didn't, the first time, and the post says so.
Prize Categories
- Best Use of Tinker: LoRA on Qwen3.5-4B (rank 32, 678 rows, 126 steps), distilled from Gemma 4. Untuned: 0% (invents its own schema). Fine-tuned: 75% booking-exact on 80 hand-labelled messages, 95% with code doing the dates and counts, and 30/30 on a fresh set written after the rules froze, against 27/30 for the untuned model carrying a 532-token rulebook. 68-token prompt, $0.084 per 1k messages.
- Best Use of Temporal: one durable workflow per inquiry: a risk-priced hold timer, deposit and guest-reply signals, release on timeout. Worker SIGKILLed mid-hold and the whole server SIGKILLed past the deadline: 0 activities run twice, and the timer fired 0.1 s after the server came back. The first drill caught a 107-second TabPFN cold start, now 1.5 s.
- Best Use of TabPFN: P(cancel) from a 300-booking register: AUC 0.830 vs 0.806 / 0.789, 10/10 seeds, calibrated per band (82% predicted, 81% actual). It prices the advance: keeps 37.7% of cancelled booking value while asking honest guests 26.9%.
-
Best Use of Sentry Agent Tracing: one trace per inquiry across Temporal activities:
invoke_agent→gen_ai.chat(model, tokens, cost) →execute_tool reply_check, where code overrules the model. No guest text in Sentry, enforced by a test. - Best Use of Gemma: Gemma 4 is the teacher for the fine-tune, the baseline it's measured against, and the writer of every reply, behind a checker it passed 70/70 times.
Top comments (0)