Part of gharasathi, a private household AI. Start here for all the series.
In the original series, a 3B model on a 6 GB mini PC could route a question to the right tool and turn its result into a sentence. That was about all. It couldn't analyse a year of household spending. This part covers what changed when the model got 40 times bigger, and what didn't.
The setup
| Machine | Mac Studio, M3 Ultra, 96 GB unified memory |
| Server |
llama-server from llama.cpp, installed with Nix, run by a launchd agent |
| Model |
Qwen3.5-122B-A10B, GGUF, 4-bit (UD-IQ4_NL) |
| Memory in use | about 62 GB |
| Context | 128k tokens, 2 parallel slots |
| API | OpenAI-compatible on :8092; services call it as ChatOpenAI
|
llama-server \
--model Qwen3.5-122B-A10B-UD-IQ4_NL-00001-of-00003.gguf \
--alias qwen3.5-122b --port 8092 \
--ctx-size 131072 --parallel 2 \
--n-gpu-layers 99 --flash-attn on --jinja \
--chat-template-kwargs '{"enable_thinking":false}' \
--temp 0.7 --top-p 0.8 --top-k 20 --presence-penalty 1.5
Why this model. It's a mixture of experts: 122B parameters in total, but only about 10B are active for each token. All the experts must sit in memory, which unified memory handles well, while each token costs roughly what a 10B dense model would. The quality is that of a big model at a speed that's fine for an overnight job and acceptable for chat. A typical dashboard follow-up question with tool calls takes about 25 seconds.
Why llama.cpp rather than Ollama. I started on Ollama. I moved to llama-server for direct control over context size, parallel slots, the chat template and the thinking switch. Nix pins the build, and launchd restarts it if it dies.
Thinking on and off. Qwen3.5 "thinks" by default. The server turns that off, and each caller turns it back on per request where it's worth it: the overnight analyst thinks, while chat and routing don't. Thinking also has no natural stop on hard prompts (I once saw 35k tokens on a single call), so thinking requests are capped.
Two ways the model is used
1. anna (chat, and the dashboard's Explain button). This is a LangGraph agent with about a dozen finance tools: spending analysis, insights, goals, a mortgage what-if calculator, bills and statements, and a read-only Cypher query. The rule in its prompt is blunt: always call a tool before answering; never do arithmetic in your head; use the calculate tool. On the dashboard you can click a fixed question, such as "Explain this and what to do", and then type follow-ups. The follow-ups carry the conversation, so "the loan is fixed, does that change it?" gets an answer about that finding.
2. The overnight analyst. This is where the bigger model earns its place.
- Fact packs. Code builds one pack per topic: mortgage and offset, childcare, utilities, groceries, dining, insurance and so on. A pack holds monthly totals, merchants, recurring charges, balances and, now, facts read from bills and statements. Every number in a pack is computed, never estimated.
- Overview, then topics. The model first names the household's 3 to 6 themes. Then it writes up to 3 findings per topic. Before writing, it may ask for extra data from a fixed list of requests, such as "transactions for merchant X" or "this category by month".
- Fact-check (code). Every dollar amount and percentage in a finding must appear in the pack, or follow from it with one step of arithmetic. If not, the finding gets one repair round, then it is rejected.
- Judge, synthesis, editor. A second pass asks whether the advice actually follows from the facts. Then the model connects findings across topics, ranks them against our past feedback ("not useful", "already handled"), merges duplicates and caps the number of new findings each night.
The fact-checker is not decoration. From last night's log:
PACK childcare: [
{'title': 'Recurring cost from exceeding 42-day absence allowance',
'outcome': "rejected by fact-check: 156.0 in saving_calculation isn't from the data"},
{'title': 'CCS rate dropped …, adding ~$…/year',
'outcome': 'withdrawn by analyst after fact-check'}, ...]
Both rejections were plausible-sounding numbers that the model had worked out rather than read from the data.
Where a big model still goes wrong: confident advice without context
The analyst once told me our mortgage rate was a fraction of a percent "above a previously negotiable rate" and that I should call the bank. Every figure in that finding passed the fact-check, but the advice was useless:
- The loan is fixed, so repricing it means break costs.
- Our loan-to-value ratio means moving to another lender would mean paying Lenders Mortgage Insurance again, which wipes out years of a small rate saving.
- The "negotiable" rate came from an earlier finding, not from any rate in the data.
Two fixes:
- Domain rules in the pack. The mortgage pack now starts with what is realistic: fixed-rate break costs, LMI above 80% loan-to-value, never quote a rate that isn't in the data, and the real levers (the offset balance, a retention rate when the fixed term ends, and refinancing once the loan-to-value ratio drops below 80%).
- Household facts. Some things only we know, such as the loan-to-value ratio, so they go to the analyst and anna as a short "household says" note.
Now when I ask whether refinancing is realistic, anna says no, explains LMI and break costs, and points to the offset. The offset is where the data shows real money: in the months the offset held the most cash, interest per day fell by about a fifth.
What I'd tell someone trying this
- A bigger model makes the explanations better. It doesn't make the arithmetic safe. Keep the counting in code at any model size.
- Make it cite. Every finding stores its evidence: transaction ids, or the exact line quoted from a PDF. The fact-checker checks against that.
- Feed it the constraints you'd tell a human adviser. Most bad advice came from missing context, not from weak reasoning.
- Unified memory is the enabler. A 4-bit 122B mixture-of-experts model in about 62 GB leaves room for Immich, Nextcloud and a Kubernetes cluster on the same machine.
Next, part 3: bills and statements. They're the one place I deliberately didn't use the LLM, and they turned out to hold the most valuable findings.
Top comments (1)
Running a 122B model locally on a Mac Studio for finance sounds like a solid test of both the hardware and the model's reasoning. How's the token speed holding up?