Most local-LLM posts are written on machines with a good GPU. Mine doesn't have one, so I'm sharing what actually happened.
The setup
- Laptop: Intel Core i7-13620H, 16 GB RAM, Intel UHD graphics only (no dedicated GPU), Windows
- Runtime: Ollama
- Models:
qwen2.5:7b,llama3.2:3b, andnomic-embed-textfor embeddings
I built two apps on this machine during Hacktoberfest.
App 1: an exam study buddy (RAG)
It answers questions and quizzes you from your own notes, using local embeddings plus Chroma/SQLite, with qwen2.5:7b generating.
| Request | Time |
|---|---|
| First answer after server start | 55 s |
| Normal answer | 42 s |
| Quiz question | 35 s |
Honest verdict: usable for studying, since you ask, think and wait. It's not usable for quick back-and-forth.
App 2: WhatsApp orders into a ledger
It turns messy order chats (English, Hinglish, Kanglish) into structured orders.
| Model | Time per order | Tuned 12-chat set | Fresh 6-chat set |
|---|---|---|---|
| qwen2.5:7b | ~17 s | 11/12 orders (99% fields) | 3/6 (93% fields) |
| llama3.2:3b | ~7.7 s | 1/12 orders (82% fields) | 3/6 (91% fields) |
Two things stood out:
- The 3B model was about 2x faster, and on fresh chats it was close to the 7B on field accuracy. On my tuned set it was much worse, so the tuned-set score flattered the 7B. Always keep a held-out set.
-
Both models missed
delivery_dateon same-day ("aaj") and Kannada ("nale") chats. Model size didn't fix this. Prompt wording did, partly.
What I'd change to make it usable
These are ideas I haven't all measured yet. [FILL: add your before/after numbers for the ones you try.]
- Stream tokens. Seeing the first words within seconds changes how a 40 s answer feels, even if the total time is the same.
-
Keep the model loaded. Ollama unloads idle models after a few minutes. The 55 s first answer probably includes loading the model, so set
OLLAMA_KEEP_ALIVEor warm it up at server start. - Shrink the prompt. On CPU, a long prompt costs real seconds. Retrieve fewer chunks and cut the instructions.
-
Cap the output. Limit
num_predictfor quiz questions and extraction, which have short answers. - Don't make people wait. For the ledger, process in the background and show orders as they finish.
- Match the model to the task. Use the 3B for easy extraction and the 7B only where it clearly wins. Only a held-out set can tell you which.
Takeaways
- No GPU is workable if the task is async or patient (studying, batch extraction), and painful if it's chat-like.
- Model size matters less than you'd expect for narrow extraction, and more than you'd expect for tuning-set overfitting.
- Report timings and held-out accuracy together. One without the other misleads.
Code: study-buddy · Order-Ledger
Top comments (1)
On a modern laptop without a discrete GPU, generation latency is almost pure memory bandwidth physics. With dual-channel DDR5 running around 60 to 70 GB/s, reading a 4.5 GB quantized 7B model once per generated token caps theoretical throughput near 12 tokens per second before touching prompt evaluation. That explains why the 3B model delivered an exact 2x speedup.
The held-out set evaluation is the real takeaway here. When prompts get iterated against a dozen sample chats, the larger model easily overfits to phrasing idiosyncrasies. Once you test against fresh inputs, the field accuracy gap shrunk to two percentage points (91 percent versus 93 percent). For structured extraction, that tiny delta rarely justifies paying more than double the latency on every transaction.
Relative date parsing on words like 'aaj' or 'nale' also highlights a structural limitation rather than a parameter deficiency. Without an explicit message timestamp injected into the system prompt, a model has no calendar anchor to resolve relative days. No amount of parameter scale can infer an external reference date that was never provided in the context.