I am delighted to announce that a user reported dysfunction so that I could go down the rabbit hole of fixing it. Messing around locally is one thi...
For further actions, you may consider blocking this person and/or reporting abuse
This was a great read. I hadn't really thought about the tradeoff between quantization, performance, and output reliability before. The benchmark comparison was especially helpful. Thanks for documenting the whole process!
Thanks! I may have to create a follow-up post at this point with all of the feedback and findings. This is a big rabbit hole for sure.
Thank you for sharing such an excellent post. I really enjoyed reading it.
I’m a Python Full-Stack Engineer with over 10 years of experience designing and building scalable software solutions for clients across a variety of industries. Along the way, I’ve learned that successful projects depend not only on strong technical execution but also on creating real business value.
With my recent contract completed, I’m exploring new opportunities to collaborate with professionals who value innovation, practical problem-solving, and long-term partnerships. I enjoy discussing ideas that combine technical excellence with sound business strategy, creating outcomes that benefit everyone involved.
I believe every connection has the potential to become something meaningful. If you're interested in exchanging ideas, exploring opportunities, or simply connecting with someone who enjoys building impactful technology, I'd be happy to hear from you.
Wishing you success in your future endeavors, and I look forward to connecting.
Great writeup, and the honesty about the 50-65% malformed-JSON rate is the useful part most benchmark posts leave out. One thing that beats the retry loop on the JSON failures specifically: constrained decoding. Ollama rides on llama.cpp, so you can pass a JSON grammar (GBNF) or use the format:json / schema option and the decoder physically cannot emit a token that breaks the structure. That takes the structural failure rate to basically zero instead of 85-95%, and it does it w/o paying the retry tax, which matters most exactly here because latency is your scarcest resource on the Nano. Retrying up to 3x means your worst case is 3x the 35-45s you fought so hard to win back.
The sharper issue hiding under the retry number: retry only catches JSON that fails to parse. It does nothing for the failure Q4 actually introduces, which is well-formed JSON with a wrong value in it. A 1b model at Q4 will happily hand you clean, parseable JSON with a hallucinated field, and the loop counts that as a success because it parsed. So "85-95% effective" is measuring "did it parse," not "is it right," and those two diverge fastest exactly at aggressive quant levels. If the values matter, that is where a Q5_K_M (still fits your 8GB for a 1b) or a cheap validation pass against the source earns its keep more than more retries. Structural validity and correctness are different tests, and quant pressure breaks the second one first while the first one still looks fine.
Yes I found this out quickly. I was reading that a smaller model like qwen could actually out perform a heavily squished version of llama3.2:b. Because quality degrades quickly. Im currently doing some other benchmarks and am considering making the switch. Thanks for your thoughtful response!
That instinct is right, and it is worth being precise about why it holds for your case specifically. Qwen2.5 at 0.5b or 1.5b is a genuinely strong instruction follower for its size, and for structured extraction the thing that degrades first under heavy quant is not general fluency, it is exactly the value-correctness you care about. A smaller model at Q5 or Q8 will often beat a bigger model crushed to Q4 on "did it put the right number in the right field," even when the bigger model wins on a general benchmark. Rank order flips by task, so the general benchmarks you are reading point you the right way but they will not settle it.
The only thing that settles it is your own schema. Build a small gold set once, say 30 to 50 real inputs with the correct JSON written by hand, then run every candidate (qwen-1.5b at Q5, llama-1b at Q4, llama-1b at Q5) against it with the JSON grammar turned on, so structural noise is out of the picture and you are measuring only the wrong-value rate, which is the number that actually decides this. That turns "I read that qwen might be better" into a number for your data. And it compounds: the gold set is a one-time cost, and every future model or quant choice after that is a cheap re-run against it instead of re-forming an opinion from someone else's benchmark. I keep a small public harness that does exactly this field-level scoring (doc-eval on my github) if it saves you the wiring.
Thanks again for your thoughtful feedback. So basically, feed it the general structure of what I want for the output, and then see how each model follows the rules accordingly?
Close, with one refinement that changes what you end up measuring. The structure part you do not test, you enforce: with the JSON grammar on, the model literally cannot produce anything except your shape, so "does it follow the output rules" stops being a question at all. What you test is whether the values inside that shape are correct, and that is why the gold set is the heart of the method: 30 to 50 real inputs where you wrote the correct output by hand, an answer key. Run each candidate model on those inputs, compare what it filled in against what you wrote, field by field, and the wrong-value rate per field is the score.
So the flow is: grammar guarantees the shape, the gold set defines the truth, and the eval just counts disagreements. A model can follow every rule and still put the wrong date in the right field, and that is exactly the failure you want counted, which is only possible because you knew the right answer ahead of time. Build the answer key from your real documents, the ugly ones included, and the whole model-choice question turns into reading one number per candidate.
I see, that makes more sense now! I am genuinely curious to try this method. 🤔 this raises more questions actually.
Interesting post! I opted for slow, highly accurate models for my personal offline apps. Seeing you use a faster model combined with retry logic for a user-facing app is a great alternative approach. Nice work.
I must note that you need to keep a very watchful eye on the quality of your output. Depending on your use case, the results may not be great. Just some tips from actual experience here. Haha XD Depending on what you are doing, it may not matter.
Retry logic > perfect JSON. 25x faster Q4 with 3 retries = 85-95% success. Ship it.
By reducing the precision of the model (quantizing/compressing) We lose the accuracy needed to output test results that make sense. For example, with Q4, and running three tests, we may only get some test questions with answers that make sense.
The difference, simply put, might output something comparable to the following if given the same information to read from:
Q4: Chickens are primordial retirees
Q8: Chicken soup has been around for a long time.
Precision matters. It's better to not squish, if possible, in my opinion. I know this to be true because after I switched to Q4, my flippycard app started generating absurdities, rendering it useless. I very much wanted to believe your statement, and I agree with your logic, but not if the output is important. Haha