I am delighted to announce that a user reported dysfunction so that I could go down the rabbit hole of fixing it. Messing around locally is one thing, but building a tolerable app for end users has other considerations when using a ‘local’ AI. This led to interesting findings about the limitations of hardware and gaining a high-level understanding of quantization and it's importance. I also explain how to creatively get around limitations. It's called flippy card, and it's an app that helps you study via custom uploaded content. Wanna see it? It's here: Flippycard
Problems:
- Ollama was stuck behind cloudflare in a local loop.
- Ollama needs to be accessed outside of systemctl with config file.
- Ollama was running painfully slow, and needed to set it to use GPU, not CPU.
- Had to build Ollama from source and tell it to use GPU instead.
Implementations:
- Make sure Ollama goes through cloudflare tunnel.
- Create configuration file so that Ollama is reachable externally.
- Build from source to save resources, Docker is too resource intensive for Jetson Nano to be performant.
- Increase speed by switching from CPU -> GPU, you need to be explicit about these things.
- Keep a watchful eye on system resources to make sure the Nano is not at risk of crashing. I also added a Notes section to my new techdocs page to start documenting implementations.
I built from source to spare resources on the jetson orin nano which took about 30 minutes. I had concerns about pushing my 8GB of ram to its limit, which is a very valid concern. I need to keep it as lightweight as possible. As much as a docker container appealed to me, it is slim-pickin’s on a budget.
Building from source steps:
Step 1: Install CUD toolkit + cmake with:
Sudo apt install -y cuda-toolkit-13 cmake
Step 2: Install Go (arm 64):
Curl -LO https://go.dev/dl/go1.24.4.linix-arm64.tar.gx && sudo tar -C /usr/local -xzf go1.24.4.linux-arm64.tar.gz && rm go1.24.4.linux-arm64.tar.gz
Step 3: Add Go and CUDA to PATH:
Echo ‘export PATH=$PATH:/usr/local/bint:/usr/local/cuda-13/bin’ >> ~/.bashrc && esport PATH=$PATH:/usr/local/go/bin:/usr/local/cuda-13/bin
Step 4: Clone and build ollama with sm_87:
Git clone https://github.com/ollama/ollama /home/anna/ollama-src && cd /hom/anna/ollama-src && CUDA_ARCHITECTURES=87 cmake
This whole process took roughly 30 minutes.
Effeciency
I wanted to share this process easily, so I connected the gmail MCP server to claude so I can send emails of the process summary to myself.
Maximizing efficiency on the jetson nano was a good choice. I really don't want to be managing a bunch of documentation from my server, so I've created a new place to document my journeys on my new techdocs page. I built this while ollama was building. Understanding resource considerations, I asked claude to also build me a new page for me to put all my tech findings only if it didn't push my resources on my tiny server to the limit - knowing the process at hand was of the utmost importance. It analyzed the situation, gave me real feedback, and told me what could be done since I was already pushing the limits of hardware capabilities with my current install. This was a great insight to not crash the PC while an important process was running. Since everything was already running I was able to add new routes easily with a few lines. Of course I realize I could have done this on another computer, but it’s fun to push the limits of hardware.
Quantization - New Topic (for me)
Quantization:
“Quantization is the process of reducing the precision of a digital signal, typically from a higher-precision format to a lower-precision format.” (Brian Clark, IBM)
After finally getting everything to work, the request took 13 minutes and 20 seconds using the Q8_0 version. This is where I learned about quantization. I then tried another variant, Q4_0 to improve the results.
Benchmarks
Here is the benchmark breakdown of Q8_0 vs Q4_K-M quantization:
Since the bottleneck was "model doesn't fit in available GPU memory," we tested a smaller quantization of the exact same model (llama3.2:1b), rather than switching to a different, weaker model family.
| Q8_0 (original) | Q4_K_M | |
|---|---|---|
| Model file size | 1.5 GB | 808 MB |
| GPU layers loaded | 3–9 of 17 | 17 of 17 (100%) |
| Generation speed | ~1.2–1.35 tok/s | ~30.7 tok/s |
| Real 971-token test | 13m 20s | ~35–45s |
Q8_0 (original) Q4_K_M (new)
Model file size 1.5 GB 808 MB
GPU layers loaded 3–9 of 17 17 of 17 (100%)
Generation speed ~1.2–1.35 tok/s ~30.7 tok/s
Real 971-token test 13m 20s ~35–45s (estimated at this rate)
That's roughly a 25x speedup, because the entire model now fits on the GPU instead of mostly running on the slow CPU path. Amazing! But what's the catch?
Issues
Q4_K_M is fast but produced malformed output sometimes.
Before switching, ran 6 back-to-back test generations with Q4_K_M to check reliability, since lower-precision quantization can be less consistent.
Here's what happened:
- 2 of 6: perfectly valid JSON, correct structure
- 1 of 6: valid JSON, but used a slightly different field name than expected
- 3 of 6: malformed JSON (e.g., a mismatched bracket) that would have crashed the app's parser outright
That's roughly a 50-65% failure rate per attempt. I cannot knowingly ship that, even with a massive speed improvement.
So how do we handle the error-prone behavior of Q4?
Rather than giving up on the faster model, automatic retry logic was added to the app itself. If the model's response is malformed, the app now silently tries again up to 3 times before showing an error. Because each Q4_K_M attempt only takes about 30-45s, even a worst-case 3 attempts is still far faster than a single guaranteed-slow Q8_0 request, while pushing the effective success rate up to roughly 85-95%.
This handles the potential parsing errors gracefully. Since Q4 is 25x times faster, the client won’t really feel it.
Side note: I noticed my Jetson had the “super” abilities after a month of setting it up. Don’t do what I did. Check for super abilities first. It’s a free download. XD
Top comments (15)
This was a great read. I hadn't really thought about the tradeoff between quantization, performance, and output reliability before. The benchmark comparison was especially helpful. Thanks for documenting the whole process!
Thanks! I may have to create a follow-up post at this point with all of the feedback and findings. This is a big rabbit hole for sure.
Thank you for sharing such an excellent post. I really enjoyed reading it.
I’m a Python Full-Stack Engineer with over 10 years of experience designing and building scalable software solutions for clients across a variety of industries. Along the way, I’ve learned that successful projects depend not only on strong technical execution but also on creating real business value.
With my recent contract completed, I’m exploring new opportunities to collaborate with professionals who value innovation, practical problem-solving, and long-term partnerships. I enjoy discussing ideas that combine technical excellence with sound business strategy, creating outcomes that benefit everyone involved.
I believe every connection has the potential to become something meaningful. If you're interested in exchanging ideas, exploring opportunities, or simply connecting with someone who enjoys building impactful technology, I'd be happy to hear from you.
Wishing you success in your future endeavors, and I look forward to connecting.
Great writeup, and the honesty about the 50-65% malformed-JSON rate is the useful part most benchmark posts leave out. One thing that beats the retry loop on the JSON failures specifically: constrained decoding. Ollama rides on llama.cpp, so you can pass a JSON grammar (GBNF) or use the format:json / schema option and the decoder physically cannot emit a token that breaks the structure. That takes the structural failure rate to basically zero instead of 85-95%, and it does it w/o paying the retry tax, which matters most exactly here because latency is your scarcest resource on the Nano. Retrying up to 3x means your worst case is 3x the 35-45s you fought so hard to win back.
The sharper issue hiding under the retry number: retry only catches JSON that fails to parse. It does nothing for the failure Q4 actually introduces, which is well-formed JSON with a wrong value in it. A 1b model at Q4 will happily hand you clean, parseable JSON with a hallucinated field, and the loop counts that as a success because it parsed. So "85-95% effective" is measuring "did it parse," not "is it right," and those two diverge fastest exactly at aggressive quant levels. If the values matter, that is where a Q5_K_M (still fits your 8GB for a 1b) or a cheap validation pass against the source earns its keep more than more retries. Structural validity and correctness are different tests, and quant pressure breaks the second one first while the first one still looks fine.
Yes I found this out quickly. I was reading that a smaller model like qwen could actually out perform a heavily squished version of llama3.2:b. Because quality degrades quickly. Im currently doing some other benchmarks and am considering making the switch. Thanks for your thoughtful response!
That instinct is right, and it is worth being precise about why it holds for your case specifically. Qwen2.5 at 0.5b or 1.5b is a genuinely strong instruction follower for its size, and for structured extraction the thing that degrades first under heavy quant is not general fluency, it is exactly the value-correctness you care about. A smaller model at Q5 or Q8 will often beat a bigger model crushed to Q4 on "did it put the right number in the right field," even when the bigger model wins on a general benchmark. Rank order flips by task, so the general benchmarks you are reading point you the right way but they will not settle it.
The only thing that settles it is your own schema. Build a small gold set once, say 30 to 50 real inputs with the correct JSON written by hand, then run every candidate (qwen-1.5b at Q5, llama-1b at Q4, llama-1b at Q5) against it with the JSON grammar turned on, so structural noise is out of the picture and you are measuring only the wrong-value rate, which is the number that actually decides this. That turns "I read that qwen might be better" into a number for your data. And it compounds: the gold set is a one-time cost, and every future model or quant choice after that is a cheap re-run against it instead of re-forming an opinion from someone else's benchmark. I keep a small public harness that does exactly this field-level scoring (doc-eval on my github) if it saves you the wiring.
Thanks again for your thoughtful feedback. So basically, feed it the general structure of what I want for the output, and then see how each model follows the rules accordingly?
Close, with one refinement that changes what you end up measuring. The structure part you do not test, you enforce: with the JSON grammar on, the model literally cannot produce anything except your shape, so "does it follow the output rules" stops being a question at all. What you test is whether the values inside that shape are correct, and that is why the gold set is the heart of the method: 30 to 50 real inputs where you wrote the correct output by hand, an answer key. Run each candidate model on those inputs, compare what it filled in against what you wrote, field by field, and the wrong-value rate per field is the score.
So the flow is: grammar guarantees the shape, the gold set defines the truth, and the eval just counts disagreements. A model can follow every rule and still put the wrong date in the right field, and that is exactly the failure you want counted, which is only possible because you knew the right answer ahead of time. Build the answer key from your real documents, the ugly ones included, and the whole model-choice question turns into reading one number per candidate.
I see, that makes more sense now! I am genuinely curious to try this method. 🤔 this raises more questions actually.
Ask away, that is what threads are for. One tip to make starting easy: do not wait until you have 50 examples, write 10 from documents you already know well and run the comparison once end to end, the method teaches you more from one full pass than from any amount of planning, and growing the answer key afterward is the easy part.
😂😂😂Here is my reply: dev.to/annavi11arrea1/simple-bench...
Interesting post! I opted for slow, highly accurate models for my personal offline apps. Seeing you use a faster model combined with retry logic for a user-facing app is a great alternative approach. Nice work.
I must note that you need to keep a very watchful eye on the quality of your output. Depending on your use case, the results may not be great. Just some tips from actual experience here. Haha XD Depending on what you are doing, it may not matter.
Retry logic > perfect JSON. 25x faster Q4 with 3 retries = 85-95% success. Ship it.
By reducing the precision of the model (quantizing/compressing) We lose the accuracy needed to output test results that make sense. For example, with Q4, and running three tests, we may only get some test questions with answers that make sense.
The difference, simply put, might output something comparable to the following if given the same information to read from:
Q4: Chickens are primordial retirees
Q8: Chicken soup has been around for a long time.
Precision matters. It's better to not squish, if possible, in my opinion. I know this to be true because after I switched to Q4, my flippycard app started generating absurdities, rendering it useless. I very much wanted to believe your statement, and I agree with your logic, but not if the output is important. Haha
Some comments may only be visible to logged-in visitors. Sign in to view all comments.