Previous Related Post: Part 1
This particular rabbit hole was created due to a previous conversation I had here with another member on DEV about benchmarking and tests. Lot's of questions sparked:
- What's the best way to do them?
- How do we do them?
- What are good numbers?
- What information is actually important?
The truth is that it heavily depends on what you are doing and what is important to you, first and foremost. For me, and my little test generation app, my purpose seemed simple: create a simple app using ai, locally, for free, and take text I have and generate flashcards and quizzes from it so I can review. I can tell you by this point I have spent less time reviewing and more time digging into interesting tools.
On this particular journey, man - I did so much I'm having trouble knowing what to explain first. But I'll try to stick to testing. So basically, I wanted to see which model would run the best on my nano, you know without crashing it, because I did do that actually at one point. That's a tangent I'll save for later but I saved by conversation with claude in my techdocs if you want some entertainment. ( techdocs --> then click jetson nano on the left. ) I did not have enough ram on my nano to handle the test run. This led me to create a swap file to accommodate for lack of ram in case of emergency. Just as a safety net - since I get into trouble.
Safety net in place, I create a quick quiz of the OSI model from a web page to use as source of truth - something to compare the results to. Thanks Vinicius Pereira for the ideas in our chat! So, I ran this test against each one of the models and varying 'quantizations' (is that a word yet?) to bring to light the mystery of this concept.
The high level - if a model is heavily quantized, you lose quality. Brass tax.
I went to these great lengths and then only come to realize, sheesh, this is only for one specific use case. This would take a minute to map performance across many different use cases. I wonder, if you have nano, what have you used Ollama for on it? Not like I need another reason to hoard data. XD
Anyways, in case you don't feel like navigating away, here's a table:
| Model | Quant | Accuracy |
|---|---|---|
| qwen2.5:3b-instruct | q4_K_M | 100% |
| qwen2.5:3b-instruct | q5_K_M | 100% |
| qwen2.5:3b-instruct | q8_0 | 100% |
| qwen3.5:2b | q4_K_M | 0% (empty output) |
| qwen3.5:2b | q8_0 | 0% (empty output) |
| llama3.2:3b-instruct | q2_K | 40% |
| llama3.2:3b-instruct | q4_K_M | 90% |
| llama3.2:3b-instruct | q5_K_M | 90% |
| llama3.2:3b-instruct | q8_0 | 90% |
| mistral:7b-instruct | q2_K | 80% |
| mistral:7b-instruct | q4_K_M | 100% |
| mistral:7b-instruct | q5_K_M | 80% |
Some cells above are empty because they would not fit on the gpu of the nano, so no point in running a test for a model that will never perform as needed due to hardware limitations.
The numbers are based on a pool of 10 questions, so this is why the numbers are so perfectly precise. Obviously this is simple high level test, but I wanted to work through this fully. Having gone through the motions, I know that the rabbit hole is deep and vast and this is barely the surface. But from simple test point of view for my use case of quiz generation, qwen2.5:3b-instruct takes the cake from the bakery to the house. Guess ima have a slice once I recombobulate my app, switching from llama3.2:3b-instruct to qwen ~ a task for another day.
I'm sure many of you reading this have seen other benchmarks elsewhere. I'd love to have some peer review here to tell me if I'm kind of on point here or if my data looks odd. Feel like I'm falling into a dark trap of testing things. I know others have done this and there is other information out there, but what kind of actual appreciation for the technology does that offer?
Top comments (41)
Solid setup. one thing tho β 10 questions on a single OSI quiz is a tiny sample, so 100 vs 90% could just be noise. and that swap file saves you from OOM but tanks tok/s on nano, so worth logging latency next to accuracy too
Hi Hayrullah, it only uses the swap file if memory hits 100% - and at that point it starts gracefully stopping the rogue process before it causes a complete crash. So any process that doesn't complete would be a red flag, and yes it sounds like latency logs would be interesting to log here.
Ah that's a clean failsafe actually β graceful kill at 100% beats a hard crash any day. So a non-completing run basically becomes your signal, makes sense. Latency logs would still show which models flirt with that ceiling before they trip it.
Thank you for this practical and honest benchmark. I truly appreciate your transparency about the trade-offs between heavy quantization and output quality on limited hardware like the Jetson Nano. This kind of realistic testing is exactly what the community needs. I follow your work to learn from these practical insights. I wish you more progress and success!
π§ππ»π
Thank you!
On a Jetson, tokens per second gets the attention, but the number that predicts the actual feel is time to first token after a cold load, since that is what the user waits on. The half nobody benchmarks is whether a smaller quant quietly changes the quality of your generated flashcards, not just the speed. I'd keep a tiny fixed set of my own inputs and score the outputs each run, so a faster quant that degrades answers shows up before users notice.
Local benchmark posts are valuable when they show the constraints, not just the numbers. On small hardware, the practical story is usually latency, memory pressure, model size, thermal behavior, and whether the workflow is still usable after the novelty wears off.
If im understanding you correctly, you are saying the tests would be more useful if they included more specific parameters? Thanks for your feedback. I am learning.
Yes, exactly. The benchmark is already useful because it shows the device and the general outcome. The next step would be making the test more reproducible.
For example: model name and quantization, prompt length, output length, temperature if relevant, memory usage, tokens per second, and whether the device was thermally throttling.
That way a reader can tell whether the result reflects the Jetson Nano, the specific model setup, or the workload.
Thank you for the direction, I am scrounging for time here but I am very curious to do some more in depth poking-around when I can! It is on my list of things to try now. Haha, thanks. βοΈ
That sounds like exactly the right pace. The quick benchmark is already useful because it sets the baseline. The deeper pass can be small: same prompt set, cold vs warm run, token rate, memory pressure, and maybe one practical task that represents what you would actually use the Nano for. No need to turn it into a huge lab project.
One gotcha worth adding to any Ollama benchmark: effective context length. Ollama silently clamps num_ctx to the model's native GGUF window β I asked for 65K, got 40K, and only found out when my session compression started failing without a single error. On constrained hardware where you're already trading accuracy for speed, that silent clamp can quietly change what you're actually measuring.
Hi Nova,
It looks like ollama uses a clamp when a system is being pushed past its limits to stop it from crashing. Thats interesting to note since I made it crash once, its not 100% effective, because it doesnt factor in for say a sudden process to start maybe, mid test. So in theory I guess the clamp is a safe guard. But we cannot be saved from ourselves? π If you think you are measuring with a certain context window, but in reality, its a different context window, I can see how that would be both misleading and irritating! Thanks for the note, I'll try to keep that in mind.
Close, but it's a subtly different limit than it looks. The clamp isn't tied to your hardware being pushed to its edge β Ollama caps num_ctx to the model's native window baked into the GGUF, no matter how much RAM you have.
Ask for 65K on a model exported at 40960 and you get 40960, on a Nano or a beefy server alike. So it's not really a crash-safeguard, it's the model's architectural ceiling enforced silently β and the real risk is exactly what you named: you think you're measuring at one context, you're actually at another.
Once I realized the setting was quietly a lie, I stopped trusting it and sized everything around the model's real native window β same "measure, don't assume" discipline your whole post is built on.
Yes I am definitely dabbling here. People can share all kinds of things with buzzwords all over, but when you actually do the digging, real learning happens. I love that. Because that is real experience. Just like all the awesome feedback on here, which I deeply appreciate.
We can't accept everything - we need to find it to be true, or not. Or/also side bonus: discover other things in the process. And I think that's the magic part. When you have a serious case of the "What-if..."'s. I am feeling slightly defeated by the overwhelming amount of feedback I am receiving on here in the best possible way. I have to go find out more so I can talk about more things. It is needed and I feel compelled from within. πΆβπ«οΈ
I like small hardware benchmarks because they force honesty. The question is not only can it run, but what kind of workflow still feels usable under the constraint. Latency, memory pressure, model size, and thermals are more useful than a single success screenshot.
I completely agree. A single success screenshot doesn't always show the full pictureβreal usability comes from testing performance, limitations, and how the system behaves under different conditions.
Iβm actually interested in QA and user testing, so I enjoy seeing people focus on practical testing rather than just whether something technically works.
Exactly. QA thinking is underrated in AI/hardware posts because it asks the question a demo usually avoids: does this keep working under ordinary use? A single success screenshot tells you it can happen once. Repeated task shape, latency, failure recovery, and limits tell you whether it belongs in a workflow.
This made my week, you took the one-full-pass advice and came back with receipts. And your data has a payoff hiding in it: Qwen2.5:3b holding 100% even at q2_K means that for THIS task the quant precision was never the bottleneck, so you get to take the smallest, fastest quant for free, which on a Nano is real RAM back. The "heavy quantization loses quality" rule did show up, but only in Llama's 40-90% spread. That's the whole argument for measuring instead of assuming: the rule is real, but WHERE it applies is model times task, and your 10-question key just answered it for yours. Keep that harness, it's now your regression gate every time you swap models.
I wasnt even thinking about qwen honestly. I think I got emotional attached to llama3.2 or something, since I can run that on the regular desktop. π€£ what I wanna do next is maybe think of some other useful, use case specific tests for these models and run them through deeper, more rigorous testing. Considering making a tiny database for testing, then it can be referenced. Then the "which model can be used for xyz question can be answered a bit more easily. I am wondering though, what IS the use case to squish a bigger model to the point of dysfunction? Why? π There is a fine line somewhere and I must know.
The use case is when the model is the only thing that fits, not the best thing that fits. On a Nano you have maybe 4-8GB shared with everything else, so the real question isn't "7b or 3b", it's "does a 7b at q2 that barely runs beat a 3b at q5 that runs comfortably?" Your data already says no for quiz generation: the 3b won outright. So the honest rule is squish the biggest model that still clears your task's accuracy bar, and not one bit smaller, which is exactly what your 10-question key measures. The line you're looking for isn't a model property, it's the point where your own eval starts dropping. Below that number the smaller-faster model wins every time.
I just wrote a huge reply and my phone screen went dark and it didn't save. π #dramatic
Losing the long version is a special kind of pain. If the gist survived the crash, drop the two-line recap when you get a second, I am curious where you landed.
Amazing π¦ποΈ
Running Ollama on the Jetson Nano really highlights the memory bandwidth bottlenecks inherent to the Maxwell architecture when handling LLM inference. I found that sticking to heavily quantized 1B or 2B parameter models is practically mandatory to avoid swapping to system RAM, which tanks the tokens per second. Have you experimented with tweaking specific GGUF quantization levels or adjusting context window sizes to squeeze out a bit more throughput for these edge deployments?
Your results actually look pretty reasonable for a small, task-specific benchmark. The important part is that you tested against your own workload instead of chasing generic leaderboard scores. I'd just increase the question set beyond 10 and run each model multiple times to reduce randomness. If qwen2.5:3b consistently gives you the best quality on your Nano, that's the benchmark that matters most for your app.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.