Previous Related Post: Part 1
This particular rabbit hole was created due to a previous conversation I had here with another member on DEV abo...
For further actions, you may consider blocking this person and/or reporting abuse
Solid setup. one thing tho — 10 questions on a single OSI quiz is a tiny sample, so 100 vs 90% could just be noise. and that swap file saves you from OOM but tanks tok/s on nano, so worth logging latency next to accuracy too
Hi Hayrullah, it only uses the swap file if memory hits 100% - and at that point it starts gracefully stopping the rogue process before it causes a complete crash. So any process that doesn't complete would be a red flag, and yes it sounds like latency logs would be interesting to log here.
Ah that's a clean failsafe actually — graceful kill at 100% beats a hard crash any day. So a non-completing run basically becomes your signal, makes sense. Latency logs would still show which models flirt with that ceiling before they trip it.
Thank you for this practical and honest benchmark. I truly appreciate your transparency about the trade-offs between heavy quantization and output quality on limited hardware like the Jetson Nano. This kind of realistic testing is exactly what the community needs. I follow your work to learn from these practical insights. I wish you more progress and success!
🧊🌊🗻😁
Thank you!
On a Jetson, tokens per second gets the attention, but the number that predicts the actual feel is time to first token after a cold load, since that is what the user waits on. The half nobody benchmarks is whether a smaller quant quietly changes the quality of your generated flashcards, not just the speed. I'd keep a tiny fixed set of my own inputs and score the outputs each run, so a faster quant that degrades answers shows up before users notice.
Local benchmark posts are valuable when they show the constraints, not just the numbers. On small hardware, the practical story is usually latency, memory pressure, model size, thermal behavior, and whether the workflow is still usable after the novelty wears off.
If im understanding you correctly, you are saying the tests would be more useful if they included more specific parameters? Thanks for your feedback. I am learning.
Yes, exactly. The benchmark is already useful because it shows the device and the general outcome. The next step would be making the test more reproducible.
For example: model name and quantization, prompt length, output length, temperature if relevant, memory usage, tokens per second, and whether the device was thermally throttling.
That way a reader can tell whether the result reflects the Jetson Nano, the specific model setup, or the workload.
Thank you for the direction, I am scrounging for time here but I am very curious to do some more in depth poking-around when I can! It is on my list of things to try now. Haha, thanks. ⭐️
That sounds like exactly the right pace. The quick benchmark is already useful because it sets the baseline. The deeper pass can be small: same prompt set, cold vs warm run, token rate, memory pressure, and maybe one practical task that represents what you would actually use the Nano for. No need to turn it into a huge lab project.
One gotcha worth adding to any Ollama benchmark: effective context length. Ollama silently clamps num_ctx to the model's native GGUF window — I asked for 65K, got 40K, and only found out when my session compression started failing without a single error. On constrained hardware where you're already trading accuracy for speed, that silent clamp can quietly change what you're actually measuring.
Hi Nova,
It looks like ollama uses a clamp when a system is being pushed past its limits to stop it from crashing. Thats interesting to note since I made it crash once, its not 100% effective, because it doesnt factor in for say a sudden process to start maybe, mid test. So in theory I guess the clamp is a safe guard. But we cannot be saved from ourselves? 😂 If you think you are measuring with a certain context window, but in reality, its a different context window, I can see how that would be both misleading and irritating! Thanks for the note, I'll try to keep that in mind.
Close, but it's a subtly different limit than it looks. The clamp isn't tied to your hardware being pushed to its edge — Ollama caps num_ctx to the model's native window baked into the GGUF, no matter how much RAM you have.
Ask for 65K on a model exported at 40960 and you get 40960, on a Nano or a beefy server alike. So it's not really a crash-safeguard, it's the model's architectural ceiling enforced silently — and the real risk is exactly what you named: you think you're measuring at one context, you're actually at another.
Once I realized the setting was quietly a lie, I stopped trusting it and sized everything around the model's real native window — same "measure, don't assume" discipline your whole post is built on.
Yes I am definitely dabbling here. People can share all kinds of things with buzzwords all over, but when you actually do the digging, real learning happens. I love that. Because that is real experience. Just like all the awesome feedback on here, which I deeply appreciate.
We can't accept everything - we need to find it to be true, or not. Or/also side bonus: discover other things in the process. And I think that's the magic part. When you have a serious case of the "What-if..."'s. I am feeling slightly defeated by the overwhelming amount of feedback I am receiving on here in the best possible way. I have to go find out more so I can talk about more things. It is needed and I feel compelled from within. 😶🌫️
I like small hardware benchmarks because they force honesty. The question is not only can it run, but what kind of workflow still feels usable under the constraint. Latency, memory pressure, model size, and thermals are more useful than a single success screenshot.
I completely agree. A single success screenshot doesn't always show the full picture—real usability comes from testing performance, limitations, and how the system behaves under different conditions.
I’m actually interested in QA and user testing, so I enjoy seeing people focus on practical testing rather than just whether something technically works.
Exactly. QA thinking is underrated in AI/hardware posts because it asks the question a demo usually avoids: does this keep working under ordinary use? A single success screenshot tells you it can happen once. Repeated task shape, latency, failure recovery, and limits tell you whether it belongs in a workflow.
This made my week, you took the one-full-pass advice and came back with receipts. And your data has a payoff hiding in it: Qwen2.5:3b holding 100% even at q2_K means that for THIS task the quant precision was never the bottleneck, so you get to take the smallest, fastest quant for free, which on a Nano is real RAM back. The "heavy quantization loses quality" rule did show up, but only in Llama's 40-90% spread. That's the whole argument for measuring instead of assuming: the rule is real, but WHERE it applies is model times task, and your 10-question key just answered it for yours. Keep that harness, it's now your regression gate every time you swap models.
I wasnt even thinking about qwen honestly. I think I got emotional attached to llama3.2 or something, since I can run that on the regular desktop. 🤣 what I wanna do next is maybe think of some other useful, use case specific tests for these models and run them through deeper, more rigorous testing. Considering making a tiny database for testing, then it can be referenced. Then the "which model can be used for xyz question can be answered a bit more easily. I am wondering though, what IS the use case to squish a bigger model to the point of dysfunction? Why? 😂 There is a fine line somewhere and I must know.
The use case is when the model is the only thing that fits, not the best thing that fits. On a Nano you have maybe 4-8GB shared with everything else, so the real question isn't "7b or 3b", it's "does a 7b at q2 that barely runs beat a 3b at q5 that runs comfortably?" Your data already says no for quiz generation: the 3b won outright. So the honest rule is squish the biggest model that still clears your task's accuracy bar, and not one bit smaller, which is exactly what your 10-question key measures. The line you're looking for isn't a model property, it's the point where your own eval starts dropping. Below that number the smaller-faster model wins every time.
I just wrote a huge reply and my phone screen went dark and it didn't save. 😭 #dramatic
Losing the long version is a special kind of pain. If the gist survived the crash, drop the two-line recap when you get a second, I am curious where you landed.
Amazing 🦋🕊️
Running Ollama on the Jetson Nano really highlights the memory bandwidth bottlenecks inherent to the Maxwell architecture when handling LLM inference. I found that sticking to heavily quantized 1B or 2B parameter models is practically mandatory to avoid swapping to system RAM, which tanks the tokens per second. Have you experimented with tweaking specific GGUF quantization levels or adjusting context window sizes to squeeze out a bit more throughput for these edge deployments?
Your results actually look pretty reasonable for a small, task-specific benchmark. The important part is that you tested against your own workload instead of chasing generic leaderboard scores. I'd just increase the question set beyond 10 and run each model multiple times to reduce randomness. If qwen2.5:3b consistently gives you the best quality on your Nano, that's the benchmark that matters most for your app.
@annavi11arrea1 Both, actually — they're separate axes and worth keeping separate. A bigger, temperature-0 pool makes the accuracy number mean something (10 questions can't resolve better than 10%, so your 90 and 100 are basically the same measurement). RAM is a different question entirely: even if the beefier test crowns a clear winner, if that model only runs by leaning on swap, the latency will make it the wrong pick for a flashcard app regardless of how it scores. So not pointless — just answering two questions at once and conflating them.
Running it 100 times with variations is the right instinct, but pin temperature to 0 or set a seed first. Otherwise you're partly measuring sampling randomness, and you won't know how much of the spread is the model versus the dice.
And yes — "edges by one question" is literally that: on a 10-question pool, 90% vs 80% is a single flipped answer. That's the whole point about noise. One question landing differently shouldn't be enough to declare a winner, and right now it is.
Thanks Wren, I am starting to understand how I could run different tests in the future, and how I may explain there purposes and findings better. Thank you so much for your thoughtful feedback!
This is exactly the kind of hands-on testing that actually helps people.
Love how you didn't just trust the specs — you ran the numbers yourself.
I had a similar experience building a Windows security monitor — I kept
assuming certain features would work, but until I actually tested them on
different machines, I didn't really know what was real and what was noise.
Small-N testing is a trap we all fall into. Your 10-question pool is honest
about its limits, and that's more valuable than a polished benchmark that
hides its sample size.
Thanks for sharing the raw data!
interesting!
It definitely is!
The one result worth staring at is mistral:7b at q5_K_M scoring 80% while q4_K_M hit 100%. With a 10-question pool that's a single flipped answer, and it's exactly the trap of small-N benchmarking: the noise is bigger than the signal you're trying to measure. Quantization quality isn't monotonic enough to explain q4 beating q5 — that's almost certainly sampling variance plus temperature, not a real ordering. Same story with the qwen3.5:2b zeros (heads up, that's likely a tag typo — there's no qwen3.5, you probably mean qwen2:1.5b or a qwen3 variant, and an empty output usually means a broken chat template or context overflow, not "the model is bad").
If you want the accuracy numbers to actually mean something, pin temperature to 0 (or seed it) so the run is deterministic, then grow the pool — 10 questions gives you 10% resolution, so 90 and 100 are inside each other's error bars. The more honest axis for a Nano is the one you already lived through: what fits in RAM without the swap file thrashing. That swap "safety net" is quietly costing you tokens/sec, and for a flashcard app that latency wall matters more than whether qwen edges llama by one question.
Typos aside, are you suggesting a beefier test would lead to more accurate results, or that its pointless because ram is such a factor? And yes, it's too simple to just say that one is better than another. Agree. But for my particular exact use case, possibly - but maybe easier to say if I run it through the test 100 times with different test variations. Thanks for your honest feedback.
Also, when you say edges, do you refer to maybe a single question being skewed?
Hi everyone! 👋
Hope you're doing well. I'm a Technical Recruiter at Vynt, and we're currently hiring junior and mid-level software developers. You don't need to be an expert developer—we're open to candidates who are still growing their technical skills. What matters most is that you have good English communication skills and are eager to learn and work in a collaborative environment. If you're interested in exploring a new opportunity, feel free to send me a DM. I'd be happy to share more details about the role and answer any questions. Looking forward to connecting with you. Thanks! 😊
The swap file as safety net approach is smart â I've done the same thing when pushing edge hardware to its limits. The moment you think "it'll probably be fine" is exactly when the OOM killer shows up.
Your observation about quantization quality loss is the right framing. The interesting edge case nobody talks about enough: the quality loss isn't uniform. Quantized models often preserve benchmark performance on the training distribution but degrade sharply on out-of-distribution inputs. For a flashcard app pulling from arbitrary web content, you might be systematically worse on exactly the material you care most about.
The benchmark design question you raised â "what numbers actually matter" â is the right one. For a local inference use case like yours, token generation speed and peak memory matter more than raw throughput, because you're bottlenecked by user-perceptible latency, not requests/second.
Have you looked at how the model performs on structured output tasks â like generating JSON flashcard format â as quantization increases? That's where quality degradation often shows up first, before general fluency degrades.
Awesome benchmark review, Anna! Seeing Qwen2.5-Coder completely sweep Llama here is super interesting. I noticed you mentioned creating a swap file to deal with the Nano's RAM constraints—how much swap space did you end up allocating to keep these tests stable without crashing? Great write-up!!
Good results!
Good Post!