Originally published on my Substack. I'm a Microsoft MVP based in Japan, writing in English about the AI agent systems I actually run in production.
Local AI models keep multiplying. But comparing numbers on model cards alone doesn't tell you which one to actually use.
Does a higher parameter count mean smarter? Does MoE mean faster? If a model is popular on AI Arena, is it good for my own work? Each question offers a partial clue, but in the end you can't decide without running the same task through the models yourself.
So this time, I ran the exact same Japanese question through 7 major models running on a single NVIDIA DGX Spark. That includes NVIDIA's Nemotron 3 Super 120B-A12B, for which I actually deployed the large quantized version.
What I compared wasn't just tokens/s.
Whether the answer finished in the specified format
Whether it made claims that would be risky in real business use
What caveats it raised
Whether any model couldn't answer without changing its settings
The results didn't shake out into a clean "bigger model = better" story. One model was fast but gave an answer too risky to show a customer as-is. Another had excellent timing but dropped the instruction to provide 3 items. And one 120B model, after thinking as hard as it could, ended up with a completely empty answer field.
So which model struck the best balance for short business-style answers? And did Nemotron 3 Super prove that its 87GB of weights are worth it compared to a 23GB-class model? Below I lay out the raw measurements and every full answer, unedited.
🤖✍️ This article was co-written with AI — AI agents (Claude Code / Codex) generated the draft based on actual collaborative work with Ebisuda and real measurement data, and Ebisuda reviewed and edited the content before publishing.
📘 The infrastructure behind my workflow
Ebi Workspace — Never forgetting experimental conditions and decisions
To keep observing while only swapping out the model, you need to keep the previous conditions, raw logs, and the reasoning behind each decision in one place. I use "Ebi Workspace," a plugin compatible with Claude Code / Codex, to manage the project's current state and measurement records in Obsidian.
👉 Claude Code / Codex That Never Forgets, Never Gets Lost — Ebi Workspace
Ebi Agent Chat Relay — Driving real hardware work from Discord
I handle model deployment, measurement, and article writing from Discord through the open-source "Ebi Agent Chat Relay." It's a conversation and coordination framework that splits multiple AI sessions into separate threads, letting them run while avoiding conflicts over the same work targets.
👉 Ebi Agent Chat Relay (GitHub)
Same input, watching speed and answer quality together
I used the GB10 in a DGX Spark. The unified memory available from the OS is about 121GiB, and Ollama is version 0.32.9. I fixed the comparison conditions as follows.
temperature: 0
seed: 42
context: 8192
Max generation budget: 320 tokens
thinking: OFF
One cold run after unloading each model, followed by 3 warm runs
Primary comparison: average response time across the 3 warm runs
Optimizing each model individually toward its own recommended settings would blend configuration differences together with model differences. So I kept the primary comparison on common conditions, and only ran separate diagnostics to check the cause for any model that couldn't answer under those common conditions.
The question posed a scenario where a small or midsize business wants to adopt generative AI without sending customer data to an external cloud AI. I asked the models to list 3 tasks suited to local AI, with each item written as a single line in the form "Task name: reason / caveat," followed by a short adoption verdict at the end. I also prohibited inventing product names or law names.
Speed first: more than a 14x gap between the fastest and slowest
The warm averages ranked as follows.
Qwen3.5 35B — 1.49s, 76.27 tokens/s, followed the format
GLM-4.7-Flash — 1.94s, 64.64 tokens/s, broke the format
Qwen3.6 35B-A3B — 2.12s, 44.72 tokens/s, followed the format
GPT-OSS 120B — 8.98s, displayed answer was empty
Nemotron 3 Super 120B-A12B — 9.09s, 20.36 tokens/s, followed the format
Gemma 4 31B — 9.81s, 10.41 tokens/s, followed the format
Qwen2.5 72B — 21.68s, 4.42 tokens/s, followed the format
For short-form responses, Qwen3.5 is overwhelmingly the fastest. But looking at the actual answer shows why deciding on speed alone is dangerous.
The fastest model, Qwen3.5, also returned its risky claim the fastest
Here's Qwen3.5 35B's answer.
Customer support log analysis: zero risk of confidential data leakage / need to check the balance between processing speed and accuracy
Internal document summarization and drafting: no need to send materials externally even when they contain personal information / depends on the quality of the training data
Business rule matching system: can reference confidential internal regulations with high security / incurs initial setup costs
Adoption verdict: when handling highly confidential customer data, local AI is a valid option.
The format is clean. But "zero risk of leakage" is an overstatement. Even with local deployment, risks remain from device compromise, permission settings, logs, backups, and data being carried out. The fastest number on the clock and an answer quality you could adopt for business as-is didn't line up.
At just over 2 seconds, Qwen3.6 was the most practical for this one question
Here's Qwen3.6 35B-A3B's answer.
Using internal data: avoids the risk of customer information leakage / requires specialized knowledge to set up the environment
Internal document summarization: drafting meeting minutes where confidentiality is essential / improving accuracy requires curating training data
Handling business FAQs: responding to inquiries that include personal information / building a verification process to prevent incorrect answers
Adoption verdict: carefully weigh the balance between securing safety and operational load, and roll it out in stages.
It raised distinct caveats — environment setup, training data, and a verification process for incorrect answers — and kept its final verdict to a staged rollout. It followed the format in 2.12 seconds.
This is purely my editorial judgment on this one question, but for short, structured Japanese business answers, Qwen3.6 would be my first candidate. It wasn't the fastest, but it struck the best balance between speed and answer quality.
Nemotron 3 Super wasn't slow — its answer was long and careful
Nemotron 3 Super, added this time, is a 120B total / 12B active MoE. Its Q4_K_M quantization in Ollama measured 86.83GB. On the DGX Spark, all the weights fit onto the GPU side, and the warm average was 9.09 seconds.
Auto-generating FAQs for customer support: can learn internal knowledge and respond instantly / operate only within a scope that excludes confidential information
Drafting sales materials: generates templates from past proposals and deal records / exclude personal information from the input
Summarizing and translating internal manuals: restructures long documents into forms appropriate for each department / don't feed in documents where alteration is prohibited
Adoption verdict: it's appropriate to start with a staged pilot operation, within a scope that protects data confidentiality while achieving operational efficiency.
Among the 7 models, this was the longest and most careful answer. On the other hand, it's somewhat conservative — for instance, demanding the exclusion of personal information even though local operation is the whole premise. On raw short-answer speed alone, it loses to the 23GB-class Qwen3.6.
That said, Nemotron 3 Super's real strength lies in agentic processing, code, function calling, and long-form reasoning. You can't conclude from this one question that "loading 87GB isn't worth it." It didn't show a speed advantage in the short-answer sprint — the result points to measuring it on longer work next.
GPT-OSS thought for 8.98 seconds and left the answer field empty
GPT-OSS 120B used up the shared 320-token budget entirely on internal thinking, leaving the displayed answer empty. This isn't an API error. It's the result of all 320 generated tokens going into the thinking side.
Even raising the max generation budget to 1000 tokens, it cut off partway through the second item. Raising it to 4000 tokens finally produced a complete answer, at a warm time of 25.89 seconds and 1087 output tokens.
Handling customer inquiries: can be trained using only internal data, reducing the risk of information leakage / model updates require regular data maintenance
Automatic summarization of internal documents: can extract key points without sending confidential information externally / summarization accuracy requires tuning for business terminology
Inventory optimization simulation: runs demand forecasting and ordering plans locally / needs to periodically retrain on seasonal variation in the data
Adoption verdict: if confidentiality is the top priority, start with local AI and consider expanding in stages.
Even with the same model, not knowing the generation budget turns it into "a model that answers nothing." Model configuration is part of its performance.
The remaining 3 models' answers
GLM-4.7-Flash
Summarizing and drafting internal documents: since it contains confidential information, data leakage risk can be avoided. / If there's uncertainty about accuracy due to insufficient recognition, training adjustments on the local model are needed.
Generating FAQs from customer support history: quality is high because it can reflect internal-specific terminology and know-how.
Adoption verdict:
If data protection is the priority, it's best suited for document creation and QA tasks.
1.94 seconds is attractive, but against the requirement of 3 items it effectively delivered only 2, and the one-line-under-80-characters constraint broke down too. By speed measurement alone it ranks 2nd; by the standard of completing the task, it's a failure.
Gemma 4 31B
Looking up internal regulations: handles highly confidential material such as employment rules, to prevent external leakage / answer accuracy
Drafting customer responses: creates drafts without hiding personal information, for efficiency / securing compute resources
Summarizing technical documents: to safely analyze proprietary design documents and know-how / model optimization
Adoption verdict: recommended when confidentiality is the top priority and the operating cost is acceptable.
It followed the format. But the caveats end in bare nouns like "answer accuracy" and "securing compute resources," leaving it weak on what to actually check. "Without hiding personal information" is also a phrase that shouldn't be adopted without permissions and auditing in place first.
Qwen2.5 72B
Customer information management: strengthens data protection and privacy compliance / high initial cost
Document creation support: can handle highly confidential information / requires learning up-to-date information
Automating quality inspection: improves productivity through fast processing / requires specialized training data
Adoption verdict: suited to work where security and privacy are prioritized.
It followed the format, but at 21.68 seconds it was the slowest among the models that completed their answer this time. "Privacy compliance" can't be guaranteed by localization alone, and the answer overall stays abstract.
My conclusion: measure task completion, not model rankings
I drew four conclusions from this one question.
The fastest model doesn't necessarily return the safest answer
Even with high tokens/s, dropping the specified format means the task isn't complete
Thinking models are put at a disadvantage just by matching the generation budget, so separate the common-condition test from the recovery diagnostic
For short structured answers, Qwen3.6 strikes a good balance. Nemotron 3 Super should be re-evaluated on longer agentic work
This isn't a comprehensive ranking. There was only one question, and I didn't measure factuality, code, images, long context, or parallel load. The common conditions of temperature=0 and thinking OFF also aren't settings that draw out each model's maximum capability.
That's exactly why I kept the raw answers. Rather than stopping at "it took 9.09 seconds," I compared what it actually said in those 9.09 seconds. From there, you can pick the model that fits your own work.
Next, I'll hand the same agentic task to Nemotron 3 Super, Qwen3.6, and GPT-OSS, and compare completion rate, tool calls, and time taken. After Qwen wins the sprint, does the 87GB Nemotron come back in the long distance? That's the next thing to observe.
One thing that caught my attention in the distribution license labeling
As of August 12, 2026, NVIDIA's official Hugging Face model card points to the NVIDIA Nemotron Open Model License. On the other hand, the embedded label on nemotron-3-super:120b-a12b-q4_K_M as pulled from Ollama showed the NVIDIA Software and Model Evaluation License.
Since my use here is evaluation, I went ahead with the measurements, but before moving the Ollama-distributed build into production or commercial use, I'll double-check whether this labeling discrepancy has been resolved. I'm not drawing a legal conclusion here.
So, what did you think? You might have been surprised that even a smaller local LLM can return answers this fast and this good. They keep getting smaller and smarter. That feeling of the same hardware getting more and more capable over time is something you can only experience in an era like this one. It's genuinely fascinating.
Top comments (1)
The empty-answer case is the one I’d keep in the headline of the benchmark, honestly. Tokens per second is useful, but a model that silently exits the format contract is a different failure class. I’d add a simple pass/fail column for instruction compliance before ranking any of the timing numbers.