Why I chose Gemma4b over Mistral 7b?

Mayank Dewangan — Mon, 14 Sep 2026 14:08:18 +0000

Introduction

Message grading appears seemingly simple in today's era of powerful LLMs, but this hypothesis crashes when we add hardware as an additional constraint to the existing problem. We are conducting this on relatively weaker hardware, specified as an RTX 4050 with 6GB VRAM and an Intel i7 processor.

Evaluation Criteria

The evaluation criteria for the models are strictly defined schema, 100% batch grading, and no overflow of the context window. Context window refers to the sum of input and output tokens an LLM can hold for a conversation. The schema is attached below.

Phase 1: phi-4-mini

We started with phi-4-mini, assuming the simplicity of the task and that the smallest model in the arsenal would be enough for what we initially deemed a simple task. This assumption was contradicted when we actually tested it in phases. While grading messages individually, all criteria were fulfilled (except for batch-grading, as it is one grading at a time). However, this introduced unnecessary LLM calls by 71%, additional latency of more than 10 minutes per 130 messages, and, most importantly, inefficient use of resources, which is itself a constraint.

Given the new complexities that arose from individual grading, we switched to batch-grading. However, phi-4-mini failed on batch-grading: it failed to give answers in the defined schema, with only 60% of the batch conforming to the schema, which caused the pipeline to break mid-run. After iterating over the system prompt and making changes in the code to catch the failure before the pipeline broke, we finally learned that phi-4-mini was not built for long structured responses.

Phase 2: Mistral 7B

Now, the focus was on getting the response in the 100% defined schema for batch-grading. Mistral satisfied all the criteria even when we did a test run of batch-grading on 40 messages. Things started to break down when we synthesised real-world messages, which might be a combination of long and short messages; this is where Mistral behaved like phi-4-mini.

Investigation and debugging revealed that average context consumption was around 16k tokens, including input and output tokens, while Mistral provided a context window of only 8k. This provided the root cause of why good models were automatically failing: Mistral was not even remembering what it had to do. This led us to narrow our model search to a model that could support long structured responses with a sufficiently long context window to hold the context while performing the task.

Phase 3: Gemma-4b

Gemma-4b was the exact sweet spot we were looking for. It satisfied all the above evaluation conditions on a test set of 52 messages, unlike Mistral. On the given hardware, it provided a 32k context window, which was enough for long messages as well. On the real run, the longest token consumption was 28k, including input and output tokens.

Addressing the Counter-argument

One may counter that I am choosing a weaker model for the job instead of better, more reliable models. However, the point was not to always use the best model; it was to use the acceptable model that can do the job. Unlike Mistral, a higher-parameter model, Gemma provides a sufficiently large context window, whereas Mistral's low context window caused broken output schemas that could not be used. The grading itself was not a tough task, as it was handled by phi-4-mini as well; the real problem was handling batch grading.

Conclusion

The selection of Gemma-4b over Mistral 7B was not a rejection of capability but an acceptance of constraint. When hardware limits the available context window, model selection becomes a problem of fit rather than raw power. Mistral 7B may outperform Gemma-4b on general benchmarks, but a model that cannot hold the context required for the task produces unusable output regardless of its intelligence. The lesson here is simple: the best model is not the one with the most parameters, but the one that satisfies the hard constraints of the problem at hand. For batch grading on a 6GB VRAM machine, Gemma-4b was that model.

What's Your Experience?

Model selection under hardware constraints is a problem many of us face, but the right answer depends entirely on the specific task, dataset, and machine. I've shared what worked for batch grading on a 6GB VRAM setup—now I want to hear from you.

Have you faced a similar trade-off between model capability and context window? Did you choose the larger model and work around its limitations, or did you opt for a smaller model that fit your constraints?

What's your go-to local LLM for constrained hardware? Is it Gemma, Mistral, phi, or something else entirely?

Did I miss a criterion you would have included? Perhaps inference speed, quantization support, or fine-tuning potential?

Drop your thoughts in the comments below. I read every response, and I'm especially curious about cases where a different model won out for you. If you've written about your own local LLM experiments, share a link—I'd love to learn from your approach.

Let's make this a conversation, not just a case study.

I Lost My Best Engineering Advice in a Group Chat. And I Can't Get It Back.

Mayank Dewangan — Thu, 27 Aug 2026 15:25:49 +0000

I'm part of an awesome community where senior devs, product managers, founders, and experienced folks from different domains discuss how they use AI and automation tools to boost productivity :- without sacrificing real learning.

The group is a goldmine. Tool recommendations. Automation workflows. Latest trends. Migration war stories. I've learned more from this group than from most tutorials.

Last week, I needed to find something specific about running local LLMs. I searched with keywords. I scrolled. I found a whole lot of messages :- but none of them answered my question directly.

I still had to manually read through dozens of messages, open the links they shared, and try to piece together the context myself. It took me over an hour, and I wasn't even sure I'd found everything.

Someone might say: "Just Google it." Or "Ask an LLM." But that defeats the purpose. The value here isn't just information :- it's the context. The "this library is a game changer" comment only makes sense if you know what the person was working on before, and who else agreed or disagreed.

That context is lost in scrollback. And I'm tired of trying to keep it all in my head.

I've seen people send important links to themselves on WhatsApp. But that becomes a messy pile with no structure. No connections. No way to see how one message relates to another.

So I'm curious: How do you deal with this?

Have you lost important knowledge in group chats? Do you have a system for recovering it? Or are you also just scrolling endlessly?

I have an idea I'm working on. But I'd love to hear your approaches first. Drop your thoughts in the comments :- I'll document what I learn.

DEV Community: Mayank Dewangan