Introduction
Message grading appears seemingly simple in today's era of powerful LLMs, but this hypothesis crashes when we add hardware as an additional constraint to the existing problem. We are conducting this on relatively weaker hardware, specified as an RTX 4050 with 6GB VRAM and an Intel i7 processor.
Evaluation Criteria
The evaluation criteria for the models are strictly defined schema, 100% batch grading, and no overflow of the context window. Context window refers to the sum of input and output tokens an LLM can hold for a conversation. The schema is attached below.
Phase 1: phi-4-mini
We started with phi-4-mini, assuming the simplicity of the task and that the smallest model in the arsenal would be enough for what we initially deemed a simple task. This assumption was contradicted when we actually tested it in phases. While grading messages individually, all criteria were fulfilled (except for batch-grading, as it is one grading at a time). However, this introduced unnecessary LLM calls by 71%, additional latency of more than 10 minutes per 130 messages, and, most importantly, inefficient use of resources, which is itself a constraint.
Given the new complexities that arose from individual grading, we switched to batch-grading. However, phi-4-mini failed on batch-grading: it failed to give answers in the defined schema, with only 60% of the batch conforming to the schema, which caused the pipeline to break mid-run. After iterating over the system prompt and making changes in the code to catch the failure before the pipeline broke, we finally learned that phi-4-mini was not built for long structured responses.
Phase 2: Mistral 7B
Now, the focus was on getting the response in the 100% defined schema for batch-grading. Mistral satisfied all the criteria even when we did a test run of batch-grading on 40 messages. Things started to break down when we synthesised real-world messages, which might be a combination of long and short messages; this is where Mistral behaved like phi-4-mini.
Investigation and debugging revealed that average context consumption was around 16k tokens, including input and output tokens, while Mistral provided a context window of only 8k. This provided the root cause of why good models were automatically failing: Mistral was not even remembering what it had to do. This led us to narrow our model search to a model that could support long structured responses with a sufficiently long context window to hold the context while performing the task.
Phase 3: Gemma-4b
Gemma-4b was the exact sweet spot we were looking for. It satisfied all the above evaluation conditions on a test set of 52 messages, unlike Mistral. On the given hardware, it provided a 32k context window, which was enough for long messages as well. On the real run, the longest token consumption was 28k, including input and output tokens.
Addressing the Counter-argument
One may counter that I am choosing a weaker model for the job instead of better, more reliable models. However, the point was not to always use the best model; it was to use the acceptable model that can do the job. Unlike Mistral, a higher-parameter model, Gemma provides a sufficiently large context window, whereas Mistral's low context window caused broken output schemas that could not be used. The grading itself was not a tough task, as it was handled by phi-4-mini as well; the real problem was handling batch grading.
Conclusion
The selection of Gemma-4b over Mistral 7B was not a rejection of capability but an acceptance of constraint. When hardware limits the available context window, model selection becomes a problem of fit rather than raw power. Mistral 7B may outperform Gemma-4b on general benchmarks, but a model that cannot hold the context required for the task produces unusable output regardless of its intelligence. The lesson here is simple: the best model is not the one with the most parameters, but the one that satisfies the hard constraints of the problem at hand. For batch grading on a 6GB VRAM machine, Gemma-4b was that model.
What's Your Experience?
Model selection under hardware constraints is a problem many of us face, but the right answer depends entirely on the specific task, dataset, and machine. I've shared what worked for batch grading on a 6GB VRAM setup—now I want to hear from you.
Have you faced a similar trade-off between model capability and context window? Did you choose the larger model and work around its limitations, or did you opt for a smaller model that fit your constraints?
What's your go-to local LLM for constrained hardware? Is it Gemma, Mistral, phi, or something else entirely?
Did I miss a criterion you would have included? Perhaps inference speed, quantization support, or fine-tuning potential?
Drop your thoughts in the comments below. I read every response, and I'm especially curious about cases where a different model won out for you. If you've written about your own local LLM experiments, share a link—I'd love to learn from your approach.
Let's make this a conversation, not just a case study.
Top comments (0)