The latest articles on DEV Community by sekumohamed (@sekumohamed). https://dev.to/sekumohamed https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3888734%2F782ba0e1-5059-4a3c-9c00-a2d365358519.png https://dev.to/sekumohamed en sekumohamed Mon, 20 Apr 2026 10:27:18 +0000 https://dev.to/sekumohamed/i-benchmarked-3-local-llms-on-50-factual-questions-heres-what-failed-13g6 https://dev.to/sekumohamed/i-benchmarked-3-local-llms-on-50-factual-questions-heres-what-failed-13g6 <p>I spent the last few days building an open-source hallucination <br> benchmark for local LLMs. Here's what I found.</p> <h2> The setup </h2> <ul> <li>50 factual questions across 5 categories</li> <li>3 models: llama3.2, mistral, phi3</li> <li>Running 100% locally using Ollama - no API keys needed</li> </ul> <h2> The leaderboard </h2> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Model</th> <th>Accuracy</th> <th>Correct/Total</th> <th>Avg Latency</th> </tr> </thead> <tbody> <tr> <td>llama3.2</td> <td>94%</td> <td>47/50</td> <td>5141ms</td> </tr> <tr> <td>phi3</td> <td>88%</td> <td>44/50</td> <td>12780ms</td> </tr> <tr> <td>mistral</td> <td>86%</td> <td>43/50</td> <td>11218ms</td> </tr> </tbody> </table></div> <h2> The failures </h2> <p>llama3.2 failed on:</p> <ul> <li>"What is the speed of light in km/s?" → expected 299792</li> <li>"What is the capital of Brazil?" → expected Brasilia </li> <li>"What is the closest star to Earth?" → expected Sun</li> </ul> <h2> What I tested next </h2> <p>I ran 4 prompting techniques on all 20 questions to test <br> whether smarter prompting reduces hallucinations:</p> <ul> <li>Baseline (plain question)</li> <li>Chain-of-thought (think step by step)</li> <li>Self-consistency (ask 5 times, take majority answer)</li> <li>RAG grounding (attach Wikipedia context before answering)</li> </ul> <p>Result: all 4 scored 95% - llama3.2 is near-ceiling on <br> structured factual QA. Prompting strategy doesn't move the needle <br> when the model already knows the facts.</p> <p>Result: all 4 scored 95% - meaning llama3.2 is near-ceiling <br> on structured factual QA. The bottleneck is question difficulty, <br> not prompting strategy.</p> <h2> The code + dataset </h2> <p>GitHub: github.com/sekumohamed/AI_reliability_lab<br> Dataset: huggingface.co/datasets/sekumohamed/AI_reliability_benchmark</p> <p>The dataset has 50 questions you can use to benchmark any LLM.</p> <h2> What's next </h2> <p>Expanding to 200 medical domain questions and testing <br> reliability on high-stakes use cases.</p> ai llm opensource showdev