The latest articles on DEV Community by Poojitha Saranya (@poojitha_saranya_2aadb3a7). https://dev.to/poojitha_saranya_2aadb3a7 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4163488%2F21c16a37-7856-4199-8a05-d38ed7ff9743.png https://dev.to/poojitha_saranya_2aadb3a7 en Poojitha Saranya Sat, 10 Oct 2026 19:27:01 +0000 https://dev.to/poojitha_saranya_2aadb3a7/telugumixbench-evaluating-llms-on-telugu-and-telugu-english-code-mixed-tasks-o8b https://dev.to/poojitha_saranya_2aadb3a7/telugumixbench-evaluating-llms-on-telugu-and-telugu-english-code-mixed-tasks-o8b <h2> What I Benchmarked </h2> <p>Can large language models reliably understand Telugu and Telugu-English code-mixed prompts?</p> <p>To explore this, I created <strong>TeluguMixBench</strong>, a small benchmark containing 20 test cases across five categories:</p> <ul> <li> <strong>Reading comprehension:</strong> Understanding Telugu passages and answering questions.</li> <li> <strong>Reasoning:</strong> Solving simple problems expressed in Telugu.</li> <li> <strong>Translation:</strong> Translating Telugu sentences into English.</li> <li> <strong>Code-mixed understanding:</strong> Handling prompts that combine Telugu and English.</li> <li> <strong>Instruction following:</strong> Following constraints such as providing a specified number of points or answering in a requested language.</li> </ul> <p>I chose this problem because multilingual AI evaluation should consider not only widely used languages but also regional languages and the way people naturally mix languages in everyday conversations.</p> <h2> Models Tested </h2> <p>I used Kaggle Benchmarks to evaluate models on TeluguMixBench.</p> <p>The leaderboard included:</p> <ul> <li>Claude Opus 4.5</li> <li>Claude Haiku 4.5</li> <li>Claude Haiku 5.5</li> <li>Gemini 3.7 Flash</li> </ul> <p>I wanted to explore how models from different providers performed on the same set of language tasks and how their scores compared.</p> <h2> Findings </h2> <p>The initial leaderboard showed the following scores:</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Model</th> <th>Score</th> </tr> </thead> <tbody> <tr> <td>Claude Opus 4.5</td> <td>100%</td> </tr> <tr> <td>Claude Haiku 4.5</td> <td>100%</td> </tr> <tr> <td>Claude Haiku 5.5</td> <td>100%</td> </tr> <tr> <td>Gemini 3.7 Flash</td> <td>95%</td> </tr> </tbody> </table></div> <p>These results are an interesting starting point, but they should be interpreted cautiously.</p> <p>The benchmark currently contains only 20 examples, and some evaluation checks rely on expected-answer matching rather than a full assessment of meaning. A model may include an expected keyword without providing a completely correct answer. Perfect scores therefore do not establish that a model reliably understands Telugu across different contexts.</p> <p>My main takeaway is that <strong>the quality of an evaluation method matters as much as the scores it produces</strong>. A useful next step is to expand the dataset, improve semantic evaluation, test more varied Telugu-English prompts, and examine incorrect responses individually.</p> <h2> My Benchmark </h2> <p>Explore TeluguMixBench on Kaggle:</p> <p><a href="https://www.kaggle.com/benchmarks/poojithasaranya/telugumixbench/leaderboard" rel="noopener noreferrer">https://www.kaggle.com/benchmarks/poojithasaranya/telugumixbench/leaderboard</a></p> <p>The benchmark is an initial step toward evaluating regional-language and code-mixed capabilities more systematically. I hope to improve its coverage and scoring methodology in future iterations.</p> devchallenge kagglechallenge ai machinelearning