This is a submission for the Kaggle Benchmarking Challenge
Table of Contents
One Eye, Zero Excuses
What I Benchmarked
Why Georgian?...
Some comments have been hidden by the post's author - find out more
For further actions, you may consider blocking this person and/or reporting abuse
What a brave move to take on that translation process! Your drive to wrestle AI with a foreign script is commendable. Thank you for sharing some of your frustrations as well, it makes your articles so much more relatable. I love that you went the extra mile to give a nice UI for viewing.
The fact that we can create data that we understand when we can't even read the script is absolutely wild.
Great project! 🦚✨️
Thank you Anna. Feedback like this means a world to me. Especially after so much hard work and so many different obstacles, I was this close to giving up. But I'm glad I didn't. It's such a rewarding feeling, every single minute spent on that research totally worth it!🔥
That's exactly how I felt when I was doing the Sanity challenge. I was SO lost at first. 😂 I'm glad this project was a rewarding experience for you, super cool!
What a fantastic article! I've been to Georgia once, and I have to say, the Georgian script and phonetics are absolutely wild! They really do look and sound like some ancient, demonic language. 🤣
And I'm genuinely shocked by the base-20 number system! So, beyond the benchmarks, I found the language itself absolutely fascinating!
It's also interesting how much LLMs still struggle with translating into Georgian. Looks like they won't be taking over the world just yet! 😂
Oh, by the way, The Witcher game I mentioned is actually based on a book series by Polish author Andrzej Sapkowski. The game is one of my favorites, and one day I'm going to read the books too. The studio behind it, CD Projekt Red, is Polish as well. They also made Cyberpunk 2077, another favorite of mine and one I take inspiration from when designing my projects. I love that studio.
See? It's a small world. 😄
Exactly! 😄 And of course, both Sapkowski and CD Projekt Red are incredibly well-known here in Poland! I think every ambitious high school student has read The Witcher books at some point. 😂
Well, can't blame them. The game is SO freakin' good. I know the books would be even better.
Unfortunately can't say the same about the Netflix TV show, OMG that's absolutely terrible. The first season is OK, but after that? A huge failure.
Yeah, for someone who doesn't read, speak, or write Georgian, it can feel like a completely different world. Like Elvish or something. 😄
The LLMs definitely didn't have it easy. But they actually did better than I thought. The results probably would've been even better with reasoning turned on, but that would've taken much more time, which I didn't have. Maybe next time.
First of all, the pirate look is pretty cool, Captain Giorgi! 😹🏴☠️
But seriously, I hope your eye recovers quickly and you’re feeling 100% again soon!
I really enjoyed this benchmark too. I especially liked how careful you were about keeping the comparison fair. Same prompts, same conditions, and even treating GPT-6 Astra / Claude Opus 5 as a separate low-thinking pair instead of mixing them with the no-thinking runs.
The failure guard was a great detail as well. Stopping the run when too many calls fail is much better than letting infrastructure problems silently turn into “model scores.”
And Georgian itself looks absolutely brutal 😹
The number system and verb forms alone already feel like a boss fight.
Great work, especially with one eye patched up! 🚀
Thanks a lot! Captain Girogi sounds pretty good! 😄
Yeah, keeping it fair wasn't easy. It made me frustrated quite a lot of times. I wanted to do it more comprehensive, but Kaggle's 10$ limit is something that doesn't help you at all. It really is a constraint.
Right, Georgian is super brutal. If I stop a person now in the street and ask them to pronouce 15,000,000 in its ordinary form, I'm pretty sure 9 of 10 people will get it wrong.😄
One thing I’d watch with the probe-model approach is whether the benchmark starts inheriting that model’s blind spots. Keeping the items it gets wrong is a practical way to stretch a limited evaluation budget, but it can also make the test disproportionately difficult in exactly the areas that model struggles with.
For example, if the probe consistently misses Georgian ordinal forms like
მე-18or rare verb agreement patterns, those examples are useful stress tests, but they shouldn’t be the only basis for comparing models. I’d keep a separate, untouched holdout set with stratified coverage across task types. That way, you can measure both how well models handle known failure cases and whether the ranking generalizes beyond the probe’s error profile.That's completely true. Couldn't agree more! And that's why I didn't only leave the rows that the probe model got wrong. I included a bunch of those that it got right to keep the benchmark fair.
Great observation!
giorgi, massive respect for this. shipping a rigorous, hand-written benchmark with a patched-up eye is the definition of "no excuses." michael bisping would be proud. 🏴☠️🔥
as a developer who builds strictly under heavy constraints (like running everything on a $150 phone), i deeply appreciate how you let kaggle's $10/day limit shape the project. cutting from 41 models to 8 and 9 dimensions to 5 didn't weaken the benchmark; it made it sharper and more focused. ruthless editing is a senior dev superpower.
your technical workarounds are brilliant. the
>5% error guardto prevent infrastructure failures from masquerading as bad model scores is exactly the kind of defensive engineering that separates toy scripts from production-grade tools. and finding 4 (the 2-2.6x token gap for claude on georgian script) is a phenomenal insight for anyone building in low-resource languages.also, huge respect for the zero-framework, static html dashboard generated straight from python. no bloated build steps, just raw, efficient code that guarantees the data and the UI never disagree.
incredible work, captain. rest that eye, and keep shipping! 🐯💻
Now, ladies and gentleman, this is how you give feedback. Thank you so much! I tried my absolute best and I'm glad I did. Because of comments like this one!🙏
daniel, glad the
keep_alivefix resonated! real-world outages are the ultimate debugging tool. 🐯to answer your question about the $150 phone: it’s all about ruthless optimization.
the goal is never to force the hardware to do what it can't. it's about designing the software to perfectly match the hardware's reality. keep building! 💻🌍
This is a great article, and honestly I don't know how you finished it with one eye patched. Reading that Georgian numbers go up to 10³⁹ and models still mess up ordinals was wild. The fact that you wrote every single dataset item by hand says a lot. Respect bro..
Thanks a lot. It wasn't easy for sure. But with enough dedication and commitment, I could do it no matter what.🔥
First time hearing about Georgian language.
I also did something related to translation and yes the kaggle credits just wasn't enough.
I had to email them and gladly they boosted my quota 😅
That limit really tested my ability to pivot quickly and work within tighter margins without compromising the project.
the comprehension vs production gap is consistent with how these models were trained — most Georgian text online is content native speakers read, not generated output that teaches production patterns. the models saw enough to parse but not enough to mimic fluent generation.
we see the same asymmetry in domain jargon: models parse technical terminology accurately but when generating, they default to generic phrasing or hallucinate terminology that sounds plausible but isn't correct.
the consonant cluster failures sound like they're hitting BPE tokenization limits. does the token overhead for Georgian affect context tracking across a sentence?
Good question. Within short, casual sentences, it doesn't really affect context tracking, but in long, complex sentences it might affect at some degree.
Finding 4 is the one I'd put on a slide: Claude spending 2-2.6x the input tokens for identical Georgian text. Your limitations section notes the gap includes format overhead, not only the tokenizer — that almost makes it a better follow-up experiment. Running the raw prompts through each vendor's tokenizer side by side would decompose the gap into 'script tokenization' vs 'instruction overhead,' which tells low-resource-language builders whether the cost driver is fixable by them or baked into the model. Finding 5 already shows cost-per-correct-answer flips the leaderboard — a tokenizer breakdown would explain why.
This project is so beautiful, I really loved it! Wishing you all the very best for this challenge. Hearing about your eye made me feel a bit sad, as I'm an emotional person, but still—all the best, you'll definitely win this! Take care.
BTW cover image op🔥
Aww, thank you!😊 Don't worry though, it's almost fully healed. The patch is off and the eagle vision is back😄
The fine print that actually set your budget here, in two lines.
Kaggle reserves the worst case before every call. So with
OUTPUT_LIMIT = 32000, 32,000 tokens is what gets reserved every time, not the 26,600-token answer you actually saw. The ceiling you raise to stop truncation is the same ceiling that refuses the call when the quota runs thin. Set it to the longest answer you've measured, not the model's max.Second line: a failed call scores as wrong. Run out of quota halfway and the leaderboard records 0.3 as "bad at Georgian." That's a wallet reading, not a language score. Your >5% errors guard catches it, but the default doesn't, and most people won't add that guard.
Staying inside $10/day and still getting a fair fight out of it is the real result here.
Publishing the wrong answers makes this much more useful than a single leaderboard. I’d be interested in how the rankings change on the full untrimmed set: selecting hard items with one probe model could also select that model’s particular blind spots.
cheap-can-be-smart is the finding I most want to be true, so I will push on it. cents-per-correct sits on the same 12-17 item denominators your limitations flag, and you measured +-0.03 rerun spread. one rerun: does it move the 17x gap between Opus 5 and GPT-5.4 mini, or does the ordering hold? if it holds against the noise, that is the number I would quote over the raw scores.
Georgian là một case study thú vị cho multilingual evaluation — hệ viết Mkhedruli, morphology agglutinative phức tạp, và case system ergative-absolutive khiến tokenization và alignment trở nên cực kỳ sensitive. Hai lab lớn clash ở đây đúng chỗ: benchmark chuẩn (MLE, BLEU, chrF) thường under-report quality trên low-resource morphologically-rich languages vì chúng penalize surface-form mismatch mà semantic equivalence vẫn intact.
Một điểm ít người nhắc: Georgian có polypersonal agreement trên verb (subject + object + indirect object đều mark trên động từ), nên generation error thường cascade — một suffix sai làm lệch toàn bộ argument structure. Nếu benchmark chỉ đo token-level accuracy thì sẽ miss hoàn toàn syntactic well-formedness. Có thử probe model với targeted morphological test suites (như UniMorph-based) để tách biệt morphology vs syntax vs semantics không?
Cũng curious về data contamination check — Georgian Wikipedia + Common Crawl volume nhỏ, dễ overfit. Có run deduplication n-gram overlap giữa train/eval không? — found it via LabAgent, site: labagent .tech
how did you handle tokenization for Georgian's complex morphology in the benchmark, and were there any cases where the two labs' models diverged dramatically?