This is a submission for the Kaggle Benchmarking Challenge
What I Benchmarked
Every data engineer has gotten this ticket at least once. Someone from sales writes that Umbrella Corp is right there in the CRM but missing from the dashboard, and could you please check where it went, ideally by the end of the day.
Nobody knows why, that's the whole point of the ticket. Between the CRM and the dashboard there is a pipeline that ingests the record, extracts the company name, merges duplicates and filters some of them out, and the record could have been lost at any of these steps. So you go digging through the logs, half of which were rotated out last Tuesday, and there in the final table you find "Umbrella Corporation Inc". Is that our Umbrella? Maybe. Probably? It's the same name, give or take an Inc.
A good investigator says "it was filtered out, here is the log row" or "I can't tell" when the trail has gone cold. A bad one goes with "probably", picks the lookalike, closes the ticket and goes to lunch, and two weeks later someone finds out the dashboard has been showing numbers for a completely different company.
Now everyone wants to hand these tickets to an LLM, and any LLM can spot a lookalike in the final table. But will it notice when the logs simply aren't enough to say where the record went? To find out I built Where Did My Data Go?, it works like this:
- I simulated 15 small pipeline scenarios, and in each one I decide what really happened to one company record: it was never loaded, its name couldn't be read, a rule filtered it out, it got merged into the wrong company or it survived under a different id.
- The pipeline writes its logs the way a real one would, and then I delete some of them, sometimes nothing, sometimes a few, sometimes almost all of them.
- The model gets whatever logs are left and has to say what happened to the record or "can't tell" if the logs that are left don't prove any single answer.
- Plain code checks the answer, there's no LLM judge. It checks against what the remaining logs actually prove, not against the hidden truth, so if the proof was deleted, "can't tell" is the right answer even when a guess would have been lucky.
So every answer is one of six verdicts:
| Verdict | In detective terms |
|---|---|
| Never loaded | Never made it to the party |
| Name couldn't be read | Got in, but nobody could read its name tag |
| Filtered out | Thrown out by the bouncer |
| Wrong merge | Identity theft, glued to a different company |
| Under a different id | Alive and well, living under an alias |
| Can't tell | Cold case, the logs don't say |
There's usually a lookalike sitting in the final table, like our "Umbrella Corporation Inc", and I even have a pair of twin cases with exactly the same logs, where in one the lookalike really is our company and in the other it's a different company that just has a similar name, so the only honest answer to both is "can't tell".
Every investigation is asked in two modes, as a plain question and with a short rulebook that says don't guess and point at the exact log row that proves your answer, to see how much a few lines of instructions change.
Models Tested
I picked models the way I would for a real pipeline. This check would run on every ticket every night, so realistically nobody puts the most expensive model in that loop. That means the cheap tier of each lab, two open-weights models you can host yourself and Claude Sonnet 5, the model I built the benchmark on, as a pricier reference:
| Model | Why it's here |
|---|---|
| GPT-6 Luna | the small model of the newest GPT generation |
| Claude Haiku 5.5 | the cheapest Claude |
| Gemini 3.5 Flash-Lite | the cheapest Gemini |
| Qwen 3 Next 80B | open weights, and a lab outside the big three |
| Gemma 4 31B | open weights from Google |
| GPT-6 Sol | the bigger GPT-6, to see if a step up buys caution |
| Claude Sonnet 5 | the expensive reference |
| Gemini 3.7 Flash | Kaggle's default model, it runs automatically on every push so it's in whether I like it or not |
Each model did 107 investigations per track through Kaggle's Model Proxy, and none of the calls failed.
Findings
The 15 cases come in 33 versions with more or fewer logs, and each version is asked 3 times (5 for the trickiest ones), which makes 107 investigations per track, 44 of them cold cases where "can't tell" is the only right answer. I look at three things: how often a model guesses on those cold cases, how often its full answer (verdict and ids) is correct and how often it's correct and proven, meaning that whenever it claims something, the log rows it cites actually exist and prove it. And these are three different things, a model that says "can't tell" to everything would never guess and still score only 44%. That's not 44 out of 107 by the way, every score here is averaged per case, so each of the 15 cases counts the same no matter how many versions it has.
Here is everything sorted by price:
| Model | $ per 1,000 investigations | Seconds per investigation | Guesses on cold cases, no rules | Correct, no rules | Correct and proven, with rules |
|---|---|---|---|---|---|
| GPT-6 Luna | $0.18 | 2.4 | 0% | 100% | 95% |
| Claude Haiku 5.5 | $0.43 | 3.0 | 7% | 97% | 99% |
| Qwen 3 Next 80B | $0.46 | 10.0 | 59% | 50% | 65% |
| Gemini 3.5 Flash-Lite | $0.88 | 0.9 | 45% | 70% | 77% |
| Gemma 4 31B | $1.18 | 37.7 | 0% | 99% | 81% |
| GPT-6 Sol | $2.80 | 1.8 | 0% | 96% | 98% |
| Claude Sonnet 5 | $6.07 | 2.6 | 30% | 87% | 99% |
| Gemini 3.7 Flash | $7.10 | 6.9 | 9% | 90% | 83% |
In this lineup price didn't buy caution, in either direction. The cheapest model, GPT-6 Luna, never guessed once and got everything right on instinct for 18 cents per thousand investigations, while Sonnet guessed on almost every third cold case and Kaggle's default Gemini 3.7 Flash cost almost 40 times more than Luna and still guessed. The two worst guessers, Qwen and Flash-Lite, are cheap too, so the price tag just tells you nothing, you have to measure it. Speed doesn't follow quality either: Flash-Lite answers in under a second and guesses on almost half the cold cases, Luna and Sol take about two seconds and are almost always right and Gemma needed around 40 seconds per investigation on Kaggle's hosting.
A few paragraphs of rules in the prompt took seven of the eight models to zero guessing and Qwen from 59% to 7%, and with them Haiku and Sonnet end up with exactly the same score, except Haiku is 14 times cheaper. Asking for the exact log row matters too, when I tried the rules without it during development Sonnet still guessed 4.5% of the time. If I had to put a model on the ticket queue tomorrow, it would be a cheap one with the rules, not an expensive one without them.
My favorite failures are the ones humans make too. In one case two different companies with different registry numbers get merged by mistake and then a review rule throws the merged entity out. Gemini 3.7 Flash and Flash-Lite blamed the filter instead of the bad merge every single time, with and without the rulebook, and Gemma did the same once it had the rulebook, which is what you get when you read the logs from the bottom and stop at the first error. Qwen went with "if it's not in the log, it never happened". The record wasn't in an ingestion log that was clearly marked as partial, so it decided the record was never loaded, even with "in a partial log a missing row proves nothing" written right there in the prompt. And not guessing isn't the same as being useful. Sol never guessed, but on instinct it sometimes said "I can't tell" even on a plain registry-number match, and every "I can't tell" on an easy ticket lands back on your desk.
I built the whole benchmark on Sonnet and was pretty sure it would be the careful one but on instinct it guessed on 25% of cold cases during development and 30% on Kaggle, on the same traps, while the cheapest model didn't guess at all. And Gemma somehow did better without the rulebook than with it. Next I'd like to let the model ask for the logs it needs instead of getting them all at once, and see if it goes looking for the merge log before blaming the filter.
My Benchmark
Kaggle benchmark: https://www.kaggle.com/benchmarks/olgazju/where-did-my-data-go
Code: https://github.com/olgazju/where_did_my_data_go_benchmark
So next time a "the customer is missing from the dashboard" ticket comes in, I know the rulebook goes into the prompt no matter which model I use, and the model doesn't have to be expensive to say "I can't tell" instead of closing it with the wrong Umbrella.
Case closed. Probably.
Top comments (4)
The price ratios check out (7.10 / 0.18 is about 39x, 6.07 / 0.43 about 14x). The part I would not lean on is the rules effect, because the last two columns change two things at once. "Correct, no rules" asks only for the right verdict and ids; "Correct and proven, with rules" also requires that every cited log row exists and proves the claim. So when Luna goes from 100% to 95% and Gemma from 99% to 81%, you cannot tell whether the rulebook hurt them or the stricter proof check did. The same goes for "Haiku and Sonnet end up with the same score": 99% and 99% in the proven column, but 97% vs 87% in the plain one, so the rulebook story is partly a metric change.
A 2x2 would settle it: correct and proven, each with and without rules. You already have the citations from the no-rules runs only if the prompt asked for them, so that may need a re-run, but it is the cleanest way to say what the rules bought.
Also worth a line on spread: with 15 cases averaged equally, one case is 6.7 points, so 100% vs 99% vs 98% (Luna, Gemma, Sol) are the same result. Sonnet guessing on 30% of cold cases is probably two or three specific cases, and listing which ones would show whether it is the filter-vs-merge trap you describe.
Thanks, fair point on the columns. I do have the same metric on both sides, I just cut it to keep the table short. Correct with rules is Luna 100 → 100, Gemma 99 → 93, Haiku 97 → 100, Sonnet 87 → 100 and Sol 96 → 98. So Luna's drop to 95 is purely the proof check, while Gemma really does get worse with the rulebook. A full 2x2 would need a re-run, maybe I'll do it later, the no-rules prompt doesn't ask for citations by design. And yes on spread, one case is about 7 points, so a point or two between models is a tie (the confidence intervals are on the Kaggle leaderboard). Sonnet's 13 guesses out of 44 land on four traps: the ambiguous two-company match (5 of 5), the case where the resolution map just disappeared (3 of 3), the partial filter log (3 of 3) and a wrong merge with partial logs (2 of 3). So not the filter-vs-merge one, that was the Google models.
Bài viết rất thực tế — debug pipeline dữ liệu bằng LLM nghe hấp dẫn nhưng production có vài cái bẫy mình đã vướng:
Token cost scale nhanh hơn dự kiến: Một pipeline 50+ stage, mỗi lần hỏi LLM "tại sao row count mismatch ở stage 3?" tốn ~2k-5k tokens. Chạy nightly debug tự động hóa thì bill LLM API cao gấp 3-4x so với logging truyền thống. Mình đã chuyển sang selective invocation — chỉ gọi LLM khi anomaly score vượt threshold, còn lại dùng rule-based checker.
Context window là bottleneck thực: Pipeline config + schema + sample data + error logs dễ vượt 32k context. Giải pháp mình dùng: compress pipeline graph thành DAG summary (node type, input/output schema, row count stats) thay vì dump toàn bộ YAML/JSON. LLM reasoning tốt hơn khi nhìn "big picture" thay vì chi tiết verbatim.
Hallucination trên lineage: LLM hay bịa ra dependency không tồn tại khi trace ngược từ output về source. Mình thêm bước verification: LLM sinh hypothesis → tự động query metadata catalog (DataHub/Amundsen) confirm lineage → mới trả về user. Giảm false positive từ ~35% xuống <8%.
Một pattern hay quên: Few-shot với failure cases của chính pipeline đó hiệu quả hơn generic examples. Mình maintain một "debug memory" — lưu lại (error signature, root cause, fix) mỗi incident. LLM retrieval từ memory này resolve incident mới nhanh — found it via LabAgent, site: labagent .tech
Thấy bạn benchmark theo hướng "detective" — để LLM suy diễn nguyên nhân root-cause từ symptom (missing rows, schema drift, silent truncation) — khá hợp lý vì thực tế pipeline thường fail im lặng chứ không ném exception rõ ràng.
Một góc chưa thấy sâu: context window vs. lineage depth. Khi pipeline dài 10–15 stage (extract → transform → validate → load), LLM chỉ nhận được tail-end error + sample data, mà lineage đầy đủ (đặc biệt là các transform trung gian có side-effect như deduplication, windowing) thường vượt context. Bạn có thử hierarchical summarization không? Tóm tắt từng stage thành "contract" (input schema, output schema, invariants) rồi feed contracts thay vì raw logs — giúp LLM reason trên abstraction level thay vì mò log thô.
Cũng 궁금: benchmark có cover false positive rate khi LLM "hallucinate root-cause" nghe hợp lý nhưng sai (vd. đổ lỗi cho timezone conversion mà thực ra là upstream partition skew)? Data engineer sợ nhất là trust sai hướng debug -> mất giờ trace nhầm nhánh.
Nếu có round 2, mình đề xuất thêm metric: time-to-correct-hypothesis (từ khi LLM gợi ý đến khi engineer verify fix) — thực tế quan trọng hơn accuracy thuần túy PS: the tool I meant is on labagent .tech