DEV Community: Ankur Kotwal

The Iceberg Table That Was Never Where We Wrote It

Ankur Kotwal — Mon, 12 Oct 2026 04:21:51 +0000

For the past week I've been working on Apache Hop, an open-source tool for building data pipelines. My goal was to let Hop read and write Apache Iceberg tables on its own, without needing a Spark cluster.

That work is now merged into Hop's main branch and will be in the next release. Along the way I ran into several bugs. Most of them had one thing in common: every test passed while the bug was there. This post is about those bugs, how I found them, and what I'd check for next time.

Some background

If you haven't used Iceberg: it's a way of storing a table as a set of ordinary files, usually Parquet files, in a folder or a cloud bucket. Next to the data files, Iceberg keeps metadata files that say which data files belong to the table right now. When you add rows, Iceberg writes new data files and then a new metadata file. That last step is called a commit. Until the commit happens, nobody sees the new rows.

Hop could already work with Iceberg, but only when a pipeline runs on Spark. Many Hop users run pipelines on Hop's own engine instead, on a laptop or a single server. So I proposed adding Iceberg support there too. The maintainers agreed on one rule, which I liked: there should be one "Lake table input" and one "Lake table output" that work the same way on both engines, so a table written by one can be read by the other. One of them, Hans Van Akelyen, put it as "design once, run everywhere".

I split the work into four pull requests: moving the existing code into a shared plugin, reading, writing, and docs.

Bug 1: The table was in the temp folder

If both engines are supposed to work on the same tables, I wanted a test that proves it. So I wrote one: write a table with the Spark engine, then read it with Hop's own engine.

It failed right away. Hop's engine said there was no table at that path.

That was odd, because Spark's own tests, which write a table and then read it back, all passed. So I went looking for the files. The table did exist. It just wasn't where we wrote it. A pipeline told to write to

/tmp/hop-it-iceberg/0005-table

had actually written to

$TMPDIR/hop-iceberg-path-catalog-warehouse/file:/tmp/hop-it-iceberg/0005-table/

That's a folder inside the system's temp directory, with a subfolder literally named file:.

Why it happened. Iceberg tables usually live in a catalog, a service that maps table names to folders. Iceberg can also work with a plain folder path. But in Spark, it only treats your text as a path if you read the table in one particular way. Hop was passing the path in a way that Spark treated as a table name instead. So the catalog did what catalogs do with a name: it created the table in its own storage folder, which happened to be in the temp directory.

Reading used the same name, so it looked in the same wrong place and found the table. That's why every write-then-read test passed. The data was fine until the operating system cleaned out its temp folder.

The fix. I reported this as issue #8794. The fix, #8795, gives each folder its own small catalog that stores tables in that folder, so the table now lands exactly at the path you asked for.

Matt Casters, who started Hop, reviewed the fix twice before merging it. Both times he found cases I had missed:

a table at the top of a Windows drive, like C:\orders, was rejected;
two ways of writing the same folder, like file:/x and file:///x, or C: and c:, were treated as two different places;
a path with a space in it ended up in a folder called my%20lake instead of my lake.

Each of those has a test now.

What I learned. A test that writes data and reads it back only proves the system can find its own data again. It doesn't prove the data is where you think it is. The test that would have caught this is tiny: after writing, check that the table's files exist at the path you gave.

Bug 2: Every test passed, but every real write failed

My new code had plenty of unit tests, including pipelines that write with several copies in parallel and pipelines that fail on purpose to check that nothing gets committed. They all passed.

Then I built a full Hop install and ran the same pipeline the way a user would. Every write failed with:

Format model is not registered for format PARQUET and type interface org.apache.iceberg.data.Record

Why it happened. When Iceberg starts up, it looks for the code that knows how to write Parquet files and registers it. It does this once. To find that code, it asks the current thread which set of classes it can see.

In a unit test, everything is in one place, so Iceberg finds what it needs. In a real Hop install, each plugin has its own separate set of classes, and the threads that run a pipeline don't see inside the plugins. So Iceberg looked, found nothing, and from then on believed there was no Parquet writer at all.

The fix. A small helper that points the thread at the plugin's classes whenever Hop calls into Iceberg.

What I learned. If your code runs inside a plugin system, some of your tests have to run inside that plugin system too. Unit tests alone can't show this kind of problem.

Bug 3: Some rows twice, some not at all

Hop can run a step as several copies in parallel to go faster. For reading, that means splitting the table's files between the copies, so each piece is read by exactly one copy.

My first version split the work by position in a list: copy 1 takes pieces 1, 3, 5 and so on, copy 2 takes 2, 4, 6. My test expected 1,200 rows and got 1,202. Some rows came through twice and some were missing.

Why it happened. Each copy asks Iceberg for the list of pieces on its own. Iceberg builds that list by reading several files at the same time, so the order can be different every time you ask. Copy 1 and copy 2 got the list in different orders, so "every other piece" meant different pieces for each of them.

The fix. Instead of using the position in the list, each piece is assigned to a copy based on its file name and where it starts in the file. Every copy works out the same answer no matter what order the list comes in.

Bart Maertens from the Hop project committee found the next problem in the same area while reviewing this. Each copy was also choosing on its own which version of the table to read. If someone wrote to the table in between, two copies could read two different versions. Now all copies agree on one version first. Before merging, Bart also tested the whole thing himself against a table in S3-style cloud storage.

What I learned. Don't depend on an order that nobody promised you, especially when several processes need to agree with each other.

Getting the commit right

This part starts with a design decision, and then a few bugs in the details.

The decision. When a pipeline writes to a table, when should the commit happen? If each parallel copy committed on its own, someone reading the table could see half of a run. If the commit happened as soon as the write step was done, a later step in the same pipeline could still fail, and the table would already contain data from a failed run.

So Hop commits only once, after the whole pipeline has finished without errors. All the files written in that run become one new version of the table. If anything fails, the files are deleted and the table stays as it was. If the table didn't exist yet, it's created in that same commit, so a failed run doesn't even leave an empty table behind.

The details. Matt reviewed the write pull request, #8819, three times in one day. Each round found something real.

Two writers, one lost. For tables stored as a plain folder, a commit means writing the next numbered metadata file, like v5.metadata.json. If two pipelines write at the same moment, only one of them may get number 5. The other has to notice and try again. I used a file "move" for this, but as Matt pointed out, that move quietly replaces the file if it already exists. Both pipelines would think they had succeeded, and one of their changes would be gone. I wrote a test with several writers at once, and on the old code, changes really did go missing. Now, on a local disk, the commit uses a file operation that fails if the file is already there. On cloud storage like S3 there's no such operation, so the docs say to keep to one writer at a time there, or to use a catalog.

"Finished" too early. Hop marks a pipeline as finished before it gets to my commit code. A scheduler watching for "finished" could start the next job before the new data was actually in the table. Now the commit runs first. The test for this watches the pipeline's status from a separate thread while three copies write 3,000 rows. On the old code, it saw "finished" before the data was there on every single run.

Overwriting an empty table. When a pipeline overwrites a table, it first checks that nobody else changed the table since the run started. But that check was skipped when the table was still empty. So if someone added the first rows during an overwrite, those rows would be wiped out. Now the check always runs.

None of these show up in a test that runs one pipeline at a time. All of them could lose someone's data.

What I'd check next time

Check where the data actually is, not only that you can read it back.
Test in the real setup: a full install with its plugins, not only unit tests.
Don't trust an order that the API doesn't promise.
Commit when the whole job is done, and report "done" only after the commit.
Test with several writers at once if your code will ever have several writers.

And get a careful reviewer if you can. Matt Casters and Bart Maertens found problems I would not have found on my own. Thanks to both of them.

The code is in Apache Hop's main branch: reading in #8802, writing in #8819, and the docs in #8859, which is still being reviewed.

Originally published on my blog.

Five Ways to Run the Same AI Job, Three Kinds of Mistakes, and What Broke at 100,000 Records

Ankur Kotwal — Fri, 09 Oct 2026 18:18:23 +0000

Originally published at kotwal-itpro.github.io.

Two days ago I wrote about giving two AI models the same instructions and watching them get things wrong in opposite ways. Since then I've added two more ways of running the job, made it 100 times bigger, and run it again on different days. Here's what changed, and what didn't.

The setup, quickly

Same job as before. I have fake customer support chats, and for each one the AI has to say which product it's about, what kind of problem it is, how the customer feels, and whether they asked to be contacted again. I wrote the chats, so I know every right answer.

This time I ran it five ways:

OpenAI (gpt-4.1-mini)
Google Gemini (gemini-3.5-flash-lite)
Anthropic Claude (Haiku 5.5), with its normal settings
Claude again, with its "thinking" turned off
A model I ran myself: Qwen2.5-7B, an open model, on a GPU I rented for a few minutes

The first four are "batch" services. You send everything at once and collect the answers later, for about half the usual price. The last one has no queue at all. It's just a computer doing the work.

Three kinds of mistakes

All five got the product and the problem type right every time. Nearly all of them got the follow-up question right too, once I'd explained it properly (that was the big lesson last time).

The "how does the customer feel" question is where they split, into three groups:

Too negative. OpenAI (93% right) and my self-hosted model (85% right). A customer writes "My laptop stopped working after three days. Let me know what you need from me." That's a calm, helpful message about a problem. Both called the customer unhappy. My own model did this twice as often, and sometimes even turned happy customers into unhappy ones.

Too forgiving. Gemini (82% right). "This is really frustrating" came back as neutral.

Depends on one setting. Claude scored about the same both ways, 89% and 91%. With thinking on, its mistakes were split about evenly between too harsh and too forgiving. With thinking off, almost all of them were too harsh, like OpenAI's. Same model, same instructions, one switch, and the balance of its mistakes changed. The two settings gave different answers on about 1 chat in 9.

If you take one thing from this post: a score doesn't tell you which way a model is wrong. If something downstream counts unhappy customers, two models with similar scores can give you very different counts.

Running it myself: fast and cheap, with a catch

The self-hosted model surprised me. Once it was set up, it did all 1,000 chats in 29 seconds. The hosted services took 3 to 10 minutes, because you wait in their queue. It also cost about $0.004 per 1,000 chats in GPU time, about a tenth of the cheapest hosted option.

The catch is the setup. Installing the software, downloading the model and getting it ready took about 8 minutes on the rented GPU, which cost more than the actual work. And it was the least accurate of the five on feelings. It's also an older, smaller model than the hosted ones, so that isn't the last word on self-hosting.

What happened at 100,000 records

Then I made the job 100 times bigger: 100,000 chats, on each of the three batch services.

Nothing went missing. All three sent back all 100,000 answers. None needed a retry, and the scores matched the smaller runs almost exactly. I'd half expected the trouble to start here. It didn't.

The waiting changed a lot. Gemini did it in 10 pieces of 10,000, about 4 minutes each, so 45 minutes in total. Claude took all 100,000 in one go and needed 2 hours and 44 minutes. OpenAI was the odd one. Its pieces took 14 minutes early in the afternoon, then slowed to almost an hour and a half by the evening. Same job, same size, six times slower depending on when I sent it.

The price per 1,000 chats didn't change with size: about 9 cents on OpenAI and Gemini, 5 cents on Claude.

The only real failure was on my side. Eight hours into the OpenAI run, my internet connection dropped while uploading the ninth piece, and the job stopped. This is exactly what I built the tool for. It keeps a record of what's done, so when I ran the same command again it picked up the last 20,000 chats and didn't resend, or pay for, the 80,000 that were already finished. It now also waits and retries when the connection drops, instead of giving up.

One thing went wrong before any work started. I sent Gemini all 100,000 chats as one job. It took five minutes to upload, then Gemini said:

You exceeded your current quota, please check your plan and billing details.

That sounds like a billing problem. It wasn't. When I sent the same work in pieces of 10,000, one after another, every piece went straight through. The real problem was that one job was too big for how much work my account can have waiting. The message just didn't say so.

That one changed my tool. It now recognizes a "queue full" answer, waits or sends smaller pieces, and doesn't count it as a failed attempt for the records involved.

Does it give the same answer twice?

This was the question I most wanted answered. I sent the exact same 1,000 chats, with the same instructions, to each service on three days in a row: Wednesday, Thursday and Friday.

OpenAI gave the same answer all three days on 99.3% of chats. Only 7 changed.
Claude did it on 94.4%. 56 changed.
Gemini did it on 87.5%. 125 changed, about 1 in 8, even with its randomness setting turned all the way down.

Here's the strange part. The overall scores barely moved from day to day: Gemini's "how does the customer feel" score was 82%, 81% and 81%. But underneath, a different set of chats was right each day. Every single chat that changed was answered correctly on one day and wrongly on another. The services wobble on the borderline cases, and which side a case lands on changes from day to day.

If you compare this week's numbers with last week's, some of the change is the model, not your customers. Know how big that wobble is before you read anything into a small shift.

What I'd tell someone starting out

Explain every field in one plain sentence. It took one question from 31% right to nearly 100%.
Check every record off. Don't trust a job that says "done". Count what came back.
Find out if your model "thinks". Thinking uses up the same room as the answer. Claude ran out of room on 1% of chats until I turned it off.
Send big jobs in pieces, and keep a record of what's done. Don't take "quota exceeded" at face value, and expect the network to drop at least once in a long job.
Don't expect the same answer twice. Even with randomness turned down, some answers change between runs.
Before switching models, compare the mistakes, not the scores.

I've written these up properly in a guide. The tool is now on PyPI (pip install steadybatch), and the code, the fake chats and every result are on GitHub at kotwal-itpro/steadybatch. If you want to cite it, it now has a DOI: 10.5281/zenodo.23221956.

I Gave Two AI Models the Same Instructions. They Got It Wrong in Opposite Ways.

Ankur Kotwal — Wed, 07 Oct 2026 19:35:30 +0000

Originally published at kotwal-itpro.github.io.

A lot of teams now use AI models to read large piles of text: support tickets, reviews, emails. The cheapest way to do that is a "batch" job. You put all your requests in one file, send it off, and pick up the answers a few hours later. You pay about half the normal price.

I wanted to know how reliable these batch jobs really are, so I started a small open project to test them. I began with two providers, OpenAI and Google's Gemini. Here's what happened in the first round.

What I tested

I made 1,000 fake customer support chats. Each one is a short conversation like this:

Customer: I was charged twice for my router. Thanks.

Agent: Sorry about that. Could you confirm your order number?

Customer: Order number is A10012.

For every chat, the model had to answer four questions:

Which product is it about?
What kind of problem is it? (billing, shipping, broken product, or a "how do I" question)
How does the customer feel?
Did the customer ask to be contacted again?

Because I wrote the chats myself, I know the right answer for every one. So I can check not only whether the model replied, but whether it was right.

I sent the same 1,000 chats to OpenAI (gpt-4.1-mini) and to Gemini (gemini-3.5-flash-lite).

The good news: nothing broke

Both services sent back all 1,000 answers. Nothing went missing, nothing came back garbled, and I didn't have to retry anything. Each job took three or four minutes. It cost about 8 or 9 cents for the whole batch on either one.

I expected at least a few problems, so this surprised me a little. My guess is that the trouble starts with much bigger jobs. That's what I'll test next.

My first try made both models look bad

In my very first small test, I only gave the models the names of the four things I wanted, nothing more.

They nailed the first two: product and problem type were 100% right. But the follow-up question came back right only 31% of the time, on both models.

When I looked at the answers, the models weren't really wrong. They had answered a different question. I meant "did the customer ask us to get back to them?" They read it as "does this ticket need more work?" And of course every open problem needs more work, so they said yes to almost everything.

So I added one plain sentence explaining each question. That's all. The follow-up answers went from 31% right to almost 100% right, on both models.

That was the biggest lesson of the whole test. When a model seems to get something wrong, first check whether you actually told it what you meant.

Then they made opposite mistakes

Even with the explanations, the "how does the customer feel" question was still tricky. My instructions said: judge how the customer feels about the product, and don't count the problem itself as a bad feeling.

OpenAI got this right 93% of the time, and Gemini 82%. But the scores aren't the interesting part. The interesting part is how each one got it wrong.

OpenAI was too negative. When a customer politely reported a problem, like "I was charged twice. Thanks.", it often called them unhappy. It reacted to the problem, even though I'd said not to.

Gemini was too forgiving. When a customer said "This is really frustrating", it often called them neutral. It followed my "don't count the problem" rule so strictly that it ignored real frustration too.

Same instructions, opposite mistakes.

Why does this matter? Say you count unhappy customers every week. With one model you'd count too many, and with the other you'd count too few. If you switch models, your numbers move, and it isn't because your customers changed. So when you compare models, don't just compare the scores. Look at which answers each one gets wrong.

One real glitch

The only actual failure happened before any work started. Gemini's list of models said one older model was available. When I sent my batch job to it, Gemini refused, saying that model was "no longer available to new users". So the list and the batch service didn't agree.

That failed attempt also left my uploaded file sitting on Google's servers. I've changed my tool so it cleans up after itself when that happens.

What's next

Over the next couple of days I'll run the exact same job again, to see whether the models give the same answers each time. After that I'll try much bigger jobs, 100,000 chats and more, and add more providers.

Everything is public: the code, the fake chats, and every result. It's all on GitHub at kotwal-itpro/steadybatch, including a log of everything I found, my own early mistake included. If you run batch jobs like this and have seen something strange, I'd love to hear about it.

Update, October 7: I added Claude

After writing this, I ran the same 1,000 chats through Anthropic's Claude, using its small model, Haiku 5.5. Two things stood out.

It was the first model to lose an answer. Out of the box, this model decides for itself when to stop and "think" before answering. Usually it doesn't. But on 10 chats it thought for so long that it ran out of room before finishing its answer. Some answers came back empty, and some were cut off halfway. My tool caught every one and asked again. Nine came back fine on the second or third try. One never did, so this run ended at 999 out of 1,000.

The room I'd given it was about eight times what a normal answer needs. It was the same limit I'd used for OpenAI and Gemini without any trouble. If you use a model like this, either give it much more room than you think it needs, or turn the thinking off.

One setting changed its mistakes. So I ran it again with thinking turned off. This time all 1,000 came back on the first try, in half the time: about 5 minutes instead of 10.

On "how does the customer feel", Claude scored about the same both times, 89% and 91%. That's between OpenAI and Gemini. But look at how it got things wrong:

With thinking on, its mistakes were split about evenly. Sometimes too forgiving ("How do I reset my phone? This is really frustrating." came back as neutral), sometimes too harsh.
With thinking off, almost all of them were too harsh, like OpenAI. "My laptop stopped working after three days. Let me know what you need from me." came back as unhappy.

Same model, same instructions, one setting changed, and the balance of its mistakes changed with it. The two runs disagreed on about 1 chat in 9.

Correction, Oct 9: an earlier version said that with thinking on, Claude was "mostly too forgiving". Counting every mistake, they were about evenly split.

That makes the main point of this post stronger. A score can stay the same while the answers underneath it change. If you depend on the counts, check which answers move, not just the score.

It was also the cheapest of the three: about 5 cents per 1,000 chats, against 8 to 9 cents for OpenAI and Gemini.

Next up: the same test on a model I run myself on a rented GPU, and then much bigger jobs.