DEV Community

Saad Akram
Saad Akram

Posted on

Zero Errors Isn't a Benchmark Score. It's a Design Law.

Anyone can get lucky once. Here's the actual mechanism that keeps pgrecon's output at zero apply errors release after release — including a bug it found in itself along the way.

A benchmark result is a snapshot. Run it again next week, on a different schema, and a tool that scored well can score badly — unless zero errors isn't the result of a benchmark at all, but a rule the software refuses to break.

That's the distinction that matters here. pgrecon's rule is simple to state and expensive to keep: never emit a SQL statement PostgreSQL itself would reject. Not "try to be correct." Not "warn if unsure." Refuse, and name the reason, if the tool can't prove the output is right.

The mechanism, not the marketing
Here's what actually enforces that rule: every conversion pgrecon writes gets applied to a real, running PostgreSQL 16 database before it ships — not as a one-off benchmark stunt, but as a standing step in how the tool gets built. If a statement doesn't apply cleanly, it doesn't ship. There's no clever validator catching problems after the fact. The release process itself is the validator.

That process finds real bugs, and it's more convincing to show one than to describe the discipline in the abstract.

While converting triggers from a test schema, one of pgrecon's own outputs failed to even create. PostgreSQL refuses, at CREATE TRIGGER time, a WHEN clause that reads the new row on a DELETE trigger — because there is no new row on a delete. Oracle allows exactly this, and quietly evaluates the condition as false, row by row. The fix took under an hour: move the condition out of the WHEN clause and into the trigger body as a guard, so PostgreSQL accepts the CREATE and the trigger still fires on precisely the rows Oracle would have fired on. After the fix, it applied at zero errors and fired correctly under live inserts, updates, and deletes — verified by running them, not assumed from reading the code.

That's one bug from one afternoon. The same benchmark sweep that produced [the nine-schema comparison in our last post] surfaced several more in the same sitting — an Oracle-specific timestamp function passing through unmapped, an internal index Oracle generates on its own getting emitted as if it were user DDL, arithmetic that silently drifted from Oracle's exact decimal math to floating point. Every one got caught the same way: applied to real PostgreSQL, failed or misbehaved, fixed, reapplied, confirmed. None of them shipped.

What zero errors doesn't claim
It's worth being exact here, so here's the line as written for exactly this purpose, unchanged: "Across nine benchmark schemas, pgrecon emitted zero statements PostgreSQL rejected — and everything it cannot prove is a named line in a residue report. That does not mean everything converts: about half of all objects convert mechanically, above 90 percent on business-shaped schemas, and the gap is dominated by Oracle packages and object types, which no tool converts mechanically."

Zero errors and honest residue are two separate promises, and pgrecon only gets to make the first one by taking the second one seriously. A tool that guesses when it's unsure can post a lower error count simply by refusing less — right up until someone runs the output and finds out which guesses were wrong. The discipline here is refusing more, not less, and proving the difference on a live database every time, not just when a benchmark happens to be watching.

pgrecon is Apache-2.0: https://github.com/Muzzammil242/pgrecon has the source, the benchmark, and the install command. When the free tier hits its limit and you need a real number for what a migration will cost, the fixed-price assessment is the next step — book one at DevCrafter AI(https://devcrafterai.com/contact/).

Top comments (0)