I Round-Tripped 2,249 Test Fixtures Through sqlfluff's Auto-Fixer. Eight Came Back Unparsable.

Auto-fix is the most dangerous feature a linter has. Every other part of the tool only reads your code. --fix writes to it, usually across a whole repository, usually in CI, usually without anyone reading the diff.

So it is worth asking a blunt question: does the fixer ever produce output the tool itself cannot read back?

There is a cheap way to find out, and the input you need is already sitting in the repository.

The invariant

If a fixture parses cleanly, its fix output must parse cleanly too.

That is the whole idea. One line. No fuzzer, no grammar model, no corpus to build.

It works because a linter's test fixtures are the best adversarial input anyone will ever hand you for free. They were written by the maintainers, deliberately, to sit on the edges of the grammar: every operator, every dialect quirk, every syntax someone once filed a bug about. Nobody could commission a better corpus, and it ships with the source.

The fixer, meanwhile, has usually never been pointed at all of it at once. Fixtures get used to test the parser. Running them through the fixer is a different question, and mostly nobody asked it.

The run

sqlfluff, at a54076a. 2,249 fixtures, 28 dialects, roughly an hour single-process.

For each fixture: parse it. If it parses, fix it, then parse the output. Classify the result:

• CORRUPTION — parsed before, does not parse after.
• UNSTABLE — fixing twice does not converge.
• FIX_CRASH — the fixer threw.

Raw result: 8 CORRUPTION, 31 UNSTABLE. Thirty-nine hits.

Thirty-seven of them were not bugs.

Triage is the entire job

Finding 39 things is easy. Knowing which 2 are real is the skill, and getting it wrong in public is how you teach a maintainer to ignore you.

UNSTABLE is almost always noise

All 31, in this run. sqlfluff's CLI loops the fixer until output stabilises, so a single pass that has not yet converged is expected behaviour, not a defect. Reporting those would have been 31 false positives in one issue.

Nine of my first ten findings on an earlier run were this same noise. The rule I now apply without exception: UNSTABLE is a hypothesis, never a finding.

Measure the before state, not just the after

ALTER USER 'jeffrey'@'localhost' fails to parse after the fix. It also fails to parse before the fix. That is a gap in the parser, not corruption in the fixer, and the two look identical if you only check the output.

I nearly folded it into a corruption report. It would have weakened the whole issue: one soft claim is enough to make a maintainer discount the hard ones sitting next to it.

The eight collapse into two root causes

Not eight bugs. Two:

root cause	fixtures
Token fusion at a whitespace boundary	ansi/arithmetic_a, sqlite/arithmetric_a, oracle/multiset_operators
RF06 vs 'user'@'host' account specs	mysql/create_user, mysql/grant, mariadb/create_user, mariadb/grant, mariadb/create_view_if_not_exists

Five fixtures pointing at one rule is not five bugs. Filing it as five would have been noise wearing the costume of thoroughness.

Bug one: a whitespace fix that deletes your code

sqlfluff removes whitespace it considers redundant. Sometimes the characters on either side of that whitespace are tokens that fuse into a different token when they touch.

-- in, parses fine
SELECT 1 * - - 5 AS a, 99 AS b FROM t

-- out
SELECT 99 AS b, 1 * --5 AS a FROM t

- - became --, which in SQL starts a line comment. Everything after it is now commented out. Note also that the fixer reordered the select targets, pushing the damaged column to the end, where it takes the rest of the line with it.

That is silent data loss produced by a formatting rule.

Two details from the corpus made the case much stronger than the -- example alone.

It is not only comments. SQLite fuses 4 | ~ ~ ~4 into 4 | ~~~4. A different token, no comment involved. The failure mode is the general one: any pair of adjacent tokens that lexes differently when joined.

It is not only operators. Oracle fuses two keywords: MULTISET EXCEPT becomes MULTISETEXCEPT. And the rule responsible is LT02, an indentation rule. A purely cosmetic concern, destroying meaning.

That last one is the argument for guarding at the token layer rather than patching rule by rule. A rule that only ever adjusts indentation should not have to know what a token is. The check belongs where fixes are applied.

Filed as sqlfluff#8415.

Bug two: the fixer strips quotes that were load-bearing

RF06 removes unnecessary quotes from identifiers. In MySQL and MariaDB, 'jeffrey'@'localhost' is not two quoted identifiers. It is one account specification, and the quotes are part of it.

-- in
CREATE USER 'jeffrey'@'localhost'

-- out, which sqlfluff itself cannot parse
CREATE USER jeffrey@localhost

Default config, no .sqlfluff needed. It also hits GRANT ... TO, DROP USER, and DEFINER = clauses.

Filed as sqlfluff#8462.

The check that found nothing

A clean re-parse does not mean the output is safe. 1 * --5 parses perfectly. It just means something else now. So I added a second check, for output that parses but has lost meaning.

My first design was wrong. Compare all code tokens before and after, and it fires on every legitimate fix: added commas, capitalised keywords, removed parentheses. The signal drowns under correct behaviour.

The invariant that works is much narrower:

A fix never introduces a comment.

Nothing sqlfluff legitimately does creates a comment, so a new one after fixing means source text was swallowed. I validated it in both directions before trusting it: it fires on the 1 * - - 5 case, and stays silent on three benign fixes.

Result across the whole corpus: zero.

That is an honest negative, and I am including it on purpose. The check covers a region the parse test cannot see, and that region turned out to be clean. A methodology piece that only reports the checks that hit is a sales pitch, not a method.

Two things that cost me time

Extracted statements lie. Pull a statement out of a fixture to build a minimal repro and it may not parse standalone even though the whole file does. My first CREATE VIEW repro was a bad extraction, not a bug. Run the real fixture, then minimise, then re-verify against the real fixture again.

Redirect, do not pipe. Piping an hour-long run through tail buffers until EOF. You watch nothing happen for sixty minutes. Write to a file.

This generalises

Nothing above is specific to SQL. The requirements are only:

1. The tool has an auto-fix mode.
2. The tool can validate its own output.
3. The project ships a fixture corpus.

That describes ruff, prettier, rustfmt, black, ESLint, gofmt, and most of the formatting layer our industry now runs unattended in CI.

The invariant is one line. The corpus is already in the repository. The expensive part is not the scan. It is the discipline to throw away 37 of your 39 findings before you tell anyone about the other 2.

---

Both issues above are open against sqlfluff, a SQL linter with roughly 9k stars that a great many teams run with --fix in CI.