DynamoDB accepts a narrow slice of PartiQL and rejects everything else at request time. GROUP BY? ValidationException. A statement-level LIMIT? ValidationException. The * operator, CAST, a subquery? All of them parse fine in your head, travel over the wire, and die on the server. The only place that knowledge lived was the AWS documentation and the error messages, which meant every editor for DynamoDB — including ours, for a while — would happily let you compose a statement the engine was guaranteed to refuse.
We wanted the refusal to happen in the editor, on the keystroke, with a red squiggle on the exact clause and a one-click fix where a rewrite exists. That editor need turned into a hand-written lexer and CST parser for DynamoDB's PartiQL dialect, and this week we open-sourced it: dynamodb-partiql-parser, pure TypeScript, zero dependencies, MIT, with the CodeMirror wiring published separately as codemirror-lang-partiql. This post is why it's hand-written, what the first linter got wrong, and the two bugs that only showed up when someone pasted garbage.
Regex was fine, until it wasn't
The first PartiQL linter in DynoTable was about 650 lines of regex and token scanning, and it was genuinely useful: nineteen distinct checks, quick fixes for the common traps (IN (...) to [...], LIKE to contains(), IS NULL to attribute_not_exists()). It shipped, it caught real mistakes, users stopped filing "why does my query fail" tickets for the cases it covered.
But a regex linter knows patterns, not structure. It couldn't see that * in SELECT price * quantity is arithmetic DynamoDB rejects, because * also means "all columns" and telling those apart requires actually parsing. Its diagnostic ranges were approximations — close enough to point at a line, too coarse to drive a quick fix that splices text at exact offsets. And every new check made the pile more fragile, because each regex had to defend against every other regex's assumptions.
The fix for "the linter needs structure" is a parser. The question was which one.
Nobody had built one
For the Workbench's real-SQL side we had already been through this: an off-the-shelf SQL parser that lied to us, replaced by sql-parser-cst, which carries a source range on every node and preserves quoted-versus-unquoted identifiers. That experience set the bar for what the PartiQL side needed — a lossless concrete syntax tree, not a lossy AST.
But PartiQL is not SQL where it counts for a parser. DynamoDB's dialect writes IN lists with brackets (WHERE OrderID IN [100, 300, 234]), has bag literals (<<'a', 'b'>>), map literals with quoted keys ({'rating': 5}), a MISSING literal, document paths with list indexes (Devices.FireStick.DateWatched[0]), and RETURNING ALL OLD * — none of which a SQL grammar knows. In the other direction it lacks half of what a SQL grammar insists on. At the time, the parsers on npm were WebAssembly builds of AWS's Rust implementation for generic PartiQL, with no notion of what DynamoDB specifically rejects.
So we wrote one: a small lexer and a recursive-descent parser, modeled on the shape sql-parser-cst taught us to want. Every node carries its byte range. The whole thing has zero runtime dependencies — a property the CI now asserts, because it's what makes the parser embeddable anywhere, including the browser, including your project.
The grammar was the easy half. A linter's parser spends its whole life parsing broken code. Mid-keystroke, half a statement, a typo in the third clause. Stopping at the first error would make the editor useless, so the parser is error-tolerant: it records a diagnostic, resynchronizes, and keeps going, so the fourth clause still lints while the second is incomplete.
Swapping the engine without breaking the plane
By the time the parser was ready, the regex linter's four functions were load-bearing across the editor — including the one that decides whether a statement is safe to auto-execute. Silently changing that behavior shows up as "the editor won't run my query," which is the kind of bug users don't report so much as leave over.
So the swap was a strangler: the old linter was renamed, frozen, and kept in the tree. The new parser-driven linter re-exported the exact same four functions. And a parity corpus ran every fixture through both linters and pinned the outputs against each other — every diagnostic the regex version produced, the parser version had to produce too, before it was allowed to produce more. The old linter is still there today, frozen, as executable documentation of what the swap promised.
The bugs that only garbage finds
Two failures never appeared in any real query and both would have taken the editor down.
A CodeMirror linter runs synchronously on the document, on every change, with no error sink above it. One uncaught exception doesn't fail a lint — it white-screens the editor. And a recursive-descent parser has a natural uncaught exception built in: the call stack. Paste [[[[[[… a few thousand brackets deep, or a NOT NOT NOT … chain, and each nesting level is a stack frame; V8 eventually throws RangeError: Maximum call stack size exceeded straight through the linter.
The fixes are boring on purpose. Expression recursion has a hard depth ceiling — five hundred levels, far beyond anything a human writes, well under the stack budget — past which the parser emits a single diagnostic instead of throwing. And the constructs where pastes realistically chain, like A UNION B UNION C … thousands of arms long, were rewritten from recursion into flat lists: one parseSelect frame and an array of set-operations, instead of one frame per arm. The stress suite now pastes 100 KB of garbage and 30,000-deep operator chains at every build, and the public package wraps the whole pipeline in a lint() entry point that never throws, because the next editor to embed this will have the same no-error-sink problem we did.
A test suite you can audit against AWS's docs
The dialect rules — what DynamoDB accepts, what it rejects, which rewrite fixes what — all come from AWS's PartiQL reference. Documentation-derived behavior has a specific failure mode: the doc moves, the code doesn't, and nobody notices.
So the corpus is structured against it. Two hundred and eight fixtures, and every one opens with the URL of the AWS documentation page the rule comes from. A coverage table maps each documented rule to its fixture, and the suite fails if a rule loses its fixture. When AWS changes the dialect, the diff is a fixture diff with a citation on it.
That discipline paid for itself the week we open-sourced. The linter's IN-list warning cited two caps: 50 values on a partition key column, 100 on a non-key column. Re-verifying every number before publication, we could confirm the 100 in AWS's current documentation — and could not find the 50 anywhere in an operative doc. It survives all over blog posts and old forum answers, but the primary source has moved on. The linter had it right by accident (it only warns past 100, since without your schema it can't tell which case applies), and the comment now says exactly which half of the claim is documented and which is folklore.
What transfers if you're building one
- A hand-written recursive-descent parser for a small dialect is days of work, not months, and you own every error message. The scary version of "write a parser" assumes a big grammar.
- Build a CST, not an AST. Byte ranges on every node are what turn diagnostics into quick fixes; a lossy tree can't splice text.
- If the parser feeds a linter, error tolerance is the feature. Recover and continue; a parser that stops at the first error lints nothing after it.
- Swap engines behind a frozen interface with a parity corpus pinning old against new. The old implementation is the spec you already agreed to.
- Anywhere input can nest, someone will paste something that nests absurdly. Depth-cap the recursion and flatten the chains; test with garbage, not just queries.
- Cite your sources in the tests. A fixture that names the doc page it encodes is a test that can be audited when the doc changes — and it will change.
The parser is on GitHub and npm (npm install dynamodb-partiql-parser), with the editor integration in codemirror-lang-partiql. If you want the dialect itself rather than the parser, PartiQL vs SQL covers what DynamoDB's subset can and can't do and PartiQL examples is the practical walkthrough; the editor all of this was built for is in DynoTable, and you can try it free.
Top comments (0)