DEV Community

Cover image for Why We Built an SQL Layer for DynamoDB
Camma Smith
Camma Smith

Posted on

Why We Built an SQL Layer for DynamoDB

The SQL vs. NoSQL debate has been running long enough that most engineers have stopped having it. The real question isn't which is better. It's which tradeoffs you're actually making when you choose one.

SQL databases give you a relational model, flexible ad-hoc queries, decades of tooling, and strong consistency guarantees. The cost is horizontal scale. Distributing a relational database across many nodes is possible, but it adds complexity, and query semantics get harder to reason about as you do it. At extreme read/write throughput, relational databases require significant engineering effort to keep performant.

NoSQL databases flip those tradeoffs. They're designed to distribute data across nodes predictably, which gives you horizontal scale and consistent latency. You gain the ability to handle massive throughput without complex sharding logic. You give up some of the richness of the relational model, particularly around ad-hoc querying.

DynamoDB is the clearest example of this tradeoff done deliberately and at scale.

Why teams choose DynamoDB

DynamoDB is a managed key-value and document store built by AWS. It delivers single-digit millisecond read and write latency at any scale, scales capacity automatically, requires no cluster management, and has a low operational burden compared to running your own database. For teams that don't want to operate infrastructure, it's a compelling default for production workloads.

The data model is built around partition keys. Every item in a table has a partition key, which determines which physical partition holds that item. An optional sort key allows range queries within a partition. Global Secondary Indexes (GSIs) let you define alternative partition and sort key layouts, enabling multiple read patterns without duplicating your full dataset.

For workloads where access patterns are known and stable, DynamoDB performs exceptionally well. An e-commerce platform with clear paths for "get order by ID", "list orders for customer", and "get product by SKU" can serve all of them from DynamoDB efficiently and cheaply. Each access pattern gets an index. Each query hits that index. Performance is predictable.

The catch is that this performance guarantee comes with a hard constraint: you must define your access patterns before you write your data.

The querying problem

DynamoDB's Query API requires a partition key value. You can filter on sort key attributes and apply additional filter expressions, but the partition key is always mandatory. Without it, the only option is Scan, which reads every item in the table and applies filters afterward.

Scans are expensive. They consume read capacity proportional to the data they touch, not the data they return. On a large table, a Scan that returns 100 rows might read millions of items to get there. That's costly in both time and money, and it gets worse as your table grows.

This creates a specific class of queries that DynamoDB handles poorly.

Aggregates. There's no native SUM, COUNT, AVG, or GROUP BY. To know how many orders were placed by region last week, you fetch the rows and compute the aggregate yourself. For small datasets that's workable. For anything large, you're pulling significant data across the wire and doing compute work in your application.

JOINs. DynamoDB has no cross-table JOIN. Single-table design, a common DynamoDB modeling pattern, works around this by denormalizing related entities into one table. But that requires anticipating every join at schema design time. It makes your data model harder to evolve and makes ad-hoc analysis across related entities painful.

Ad-hoc queries. Questions that weren't anticipated at schema design time have no efficient answer. The data exists. Getting it in an arbitrary shape requires either a full-table Scan or redesigning your access patterns, which often means a new GSI and a backfill.

These aren't edge cases. They're the normal requirements of anyone who needs to understand what's in their database beyond the access patterns the application was originally built around.

What teams do about it

Several approaches exist. Each involves real tradeoffs.

Application-level computation. Fetch the relevant rows from DynamoDB and compute aggregates or join logic in application code. This works at small data volumes. At larger volumes, you're moving significant data across the network and doing CPU-heavy work in your API layer. It also spreads data logic across application code rather than keeping it at the data layer.

Export to S3 and Athena. DynamoDB supports point-in-time exports to S3. Athena can then run SQL against those files. The tradeoff is freshness: exports aren't real-time, so your query results are as stale as your last export. For some reporting use cases that's acceptable. For anything that needs current data, it isn't. You also now have two systems to understand and operate.

DynamoDB Streams to a secondary store. Stream DynamoDB changes to Redshift, OpenSearch, or a relational database, and run queries against the replica. This can give you full SQL against a continuously synced copy of your data. The tradeoffs are real: operational complexity, eventual consistency between primary and replica, and the cost of maintaining both systems. When the pipeline works, it works. When it lags or fails, your results are wrong and diagnosing the problem requires understanding two systems at once.

PartiQL. AWS added PartiQL, an SQL-like syntax, to DynamoDB in 2020. PartiQL translates SQL-style statements into native DynamoDB API calls. A SELECT with a partition key condition becomes a Query. Without one, it becomes a Scan. There's no JOIN support, no GROUP BY, and no aggregates. PartiQL gives DynamoDB an SQL-style syntax. It doesn't change DynamoDB's underlying access model. It's a syntax layer, not a query engine.

Each of these approaches solves part of the problem. None of them gives you real-time SQL with full query capabilities directly against your DynamoDB tables.

What we built

That's what led us to build DynamoSQL, a read-only SQL engine that translates ANSI SQL SELECT statements into DynamoDB API calls and executes them directly against your tables, in real time.

When you submit an ANSI SQL SELECT statement to DynamoSQL, it parses the SQL into an AST, builds a logical query plan, and runs an optimizer that inspects your table's actual index metadata. The optimizer selects the best access path: a partition key Query if your WHERE clause supports it, a GSI Query if a matching index exists, or a Scan as a last resort. Then it executes the plan against DynamoDB and returns results as JSON rows.

The execution layer handles what DynamoDB can't do natively. JOINs are computed across multiple DynamoDB API calls. GROUP BY, aggregates (SUM, COUNT, AVG, MIN, MAX), subqueries, CTEs, UNION, INTERSECT, EXCEPT, ORDER BY, HAVING, and a wide function library all run in the engine. You write standard SQL. DynamoSQL figures out how to execute it efficiently.

This is different from the Athena approach because it's real-time. No export pipeline, no stale snapshots. It's different from streaming to a secondary store because there's no replica to maintain and no pipeline to monitor. It's different from PartiQL because it's an actual query engine, not a syntax translation layer.

It's also read-only. DynamoSQL supports SELECT only. DynamoDB's write model is precise by design: partition keys, sort keys, defined capacity allocations. Wrapping writes in SQL semantics would make the behavior harder to reason about, not easier. The read side is where the gap is, and that's what we're solving.

The MCP server

The SQL engine solves the querying problem for developers who write code. The MCP server extends that to a different kind of user: anyone interacting with an AI agent.

MCP, or Model Context Protocol, is the standard interface AI agents use to call external tools. When DynamoSQL is connected to an agent through its MCP server, the agent can query DynamoDB directly using plain SQL through a single tool call.

Here's what that looks like in practice. A developer or analyst asks the agent a question in natural language: "How many orders were placed in California last week?" The agent doesn't need to know anything about DynamoDB's partition key structure or which indexes exist on the table. It translates the question into SQL, calls run_sql, and gets back rows. DynamoSQL handles the rest: optimizing the query plan, selecting the right index, executing against DynamoDB, and returning structured results.

This matters because SQL is a reliable middle layer between natural language and a NoSQL database. LLMs write SQL well. They've been trained on large amounts of it, the syntax is unambiguous, and errors are easy to catch. Asking an agent to write DynamoDB SDK code to answer the same question is a different proposition: the code is longer, harder to verify, and requires the agent to know the table's internal structure upfront.

The SQL layer gives the agent a well-defined interface. The agent writes what it wants. DynamoSQL figures out how to fetch it. The result is more reliable than asking the agent to navigate DynamoDB's access model directly, and more accessible than expecting every user to write SQL by hand.

How it deploys

DynamoSQL runs as a managed SaaS. You connect your AWS account, register your DynamoDB tables under a schema, and run queries through the REST API or the MCP server. Your data stays in your AWS account. DynamoSQL assumes a read-only IAM role you create and control, runs queries, and returns results. We don't store your data and we don't have standing access to your tables.

If you're using DynamoDB and want real-time SQL read access against your tables, the product is live. AWS Marketplace Listing

Where this leaves you

DynamoDB is a good database. The teams that choose it make a deliberate call: predictable performance, no cluster management, strong operational guarantees at scale. The querying limitation isn't a reason to abandon it. It's a gap that can be filled without changing anything about how your data is stored or your tables are structured.

DynamoSQL sits on top of what you already have. You bring the DynamoDB tables. We bring the SQL engine, the query optimizer, and the MCP server. The access patterns you've already defined become inputs to the optimizer, not constraints you have to work around.

If you're hitting the querying wall and want to see how it works against your own tables, we'd like to hear from you. Dynamosql.com

Top comments (0)