If you've ever deployed on AWS Lambda or App Runner, you know the pain:
You need backend storage, but RDS is overkill.
RDS is great. It's also $30–$100/month minimum before you write a single query. For small apps, side projects, or cost-sensitive workloads, that's a hard pill to swallow.
So you look at the alternatives.
The obvious candidates
DynamoDB? Great, until your access patterns don't fit and you're fighting it constantly.
SQLite? Perfect size. Except Lambda and App Runner are ephemeral. No persistent local disk. Dead on arrival.
Mount S3 directly? Tools like Mountpoint-S3 exist, but SQLite on a mounted S3 bucket only supports a single writer. The moment you have more than one container, you're in trouble.
So what do you do?
What I built
I wanted SQLite. I wanted S3. I wanted multiple App Runner instances or Lambda functions to be able to write concurrently without corrupting each other.
So I built distributed-sqlite: a Python library that lets multiple ephemeral compute instances read and write a SQLite database stored on S3, with conflict detection and automatic retry for safe concurrent writes.
How it works
The core idea is optimistic concurrency over S3 object versioning:
- each writer reads the current database state from S3
- writes are attempted locally
- before committing back to S3, the writer checks whether the state changed underneath it
- if it did, the conflict is detected and handled
For INSERT-only overlaps — the most common case in typical web app workloads — instead of raising a ConflictError, the writer:
- rebases on the latest state
- retries the transaction
Both workers converge to identical final state. No data loss. No conflict errors surfaced to your app.
I tested it
Two frontend web servers. Concurrent writes. Both converged to identical state every time.
That was the moment it stopped feeling like a hack and started feeling like a real primitive.
The honest tradeoffs
This is not a Postgres replacement. Let's be clear about what it is and isn't:
It is:
- great for append-heavy workloads
- great for low-to-medium write frequency
- great for Lambda/App Runner/any ephemeral compute
- operationally near-zero — just S3
- cheap. Like, really cheap.
It is not:
- globally serializable distributed SQL
- safe for arbitrary concurrent UPDATE/DELETE without care
- a replacement for RDS on high-write, complex-transaction workloads
Who this is for
If you're building:
- a side project or indie app on Lambda/App Runner
- an internal tool that doesn't justify RDS costs
- an edge service with append-heavy writes
- anything where "just use Postgres" is financially absurd
...this might be exactly what you need.
Install it
pip install distributed-sqlite
PyPI: https://pypi.org/project/distributed-sqlite/
Github: https://github.com/chrisk60331/distributed-sqllite
Top comments (0)