When I set up the ETL databases for three Astro SSG directory sites, I had two obvious SQLite-at-the-edge options: Turso (libSQL, runs anywhere) and Cloudflare D1 (SQLite inside Workers). I went with the libSQL side and installed @libsql/client. Here's the practical difference that drove the decision.
Two corrections up front, because the first version of this post overstated what I'd actually built. Nothing in the pipeline points at a hosted Turso database: the scheduled workflow passes no Turso credentials, so @libsql/client takes its local-file branch in CI exactly as it does on my laptop. And there is no shared database: each of the three apps opens its own data/local.db under its own app directory. Remote Turso is still a scaling step in docs/RUNBOOK.md — "Moving to Turso (remote libsql DB) for persistent state" — not something that's running. The comparison below is real, but read it as why I picked this client library, not as a report from production Turso.
The local dev problem D1 doesn't solve cleanly
Cloudflare D1 is native to the Workers runtime. If you're using Cloudflare Workers for server-side rendering, D1 is the obvious choice — it's edge-collocated with zero config and the env.DB binding is automatic.
My setup is different. The sites are static Astro 5 SSG on Cloudflare Pages — no Workers, no server runtime. The ETL pipeline that populates the database runs in GitHub Actions. Nothing in my stack executes inside the Workers environment.
To use D1 from GitHub Actions you either use the Cloudflare REST API or the wrangler CLI. Both work, but neither gives you a local SQLite file you can query directly during development. You'd be hitting a remote database for every SELECT during local ETL runs or testing schema changes. Wrangler does have a --local flag that writes to a local D1 file, but the path and format differ from the production D1 setup, so you're managing two different code paths.
Why Turso's local fallback changes the calculus
The @libsql/client package accepts a url that can be either a libsql:// remote URL or a file:// path:
export function getClient(): Client {
if (cached) return cached;
const url = process.env.TURSO_DATABASE_URL ?? "file:./data/local.db";
const authToken = process.env.TURSO_AUTH_TOKEN;
cached = createClient({ url, authToken });
return cached;
}
The intent was that CI would set TURSO_DATABASE_URL to a remote URL while my laptop left it unset and fell through to file:./data/local.db. That's not what happens. .github/workflows/refresh-content.yml sets neither TURSO_DATABASE_URL nor TURSO_AUTH_TOKEN — the only env it passes to the refresh job is HUGGINGFACE_TOKEN, GITHUB_TOKEN, ETL_LIMIT and GENERATE_LIMIT — so CI takes exactly the same local-file branch my laptop does. Each matrix job runs with working-directory: apps/<app>, so each site gets its own data/local.db, and since *.db is gitignored and every run starts from a fresh checkout, that file is thrown away at the end of the run. What actually persists between runs is the committed src/data/*.json output.
The upside of that accident is that the fallback isn't a dev-only convenience — it's the whole thing, in both places, one code path. Same @libsql/client package, same query API, same schema.
This means I can run ETL scripts locally and inspect the database with any SQLite viewer. Migrations are a plain loop over CREATE TABLE IF NOT EXISTS statements. The shared package exposes a generic helper for that:
export async function applyMigrations(
migrations: readonly string[]
): Promise<void> {
const client = getClient();
for (const sql of migrations) {
await client.execute(sql);
}
}
Being honest about this one too: I wrote applyMigrations() in packages/shared/src/db/index.ts and then never wired it up. Nothing in the repo calls it — each app hand-rolls the same loop in its own src/etl/schema.ts instead.
No Docker containers. No Wrangler flags. No separate local-vs-remote code path — and no remote to diverge from in the first place.
What the migration pattern actually looks like
Each app defines its own migration array in src/etl/schema.ts and exports a migrate() that walks it with db.execute(sql). The run.ts entrypoint for the AI tools ETL calls await migrate() at startup, right after mkdir("./data"). Its array is four statements: CREATE TABLE IF NOT EXISTS models, CREATE TABLE IF NOT EXISTS model_content, and two CREATE INDEX IF NOT EXISTS. If the table already exists, CREATE TABLE IF NOT EXISTS is a no-op. Idempotent, no migration runner needed.
This is the same philosophy as the ETL publish step — the article publish pipeline checks published_urls in the frontmatter before posting, so re-running never double-posts. The database migration check follows the same pattern: check-then-act, idempotent by design.
Where D1 would actually win
If any part of the stack were running inside Cloudflare Workers — a search endpoint, an API route, a middleware layer — D1 would be the stronger choice. The env.DB binding is faster than a network call to Turso's edge, and you don't need to manage auth tokens for a same-datacenter query.
My architecture is fully static because the freshness vs. speed trade-off works in SSG's favor for directory content. No Workers means D1's core advantage doesn't apply.
If I add a Cloudflare Worker for site search or a revalidation webhook, I'd reconsider. A hybrid Turso (ETL reads/writes in GitHub Actions) + D1 (Workers queries) setup would be more complex than I want right now.
Three things I don't know yet
Concurrent write performance. The ETL pipeline has max-parallel: 1 set explicitly in the workflow. Writes are serial and controlled. I haven't tested what happens with concurrent writes — and since I'm running against a local file rather than a remote instance, I have no data at all on how Turso's hosted side behaves under concurrent writes. That's a question for whenever I actually move the state remote.
Migration safety at schema evolution. Adding a nullable column to an existing table is straightforward. Renaming a column or changing a type requires a table rebuild. I haven't had to do either yet. When I do, the per-app migrate() approach will require careful ordering.
D1 pricing at scale. Turso's free tier covers 500 databases and 1 billion row reads per month. Cloudflare D1's free tier is similar. At the current scale of three sites with daily ETL runs, neither would cost anything. If traffic grows enough to matter, I'll publish actual numbers — not estimates.
The database choice was not the interesting part of this project. The local file fallback is the entire reason I made it; everything else is roughly equivalent between the two for my use case.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)