DEV Community: Eitamos Ring

A Protobuf for Database Schemas

Eitamos Ring — Wed, 18 Mar 2026 07:52:54 +0000

Every serious system has an interface definition for its wire format. gRPC has protobuf. REST has OpenAPI. GraphQL has its SDL. But databases -- the thing everything else is built on top of -- have nothing.

Your database schema is one of the most important artifacts in your system. It defines every table, column, type, constraint, relationship, and index. It encodes years of domain decisions. And yet there is no standard, portable, machine-readable format for it.

We built one. We call it ctxexport.json.

The problem is older than LLMs

Before you assume this is an AI-context story, consider how many times you have needed your schema outside the database itself:

Onboarding a new engineer who needs to understand the data model.
Diffing staging against production to catch drift before a deploy.
Running a linter in CI to enforce naming conventions or catch missing indexes.
Generating documentation that is not immediately stale.

Every time, you end up writing a bespoke script that queries information_schema or pg_catalog, parses the output, and feeds it into whatever tool you need. The script is Postgres-specific. It breaks when you add a second schema. Nobody maintains it.

pg_dump --schema-only exists, but it is a restore format, not a consumption format. It is Postgres-specific SQL with SET statements, ownership clauses, and an ordering designed for replay, not reading. Try parsing it reliably. Try feeding it to a linter. Try diffing two of them without drowning in noise.

MongoDB is worse. There is no mongodump --schema-only. Your schema lives in the shape of whatever documents happen to exist. Good luck extracting that into something a tool can reason about.

Extract once, use many ways

The core insight behind ctxexport.json is the same one behind protobuf: separate the definition from the consumption.

A protobuf .proto file is written once and compiled to Go structs, Python classes, TypeScript types, gRPC stubs, or REST gateways. The definition is the single source of truth. The consumers are many and varied.

ctxexport.json works the same way. You extract your schema once -- from Postgres, MongoDB, or whatever backend -- and produce a single canonical JSON file. That file contains entities (tables, views, collections), fields (columns with types, nullability, defaults), edges (foreign keys and inferred references), and access paths (indexes). Everything a tool needs to understand your data model, nothing it does not.

From that single artifact, you can:

Compile to a lighthouse map -- a compact table-and-relationship summary that fits in an LLM prompt.
Compile to full SQL DDL -- standard CREATE TABLE statements for any subset of tables.
Serve over MCP -- give an AI agent schema awareness without database credentials.
Diff across environments -- compare staging and production schemas as structured data, not text.
Lint offline -- check naming conventions, missing indexes, or orphaned foreign keys in CI.
Validate in CI -- catch schema regressions before they reach production.
Commit to git -- your schema becomes a versioned artifact with a real history.

None of these consumers need to know whether the source was Postgres or MongoDB. None of them need a live database connection. The extraction happened once, upstream, and everything downstream reads the same contract.

The sidecar pattern

Databases have never been good at carrying human knowledge alongside the schema. Your users.deleted_at column is a soft-delete flag, but the database only knows it is a timestamp with time zone. Your orders.payload column is JSONB with a specific structure, but the database sees an opaque blob.

A sidecar file (dbdense.yaml) layers descriptions and value annotations onto the extracted schema:

entities:
  payments:
    fields:
      status:
        values: ["pending", "authorized", "paid", "failed", "refunded"]
  users:
    fields:
      deleted_at:
        description: "Soft delete timestamp. NULL = active."

This merges at export time. The compiled DDL gets inline comments like -- Values: pending, authorized, paid, failed, refunded. Every downstream consumer -- linter, LLM, documentation generator -- picks it up automatically. Write it once in a YAML file committed to the repo.

Why JSON, not SQL

SQL DDL is human-readable but machine-hostile. Parsing CREATE TABLE statements reliably across dialects is a nightmare. Defaults are quoted differently. Constraints can be inline or out-of-band. Comments use different syntax. There is no standard way to represent a foreign key relationship as structured data.

JSON is boring and that is the point. It is a declarative state representation -- you look up a table by name, not by parsing DDL statement order. Every language has a JSON parser. The schema is simple: a version string, an array of entities, and an array of edges. You can validate it with a JSON Schema. You can diff it with jq. You can read it in any language without a SQL parser.

A minimal entity looks like this:

{
  "name": "payments",
  "type": "table",
  "fields": [
    {"name": "id", "type": "uuid", "is_pk": true},
    {"name": "status", "type": "text", "not_null": true, "values": ["pending", "paid", "failed"]}
  ]
}

Flat, predictable, zero ambiguity.

Stop treating your schema like a black box

The immediate use case is LLM context -- giving AI agents schema awareness without live database access. But the format is deliberately general. If your tool can read JSON, it can read a database schema. That was not true before.

The project is at github.com/valkdb/dbdense. The contract is documented in docs/ctxexport-contract.md. It supports Postgres and MongoDB today. The extractor interface is small enough that adding a new backend is a single file.

Your database schema is too important to be locked inside the database. Export it. Version it. Build on it.

Stop Sending 93K Tokens of Schema to Your LLM Agent!

Eitamos Ring — Wed, 18 Mar 2026 07:52:06 +0000

I've watched agents query information_schema over and over, spending 4-6 turns just to figure out which tables exist, what columns they have, and how they join. On a 500-table database, the full DDL is around 93,000 tokens. Most questions touch 3-5 tables. On a complex multi-table join, I measured a 64% token reduction by just giving the agent the schema upfront.

That's what dbdense does.

I built dbdense to fix this.

What it does

dbdense is a three-step offline pipeline: extract, compile, serve.

Extract connects to your database once and snapshots the schema into a portable JSON file (ctxexport.json). Tables, columns, types, primary keys, foreign keys, indexes -- everything an LLM needs to write correct queries.
Compile turns that snapshot into two artifacts:
- A lighthouse -- a compact table map (~4K tokens for 500 tables). It looks like this:
```
 T:users|J:orders,sessions
 T:orders|E:payload,shipping|J:payments,shipments,users
 T:payments|J:orders
```
Every table, its FK neighbors, and embedded docs. 23x smaller than full DDL. This stays in the agent's context so it always knows what's available.
- Full DDL -- standard CREATE TABLE statements with constraints, rendered on demand only for the specific tables the agent asks about.
Serve (optional) exposes the lighthouse as an MCP resource and the DDL via an MCP slice tool. The agent reads the map, picks the tables it needs, and gets back just those definitions.

After the extract, everything runs locally. The compiled artifacts are plain text you can commit to your repo. No database connection needed at runtime.

No credentials in the agent runtime

The export step is the only step that touches the database. After that, compile and serve work from the local snapshot. Your production database credentials never need to be in the agent's environment. The tool works offline and air-gapped.

The numbers

I ran an agentic benchmark: n=3, same 5 questions, same seeded Postgres database (20K+ rows, 8 tables), same model (Claude Sonnet 4). One arm had only a Postgres MCP tool. The other had the same tool plus dbdense schema context injected into the prompt.

Metric	Without schema context	With dbdense	Delta
Correct answers	13/15	13/15	equal
Avg turns	4.1	2.2	-46%
Tokens per run	285,922	187,603	-34%

Same accuracy. 34% fewer tokens. 46% fewer turns.

The savings scale with query complexity. On simple single-table filters, both arms performed about the same. On a complex multi-table join, the baseline agent spent 6+ turns querying information_schema to discover the schema. dbdense answered in 2 turns, using 64% fewer tokens for that query.

The two wrong answers (both on the same question, in both arms) returned identical incorrect results, pointing to question ambiguity rather than a schema context issue.

Sidecar enrichment

Databases lie by omission. A column named status with type text tells the LLM nothing about what values are valid. The agent either guesses or wastes a SELECT DISTINCT turn to find out.

dbdense supports a dbdense.yaml sidecar file where you annotate columns with descriptions and enum values:

entities:
  payments:
    fields:
      status:
        values: ["pending", "authorized", "paid", "failed", "refunded"]
  orders:
    fields:
      status:
        description: "Order lifecycle status."
        values: ["pending", "confirmed", "shipped", "delivered", "cancelled"]

These annotations merge into the compiled DDL as inline SQL comments. The LLM sees -- Values: pending, authorized, paid, failed, refunded right next to the column definition. No extra queries needed.

This also works for documenting JSONB structures, MongoDB embedded documents, or anything else the raw schema doesn't capture.

What it doesn't do

The snapshot is static. If your schema changes, re-run export. This is intentional -- schemas are stable; questions change.

The slice tool still depends on the LLM picking the right tables from the lighthouse. dbdense reduces the context problem; it doesn't solve table selection for the model.

It's not a pg_dump --schema-only replacement. The renderer covers columns, PKs, FKs, NOT NULL, defaults, unique constraints, and indexes, but skips triggers, RLS policies, and custom types.

Try it

go install github.com/valkdb/dbdense/cmd/dbdense@latest
dbdense export --driver postgres --db "postgres://user:pass@localhost:5432/mydb" --schemas public
dbdense compile --mode lighthouse --in ctxexport.json --out lighthouse.txt
dbdense compile --in ctxexport.json --out schema.sql

You now have two files: a lighthouse map and full DDL. Point your agent at them. If you use Claude Code, dbdense init-claude writes the MCP config for you.

The project is open source at github.com/valkdb/dbdense.

From Flat Columns to Full Relationships: How We Taught Our SQL Parser to Understand Table Constraints

Eitamos Ring — Thu, 12 Mar 2026 14:30:24 +0000

TL;DR — We went from parsing CREATE TABLE as a bag of columns to extracting full relational metadata: primary keys, foreign keys, unique constraints, CHECK expressions, and referential actions. Here's how we did it in Go with ANTLR, and why it matters for anyone building developer tools on top of SQL.

The Problem Nobody Talks About

You've seen a hundred blog posts about writing SQL.

This one is about reading it programmatically.

We maintain valk-postgres-parser, an open-source Go library that parses PostgreSQL into a structured IR (intermediate representation). It powers linters, migration validators, and query analyzers that need to understand what SQL means, not just what it says.

For months, our CREATE TABLE extraction looked like this:

SQL in → columns out

That's it. Column names, types, nullability. No relationships. No constraints. If you wanted to know which column was the primary key or what foreign key pointed where — you were on your own.

CREATE TABLE orders (
    id    serial PRIMARY KEY,
    email text   UNIQUE NOT NULL,
    user_id int  REFERENCES users(id) ON DELETE CASCADE,
    total numeric CONSTRAINT positive_total CHECK (total > 0)
);

Before: We'd hand you [id, email, user_id, total] with types. That's barely more useful than a regex.

After: You get the full picture:

{
  "Constraints": {
    "PrimaryKey": { "Columns": ["id"] },
    "ForeignKeys": [{
      "Columns": ["user_id"],
      "ReferencesTable": "users",
      "ReferencesColumns": ["id"],
      "OnDelete": "CASCADE"
    }],
    "UniqueKeys": [{ "Columns": ["email"] }],
    "CheckConstraints": [{
      "ConstraintName": "positive_total",
      "Expression": "(total > 0)"
    }]
  }
}

That's the difference between a tokenizer and a parser that understands your schema.

The Architecture: Two Layers, One Truth

Here's how data flows through the parser. Constraints live at both layers:

                    ┌──────────────────────┐
                    │      Raw SQL         │
                    └──────────┬───────────┘
                               │
                    ┌──────────▼───────────┐
                    │   ANTLR 4 Grammar    │
                    │  (PostgreSQL Lexer +  │
                    │       Parser)         │
                    └──────────┬───────────┘
                               │  parse tree
                    ┌──────────▼───────────┐
                    │   Core IR Layer       │
                    │                       │
                    │  DDLAction            │
                    │   ├─ ColumnDetails[]  │
                    │   └─ Constraints      │
                    │       ├─ PrimaryKey   │
                    │       ├─ ForeignKeys  │
                    │       ├─ UniqueKeys   │
                    │       └─ CheckConst.  │
                    └──────────┬───────────┘
                               │  convert
                    ┌──────────▼───────────┐
                    │  Analysis Layer       │
                    │                       │
                    │  SQLDDLAction         │
                    │   ├─ ColumnDetails[]  │
                    │   └─ Constraints      │
                    │       ├─ PrimaryKey   │
                    │       ├─ ForeignKeys  │
                    │       ├─ UniqueKeys   │
                    │       └─ CheckConst.  │
                    └──────────────────────┘

Core IR = 1:1 with the parse tree. It's the what.
Analysis layer = the consumer-friendly view. It's the so what.

Both layers carry the same constraint shape, but the analysis layer normalizes and simplifies for downstream consumers. Pick your depth: low-level IR for tool authors, analysis structs for application developers.

How It Works: Constraint Extraction in 3 Steps

Step 1: Walk the ANTLR Tree

PostgreSQL's grammar defines constraints in two places:

CREATE TABLE orders (
    id serial PRIMARY KEY,              -- ← INLINE constraint
    user_id int REFERENCES users(id),   -- ← INLINE constraint
    CONSTRAINT uq_email UNIQUE (email)  -- ← TABLE-LEVEL constraint
);

Inline constraints are attached to a column definition node. Table-level constraints are siblings of the column list. We walk both.

Step 2: Build the IR Structs

Each constraint type maps to a dedicated Go struct:

type DDLPrimaryKey struct {
    ConstraintName string
    Columns        []string
}

type DDLForeignKey struct {
    ConstraintName    string
    Columns           []string
    ReferencesSchema  string
    ReferencesTable   string
    ReferencesColumns []string
    OnDelete          FKAction   // CASCADE, SET NULL, RESTRICT, ...
    OnUpdate          FKAction
}

type DDLCheckConstraint struct {
    ConstraintName string
    Expression     string       // raw expression text: "(price > 0)"
}

These get bundled into DDLConstraints and attached to the DDLAction:

type DDLConstraints struct {
    PrimaryKey       *DDLPrimaryKey
    ForeignKeys      []DDLForeignKey
    UniqueKeys       []DDLUniqueConstraint
    CheckConstraints []DDLCheckConstraint
}

Step 3: Merge Inline + Table-Level

This is the tricky part. A single table can define its PK inline on one column and also have a composite unique key at the table level. We use an internal tableConstraints struct with a merge() method to combine both sources without duplication:

    inline constraints ──┐
                         ├──▶ merge() ──▶ DDLConstraints
  table-level constraints┘

ALTER TABLE: The Forgotten Half

Most SQL parsers stop at CREATE TABLE. But in the real world, constraints are added after table creation just as often:

ALTER TABLE public.products
  ADD CONSTRAINT positive_price CHECK (price > 0);

ALTER TABLE orders
  ADD CONSTRAINT fk_user FOREIGN KEY (user_id)
  REFERENCES users(id) ON DELETE SET NULL;

Before our changes, ALTER TABLE ... ADD CONSTRAINT was silently dropped — the extraction guard only checked for column adds/drops. Now it emits a proper DDLAction with ADD_CONSTRAINT flag and fully populated Constraints.

Real-World Use Cases

Why does any of this matter? Here's what you can build with structured constraint metadata:

  ┌─────────────────┐     ┌──────────────────────┐
  │ Migration Safety│     │  Schema Docs Gen     │
  │                 │     │                      │
  │ "This migration │     │ Auto-generate ER     │
  │  drops a FK that│     │ diagrams from SQL    │
  │  3 services     │     │ migration files      │
  │  depend on"     │     │                      │
  └────────┬────────┘     └──────────┬───────────┘
           │                         │
           │    ┌────────────────┐   │
           └────┤  postgresparser├───┘
                │  Constraints   │
           ┌────┤  IR            ├───┐
           │    └────────────────┘   │
  ┌────────▼─────────┐     ┌─────────▼─────────────┐
  │ Query Linting    │     │  Access Control       │
  │                  │     │                       │
  │ "WARNING: INSERT │     │ "Column user_id has   │
  │  missing required│     │  FK to users — ensure │
  │  FK column"      │     │  caller has read on   │
  │                  │     │  both tables"         │
  └──────────────────┘     └───────────────────────┘

Migration safety — Diff two versions of your schema SQL. If a FK is removed that other services reference, flag it before it hits production.

Schema documentation — Generate ER diagrams directly from .sql files. No database connection needed. Works in CI.

Query linting — Cross-reference INSERT/UPDATE column lists against known constraints. Catch missing NOT NULL or FK violations statically.

Access control analysis — FK relationships imply data access paths. Map them automatically for security reviews.

The Numbers

Metric	Value
GitHub Stars	200+
Constraint types supported	PK, FK, UNIQUE, CHECK
FK referential actions	CASCADE, SET NULL, SET DEFAULT, RESTRICT, NO ACTION
Works on	CREATE TABLE (inline + table-level) + ALTER TABLE ADD CONSTRAINT
Grammar changes needed	Zero. Pure tree-walking.

That last point is worth emphasizing: we didn't touch the ANTLR grammar at all. The PostgreSQL grammar already parses constraints — we just weren't reading them. The entire feature was built by walking deeper into the existing parse tree.

Try It

import parser "github.com/ValkDB/postgresparser"

sql := `CREATE TABLE orders (
    id serial PRIMARY KEY,
    user_id int REFERENCES users(id) ON DELETE CASCADE,
    total numeric CHECK (total > 0)
);`

result, _ := parser.Parse(sql)

constraints := result.DDLActions[0].Constraints
fmt.Println(constraints.PrimaryKey.Columns)           // [id]
fmt.Println(constraints.ForeignKeys[0].OnDelete)       // CASCADE
fmt.Println(constraints.CheckConstraints[0].Expression) // (total > 0)

Install:

go get github.com/ValkDB/postgresparser@latest

What's Next

We're not done. The constraint work opened doors for a brand new world!
Follow along, and feel free to request stuff

Built and maintained by the ValkDB team. Star the repo if this saved you from writing another regex.

How does a linter know your column doesn't exist

Eitamos Ring — Mon, 09 Mar 2026 08:40:04 +0000

You write a query that SELECTs ghost_status from the orders table. Your code compiles. Your tests pass. But ghost_status was never created in any migration. In production, that query crashes.
Valk Guard catches this at PR time - with no database connection.
This post walks through exactly how. Not hand-waving. The actual code path, from source file to finding.
The setup
Here's a Go file using Goqu to build a query:
func ListBrokenUserOrderStatus(ctx context.Context) error {
_, _, err := goqu.From("users").
LeftJoin(
goqu.T("orders"),
goqu.On(goqu.I("orders.user_id").Eq(goqu.I("users.id"))),
).
Select("users.id", "users.email", "orders.ghost_status").
Where(goqu.I("orders.missing_flag").Eq("pending")).
ToSQL()
return err
}
And here's the migration that created the orders table:
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
user_id INTEGER NOT NULL REFERENCES users(id),
total NUMERIC(10,2) NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
created_at TIMESTAMP DEFAULT now()
);
Notice: the query references orders.ghost_status. The migration never created that column. There is no ghost_status. Valk Guard reports:
VG105: projection column "ghost_status" not found in table "orders" schema; check SELECT list and schema/model mappings
How does it know?
Let's walk through each phase.
Phase 1: Query extraction
The Goqu scanner doesn't look for SQL strings. It walks the Go AST looking for method chains rooted in goqu.From().
When it finds one, it flattens the chain into a list of method calls: From("users") → LeftJoin(...) → Select(...) → Where(...). Each method gets parsed: From gives the base table, LeftJoin gives the join target, Select gives the projection columns, Where gives the predicates.
From these parts, the scanner synthesizes a SQL statement:
SELECT users.id, users.email, orders.ghost_status
FROM users LEFT JOIN orders ON orders.user_id = users.id
WHERE orders.missing_flag = 'pending'
This SQL never existed in your source code. Valk Guard constructed it from the AST of your Go code. That's the key difference from regex-based tools - regex can't walk a method chain and reconstruct what the query builder will produce.
Phase 2: Schema snapshot
Separately, Valk Guard finds all .sql files under your migration paths. Each file gets parsed through postgresparser, and every DDL statement gets applied to a Snapshot - an in-memory representation of your schema's current state.
The snapshot builder processes DDL actions in order:
CREATE TABLE orders (id, user_id, total, status, created_at) → registers the table with five columns
ALTER TABLE orders ADD COLUMN shipped_at TIMESTAMP → adds a sixth column
ALTER TABLE orders DROP COLUMN shipped_at → removes it

The end result is a map of table names to column definitions. For orders, that's: id, user_id, total, status, created_at. Five columns. No ghost_status.
This is the same principle as running all your migrations on an empty database - except it happens in memory, with no database, in microseconds.
Phase 3: Rule evaluation
Now VG105 runs. It takes the synthesized SQL (already parsed into a structured IR by postgresparser) and the schema snapshot, and does a straightforward lookup:
For each column in the SELECT list with usage type "projection", resolve which table it belongs to (using the alias or the single-table shortcut)
Look up that table in the snapshot
Check if the column exists in the table's column map
If not → finding

For ghost_status, the column usage says it belongs to orders (from the orders.ghost_status qualifier). The snapshot has an orders table. But orders.ghost_status is not in the column map. Finding.
The same logic powers VG106 (unknown filter column - catches WHERE orders.missing_flag = 'pending' from the same query) and VG107 (unknown table reference).
It also works with ORM models
The same snapshot system powers schema-drift rules (VG101–VG104). Instead of checking queries against migrations, these rules check ORM models against migrations.
Say you have a Go struct:
type Order struct {
ID int db:"id"
UserID int db:"user_id"
Total string db:"total"
Status string db:"status"
GhostStatus string db:"ghost_status"
}
Valk Guard's Go model extractor walks the AST, reads the db struct tags, and produces a ModelDef with columns: id, user_id, total, status, ghost_status.
VG101 then compares each model column against the migration snapshot. ghost_status isn't in the orders table → finding:
VG101: model "orders" references column "ghost_status" not found in table "orders" schema; check migration DDL or update model mapping
Two different rules, two different input paths (query vs. model), same schema snapshot, same answer.
What this means in practice
You don't need a running database. You don't need to run migrations. You don't need to connect to staging. Valk Guard reads your source code and your migration files, builds everything in memory, and cross-references them statically.
This runs in CI in seconds. It catches the kind of bug that usually shows up as a column "ghost_status" does not exist error in your logs at 2am - and moves it to a PR comment at 2pm instead.
go install github.com/valkdb/valk-guard/cmd/valk-guard@latest
valk-guard scan .
Repo: github.com/ValkDB/valk-guard

We didn't want an AI SQL reviewer. We wanted deterministic

Eitamos Ring — Sat, 07 Mar 2026 16:33:20 +0000

So we built Valk Guard.

Most SQL linters scan .sql files. The problem is, most SQL doesn't live in .sql files.

It lives in db.Query() calls. In Goqu builder chains. In SQLAlchemy ORM methods. In migration files mixed with application logic. By the time SQL reaches production, it's been assembled, concatenated, or synthesized by code that no .sql-only tool will ever see.

I built Valk Guard to solve that. It's a static analysis tool that walks your source code's AST, reconstructs the SQL your ORMs and query builders will generate, parses it through a real PostgreSQL grammar, and reports findings in CI-friendly formats. No database connection. No runtime. Just structure.

go install github.com/valkdb/valk-guard/cmd/valk-guard@latest
valk-guard scan .

19 rules enabled by default. Zero config. Takes seconds.

What it actually catches

Here's a Goqu chain in Go:

goqu.From("orders").Delete()

There's no raw SQL anywhere in that line. But Valk Guard walks the Go AST, recognizes the Goqu method chain, synthesizes DELETE FROM orders, feeds it through postgresparser, and fires VG003: DELETE without WHERE may affect all rows.

Same thing with SQLAlchemy:

session.query(User).delete()

No SQL string. Valk Guard's embedded Python AST extractor reconstructs it, and the same rule fires.

The full rule set covers three categories:

Query safety — UPDATE without WHERE (VG002), DELETE without WHERE (VG003), SELECT * (VG001), unbounded SELECT without LIMIT (VG004), leading wildcard LIKE '%...' (VG005), SELECT ... FOR UPDATE without WHERE (VG006).

Dangerous DDL — DROP TABLE / TRUNCATE in application code (VG007), CREATE INDEX without CONCURRENTLY (VG008).

Schema drift — ORM model references a column that migrations dropped (VG101). NOT NULL column missing from model (VG102). Type mismatch between model and DDL (VG103). Model table has no CREATE TABLE in migrations (VG104). Query SELECTs a column that doesn't exist in the schema (VG105). And several more cross-reference checks between your code and your migrations.

That last category is the one I haven't seen elsewhere. Valk Guard reads Go struct tags (db, gorm) and Python __tablename__ / Column(...) definitions, builds a schema snapshot from your migration DDL, and cross-references them. If your ORM model says email exists but your migration dropped it, that's VG101 at PR time — not a runtime panic in production.

Why AST, not AI

This was a deliberate choice, and it's worth explaining.

CI is not a brainstorming session. If a PR check comments on your code and changes its mind between runs, people stop trusting it. If it floods you with false positives, people add --skip-lint and move on. The tool is dead even if the idea was good.

I needed the opposite: same input, same output, every time. Testable. Explainable. Boring in the best way.

AI is useful for exploration and suggestions. But a blocking CI step needs determinism. Even structured-output approaches for LLMs improve schema conformance — they don't make a generative model behave like a static analyzer. The questions Valk Guard answers are structural: "does this statement have a WHERE clause?" "does this builder chain produce a bounded query?" "does this model match this schema?" Those are AST questions, not generation questions.

The same logic applies to regex. Regex is fine when the thing you're checking is a flat string. It falls apart when SQL is buried inside Go method chains or Python ORM calls. You can't regex your way through goqu.From("users").Where(goqu.C("id").Eq(42)).Select("name") and reliably reconstruct the query. You need to parse the source language's AST, understand the builder pattern, and synthesize the SQL. That's what Valk Guard does.

A small number of checks do use targeted regex after parsing — when a parser-extracted clause doesn't expose the exact field a rule needs. But that's regex as a surgical helper on already-parsed output, not regex as the foundation.

The pipeline

Source files go in. Findings come out. Here's what happens in between:

1. Extraction — Four scanners run concurrently. The raw SQL scanner handles .sql files with proper dollar-quoting and nested block comments. The Go scanner uses go/ast to extract SQL from db.Query, db.Exec, and db.QueryRow. The Goqu scanner walks builder chains and synthesizes SQL. The SQLAlchemy scanner invokes an embedded Python script (stdlib only — no pip dependencies) that parses ORM chains via Python's ast module.

2. Parsing — Every extracted statement goes through postgresparser, a pure-Go PostgreSQL parser I built on ANTLR. It produces a structured IR: tables, columns, joins, WHERE clauses, command type. No CGO, no database connection. Most queries parse in 70–350 µs.

3. Rule evaluation — Rules are dispatched by SQL command type for efficiency. Query rules (VG001–VG008) run against every parsed statement. Schema-drift rules (VG101+) cross-reference ORM model definitions against a migration-derived schema snapshot. Query-schema rules (VG105–VG108) validate that columns and tables referenced in queries actually exist.

4. Output — Findings are deduplicated, sorted by file and line, and formatted as terminal output, JSON, SARIF (for GitHub Code Scanning), or rdjsonl (for reviewdog PR comments).

The whole thing is ~8,100 lines of Go (plus ~700 lines of embedded Python), with nearly 1:1 test coverage. Three runtime dependencies: cobra, postgresparser, and yaml. That's it.

CI integration

Valk Guard was designed for pull request workflows. Exit code 0 means clean, 1 means findings, 2 means config/parser error. Hook it into reviewdog and you get inline PR comments:

- name: Run valk-guard
  run: |
    set +e
    valk-guard scan . --format rdjsonl > valk-guard.rdjsonl
    code=$?
    set -e
    if [ "$code" -gt 1 ]; then exit "$code"; fi

- name: Post review comments
  env:
    REVIEWDOG_GITHUB_API_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: |
    reviewdog -f=rdjsonl -name="valk-guard" \
      -reporter=github-pr-review -filter-mode=added \
      < valk-guard.rdjsonl

Findings are non-blocking by default. Config errors fail the job. You can see real example PRs with live review comments in the valk-guard-example repo.

Where it fits

Valk Guard is not a runtime firewall, not a database advisor, and not a replacement for EXPLAIN ANALYZE. It's a guardrail for the most common and most expensive SQL mistakes — the ones that happen when someone pushes a DELETE FROM orders without a WHERE at 4pm on a Friday.

It's PostgreSQL-only. It doesn't auto-fix. It doesn't need a running database. It reads your source code, understands your ORMs, and tells you what's going to break — before it merges.

Less magic. More signal. Same answer every run.

Repo: github.com/ValkDB/valk-guard

Building a PostgreSQL Parser in Go: What Broke After We Open-Sourced It

Eitamos Ring — Wed, 18 Feb 2026 06:18:53 +0000

Building a PostgreSQL Parser in Go: What Broke After We Open-Sourced It

postgresparser is a pure-Go PostgreSQL SQL parser. It turns SQL text into structured metadata (tables, columns, joins, filters, DDL actions, parameters) without executing queries.

We thought it was solid. Open source proved we were wrong.

Here is what open source forced us to learn.

The biggest shift was not “more bug reports.” It was use-case expansion.

We built for our workflow. Users showed up with very different workloads.
In the first week after release, most feedback centered on deterministic batch parsing.

Our internal assumptions broke immediately

Inside a single team, ambiguous behavior survives because everyone “knows” the rules. Public users do not have that context.

The first pressure point was multi-statement SQL. We had ParseSQL (single statement) and figured batch parsing was “close enough.” It was not.

People were using the parser for:

CI linting pipelines
production tools
llm wrappers

People asked practical questions we could not answer cleanly:

Which exact statement failed?
Is this a warning or a hard failure?
Can I map diagnostics to the original SQL text reliably?

Those questions forced us to define strict contracts instead of relying on implied behavior.

If your tool consumes SQL in bulk, batch correlation is everything.

Broken behavior example

This input exposed the issue quickly:

SELECT 1;
SELECT FROM;
SELECT 2;

Early batch behavior made correlation awkward because results were compacted and diagnostics were not statement-first. If you’re building CI checks or migration tooling, “something in the batch failed” is not actionable.

Now each statement has deterministic correlation (Index, RawSQL, Query, Warnings), so downstream code can point to the exact source statement.

Before/after API diff

- type ParseBatchResult struct {
-   Queries          []*ParsedQuery
-   Warnings         []ParseWarning
-   TotalStatements  int
-   ParsedStatements int
- }
+ type StatementParseResult struct {
+   Index    int
+   RawSQL   string
+   Query    *ParsedQuery   // nil => IR conversion failure
+   Warnings []ParseWarning // statement-scoped warnings
+ }
+
+ type ParseBatchResult struct {
+   Statements       []StatementParseResult
+   TotalStatements  int
+   ParsedStatements int
+   HasFailures      bool
+ }

That shape is less convenient for quick demos, but much better for real integration.

Real SQL in the wild is much uglier than test fixtures

Open source usage also brought SQL shapes we did not have in internal tests:

trailing semicolons and odd whitespace
invalid syntax in the middle of an otherwise valid batch
mixed DDL + DML scripts
ONLY variants in DDL paths

The parser had to become resilient without becoming vague. That meant:

better statement-level warning attribution
explicit failure semantics (Query == nil)
tighter handling across DDL relation extraction paths

One concrete snippet (current behavior)

batch, err := postgresparser.ParseSQLAll(sql)
if err != nil {
    log.Fatal(err)
}

fmt.Printf("total=%d parsed=%d has_failures=%t\n",
    batch.TotalStatements, batch.ParsedStatements, batch.HasFailures)

for _, stmt := range batch.Statements {
    fmt.Printf("idx=%d failed=%t warnings=%d raw=%q\n",
        stmt.Index, stmt.Query == nil, len(stmt.Warnings), stmt.RawSQL)
}

That is the integration model people asked for: deterministic, inspectable, and boring in the best way.

Why this matters

Open source removed our ability to hand-wave edge cases.

The loop became:

implement
get challenged
simplify
lock behavior with tests
document the contract

That loop made postgresparser better than it would have been as an internal-only tool.
Internal tools can survive ambiguity. Public libraries cannot.

If you're building something on top of postgresparser, open an issue. Real-world SQL keeps improving the contract.

Building a Pure Go PostgreSQL SQL Parser (No CGO, No Server, No Runtime Dependencies)

Eitamos Ring — Mon, 09 Feb 2026 17:29:25 +0000

Why we built this

We needed PostgreSQL SQL parsing in environments where CGO was not an option:

Alpine containers
AWS Lambda
Distroless images
Scratch builds
ARM deployments
Anywhere CGO_ENABLED=0 is required

Most existing approaches either:

Depend on native Postgres parser bindings
Require CGO
Require running a Postgres server
Are too heavy for infrastructure tooling

So we built a pure Go PostgreSQL parser.

The goal

Not to replace Postgres parsing.

Not to be 100% server-compatible.

The goal was simple:

Give infrastructure and tooling systems structured query data safely and deterministically.

What it extracts

The parser outputs an intermediate representation (IR) with:

Tables (with aliases)
Columns
Joins
WHERE filters
GROUP BY
ORDER BY
CTEs
Subqueries

Example

result, err := postgresparser.ParseSQL(`
    SELECT u.name, COUNT(o.id) AS order_count
    FROM users u
    LEFT JOIN orders o ON o.user_id = u.id
    WHERE u.active = true
    GROUP BY u.name
    ORDER BY order_count DESC
`)

fmt.Println(result.Command)       // "SELECT"
fmt.Println(result.Tables)        // users, orders with aliases
fmt.Println(result.Columns)       // u.name, COUNT(o.id) AS order_count
fmt.Println(result.Where)         // ["u.active=true"]
fmt.Println(result.JoinConditions) // ["o.user_id=u.id"]
fmt.Println(result.GroupBy)       // ["u.name"]
fmt.Println(result.ColumnUsage)   // each column with its role: filter, join, projection, group, order

Now tooling can answer:

What tables does this query touch?
What joins exist?
What filters are applied?

Why ANTLR + Pure Go

We evaluated:

libpg_query bindings
WASM approaches
regex / string parsing
custom parsers

Tradeoffs we cared about

Requirement	Why
Pure Go	Simpler deploy, fewer runtime risks
No CGO	Works in restricted environments
Deterministic behavior	Important for tooling / analysis
Performance	Needed for production workloads

ANTLR gave us:

Mature grammar ecosystem
Strong parsing guarantees
Good performance with SLL mode

Performance

Most real-world queries parse in roughly:

~70–350 microseconds

(using SLL prediction mode)

Where this is useful

Typical use cases:

CI SQL validation
Query lineage hints
Migration safety checks
Static query analysis before deploy
“What tables does this service touch?” automation

Open Source

We’ve been using this internally for months and decided to open source it.

If you break it with weird SQL, please open issues — that’s how coverage improves.

👉 https://github.com/ValkDB/postgresparser

Why Database Indexes Keep Coming Up in My Performance Work

Eitamos Ring — Tue, 29 Jul 2025 06:44:05 +0000

I bounce between data pipelines, API fires, and new features all week, and there’s this one thing that keeps biting us. Slow pages. And 8 times out of 10 it’s the same root cause: we forgot the right index.

We had this analytics dashboard—we tuned the React, cached the API, CDN was spotless. Still slow. The query behind it? joining big tables and scanning like theres no tomorrow. No index on the join keys. Oops.

A quick demo to prove I’m not just ranting
I spun up a tiny test on my dev box: made an orders table, loaded 100k rows, timed a few queries. First with no indexes, then with some obvious ones.

CREATE TABLE orders ( id SERIAL PRIMARY KEY, customer_id INTEGER NOT NULL, order_date TIMESTAMP NOT NULL, total_amount DECIMAL(10,2) NOT NULL, status VARCHAR(50) NOT NULL, country VARCHAR(100) );
Before indexes (avg):

Customer lookup: 6.11 ms

Status filter: 8.47 ms

Date range: 6.73 ms

After indexes:

Customer lookup: 0.88 ms (~7x faster)

Status filter: 2.41 ms (~3.5x)

Date range: 1.48 ms (~4.5x)

Note: these are from my laptop which is also running Docker, two IDEs, Slack, and that Electron app we dont talk about… so, not a lab.

The classic slow page (you’ve seen this movie)
Admin page loads in 10 seconds, everyones pointing fingers. Frontend swears it’s fine, backend says “works on my machine”. The DB? doing full table scans through millions of rows because the where clause is on status and order_date and, yeah, neither is indexed.

Usual suspects

Foreign keys without matching indexes on the child table

Date columns everybody filters by (no index)

Status fields in every WHERE (also no index)

The tiny bit of code that tells the truth
Here’s how I timed the “customer orders” lookup in Go:

func testCustomerIDQuery(db *sql.DB, description string) { var total time.Duration for i := 0; i < numQueries; i++ { id := rand.Intn(100000) + 1 start := time.Now() rows, _ := db.Query(" SELECT id, customer_id, order_date, total_amount, status, country FROM orders WHERE customer_id = $1 LIMIT 10", id) if rows != nil { rows.Close() } total += time.Since(start) } fmt.Printf("%s avg: %v\n", description, total/time.Duration(numQueries)) }
And the “magic” is not magic, it’s just this:

CREATE INDEX idx_customer_id ON orders(customer_id);
CREATE INDEX idx_status ON orders(status);
CREATE INDEX idx_total_amount ON orders(total_amount);
CREATE INDEX idx_order_date ON orders(order_date);
CREATE INDEX idx_country ON orders(country);
CREATE INDEX idx_customer_date ON orders(customer_id, order_date);``
Add those, re‑run the exact same queries, and your 10‑second page quietly becomes sub‑second. It’s almost embarrassing how often that’s the fix.

Index size reality check
People ask “wont indexes be huge?”. From the same test (100k rows):

idx_customer_date: 3.1 MB

orders_pkey: 2.2 MB

idx_order_date: 2.2 MB

idx_total_amount: 2.2 MB

idx_customer_id: 1.9 MB

idx_country: 712 KB

idx_status: 688 KB

Call it ~12 MB total. For the speedup you get, thats cheap.

A couple gotchas (learned the hard way)
Composite order matters. (customer_id, order_date) helps WHERE customer_id = ? ORDER BY order_date DESC LIMIT 10. Flip it and you’ll be sad.

Check the plan. EXPLAIN (ANALYZE, BUFFERS)—you want Index Scan / Index Only Scan, not Seq Scan.

Not every column deserves an index. Super low selectivity (like a boolean) usually wont help with a plain b‑tree.

Writes pay the bill. Indexes speed reads, but inserts/updates get a bit slower—so pick the ones you actually use.

The boring checklist that works
Before you reach for sharding, a rewrite, or a shiny new DB:

Profile the slow endpoint.

EXPLAIN (ANALYZE, BUFFERS) the worst query.

If it’s scanning a big table, add the smallest useful index.

Re‑test. Ship. Sleep.

It’s not flashy. It won’t wow anyone at a meetup. But it’ll make your app feel fast, which is what users care about anyway.