How We Got an LLM to Draw Charts Without Ever Touching a Pixel

Let's get something out of the way first.

Having data is good. Having a database full of reviews, commits, and org activity sitting there quietly, untouched, unread, never once glanced at by a human being with a coffee and an opinion? That's not "having data." That's a very expensive data graveyard.

At LiveReview, we build what we call a Blast-Radius Aware AI Code Review for Business-Critical Systems.

Which is a fancy way of saying: we review your code, we figure out how bad it would be if a change goes wrong, and we don't shut up about it until someone fixes it.

Along the way we accumulate a review data: who reviewed, how much, how fast, how often, which repos are on fire.

And for a while, that pile just sat there.

Engineering leaders would ask "is adoption increasing?" and get back a vibe, not an answer.

So we built Livi, a chat bot that answers real questions about that data with real charts, not paragraphs of hedging.

This post technically about how Livi draws those charts.

Specifically: why we never let the LLM touch a pixel, how the same chart definition ends up as both a live interactive graph in your browser and a flat PNG in a Slack thread, and why teaching a language model to pick the right chart shape is a surprisingly deep rabbit hole.

The core decision: don't ask the LLM to draw, ask it to describe

The tempting, wrong idea is: "let's have the LLM generate an image." Please don't.

Image-generating models are a different beast entirely, and even if you got one to draw a bar chart, you'd have no way to verify the numbers on it are real.

You'd be trusting a model that hallucinates plausible-sounding review counts to also render them faithfully into pixels.

That's not a chart, that's chart-shaped fan fiction.

The actually good idea, and the one every serious LLM-charting integration eventually converges on, is: the LLM writes Vega-Lite, a JSON grammar for describing charts declaratively.

You don't say "draw a blue bar going up." You say:

{
  "mark": "bar",
  "encoding": {
    "x": { "field": "month", "type": "temporal" },
    "y": { "field": "review_count", "type": "quantitative" }
  }
}

That's it.

That's the entire chart.

No pixels, no drawing, just a description of what the data means and how it should be mapped to a picture.

Vega-Lite does the actual drawing.

The LLM's job shrinks down to something it's genuinely good at: filling in a well-defined schema.

Models are much better at "pick mark: bar or mark: line" than "hallucinate 600 pixels of a correct y-axis."

And critically: the LLM never sees the actual numbers before they're rendered.

It writes the SQL, we run it, and the real result set gets stitched into data.values by our own Go code.

The model can be as creative as it wants about presentation.

It gets zero creative license over the numbers.

What actually happens between "how many reviews last month" and a chart on your screen

Here's the pipeline, roughly:

A few things worth dwelling on here, because each one exists because something went wrong first.

Why two SQL-writing steps instead of one? Because the model needs to know how many rows the answer will have before it can decide whether a chart makes sense or whether it should hand you a CSV instead.

Nobody wants a bar chart with 4,000 bars.

So step one is basically "how big is this going to be," and step two is "okay, now actually get me the data and tell me how to draw it."

Why is there a SQL guard at all? Because an LLM writing raw SQL against a multi-tenant database is one confidently-worded prompt away from org_id = 1 OR 1 = 1.

We run every generated query through a guard that rejects anything that isn't read-only, checks every table against a denylist, and specifically looks for the shape of a tenant-isolation bypass (constant-vs-constant comparisons, bare OR TRUE, all the classics).

It's not glamorous work, but it's the difference between "cool AI feature" and "why is Org A looking at Org B's review data" showing up in an incident channel.

(meme placeholder: "Well Yes, But Actually No" / bike fall guy. Top text: "the query has an org_id filter." Bottom text (mid-fall): "WHERE org_id = 1 OR 1=1")

Why does the LLM only ever see a narrowed slice of the schema, not the whole database?

Because our actual schema has north of fifty tables, and dumping all of them into every prompt is both expensive and a great way to get the model confused about which created_at belongs to which table. More on this in a second, it deserves its own section.

Teaching the model which tables even exist: dbctx

Here's a problem that doesn't show up until your schema stops being a toy demo.

We're at 58 tables and counting: reviews, pull requests, AI comments, review feedback, billing, licensing, job queues, the works.

If we pasted the full schema into every single prompt, we'd be burning thousands of tokens per question just describing license_seat_assignments to a model that was asked "how many reviews happened last month" and does not, will never, care.

So instead of "here is the entire database, good luck," we use dbctx, a Go library built specifically for this problem: given a natural-language question and a live Postgres connection, hand back only the tables that actually matter, formatted as compact, LLM-friendly text instead of a raw information_schema dump.

shrsv / dbctx

Compile a PostgreSQL database into compact, queryable context.

dbctx

Compile a PostgreSQL database into compact, queryable context.

dbctx is a Go library and CLI tool that compiles a PostgreSQL database into a portable, queryable context index (.dtx file). It extracts schema, relationships, field semantics, representative values, JSONB structure, and builds a full-text search index — all from deterministic introspection, statistics, and heuristics, with no generative LLM and no external services required for the core index. It also supports an optional local semantic embedding signal and an optional, user-controlled terminology dictionary — both additive, both off-by-default-cost, described below.

Use it to give text-to-SQL systems, AI agents, and database-aware applications a compact, relevant slice of your database schema at query time, instead of dumping the entire information_schema into every prompt.

Key features:

• Natural-language query — find relevant tables, columns, and relationships from a text question
• Semantic retrieval (optional, on by default) — a local embedding model (BGE-small-en-v1.5, ~33M params…

View on GitHub

It does this with a genuinely layered retrieval pipeline, not just a keyword grep.

Lexical and fuzzy matching against table and column names, full-text search, matching against actual sampled values in the data, an optional semantic embedding pass, and a curated terminology layer we feed it ourselves (so "LOC" resolves to billable_loc, and "MR" and "PR" both resolve to the same underlying pull-request concept, because our users say both depending on which Git host they came from).

Whatever scores above zero gets pulled in, plus anything reachable through a foreign key from something that scored, because a join target with zero lexical overlap with the question is still often exactly what the query needs.

The payoff is not subtle.

On our real schema, a typical question narrows 58 tables down to somewhere around 25 to 30, which is roughly half the context gone before the model has written a single character of SQL.

That's not a rounding-error optimization, that's the difference between a prompt the model can actually reason clearly about and one where the important tables are buried in a wall of billing and license-seat noise.

The part where pixels finally show up: vl-convert

So now we have a Vega-Lite spec. Great.

If the user is on the web dashboard, we're basically done, more on that in a second.

But what about Slack? What about Discord? Those platforms don't run a JavaScript charting library inside a chat message.

A Slack message is not a browser tab.

You cannot politely ask Slack to interpret a Vega-Lite spec and render SVG for you.

So for anywhere that isn't our own frontend, we need an actual image file.

This is where vl-convert comes in: a Rust binary (with Python and Node bindings, but we shell out to the CLI) that takes a Vega-Lite spec and rasterizes it straight to PNG, no headless browser required.

That last part matters more than it sounds.

The old-and-busted way to render a chart server-side is to spin up a headless Chrome instance, load a page with a charting library, screenshot it, and pray your Docker image doesn't balloon to two gigabytes.

vl-convert skips all of that.

It's a single binary, it takes JSON in, it gives PNG bytes out, and it's fast enough that nobody notices it happening.

(meme placeholder: Kombucha Girl, disgusted-then-intrigued two-panel. Disgusted panel: "spin up headless Chrome to screenshot a chart." Intrigued panel: "one Rust binary, JSON in, PNG out.")

Same spec, two very different destinies

Here's the part I actually think is neat.

We generate one Vega-Lite spec per chart.

What happens to it next depends entirely on where it's going.

(meme placeholder: Trade Offer / Minecraft villager trading. Give: "one Vega-Lite spec." Take: "a live interactive chart in the browser, or a flat PNG in a Slack thread, depending on where it lands.")

On the web, the frontend just hands the raw spec to react-vega and lets the browser do the work.

You get hover tooltips, you get resizing, you get an actually interactive chart, and our backend does zero image rendering for that path.

It just ships JSON.

For Slack and Discord, the exact same spec gets routed through vl-convert instead, turned into a flat PNG, and attached as a file to the message.

The bot doesn't know or care that it's the "same" chart taking a different road.

From the pipeline's point of view, a Vega-Lite spec is just a Vega-Lite spec.

Where it ends up decides whether it becomes living, breathing SVG or a JPEG-adjacent screenshot sitting quietly in a chat thread forever.

This split is also why we can add new chart destinations cheaply.

Want an emailed weekly digest with embedded charts? Same spec, same vl-convert path, new delivery mechanism.

The hard problem (getting a correct, sensible chart spec out of an LLM) is solved exactly once.

Where this leaves us

The whole point of Livi was never "add a chatbot," it was "close the loop between the data we're already collecting and the person who actually needs to act on it."

A CTO asking "are engineers actually using this thing" should get an answer that looks like a calendar heatmap of usage rhythm, not a spreadsheet and a shrug.

The recipe, if you're building something similar, is genuinely not complicated:

1. Never let the model touch pixels. Let it write a declarative spec.
2. Never let the model see real numbers before you've run its query yourself.
3. Guard the query like you mean it, not like a vibes-based regex.
4. Pick one rendering pipeline (vl-convert, in our case) and let destination decide static-vs-interactive, not the model.
5. Budget real prompt space for chart taste. This is the part that actually takes iteration.

LiveReview reviews your code, tells you how bad a change could go, and won't shut up until it's fixed — and Livi turns all that accumulated review data into charts a human can actually act on.

If you liked the post, drop a ⭐ on the repo and try LiveReview now.