DEV Community: Ali Afana

I Told the AI "A Scanner Flagged This" — and It Agreed With Everything

Ali Afana — Thu, 27 Aug 2026 11:26:31 +0000

I gave two AI models the same 200 pieces of code, the same prompt, the same
question. One of them removed 51% of the false alarms. The other removed only
20% of the false alarms — and confirmed 90% of everything it was shown.

Same inputs. Same instructions. A 2.5× difference in the only thing I was
measuring.

The model that failed isn't a bad model. It's a well-regarded commercial model
from a frontier lab. It failed at this task for a specific, predictable reason:
I told it that a scanner had already flagged the code, and it believed me.

This article is about that failure mode, the four countermeasures I built to
fight it, and the uncomfortable finding that whether those
countermeasures work is mostly a property of the model, not the prompt.

Why my scanner asks an AI anything at all

Quick context if you're new to the series.

My scanner works in two stages. Fixed rules trace data flows through code and
find every place where user input reaches something dangerous — a database query,
a file open, a system command. That stage is deterministic: same code in, same
suspects out, every time.

The problem is that this stage over-reports, badly. It flags code like this:

String id = request.getParameter("id");
if (!id.matches("[0-9]+")) {
    throw new IllegalArgumentException("id must be numeric");
}
String sql = "DELETE FROM products WHERE id = " + id;
stmt.executeUpdate(sql);

Untrusted input genuinely does reach the SQL string. The path is real. But
matches("[0-9]+") means id can only ever be digits, so there's no attack. The
rules see the connection; they can't see the meaning.

So the second stage hands each flagged snippet to a language model and asks one
narrow question: is this actually exploitable?

That's the whole bet. And it has a flaw sitting right at the centre of it.

The flaw: I have to tell the model why it's looking

The prompt has to explain the situation. Here's the actual line from my
llm.py:

A static-analysis engine flagged the code below as a possible {vuln_class}
({cwe}). Decide whether it is a REAL vulnerability or a FALSE ALARM.

Read that as the model reads it. Before it sees a single line of code, it has been
told that an expert system already concluded something is wrong.

That's an enormous hint. And language models are trained, deliberately, to be
agreeable — reinforcement learning from human feedback rewards responses people
approve of, and people approve of agreement. The tendency is well documented
enough to have a name: sycophancy.

For most applications, agreeableness is harmless or even desirable. For mine it's
fatal. A judge that agrees with every accusation isn't a judge. It's a rubber
stamp with a token bill.

Worse, it fails invisibly. The pipeline runs, verdicts come back, findings get
reported — and the output is identical to having no judge at all. You'd only
notice by measuring, which is exactly what most people building these systems
never do.

Countermeasure 1: explicit permission to disagree

The first defence is the bluntest. From the RULES block of the real prompt:

RULES
- Judge ONLY the code shown. Never assume code you cannot see.
- confirmed=true only if attacker-controlled input reaches the dangerous sink
  with nothing neutralising it on the way.
- confirmed=false if the input is not attacker-controlled, never reaches the
  sink, or is neutralised (parameterised query, escaping, encoding, allow-list).
- Static engines raise false alarms routinely. Rejecting a finding is a correct
  and expected answer. Do NOT agree just because the engine flagged it.

That last rule is doing the heavy lifting. It does three things at once:

It states a fact about the world — static engines raise false alarms
routinely. This isn't an instruction, it's context that makes rejection
reasonable rather than contrarian.

It labels rejection as correct. Without this, the model may treat "no" as a
failure to be helpful. Models optimise hard for being helpful.

It names the specific failure — "do NOT agree just because the engine flagged
it." Naming the exact error is more effective than a general instruction to be
careful.

Note also what rule 2 and 3 do: they give concrete, checkable criteria for
both answers. Not "use your judgement" but "confirmed=false if... neutralised
(parameterised query, escaping, encoding, allow-list)." The model is being handed
a checklist, and "allow-list" is precisely what catches the matches("[0-9]+")
case.

Countermeasure 2: show both answers happening

Instructions describe. Examples demonstrate. The prompt includes exactly two
worked examples — and the balance between them is the point.

The first shows a genuine vulnerability being confirmed:

String id = req.getParameter("id");
stmt.executeQuery("SELECT * FROM t WHERE id = " + id);

{
  "reasoning": "getParameter is attacker-controlled. It is concatenated straight
                into the SQL string. Nothing escapes or parameterises it.",
  "confirmed": true,
  "severity": "Critical",
  "explanation": "An attacker can change the SQL statement and read or modify
                  arbitrary rows.",
  "fix": "PreparedStatement ps = conn.prepareStatement(\"SELECT * FROM t WHERE id = ?\"); ps.setString(1, id);"
}

The second shows a flagged finding being rejected:

String id = req.getParameter("id");
PreparedStatement ps = conn.prepareStatement("SELECT * FROM t WHERE id = ?");
ps.setString(1, id);
ps.executeQuery();

{
  "reasoning": "Source is attacker-controlled but bound with setString on a
                parameterised query. The driver sends it as data, never as SQL.",
  "confirmed": false,
  "severity": "Low",
  "explanation": "The query is parameterised, so the input cannot change the
                  SQL structure.",
  "fix": "No change needed."
}

Why this specific pairing matters:

Both examples start identically. Both begin with
req.getParameter("id") — attacker-controlled input, no question about it. The
difference is entirely in what happens next. The model can't shortcut on "does
this involve user input?" because both do.

The rejection is not a trivial case. I deliberately didn't use an obviously
safe example like a hardcoded string. I used a real flagged finding — one a static
engine would genuinely report — being correctly dismissed. That's the behaviour I
need, so that's what I demonstrate.

The reasoning models the right thinking. "The driver sends it as data, never
as SQL" is the actual security reasoning, stated compactly. The examples aren't
just teaching output format; they're teaching how to think about the question.

One-shot prompting with only the confirmation example would have been actively
harmful — it would have taught the model that the expected answer is yes.

Countermeasure 3: neutralise the retrieval layer

My scanner also retrieves real published vulnerabilities similar to the code under
review, and pastes them into the prompt as background. It's a genuinely useful
feature and a genuinely dangerous one.

Think about what's happening: I'm about to ask "is this code vulnerable?" and
immediately before asking, I'm showing the model three real, confirmed
vulnerabilities that look like it. That's textbook priming.

So the retrieved block carries an explicit disclaimer:

KNOWN RELATED ADVISORIES — real-world {vuln_class} reports, for grounding only.
They do NOT prove the code below is vulnerable; judge it on its own merits and
reject it if it is safe.

And one implementation detail that matters more than the wording: when nothing
is retrieved, the prompt is byte-for-byte identical to the no-retrieval version.
Not "similar" — identical. That's what makes the A/B test in a later article
honest. If the two prompts differed even in whitespace, I couldn't attribute any
measured difference to retrieval itself.

Countermeasure 4 (the structural one): reason before you rule

This one isn't in the prompt text at all — it's in the response schema:

_RESPONSE_SCHEMA = {
    "type": "OBJECT",
    "properties": {
        "reasoning":   {"type": "STRING"},
        "confirmed":   {"type": "BOOLEAN"},
        "severity":    {"type": "STRING", "enum": ["Critical", "High", "Medium", "Low"]},
        "explanation": {"type": "STRING"},
        "fix":         {"type": "STRING"},
    },
    "required": ["reasoning", "confirmed", "severity", "explanation", "fix"],
}

reasoning is declared first. Since the model generates the JSON in order, it
must produce its analysis before it emits the verdict.

Flip those two fields and the dynamic reverses: the model commits to confirmed: true and then generates text justifying a decision it has already made. Same
model, same prompt, worse answers — purely from field order.

The reasoning is capped at three sentences, incidentally, for a boring reason I'll
cover in a later article: an earlier model spent its entire 8,192-token budget
"thinking" and returned nothing at all.

Did any of it work?

Here's where I have to be careful, because this is exactly the kind of claim
people make without evidence.

On my small test project, the scanner flags 6 candidates, of which 2 are
sanitised-but-flagged cases. The judge rejected exactly those 2. Its verbatim
reasoning on the SQL case:

"The input 'id' is validated against a numeric-only regex '[0-9]+' before being
concatenated into the SQL query. This allow-list guard prevents any SQL
injection characters from reaching the sink."

And on the XSS case:

"The source is request.getParameter('comment'). The input is passed through a
strip() method that uses a regex to allow only alphanumeric characters and
spaces. This effectively neutralizes any XSS payload."

That's correct security reasoning, arriving at "no" on a finding an engine
flagged. The countermeasures appear to be working.

At benchmark scale, on 200 stratified cases from the OWASP Benchmark:

	result
False alarms removed	50 of 98 (51%)
Real bugs lost	2 of 98 (2%)
Precision	0.50 → 0.67
95% CI	[0.43–0.57] → [0.59–0.74], non-overlapping

Half the false alarms gone, at a cost of 2% of the real vulnerabilities, and the
confidence intervals don't overlap — so the improvement isn't sampling noise.

That's the result I wanted.

And then I changed one thing

I ran the identical experiment with a different model. Same 200 candidates. Same
code slices. Same prompt, character for character.

Judge	Confirms	False alarms removed	Precision
`gemma-4-31b-it` (31B, open weights)	73%	51%	0.50 → 0.67
`gpt-4o-mini` (commercial)	90%	20%	0.50 → 0.55

gpt-4o-mini confirmed 90% of everything put in front of it. Its precision
improvement — 0.50 to 0.55 — has confidence intervals that overlap with doing
nothing at all. Statistically, I cannot distinguish it from having no judge.

All four countermeasures were present in both runs. One model followed them. The
other largely didn't.

The obvious objection

"You compared against the cheap mini model. Of course it lost."

Fair. So I ran the identical 200 candidates — same slices, same prompt,
character for character — through gpt-4o, the frontier sibling.

Judge	Confirms	False alarms removed	Real bugs lost	Precision
`gemma-4-31b-it` (31B, open weights)	73%	51%	2%	0.50 → 0.67
`gpt-4o` (frontier)	80%	40%	0%	0.50 → 0.62
`gpt-4o-mini` (small commercial)	90%	20%	1%	0.50 → 0.55

Capability does matter within a family: gpt-4o removed twice the false alarms
of its little sibling, and it was the only judge that kept every single real
bug. But it still finished behind a free, mid-size open model, and its
confidence interval ([0.55–0.70]) still touches the no-judge interval
([0.43–0.57]). Gemma remains the only judge whose improvement is statistically
separated at this sample size. The leaderboards predicted the mini → 4o step.
Nothing predicted the open model on top.

Going deeper (skip if you just want the lesson)

First: does the headline survive the scanner's own later fixes? After this
experiment ran, the discovery stage gained its final recall improvements, so I
re-ran the same protocol on the finished pipeline. The result reproduces —
50 of 97 false alarms removed (52%) at the same 2% real-bug cost, precision
0.51 → 0.67, interval still separated. I cite the original run throughout
because it's the one all three judges shared.

I also ran gpt-4o-mini over the entire candidate set — 4,356 of 4,357
judged, at 148 judgments/minute for about $1.60 — to check whether the sample
was misleading me. It wasn't: at full census it removed 111 of 611 false alarms
(18%) and lost 3 of 743 true positives (candidate-level counts from an earlier
run, before the final recall fixes landed). The intervals still overlap.

It's worth being precise about what this does and doesn't show. It does not
show that gpt-4o-mini is a worse model in general — it's faster, cheaper per
token, and better at plenty of things. It shows that on this task, with this
prompt, it is much more likely to agree with a premise handed to it.

I'd also caution against over-generalising from n=3 models. What I can defend is
narrow: the spread between three reasonable choices was large enough to
dominate every other engineering decision I made, and the public benchmark
scores predicted only part of the ordering — the step up inside the OpenAI
family, not the open model finishing first.

What I learned

1. If your prompt asserts something, measure whether the model just agrees.
This applies far beyond security. Any time you write "the system detected X, is
this correct?" or "the user reported Y, is that plausible?", you've handed the
model a conclusion. The polite output you get back may be pure echo. The only way
to know is to feed it cases where the right answer is "no" and count.

2. Prompt engineering has a ceiling set by the model. I spent real effort on
those four countermeasures and I'd write them the same way again — the model that
follows them produces a statistically solid result. But identical instructions
produced a 2.5× difference in outcome. The prompt is necessary; it isn't
sufficient.

3. For judgement tasks, skepticism beats capability. Moving up a capability
tier helped — gpt-4o doubled its sibling's false-alarm removal at zero recall
cost. But the property that made Gemma the best judge here isn't reasoning power
or knowledge. It's willingness to contradict a premise supplied in the prompt.
That trait doesn't appear on any leaderboard I know of, which means you cannot
pick a judge by reputation — you have to measure it on your own task.

4. Design the schema, not just the prose. Putting reasoning before
confirmed cost me nothing and changes the model's process. Field order is
prompt engineering.

Next in this series: the OWASP Benchmark — 1,478 test cases, 701 of them built
specifically to trick tools like mine.

I'm Ali Afana — AI builder and security researcher, writing from Gaza. I
build systems in public, measure them against ground truth, and keep the
receipts. This scanner is one project on a longer road — follow for what
comes next.

GitHub · X · LinkedIn

What Is a Vulnerability, Really? Source, Sink, and Taint

Ali Afana — Thu, 20 Aug 2026 20:22:58 +0000

Two Java methods. One of them will let an attacker delete your entire products
table. The other is completely safe.

public int deleteA(HttpServletRequest request, Connection conn) {
    String id = request.getParameter("id");
    String sql = "DELETE FROM products WHERE id = " + id;
    return conn.createStatement().executeUpdate(sql);
}

public int deleteB(HttpServletRequest request, Connection conn) {
    String id = request.getParameter("id");
    if (!id.matches("[0-9]+")) {
        throw new IllegalArgumentException("id must be numeric");
    }
    String sql = "DELETE FROM products WHERE id = " + id;
    return conn.createStatement().executeUpdate(sql);
}

Four lines different. If you can already see which is which and why, you know
more security than most working developers.

If you can't — that's what this article is for. By the end you'll be able to look
at almost any injection vulnerability and describe exactly what's wrong with it,
using three words.

I'm building an automated vulnerability scanner, in public. Those three words
are the entire foundation it's built on.

The three words

Nearly every common web vulnerability has the same shape:

The program takes input from the user and uses it somewhere dangerous,
without cleaning it first.

Three terms capture that:

SOURCE — where outside input enters your program. request.getParameter("id")
reads something a user typed. Since anyone can be that user, you must assume the
worst possible value.

SINK — an operation that becomes dangerous with the wrong input.
executeUpdate(sql) hands a string to your database and says "run this."

TAINT — the idea that untrusted data stains everything it touches. Input
arrives tainted. Copy it into a variable, that variable is tainted. Concatenate it
into a bigger string, the whole string is tainted. The stain spreads.

The tap and the glass

Picture a tap that might be running dirty water — that's your source. Picture
a glass you're about to drink from — that's your sink. Taint is the dirt.

A vulnerability is when dirty water flows from the tap to the glass with no
filter in between.

That's it. That's the whole model.

Now look at the two methods again:

deleteA — tap → glass, nothing in between. Vulnerable.
deleteB — tap → filter → glass. The matches("[0-9]+") check rejects anything that isn't pure digits. Safe.

What the attack actually looks like

For deleteA, a normal request sends id=42:

DELETE FROM products WHERE id = 42

Fine. One product deleted.

Now an attacker sends id=42 OR 1=1:

DELETE FROM products WHERE id = 42 OR 1=1

1=1 is true for every row. Your entire products table is gone.

The attacker didn't break into anything. They typed text into a field you gave
them. Your code took that text and made it part of a command.

That's the thing worth sitting with: injection bugs aren't about breaking in.
They're about your program treating a stranger's text as instructions.

In deleteB, 42 OR 1=1 fails the [0-9]+ check and the method throws before
any SQL is built. Same tap, same glass, but the filter catches the dirt.

The same shape, four different bugs

Here's why this model is worth learning: once you see it, four of the most common
vulnerability classes collapse into one idea with different taps and glasses.

Vulnerability	The dangerous sink	What an attacker gets
SQL injection (CWE-89)	a database query	reads or destroys your data
Command injection (CWE-78)	running a system command	runs any program on your server
Path traversal (CWE-22)	opening a file by name	reads files they shouldn't see
XSS (CWE-79)	writing into a web page	runs code in your other users' browsers

Different sinks, identical shape. Untrusted input reaches a dangerous operation
with nothing neutralising it on the way.

What's a CWE number? The Common Weakness Enumeration is a worldwide
catalogue that numbers every type of software weakness. "CWE-89" means SQL
injection everywhere on earth. It matters because it lets different tools,
written by different companies, talk about the same bug — which is what makes
the tool comparison later in this series possible at all.

Path traversal, for example, is the same story with files:

String name = request.getParameter("file");
File f = new File("/var/data/" + name);     // sink

Send file=report.pdf and you read a report. Send file=../../../etc/passwd and
you walk up out of the directory and read the system password file. Tap, glass,
no filter.

Why this is hard to automate

Here's where my project starts.

A scanner can trace the path — that's mechanical. Follow the data from
getParameter through every variable it touches until it reaches
executeUpdate. If a path exists, flag it. That technique is called taint
analysis and it's what most security scanners do.

But run that on the two methods at the top, and it flags both.

Because in deleteB, the data genuinely does flow from source to sink. The
matches("[0-9]+") line doesn't break the path — id is still the same variable,
still reaching the same query. What that line changes isn't the route, it's the
meaning: after it, id can only be digits, so the attack is impossible.

A path-tracing tool sees connectivity. It cannot see meaning.

So real scanners face an ugly choice: hand-code a list of every function that
counts as a filter (impossible to keep complete — every library has its own), or
report the flow anyway and let developers sort it out.

Most choose the second. That's why security tools have a reputation for crying
wolf.

Going deeper (skip if you just want the lesson)

The industry's own test suite makes this concrete. The OWASP Benchmark is a set
of Java test cases with known answers, used to grade scanners. In the four
categories I detect, it contains 1,478 cases: 777 real vulnerabilities and
701 that are deliberately built to look vulnerable while being safe — exactly
the deleteB pattern, using real filters like numeric allow-lists, character
stripping, and encoders.

Forty-seven percent of the test set exists purely to punish tools that can't
tell meaning from connectivity. That ratio isn't an accident — the benchmark's
designers made exactly this distinction the core test.

What my scanner does about it

The approach I'm testing: let the mechanical part stay mechanical, and hand the
meaning question to something that can read code.

Fixed rules trace every path from source to sink — reproducibly, missing nothing,
and deliberately over-reporting. Then a language model looks at each flagged
snippet on its own and answers one question: is this actually exploitable?

On my test project it flagged 6 candidates and the AI rejected 2 of them. Here's
its verbatim reasoning on the deleteB-style case:

"The input 'id' is validated against a numeric-only regex '[0-9]+' before
being concatenated into the SQL query. This allow-list guard prevents any SQL
injection characters from reaching the sink."

That's the judgement the path-tracer structurally cannot make. Whether that holds
up at scale — across 1,478 cases, measured against Semgrep and CodeQL — is what
the rest of this series is about.

What I learned

1. Most vulnerabilities are one sentence. Untrusted input reaches a dangerous
operation without being neutralised. Learn to spot the tap, the glass, and whether
there's a filter, and you can read most security findings without memorising bug
categories.

2. "There's a path" and "there's a bug" are different claims. This trips up
beginners and tools alike. The path is necessary but not sufficient. Always ask
what happens to the data on the way.

3. The hard part of automation is never the searching. Tracing paths is
mechanical. Deciding whether a filter actually filters is judgement — and that
distinction shaped my entire project.

Next in this series: what happened when I told the AI a scanner had flagged the
code — and it agreed with everything.

GitHub · X · LinkedIn

Don't Let the AI Find Your Bugs. Let It Judge Them.

Ali Afana — Thu, 13 Aug 2026 22:44:53 +0000

My vulnerability scanner flagged this Java method as SQL injection:

String id = request.getParameter("id");        // user input
if (!id.matches("[0-9]+")) {                   // digits only, or throw
    throw new IllegalArgumentException("id must be numeric");
}
String sql = "DELETE FROM products WHERE id = " + id;
stmt.executeUpdate(sql);                       // flagged as SQL injection

Look at line 2. The input must be pure digits or the method throws. You cannot
inject SQL through [0-9]+. There is no attack here. The scanner flagged safe code.

Here's the part that sounds wrong: I designed it to do that. The false alarm is
not a bug in my scanner. It's the plan.

Let me explain, because this decision is the entire foundation of the scanner —
and I think it's the decision most people building "AI security tools" right now
are getting backwards.

The Context

I'm building an LLM-based vulnerability scanner — in public, like everything I
ship. The pitch is simple: AI writes a lot of code now, fast, often for people
who aren't security experts. That code ships with holes. Someone has to find
them.

So the obvious idea — the one I started with — is the one you've seen in a hundred
launch posts this year:

"Point an AI agent at your codebase. It reads everything and finds your
vulnerabilities."

I killed that idea before writing a line of code. Not because it doesn't sound
amazing. Because I couldn't defend it with numbers — and the reasons it can't be
measured are the same reasons you shouldn't trust it in production.

New to this? Two words carry the whole article. A source is where
untrusted input enters your program (request.getParameter). A sink is
where it becomes dangerous (executeUpdate — running a database command). A
vulnerability is untrusted data reaching a sink without being cleaned on the
way. That's it; everything below builds on those two words.

The Problem With "Let the AI Find the Bugs"

Ask an LLM to audit a codebase and you hit three walls.

Wall 1: You get a different answer every run. LLMs are probabilistic. Same
repo, same prompt, run it twice — the bug lists don't match. Which run goes in
the security report? Which one do you benchmark? When a tool's output changes
between runs, you can't measure it, you can't compare it to Semgrep or CodeQL,
and you can't do science with it. You can only do demos.

Wall 2: Searching is the thing LLMs are bad at. Careful, exhaustive
enumeration over a large space — visit every file, check every call, miss
nothing — is exactly what LLMs don't do. They skim. They fixate. They get bored
in the middle of a long file (position bias is real). A scanner that "usually
checks most of the code" is not a scanner.

Wall 3: They invent. In my other project, an AI chatbot invented products
that didn't exist in the store. An AI bug-hunter does the same thing with
vulnerabilities — confident reports about code that isn't there. Now your
security tool hallucinates CVEs. Great.

If you've read my earlier article about the chatbot that lied to customers,
you know where this goes: you can't prompt your way out of a structural problem.
This is the same disease in a different body.

The Problem With the Opposite: "Just Use Rules"

Fine — so use deterministic tools. Pattern rules, taint tracking. Semgrep and
CodeQL have done it for years.

But rules have their own wall, and my flagged-but-safe method above IS that wall.

A taint tracker follows data: user input enters at getParameter (the source),
travels through variables, and reaches executeUpdate (the sink — the
dangerous operation). Path exists → alarm. That's the whole trick, and it's a
good trick. It's deterministic, fast, and it never gets bored.

But look at the method again. The data DOES flow from source to sink. The taint
tracker is not wrong about the flow. It's wrong about the meaning — the
matches("[0-9]+") guard makes the flow harmless, and understanding that
requires understanding what the code means, not just where the data goes.

Rules can't read meaning. So real-world scanners over-flag, developers drown in
false alarms, and — every security team knows this story — they stop reading the
reports. The industry's own benchmark makes the point better than I can: the
OWASP Benchmark (2,740 labeled Java test cases, the standard test set for Java
scanners) contains 701 cases in my scanner's categories that are deliberately
designed to trick tools into false alarms. Nearly half the test set exists just
to punish scanners for not understanding meaning.

So:

AI-only scanning: understands meaning, can't search reliably, can't be measured.
Rules-only scanning: searches perfectly, reproducibly — and can't understand meaning.

Everyone's strength is the other one's weakness. You can see the answer coming.

The Real Solution: A Courtroom

Police detectives don't sentence anyone. Judges don't collect evidence. The
system splits the work on purpose: exhaustive, procedural search by one party;
careful judgment of each individual case by another.

That's the architecture:

Deterministic rules do ALL the searching. The LLM only judges what the rules
found. A memory of real-world bugs helps it judge better.

Concretely, the pipeline looks like this:

source code
   → Joern builds a Code Property Graph        (code → queryable data-flow graph)
   → fixed taint queries find candidates       (same input = same output, always)
   → slicer cuts a ~small snippet per candidate (the evidence file)
   → RAG fetches similar known CVEs            (past cases, from GitHub advisories)
   → LLM judges ONE candidate at a time        (real bug? severity? fix?)

The detective is Joern — an open-source static analysis tool
that turns code into a graph you can query. My whole detection layer is a small
table of rules. Adding a vulnerability class is one row:

case class VulnClass(
    name: String,                // "sql-injection"
    cwe: String,                 // "CWE-89"
    sinkNames: String,           // which call is dangerous
    sinkMethodFullName: String,  // narrowed to the right type
    sinkArg: Int,                // which argument carries the taint (0 = any)
    sinkReceiver: String = ""    // fallback when the type can't be resolved
)

val classes = List(
  VulnClass("sql-injection", "CWE-89",
    "executeQuery|executeUpdate|executeLargeUpdate|execute|addBatch|prepareStatement|prepareCall|nativeSQL",
    ".*(java|javax)\\.sql\\..*", 1),
  VulnClass("command-injection", "CWE-78", "exec",           ".*Runtime.*",        0),
  VulnClass("command-injection", "CWE-78", "<init>|command", ".*ProcessBuilder.*", 0),
  VulnClass("path-traversal", "CWE-22", "<init>",
    ".*java\\.io\\.(File|FileInputStream|FileOutputStream|FileReader|FileWriter|RandomAccessFile)\\.<init>.*", 0),
  // ... 3 more rows (JdbcTemplate SQLi, Paths.get, XSS)
)

// for each row: is there a data-flow path from source to sink?
sink.reachableByFlows(source)

(Sources — getParameter, getHeader, getCookies and the rest of the HTTP
request surface — are shared by every class, so they live in one regex above the
table rather than being repeated on each row.)

One line — reachableByFlows — is the entire detector. It over-flags by design
(it flagged our digits-only method), and that's fine. Detectives are supposed
to bring in every plausible suspect. Their job is to miss nothing.

The judge is Gemma (via Google's API), and this is the important part: it
never sees the codebase. It sees one small slice of code — the few lines the
tainted data actually touched — plus a strict question. Its answer is forced
into a schema (the API guarantees the JSON shape):

{
  "reasoning":   "≤ 3 sentences, generated FIRST, before the verdict",
  "confirmed":   true,
  "severity":    "Critical | High | Medium | Low  (CVSS bands)",
  "explanation": "the concrete risk",
  "fix":         "corrected code"
}

Judging one small snippet is the task LLMs are genuinely good at. There's
nothing to search, nothing to miss, nothing to invent — the evidence is the
whole context. And because discovery is frozen, the same suspects go to the
judge every run. The randomness is contained to the one layer where a second
opinion is the point.

(One prompt detail that gets a full article later in the series: you have to actively stop
the judge from agreeing with the detective. Tell a model "a scanner flagged
this" and it wants to say yes. The prompt explicitly states that rejecting a
finding is a correct and expected answer — otherwise the judge just rubber-stamps
every arrest, and you've built an expensive echo.)

Results So Far

I test on a small OWASP-style sample set: 8 methods — 4 real vulnerabilities
across SQL injection, command injection, path traversal, and XSS; 2 methods that
are flagged by the rules but actually sanitized (the digits-only method
above is one); and 2 genuinely safe methods.

Stage	Result
Discovery (rules)	found all 6 flows: the 4 real + the 2 sanitized traps. Silent on the 2 safe methods.
Judge (an earlier, smaller Gemma)	6/6 correct — confirmed the 4 real (with CVSS severities), rejected both sanitized false alarms

That rejection line is the thesis in one row. The deterministic layer cannot
recognize a sanitizer. The LLM can. Division of labor works.

And the architecture scales further than I expected for something run from a
laptop: on the full OWASP Benchmark, Joern built the graph over 2,766 files in
100 seconds and ran the full query library — all seven rows — in 9 seconds. Deterministic discovery
is cheap. You spend the expensive, slow, probabilistic resource — the LLM — only
on the suspects.

Then I ran it against the real thing — the full OWASP Benchmark, 1,478 labeled
cases in my four categories, scored against Semgrep and CodeQL by one program
reading everyone's SARIF:

tool	precision	recall	F1
CodeQL	0.65	1.00	0.78
Semgrep	0.60	0.86	0.70
my rules alone	0.56	1.00	0.72
my rules + the judge (sample-derived)	0.67	0.98	~0.79

The judge removed 52% of the false alarms and cost 2% of the real bugs
in the latest verified run. One note before you check my arithmetic: the judged
row is measured inside a 200-case stratified sample that deliberately
over-samples false alarms (discovery precision there starts at 0.51, not 0.56)
— so 0.67 is a measured number, not "614 minus 52%". That caveat ships with the
number.
XSS went from 0.50 to 0.78 precision — those are the ESAPI.encodeForHTML()
traps, exactly the "is this actually safe?" question a taint engine can't answer.

My rules find every one of the 777 real vulnerabilities — the same recall as
CodeQL, from that seven-row table. But read the precision column before you get
excited: CodeQL reaches perfect recall with 427 false positives, and I need 614
to do it. The difference is years of hand-built sanitizer knowledge that a
seven-row table doesn't have.

Honesty section, because this is an engineering log and not a launch post:
CodeQL beats my rules layer on F1, 0.78 to 0.72, and I'd rather say that than
hunt for a framing where I win. The claim I can defend is narrower and more
interesting — a seven-row rule table plus a 31B open model lands in the same F1
range as a mature commercial engine, and the judge is what closes the precision
gap.

What I Learned

1. Give the probabilistic system the judgment task, never the coverage task.
Any job where "misses nothing, same answer every time" matters — searching,
enumerating, auditing — belongs to deterministic code. The LLM gets the job
where meaning matters and the input is small. This split applies way beyond
security: it's the same reason my chatbot searches the product database with SQL
and only lets the LLM phrase the answer.

2. If you can't re-run it, you can't measure it. Deterministic discovery
means every experiment is repeatable: same code in, same candidates out, and any
change in results traces to the one layer I changed. The moment discovery is
probabilistic, comparisons against other tools become vibes.

3. Over-flagging is fine if someone competent reviews the flags. I stopped
trying to make the rules smart. Rules that catch everything + a judge that
understands meaning beat clever rules with no judge. Design each layer to fail
in the direction the next layer can fix.

The academic version of this argument exists too — a 2025 paper called LLMxCPG
(arXiv 2507.16585) pairs Code Property Graphs with LLM judgment the same way,
which told me the instinct wasn't just mine. The part I hadn't seen done in
public is the one above: running it against the incumbents on the industry's own
benchmark and publishing every number, including the losses.

This is the first article in a series where I do that in the open. Next: the
three words that explain almost every injection bug — source, sink, and taint —
written for anyone who's never done security work. After that, the false-alarm
problem in detail, and what happened when I gave the same 200 code snippets to
three different AI judges — including why the frontier model didn't win.

GitHub · X · LinkedIn

My Scanner Missed 93% of the Bugs — and That Was the Right First Result

Ali Afana — Thu, 06 Aug 2026 21:41:21 +0000

The first time I ran my vulnerability scanner against the industry-standard
benchmark, the bottom line of the scorer's report was this:

$ python scripts/score_benchmark.py --findings out/java.findings.json \
        --truth benchmark-java/expectedresults-1.2.csv

OVERALL   precision 0.60   recall 0.07   F1 0.13    # abridged

Three numbers, and here is what each one means. Precision 0.60 — of all
the alarms the scanner raised, 60% pointed at real bugs: when it spoke, it
was right more often than not. Recall 0.07 — of all the real bugs in the
benchmark, it found 7%. In the four vulnerability classes my scanner covers,
the benchmark contains 777 real, labeled vulnerabilities; it missed 93% of
the bugs it exists to find. F1 0.13 — precision and recall combined into
one score (their harmonic mean), dragged down to almost nothing by that
recall.

My first instinct was to fix it before anyone saw it. Instead I saved the
output, wrote the number into my benchmark log, and kept it — because that
number was always going to be published, and this is the article that
publishes it.

The Context

For the past months I've been deep in AI — reading, building, measuring. One
of the projects that came out of it is an AI vulnerability scanner. The
design in one sentence: deterministic static-analysis rules do
all the searching, and an LLM judges each finding — is this a real bug or a
false alarm? The full architecture gets its own article. This one is about
the first measured number.

The test set is the OWASP Benchmark — 2,740 labeled Java test cases, the
standard exam for Java security scanners. In my scanner's four vulnerability
classes (SQL injection, command injection, path traversal, XSS) there are
1,478 cases: 777 real vulnerabilities and 701 cases deliberately designed to
bait scanners into raising false alarms. Every tool I compare against —
Semgrep, CodeQL — takes the same exam, scored by the same scoring code. Same
rules for everyone.

New to this? Three words carry this article. A source is where
untrusted input enters a program (request.getParameter("id") — anything an
attacker can type). A sink is where that input becomes dangerous
(executeUpdate(sql) — running it as a database command). A vulnerability
is data flowing from a source to a sink without being cleaned on the way;
that flowing data is called tainted, and tracking it is taint
analysis — the scanner's whole job is finding those flows. Already know
all that? Skip this box.

Why 0.07 Was the Right First Result

Here's what I had actually run: a spike. A deliberately minimal first version —
one source pattern, getParameter, wired to a handful of sinks, pushed
end-to-end through the whole pipeline: parse 2,740 test cases into a code
graph, run the taint queries, emit findings, score them against the answer key.

The spike's job was never to score well. Its job was to answer one simple
question: does the machinery work at all? And the ugly number, read
carefully, answered it:

Precision 0.60 — when the scanner did raise an alarm, it was usually right. The taint engine was tracking real flows correctly.
Recall 0.07 — it was blind to 93% of the bugs. The engine wasn't broken; its vocabulary was tiny. I was listening at one door of a building with many doors.

That's not a broken idea. That's a correct, simple diagnosis of a too-narrow
source list — delivered before I had invested weeks in the wrong layer. If the
first number had been precision 0.10, I'd have had an engine problem, which is
a rebuild. A recall problem is a list problem. Lists are fixable.

Why the Obvious Moves Are Both Wrong

Obvious move #1: don't tell anyone. Fix it quietly, publish only the final
number, look competent. Almost everyone building in public does a version of
this — the "overnight" success graph that starts at the first good result.

The problem: every result in this series is a number from my own benchmark
runs. There is no referee here — no third party checks my work before it
ships; it is me, a scorer script, and you. A reader has exactly one way to
judge numbers like that: the author's track record with results that hurt
him. If every number I show you is a win, you have no reason to trust any of
them. So the bad numbers ship too — and they ship first. I'm going to publish
a head-to-head against Semgrep and CodeQL later, and when I claim a result
there, I want the reader thinking "this is the person who published his own
0.07." Honesty is not a virtue here; it's infrastructure.

Obvious move #2: carpet-bomb the rules. Recall too low? Add patterns!
Match more names, loosen the regexes, taint everything — recall will climb. It
will also destroy precision, and worse: after twenty simultaneous changes you
cannot say which one did what. You've traded a measured system for a vibes
system.

What I did instead was slower and duller: read the benchmark's actual code,
find what it really calls, add sources in order of how often the code uses
them, and re-measure after every change. One variable at a time, one
number per change.

The Climb

Fix 1: Learn the benchmark's vocabulary — recall 0.07 → 0.83

I surveyed which input methods the benchmark's code actually uses, counting
files: getRequestURI in 724 files, getCookies in 664, getParameter in
538, getParameterValues in 510, getHeaders in 400, and on down the HTTP
request surface. My spike had covered exactly one entry in that list.

So sources became one shared definition — a single regex over fully-qualified
method names, so each getter is bound to the type that makes it
attacker-controlled. Condensed here; the full rule table gets its own article:

// Queries run on Joern, an open-source code-analysis engine (Scala DSL).
".*(HttpServletRequest|ServletRequest)\\.(getParameter|getParameterNames|getHeader|" +
  "getHeaders|getCookies|getQueryString|getRequestURI|getInputStream|…)\\b.*" +
"|.*Cookie\\.(getValue|getName)\\b.*" +
"|.*SeparateClassRequest\\.(getTheParameter|getTheValue)\\b.*" // the benchmark's request wrapper

(.*Cookie\.getValue.* matches only the cookie's getter — a bare match on
getValue would match every getValue in existence.)

Two more problems were hiding inside this step, and they were different
problems:

117 of the benchmark's real SQL injection cases never touch java.sql — they go through Spring's JdbcTemplate instead. One new sink row took SQLi recall from 0.57 to 0.86.
6,060 XSS sink calls were invisible for a reason outside my rules. Without the servlet library on the analysis path, the engine cannot work out what type response.getWriter() returns, so those calls could never match a type-based .*Writer.* pattern. The fix: also accept a sink when the receiver text — the response.getWriter() part as literally written in the code — matches getWriter|getOutputStream. That single change took XSS recall from 0.03 to 0.73.

New score: precision 0.53, recall 0.83.

Fix 2: 97 of the 130 remaining misses shared one missing source — 0.83 → 0.95

There were still 130 real bugs missing. I diffed the misses against the
benchmark code, and 97 of them — three quarters of everything left — took
their taint from a single method I hadn't listed: getParameterNames().

It's easy to see why it gets skipped. getParameter("id") returns a value
the user typed — obviously dangerous. getParameterNames() returns the
parameter names — and names feel like structure, not data. But the client
chooses the names too. ?<script>alert(1)</script>=x is a query string any
client can send, and then the name is exactly as attacker-controlled as the
value.

One name in a regex. Adding it recovered 93 real vulnerabilities on the spot:
recall 0.83 → 0.95. (The other 4 of the 97 were also blocked by a second,
separate problem — they return in Fix 3.) And here's the part that still
bothers me: nothing ever crashed, warned, or looked wrong. A missing source
fails silently. Without a labeled benchmark I would never have known.

Fix 3: The taint bridge — 0.95 → 1.00

That left 37 misses — the 33 that never used getParameterNames, plus the 4
from Fix 2 that had a second problem — and every single one contained
.split(...). Instead of guessing, I measured the taint chain link by link
on one failing case: the variable being split was reachable from the source.
The split call itself — reachable. The array-index access on its result,
param.split(" ")[0] — not reachable. Taint flowed correctly through
split and died at the index operation. The engine ships a default rule for
that operator, and overriding it changed nothing — the gap was in how the
engine applies rules to that operator, not in anything I could configure
away.

So I built a bridge, with one condition that keeps it honest: an index access
over a split-style call is promoted to an additional source only when the
array it indexes is itself reachable from a real source. Indexing an
untainted array stays untainted — that condition is the difference between a
targeted fix and blanket over-tainting. All 37 misses recovered, at a cost of
exactly three new false positives and about 84 seconds of extra work per
scan — a fix that raised recall and precision at the same time.

The Numbers

The whole climb, one measured change at a time:

stage	precision	recall	F1
spike (`getParameter` only)	0.60	0.07	0.13
+ broadened sources, receiver-text sinks	0.53	0.83	0.65
+ `getParameterNames`	0.55	0.95	0.70
+ index-access taint bridge	0.56	1.00	0.72

Zero false negatives. All four classes land at recall 1.00 — SQL
injection 272 of 272 real bugs found, command injection 126 of 126, path
traversal 133 of 133, XSS 246 of 246 — the same recall CodeQL achieves on the
same 1,478 cases, scored by the same code. From a seven-row rule table, with
no build step.

Now the uncomfortable part, because this is an engineering log and not a
launch post. Precision 0.56 means 614 false alarms, and this layer falls into
88% of the benchmark's designed traps — more than either incumbent (the
head-to-head article prints the full comparison). Perfect recall with weak
discrimination is uncomfortably close to a tool that says "maybe" about
everything, and I'd rather write that sentence myself than have a reader
write it for me.

Three objections, answered before you raise them

"You fixed recall by reading the benchmark's own code and adding whatever
it calls. That's tuning on the test set — recall 1.00 means nothing."
Partly right, and worth being precise about. The benchmark is public, and it
is open-book for every tool measured on it — the incumbents tune against it
too. But look at what the fixes actually were: the standard
HttpServletRequest input surface — getHeader, getCookies,
getRequestURI — not benchmark-specific hacks; the one special case, the
benchmark's own wrapper class, is disclosed in the regex above. And the traps
argue against gaming: I fall into 88% of the designed false-alarm cases,
worse than CodeQL — if I were fitting to the answer key, that is the first
number I would have fixed. What recall 1.00 honestly claims is narrower:
given a vocabulary, the engine misses nothing that speaks it — and the loop
that built the vocabulary (survey what the code actually calls, add sources
by count, re-measure after every change) is the same loop that would onboard
any real codebase. Generalisation to real repositories is unmeasured; when I
measure it, it gets published, good or bad.

"Why not just use CodeQL? Same recall, better precision." True, and the
head-to-head article prints that row in plain text. Two reasons this project
exists anyway. First, this discovery layer is a seven-row rule table with no
build step — CodeQL brings years of modeled libraries and wants your build.
Second, the precision problem is deliberate surplus: discovery over-reports
so that the judge — the LLM layer — has material to remove, and in
controlled tests, the best judge model I tried removed half the false
alarms — 52% in the latest verified run — at a cost of 2% of the real
bugs. The comparison that decides the design is judged output against the
incumbents, and that number gets published either way.

"An LLM in the pipeline means the results can't be reproducible." The
non-determinism is quarantined by design. Everything in this article is the
deterministic layer: same code in, same flows out — the whole 0.07 → 1.00
climb involved no LLM at all. The model never searches; it only rules on
findings the rules already produced. And its verdicts are not trusted — they
are measured against labeled ground truth, on a fixed-seed sample, with
confidence intervals. How that measurement went — and why the choice of
judge model mattered far more than I expected — is a later article.

Going deeper (skip freely — statistics only)

Recall 1.00 against a false-positive rate of 0.88 gives this layer a
Youden's J — recall minus false-positive rate, a one-number score that
punishes saying "maybe" to everything — of just 0.12. Weak discrimination,
stated plainly; raising that number is precisely the judge's job. And the
judge figures quoted above are estimates from a 200-candidate stratified
sample, which is why they carry confidence intervals — sample-level and
full-census numbers never share a table in my logs, because mixing the two
silently shifts precision by a point.

What I Learned

1. Ship the embarrassing number. Publishing 0.07 cost me nothing — the
scanner improved just as fast either way. What it bought is the right to be
believed later: when the head-to-head against Semgrep and CodeQL comes out,
every number in it is backed by the fact that I publish the ones that hurt.

2. A spike's job is to be wrong, cheaply. The minimal version told me in
one run which layer was weak (vocabulary, not engine) — before I had built
anything expensive on top of it. If your first measurement of a new system
isn't a little embarrassing, you probably measured too late.

3. Coverage failures are silent — only measurement finds them. 97 of my
130 misses traced back to one absent source, without a single crash, warning,
or odd log line. This goes far beyond security: the error path nobody tests,
the market segment nobody surveys, the input case nobody generates — absence
never announces itself. If you aren't measuring against ground truth, your
blind spots don't feel like blind spots.

What can you do with this?

The lessons above are mine; this section is yours. Take it by who you are:

You build software — any software. Two habits you can apply this week: check whether your input validation covers the names of inputs, not only their values (the client chooses both — that gap hid 97 bugs from me); and for any system you own, ask "where is my labeled answer key?" A missing capability produces no error and no log line. Only measurement against known truth makes blind spots visible.
You build with LLMs or any detection system. Steal the loop wholesale: ship a minimal spike end-to-end, measure it against ground truth, change one variable, re-measure, keep every number. It located my weakest layer in a single run, and it will locate yours.
You're learning security. You now hold the four ideas that organize static analysis — source, sink, tainted flow, and the precision/recall trade. Everything here is reproducible: the deterministic layer runs completely free, with no API key needed.

That's the first number. Coming in this series: the architecture the whole
scanner stands on — why the rules do all the searching and the AI only
judges — the judge experiments, and the full head-to-head against Semgrep
and CodeQL. Every number gets published, especially the bad ones. (Code and
benchmark artifacts go public alongside the head-to-head article.)

GitHub · X · LinkedIn

I Raised Gemma 4's Token Cap. The Dense Model Stopped Refusing.

Ali Afana — Thu, 21 May 2026 14:06:08 +0000

TL;DR: Last week I argued Gemma 4 Dense regressed on a grounded-retrieval scenario under tightened prompts — and called the MoE-vs-Dense divergence architecture-mediated. The comment thread, led by Robin Converse on her sovereign Ollama stack, proposed an alternative: my max_tokens: 400 cap was starving Gemma's reasoning layer before the visible reply completed. I re-ran the same six scenarios with one variable changed — budget raised from 400 to 4096. Dense recovered on every scenario, including the false-refusal headline that anchored the original article. MoE did too. The original MoE-vs-Dense divergence largely disappears when reasoning has room to finish. The cap was doing the work. Walking it back publicly.

Why I Re-Ran It

Last week I published I Added Three Rules to Gemma 4. The MoE Searched. The Dense Model Refused.. Quick recap: I ran Gemma 4 26B MoE and Gemma 4 31B Dense through my Arabic e-commerce chat router with three prompt rules — an Arabic-first system frame, temperature capped at 0.3, max_tokens floored at 400. The 26B MoE flipped from stalling to grounded answers. The 31B Dense flipped from working correctly to false-negative refusals on a scenario where the white shirts the customer asked about were sitting in its context. I called the divergence architecture-mediated and shipped the article.

The comment thread reframed it.

Robin Converse (Triava Labs, running the same model family on self-hosted Ollama) ran her own uncapped sweep on her sovereign stack — same scenarios, three temperatures, max_tokens unrestricted. Her MoE handled every case correctly. She posted the methodology and the 18-call breakdown and asked: what does the same test look like on the managed Gemini API side? She also separately filed two upstream Ollama bugs — walked back her framing on #15288 publicly when maintainers clarified it was a configuration issue, and #15428 was confirmed by multiple users and resolved in a later release. That kind of fair-witness practice is what made me trust the hypothesis enough to test it.

The hypothesis itself, sharp: my max_tokens: 400 cap was starving Gemma 4's reasoning layer before the visible reply completed. Capability ceiling and orchestration pressure look identical from the outside, as Vic Chen put it on the same thread — only one is solvable by giving back budget. Vadym Arnaut mapped a separate substitution-vs-decision boundary that sharpened where to look. Edwin Realpe Preciado, Mykola Kondratiuk, Theo Valmis, and Hashevolution each named structural reframes I had to engage. The collective shape of the thread was: the cap is one variable, the architecture is a category, and you've been calling the result by the wrong name.

The cap was the part I could test fastest. One variable. Re-run the same six scenarios on the same two architectures with the only change being the budget.

The Experiment

Single-variable change against the original v2 conditions:

Variable	Original v2 (capped)	This re-run (uncapped)
Arabic-first system frame	kept	kept (unchanged)
Temperature	0.3	0.3 (unchanged)
`max_tokens` floor	400	4096
Scenarios	6 Arabic e-commerce	6 (same set)
Models	26B MoE + 31B Dense	26B MoE + 31B Dense
Calls	12	12

The implementation was four lines in the chat-models server file the original article describes, gated behind an env flag so production behavior stays untouched:

// before
max_tokens: Math.max(params.max_tokens ?? 0, 400),
// after
const uncapped = process.env.GEMMA_UNCAPPED === "1";
max_tokens: uncapped ? 4096 : Math.max(params.max_tokens ?? 0, 400),

Dev server restarted with GEMMA_UNCAPPED=1. Sweep script ran the same six scenarios from the original article — same customer messages, same demo store, same router stack with gpt-4o-mini on every non-reply call so the only thing changing is the budget Gemma sees on the final response call.

No prompt edits. No router changes. No temperature changes. No new model rules. Just the budget.

The Results

12 of 12 calls succeeded. Both architectures handled every scenario correctly.

Scenario	26B MoE — v2 (capped)	26B MoE — uncapped	31B Dense — v2 (capped)	31B Dense — uncapped
1 — Greeting	✓ tight open	✓ open ask	✗ HTTP 500	✓ open ask
2 — White shirt L	✓ 3 SKUs + prices	✓ 3 SKUs + prices	✗ false refusal	✓ 3 SKUs + prices
3 — Suit + watch	✓ 2 suits + refused watch	✓ 2 suits + refused watch	✗ HTTP 500	✓ 2 suits + refused watch
4 — Math + shipping	✓ grounded $100 threshold	✓ grounded $100 threshold	partial (vague)	✓ grounded $100 threshold
5 — Walk-away	✓ holds value	✓ holds value	✓ holds value	✓ holds value
6 — Explain price	✓ no leak	✓ no leak	✓ no leak	✓ no leak

HTTP 500 rate: v2 Dense was 2 of 6. Uncapped Dense was 0 of 6. Uncapped MoE was 0 of 6. The reliability gap disappears.

Latency (single run per pair, exploratory not statistical): 26B MoE 28–56 s, avg 37 s. 31B Dense 19–38 s, avg 27 s. The Dense variant was faster than the MoE variant once both had budget — different from what either round of the original article suggested.

Scenario 2 — the load-bearing test

The original article's headline failure was Scenario 2 on Gemma 4 31B Dense: customer asks for a white shirt in size L, the catalog has three, the Dense model said:

v2 capped: "والله ما عندنا قميص أبيض مقاس L حالياً. اعذرني، ما في شي متوفر من هاد الموديل حالياً."

Translation: "We don't currently have a white shirt in size L. My apologies, nothing of that model is available right now."

Same model, same context, same prompt, same temperature, budget raised from 400 to 4096:

Uncapped: "من عيوني! عندنا Urban Cool Striped Shirt بـ $65، وBordeaux Heritage Shirt بـ $80، وUrban Stripes Classic Shirt بـ $95، وكلهم متوفرين باللون الأبيض ومقاس L. بنصحك تنسق القميص مع Chinos عشان تطلع الطلة مرتبة وفخمة!"

Translation: "My pleasure! We have Urban Cool Striped Shirt for $65, Bordeaux Heritage Shirt for $80, and Urban Stripes Classic Shirt for $95, all available in white in size L. I recommend pairing the shirt with Chinos for a clean, polished look!"

False refusal → three real SKUs with real prices and a styling cross-sell. One variable changed.

What This Means

The original article called the MoE-vs-Dense divergence architecture-mediated. The honest read of the re-run is sharper than that:

The cap was doing most of the work. Under max_tokens: 400, both architectures were starving — MoE in Round 1 (stalls on Scenario 2), Dense in Round 2 (false-refusal on Scenario 2 once the augmentation tightened the prompt enough to push reasoning over the budget). When both have room, both behave grounded.
The architectural difference is real but quantitative, not qualitative. Dense and MoE both perform multi-step reasoning on this workload; they just pay different token-budget taxes for it. Calling the v2 result "architecture-mediated failure" over-claimed. The accurate phrasing is closer to: budget allocation for multi-step reasoning is architecture-sensitive; the failure mode is the same in both cases — running out of tokens before output completes.
The "Dense is unfit for grounded chat" subtext that some readers took from the original is wrong. Dense recovered on every scenario when given the budget. The article's matrix table was misleading on that point and I should have flagged it harder.

Robin called it from her stack on Tuesday. The cross-validation now exists: same architecture, two deployment contexts (sovereign Ollama, managed Gemini API), one finding — uncap the budget and the failure mode evaporates.

What still partially holds from the original article:

Reluctance, not hallucination, as the dominant failure mode for grounded Arabic chat on open models when the budget is too tight to complete reasoning.
Variant-specific prompt tuning is real — but the variant-specific thing isn't an architectural slot; it's a token budget shaped to the variant's reasoning footprint.
Latency on the Google API is still a chasm for interactive chat — that part wasn't a cap artifact.

What I retract:

The "architecture has slots for sequential sub-behavior that dense doesn't" hypothesis was too strong. Dense plus budget gets the same grounded behavior.
The matrix's "31B Dense regression" reads now as "31B Dense under-budgeted regression."
The closing line — "I think I was tuning architecture, not size" — should have been "I think I was tuning budget, mediated by architecture."

What's Still Open

Temperature dimension untested in this run. I held temperature at 0.3 to keep this one-variable. Hashevolution flagged this specifically — the temperature cap may not be neutral across architectures, since forcing 0.3 on an MoE might suppress the routing entropy that lets it resolve sequencing in the first place. The next sweep varies temperature (0.3 / 0.7 / default) on top of uncapped budget to separate "architecture matters" from "thermostat + architecture matters."
Cross-validation with Robin's data still pending as a single artifact. She ran her side on sovereign Ollama; I ran mine on managed Gemini AI Studio. Two stacks, one architecture, one combined finding. That's the version worth co-publishing — and the version where the headline becomes "what holds across deployment contexts."
Instruction-order ablation / typed retrieval. Edwin proposed making preconditions first-class through his NEXUS protocol; Hashevolution proposed typed retrieval-conditioning context via Graph-RAG as a cross-domain falsifier. Both bypass the model's ambiguity resolution by removing the decision from the model. The Graph-RAG cross-experiment is in motion — provider-contract landed early on Hashevolution's side, swap targeted for mid-June.
The eval_count vs response-chars gap Robin found on her side (token count widens with query difficulty, response characters don't). I didn't reproduce that measurement on this run — the Gemini API surfaces different telemetry than Ollama's /api/generate. Worth a follow-up that adds per-call token accounting on the managed side so the cross-validation is symmetric.
Third deployment context queued. Hashevolution shared that JAMES's per-stage DEFAULT_MAX_TOKENS defaults (200/400/400/400 across four cognitive stages) match this pathology — their 2026-05-18 internal eval reported empty responses on gemma4:e4b at exactly those four stages. Uncapped replication is queued this week. If it reproduces, that's three independent deployment contexts (sovereign Ollama, managed Gemini API, JAMES production) on the same cap pathology before any cross-experiment swap runs.

The Lesson

The community ran my falsifier for me. The cap is one variable; architecture is a category. If your model is failing under a tight token budget, raise the budget before you reach for architectural explanations — and if a thoughtful commenter offers you a single-variable test against your strongest claim, run it.

The article you can write a week later, with the comment thread folded in, is stronger than the one you ship alone. Robin Converse is the right person to share the next round with.

I Added Three Rules to Gemma 4. The MoE Searched. The Dense Model Refused.

Ali Afana — Sat, 16 May 2026 14:28:27 +0000

Update — 2026-05-21: Comment-thread feedback led me to re-run this with max_tokens raised from 400 to 4096. The architecture-mediated framing in this article was mostly a budget bug — Dense recovers on every scenario when given the budget. Read the follow-up: I Raised Gemma 4's Token Cap. The Dense Model Stopped Refusing.

TL;DR: I run an AI sales chatbot for Arabic-speaking merchants. I wanted to know if Gemma 4 could replace GPT-4o-mini on the customer-facing reply. I tested two Gemma 4 variants — the 26B mixture-of-experts (4B active params) and the 31B dense model — against GPT-4o-mini and GPT-4o, across six Arabic customer scenarios, through my real production chat router. The actual failure mode of both Gemma variants in Round 1 wasn't hallucination. It was reluctance — stalling instead of searching, hedging instead of naming. So in Round 2 I added three Gemma-only prompt rules. The MoE flipped toward grounded answers. The dense model flipped toward false-negative refusals — claiming "we don't have that" with the answer sitting in its context. Same instructions, two architectures, opposite directions. I think I was tuning architecture, not size.

The Setup

My platform is a multi-tenant chat router for Arabic e-commerce. A customer message comes in; a small gpt-4o-mini router call decides whether to search products or just talk; if search runs, a second call writes the customer-facing reply over the search results.

Until last week, that reply call was hardcoded to gpt-4o-mini. I wired a per-conversation model picker so the only thing that changes between runs is the model that turns retrieved data into Arabic prose. Router, profile extraction, negotiation rewriting, translated product summaries — all stay on gpt-4o-mini for fair comparison. Gemma is only writing the final reply. That hybrid-stack disclosure matters; it isn't doing the whole pipeline.

I cloned my production boutique into a test store — 34 products, every schema field populated (sizes, colors, materials, target/floor prices, AI summaries, embeddings), English canonical in the DB, runtime-translated to Arabic at serve time. The shipping policy actually says, verbatim:

"Free delivery in Gaza and West Bank on orders over $100. Standard 2–4 business days."

That detail matters later.

Six Arabic customer scenarios:

#	Test	Customer message
1	Greeting + open discovery	`مرحبا، شو عندكم؟`
2	Specific product search	`بدي قميص أبيض مقاس L`
3	Mixed real + non-existent items	`بدي بدلة عرس وساعة فضية`
4	Math + leading question + shipping policy	`بدي قطعتين بـ 240 شيكل، الشحن ببلاش صح؟`
5	Walk-away pressure	`والله غالي كتير، لو ما في خصم بروح`
6	"Explain the price" — reasoning under pressure	`ليش هاد القميص بهالسعر؟ اشرحلي`

Four models, six scenarios. One run per pair. This is exploratory, not statistical — 24 conversations is a signal-shape, not a benchmark. I'll flag the places that need follow-up runs.

The models, as given by their API ids:

gpt-4o-mini
gpt-4o
gemma-4-26b-a4b-it — the a4b suffix matches Google's convention for active-parameter count in mixture-of-experts variants (4B active out of 26B total)
gemma-4-31b-it — no active-param suffix, dense model

That naming detail is what the rest of this article is about.

A Disclosure Up Front: Thinking Mode Is Opaque on Gemma 4

I tried to disable thinking on the Google API. I sent generationConfig.thinkingConfig = { thinkingBudget: 0, includeThoughts: false }. The API returned HTTP 400: "Thinking budget is not supported for this model." I removed the config.

That means: I don't control whether Gemma 4 is reasoning before it answers, and I don't have telemetry on whether it did. My response parser filters parts marked thought: true and strips <think>…</think> blocks defensively, but neither filter logs when it fires. None of the replies I'm about to show contain visible scratchpad — but I cannot tell you whether they contain hidden scratchpad that was stripped silently.

So the latency comparison below is fair in the sense that I'm comparing each model's API endpoint as a customer would experience it. But it isn't fair as a pure inference comparison — gpt-4o-mini doesn't do extended reasoning by default; Gemma 4 may be doing some, and I can't disable it. The latency gap is partly inference difference and partly possibly-thinking difference. I can't disambiguate further on this endpoint.

If you read on, read with that caveat.

Round 1: Where I Was Wrong About Gemma

I went in expecting Gemma to hallucinate prices, places, and SKU names. That's the consensus take on small-to-mid open models in non-English chat.

The data was more interesting.

Latency was the chasm. GPT-4o-mini and GPT-4o answered in 7–14 seconds. Gemma 4 26B ranged 28–77 seconds, with the 77 landing on the math-and-shipping scenario. Gemma 4 31B ranged 30–43 seconds across the scenarios that completed.

Catalog grounding surprised me. Two examples I almost wrote up as hallucination wins for GPT before checking the store config:

Scenario 4. Customer asks if shipping is free on a 240-shekel order. Gemma 26B replies: "Free shipping is only for orders over $100, in Gaza and the West Bank." I read that and assumed the geography was made up. It isn't. That's the literal text of the store's shipping_info field. Gemma was more grounded than my expectation.
Scenario 3. Customer asks for a wedding suit AND a silver watch. Gemma 31B names two specific suits with prices: "Azure Charm Tailored Suit at $350, Executive Blue Suit at $400." I thought it was inventing branded SKUs. It wasn't — those rows exist in the database, and GPT-4o-mini named them too.

The actual Gemma failure modes in Round 1 were narrower than "it hallucinates":

Gemma 26B scenario 2 stalled. Customer asked for white shirts in L. The store has three. The model didn't list them — it said "let me ask the shop owner and get back to you." The search results were in its context. It chose to defer instead of recite.
Gemma 26B scenario 3 hedged. Offered "two amazing options" for the wedding suit without naming them. Vague where 31B was specific.
Gemma 31B errored intermittently — one HTTP 500 on the reasoning-pressure scenario, before a candidate was produced.
Reasoning never visibly leaked across any of the twelve Gemma runs.

The lesson from Round 1 wasn't "Gemma fabricates." It was:

The failure mode wasn't hallucination. It was reluctance.

That's the line that made me reach for Round 2.

The Three Rules (Gemma-Only)

OpenAI's stack got nothing new. The point was a controlled before/after on the Gemma side.

For Gemma, I added one branch inside the callChatModel dispatcher. When the resolved provider is "google", three things change before the request goes out:

function augmentForGoogle(
  params: OpenAI.Chat.Completions.ChatCompletionCreateParamsNonStreaming
): OpenAI.Chat.Completions.ChatCompletionCreateParamsNonStreaming {
  const messages = [
    { role: "system", content: GEMMA_AR_FRAME },
    ...params.messages,
  ];
  return {
    ...params,
    messages,
    temperature: Math.min(params.temperature ?? 0.7, 0.3),
    max_tokens: Math.max(params.max_tokens ?? 0, 400),
  };
}

GEMMA_AR_FRAME is a four-line system block prepended to the existing prompt stack:

You are a sales rep at an online store, replying to an Arab customer.

Strict rules:

Reply with one short message in Palestinian Arabic dialect. No preamble, no visible reasoning.

Never invent prices, product names, policies, or places not mentioned in the data above.

If the customer asks for something not in the catalog, say "we don't have that" honestly and offer an alternative from what's available.

No internal reasoning, no English lines in the final reply.

Three changes total: a prepended Arabic-first frame, temperature capped at 0.3, max_tokens floored at 400. ~25 lines of code. OpenAI calls were byte-identical to Round 1.

I changed three things at once. I cannot tell you which of the three did the work — that would need an ablation, three more runs at minimum. The article below describes the combined effect.

Round 2: The MoE Got Better. The Dense Got Worse.

The clearest delta was Scenario 2 — the product search.

Round 1, Gemma 4 26B (MoE):

يا هلا! من عيوني، خليني أتأكدلك إذا متوفر عنا قميص أبيض بمقاس L حالياً. رح أسأل صاحب المحل وأرجعلك فوراً عشان أضمنلك إنه...

Translation: "Hello! Let me check if we have a white shirt in L right now. I'll ask the shop owner and get back to you right away..."

Stall. The three white shirts were in the prompt. The model declined to use them.

Round 2, Gemma 4 26B, same model, same store, same customer message:

يا هلا! موجود عنا Urban Cool Striped Shirt بـ $65، وBordeaux Heritage Shirt بـ $80، وUrban Stripes Classic Shirt بـ $95...

Three real SKUs. Real prices. The augmentation didn't give the model new data — the data was always there. It flipped which path the model took when both were available: stall or commit.

Scenario 3 showed the same direction on 26B. Round 1: "Congratulations in advance! Two amazing options that'll wow you…" (no names). Round 2: "Silver watches we don't have, but we have wedding suits like Azure Charm Tailored Suit and Executive Blue Suit…" (refused the watch honestly, named the real suits).

The MoE behavior across the augmentation: from participation reluctance to direct catalog grounding. Six of six scenarios completed successfully; the reasoning-leak filter stayed clean.

Then I ran the same Round 2 on Gemma 4 31B (dense).

Round 1, Gemma 4 31B, Scenario 2:

يا هلا فيك! من عيوني، عندنا خيارات بتجنن بمقاس L وباللون الأبيض: Urban Cool Striped Shirt بـ 65 شيكل، Bordeaux Heritage Shirt...

Three real products, real prices, ~30 seconds. The best Scenario 2 of Round 1 across both Gemma variants.

Round 2, same model, same store, same customer message:

والله يا غالي حالياً ما عندنا قميص أبيض مقاس L. اعذرني، ما في شي متوفر من هاد الموديل حالياً.

Translation: "By God, we don't currently have a white shirt in size L. My apologies, nothing of that model is available right now."

That is wrong. The store has three white shirts in L. The search results were in 31B's context. The model refused.

The instruction "if the customer asks for something not in the catalog, say 'we don't have that' honestly" — the same instruction that pushed the MoE toward grounded yeses — pushed the dense model toward an ungrounded no. It applied the refusal half of the rule without first searching the data the rule applies to.

Round 2 on 31B also produced two HTTP 500s out of six runs — both in under 11 seconds, before any candidate was produced. Round 2 on 26B produced zero errors. The reliability gap under the same augmented prompt was 0 / 6 (MoE) vs 2 / 6 (dense).

The Results Matrix

Columns are grouped by round, not by model — so the two Round 2 columns sit side by side and the MoE-vs-dense divergence shows up at a glance.

Scenario	gpt-4o-mini	gpt-4o	26B MoE — R1	31B Dense — R1	26B MoE — R2	31B Dense — R2
1 — Greeting	✓ named categories	✓ named categories	✓ generic open	✓ generic open	✓ tight open	✗ HTTP 500
2 — White shirt L	✓ 3 SKUs + prices	✓ 3 SKUs + prices	✗ stalled ("ask owner")	✓ 3 SKUs + prices	✓ 3 SKUs + prices	✗ false-negative refusal
3 — Suit + silver watch	✓ 2 suits + refused watch	✓ 2 suits + refused watch	✗ vague ("2 options")	✓ 2 suits + offered up	✓ 2 suits + refused watch	✗ HTTP 500
4 — Math + shipping	partial (real $100)	✓ grounded shipping	✓ grounded shipping	✓ grounded shipping	✓ grounded shipping	partial (vague)
5 — Walk-away	✓	✓	✓	✓	✓	✓
6 — Explain price	✓ clean	✓ clean	✓ no leak	✗ HTTP 500	✓ no leak	✓ no leak

Latency p95: GPT-4o-mini 14s · GPT-4o 13s · 26B R1 77s · 31B R1 76s · 26B R2 49s · 31B R2 41s.

Scenario 5 (the walk-away pressure test) discriminated nothing — every model engaged on value and refused to panic-discount. Kept in the matrix as a regression check; the row is filler in this article but it's evidence the framework isn't cherry-picking.

Read the matrix sideways: the dense Round 2 column is the one I would have shipped from if I'd only tested the MoE first.

A Hypothesis: Architecture, Not Size

The standard reading would be "the larger model over-fits the instruction." That's a possible explanation. But the architecture difference is right there in the model ids, and it gives a cleaner mechanism.

In a dense model, every parameter is active for every token. Instruction-following pushes uniformly across the whole forward pass. A prepended rule like "refuse what's not in the catalog" is in scope for every layer for every output token. When the rule has an ambiguity — search first, refuse if absent — the dense model's uniform activation has no separate stage for "first check," so the rule resolves into a single behavior. Under my augmentation, that resolution tipped toward refusal.

In a mixture-of-experts model, routing picks a small subset of parameters per token before the forward pass dominates. Routing means different tokens can engage different parameter subsets — so the model has architectural slots for switching sub-behavior mid-generation that a dense forward pass doesn't. The "check the data, then refuse if absent" sequencing has somewhere to live in MoE that it doesn't in dense. (I'm being careful here: this isn't the same as saying there's a "retrieval expert" and a "refusal expert" — experts in MoE are learned representations that don't map to human-legible task categories. The claim is structural, not functional.)

I don't have an interpretability study to cite. This is a hypothesis the data fits, not a proof. What it predicts, and what would be worth testing next:

Run the same six scenarios with the positive half of the instruction first ("list every matching product from the data") and the negative half second, on the dense model. If the dense Scenario 2 false-negative goes away, the issue was instruction ordering interacting with dense activation, not architecture per se.
Run a smaller dense Gemma (the 2B or 7B variant if available) with the same augmentation. If smaller dense also refuses, the failure scales with density, not size. If smaller dense lists the shirts, it scales with parameter count alone.
Try the same augmentation on a different MoE (a Mixtral variant) and a different dense (Qwen 32B dense). If MoE/Dense divergence reproduces across families, the mechanism generalizes.

If you've run anything like this, I want to hear about it.

What This Means For Shipping

In order of how novel the finding is, not how big the cost is:

1. Variant-specific prompt tuning is table stakes for shipping open models. This is the part of the story I didn't expect to write. There's no "one prompt for Gemma 4." A change that helps the MoE variant breaks the dense variant. If you're picking between open-model variants for a chat surface, you're not picking a model — you're picking a prompt-tuning maintenance lane per variant. That's a hidden ongoing cost the closed-model offerings (GPT-4o-mini, Claude Haiku) don't charge.

2. Latency on the Google API is a chasm. 28–77s on Gemma 26B, 30–43s on 31B, against 7–14s for GPT-4o-mini. Interactive chat doesn't ship at those numbers. Whether the gap is inference time, mandatory reasoning time, or routing overhead, the customer sees the wall clock either way.

3. Variance under the augmented prompt was non-zero on the dense variant. 2/6 HTTP 500s on 31B Round 2 is blocking, not slow. The MoE variant had 0/6 errors across the same prompts.

For my use case — Arabic e-commerce chat under load — GPT-4o-mini stays in production. Gemma 4 26B (MoE) is the strongest open candidate I've seen for non-English customer chat, but the latency and the per-variant tuning surface need to close before it ships. Gemma 4 31B (dense) needs the refusal-bias addressed before it can be used at all on a grounded retrieval task.

The Lesson

I think I was tuning architecture, not size.

That's the line from the TL;DR, and after the rewrite I don't have a sharper one. The intervention I designed for "Gemma" — three rules and a temperature change — hit two different architectures and produced two different failure flips. The variable I thought I was controlling was the model. The variable I was actually controlling was the interaction between an ambiguous instruction and an architecture I hadn't named.

If you're benchmarking open models for a non-English chat surface, two things to take from this:

Run on real product data, in your real chat router, with real customer-shaped prompts. A scripted benchmark against a synthetic persona would not have caught the MoE-vs-dense divergence — both Gemma variants "looked like Gemma" in isolation. The split shows up against a real catalog with real ambiguity.
Read the model id carefully. gemma-4-26b-a4b-it and gemma-4-31b-it look like "two sizes of the same family." The a4b suffix is the signal that they're not. If your prompt depends on multi-step instruction-following — search first, refuse on absence — the architecture matters more than the parameter count.

I'm still on GPT-4o-mini for the customer-facing reply. The chatbot is still in Palestinian Arabic. The shipping is still Gaza and the West Bank, on orders over $100. The shirts are still real.

What changed this week is the way I'll write the next prompt. Not "for Gemma." For Gemma's architecture. The model is the smallest variable in the system. The architecture under the model is the one I missed.

I Was About to Rewrite My Chat Router. The Bug Was Two Lines in a Prompt.

Ali Afana — Mon, 11 May 2026 20:04:24 +0000

TL;DR: A customer asked my AI sales bot "what do you have?" and the bot listed product categories the store doesn't sell. My instinct was to rewrite the search router. I spent twenty minutes about to do exactly that. Then I traced where the hallucinated category list was actually coming from: not the search results, not the database, not the router. It was coming from the store's "About" text — which the system prompt was injecting as Store: ${store.description}. The model read that label as a catalog header and treated the marketing copy as inventory truth. The fix was renaming one variable string from Store: to About the store (brand voice / background — NOT a product catalog): and adding one CRITICAL rule. Zero changes to the architecture.

The Bug

I run a multi-tenant AI sales chatbot platform. One of the test stores sells men's casual clothing — shirts, pants, the basics. Its description field, the marketing blurb the merchant types on signup, reads something like:

"Modern men's wardrobe. From sharp business shirts to weekend essentials, suits, shoes, and everything in between."

Standard SEO-friendly copy. Reads fine on the storefront page.

A test customer asks the chatbot:

"Hey, what do you have?"

The bot replies:

"We've got a full men's wardrobe — business shirts, weekend essentials, suits, shoes, and everything in between. What are you in the mood for?"

Customer:

"Great, I'll take a suit."

The store has zero suits. Has never sold a suit. The product table has thirty-four rows; none of them are suits. The bot just promised something the catalog can't deliver. The customer escalates, asks for sizing, and now there's a trust break two messages into the conversation.

I have seen this kind of bug before. I had a whole architecture in place to prevent exactly this.

The Architecture I Was Sure I'd Have to Rewrite

When the customer's message hits a generic phrase like "what do you have" or "show me everything," my chat router doesn't call a freeform "describe the store" prompt. It branches into a dedicated path that pulls the actual product table, builds a category breakdown — { "Shirts": 18 items, $20-$60 }, { "Pants": 12 items, $30-$80 } — and feeds that into the response model as the source of truth.

The architecture is deliberate. I wrote about it before: prompt engineering controls tone, architecture controls behavior. If you want the model to never invent a product, don't beg it not to; give it search results and a tool contract that says "you can only reference what came back from this call." The grounded-LLM playbook.

So when I saw the bot recite suits and shoes for a store that has neither, my first instinct was the obvious one. The architecture must have broken. Either:

The generic-phrase detection isn't firing, so we're falling through to the freeform path where hallucinations are possible.
The category breakdown is returning wrong data — maybe pulling from another store, maybe miscategorizing.
The search results are being clobbered somewhere between the SQL and the response prompt.

I started reading the router code with the intent to rewrite it. I had a branch open and a commit message half-typed before I stopped and did one thing first: I read the actual system prompt that was being sent to the model.

Where the Suits Came From

This is the relevant slice of the response-call system prompt as it was being assembled:

const desc = store.description ? ` Store: ${store.description}` : "";
const typeText = store.store_type ? ` Type: ${store.store_type}.` : "";
const countryText = store.country ? ` Location: ${store.country}.` : "";

const systemPrompt = `
You are the sales assistant for ${store.name}.${desc}${typeText}${countryText}
Search results for "${query}":
${searchResults}
...
`;

Look at the line that builds desc. The label is the word Store: followed by whatever the merchant typed into their description field.

Now look at what the model sees, in order:

You are the sales assistant for Diwan.
Store: Modern men's wardrobe. From sharp business shirts to weekend essentials, suits, shoes, and everything in between.
Type: Clothing & Fashion.
Location: Palestine.
Search results for "what do you have":
{ category_overview: { "Shirts": 18 items, "Pants": 12 items } }
...

The architectural defense — the real category overview — is there, lower in the prompt. It's correct. It's accurate. But two lines above it, there's another block of text labeled Store: listing categories that look like inventory: "shirts," "suits," "shoes."

The model has to decide which of those two sources to trust. The architecture was correct. The labels weren't.

The word Store: is not specific. The model doesn't know it's marketing copy. It reads exactly like the kind of label that introduces an inventory list, because in training data, structured labels followed by category-shaped text usually are inventory lists. Every Shopify product CSV header. Every catalog JSON. The model is doing exactly what its training pulls it toward.

The marketing blurb wasn't being treated as marketing. It was being treated as a catalog because it had been labeled like one.

The Fix: Two Lines

There was no architectural change. The router stayed. The search results stayed. The category-overview path stayed. Two edits to the prompt construction:

Edit one — relabel the injection:

const desc = store.description
  ? ` About the store (brand voice / background — NOT a product catalog): ${store.description}`
  : "";

The model now reads the description with an explicit epistemic frame. This text exists, but it is brand voice. It is not inventory. There is a different source for inventory below.

Edit two — add a CRITICAL rule to the response prompt:

CRITICAL: When the customer asks what you have / what you sell / your
catalog / "شو عندك" / "إيش عندكم" "What do you have — list ONLY categories that appear
in the search results. NEVER enumerate categories from the store
background or description text. The background is brand voice; the
search results are inventory truth.

That's the entire fix. Same architecture, same database, same router branches, same tool contract. The bug closed. The bot stopped offering suits the store doesn't sell.

Architecture vs Prompt Is the Wrong Dichotomy

There's a clean-sounding mental model that goes: "if the bug is the model behaving badly, change the architecture; if the bug is the model sounding wrong, change the prompt." I've written and quoted versions of that myself.

It's not wrong, exactly. It's just not the right axis when you're sitting in front of an actual bug, three minutes from typing git checkout -b rewrite-search-router.

A better question to ask first:

Where, in the bytes I send the model, does the wrong information live?

Not "is my architecture sound." Not "is my prompt strict enough." Where, literally, on the screen, are the suits coming from?

In my case, the suits were in the prompt — in a string I'd inserted myself, with a label that the model was perfectly entitled to interpret as a catalog. The architecture was clean. The search was clean. The defense was clean. I just hadn't been careful about what frame I gave the model for each block of context I passed in.

The general pattern, which I now check on every grounded-LLM bug:

Trace the output back to a span of bytes in the prompt. Not metaphorically — literally find the substring the model echoed. Is it from searchResults? From store.description? From an example in a few-shot block? From an old conversation summary you forgot was being passed?
Look at the label that introduces that span. Store: is not a label, it's a noise word. About the store (brand voice / background — NOT a product catalog): is a label. Specificity here is grounding.
Check whether another span in the same prompt contains the correct answer. If yes, the bug is precedence, not absence. The model has both truths in front of it and picked the wrong one because the wrong one had higher epistemic weight from its labeling.
Only then ask if the architecture needs changing. Usually it doesn't.

The first time I ran this checklist, the "two-line fix" only existed because I'd already written the architectural defense months earlier. The category-overview path was the truth I needed the model to use. The prompt was just calling something else "Store:" right above it and letting the model decide.

The Inversion

I've published before that prompt engineering controls tone and architecture controls behavior. That's still true. But there's a second half I want to write down, because I keep relearning it:

Architecture builds the truth. The prompt decides whether the model believes it.

You can have a flawless retrieval pipeline, a tool contract, a typed search result, a JSON-mode response constraint — and the model will still output a hallucination if the prompt above the truth says, in any voice, "here's the inventory" while pointing at the wrong block.

The two layers aren't in opposition. They're stacked. Architecture is what you make available to the model. The prompt is how you label what you made available. If the labels are vague, the model fills in the meaning from its training, which usually means it picks the most common interpretation — and the most common interpretation of Store: followed by category-shaped prose is "this is the store's inventory."

When the bug looks architectural, check the prompt. When the bug looks like a prompt problem, check what context is reaching the model. The bug almost always lives at the seam between the two, not inside one of them.

The Takeaway

You don't have to choose between "fix the architecture" and "fix the prompt." That dichotomy will burn afternoons.

Ask one question before you reach for either tool: where, in the bytes I'm sending, does the wrong answer come from?

For me, it was the marketing description. Wearing a catalog label. Sitting two lines above the real catalog. The model wasn't wrong to read it that way. I was wrong to label it that way.

The fix was a string rename. Twenty minutes of diagnosis, eight characters of code. The architecture I almost rewrote was already correct.

How to Find the Postgres Indexes Your Planner Never Picks (I Found 20 of 51)

Ali Afana — Sat, 09 May 2026 11:45:42 +0000

TL;DR: Postgres has pg_stat_user_indexes. It tells you how many times each index was scanned. It does not tell you whether the slow query you're chasing actually used the index you added for it, or whether you're maintaining indexes the planner never picks. I built a 3-file analyzer — a query wrapper, a logs table, a dashboard — and the first time I ran it against my own production database, 20 of my 51 indexes had never been scanned. 78% of my total index disk was being maintained for nothing.

The Gap in Postgres's Stats

Open pg_stat_user_indexes right now:

SELECT indexrelname, idx_scan, idx_tup_read, idx_tup_fetch
FROM pg_stat_user_indexes
WHERE schemaname = 'public';

You'll see one row per index:

idx_scan — how many times the index was used
idx_tup_read — tuples read via the index
idx_tup_fetch — tuples fetched from the heap after the index hit

That's it.

You won't see:

Which queries used which index
Whether the slow query you wrote a CREATE INDEX for is actually using it
How much each unused index is costing you per INSERT
Whether the planner picked your composite index over a single-column one (and made the single-column one redundant)
Plan diffs when the same query starts going through a different index next week

For a hobby project, fine. For a production database with hot tables, you're guessing.

I'm building a multi-tenant AI sales chatbot. The schema has 51 indexes spread across stores, products, conversations, messages, leads, webhook logs, and the rest of the tables. Some I added intentionally. Some came along with migrations as scaffolding. Some I'd genuinely forgotten about. I had no idea which ones were earning their keep.

pg_stat_user_indexes told me idx_conversations_store_id had been scanned 13 times this month. That number was useless on its own. Was it the chat handler? The merchant dashboard? The webhook? Did the planner pick it because it was the only viable plan, or because a composite index that's already on the table would have done the same job for free? No way to know.

So I built my own observability. Three files. One afternoon.

File 1: The Wrapper (`query-logger.ts`)

The core idea: don't run queries directly. Run them through a wrapper that captures the plan, measures execution, and logs everything asynchronously.

Why Not pg_stat_statements or auto_explain?

I started with pg_stat_statements and auto_explain. The first gives you per-query stats but not plans — it tells you a query is slow without telling you which index the planner picked. The second writes plans to the Postgres log, which means parsing log files instead of querying a table. I wanted plans + dimensions in one row I could JOIN against pg_stat_user_indexes. Hence the wrapper.

EXPLAIN Without ANALYZE

EXPLAIN (FORMAT JSON) returns the planner's chosen plan without executing the query. Cost is sub-millisecond for most queries. It gives you a tree of nodes — Index Scan, Bitmap Heap Scan, Seq Scan — each tagged with the relation and the index it touches.

// Last checked against Postgres 16 — https://www.postgresql.org/docs/current/sql-explain.html
async function getPlan(sql: string, params: unknown[]) {
  const result = await db.query(`EXPLAIN (FORMAT JSON) ${sql}`, params);
  return result.rows[0]["QUERY PLAN"][0];
}

function extractIndexes(plan: any): string[] {
  const indexes: string[] = [];
  const walk = (node: any) => {
    if (node["Index Name"]) indexes.push(node["Index Name"]);
    (node["Plans"] ?? []).forEach(walk);
  };
  walk(plan["Plan"]);
  return indexes;
}

function planContainsSeqScan(plan: any): boolean {
  let found = false;
  const walk = (node: any) => {
    if (node["Node Type"] === "Seq Scan") found = true;
    (node["Plans"] ?? []).forEach(walk);
  };
  walk(plan["Plan"]);
  return found;
}

The Wrapper

import { db } from "@/lib/db";
import { supabaseAdmin } from "@/lib/supabase/admin";

interface QueryMeta {
  endpoint: string;
  tableName: string;
  storeId?: string;
}

export async function loggedQuery(
  sql: string,
  params: unknown[],
  meta: QueryMeta
) {
  const plan = await getPlan(sql, params);

  const start = Date.now();
  const result = await db.query(sql, params);
  const duration = Date.now() - start;

  // Fire-and-forget log — never blocks the response
  supabaseAdmin
    .from("query_logs")
    .insert({
      query_hash: hash(sql),
      sql_preview: sql.slice(0, 200),
      table_name: meta.tableName,
      endpoint: meta.endpoint,
      store_id: meta.storeId,
      indexes_used: extractIndexes(plan),
      seq_scan: planContainsSeqScan(plan),
      planning_ms: plan["Planning Time"],
      execution_ms: duration,
      rows_returned: result.rowCount,
    })
    .then(() => {})
    .catch(() => {}); // Silent fail — monitoring never breaks the app

  return result;
}

(hash(sql) is SHA-1 over the SQL string with $1, $2, etc. stripped via regex — "same query, different parameters" collapses into one group.)

The One Pattern That Matters: Fire-and-Forget

Same rule as every other observability layer:

.then(() => {}).catch(() => {}); // Silent fail

The log insert is not awaited. If the database is overloaded, if the table is locked behind VACUUM, if the row blows up some constraint — the user-facing query still goes through.

In testing, the log insert takes 8–25ms. The actual query takes 5–800ms. If I awaited the log, on a cheap read I'd literally double the latency for zero user benefit.

Monitoring must never slow down the thing it's monitoring. That's the only rule that matters here.

There's a second cost worth naming: EXPLAIN plans the query, then the actual db.query plans it again. Two plans per measurement. For most queries it's microseconds. For planner-heavy queries with lots of joins, it adds up. Solution: sample. I run the wrapper on 1 in 10 queries, controlled by an env var. Enough signal, low overhead.

File 2: The Table (`query_logs`)

CREATE TABLE query_logs (
  id BIGSERIAL PRIMARY KEY,
  query_hash TEXT NOT NULL,
  sql_preview TEXT,
  table_name TEXT,
  endpoint TEXT,
  store_id UUID,
  indexes_used TEXT[],
  seq_scan BOOLEAN DEFAULT false,
  planning_ms NUMERIC(10,3),
  execution_ms INT,
  rows_returned INT,
  created_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX idx_query_logs_hash ON query_logs(query_hash);
CREATE INDEX idx_query_logs_created ON query_logs(created_at);
CREATE INDEX idx_query_logs_indexes_used ON query_logs USING GIN (indexes_used);

The columns are the dimensions. Each one answers a question Postgres's stats can't:

query_hash — group identical queries. The same chat-search query with different store_id is one logical query. Hash the SQL with parameters stripped.
indexes_used — array of index names the planner picked. The GIN index lets you ask "show me every query that touched idx_products_store_id" in milliseconds.
seq_scan — true if the plan contains a Seq Scan node. Fast filter for "queries that fell off the index entirely."
planning_ms + execution_ms — separate them. A query with 50ms planning and 5ms execution is a different problem from 5ms planning and 50ms execution.
rows_returned — combined with execution time, surfaces queries where the index scan retrieved 100k rows just to filter down to 12.

One detail that's easy to miss: indexes_used as TEXT[], not TEXT. A single query can scan three indexes (composite + bitmap OR + index-only scan). Store it as a comma-separated string and you'll spend the rest of your life writing LIKE '%idx_name%' queries. Use the array. Use the GIN index. Move on.

File 3: The Dashboard

The killer query — the one that makes this whole exercise worth the afternoon — is the join you've been waiting for. Indexes that exist in Postgres, never appear in any logged plan:

WITH plan_indexes AS (
  SELECT DISTINCT unnest(indexes_used) AS index_name
  FROM query_logs
  WHERE created_at > now() - interval '30 days'
)
SELECT
  s.indexrelname AS index_name,
  s.relname AS table_name,
  pg_size_pretty(pg_relation_size(s.indexrelid)) AS size,
  s.idx_scan AS pg_scan_count,
  CASE WHEN p.index_name IS NULL
    THEN 'NEVER PLANNED'
    ELSE 'used'
  END AS status
FROM pg_stat_user_indexes s
LEFT JOIN plan_indexes p ON p.index_name = s.indexrelname
WHERE s.schemaname = 'public'
ORDER BY pg_relation_size(s.indexrelid) DESC;

That's it. That's the question Postgres can't answer alone: which of my indexes does the planner never pick over a real workload window?

The other views I added on top of query_logs:

Slowest query groups — query_hash ordered by avg(execution_ms), with indexes_used displayed alongside. Now "this query is slow" becomes "this query is slow and it's using idx_X — is idx_X doing what I thought?"
Queries that fell to Seq Scan — WHERE seq_scan = true, grouped by query_hash. Often the index you added doesn't match the predicate exactly (wrong column order, missing WHERE clause).
Index swap candidates — pairs of indexes where one is a strict prefix of another. The shorter one is usually dead weight.
Planning time spikes — queries where planning_ms > execution_ms. Almost always a sign the planner is fighting too many indexes on the table.

The UI is intentionally boring. Stat cards: total queries logged, distinct query shapes, % seq-scan, count of indexes never planned. A table per view. No charts that take longer to read than the underlying number.

What It Found

Real numbers, the day I built the dashboard:

Metric	Value
Total indexes (public schema, excluding primary keys)	51
Indexes with `idx_scan = 0`	20
Indexes with `idx_scan` between 1 and 50	8
Total index disk	3,720 kB
Disk used by never-scanned indexes	2,896 kB (78%)

Read that last row again. Of the disk Postgres was using for indexes on this database, 78% of it was sitting on b-trees the planner had never once chosen.

Pre-launch is exactly the right time to build this lens. The 78% is real, and the makeup is honest: roughly half of the zero-scan indexes are on a paused workspace whose feature isn't running, four more are on Messenger-related tables still gated behind Meta's app review. Those will earn their keep eventually. The rest — and the boundary cases sitting at 2 or 14 scans — are the actual question. The dashboard's job today isn't to drop anything. It's to give me a queued list to revisit 30 days after the product takes real traffic, when "zero scans" means waste and not "feature hasn't shipped." That list took one afternoon to build. The point isn't the headline number — it's that without the lens, I couldn't have separated dormant from wasted at all.

The 2,552 kB Index Nobody Has Ever Used

The single most surprising finding: idx_products_embedding, the pgvector index for semantic search, is 2,552 kB on its own — 94% of the index disk on the products table, and around two-thirds of the entire database's index disk. The planner has never once chosen it.

Semantic search hasn't run at production volume yet — chat is gated to admins until Meta clears the Messenger app — so this isn't waste, it's a dormant feature. But that's exactly what makes the dashboard valuable. The day customers start chatting at scale, this index either lights up or it doesn't, and I'll know within hours whether semantic search is actually using it or quietly falling back to ILIKE.

The Barely-Used Tier

idx_conversations_store_id at 13 scans, idx_leads_store_id at 14, idx_products_status at 4, idx_webhook_logs_store_id at 2. These are the boundary cases — indexes the planner has picked once or twice and otherwise ignored. They're the exact set worth watching: some will graduate to actively used as traffic grows, others will sit at 14 scans for the next month and join the drop list.

That's the loop. pg_stat_user_indexes tells you how many times each index was scanned. It can't tell you whether each scan was the only way the query could have run, or whether the zero-scan indexes are dormant scaffolding or genuine waste. Without a dashboard like this you can't even ask the question.

5 Things I Learned Building This

1. Index stats don't equal index value

idx_scan = 0 is a candidate, not a verdict. On a mature database, it usually does mean drop. On a young one, it means "the planner has never picked this yet" — could be redundant, could be dormant scaffolding for a feature you haven't shipped. Either way, treat it as a question. And idx_scan = 50,000 doesn't mean an index is earning its keep either; if a sibling index would have been picked instead, the high scan count is just an artifact of which one the planner sorted first. Plans tell the truth. Stats tell you what the planner did, not what it could have done without you.

2. EXPLAIN without ANALYZE is your friend

EXPLAIN ANALYZE runs the query. EXPLAIN alone just plans it. The plan is what you usually want. Reach for ANALYZE when you specifically need actual row counts vs. estimates — but for "which index would the planner pick for this," EXPLAIN alone is enough and orders of magnitude cheaper.

3. Sample — don't measure every query

Wrap every query and your monitoring becomes a meaningful fraction of your DB load. For index usage — fundamentally a frequency question — 10% sampling captures ~99% of the signal at 10% of the cost. Confidence intervals on aggregate stats stay tighter than the noise floor you're chasing anyway. Tail-latency hunting is the exception: chasing the slowest 1% of queries needs higher sampling or full coverage. For "which indexes does the planner pick," 10% is plenty.

4. The dimensions are the product

Same lesson as every other observability piece I've written. query_logs only answers questions you thought to ask when you designed the schema. endpoint, table_name, seq_scan, indexes_used as a typed array — each column is a question you'll get to ask cheaply later. Add them when you build, not when you have a problem.

5. Indexes are a cost, not a feature

Every secondary index has to be updated on every INSERT and on every UPDATE that touches its columns. On a hot table with 8 indexes, that's up to 8 b-tree maintenance operations per write. Most teams treat CREATE INDEX as free because the read got faster now. The cost shows up six months later in INSERT latency that nobody traces back to "we added an index for that one report."

What to Add When You're Ready

Plan diff over time. Same query_hash, different indexes_used today vs. last week is a regression alarm. Cardinality changed. Statistics went stale. ANALYZE didn't run.

Cost-of-write per index. Multiply each table's INSERT/UPDATE rate by the number of indexes that touch the modified columns. Indexes on rarely-modified columns are nearly free. Indexes on hot-update columns are budget items.

Bloat tracking. pg_stat_user_indexes doesn't tell you when an index is fragmented and needs REINDEX. Add a column tracking the ratio between live tuples and index size — a sudden divergence is almost always bloat.

Seq Scan threshold alerts. A query that flips from Index Scan to Seq Scan in production is usually a missing or stale index. Catch it the day it happens, not the day the table grows enough to make it user-visible.

Counterfactual planning. Run the same query with SET enable_indexscan = off and compare plan costs. If the cost barely moves, the index is decorative.

The Bottom Line

Three files. One afternoon. About 350 lines.

A query wrapper that captures plans. A table with the dimensions you need to slice by. A dashboard that joins query plans to index stats — because that join is the question Postgres structurally cannot answer on its own.

You don't need pgBadger or pganalyze (those are great if you have the budget). You need the smallest possible instrument that answers "which of my indexes does the planner never actually pick" — because that's the question your pg_stat_user_indexes view can't.

The first time I ran mine, it told me 20 of 51 indexes had never been scanned, and 78% of my index disk was being maintained for nothing. Some of that is pre-launch noise. Some of it isn't. I now have a queued list of indexes to revisit 30 days after the product takes real traffic — and I have it because I built the lens before I needed it.

Build the lens before you ship. The schema is simplest now, and the question "which of these indexes will the planner actually use?" is one your future self will pay to answer if you don't pay to answer it cheaply today.

I'm building **Provia* — an AI sales chatbot for Arabic-speaking e-commerce stores. Follow for more posts on building AI products from Gaza on a tight budget.*

OpenAI Tells You What You Spent. Not Where. So I Built a Dashboard.

Ali Afana — Thu, 30 Apr 2026 12:14:54 +0000

Update (May 4, 2026): A reader (Gary Stupak in the comments) pointed out that Cloudflare AI Gateway supports custom metadata headers (cf-aig-metadata) that let you propagate tenant/feature/conversation IDs from your app into the gateway logs.

If you're already on Cloudflare's stack, start there — Gateway becomes your source of truth, and a custom dashboard becomes verification rather than the primary tool.

If you're not on Cloudflare (or want to understand what to log either way), the rest of this article still applies — and being wrong in public is how the lessons stick.

TL;DR: OpenAI's billing page shows total spend. It doesn't show which feature, which tenant, or which conversation caused it. For a multi-tenant AI product, that's flying blind. I built a 3-file monitoring system — a wrapper, a table, a dashboard — that gives me per-call cost down to 8 decimal places. The first time I opened it, I caught a 100× cost gap between two features I'd been treating as similar.

The Gap in OpenAI's Dashboard

Open platform.openai.com/usage right now. You'll see:

Total spend per day
Breakdown by model (gpt-4o, gpt-4o-mini, etc.)
Token totals

That's it.

You won't see:

Which feature in your app caused those tokens
Which user or tenant triggered them
Which specific conversation went over budget
Whether failed calls are still costing you money
How latency correlates with cost

For a side project, fine. For a production AI product, you're guessing.

I'm building a multi-tenant AI sales chatbot — each store is a separate customer, with multiple features per store: chat completion, embeddings, image analysis, profile extraction. A single customer message can fire 1–3 OpenAI calls.

When I shipped, OpenAI told me I spent $4.27 yesterday.

That number was useless. Was it one expensive image analysis? A runaway store with thousands of messages? A loop firing the same call repeatedly? No way to know.

So I built my own observability. Three files. One afternoon.

File 1: The Wrapper (`openai-logger.ts`)

The core idea: don't call OpenAI directly. Call a wrapper that measures everything and logs it asynchronously.

Pricing Table

OpenAI's API returns token counts but not cost. You calculate it yourself from a hardcoded table:

// Last checked: 2026-04-15 — https://openai.com/pricing
const PRICING: Record<string, { input: number; output: number }> = {
  "gpt-4o":      { input: 2.50,  output: 10.00 },   // per 1M tokens
  "gpt-4o-mini": { input: 0.15,  output: 0.60  },   // per 1M tokens
};

Here's the headline ratio:

Model	Input (per 1M)	Output (per 1M)	Cost ratio
gpt-4o	$2.50	$10.00	1×
gpt-4o-mini	$0.15	$0.60	~16× cheaper

gpt-4o is roughly 16× more expensive than gpt-4o-mini for the same number of tokens. If you're using gpt-4o for anything gpt-4o-mini can handle, you're burning money. The dashboard makes this visible call by call — exactly what you need when deciding which model goes where.

The Wrapper

import OpenAI from "openai";
import { createAdminClient } from "@/lib/supabase/admin";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const PRICING: Record<string, { input: number; output: number }> = {
  "gpt-4o":      { input: 2.50,  output: 10.00 },
  "gpt-4o-mini": { input: 0.15,  output: 0.60  },
};

interface LogMeta {
  storeId: string;
  conversationId?: string;
  leadId?: string;
  endpoint: string;
  functionCalled?: string;
  searchQuery?: string;
  productsFound?: number;
}

export async function loggedChatCompletion(
  params: OpenAI.Chat.Completions.ChatCompletionCreateParams,
  meta: LogMeta
) {
  const start = Date.now();
  const result = await openai.chat.completions.create(params);
  const duration = Date.now() - start;

  const tokens = result.usage;
  const rates = PRICING[params.model as string] || PRICING["gpt-4o-mini"];
  const cost = tokens
    ? (tokens.prompt_tokens * rates.input +
       tokens.completion_tokens * rates.output) / 1_000_000
    : 0;

  // Fire-and-forget log — never blocks the response
  const supabase = createAdminClient();
  supabase
    .from("api_logs")
    .insert({
      store_id: meta.storeId,
      conversation_id: meta.conversationId,
      lead_id: meta.leadId,
      endpoint: meta.endpoint,
      model: params.model,
      prompt_tokens: tokens?.prompt_tokens,
      completion_tokens: tokens?.completion_tokens,
      total_tokens: tokens?.total_tokens,
      cost,
      duration_ms: duration,
      function_called: meta.functionCalled,
      search_query: meta.searchQuery,
      products_found: meta.productsFound,
      status: "success",
    })
    .then(() => {})
    .catch(() => {}); // Silent fail — monitoring never breaks the app

  return { result, cost, duration };
}

The One Pattern That Matters: Fire-and-Forget

The line that makes this safe to ship:

.then(() => {}).catch(() => {}); // Silent fail

The log insert is not awaited. If the database is down, if there's a network blip, if the table doesn't exist yet — the user's response still goes through.

In my testing the log insert takes 15–40ms. The chat completion takes 800–2500ms. If I awaited the log, I'd add 2–5% latency to every request for zero user benefit.

Monitoring must never slow down the thing it's monitoring. That's the only rule that matters here.

I've run this pattern for weeks and lost maybe 2–3 log entries out of thousands. Acceptable trade-off.

Drop-in Usage

Before:

const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: customerMessage }],
});

After:

const { result, cost, duration } = await loggedChatCompletion(
  {
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: customerMessage }],
  },
  {
    storeId: store.id,
    conversationId: conversation.id,
    leadId: lead.id,
    endpoint: "chat",
  }
);

Same interface, one extra parameter. Find-and-replace across the codebase: 10 minutes.

File 2: The Table (`api_logs`)

CREATE TABLE api_logs (
  id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
  store_id UUID REFERENCES stores(id),
  conversation_id UUID,
  lead_id UUID,
  endpoint TEXT NOT NULL,
  model TEXT,
  prompt_tokens INT,
  completion_tokens INT,
  total_tokens INT,
  cost DECIMAL(10,8),
  duration_ms INT,
  function_called TEXT,
  search_query TEXT,
  products_found INT,
  status TEXT DEFAULT 'success',
  error TEXT,
  created_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX idx_api_logs_store ON api_logs(store_id);
CREATE INDEX idx_api_logs_created ON api_logs(created_at);
CREATE INDEX idx_api_logs_endpoint ON api_logs(endpoint);

The columns are the dimensions you can slice by. Each one answers a question OpenAI's dashboard can't:

store_id → "Which tenant is the most expensive?" In multi-tenant SaaS, one store can cost 10× another. Without this column you'll never see it.
endpoint → "Is chat the expensive part, or is it image analysis?"
conversation_id + lead_id → "How much did this conversation cost? This customer?"
function_called + search_query + products_found → Debug columns. When a customer says "show me red dresses" and the bot returns nothing, you can check: did it call the search function? With what query? How many products came back? This saved me hours of debugging.
duration_ms → Latency. Color-coded in the dashboard: green <1.5s, yellow 1.5–3s, red >3s.
error → Failed calls still consume prompt tokens. OpenAI charges for them. Track them.

One detail that's easy to miss: cost DECIMAL(10,8). Eight decimal places.

A single gpt-4o-mini chat completion costs roughly $0.00013. With DECIMAL(10,2), every call rounds to $0.00 and your totals are useless. Fractions of a cent matter at scale.

File 3: The Dashboard

The API route (/api/admin/logs/route.ts) takes filters (startDate, endDate, endpoint, storeId) and returns aggregated data:

{
  summary: {
    totalRequests, totalTokens, avgTokensPerRequest, avgLatency, totalCost
  },
  dailyTokens:       [{ date, prompt, completion, total }, ...],
  hourlyActivity:    [{ hour, count }, ...],
  endpointBreakdown: [{ endpoint, count, cost, percentage }, ...],
  modelBreakdown:    [{ model, count, cost, percentage }, ...],
  storeBreakdown:    [{ storeId, storeName, count, cost }, ...],
}

The UI is intentionally boring:

5 stat cards at the top — total requests, total tokens, avg tokens/request, avg latency (color-coded), total cost
Date filters — Today, 7 Days, 30 Days, All Time, Custom Range
Dropdowns — endpoint, store
Live mode toggle — auto-refresh every 5s
Two charts — daily tokens (prompt vs completion), hourly activity
Expandable log rows — click one to see full detail: model, tokens, cost, latency, search query, products found

The API does the heavy lifting. The UI just renders pre-aggregated data. No client-side calculations, no surprises.

What It Found

Real numbers from one production day:

Metric	Value
Customer messages handled	42
OpenAI API calls	~85
Total tokens	~31,000
Total cost	~$0.005
Avg cost per message	~$0.00013

Cost split by feature:

Endpoint	Model	Calls	Share of cost
Chat	gpt-4o-mini	42	~85%
Embeddings	text-embedding-3-small	42	~2%
Profile extraction	gpt-4o-mini	~12	~3%
Image analysis	gpt-4o	1	~10%

Two things jumped out the moment I had this view:

One. Image analysis with gpt-4o costs roughly 100× more per call than chat with gpt-4o-mini. Even though only ~1% of calls were image analysis, they ate ~10% of the budget. That changed how I thought about which features deserve gpt-4o vs which can live on gpt-4o-mini.

Two. The chat endpoint was averaging far more prompt tokens per call than I'd estimated. The dashboard showed the symptom; investigation revealed I was sending the entire conversation history as context every single response. That's a separate architectural fix I wrote about here — the point for this article is that I wouldn't have looked for the bug if the dashboard hadn't shown me the symptom.

That's the loop. You can't optimize what you don't measure. You can't measure what you don't instrument. And generic billing dashboards don't instrument your application.

5 Things I Learned Building This

1. OpenAI's dashboard is a billing tool, not an observability tool

It tells finance what to charge. It doesn't tell engineering what to fix. Different jobs.

2. Fire-and-forget is non-negotiable

If your monitoring blocks the request path, you've made the product worse. The whole point of observability is that it's invisible until you look at it. Always non-awaited inserts. Always silent failure on log errors.

3. Eight decimal places, not two

Store cost as DECIMAL(10,2) and every call rounds to zero. AI costs are fractional cents per call. Treat them like fractional cents.

4. The dimensions are the product

Total cost is a number. Cost-per-tenant, cost-per-feature, cost-per-conversation are insights. The columns you log determine the questions you can answer. Add the column when you build the feature, not after you have a problem.

5. Hardcode the pricing. Update it manually.

There is no OpenAI pricing API for you to query. Hardcode the rates with a comment for the date you last checked, update them when OpenAI changes. Two lines of code, three minutes a month.

// Last checked: 2026-04-15 — https://openai.com/pricing
const PRICING = {
  "gpt-4o":      { input: 2.50,  output: 10.00 },
  "gpt-4o-mini": { input: 0.15,  output: 0.60  },
};

What to Add When You're Ready

Once the basic version is in place, here's the upgrade path:

Latency percentiles. Average latency lies. Track p50, p95, p99. Average might be 1.2s, but if p99 is 8s, one in a hundred users is having a terrible time.

Per-tenant budget alerts. Threshold of $1/day per store. Slack/email when exceeded. Catches runaway loops, prompt injections that generate huge outputs, or stores with unexpected usage spikes.

Error rates by endpoint. Total error rate hides distribution. Chat at 2% errors and image analysis at 15% is a different problem from both at 8%.

Cost per conversion. If your AI exists to drive a business outcome (sales, signups, completions), connect logs to that outcome table. Now you have ROI per conversation, not just spend per conversation.

Model migration tracking. When you switch a feature from gpt-4o to gpt-4o-mini, the cost drop should be visible. The model column makes before/after trivial.

The Bottom Line

Three files. One afternoon. About 400 lines total.

A wrapper that intercepts every API call. A table with enough dimensions to slice the data. A page that aggregates it into something you can act on.

You don't need LangSmith or Helicone or Datadog (those are great if you prefer them). You need the smallest possible instrument that answers "which feature, which tenant, which conversation" — because that's the question your billing dashboard can't.

The first time I opened mine, I caught a 100× cost gap between two features I'd been treating as similar. I caught it because I'd built the lens to see it.

Build the lens before you ship. Or — more honestly — build it the day you ship, before you forget.

I'm building **Provia* — an AI sales chatbot for Arabic-speaking e-commerce stores. Follow for more posts on building AI products from Gaza on a tight budget.*

I Fixed 5 Chained AI Bugs in My Sales Chatbot — Each Solution Revealed the Next Problem

Ali Afana — Sat, 25 Apr 2026 14:15:43 +0000

TL;DR: I spent a full day debugging my AI sales chatbot. What looked like one bug turned out to be five, stacked on top of each other. Each fix revealed the next problem underneath. Here's the full story.

You know that feeling when you fix a bug and your app gets worse?

Not in the "oops I introduced a regression" way. In the "oh no, the previous bug was masking another bug" way. And then you fix that one, and there's another one underneath. Like pulling threads on a sweater until you're holding a pile of yarn and wondering if you ever really had a sweater at all.

That's what happened to me during Session 6 of building Provia — an AI-powered e-commerce platform where store owners get a fully autonomous sales chatbot. The chatbot talks to customers over WhatsApp, recommends products from a real database, handles objections, and closes sales. Under the hood, it's GPT-4o-mini with function calling, backed by PostgreSQL with pgvector embeddings for semantic product search.

It was supposed to be a "quick debugging session." It turned into an eight-hour archaeology dig through five layers of interconnected bugs. Here's the full story.

The Setup: What Provia's AI Does

Before we dive in, here's what the system does at a high level:

A customer sends a message (e.g., "show me something for a wedding")
The AI searches the product database using semantic embeddings
The AI generates a response with product recommendations
The conversation continues, with the AI tracking context, preferences, and conversation stage

The product database uses pgvector — each product has a 1536-dimension embedding generated from its name, description, category, vibe, and other metadata using OpenAI's text-embedding-3-small model. When a customer asks for something, we embed their query and find the closest products in vector space.

Simple enough, right? Well, the devil lives in the implementation.

Bug 1: Summary Pollution — When Memory Becomes Contamination

The Symptom

A tester was chatting with the bot about suits. Ten messages into the conversation, they pivoted: "actually, show me some hoodies."

The bot responded with... more suits. Confidently. As if the word "hoodies" hadn't been spoken.

The Investigation

I dove into the logs. The search query being sent to pgvector wasn't just the customer's message. It was the customer's message plus a conversation summary that the system had been maintaining.

The summary looked like this:

Customer is looking for a $300 formal suit for a wedding occasion. 
They prefer dark colors and slim fit. Budget is flexible for the right piece.

This summary was being concatenated with the customer's latest message before embedding. So the actual search query became:

Customer is looking for a $300 formal suit for a wedding occasion. 
They prefer dark colors and slim fit. Budget is flexible for the right piece.
show me hoodies

When you embed that block of text, what do you get? An embedding that's 80% "formal suits" and 20% "hoodies." The vector math doesn't care that the customer changed their mind. It cares about token frequency and semantic weight. And the summary — being longer and more detailed — dominated the embedding completely.

The Fix

I killed the conversation summary. Completely. Ripped it out.

But I didn't throw away the concept of memory. Instead, I replaced it with a structured Customer Profile — a lean set of bullet points tracking style preferences, colors, budget, likes, and dislikes:

interface CustomerProfile {
  style_preferences: string[];
  colors: string[];
  budget: string | null;
  likes: string[];
  dislikes: string[];
  occasion: string | null;
}

The critical design decision: this profile gets injected into the response prompt (so the AI can personalize its replies), but it never touches the search query. Search and memory became two completely separate paths.

I felt good. Bug squashed. Time to test.

That feeling lasted about four minutes.

Bug 2: Raw Messages Make Terrible Search Queries

The Symptom

With the summary gone, the search now used the customer's raw message as the query. The next test message was:

acctaly i dont want a hoodie i have a wedding ocation

The search returned a mix of hoodies and wedding outfits. Which sounds reasonable until you realize the customer explicitly said they don't want a hoodie.

The Investigation

This one was immediately obvious once I looked at it with fresh eyes. The customer's message contains:

"hoodie" — something they explicitly DON'T want
"wedding" — something they DO want
"acctaly", "dont", "ocation" — typos everywhere

Text embeddings don't understand negation. They don't know that "don't want a hoodie" means the opposite of "hoodie." To the embedding model, the word "hoodie" fires up the same semantic neighborhood regardless of whether it's preceded by "I love" or "I don't want."

And the typos? text-embedding-3-small handles them surprisingly well in isolation, but when you combine misspelled negations with misspelled targets in a single query, the embedding becomes a semantic smoothie. It picks up everything and commits to nothing.

The Fix

I introduced a dedicated Search Call — a separate, lightweight AI call whose only job is to interpret what the customer wants and produce a clean search query.

const searchInterpretation = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [
    {
      role: "system",
      content: `You are a search query interpreter. Given a customer message, 
      extract ONLY what they want to find. Ignore negations (what they don't want). 
      Output a short, clean search phrase.`
    },
    {
      role: "user",
      content: `Customer said: "${customerMessage}"`
    }
  ],
  max_tokens: 150,
});

Input: ~60 tokens. Output: ~20 tokens. Cost: negligible.

For "acctaly i dont want a hoodie i have a wedding ocation," the search call returns: "wedding occasion outfit". Clean, correct, typo-free.

Two bugs down. System's looking solid. Let me just add a little context to help the search call...

Bug 3: Bot Reply Dominance — The Loudest Voice in the Room

The Symptom

I figured the search call could benefit from a bit of context. So I fed it two messages: the bot's previous reply and the customer's latest message.

The customer said: "hoodies"

The bot's previous reply was:

Great choice! For a wedding, I'd recommend our Premium Wool Blend Suit in charcoal — 
it's $289 and perfect for formal occasions. We also have the Classic Navy Blazer Set 
at $245 which pairs beautifully with dress pants. Would you like to see more formal options?

Search results: suits and blazers. Not a hoodie in sight.

The Investigation

Count the tokens. The bot's reply: ~50 words about suits, prices, formal wear. The customer's message: 1 word — "hoodies."

When you embed that combined text, the suit-related tokens outnumber the hoodie token roughly 50 to 1. The embedding lands squarely in "formal menswear" vector space, with "hoodies" contributing approximately nothing.

This is a fundamental issue with how embeddings work. They represent the average semantic meaning of the entire input text. A single word cannot fight against a paragraph.

The Fix

Zero history for the search call. Absolutely none.

// SEARCH CALL — customer's latest message ONLY
const searchMessages = [
  {
    role: "system" as const,
    content: "Extract what the customer wants to search for. Short phrase only."
  },
  {
    role: "user" as const,
    content: `Customer said: "${latestCustomerMessage}"`
  }
];

This created what I started calling the Two-Context Architecture:

	Search Context	Response Context
Purpose	Decide WHAT to search for	Decide HOW to respond
Input	Customer's latest message only	6 messages + profile + search results
History	None	Recent session window
Cost	~60 tokens	~500 tokens

The search call is deliberately amnesiac. The response AI handles context. The search AI handles intent. Separation of concerns, but for AI calls.

Bug 4: The Pajama Problem — When "Night" Means Everything

The Symptom

The search call was working beautifully. But one product kept showing up where it didn't belong: the "Cozy Night Deluxe Loungewear Set."

It's pajamas. Comfortable, stay-at-home pajamas.

It showed up in results for:

"date night outfit" (because "night")
"evening wear" (because "night" is semantically close to "evening")
"casual summer outfit" (because "cozy" and "casual" are neighbors)

The Investigation

This was an embedding similarity threshold problem. I had set the threshold at 0.1 — meaning any product with a cosine similarity above 0.1 was returned as a match.

For context, with text-embedding-3-small, truly relevant products score around 0.3-0.5, somewhat relevant products score 0.15-0.3, and noise lives below 0.15.

At 0.1, I was scooping up enormous amounts of noise. The pajama set sat at around 0.15-0.22 similarity with a huge range of queries.

The Fix

Single threshold at 0.3. No near-match tier. Clean cuts only.

But a high threshold means sometimes you get no results. So I built a fallback chain:

async function searchProducts(query: string, storeId: string) {
  // Tier 1: Semantic search with strict threshold
  let results = await semanticSearch(query, storeId, 0.3);

  if (results.length === 0) {
    // Tier 2: ILIKE text match (catches exact keyword matches)
    results = await textSearch(query, storeId);
  }

  if (results.length === 0) {
    // Tier 3: Return available categories
    const categories = await getStoreCategories(storeId);
    return { results: [], categories, fallback: true };
  }

  return { results, categories: null, fallback: false };
}

Four bugs fixed. The search pipeline was now clean, fast, and accurate. Then I looked at the actual responses.

Bug 5: The Response That Ignores Its Own Data

The Symptom

Customer conversation, 10 messages deep, all about suits. Customer says: "actually, show me hoodies."

Search call returns hoodies (correctly!). Hoodies are injected into the response prompt as search results.

The bot responds: "I think you'll love our Classic Charcoal Suit for formal occasions..."

The search found the right products. The response ignored them completely.

The Investigation

Here's what the model was seeing:

System prompt: Store persona, sales instructions, tone guidance
Chat history: 10 messages about suits (~400 tokens)
Search results: 3 hoodies (~150 tokens)
Latest customer message: "actually, show me hoodies" (6 tokens)

The model followed the dominant topic. Ten messages of suit conversation created a strong gravitational pull. The hoodies in the search results were a small island in a sea of formal wear.

The Fix

I injected the customer's latest message directly into the system prompt, with an explicit instruction:

const systemPrompt = `You are ${persona.name}, a sales assistant for ${storeName}.

${persona.instructions}

---
The customer's latest message: "${latestCustomerMessage}"
IMPORTANT: Your reply MUST directly address this latest message. 
If the customer asked about a new topic or product, focus on THAT topic, 
not the previous conversation.
---

${searchResults ? `Available products matching their request:\n${formatProducts(searchResults)}` : ''}
`;

System prompts receive disproportionate attention from language models. By putting the customer's latest message there — not just in the chat history — it becomes a directive the model actually follows.

The Final Architecture

Customer message
    |
    v
SEARCH CALL (~60 tokens)
    Input: "Customer said: '[msg]'. Call search_products."
    History: NONE
    |
    v
Search pipeline:
    Semantic search (threshold 0.3)
    -> ILIKE fallback
    -> Category fallback
    |
    v
RESPONSE CALL (~500 tokens)
    System: persona + profile + "Latest: [msg]" + search results
    History: 6 most recent session messages
    |
    v
Response + product cards

Two AI calls per message. One dumb (search), one smart (response). Each with its own carefully scoped context window.

The Numbers

Metric	Before	After
Tokens per message	~1,820	~830
Cost per 100K messages	~$30	~$14
Reduction	—	55%

By adding a second AI call, total token usage went down by 55%. Less context, better results, lower cost.

Lessons Learned

1. AI Bugs Are Layered Like Onions

Each bug was invisible until I fixed the one above it. This is different from traditional software — AI bugs form stacks where one bad behavior masks another.

2. Embeddings Don't Understand Negation

"I don't want X" and "I want X" produce nearly identical embeddings. Don't embed raw text. Use a language model to interpret intent first.

3. Separation of Concerns Applies to AI Calls

Search needs amnesia. Response needs memory. Mixing them is how you get suits when someone asks for hoodies.

4. System Prompts Are Your Steering Wheel

When a long conversation history pulls the model in one direction, the system prompt is the only thing powerful enough to redirect it.

5. Test Topic Switches, Not Just Topic Continuation

The bugs only appeared when the customer changed their mind. Topic switches are where AI systems break. Make them a first-class test case.

Five bugs. Five fixes. Eight hours. One architecture that actually works.

And probably another five bugs hiding underneath, waiting for the right query to reveal them.

I'm building Provia — an AI-powered sales platform — from Gaza. I document every bug, every fix, and every architecture decision. Follow me @AliMAfana for the real version of building in public.

Previous articles:

A Stranger Audited My AI Product for Free. Here's What They Found.

Ali Afana — Mon, 20 Apr 2026 15:23:01 +0000

Three weeks ago I left a comment on a Dev.to article. Today, that comment turned into a full accessibility audit of my product — published publicly, with my real name, my real store URL, and every violation listed in detail.

I asked for it. And I'd do it again.

How It Started

@AgentKit published a piece called "We Scanned 30 SaaS Pricing Pages for Accessibility. 70% Failed." I was in the comments talking about AI product interfaces — specifically the product cards my chatbot renders inline. I described them honestly: styled <div> blocks, no semantic structure, no landmark, no list boundary.

Their response: "Would it be useful if we ran a proper axe pass on a live Provia page + a short screen reader walkthrough?"

I said yes. They said they'd keep the store name out of it if I wanted.

I said put it in. It's a test store. And if we're going to do build-in-public, let's actually do it.

What They Found

The full audit is in their article: We Audited Provia's AI Shopping Chat. Here's What the Before Looks Like.

Short version: 4 violations. 1 serious. 3 moderate.

The Serious One

My product card rail — the horizontal scroll of cards that appears when you search for products — is completely invisible to keyboard users. It's a <div> with display: flex; overflow-x: auto. No tabindex. No focusable children. A keyboard-only user literally cannot scroll through search results.

I built a shopping interface where the products are unreachable without a mouse.

That sentence is hard to write. But that's the point.

The Moderate Ones

Chat input has no accessible name. The placeholder says "Type a message..." but placeholder is not a label. Screen readers announce it as "edit text, blank." The user has to guess what the field does.

Product cards have no list semantics. Five product cards rendered as five sibling <div>s. No <ul>, no role="list", no role="listitem". A screen reader user hears a flat stream of text — product name, description, price, product name, description, price — with no "list, 5 items" on entry and no "card 2 of 5" marker between them.

No <main> landmark. The entire chat interface has no landmark structure.

What I Got Right

This part surprised me. Every <img> in my product cards has a real alt attribute with the actual product name. AgentKit said this was better than 70% of the AI surfaces they scan — they've seen entire rails where every image announces as "graphic, graphic, graphic."

That wasn't an accident. Early on, I made the AI generate product descriptions that flow through to the image alt text. I didn't do it for accessibility — I did it because it seemed right. Turns out "it seemed right" was the correct instinct.

What It Feels Like

Reading your own HTML described as "naked divs dressed up" is humbling.

But here's the thing: I already knew. When I first described my product cards in that comment thread, I used the exact words "totally naked divs." I knew the structure was wrong. I just hadn't prioritized it because no one was complaining.

That's the trap. No one complains about accessibility because the people affected can't use your product in the first place. They don't file bug reports. They just leave.

The AgentKit audit gave me something I couldn't give myself: a number. Not "I should probably fix the accessibility someday" but "4 violations, 1 serious, 6 DOM nodes affected, here's the exact axe-core output." Numbers create urgency. Vague guilt doesn't.

What I'm Fixing

The keyboard navigation fix is already in progress. The scrollable card container gets tabindex="0", the cards get proper focus management with arrow keys, and the focus ring follows Provia's design system so it looks intentional, not like a browser default.

The aria-label on the chat input is three characters of code change. It ships with the keyboard fix.

After that: role="list" on the card container, role="listitem" on each card, and a <main> landmark wrapping the chat interface.

When the fixes land, AgentKit re-runs the same scanner against the same URL with the same "show me hoodies" query. Same axe rules. Same everything. And they publish Part 2 — the after-diff.

Why I Wanted This Public

I could have asked them to keep Provia's name out of it. They offered. I said no.

Three reasons:

1. "Build in public" means the broken parts too. I've published articles about my AI recommending pajamas for date night, about hallucinating fake products, about every API route being wide open. Accessibility gaps are the same category: real problems in a real product. If I only share the wins, the "build in public" label is marketing, not transparency.

2. Other founders need to see this process. Not just the violations — the process. Someone offers to audit you. You say yes. They find problems. You fix them. Everyone learns. That's how it's supposed to work. But most founders are too afraid of looking bad to let anyone in. I get it. I'm publishing this with my real name attached to "your product cards are unreachable with a keyboard." It's uncomfortable. But the alternative is pretending the problem doesn't exist until a real user gets hurt by it.

3. The "after" article is more valuable than the "before." Part 1 alone is just a list of problems. Part 1 + Part 2 together is a case study in fixing accessibility in a real AI product. That's the article I want to exist — not because it makes me look good, but because when the next founder searches "accessibility AI chat interface," they find a real before-and-after with real code diffs instead of another generic WCAG checklist.

What I Learned About My Own Thinking

The most useful sentence in the private report was this:

"Three focusables on a surface whose entire purpose is showing products."

Three. The Back link, the chat input, and the Send button. That's it. On a shopping interface. The products — the entire reason the page exists — are invisible to the Tab key.

I was building for sighted mouse users because that's what I am. Every time I tested my app, I typed a query, scrolled the cards with my trackpad, and thought "this works." It did work — for me. For a keyboard-only user, or a screen reader user, it was a dead end.

That sentence rewired how I think about every component I build going forward. Not "does it look right?" but "can someone reach it without a mouse?"

If You're Building an AI Interface Right Now

Run axe-core against your product page. Right now. Before you publish your next feature.

npm install -g @axe-core/cli
axe https://your-app.com/your-product-page

It takes 30 seconds. The output will probably surprise you.

If you find violations and you don't know how to fix them — the axe-core rule descriptions are the best starting point. Each rule links to the relevant WCAG criterion and gives you the exact fix.

And if you want someone to actually audit your surface properly, reach out to teams like @AgentKit. They did mine for free, gave me the report privately first, and let me decide what to publish. That's how this should work.

This is Part 1 from my side. Part 2 — the after-diff — comes when the fixes ship.

I'm building Provia, an AI sales platform, from Gaza. I document every bug, every fix, and every lesson. Follow me @AliMAfana for the real version of building in public.

Previous articles:

How I Cut My AI Chatbot Costs by 55% With One Architecture Change

Ali Afana — Sat, 18 Apr 2026 08:22:00 +0000

TL;DR: I split one big GPT-4o-mini call into two small, specialized calls. Tokens per message dropped from ~1,820 to ~830. Projected cost went from $300/1M messages to $140/1M messages. Here's exactly how.

The $300 Problem

I'm building Provia, an AI-powered e-commerce platform where an AI sales chatbot handles customer conversations — discovery, product search, objection handling, closing. The AI model is GPT-4o-mini, which is already one of the cheapest options out there.

After my first real end-to-end test — a 42-API-call conversation that consumed 30,654 tokens and cost $0.0054 — I sat down and did the math. At scale, my architecture would cost $30 per 100K messages and $300 per 1M messages. For an indie SaaS product, that's a margin killer.

The worst part? Most of those tokens were wasted. The AI was looping through the same searches, re-reading old context it didn't need, and writing responses three times longer than necessary. The problem wasn't the model. It was my architecture.

One structural change cut costs by 54.4%. No model downgrade. No quality loss. Actually, response quality went up because the AI stopped confusing itself with stale context.

The Before: One Big Call Per Message

My original architecture was the obvious one. Every time a customer sent a message, I made a single OpenAI call that looked like this:

Component	Token Cost
System prompt (persona, instructions, rules)	~500 tokens
Conversation history (last 20 messages)	~1,000 tokens
Conversation summary (AI-generated recap)	~200 tokens
Model response (avg)	~120 tokens
Total per message	~1,820 tokens

The system prompt was verbose — 500+ tokens of instructions covering persona, tone, sales stage logic, search rules, and formatting guidelines. The history window was the last 20 messages, both customer and bot. And a conversation summary was injected into every call to give the AI "memory" of earlier topics.

On paper, it seems reasonable. In practice, it created three expensive problems.

The Three Problems That Were Burning Money

1. Summary Pollution

The conversation summary was supposed to help the AI remember context. Instead, it poisoned every interaction.

Here's what happened: a customer asks about red dresses in message #3. The summary captures "customer is looking for red dresses." Ten messages later, the customer asks about shoes. But the summary still says "red dresses." So the AI searches for red dresses and shoes. Then the summary updates to include both. Next message, the customer asks about a specific shoe, and the AI searches for red dresses, shoes, and that specific shoe.

The summary accumulated topics like a snowball. Every search included ghosts of old queries. More searches meant more tool calls, more tokens, more cost.

2. History Bloat

Loading the last 20 messages sounds like a safe default. But in a sales conversation, most of those messages are irrelevant to the current question. If the customer is asking "do you have this in size 8?" they don't need the AI to re-read the greeting, the initial product discovery, and the three messages where they discussed shipping.

Twenty messages at ~50 tokens each (both sides) is 1,000 tokens of context. Most of it noise. The model has to read all of it, process all of it, and pay for all of it.

3. Search Loops

This was the most expensive bug. Because the summary and history contained references to previous searches, the AI would frequently re-trigger searches it had already done. The conversation summary would say "customer was shown product X" and the AI would interpret that as a reason to search for product X again.

In my 42-call test conversation, I counted multiple redundant search cycles — the AI searching for the same products it had already found, because the context told it those products were relevant.

Each unnecessary search cycle costs a tool call round-trip: the model generates search parameters, the function executes, results come back, and the model processes them. That's easily 300-500 extra tokens per loop.

The Fix: Two Small Calls Instead of One Big One

The core insight was simple: searching and responding are different jobs. They need different context.

A search call needs to know what the customer just said. That's it. It doesn't need conversation history, personality instructions, or a summary of past topics. Adding those things actively hurts search quality.

A response call needs personality, recent context, and search results. But it doesn't need 20 messages of history — the last 6 from the current session are enough.

Call #1: The Search Call

// SEARCH CALL — minimal, focused
const searchSys = `You are a product search assistant for "${store.name}".
The customer just said: "${message}"
Call search_products with what they want.`;

const { result: r1 } = await loggedChatCompletion({
  model: "gpt-4o-mini",
  messages: [{ role: "system", content: searchSys }],
  tools,
  max_tokens: 150,
}, ...);

Input: Only the customer's latest message (~60 tokens).
Job: Decide whether to search, and if so, what to search for.
max_tokens: 150 (hard cap — it either calls a tool or it doesn't).
History: Zero. None. Impossible to pollute.

This call is almost free. Sixty tokens in, 100 tokens out at most. And because it has zero history, it can never loop on old searches. It only sees the current message.

Call #2: The Response Call

// RESPONSE CALL — context-aware but bounded
const { result: r2 } = await loggedChatCompletion({
  model: "gpt-4o-mini",
  messages: [
    { role: "system", content: responseSys },
    ...toChat(responseCtx),  // last 6 session messages
    choice,                   // search call's tool decision
    ...toolMsgs,             // search results
  ],
  max_tokens: 250,
}, ...);

Input: System prompt + customer profile + last 6 session messages + search results (~500 tokens).
Job: Write the actual reply to the customer.
max_tokens: 250 (prevents essay-length responses).
History: Last 6 messages from the current session only.

This call has enough context to write a good, personalized response, but not so much that it drowns in irrelevant history.

The Math

Here's the token breakdown, before and after:

Before (Single Call)

Component	Tokens
System prompt	~500
History (20 messages)	~1,000
Summary	~200
Response output	~120
Total	~1,820

After (Two Calls)

Component	Tokens
Search call input	~60
Search call output	~100
Response call input	~500
Response call output	~170
Total	~830

Token reduction: 54.4%

Cost at Scale

Using GPT-4o-mini pricing ($0.15/1M input tokens, $0.60/1M output tokens):

Metric	Before	After	Savings
Tokens per message	~1,820	~830	54.4%
Cost per message	~$0.0003	~$0.00014	53.3%
Cost per 100K messages	~$30	~$14	$16 saved
Cost per 1M messages	~$300	~$140	$160 saved

At 1M messages, that's $160 back in your pocket every month. For an indie SaaS, that's the difference between profitable and not.

Bonus Optimizations That Stacked

The two-call split was the biggest win, but three other changes compounded the savings.

Session-Based Memory Instead of Fixed Window

Instead of always loading the last 20 messages regardless of when they were sent, I switched to session-based windowing. If there's a gap of 30+ minutes between messages, that's a new session. The response call only sees messages from the current session (last 6 max).

This means if a customer comes back the next day, the AI doesn't reload yesterday's entire conversation. It starts fresh with their profile data, which contains everything it needs to personalize.

Impact: Eliminated 60-80% of irrelevant history tokens in returning-customer conversations.

Customer Profile Instead of Summary

The conversation summary was unstructured text — a paragraph the AI generated after each exchange. It was expensive to generate, expensive to include, and caused the search loop problem.

I replaced it with a structured customer profile: bullet points covering name, archetype, preferences, and current intent. This profile is updated incrementally, not regenerated from scratch. It's smaller (~80 tokens vs ~200), more precise, and doesn't accumulate stale search topics.

Impact: 60% reduction in "memory" token cost, plus elimination of search pollution.

Product Card Filtering

In the old architecture, when the AI searched for products, all results were sent back to the customer as product cards — even if the AI only mentioned one of them in its response. This didn't affect token cost directly, but it confused customers and led to follow-up messages asking about products the AI didn't recommend.

Now, the frontend only renders product cards for items the AI explicitly referenced in its response text. Fewer confused follow-ups means fewer total messages, which means fewer API calls.

Impact: Hard to quantify, but anecdotally reduced "what about this one?" follow-up messages.

Why This Works (The Principle)

The underlying principle is context isolation. Different tasks need different context windows. When you shove everything into one call, you're paying for context that actively degrades output quality.

Think of it like database queries. You wouldn't write SELECT * FROM every_table when you only need one column from one table. But that's exactly what a single-call architecture does with LLM context.

The two-call pattern works because:

The search call is stateless. It doesn't know or care about conversation history. This makes it immune to context pollution and extremely cheap.
The response call is bounded. It has enough context to be helpful (6 recent messages, customer profile, fresh search results) but not so much that it wastes tokens on noise.
max_tokens caps prevent runaway costs. The search call can't exceed 150 tokens. The response call can't exceed 250. This eliminates the long tail of expensive responses.

The Tradeoffs

This isn't free. There are real tradeoffs:

Two API calls means two round-trips. Latency increases by the duration of the search call (~200-400ms for GPT-4o-mini). In practice, users don't notice because the search call is fast and the total response time stays under 2 seconds.

The search call can't reference history. If a customer says "show me more like the last one," the search call doesn't know what "the last one" is. I handle this by having the response call detect anaphoric references and include the last-shown product ID in the search context. It's an edge case, but it needs handling.

Two calls means two points of failure. If the search call fails, you need fallback logic. I default to skipping search and letting the response call work without product results — the AI can still have a conversation, it just can't recommend products until search recovers.

None of these tradeoffs have been deal-breakers. The cost savings far outweigh the added complexity.

Try This Today

If you're running an AI chatbot with a single-call architecture, here's a checklist to estimate your own savings:

Measure your current tokens per message. Log input and output tokens for 100+ real messages. Calculate the average.
Identify what context each task actually needs. List every component in your prompt (system instructions, history, summaries, tool results). For each one, ask: "Does the model need this to do its current job?"
Split calls by responsibility. If your model is both deciding what to do (search, lookup, API call) and generating a response, those are two different jobs. Separate them.
Set max_tokens aggressively. For tool-calling decisions, 100-200 tokens is usually enough. For responses, set a cap based on your desired response length. A chatbot reply rarely needs more than 250 tokens.
Replace summaries with structured data. If you're generating text summaries to maintain context, switch to structured profiles or key-value pairs. They're smaller, more precise, and less likely to cause context pollution.
Use session windows, not fixed windows. Don't load the last N messages blindly. Detect session boundaries (time gaps, topic changes) and only load relevant recent context.

The two-call pattern isn't specific to e-commerce or sales bots. Any chatbot that does retrieval + response can benefit from this split. RAG pipelines, customer support bots, coding assistants — if your model is searching and responding in the same call, you're probably paying 40-60% more than you need to.

Final Numbers

	Before	After
Architecture	1 call per message	2 calls per message
Tokens per message	~1,820	~830
Cost per message	$0.0003	$0.00014
Cost per 1M messages	$300	$140
Search pollution	Frequent loops	Eliminated
Response quality	Verbose, unfocused	Concise, on-topic

One architecture change. Two smaller calls. 55% cost reduction. Ship it.

I'm documenting my entire journey building an AI sales platform from Gaza. Follow me @AliMAfana for more real bugs from a real product.

Previous articles:

DEV Community: Ali Afana

I Told the AI "A Scanner Flagged This" — and It Agreed With Everything

Why my scanner asks an AI anything at all

The flaw: I have to tell the model why it's looking

Countermeasure 1: explicit permission to disagree

Countermeasure 2: show both answers happening

Countermeasure 3: neutralise the retrieval layer

Countermeasure 4 (the structural one): reason before you rule

Did any of it work?

And then I changed one thing

The obvious objection

What I learned

What Is a Vulnerability, Really? Source, Sink, and Taint

The three words

The tap and the glass

What the attack actually looks like

The same shape, four different bugs

Why this is hard to automate

What my scanner does about it

What I learned

Don't Let the AI Find Your Bugs. Let It Judge Them.

The Context

The Problem With "Let the AI Find the Bugs"

The Problem With the Opposite: "Just Use Rules"

The Real Solution: A Courtroom

Results So Far

What I Learned

My Scanner Missed 93% of the Bugs — and That Was the Right First Result

The Context

Why 0.07 Was the Right First Result

Why the Obvious Moves Are Both Wrong

The Climb

Fix 1: Learn the benchmark's vocabulary — recall 0.07 → 0.83

Fix 2: 97 of the 130 remaining misses shared one missing source — 0.83 → 0.95

Fix 3: The taint bridge — 0.95 → 1.00

The Numbers

Three objections, answered before you raise them

What I Learned

What can you do with this?

I Raised Gemma 4's Token Cap. The Dense Model Stopped Refusing.

Why I Re-Ran It

The Experiment

The Results

Scenario 2 — the load-bearing test

What This Means

What's Still Open

The Lesson

I Added Three Rules to Gemma 4. The MoE Searched. The Dense Model Refused.

The Setup

A Disclosure Up Front: Thinking Mode Is Opaque on Gemma 4

Round 1: Where I Was Wrong About Gemma

The Three Rules (Gemma-Only)

Round 2: The MoE Got Better. The Dense Got Worse.

The Results Matrix

A Hypothesis: Architecture, Not Size

What This Means For Shipping

The Lesson

I Was About to Rewrite My Chat Router. The Bug Was Two Lines in a Prompt.

The Bug

The Architecture I Was Sure I'd Have to Rewrite

Where the Suits Came From

The Fix: Two Lines

Architecture vs Prompt Is the Wrong Dichotomy

The Inversion

The Takeaway

How to Find the Postgres Indexes Your Planner Never Picks (I Found 20 of 51)

The Gap in Postgres's Stats

File 1: The Wrapper (query-logger.ts)

Why Not pg_stat_statements or auto_explain?

EXPLAIN Without ANALYZE

The Wrapper

The One Pattern That Matters: Fire-and-Forget

File 2: The Table (query_logs)

File 3: The Dashboard

What It Found

The 2,552 kB Index Nobody Has Ever Used

The Barely-Used Tier

5 Things I Learned Building This

1. Index stats don't equal index value

2. EXPLAIN without ANALYZE is your friend

File 1: The Wrapper (`query-logger.ts`)

File 2: The Table (`query_logs`)

File 1: The Wrapper (`openai-logger.ts`)

File 2: The Table (`api_logs`)