DEV Community: Atharva Panegai

We ran Pinaka on an open, unresolved Prisma bug. No ground truth. Here's what it found.

Atharva Panegai — Mon, 06 Jul 2026 12:23:41 +0000

The BullMQ benchmark we published last week had a clean scoring mechanism: a real merged PR, a real maintainer comment confirming the bug, and a known correct fix to compare against. This one doesn't have that.

Prisma issue #29480 is open. It has persisted through four minor releases — 7.5.0, 7.6.0, 7.7.0, 7.8.0 — unresolved as of the time we ran this benchmark. There is no merged fix to score against. So we had to validate Pinaka's diagnosis differently: against behavioral evidence in the codebase itself, not a known-correct answer. That's a different kind of benchmark, and it's worth being precise about what it does and doesn't prove.

The bug

When using the PrismaPg driver adapter, filtering nullable String? fields for null values throws a PrismaClientValidationError:

// These all throw:
prisma.user.findMany({ where: { name: null } })
prisma.user.findMany({ where: { name: { equals: null } } })
prisma.user.findMany({ where: { name: { not: null } } })

// This works fine (same syntax, different field type):
prisma.user.findMany({ where: { createdAt: null } }) // DateTime? — no error

The error:

PrismaClientValidationError:
Invalid `prisma.user.findMany()` invocation:
{
  where: {
    name: null
          ~~~~
  }
}
Argument `name`: Invalid value provided.
Expected StringNullableFilter or String, provided Null.

The bug is reproducible, consistent, and type-specific — it only fires on String? fields, not DateTime? or integer fields with identical null filter syntax. It has been open and unresolved across four Prisma minor releases.

What Pinaka was working with

This run had live SDK runtime context — real logs, real stack trace, real request context captured at crash time via @pinaka/sdk. The logs included something important:

[DEBUG] Executing prisma.user.findMany with where: { name: null }
[ERROR] PrismaClientValidationError thrown
[DEBUG] Retried with { name: { equals: null } } - same error
[DEBUG] Tried { createdAt: null } on DateTime? field - works fine

Those last two lines — the retry attempt and the DateTime contrast — are what made this diagnosis possible. Without them, a static-code-only RCA would have seen the error message and stopped at "user passed wrong type." The logs proved it wasn't user error by documenting what was already tried.

Pipeline stats for this run:

Vector search: 25 candidates
Direct stack trace file fetch: 3 files force-included
Total reranker candidates: 45 (25 vector + 20 direct fetch)
Final context passed to LLM: 8 chunks
Reranker: Voyage rerank-2, no fallback triggered
Time to RCA from job pickup: ~3 seconds

What Pinaka diagnosed

Failure class: Logic error in the PrismaPg adapter's SQL generation layer.

Root cause: The PrismaPg adapter incorrectly generates WHERE "name" = NULL instead of WHERE "name" IS NULL for findMany operations on nullable String? fields. In PostgreSQL, = NULL always evaluates to false — IS NULL is the correct predicate. This explains why validation fails before the query even reaches the database.

How Pinaka isolated it: By contrasting three cases found in retrieved test evidence:

Case	Field type	Operation	Result
`findMany({ where: { team_id: null } })`	Integer	findMany	✅ Works
`updateMany({ where: { name: null } })`	String?	updateMany	✅ Works
`findMany({ where: { resale: null } })`	String?	findMany	❌ Fails

The intersection of those three data points isolates the bug to findMany + String? specifically — not a general null handling problem, not a schema problem, not a DMMF problem.

Fault ownership: Framework bug in the PrismaPg adapter, not user error. Any application-side workaround is exactly that — a workaround. The correct fix belongs upstream in Prisma.

Key hypotheses eliminated:

"The field isn't nullable" — ruled out: null records can be created successfully, and the error is client-side validation, not a DB constraint
"Prisma doesn't support null filtering at all" — ruled out by the DateTime? working case with identical syntax
"The fix lives in the client runtime" — ruled out: the error originates in @prisma/client/runtime/library.js, but behavioral evidence points to adapter-level SQL generation as the deeper cause

Directional fix: Change = NULL to IS NULL in the adapter's SQL generation path for findMany + String? cases. Pinaka did not invent API syntax or a specific line-level fix it couldn't confirm from retrieved code — it stated this directionally without overreaching.

Where Pinaka was appropriately uncertain

The adapter's actual source (packages/adapter-pg/) was not retrieved. The exact file and line of the SQL generation fault was not pinpointed.

Confidence was flagged as HIGH in the output, but the caveat correctly noted the adapter source was missing — this is a known limitation of how retrieval works: the ticket framing pulled vector search toward client runtime files rather than adapter internals. An adapter-focused ticket would likely have surfaced the SQL generation code directly.

We're saying this in the body of the post, not buried in footnotes.

What the SDK actually contributed here

The SDK-captured log lines did more diagnostic work than the entire codebase read. Specifically:

The line "Tried { createdAt: null } on DateTime? field - works fine" — captured automatically because the engineer's retry attempt happened in the same request context — is what enabled asymmetric behavior diagnosis. Without that log line, the investigation would have stayed focused on the error message alone, which points to client-side validation, not adapter-level SQL generation.

Static code reading tells you what the code says it should do. Runtime context tells you what actually happened when someone tried it three different ways and got three different results. Those aren't redundant — they're complementary, and in this case the runtime evidence was the lever.

What this proves — and what it doesn't

Unlike the BullMQ benchmark, there is no merged PR to score against. We cannot tell you "Pinaka's diagnosis matches the correct fix" because no correct fix exists yet.

What we can tell you: the diagnosis is consistent with all behavioral evidence available — the asymmetric behavior across field types, the error message origin, the test file contrast cases — and the reasoning chain is transparent enough to evaluate independently.

The harder demonstration here isn't scoring accuracy. It's that Pinaka can reason from behavioral contrast evidence across test files, not just trace a known stack to a known answer. Those are different capabilities, and the second one matters more in the cases where no one has found the answer yet.

If Prisma ships a fix for #29480 that differs from what Pinaka diagnosed, we'll update this post.

Honest limitations

No ground truth — diagnosis validated against behavioral evidence only, not a merged fix
Adapter source not retrieved — exact fault line not pinpointed
Single bug, single repo — not statistically powered
Ticket framing affects retrieval — an adapter-focused ticket would likely have surfaced the SQL generation code directly

We're continuing to run this across more repositories. We'll publish what we find — including results that don't look as clean as this one.

Try it

If you want to see what Pinaka finds on your own codebase, Join waitlist at getpinaka.com. We're building this in public — benchmarks like this one are how we're deciding what to build next, not just what to put in a launch post.

Previously: Can AI actually find the root cause of a bug, or does it just sound confident? (BullMQ benchmark)

Can AI actually find the root cause of a bug, or does it just sound confident?

Atharva Panegai — Sat, 20 Jun 2026 14:14:06 +0000

We built Pinaka to write root cause analyses automatically — read a Jira ticket, search the codebase, trace the failure, post the RCA as a comment. Before putting that in front of real teams, we needed an honest answer to one question: does it actually find the right bug, or does it just sound right?

"Sound right" is the trap with every AI tool that explains why something broke. LLMs are extremely good at producing a plausible-sounding root cause. Plausible and correct are not the same thing, and the gap between them is exactly where a team loses trust in a tool — usually after the first time it sends someone chasing the wrong file for an hour.

So we picked a real bug, in a real production codebase, with a real human-written root cause already on record — and tested two versions of Pinaka against it.

The test setup

We used BullMQ (github.com/taskforcesh/bullmq), a Redis-based job queue library for Node.js with thousands of GitHub stars and heavy production use. It's also part of our own stack, which meant we could judge the RCA quality ourselves, not just trust the AI's confidence score.

The bug: issue #2487 — when a job has stackTraceLimit set and fails multiple times on retry, the stack trace shown never updates. It always shows the first failure, even on the third or fourth retry. A real BullMQ maintainer confirmed it as a genuine bug, and it was fixed in v5.4.6.

We indexed the BullMQ repo into Pinaka, then ran two separate RCA passes:

Run A — code only. A realistically vague Jira ticket, the way an engineer would actually file it — no file names, no hypotheses, just "the stack trace doesn't update after retries." Pinaka had to find the bug using semantic search over the codebase alone.

Run B — code plus runtime context. Same indexed repo, but this time Pinaka also received the kind of execution-time data our SDK is built to capture automatically — real option values, job state, and the actual sequence of events as they occur in a live Redis instance.

We graded both outputs against the real fix the BullMQ team shipped.

What code-only RCA found

Pinaka correctly identified the exact file (job.ts) and exact method (moveToFailed()), and gave a coherent step-by-step trace of how the bug manifested across retries. It even ran through three plausible alternative explanations and ruled each one out with specific evidence from the test suite — which is the kind of reasoning you'd want from a senior engineer doing this by hand.

What it didn't get: the precise line-level cause. It described the bug correctly in behavior but proposed a fix that was functionally valid but more complicated than what BullMQ's maintainers actually shipped. It also had no way of seeing that a closely related, deeper bug existed one layer down — inside the Redis Lua script BullMQ uses for atomic job updates, not in the TypeScript code at all.

Against the human ground truth, we'd score this run roughly 6.5 out of 10 — right neighborhood, not quite the exact fix.

What runtime context found

The second run, with real execution-time data included, didn't just refine the same answer — it surfaced a different, deeper bug entirely: what happens when stackTraceLimit is explicitly set to 0. This turned out to be a real follow-up issue BullMQ's own maintainers filed after the first fix shipped.

The root cause lived in the Lua script's trim condition, which only fired when the limit was greater than zero — so a limit of exactly zero silently skipped the trim step and let stale stack traces leak into Redis. Pinaka named the exact condition (ARGV[4] > 0) and proposed a fix that matched the structure of what the maintainers actually merged.

Against the same ground truth, this run scored closer to 9.2 out of 10.

Why this matters more than the score

The interesting part isn't really "9.2 beats 6.5." It's what kind of mistake runtime context fixed. Static code reading can tell you what the code says it should do. It cannot tell you what's actually sitting in Redis at the moment of failure, or that a boolean condition silently short-circuits when a value is zero, or that the real bug spans two different layers of the system — application code and the Lua script running atomically inside the database. That's not a knowledge gap an LLM can read its way out of. It's a visibility gap, and the only fix for a visibility gap is actually capturing what happened at runtime.

That's the whole bet behind shipping @getpinaka/sdk instead of building a tool that only reads your GitHub repo: a lot of real production bugs don't live in the code you can see. They live in the gap between what the code says and what actually happened.

Where we'd push back on our own result

A few things we don't want to bury in fine print:

Run A's input was more informative than real production logs usually are. We wrote the simulated logs to explicitly state facts an engineer would have to dig for in practice. If anything, this means the 6.5 score is generous to the code-only condition — the real-world gap between code-only and SDK-instrumented RCA is probably larger than what we measured here.

Run A and Run B technically targeted two related but different bugs, not a strict apples-to-apples test on one bug with and without context. They're both part of the same GitHub issue thread, and the second was filed as a direct follow-up to the first — but a tighter version of this test would hold the bug constant and vary only the context.

This is one bug, on one repo, run once per condition. It's a real qualitative case study, not a statistically powered benchmark. We're running the same methodology against issues in TypeORM and Prisma next, and we'll publish what we find — including if it doesn't look as clean as this one.

Run B's runtime context was constructed, not captured live by the SDK. The next version of this test uses a real instrumented service and a real crash, end to end.

We'd rather tell you where this benchmark is thin than have you find it yourself in the GitHub issue thread — which, if you're the kind of engineer who reads root cause analyses for a living, you probably will anyway.

Try it

If you want to see what Pinaka does on your own codebase join the waitlist at getpinaka.com We're a small team building this in public, and benchmarks like this one are how we're deciding what to build next, not just what to put in a launch post.