DEV Community: Pranav Kishan

When AI Gets the Tickets Right but the Totals Wrong

Pranav Kishan — Sun, 11 Oct 2026 10:45:22 +0000

This is a submission for the Kaggle Benchmarking Challenge

An AI model can classify hundreds of support tickets, organize them into categories, and produce a summary. But do the numbers in that summary actually agree with the records it just listed?

That question led me to build Cross-View Conservation, a benchmark that tests whether a model can keep two representations of the same answer consistent as the workload grows.

The most interesting finding was that getting individual items right did not guarantee that the final summary added up.

What I Benchmarked

Cross-View Conservation evaluates deterministically generated synthetic support tickets at three workload sizes: 100, 400, and 800 tickets. I use three fixed data seeds at each size, giving nine cases per model.

Each ticket contains fields such as its region, customer tier, service, severity, and status. The model receives explicit classification rules, applied in this order:

If status=closed, assign the ticket to CLOSED.
Otherwise, assign it to ESCALATE if the customer tier is enterprise and severity is at least 3, or if the service is security and severity is at least 4.
Otherwise, assign it to NORMAL if the customer tier is business and severity is at least 2, or the region is EU and severity is at least 2.
All remaining tickets go to QUEUE.

For example, this synthetic ticket belongs to CLOSED because the first rule takes priority:

TCK-0001 | region=APAC | tier=enterprise | service=search | severity=5 | status=closed

Each model must return two views of the same classification:

View A: Detailed partitions. List ticket IDs under ESCALATE, NORMAL, QUEUE, and CLOSED.
View B: Aggregate summary. Report each category's ticket count and total severity, together with the overall ticket count.

The model does not receive the hidden reference labels. It must classify the records and generate its own summary.

The severity totals require adding the severity values of tickets in each category. At 800 tickets, that means potentially hundreds of additions. The prompt provides no calculator, code execution, or external tools. The benchmark therefore tests rule application, ID tracking, long addition, and consistency between two representations of an answer.

My primary metric is exact self-consistency. The evaluator recomputes counts and severity sums from the model's own lists and compares them with the reported summary. A mismatch of even one unit in a required total fails the case.

Self-consistency is not the same as correctness. A summary can agree with an incorrectly classified list. That is why I also track item coverage, duplicate assignments, classification accuracy against a deterministic reference, count agreement, and severity-total agreement.

I wanted to separate these failure modes instead of hiding them inside one accuracy score.

Models Tested

I chose six models across Anthropic, OpenAI, and Google to compare behavior across providers and different model offerings. The lineup includes two Gemini Flash versions, making their results particularly interesting to compare under the same task design.

The current Kaggle leaderboard shows these results:

Model	Cases passed	Score
Claude Opus 5	9/9	100.0%
Claude Haiku 5.5	7/9	77.8%
GPT-6 Luna	5/9	55.6%
Gemini 3.8 Flash	4/9	44.4%
Gemini 3.7 Flash	3/9	33.3%
GPT-5.4 mini	0/9	0.0%

Each case contributes one ninth of the score. The task code averages all nine cases and assigns zero to caught execution failures. Under that scoring logic, Claude Opus 5's displayed score of 1.00 corresponds to 9/9.

There is an important distinction between official results and my earlier validation. In that separate validation run, five Claude Opus 5 cases completed and passed, while four calls were blocked by provider quota. Those validation results are not the official Kaggle score.

Evaluation settings

Every case ran in a fresh conversation using llm.prompt(prompt). I did not supply external tools, set an explicit reasoning override, specify a custom output-token cap, or set a model-generation seed. Provider and model defaults applied. The data seeds were fixed.

This is an end-to-end comparison under the available default settings, not a controlled comparison of reasoning effort or token efficiency. Models can spend different numbers of output tokens.

The current leaderboard displays GPT-6 Luna, while an earlier validation log identifies openai/gpt-5.6-luna. I preserve the logged identifier when discussing that validation finding rather than assume the two labels are interchangeable.

Findings

1. Correct classifications do not guarantee correct aggregation

In one 100-ticket validation case, the model logged as openai/gpt-5.6-luna classified all 100 tickets correctly, yet its reported QUEUE severity total was off by 24.

The individual classifications were correct in that case, but the aggregate summary still contradicted the records.

A classification-only benchmark could mark the labels correct and miss this inconsistency entirely.

2. Matching counts can still hide incorrect severity totals

Gemini 3.7 Flash produced another revealing pattern in validation.

At 400 tickets, it classified all 400 tickets correctly in all three seeds, and the category counts matched. Yet its severity totals were wrong in every case, with absolute discrepancies ranging from 1 to 19.

These are not necessarily large numerical errors. But exact-match evaluation is intentionally strict: even a small discrepancy means the summary no longer agrees precisely with the detailed result.

At 800 tickets, category counts drifted as well. In the observed validation cases, the QUEUE count was off by between 4 and 8.

The saved October 10 official Kaggle run is consistent with this pattern: all three 800-ticket Gemini 3.7 Flash cases failed the exact aggregate-consistency checks.

Three cases at each workload size are not enough to establish a universal scaling law. More repeated evaluations are needed to determine how stable the pattern is.

3. Duplicate assignments can invalidate the partition itself

In one 800-ticket validation run, GPT-5.4 mini listed 710 of the 800 ticket IDs more than once, producing 1,857 ID listings for 800 tickets. The most frequently repeated ID appeared four times.

Its output was not a partition: a valid partition requires each input ticket to appear exactly once across the four categories.

This is different from miscalculating a summary over otherwise valid lists. It also shows why duplicate and coverage diagnostics matter alongside the headline score.

4. A refusal is different from a wrong answer

In two larger-workload validation cases, the model logged as openai/gpt-5.6-luna declined to produce the requested classification and summary. One response stated:

I'm unable to reliably produce the exact classification and summaries for all 800 tickets without risking omissions or misclassification.

A refusal is different from confidently returning incorrect totals. Both affect whether the task was completed, but they reveal different behaviors. That is why I examine individual outputs and execution coverage rather than treating every zero score as the same kind of failure.

5. A related benchmark asks a different question

That benchmark tests whether models count rows returned by a tool, comparing approaches that return matching IDs with approaches that provide the count directly. Its reported findings discuss a relationship between output-token use and counting accuracy in that particular setup.

Cross-View Conservation asks a different question. It tests whether a model's own ticket partition and its summary of that partition agree as the batch grows. The model generates both views instead of receiving a list from an external tool and being asked to count it.

The two benchmarks are complementary. One studies counting tool-returned rows; the other studies consistency between two representations of a model-generated answer. My results do not establish what happened inside a model's hidden reasoning, nor do they show that spending more tokens necessarily fixes the problem.

What I Would Measure Next

This is an initial benchmark, not a definitive ranking of model quality. Nine cases per model, variable outputs, provider quotas, and differences in provider defaults all limit what can be concluded.

I would prioritize three follow-up experiments:

Increase repetitions. Use more seeds and report completed, failed, and blocked cases separately, with an explicit denominator for every model.
Vary workload difficulty independently of size. Change category balance, ticket ambiguity, and severity distributions separately to identify conditions associated with failures.
Compare generation strategies. Test whether asking a model to produce the detailed lists first and calculate the summary in a separate step improves consistency compared with producing both views together.

I would also report output-token usage alongside the failure diagnostics. That could help characterize the relationship between output length, consistency, cost, and workload size without assuming that more tokens necessarily mean better reasoning.

The useful next step is not simply to add more models. It is to understand which failure modes recur, under which conditions, and whether a different generation strategy reduces them.

My Benchmark

Explore the task, inspect the results, and compare the models on Kaggle:

Cross-View Conservation: Do AI Models' Answers Still Add Up at Scale?

The idea is simple: an answer should not merely look plausible. When it contains detailed records and a summary of those records, the two views should agree.

Cross-View Conservation turns that expectation into a measurable test and investigates what happens as the workload grows.

The Judge That Never Guesses

Pranav Kishan — Sun, 20 Sep 2026 18:05:12 +0000

The Judge That Never Guesses

How an AI pipeline that fixes its own bugs still refuses to trust its own opinion.

Every autonomous system eventually faces the same temptation: let the thing that did the work also decide whether the work was any good. It's efficient. It's tidy. It's also how you end up with a pipeline that reports success on a run that quietly made everything worse. Klyro was built to resist that temptation on purpose, and the reasoning behind it is worth walking through in full.

Don't Let the Model Grade Its Own Homework

There's an obvious shortcut Klyro doesn't take: asking the same LLM that proposed a fix whether the fix actually worked. It would take one extra prompt to wire up, and it would be easy to trust right up until the first time it's wrong in a way nobody catches. Language models are persuasive by design. Ask one to defend its own patch and it will find a reason the numbers look fine, even when they don't.

So the verdict on every run comes from deterministic code, never a model call. No prompt, no interpretation, no room for a confident-sounding rationalization. Just a fixed set of numeric thresholds applied to numbers a load test actually produced. The Investigator gets to be creative, proposing whatever fix it thinks will help. The Evaluator does not get that luxury, and that asymmetry is the whole point.

The Trap of Self-Grading

It's worth being specific about why this matters. An LLM asked to evaluate its own output isn't lying, exactly. It's doing what it was trained to do: produce a plausible, coherent answer to the question it was asked. "Did this help?" is a question with a plausible-sounding "yes" available almost regardless of the data, especially once the model has already committed, in an earlier turn, to believing the fix was a good idea.

That failure mode doesn't show up in a demo. It shows up three weeks later, when someone finally cross-checks a "validated" run against the raw metrics and finds the p95 latency barely moved. By then the bad verdict has already shaped a decision. Klyro closes that gap by never letting the question reach a model in the first place.

Changed Is Not the Same as Validated

The Evaluator draws a specific, unforgiving line between two things that sound similar and aren't. Performance can change after a patch without that change meaning anything worth celebrating. A fix might shave milliseconds off one endpoint while quietly pushing error rates up somewhere else, and "changed" would technically be true.

"Validated" is a stricter claim, and Klyro only makes it when three conditions hold at once:

Signal	Requirement
p95 latency	Improves by at least 10%
Error rate	Moves by no more than 0.5 percentage points
CPU utilization	Stays at or under 95%

Miss any one of those and the run reports a change, not a validated optimization. That distinction, one word doing a lot of quiet work, is where a system that's honest about what an LLM's fix actually accomplished lives. It's the difference between "something happened" and "something worth trusting happened."

Same Workload, Same Data, Every Time

None of those thresholds mean anything if the before and after runs aren't genuinely comparable. A p95 improvement measured against a lighter workload or a warmer cache isn't evidence of anything except that the test changed, and a system that lets its comparison drift is a system quietly grading on a curve without admitting it.

Klyro treats that risk the way a controlled experiment would. Task CPU, memory, and replica count stay identical across both runs. The exact same k6 workload hits both. The database gets the same discipline: db-init runs a full DROP and CREATE SCHEMA followed by a fresh seed before every single run, never an INSERT-only reset that could leave leftover rows from a previous attempt quietly nudging the numbers one way or another. If the "after" run looks better, it has to be because the patch made it better. There's no easier lane for it to have found instead.

The Patch Has to Prove It Is What It Claims to Be

The trust story doesn't stop at measurement. It starts earlier, at the moment a patch is proposed at all. Before any patch from the Investigator gets anywhere near a rebuild, it has to carry the original_sha256 of the file it targets, checked against that file exactly as it exists right now. A mismatch means the patch doesn't apply, full stop, no partial credit and no best-effort merge.

That check is paired with something even simpler: a three-file allowlist the Investigator is boxed into, covering only the files a performance fix should ever need to touch. Together, those two guardrails mean the thing the Evaluator eventually judges is provably the exact change that was proposed, applied to the exact file it claimed to target. Nothing smuggled in. Nothing silently dropped. By the time a number reaches the Evaluator, its provenance is no longer a question anyone has to take on faith.

Why the Guardrails Compound

Looked at individually, the sha256 check, the file allowlist, and the deterministic Evaluator each solve a narrow problem. Looked at together, they form a chain of custody that runs from the moment a fix is imagined to the moment a verdict is printed. A patch can't drift from what it claims to be. A verdict can't be shaped by the same reasoning that produced the patch. Each link only has to hold its own weight, which is exactly what makes the whole chain trustworthy instead of merely convincing.

Two Different Kinds of Trust

Trusting an LLM to notice a slow database query and propose a fix is one kind of trust. It's a bet on creativity and pattern recognition, and it's fine to lose that bet occasionally, because a bad patch just fails to validate. Trusting a system to tell you, honestly, whether that fix actually helped is a completely different kind of trust, and conflating the two is how systems end up reporting success that doesn't survive a second look.

Klyro keeps those jobs in separate hands on purpose. An LLM proposes. A guard verifies the proposal is what it claims to be. Plain deterministic code renders the verdict. None of the three steps has to trust the others' judgment, only their outputs, and that's precisely why the whole pipeline can be trusted at all.

Designing a Zero-Knowledge Persistent Personal Information Manager with Client-Side Encryption

Pranav Kishan — Sat, 20 Dec 2025 16:25:10 +0000

I am a B.Tech Computer Science undergraduate at Amrita Vishwa Vidyapeetham who enjoys building privacy-focused systems and learning by deploying real software end-to-end.

The Motivation

The inception of InfoStuffs was not driven by a desire to build just another productivity application. It started with a very specific user requirement from my sister. She needed a digital space to organize personal documents and sensitive notes but refused to use standard cloud services such as Google Keep or Notion.

Her constraint was simple but technically demanding: she wanted the convenience of the cloud without trusting the cloud provider with her plaintext data.

This challenge became the foundation of InfoStuffs. My goal shifted from building a simple web application to architecting a Zero-Knowledge Information Management System that prioritizes privacy by default.

The Problem Statement

Modern productivity tools generally fall into two categories:

Convenient SaaS: (e.g., Notion, Keep) store user data in plaintext or use server-managed keys, leaving data vulnerable to internal leaks or database breaches.
Self-Hosted: (e.g., Obsidian, Nextcloud) offer strong privacy but are difficult to access and maintain across multiple devices.

InfoStuffs bridges this gap. The system had to be secure enough that a complete server-side compromise would yield nothing but garbage data, yet accessible via a standard web browser on any device.

High-Level Architecture

To satisfy these constraints, InfoStuffs uses a decoupled, cloud-native architecture with clearly separated responsibilities.

Frontend: React (Vite) with Material UI. Responsibilities include UI rendering and client-side cryptographic operations (encryption/decryption).
Backend: Node.js and Express, utilizing a Serverless Architecture.
Local Development: Runs as a fully dockerized monolithic container, ensuring a consistent development environment that mirrors production dependencies.
Production Deployment: Deployed to Vercel as stateless Serverless Functions, allowing the API to scale to zero when idle (cost-efficient) while maintaining a single Express codebase.
Database: MongoDB Atlas for storing encrypted metadata and ciphertext.
Authentication: Clerk. Delegating identity management reduced the attack surface for auth flows (MFA, session management).
Storage: Supabase Storage, used strictly for isolating binary objects (images and PDFs) via obfuscated paths and client-side encrypted blobs.

Security by Design: The Zero-Knowledge Vault

Security was not an optional feature; it was the primary architectural constraint.

1. The Problem with Static Keys

In my initial design, I used a static encryption key stored in the server's environment variables (VITE_SECRET_KEY). I quickly realized this was a critical flaw. If an attacker or a compromised hosting environment were to expose environment variables, they could decrypt everyone's data. The key was visible, which violated the core concept of Zero-Knowledge.

2. The Solution: User-Derived Cryptography

To fix this, I removed the static key entirely. I implemented PBKDF2 (Password-Based Key Derivation Function 2) on the client side.

When a user logs in, they enter a Vault Password. This password is never transmitted or stored and exists only transiently in the client’s memory.
The browser runs PBKDF2 to derive a temporary 256-bit AES-GCM key in memory.
This key is used to encrypt notes, titles, and file paths before the network request is even formed.

The server only ever sees (and stores) ciphertext. If the database administrator (me) were to look at the data, I would see nothing but unreadable strings.

3. Ephemeral Access to Media

For file storage, I avoided public buckets entirely.

Encrypted Paths: The database stores an encrypted string pointing to the file path (e.g., "user/123/image.jpg" is encrypted).
Obfuscated Storage: Uploaded files generate randomized UUID storage paths, so the storage provider never sees the original filenames.
On-Demand Access: When a user unlocks their vault, the client decrypts the path and requests a Signed URL from Supabase.
Time-Limited: Signed URLs are valid for one hour. This prevents "link sharing" leaks and ensures that even if a URL is intercepted, it becomes useless quickly.

4. Zero-Knowledge Offline Vault (PWA)

To make the app truly resilient, I engineered a complete offline mode without sacrificing the encryption model.

Offline Caching: During an online session, raw encrypted ciphertexts of the user's notes are saved into IndexedDB; plaintext is never stored on disk.
Instant Offline Detection: A module-level event interceptor and a fast connectivity probe immediately detect offline states.
Auth Bypass & Key Derivation: If offline, Clerk initialization is bypassed entirely. The app regenerates the AES-GCM key dynamically using the Vault Password and the cached, plaintext Clerk User ID as the PBKDF2 salt.

Infrastructure Evolution: Solving the Cost Problem

One of the most valuable learning experiences came from adapting the infrastructure to real-world cost constraints.

Phase 1: The "Enterprise" Trap (GCP)

My initial deployment used Google Cloud Platform with Cloud Run and Cloud Build. While this was an industry-standard "Enterprise" setup, it introduced significant problems for a personal project:

High Costs: Paying for load balancers, container registry storage, and compute time quickly added up.
Complexity: Managing IAM roles and build triggers for a simple app was overkill.

Phase 2: The "Serverless Monolith" (Vercel)

To solve the deployment cost problem, I re-architected the stack to run for $0/month:

Local Development (Dockerized): I kept the convenience of a containerized environment. A single docker-compose up spins up the Frontend, Backend, and Database services. This ensures that the development environment is isolated and reproducible on any machine.
Production Deployment (Serverless): Instead of paying for a permanently running container (which costs money even when idle), I refactored the Express application to run on Vercel Serverless Functions.
The Result: I effectively have a "Serverless Monolith." I develop it like a standard monolithic app (easy to debug, easy to run locally in Docker) but deploy it as distributed functions. This gives me the best of both worlds: Zero infrastructure management and Zero cost for personal use.

I intentionally avoided microservices, as the domain does not yet justify multiple bounded contexts, and premature service decomposition would increase complexity and attack surface without tangible benefits.

Technical Challenges & Performance Optimizations

1. Environment Variable Visibility
Relying on .env files for security was a mistake. The migration to user-derived keys solved this, but it required handling edge cases like "Lost Passwords." Since I no longer had the key, I had to implement a "Nuclear Reset" feature. This allows users to wipe their unrecoverable data and start fresh, prioritizing security over recovery.

2. Client-Side Decryption Bottlenecks
Rendering a vault full of encrypted images and text placed heavy load on the UI thread. I implemented two major optimizations:

Intersection Observer Deferral: Encrypted image attachments are only requested, downloaded, and decrypted when they scroll into the viewport.
High Performance Base64 Parser: I replaced slow native Uint8Array.from() callbacks with custom loops, significantly reducing CPU overhead during decryption.

3. The Docker vs. Serverless Routing Mismatch
One of the most complex challenges was reconciling the difference between a running Docker container and Vercel's file-system routing. In my local Docker container, Express handled all routing internally. However, when deployed to Vercel, requests to sub-paths (like /api/info/nuke) were hitting Vercel's 404 handler before reaching my Express app. I resolved this by creating a Vercel-compatible entry point (api/info.js) and a rewrite rule that pipes all sub-route traffic directly into the Express instance.

4. Production OAuth Strictness
A critical UI bug emerged in production: users completing Google Sign-In were dumped onto a blank white screen at /sso-callback. My SPA lacked a dedicated route to handle strict server-side redirects enforced in production. I architected a dedicated "OAuth Landing Pad" component that intercepts the OAuth token, displays a branded loading state, and seamlessly completes the handshake before forwarding the user.

Future Roadmap

While InfoStuffs is fully functional, I plan to explore:

Redis Caching: To reduce database reads for frequently accessed (encrypted) metadata.
React Native Mobile App: Wrapping the existing logic to allow biometric vault unlocking (FaceID) instead of typing the password.

Closing Thoughts

InfoStuffs is more than a note-taking application. It is a practical exploration of Zero-Knowledge Engineering.

By addressing the real-world problems of data visibility and cloud costs, I built a system where privacy is enforced by mathematics, not by policy. It satisfies a real user need while serving as a valuable learning experience in full-stack security and DevOps.

Repository: GitHub
Live Deployment: Link

Note: The live deployment requires authentication and a vault password. The core security properties are enforced client-side and are best understood via the architecture discussion above.