DEV Community: Pranav Kishan

The Judge That Never Guesses

Pranav Kishan — Sun, 20 Sep 2026 18:05:12 +0000

The Judge That Never Guesses

How an AI pipeline that fixes its own bugs still refuses to trust its own opinion.

Every autonomous system eventually faces the same temptation: let the thing that did the work also decide whether the work was any good. It's efficient. It's tidy. It's also how you end up with a pipeline that reports success on a run that quietly made everything worse. Klyro was built to resist that temptation on purpose, and the reasoning behind it is worth walking through in full.

Don't Let the Model Grade Its Own Homework

There's an obvious shortcut Klyro doesn't take: asking the same LLM that proposed a fix whether the fix actually worked. It would take one extra prompt to wire up, and it would be easy to trust right up until the first time it's wrong in a way nobody catches. Language models are persuasive by design. Ask one to defend its own patch and it will find a reason the numbers look fine, even when they don't.

So the verdict on every run comes from deterministic code, never a model call. No prompt, no interpretation, no room for a confident-sounding rationalization. Just a fixed set of numeric thresholds applied to numbers a load test actually produced. The Investigator gets to be creative, proposing whatever fix it thinks will help. The Evaluator does not get that luxury, and that asymmetry is the whole point.

The Trap of Self-Grading

It's worth being specific about why this matters. An LLM asked to evaluate its own output isn't lying, exactly. It's doing what it was trained to do: produce a plausible, coherent answer to the question it was asked. "Did this help?" is a question with a plausible-sounding "yes" available almost regardless of the data, especially once the model has already committed, in an earlier turn, to believing the fix was a good idea.

That failure mode doesn't show up in a demo. It shows up three weeks later, when someone finally cross-checks a "validated" run against the raw metrics and finds the p95 latency barely moved. By then the bad verdict has already shaped a decision. Klyro closes that gap by never letting the question reach a model in the first place.

Changed Is Not the Same as Validated

The Evaluator draws a specific, unforgiving line between two things that sound similar and aren't. Performance can change after a patch without that change meaning anything worth celebrating. A fix might shave milliseconds off one endpoint while quietly pushing error rates up somewhere else, and "changed" would technically be true.

"Validated" is a stricter claim, and Klyro only makes it when three conditions hold at once:

Signal	Requirement
p95 latency	Improves by at least 10%
Error rate	Moves by no more than 0.5 percentage points
CPU utilization	Stays at or under 95%

Miss any one of those and the run reports a change, not a validated optimization. That distinction, one word doing a lot of quiet work, is where a system that's honest about what an LLM's fix actually accomplished lives. It's the difference between "something happened" and "something worth trusting happened."

Same Workload, Same Data, Every Time

None of those thresholds mean anything if the before and after runs aren't genuinely comparable. A p95 improvement measured against a lighter workload or a warmer cache isn't evidence of anything except that the test changed, and a system that lets its comparison drift is a system quietly grading on a curve without admitting it.

Klyro treats that risk the way a controlled experiment would. Task CPU, memory, and replica count stay identical across both runs. The exact same k6 workload hits both. The database gets the same discipline: db-init runs a full DROP and CREATE SCHEMA followed by a fresh seed before every single run, never an INSERT-only reset that could leave leftover rows from a previous attempt quietly nudging the numbers one way or another. If the "after" run looks better, it has to be because the patch made it better. There's no easier lane for it to have found instead.

The Patch Has to Prove It Is What It Claims to Be

The trust story doesn't stop at measurement. It starts earlier, at the moment a patch is proposed at all. Before any patch from the Investigator gets anywhere near a rebuild, it has to carry the original_sha256 of the file it targets, checked against that file exactly as it exists right now. A mismatch means the patch doesn't apply, full stop, no partial credit and no best-effort merge.

That check is paired with something even simpler: a three-file allowlist the Investigator is boxed into, covering only the files a performance fix should ever need to touch. Together, those two guardrails mean the thing the Evaluator eventually judges is provably the exact change that was proposed, applied to the exact file it claimed to target. Nothing smuggled in. Nothing silently dropped. By the time a number reaches the Evaluator, its provenance is no longer a question anyone has to take on faith.

Why the Guardrails Compound

Looked at individually, the sha256 check, the file allowlist, and the deterministic Evaluator each solve a narrow problem. Looked at together, they form a chain of custody that runs from the moment a fix is imagined to the moment a verdict is printed. A patch can't drift from what it claims to be. A verdict can't be shaped by the same reasoning that produced the patch. Each link only has to hold its own weight, which is exactly what makes the whole chain trustworthy instead of merely convincing.

Two Different Kinds of Trust

Trusting an LLM to notice a slow database query and propose a fix is one kind of trust. It's a bet on creativity and pattern recognition, and it's fine to lose that bet occasionally, because a bad patch just fails to validate. Trusting a system to tell you, honestly, whether that fix actually helped is a completely different kind of trust, and conflating the two is how systems end up reporting success that doesn't survive a second look.

Klyro keeps those jobs in separate hands on purpose. An LLM proposes. A guard verifies the proposal is what it claims to be. Plain deterministic code renders the verdict. None of the three steps has to trust the others' judgment, only their outputs, and that's precisely why the whole pipeline can be trusted at all.

Designing a Zero-Knowledge Persistent Personal Information Manager with Client-Side Encryption

Pranav Kishan — Sat, 20 Dec 2025 16:25:10 +0000

I am a B.Tech Computer Science undergraduate at Amrita Vishwa Vidyapeetham who enjoys building privacy-focused systems and learning by deploying real software end-to-end.

The Motivation

The inception of InfoStuffs was not driven by a desire to build just another productivity application. It started with a very specific user requirement from my sister. She needed a digital space to organize personal documents and sensitive notes but refused to use standard cloud services such as Google Keep or Notion.

Her constraint was simple but technically demanding: she wanted the convenience of the cloud without trusting the cloud provider with her plaintext data.

This challenge became the foundation of InfoStuffs. My goal shifted from building a simple web application to architecting a Zero-Knowledge Information Management System that prioritizes privacy by default.

The Problem Statement

Modern productivity tools generally fall into two categories:

Convenient SaaS: (e.g., Notion, Keep) store user data in plaintext or use server-managed keys, leaving data vulnerable to internal leaks or database breaches.
Self-Hosted: (e.g., Obsidian, Nextcloud) offer strong privacy but are difficult to access and maintain across multiple devices.

InfoStuffs bridges this gap. The system had to be secure enough that a complete server-side compromise would yield nothing but garbage data, yet accessible via a standard web browser on any device.

High-Level Architecture

To satisfy these constraints, InfoStuffs uses a decoupled, cloud-native architecture with clearly separated responsibilities.

Frontend: React (Vite) with Material UI. Responsibilities include UI rendering and client-side cryptographic operations (encryption/decryption).
Backend: Node.js and Express, utilizing a Serverless Architecture.
Local Development: Runs as a fully dockerized monolithic container, ensuring a consistent development environment that mirrors production dependencies.
Production Deployment: Deployed to Vercel as stateless Serverless Functions, allowing the API to scale to zero when idle (cost-efficient) while maintaining a single Express codebase.
Database: MongoDB Atlas for storing encrypted metadata and ciphertext.
Authentication: Clerk. Delegating identity management reduced the attack surface for auth flows (MFA, session management).
Storage: Supabase Storage, used strictly for isolating binary objects (images and PDFs) via obfuscated paths and client-side encrypted blobs.

Security by Design: The Zero-Knowledge Vault

Security was not an optional feature; it was the primary architectural constraint.

1. The Problem with Static Keys

In my initial design, I used a static encryption key stored in the server's environment variables (VITE_SECRET_KEY). I quickly realized this was a critical flaw. If an attacker or a compromised hosting environment were to expose environment variables, they could decrypt everyone's data. The key was visible, which violated the core concept of Zero-Knowledge.

2. The Solution: User-Derived Cryptography

To fix this, I removed the static key entirely. I implemented PBKDF2 (Password-Based Key Derivation Function 2) on the client side.

When a user logs in, they enter a Vault Password. This password is never transmitted or stored and exists only transiently in the client’s memory.
The browser runs PBKDF2 to derive a temporary 256-bit AES-GCM key in memory.
This key is used to encrypt notes, titles, and file paths before the network request is even formed.

The server only ever sees (and stores) ciphertext. If the database administrator (me) were to look at the data, I would see nothing but unreadable strings.

3. Ephemeral Access to Media

For file storage, I avoided public buckets entirely.

Encrypted Paths: The database stores an encrypted string pointing to the file path (e.g., "user/123/image.jpg" is encrypted).
Obfuscated Storage: Uploaded files generate randomized UUID storage paths, so the storage provider never sees the original filenames.
On-Demand Access: When a user unlocks their vault, the client decrypts the path and requests a Signed URL from Supabase.
Time-Limited: Signed URLs are valid for one hour. This prevents "link sharing" leaks and ensures that even if a URL is intercepted, it becomes useless quickly.

4. Zero-Knowledge Offline Vault (PWA)

To make the app truly resilient, I engineered a complete offline mode without sacrificing the encryption model.

Offline Caching: During an online session, raw encrypted ciphertexts of the user's notes are saved into IndexedDB; plaintext is never stored on disk.
Instant Offline Detection: A module-level event interceptor and a fast connectivity probe immediately detect offline states.
Auth Bypass & Key Derivation: If offline, Clerk initialization is bypassed entirely. The app regenerates the AES-GCM key dynamically using the Vault Password and the cached, plaintext Clerk User ID as the PBKDF2 salt.

Infrastructure Evolution: Solving the Cost Problem

One of the most valuable learning experiences came from adapting the infrastructure to real-world cost constraints.

Phase 1: The "Enterprise" Trap (GCP)

My initial deployment used Google Cloud Platform with Cloud Run and Cloud Build. While this was an industry-standard "Enterprise" setup, it introduced significant problems for a personal project:

High Costs: Paying for load balancers, container registry storage, and compute time quickly added up.
Complexity: Managing IAM roles and build triggers for a simple app was overkill.

Phase 2: The "Serverless Monolith" (Vercel)

To solve the deployment cost problem, I re-architected the stack to run for $0/month:

Local Development (Dockerized): I kept the convenience of a containerized environment. A single docker-compose up spins up the Frontend, Backend, and Database services. This ensures that the development environment is isolated and reproducible on any machine.
Production Deployment (Serverless): Instead of paying for a permanently running container (which costs money even when idle), I refactored the Express application to run on Vercel Serverless Functions.
The Result: I effectively have a "Serverless Monolith." I develop it like a standard monolithic app (easy to debug, easy to run locally in Docker) but deploy it as distributed functions. This gives me the best of both worlds: Zero infrastructure management and Zero cost for personal use.

I intentionally avoided microservices, as the domain does not yet justify multiple bounded contexts, and premature service decomposition would increase complexity and attack surface without tangible benefits.

Technical Challenges & Performance Optimizations

1. Environment Variable Visibility
Relying on .env files for security was a mistake. The migration to user-derived keys solved this, but it required handling edge cases like "Lost Passwords." Since I no longer had the key, I had to implement a "Nuclear Reset" feature. This allows users to wipe their unrecoverable data and start fresh, prioritizing security over recovery.

2. Client-Side Decryption Bottlenecks
Rendering a vault full of encrypted images and text placed heavy load on the UI thread. I implemented two major optimizations:

Intersection Observer Deferral: Encrypted image attachments are only requested, downloaded, and decrypted when they scroll into the viewport.
High Performance Base64 Parser: I replaced slow native Uint8Array.from() callbacks with custom loops, significantly reducing CPU overhead during decryption.

3. The Docker vs. Serverless Routing Mismatch
One of the most complex challenges was reconciling the difference between a running Docker container and Vercel's file-system routing. In my local Docker container, Express handled all routing internally. However, when deployed to Vercel, requests to sub-paths (like /api/info/nuke) were hitting Vercel's 404 handler before reaching my Express app. I resolved this by creating a Vercel-compatible entry point (api/info.js) and a rewrite rule that pipes all sub-route traffic directly into the Express instance.

4. Production OAuth Strictness
A critical UI bug emerged in production: users completing Google Sign-In were dumped onto a blank white screen at /sso-callback. My SPA lacked a dedicated route to handle strict server-side redirects enforced in production. I architected a dedicated "OAuth Landing Pad" component that intercepts the OAuth token, displays a branded loading state, and seamlessly completes the handshake before forwarding the user.

Future Roadmap

While InfoStuffs is fully functional, I plan to explore:

Redis Caching: To reduce database reads for frequently accessed (encrypted) metadata.
React Native Mobile App: Wrapping the existing logic to allow biometric vault unlocking (FaceID) instead of typing the password.

Closing Thoughts

InfoStuffs is more than a note-taking application. It is a practical exploration of Zero-Knowledge Engineering.

By addressing the real-world problems of data visibility and cloud costs, I built a system where privacy is enforced by mathematics, not by policy. It satisfies a real user need while serving as a valuable learning experience in full-stack security and DevOps.

Repository: GitHub
Live Deployment: Link

Note: The live deployment requires authentication and a vault password. The core security properties are enforced client-side and are best understood via the architecture discussion above.