DEV Community: ADITYA DUTT PANDEY

Harness engineering is not a joke. I have the failed runs to prove it.

ADITYA DUTT PANDEY — Wed, 14 Oct 2026 00:00:00 +0000

I built a coding harness in six hours, and on small tasks it was better than the tools I'd been using.

Then I started trying to make it good, and that's where it got humbling. I've lost more rounds than I've won, and this post is the honest account. I tried, I lost, and I went back in.

Small tasks flatter a harness

A task that runs for ten minutes hides almost every design mistake. The conversation is short, a few files are in play, and nothing has had time to rot.

A thirty-minute task is a different animal. The model has read dozens of files, formed three theories and dropped two of them. It's carrying its own history, and paying for that history again on every call. Most of what I'd built was tuned for the ten-minute case, and I couldn't see that until a long one broke it.

The thing I fought first: tokens

An agent run isn't one request. Every tool call is another API call, and every API call sends the whole conversation again. So a tool result isn't paid for once. It's paid for on every call that comes after it.

In one real mew run, a single file read cost 1,922 tokens and was re-sent 15 times, which is 28,830. Overall, 52% of that run's input was earlier tool results travelling along again. The price of reading a file is its size times the number of calls that follow.

That pushes you two ways at once. You want fewer calls, and you want less in each one.

Fewer calls. Search, outlines and symbol reads let the model find a function without reading a whole file. A read-only scout on a cheap model investigates first, so the main model starts from findings.
Less per call. mew remembers which lines of each file the model has already seen. Ask again and you get only what's new, or a diff if the file changed.

Those held up. Then I got ambitious.

Round one: the variable idea

If files are what fill the context, why keep them in the context at all?

My idea was a variable. Read a file, keep the contents somewhere else, and leave the model a handle. The data lives outside the conversation, so it can't be re-sent on every call.

On paper it was perfect. In practice, the same file was being read four or five times, the tokens were coming back as uncached, and the bill went up. That was the opposite of the point.

I still can't fully explain why they were uncached. I had a theory that prompt caching depends on the start of the conversation staying identical, so anything that rewrites an earlier message breaks it. I never confirmed that, and given what happened next, it may not be the explanation at all.

What did happen was dull. I removed one of the pieces I'd stacked on top of each other, and it behaved a bit better than before. Then I found one very small glitch in my Python and fixed it, and the variable approach now works. All that theory, and the thing standing in the way was a small bug in my own code.

I won't turn that into a lesson about clever design. The lesson is to check the bill, because the bill was the only thing that told me the idea was failing. What gave me something to check it against was the cost report. /cost now says why a call was billed in full: an idle gap, a model change, or a changed prefix. In the run I screenshotted, it reported 25,381 tokens re-billed because the prefix changed.

Round two: the thirty-minute bug

This is the one I'm least proud of.

I gave mew a bug I'd have found myself in an hour or two of digging in my own codebase, and let it plan and investigate. It failed badly. It wasn't designed for work like that, and I tried a few things on the spot that didn't rescue the run.

A long investigation isn't a bigger version of a short task. It needs things a short one never asks for. So I went back and added what I thought was missing:

Memory that survives compaction. When the context fills up, something has to go, and a paraphrase of the whole history loses the facts the investigation stands on. mew now keeps the newest ~20k tokens word for word, folds only older turns into a running summary, and adds a plain list of the files read and changed.
A way to stop reading in circles. The prompt now teaches hypothesis-first investigation: say what you think is wrong, then look for where the real behaviour differs from the expected behaviour. Loop detection catches repeated actions and escalates to a stronger model instead of burning the budget.
A budget that ends cleanly. At 70% mew warns, and when the budget is gone it makes one last call with no tools, so the work so far becomes an answer.

I'm not going to tell you that fixed it. I built those because that run showed me what was missing, and I haven't shown that mew now beats me at a two-hour bug.

The scoreboard

Attempt	Result
Search first, outlines, symbol reads, a scout	Kept
Line map and diffs instead of re-reading files	Kept
Variable handles instead of files in the context	Lost, then fixed a small glitch and it works now
A long, hard bug investigation	Lost, tried again, still not proven

What I'd tell anyone starting

Measure before you optimise. A harness has too many moving parts to improve by feel, and the cost report told me things I'd never have guessed, like one file being re-sent 15 times.

Test long tasks early. Ten-minute runs tell you the loop works and nothing else. The design problems show up at thirty minutes, and by then the architecture is set.

Expect the boring answer. The variable idea sounded like a design problem, and a small Python glitch was in the way.

I started this in six hours and thought I understood it. The ten-minute runs agreed with me. The thirty-minute one didn't, and it was right.

I built a coding agent in six hours. It argues with me the way I argue with myself.

ADITYA DUTT PANDEY — Tue, 13 Oct 2026 00:00:00 +0000

When I write code well, it's because I argued with myself first. One half of me writes it. Another half attacks it. A third decides whether it ships.

Every AI coding tool I tried gave me the first half. So I built mew, a harness that runs all three, and it took six hours.

It's now how I write code. Against Claude Code and Pi, it has been better for me. That's my experience from using it every day, on my own work, and I'll be careful about what it does and doesn't prove at the end.

Decision 1: split the writer from the critic and the judge

mew doesn't wait to be asked for feedback. While I work, other agents run alongside me and push back on my decisions. They aren't a list of if-this-then-that rules. They read what I'm doing and challenge it, which is what I'd be doing to myself in my head.

The critic and the judge are separate on purpose. When a run changes files, your checks run first, and a failure goes back to the model as a repair request. Then a second model, a different one, reads the real diff. My reasoning is that a model reviewing its own work tends to agree with itself, and a different one has different blind spots. I haven't measured that.

Here's a real run on tunyt's web app. The checks passed and the review passed. The final answer still opened with "Partly correct. I wouldn't ship it as is", and said it had read the code and never run it.

That's the part I care about. Typecheck tells you the types still line up and lint tells you the code follows the rules. Neither runs the change. Confirming this fix means liking an event, coming back hours later, and looking at the count and whether the like is still selected. mew said exactly that, in the output, without me asking.

Every run ends in one of five statuses: verified, unverified, verification_failed, failed or done_no_changes. None of them is a bare "done".

Decision 2: I set the budget before the work starts

I like to decide how many tokens a kind of task deserves, and I want the tool to respect that. So mew sorts each request by difficulty without spending a model call on it. Easy requests go to a fast chain of models, hard ones to a strong chain.

There's also a hard ceiling. At 70% of the budget mew warns me, and when the budget is gone it makes one last call with no tools, so the work done so far becomes an answer and not a bare failure.

I also use it to divide a task and rethink how I'd approach it before any code gets written. Having to say how much a task is worth turns out to change how I plan it.

Decision 3: run on the cheapest models and make the context do the work

Most of this runs on small, cheap models, DeepSeek V4 Flash and GPT-5.6 Luna among them. The aim is the best output the cheapest models can give. My bet is that a cheap model given a tight, clean context does well, and that a lot of what looks like a model's weakness is really a context problem. It's a bet. I haven't isolated it.

So most of mew is about the context. The surprising part is where the tokens go. A long run sends the whole conversation again on every call, so a tool result isn't paid for once, it's paid for on every call after it. In the run above, one file read cost 1,922 tokens and was re-sent 15 times, which is 28,830. Half of that run's input, 52%, was earlier tool results travelling along again. A file's real cost is its size times the number of calls that follow it.

So mew keeps a map of which lines of each file are already in context. Ask again and you get only what's new, or a diff if the file changed. It finds code with search and outlines before reading anything, and a read-only scout on a cheap model goes first, so the main model starts from findings. A cold index of a large repo takes 13.7 seconds and a warm one 1.2.

Caching works on the start of the conversation, so one early change re-bills everything after it. That run re-billed 25,381 tokens because the prefix changed. The Codex backend only caches when you send session_id and prompt_cache_key, so mew sends both and warns if a model isn't caching. I also cut the system prompt from about 3k tokens to about 0.7k. That run: 21 calls, 418.8k tokens in, 85% cached.

Decision 4: any provider, and say so when it switches

mew runs Claude, GPT and Codex, DeepSeek, and anything behind an OpenAI-style API. I pick which model fills which role: fast, strong, review, summarise.

The run in the screenshot was routed to the strong model, which hit a 429 rate limit. mew said so and finished the work on GPT-5.6 Luna. Nothing fails over silently.

Decision 5: the rules are code

"Please don't touch my SSH keys" is a request, and a model can forget a request. Editing a file the agent hasn't read this session is refused, and so is editing one that changed on disk after it was read. File tools can't leave the folder you started in. Shell commands are scanned for paths before they run, and on macOS an OS sandbox sits under all of it.

Every path:line in the final answer is checked against your real tree too. A file that doesn't exist goes back for one correction, and anything still unverified shows up as a warning. That's the sentence in an answer people believe fastest, so it's the one worth checking.

What I haven't proven

I built this for one person, and I'm that person. Six hours of work that suits how I think doesn't mean it suits everyone, and "better than Claude Code" is my daily experience, not a result. Nobody has run a head-to-head benchmark against other harnesses. There's a bench you can aim at your own tasks and models, and 224 offline tests.

It also has gaps. There's no plan mode, no image input and no structured JSON output for CI, and the kernel-level sandbox is macOS only.

Six hours was enough because I already knew what I wanted it to argue about.

We gave users a choice. That was the bug.

ADITYA DUTT PANDEY — Sat, 10 Oct 2026 00:00:00 +0000

The hardest problem we had on a recent tunyt event had no stack trace.

Nothing crashed. No alert fired. Every line of code did exactly what we wrote it to do, and people still kept writing in, stuck and confused, asking us questions we couldn't answer for them.

It was a team event. People registered as teams, and the team had to pay. That sounds like a checkbox. It turned out to be four separate decisions, and the interesting part is how a series of reasonable choices added up to one that wasn't.

Decision 1: the leader pays

Our first plan was the simplest thing that could work: the team leader pays for everyone. One team, one payment, one receipt.

In a payment system every extra state is a place for money to get lost, and this design had almost none. A team was either paid or it wasn't. I still think that was right.

Decision 2: let teams split it too

After a lot of back and forth inside our team, we added a second option. A team could split the payment, so every member pays for their own ticket. It felt friendlier. Nobody has to put up money for four people and chase the others afterwards.

So we shipped both, and let the team leader pick.

That last sentence is the bug. I didn't see it yet.

What people actually did

Two kinds of chaos showed up, and on the surface they had nothing to do with each other.

The first was a team that changed shape. A team of three formed and the leader paid for all three. Then they removed one member and added two new ones, and wrote to us asking why they were suddenly being asked to pay for two more people. From their side they'd paid for "the team", and the team had changed. From our side, three tickets had been paid for, one belonged to someone no longer on the team, and two new people had none. The person they'd removed still held a valid ticket of their own, and nobody on the team knew.

The second was a payment button that never came. Members started writing in to say they had no button, no link and no email. Support asked the same question every time: has your team leader chosen to split the payment, or to pay for the whole team? Almost every time, the answer was that nobody knew.

Decision 3: say no

The first problem pulled us toward building more. If teams change after paying, the proper answer is refunds, ticket transfers, moving a ticket from one person to another, and handling every team that changes its plan halfway through.

I sat with what that would mean. Each of those is its own small product. Each moves money or ownership, and each has its own edge cases, its own support load and its own new ways to get a payment wrong. We'd be spending weeks to support behaviour that a few teams had stumbled into.

So we built none of it. We kept the model we had and stopped trying to make every possible team change free. It's the decision I'm surest about, and the hardest to make with someone upset in your inbox. "We won't support that" sounds like failure. Most of the time it's just scope, and a small team that says yes to every edge case ends up with a product made of edge cases.

Decision 4: move the choice

Saying no didn't fix the second problem, and the second problem was the real one.

When I drew the flow out as states, it became obvious. With the leader choosing at runtime, a team could be in any combination of: no choice made yet, leader pays, or split; paid, partly paid or unpaid; the same members as when it paid, or different ones. We understood every one of those states. The people inside them couldn't see any of them. Every support message was someone stuck in a state the product never showed them.

So we moved the choice. Now the organiser decides, once per event, how teams pay: either the leader pays for everyone, or it's split and each person pays as they join. Attendees never make the choice, so they can't be confused about what was chosen, and "no choice made yet" stops existing as a state at all.

We had almost all the logic already. In the end it was mostly a flag on the event:

// Simplified. Set by the organiser when they create the event.
type TeamPayment = "LEADER_PAYS" | "SPLIT";

One more rule closes the gap that started all this. If a team has already paid and someone new joins, that person buys their own ticket. They join as a new member with a ticket of their own and never reopen a payment that's already settled. Money that's settled stays settled.

What I took from it

A choice you give users is a state you have to show them. We thought we were adding flexibility, and we were adding a hidden state that only one person on the team could see. How a team pays is the organiser's call, and the moment we let whoever clicked first decide, everyone else inherited a choice they never made.

Support asking the same question over and over was the product asking it in the wrong place. "Did your leader split or pay?" belonged on the event page, and it never needed to be a support chat.

Decision 1 was right. Decision 2 was reasonable. Putting decision 2 in the wrong hands was the bug.

The bug that scared me most never reached a single user

ADITYA DUTT PANDEY — Sat, 10 Oct 2026 00:00:00 +0000

The bug that scared me most never reached a single user.

It was the very beginning of tunyt. We were testing internally, someone opened an event dashboard, and it showed another host's data.

We caught it ourselves, and no user ever saw it. But for a few seconds that didn't matter, because the only question in my head was how many other places this was happening and for how long.

Decision 1: cache at the edge

Not long before, we'd been working on caching, and part of that work was in nginx, in front of the app.

The reasoning was sound. A public event page looks exactly the same for everyone. If nginx can answer it from memory, the request never touches the app or the database at all. It's about the cheapest response you can serve, and during a ticket sale, when everyone is refreshing the same event page, it's exactly the win you want.

So I wrote a rule to cache the event pages. Remember that.

Decision 2: go to the code

When private data shows up in the wrong place, every instinct says the application is leaking it. So that's where I went, and I went all the way down: how the dashboard loads its data, how every query is scoped to the logged-in host, how sessions work. All of it was correct.

So I went back to the browser and watched the network calls, again and again. One thing kept bothering me. The dashboard response was fast. Suspiciously fast, for a page that should have been doing real work for one specific host.

I put debuggers in and ran it locally, and the right host saw the right data every single time. Of course it did. There was no nginx on my machine.

The click

I was doing the same checks for what felt like the tenth time when it clicked: I had done caching on nginx too.

I opened the config and there it was. The regex I'd written to match public event pages also matched the event dashboard pages. Both were being cached as public. Something like this, simplified and not our real config:

# Written to cache public event pages.
# Also matched /events/<id>/dashboard, which is different for every host.
location ~* ^/events/ {
    proxy_cache events;
    proxy_cache_valid 200 5m;
    add_header Cache-Control "public, max-age=300";
    proxy_pass http://app;
}

Why it leaks

The mechanism catches a lot of people. By default, nginx decides that two requests are the same if they have the same scheme, host and URL. That's the cache key. Your cookies aren't in it, and neither is who you're logged in as.

So two different hosts opening the same dashboard URL look identical to the cache. Whoever loads it first fills it, and the next person gets their copy. The app never even sees the second request, which is exactly why it was so fast.

Cache-Control: public makes it worse. It tells every cache along the way, not just nginx, that the response is safe to store and share with anyone.

Decision 3: the fix, and the rule behind it

The immediate fix was to narrow the rule so it matched only the public pages it was written for.

The rule I took away is bigger than the regex. Anything behind a login never goes into a shared cache. There's no "be careful with it". It's never. A personalised response should say private or no-store, and if a cache rule could ever match it, the rule is wrong.

I made it absolute because the failure is lopsided. Get a public page's caching wrong and it's a little slower. Get a private page's caching wrong and you've handed one customer's data to another. Rules with that shape of risk don't get exceptions.

What it left me with

A response that's too fast is a clue. I noticed it early and didn't trust it. Real work takes time, so when a personalised page comes back instantly, something in front of your app answered for it.

When local and production disagree, suspect the difference first. The bug lived in a layer that didn't exist on my laptop, and I could have debugged the application for a week and found nothing, because nothing was wrong with it.

A caching rule is also a security rule. It decides who gets to see a response, and when the rule is a regex, one pattern decides whether a private page is public. That deserves the same review as an auth check. I'd stopped thinking of nginx as part of the system because I wasn't writing it every day, and config is code that just doesn't come with tests.

If I could add one thing to every proxy config I've ever written, it would be a header saying whether the cache answered. One X-Cache-Status: HIT in the network tab would have pointed straight at nginx in seconds.

The regex did exactly what I wrote. I wrote it for the pages I was thinking about, and I never thought about every page that happened to match.

The scarcest thing we own is a database connection

ADITYA DUTT PANDEY — Sat, 10 Oct 2026 00:00:00 +0000

When people talk about scaling a backend, they talk about CPU, memory and instances.

On tunyt, the resource that has hurt us most is much smaller: database connections. Once I started looking at the whole system through that one lens, a lot of arguments stopped being opinions and turned into arithmetic.

Why connections are scarce

Postgres gives every connection its own server process. That's great for isolation, but it makes connections heavy. Each one costs memory on the database server, and past a certain number Postgres spends more effort managing connections than doing your work.

So our pool is small on purpose, and every instance of the API and every worker shares one fixed budget. When the budget is used up it doesn't matter how fast your code is, because requests wait in line.

The one equation

connections in use ≈ requests per second × seconds each request holds a connection

That's Little's law. You can lower the rate, or you can lower how long each request holds a connection. Every decision below is really about the second one.

Decision 1: never hold a connection while waiting on someone else

A transaction holds a connection from the moment it opens until it commits. Call an outside service inside it and you hold that connection for the whole round trip. For us, outside services means payments, SMS, email and LLM providers.

Run the equation. If a payment call normally takes 300 ms, that's already a long hold. On a day the provider takes three seconds, every request doing this holds a connection ten times longer, so you need ten times the connections for the same traffic. The pool fills, and requests that have nothing to do with payments start queueing too. At that point your uptime is partly decided by someone else's latency, and I'm not willing to give that away.

// Don't: the connection is held for the provider's whole latency.
await db.$transaction(async (tx) => {
  const order = await tx.order.create({ data });
  await payments.capture(order.id); // slow, and outside our control
  await tx.order.update({ where: { id: order.id }, data: { paid: true } });
});

// Do: write intent, call out, then record the result.
const order = await db.order.create({ data: { ...data, status: "PENDING" } });
const result = await payments.capture(order.id);
await db.order.update({ where: { id: order.id }, data: { status: result.ok ? "PAID" : "FAILED" } });

This has a real cost. The second version introduces a new state, PENDING, and now you have to handle orders that get stuck in it. That's more code and more thinking. I'll take that trade every time, because a stuck order is a problem for one person and a full pool is a problem for everyone.

Decision 2: every list query has a ceiling

An unbounded findMany was marked P0 in our audit, the highest severity we have. It wasn't slow on the day we found it, and that's exactly why it was P0.

The danger of an unbounded query isn't today. It's the day the table grows, when it pulls every row into memory in one go and holds a connection the whole time. Tables grow fastest on your busiest days, so it goes off at the worst possible moment.

Now every list endpoint paginates, and every internal fetch has an explicit take. The number depends on the query. There is always a number.

Decision 3: ask once, not N times

Our payment flow needs several tickets at once. Looking them up one by one in a loop would be N round trips and N separate turns on the pool, for something the database can answer in one. So it asks once.

// One query for every id, then a Map to put them back in order.
const tickets = await db.ticket.findMany({ where: { id: { in: ids } } });
const byId = new Map(tickets.map((t) => [t.id, t]));
const ordered = ids.map((id) => byId.get(id));

The Map matters. IN doesn't promise to return rows in the order you asked for, and code that quietly assumes it does will pass every test and lie to you in production.

Decision 4: an index ships with its query

Any new filter has to be covered by an index before it ships. If the index is missing, it goes in the same pull request, not "later". A query without an index holds its connection for as long as the scan takes, which puts you straight back into the equation.

On high-write tables we build it with CREATE INDEX CONCURRENTLY. It's slower and can't run inside a transaction, but it doesn't block writes while it builds.

Why write any of this down

None of these rules is clever, and most backend engineers have heard all of them. They're written down on tunyt because a team of four, building a mobile app, a web app and the AI infrastructure at the same time, doesn't get to remember everything.

The fix was never more hardware. It was to stop wasting the one thing we can't simply buy more of.

I spent a whole night looking for money that was never missing

ADITYA DUTT PANDEY — Sat, 10 Oct 2026 00:00:00 +0000

I once spent an entire night looking for money that was never missing.

It was tunyt's second event. By morning I had accounted for every rupee, and the problem I'd chased all night turned out to be hiding inside a fix I'd already made.

Decision 1: let the webhook drive everything

When someone pays, Razorpay sends our server a webhook, an HTTP request saying the payment succeeded. We made that the trigger for everything after a payment:

create the ticket,
send the QR code,
increase the number of tickets sold,
add the amount to the organiser's dashboard.

I'd defend most of that today. A browser can't be trusted to say it paid, because anyone can fake a success screen. The webhook comes from Razorpay's servers to ours, which makes it the one confirmation you can believe.

The weakness was that it was the only path. One event in, four effects out, and nothing downstream ever checked.

The night it broke

Three people bought tickets and never got them. The money left their accounts. On our side there was no ticket, no QR code and no email. I traced it back and found that Razorpay had disabled our webhook. Payments were arriving and our system never heard about them.

Decision 2: people first, then the books

I had two jobs, and they weren't equal. One was to understand what had happened. The other was that three people had paid for an event and had nothing to show at the door. That one had a deadline. The investigation didn't.

So I turned the webhook back on and emailed all three their QR tickets myself, by hand. They were getting in, whatever else was true.

I'd make that call again. It's also the call that set up the rest of my night.

Then the money didn't add up

While making sure I'd covered everyone, I went through the payments, and Razorpay showed more money than our dashboard did. It looked like three payments had arrived with no tickets behind them at all.

Three people with no tickets I understood. Three payments with no people was worse. Where had that money come from, and where was it on our side?

Decision 3: go through it line by line

I knew very little about reconciliation then. I didn't know how much data I could pull straight from Razorpay, or that I could store it and compare it with our own records using a short script. I wasn't good with Excel either.

So I did the only thing I knew how to do. I exported everything to CSV and went through it by hand. This is the payment ID. This is the order string. Does it exist on our side? Next line. It's slow in the way that makes you start doubting your own spreadsheet.

The twist

By morning I had the answer, and it was almost funny. There were no extra payments. They were the same three people.

When the webhook stopped firing, none of the four steps ran. I'd fixed step 2 by hand, because step 2 is the one people can see. Steps 3 and 4 had never happened. The ticket count didn't go up and the amount never reached the dashboard. The money was all there. Our records just hadn't caught up, and my manual fix had made the customers whole while leaving the books exactly as wrong as before.

What a webhook actually promises

A webhook is a notification, not a record. Providers try to deliver them and retry when they fail, but the guarantees are looser than they look. One can arrive late, or twice, or not at all, and it can be switched off on the provider's side without anything in your app noticing.

A design where one notification triggers four effects, and nothing ever double-checks, doesn't fail loudly. It drifts. Your records slowly stop matching reality, and nobody finds out until the numbers stop adding up.

Decision 4: trust the provider, check ourselves

So we stopped relying on the webhook alone. We pull payment data from Razorpay, store it in our own database, and compare the two with a script. If money arrived and our side doesn't fully reflect it, the script finds it, and neither an organiser squinting at a dashboard nor an attendee at the door has to. The check now runs automatically, with AI in the loop to flag anything that doesn't match.

If I were designing it today I'd go one step further and make every post-payment step safe to run twice, keyed by the payment ID. A missed webhook would then mean "run it again", and I wouldn't be fixing it by hand at night.

What I keep from that night is that a manual fix only repairs what you can see. I had completed one step of a four-step transaction, and the other three don't complain when they're skipped.

The money was there the whole time. What was missing was our record of it, and I'd only repaired the part that hurt.

I trained an AI to lie to another AI

ADITYA DUTT PANDEY — Sat, 10 Oct 2026 00:00:00 +0000

Most security AI assumes something it shouldn't: that its inputs are telling the truth.

A defender model reads the logs, sees an intrusion and responds. But a real attacker who has made it inside isn't polite. One of the things they can do is go after the logs themselves. If the defender's eyes are the logs, the cheapest attack is to lie to its eyes.

So I built PHANTOM for the Meta × PyTorch × Hugging Face OpenEnv Hackathon India 2026. An AI defender works a live breach. At the same time, an adversary AI feeds it fabricated SIEM entries in real time, gaslighting it about what's happening.

Decision 1: attack the logs, not the network

I could have made the adversary attack harder, with more intrusions and faster moves. That trains a defender to be quick.

Attacking through the logs trains something more valuable: a defender that doesn't believe everything it reads. It has to cross-check, notice contradictions and act on evidence that holds up, instead of whatever the latest log line claims. That's the skill that matters once someone is already inside.

Decision 2: both sides learn

A fixed adversary gets memorised. The defender learns its five tricks and looks brilliant, right up until it meets a sixth.

So both agents learn against each other. As the defender gets better at spotting fake logs, the adversary gets better at writing convincing ones. It's multi-agent reinforcement learning, and the pressure goes both ways, which is closer to how real security works. Every defence teaches attackers what to try next.

Decision 3: GRPO

For training I used GRPO, group relative policy optimisation. Instead of training a separate model to estimate how good each situation is, you let the policy try the same situation several times, as a group, and compare those attempts against each other. Better than the group's average gets reinforced, and worse gets discouraged.

It's simpler than the classic setup, and a good fit when the only thing you can really judge is whether one response beat another.

Decision 4: start easy

A defender thrown straight at a skilled liar learns nothing. It just fails, over and over, with nothing to learn from.

So there's an adaptive curriculum. Early on the deception is clumsy and the breach is simple, and as the defender improves the environment gets harder. The aim is to keep it at the edge of what it can handle, because that's where learning actually happens.

Decision 5: an LLM as the judge, carefully

How do you score incident response? "Did it block the right IP" is easy. "Did it reason sensibly about conflicting evidence" isn't something you can write a simple rule for.

So responses are scored by a language model acting as a judge. It can weigh reasoning in a way rules can't, and it brings its own biases. A judge you can't fully check is a risk, and I'd keep thinking about that before trusting the scores for anything that matters.

What I took from it

The interesting failure in AI security isn't a model that can't see the attack. It's a model that sees exactly what it's shown and believes it.

A defender that trusts its inputs isn't a defender. It's an audience.

The bug where nothing was wrong

ADITYA DUTT PANDEY — Sat, 10 Oct 2026 00:00:00 +0000

Some bugs throw errors. The worst ones don't.

There's no exception, no alert, no red graph and no failing test. Every service is healthy. An organiser changes their ticket types, and people keep seeing the old ones for up to ten minutes.

Nothing is wrong. Everything is wrong. Welcome to cache invalidation.

Decision 1: cache the hottest read

Every event page on tunyt needs that event's ticket types. That makes it one of our hottest reads, read far more often than it changes, which is the textbook case for a cache. So we put one in front of it. The pattern is called cache-aside, and every backend has some version of it:

async function getTicketTypes(eventId: string) {
  const key = `cache:ticket:type:event:${eventId}`;
  const hit = await cacheGet(key);
  if (hit) return hit;

  const rows = await db.ticketType.findMany({ where: { eventId }, take: 200 });
  await cacheSet(key, rows, { ttlSeconds: 600 });
  return rows;
}

Read from the cache. On a miss, read from the database and write the result back.

Decision 2: ten minutes

The TTL is a trade. Longer means less load on the database and more risk of serving something stale, and shorter means fresher data and more load. Ticket types don't change often, so ten minutes saves a lot of database work.

The TTL is also our upper bound on how long any staleness bug can live, which matters more than it sounds. Hold that thought. Those ten lines are fine, and they're never where the bug is.

The bug is the writer you add later

The first way this breaks is a person, and it shows up months later.

Somebody adds a new way to change ticket types. A bulk edit, an admin tool, a one-off script. It updates the database correctly and passes its tests. It never touches the cache, because the person writing it had no reason to know the cache existed.

So the database says one thing and the cache says another, and the cache wins, for up to ten minutes. If what changed was a price or a sold-out flag, that's a support ticket at best. This class of bug is on our own performance audit list, which tells you how normal it is.

The race nobody notices

The second way it breaks is subtler, and it lives inside the ten lines. Two requests, almost the same moment:

A reader misses the cache and reads the ticket types from the database.
A writer updates the ticket types and deletes the cache key.
The reader, holding data from before the update, writes it back into the cache.

The cache now holds stale data, and nothing will remove it until the TTL runs out. Every step was correct. The order was the bug.

You can engineer around this with versioned keys or a short delayed second delete. But it's the honest reason the TTL exists. The TTL isn't there to make data fresh. It's there to put a ceiling on every bug you haven't found yet.

Decision 3: don't shorten the TTL

The tempting fix for both bugs is a shorter TTL. Make it a minute and the damage is smaller.

We didn't. The cache exists to take load off the database, and every time you shorten the TTL you hand some of that load back and still serve stale data, just for less time. You'd be paying permanently to shrink a bug instead of removing it. The TTL is the safety net, and it shouldn't be the plan.

Decision 4: every writer invalidates, through one helper

There's one helper that deletes the ticket-type cache for an event, and every code path that changes ticket types calls it. It's a written rule, and it's on the checklist anyone, person or coding agent, runs through before shipping a cache: invalidation wired at every mutation site.

A helper beats deleting the key inline because a helper is searchable. You can find every caller in seconds, and a code path that changes ticket types without calling it stands out in review.

Decision 5: one shape for every key

Every key looks like cache:{type}:{identifier}. That sounds cosmetic, and it isn't. Search the codebase for a type and you find every place that reads or writes it. If you can't find all the readers, you can't be sure you've invalidated them.

How I review caches now

A cache is a second copy of the truth. Every second copy needs someone keeping it honest, and in code that someone is every function that writes the first copy.

So when a new cache shows up in a review, I don't start with the read. The read is ten lines and it's always fine. I ask to see every write.

I trained a language model that writes by filling in blanks

ADITYA DUTT PANDEY — Sat, 10 Oct 2026 00:00:00 +0000

Every language model you've used writes the same way. Left to right, one token at a time, each word chosen after the one before it.

I wanted to know what happens if you don't do that. So I trained a small model that starts with a sentence that's entirely blanks, and fills them in.

How it writes

The model is a BERT-style transformer, which means it looks at the whole sequence in both directions at once, not just at what came before.

Generation works like this:

Start with a sequence where every position is [MASK].
The model predicts a token for every masked position at the same time.
Keep the predictions it's most confident about. Put the mask back on the rest.
Repeat, until nothing is masked.

[MASK] [MASK] [MASK] [MASK] [MASK] [MASK]
[MASK] little [MASK] [MASK] [MASK] park .
The little [MASK] went to the park .
The little dog went to the park .

It's diffusion, the same idea behind image generators, applied to words. Image models start from noise and remove it step by step. This one starts from blanks and fills them in step by step.

Decision 1: diffusion instead of left to right

The appeal is that nothing forces the model to commit to the first word before it knows the last. It can settle the shape of a sentence early, the subject here and the full stop there, and fill in the middle once the edges make sense. And it fills many positions per step instead of one, which is where the speed promise of diffusion language models comes from.

Why it's awkward

This is where it got interesting, and honest.

Image diffusion works with gentle noise. A pixel can be slightly wrong and get nudged towards right. Words don't work like that. A token is either "dog" or it isn't, and there's no such thing as slightly-dog. So "noise" for text becomes a blunt on or off mask, and a lot of the smoothness that makes diffusion work for images just isn't there.

Other awkward parts showed up quickly. You have to decide the length of the output before you start, because you're filling a fixed row of blanks. And deciding which predictions to keep at each step matters enormously. Commit too early and the model locks in a mistake it can never undo. Commit too late and it barely makes progress.

Decision 2: train it on a laptop

I trained it on TinyStories, a dataset of simple short stories, locally on Apple Silicon. No cloud cluster.

That was deliberate. A small model on a small dataset is something you can actually finish, and finishing is where you learn. You get to watch it train, break it, change one thing and see what happens, all in an afternoon instead of a week of waiting on rented GPUs. It's the same instinct as every other small version of a big thing I've built: shrink it until you can hold all of it, then look closely.

What I took from it

Left-to-right generation isn't a law of nature. It's one choice, and seeing the alternative up close makes it much clearer what that choice buys you and what it costs.

The model writes worse stories than the ones you're used to. It taught me more than any of them.

Why I ran a ticketing backend on two Raspberry Pis

ADITYA DUTT PANDEY — Sat, 10 Oct 2026 00:00:00 +0000

It sounds like a joke: Tixin's backend ran on two Raspberry Pis.

It also held about 16,000 requests a second in a load test. This is how, and more importantly why I made each choice.

Decision 1: hardware that's too small on purpose

Tixin was the ticketing pilot that later grew into tunyt. I built its Kubernetes cluster by hand, on two Raspberry Pi 5s.

On a big cloud instance, a slow query costs you nothing you can see. You have headroom, so you shrug, and the slow query becomes a permanent resident. On two small boards there's no headroom to hide anything in. Every slow query, every chatty service and every lazy default shows up immediately, and the only way past it is to fix it. You can't buy your way out.

I wanted that pressure, and I'd do it again.

The problem ticketing actually has

Ticketing traffic isn't a curve. It's a cliff.

For most of the day almost nothing happens. Then sales open and a lot of people try to do things at the same moment. Most of them are reading: looking at the event, checking prices, refreshing to see what's left. A smaller number are writing, which means actually booking. If reads and writes share one path and one database, they slow each other down at exactly the moment it matters.

Decision 2: split reads from writes

So Tixin split them. The pattern is called CQRS, command query responsibility segregation, which is a long name for a simple idea: the code that changes things and the code that reads things don't have to share a path.

Commands, like "book this ticket", go onto Kafka. Queries, like "show me this event", never touch that path and are served from a Redis cache. A person refreshing an event page and a person booking a seat are now doing completely different work on completely different infrastructure.

Decision 3: Kafka on the write path

Kafka is an append-only log, and appending is about the cheapest thing a computer can do. A burst of bookings becomes a burst of appends. The log absorbs the spike, and the work behind it catches up at its own pace instead of falling over at the peak. At peak, the pipeline sustained more than 50K events a second.

What CQRS costs you

CQRS gets sold as free. It isn't.

When reads come from a cache and writes go somewhere else, there's a moment where the two disagree. A booking has happened and the read side hasn't caught up yet. Usually the window is tiny, but in ticketing it forces real questions. What does someone see in that window? Where do you make sure a seat can't be sold twice? On which side does the truth live?

The pattern doesn't answer those questions. It makes you answer them on purpose instead of by accident, which is most of what good architecture does.

Decision 4: tune the boring parts

Splitting the paths was most of the win. The rest was unglamorous. After tuning PostgreSQL and MongoDB on the booking read path, query latency dropped by 70%. Rebuilding the CI/CD automation cut deploy time by 60%.

None of that is clever, and all of it was forced by the hardware. On a big instance I'd have shrugged. On a Pi it was the difference between holding and falling over.

Decision 5: refuse to quote the number

This is the decision people find strangest. I don't quote 16K as Tixin's capacity.

A load test isn't production. I chose the workload, so the result tells you where the limits are under conditions I picked. It says nothing about real people on flaky phones, retries, double-taps on the pay button and everything else real users do.

So 16K is evidence of one specific thing: the architecture wasn't the bottleneck. That's useful to know. It isn't a promise, and the fastest way to lose a senior engineer's trust is to dress one up as the other.

What I took into tunyt

Tixin is retired now, but two things from it run through everything I build.

One is the habit of measuring first and fixing what the measurements show, and not what you suspect. The other is the pattern itself. I later pulled the read and write split out of Tixin into its own repository, cqrs, so it could be read without a product wrapped around it.

The Pis didn't make Tixin fast. They made it impossible to pretend it was.

Our traffic grew seven times and we got faster. I don't fully trust that.

ADITYA DUTT PANDEY — Sat, 10 Oct 2026 00:00:00 +0000

Here are tunyt's numbers from one stretch of this year:

	Before	After
Weekly requests	69K	485K
Latency	110 ms	59 ms
Apdex	0.55	0.89
Error rate	2.84%	0.32%
Deploy time	20+ min	under 9 min

The error rate moved between June and August 2026, while traffic was still climbing.

Traffic went up about seven times and every other number got better. That's the kind of table that makes a great LinkedIn post, and it's exactly the kind of table I've learned to be suspicious of, my own included. So this post is about how we got there, and what I'm not willing to claim.

Decision 1: measure before touching anything

Before we changed a line, we made the server tell us how it was doing. A metrics endpoint reports, for every route, request duration at the 50th, 95th and 99th percentile. It also reports how many jobs are waiting in each queue and the state of every circuit breaker.

You usually already suspect what's slow, and suspicion is how you spend a week optimising the wrong thing. The measurements decide what we fix. What I think is slow doesn't get a vote.

Decision 2: per route, and in percentiles

An average across the whole API is a comfortable lie. Ninety fast endpoints will hide the one that's slow for every single person who touches it, and that one is usually the endpoint people came for. So every route gets its own numbers.

Within a route, we watch percentiles. The 50th is your typical user. The 95th is your unlucky one, the person on a slow phone and bad data who decides your app is broken and doesn't come back. The 99th is where the strangest problems hide. If you only watch the average, you're optimising for someone who doesn't exist.

Decision 3: one number for "does it feel fast"

Percentiles are precise but hard to watch day to day. So alongside them we track Apdex, a single score from 0 to 1.

You pick a target time. Requests faster than it count as satisfied. Requests up to four times slower count as tolerating. Anything slower is frustrated.

Apdex = (satisfied + tolerating / 2) / total

Going from 0.55 to 0.89 means the bulk of requests moved into satisfied. It's a blunt number, but it's the closest single number I know to "does this feel fast?"

What changed in that window

I'll list what we did, and I won't pretend I can tie each number to one change.

Every list query got a ceiling. Pagination on endpoints, an explicit take everywhere else.
Loops of lookups became one query. N round trips turned into one IN query and a Map.
Hot reads got a cache with real invalidation. Every write path deletes what it changes.
Expensive reads got single-flight. One request rebuilds a missing value while the rest serve the last good copy, so a cache expiry can't stampede the database.
Analytics moved to rollups. Finished days come from a daily table, and only today is computed live.
The database got a budget. One connection limit for the whole deployment, split between the API and the workers, with capped fan-out.
Hot pages start on the server. Event pages fetch their data in parallel on the server and hand it straight to the client's cache, so the browser doesn't wait for a second round of requests.

Decision 4: treat deploy time as reliability

Deploys went from more than twenty minutes to under nine. Docker layer caching meant unchanged dependencies stopped being rebuilt. Lint and tests became parallel jobs instead of one long line. Model weights got baked into the base image instead of downloaded on every build.

Deploy time quietly controls how you ship. When a fix takes twenty minutes to reach production, people batch changes together, and big batches are harder to review, harder to roll back and more likely to break something. Nine minutes makes small changes cheap again. Deploy time is a reliability metric pretending to be a convenience.

Decision 5: refuse to take full credit

Some of the improvement came from the changes above. Some almost certainly came from things I didn't control: traffic patterns shifting, caches staying warmer at higher volume, effects I never measured.

If I told you the rollup table saved exactly 30 ms, I'd be making it up. I'd rather lose a little credit than gain a little dishonesty, because the moment someone catches you overclaiming one number, they stop believing the rest.

What I can say honestly is that the system got faster and more reliable while doing about seven times the work. Every change on that list is now a written rule the team and our agents check before shipping, so the numbers don't quietly slide back.

The table is real. The story that every number came from a fix I made is the part I won't tell you.

I keep building worse versions of software that already exists

ADITYA DUTT PANDEY — Sat, 10 Oct 2026 00:00:00 +0000

I've spent weeks building worse versions of software that already exists. On purpose.

A Redis-compatible server in Go. A load balancer in Bun. A four-service gRPC mesh. An SFU on mediasoup. A small operating system in C.

None of them will ever replace the original. If you're running production on my Redis, please stop. But each one changed how I make decisions as a CTO, and that's worth explaining.

Decision 1: rebuild what holds my system up

I don't rebuild things at random. The rule is simple: if a tool is load-bearing for what I run, I want to understand it from the inside.

Every convenience is rent. What I can't rebuild owns me.

That's not a rule against using tools. I run Kafka, Redis and Kubernetes in production and wrote none of them. It's a rule about knowing them well enough to answer three questions: what does it cost, where does it break, and what will I do when it does?

Redis taught me about TCP

Redis speaks a protocol called RESP, and it's beautifully simple. A line starting with + is a string, - is an error, : is an integer, $ is a blob of bytes with its length up front, and * is an array.

*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n

That's SET key value. My version speaks it well enough that redis-cli connects and doesn't notice the difference. I wrote the parser, the storage and the command handling by hand.

The first real lesson wasn't about Redis at all. TCP gives you a stream of bytes, not messages. One command can arrive in two pieces, or three commands can arrive glued together in one read. Your parser has to handle both, or it works perfectly on your laptop and breaks under load. A whole class of "works locally, fails in production" bugs made sense to me after that.

The second lesson: the real Redis runs commands one at a time, on a single thread. Write the command loop yourself and that stops looking like a limitation. No locks around the data, no races between commands, very predictable latency.

The load balancer taught me what "healthy" means

Configuring a load balancer, you tick "health check" and move on. Building one, you have to decide what healthy means. Is a server that answers /health instantly but takes four seconds on real requests healthy? How many failures before you stop sending it traffic? How do you bring it back without stampeding it?

Mine does auto-scaling for Bun backends, active health checks, routing and live monitoring. I learned more about connection handling from it than from a year of configuring someone else's.

The SFU taught me why video calls need a middleman

In a naive group video call, everyone sends their video to everyone else. That's a mesh. With five people, each one uploads four copies of their own video. Home internet is usually much worse at uploading than downloading, so the mesh falls over long before you'd expect.

An SFU, a selective forwarding unit, fixes it. Everyone uploads once, to the server, and the server forwards streams to everyone else. You only really get why every serious video product works this way after you've built the mesh and watched it struggle.

Decision 2: stop at small

I never try to finish these. A small version is enough to teach the shape of the thing, and the last 10% of a database is years of other people's work I have no business redoing.

That's a real limitation. A small version hides the hard parts: replication, persistence, failure handling at scale. When I say I understand Redis, I mean I understand why it's shaped this way. I don't mean I could run it better than the people who maintain it.

Decision 3: let it change real choices

This is where it pays off.

When we choose infrastructure for tunyt, I'm not choosing from a feature table. I'm choosing from a picture of how the thing works inside. And when someone proposes adding a new moving part, my first question is what it costs per instance and where it breaks, because I've built enough small versions to know every moving part has an answer to both.

The other projects in the set:

go-grpc-micro. Account, catalog and order services over gRPC behind a GraphQL gateway, on PostgreSQL and Elasticsearch. Small enough to hold in your head, real enough to break.
cqrs. The read and write split Tixin ran on, extracted into its own repo. I could only pull it out cleanly because I'd built the pieces underneath.
OsU. A small operating system written from scratch in C.

I don't build these to own better software. I build them so the software I depend on stops being a black box.