DEV Community: Ketu Shah

You can't hire judgment later

Ketu Shah — Sat, 14 Nov 2026 21:51:15 +0000

Originally published at https://ketushah.com/writing/you-cant-hire-judgment-later/

You can't hire judgment later

The productivity gain is real. I take it every day. Code that used to take an afternoon takes an hour, and the hour is not a trick. It is real work done faster.

But there is a debt building underneath the gain, and I am building it too. Most of what used to make a junior into a senior is being automated away. And judgment is built from repetition.

The repeated work is small and unglamorous, which is why nobody mourns it. Debugging and figuring things out. Stumbling onto something and making mistakes and learning from it. Reading and observing. Typing and sometimes writing. Taking notes. Questioning and searching why something is a certain way. None of these look like training. They look like the slow parts of the job, the parts you would automate first. That is exactly the problem. The slow parts were the training. Debugging was never only fixing. It was how you learned the system, one wrong turn at a time. A mistake taught you where the edges were. Reading code taught you how someone else thought. Notes were thinking with your hands. Take the doing away and you take the learning away with it, quietly, without anyone deciding to.

That is how I learned. By looking and learning. By saying I don’t know how to do it, and someone explaining it to me, or learning from Stack Overflow or docs or tutorials or any random blog. Coming across dozens of results before finding the appropriate one. Sometimes the explanation came from a person. Sometimes it came from a page. Either way, the learning started with admitting I did not know.

The dozens were not wasted time. Choosing among them was the training. You learned what appropriate looked like by seeing twenty things that were not. The model hands you one answer now, and the choosing goes with the dozens. The practice was never only finding the answer. It was learning to recognize it. Appropriateness is a judgment, and you cannot borrow it from a single answer, because a single answer never shows you the near-misses.

Meetings are recorded and transcribed now, and that gives people a reason to be less present. You can always read the transcript later, so nobody listens now. Attention is outsourced the same way the typing is. That is a second kind of practice disappearing. Paying attention used to be part of the job. Now the machine pays attention for you, and you get the summary, and the summary is not the same as having been there. Being present is bare minimum. It is just respectful. And it does not waste time in follow-ups that could have been handled if one was present. That is one small item in a long list of other things.

I do purely AI-assisted development. I am not describing someone else’s problem. I see the bite already. It is not a prediction. It is recurring. In discussions about newly built items, the answers about what is happening under the hood get less and less nuanced. During alerts, the most prominent statements are "Claude thinks xyz." The model’s guess stands in for someone’s understanding, and the room moves on it. Architecture reviews are sometimes going away, because shipping is so cheap in time that review starts to feel like the expensive part. The review that used to catch the misunderstanding now costs more than the shipment it would have reviewed. System design has started to take a step back. Nuanced discussions and decisions are rare. And it is less common now that when an error occurs, a developer knows from the message where something might be wrong. The error message used to be a map. Now it is noise. People see less and less of the bigger picture, and of the contracts between systems. Nobody holds the whole thing in their head anymore, because nobody had to build it piece by piece.

What goes missing is resiliency, and understanding of consequences on decision making. Every action has a reaction. In a system, if you don’t foresee it, it is a recipe for failure in one place, or everywhere, or both. A contract between systems exists to say what touches what before anything moves.

Resiliency is not something you add later; it is what you get from foreseeing where a change lands before you make it. And when nobody holds the whole in view, nobody can say what a decision will touch, so decisions get made blind to what follows, and what follows arrives anyway. The reaction does not wait to be foreseen. It just arrives, and the team meets a failure it never decided to risk.

This is not only my view from my rooms. A September 2026 survey of 300 senior engineering leaders found that 35% of AI-generated code reaches production before engineers have fully understood it. And 93% said an AI hallucination had misdiagnosed a root cause at least once. It comes from a debugging company, and it polled leaders at large companies with complex codebases, so read it as the hard end of the industry, not the whole of it. Even at that end, the numbers say what I see: engineers in it spend an average of 16.9 hours a week debugging AI-generated code, about twice as long as they spend writing it. The code ships. The understanding does not always ship with it.

This is software engineering as I see it. Other trades have their own version of the same trade. Wherever the machine does the doing, the practice goes, and wherever the practice goes, the judgment goes later.

I use LLMs every day. What I want to keep is the craft underneath the speed: the debugging, the figuring out, the stumbling onto something and learning from it. Those were how judgment got built. The choosing among dozens was the training. Use the model for the doing. But the judgment is not part of the doing to hand over with it.

The practice is disappearing now. The shortage arrives later. You cannot hire judgment later if the practice that builds it is gone today. Nobody decided to stop training people. The doing just moved to the model, and the learning moved with it, quietly. The tools keep getting better either way. The question is only whether the craft and the judgment stay with the person doing the work. That is the part I want to keep.

Originally published at https://ketushah.com/writing/you-cant-hire-judgment-later/

Auditing the invisible

Ketu Shah — Sat, 14 Nov 2026 21:46:40 +0000

Originally published at https://ketushah.com/writing/auditing-the-invisible/

Auditing the invisible

Every system I build has two jobs, and they fight each other: it has to keep working when something breaks, and it has to tell someone something broke. Users see the first job. Nobody except the maintainers sees the second. And the better the first job goes, the worse the second gets. That tension is the whole subject. Resilience and visibility are not allies. Every fallback trades today’s calm for tomorrow’s ignorance.

The invisible failure is the fallback that works. Nothing errored, nothing paged. The monitor stayed green because the backup path did exactly what it was built to do. The user got an answer. And the system swallowed the break. Underneath, the real path is dead. Nobody knows, because nobody was ever told. So who was supposed to notice?

So who tells you? The rule I work by is simple: silent to users, loud to maintainers. A failure can be silent to the person using the system. It has to be louder for the people who maintain it. Graceful for the user and quiet for the maintainer is not graceful. It is hidden.

Loud has an address, and it is not the receiver. When a service fails and its caller falls back, the caller absorbed the failure. But alerting there chases the symptom. The failure has to be loud where it happened, because the receiver will never complain. It got an answer, or a spare path, and moved on. Only the origin knows something broke. That is where the noise has to live.

GitLab learned this in public. On January 31, 2017, an engineer deleting what he thought was a replica database deleted the primary instead, and about 300 gigabytes of GitLab.com’s production data went with it. The team reached for its backups and found what GitLab itself later described as five backup and replication techniques, none working reliably or set up in the first place. The nightly backup had been failing silently on a version mismatch, leaving zero-byte files. The failure emails never arrived, rejected by the mail server, so the absence of failure emails read as success. The snapshots that could have saved them were never enabled for the database servers, and the replica was already wiped. The restore came from a snapshot taken six hours earlier, by chance, for unrelated work. Eighteen hours down, six hours of data lost for good. The safety net was the invisible failure. Five mechanisms, believed alive, all dead. And the watcher failed silently too. GitLab is one instance. The take is wider than backups. Replicas, failovers, caches, standby systems, the backup path. All of them exist for the moment the main thing goes wrong. A resiliency mechanism that fails silently is worse than not having one. It hands you confidence you have not earned. You only discover it is dead the moment you need it. That is the worst moment to discover anything. The least-exercised path in any system is the most likely to be silently dead. Resiliency paths are exercised least of all, by design. Until you have actually fallen back onto it, it is a belief, not a fact.

That is how a fallback becomes the system. It does not happen in one decision. Nobody sits down and votes to make the degraded path permanent. It happens because it works. Users are fine, so nobody notices. Because nobody notices, nobody owns the fallback rate, so nobody reviews it. New engineers learn the degraded path as if it were the design. Runbooks get written around it. One day it breaks, and there is nothing underneath it, because the thing underneath it stopped existing a long time ago. Nobody marked the date. The sociologist Diane Vaughan gave this a name studying Challenger, where O-ring erosion kept recurring without disaster until it was treated as normal: normalization of deviance. A degraded condition, survived repeatedly, stops looking degraded.

When one surfaces, I don’t improvise. Fix what broke. Then the cause, or an alert in time if the cause can come back. Then the widening. I ask the same questions of everything like it: does it even have a watcher, does that watcher fire where the failure happens instead of where it lands, and does anyone actually own the alert when it fires. A monitor nobody owns is not a monitor. The fix is not the end of it. A broken thing is a chance to audit the whole class. And if you stop at the one that broke, the rest of the class stays exactly as blind.

It happens more often now. Especially when code is cheap. Every new path a system gains is another place for a failure to sit quietly, and cheap code adds paths fast. Defensive code is the worst of it. Every catch-all that returns a default is a new place something can break without anyone noticing.

Loud has a limit, though. Over-logging is not observability. Most of it is noise, and noise is its own kind of invisibility, because a team that hears everything stops hearing anything. Logs, metrics, and APM have to work together and tell one story of the system. Three separate piles of data are not a story. When they cohere you can see what the system did, and when they do not, you are back to guessing, with more data than before.

I run this on my own systems, in the same order. Fix, cause, widen. The audit starts at home, while fixing it is still cheap.

Originally published at https://ketushah.com/writing/auditing-the-invisible/

The most humane invention in computing

Ketu Shah — Sat, 14 Nov 2026 21:46:14 +0000

Originally published at https://ketushah.com/writing/the-most-humane-invention-in-computing/

The most humane invention in computing

To err is human. Pope wrote that in 1711, and three centuries later it is still the most important sentence in software design. People are going to make mistakes. Not occasionally, not the careless ones among us. Everyone, constantly. It is the starting condition, and any system that takes input from a user has to be designed from it.

Which is probably why undo was never invented once. It kept getting invented. A system at Brown had it in 1968. A Lisp programming tool had it by 1971. Xerox PARC put it in a text editor in 1974. These people were not copying each other. They were all watching the same thing happen: a person at a keyboard, doing something they did not mean to do, with no way back. When the Macintosh shipped, Undo sat at the top of the Edit menu. It still does.

So the rule I design by: wherever there is user input, there has to be a way to correct it. Self-serve wherever possible, a button, an edit, a window of time to change your mind. And where self-serve genuinely cannot work, an easy path to something that can correct it, another system or an actual human. Amendment or undo, in one form or another, everywhere input exists. It is not a feature to add when there is time. It is part of what accepting the input means.

Where it gets hard is the guardrails, and I will be honest: there is no formula for those. It is circumstance based. What decides it is the consequences of the undo itself. Before I let a user reverse something on their own, I want the action mapped, upstream and downstream. What consumed this input? What already acted on it? If the damage from reversing is contained, hand the user the button and get out of the way. If the reversal ripples into other systems and other people, self-serve is the wrong tool. That correction needs a process, checks, maybe a human reviewing it. The mistake gets corrected either way. The route is what changes.

The map has a clock on it too. Ask whether the mistake’s ramifications are permanent, and whether they have already landed. Order something online and you can cancel it, but the window depends on the vendor: zero minutes from one, fifteen from another. Then the order ships and the window does not narrow. It ends. Anyone who has tried to get UPS or FedEx to intercept a package or reroute it midstream has felt the far side of that line. The act left the database and entered the physical world, and only the database part was ever reversible.

Some of that is just physics, and physics is not a design flaw. Some of it is complexity that got baked in for reasons nobody remembers. Either way, once the act itself cannot be undone, the question changes to whether its consequences can be amended. You cannot un-ship a box. You can return it, refund it, send the right one. And the amendment has a simple test. It worked when whatever the user intended to achieve is achieved. Not when the ticket closed. When the person ends up where they were trying to go.

Gmail’s Undo Send is the famous version of this. Launched in 2009, and it has never recalled a single email. It delays the send, five seconds at first, up to thirty now, so the undo is really a grace period with good marketing. I mean that as praise. Where reversal is impossible, time is the substitute. A window before the act becomes real, and where that fails, an amendment after.

Given all that, why do so many systems give you no way back at all? Collect the reasons and they are mostly unglamorous. Complexity. Oversight. Cost. A dependency on some external party whose system will not cooperate. A product team that never foresaw the need. A mistake so rare that building the correction never survived prioritization. Those are engineering economics, and some of them are even true. But there is one more reason on the list, and it is not like the others. Sometimes there is no undo because no undo is the business model. A flow that takes ten seconds to enter and ten days to leave is not an accident. It is somebody’s decision, taken for gains.

Here is the part that should bother us. From the user’s chair, there is no way to tell those reasons apart. The screen does not say whether the correction path is missing because it could not be built, or because somebody chose not to build it. It is just not clear, not until the user makes the mistake and goes looking for the undo that may or may not be there. That is when a system shows you what it thinks of you. Ideally before you need to know.

Originally published at https://ketushah.com/writing/the-most-humane-invention-in-computing/

Dashboards don’t lie, but selection does

Ketu Shah — Sat, 14 Nov 2026 21:40:41 +0000

Originally published at https://ketushah.com/writing/notes-4/

Dashboards don’t lie, but selection does

Every company has this meeting. Dashboard on the screen, numbers look good, room relaxes. Nobody asks where the numbers came from. Nobody asks what’s missing. The data is real, so the story must be true.

That’s the trick. Nobody lied. Somebody chose.

A dashboard is built on real data. That’s what makes it dangerous. Nobody questions real data. The problem was never the numbers. It’s what got picked and what got left out. Same data, two opposite stories, both pointing at the same dashboard. I’ve watched it happen enough times to stop being surprised. And AI is making it easier, not harder: one dataset, two confident narratives on demand, both citing the same source.

The mechanisms aren’t exotic. They’re ordinary. That’s why they keep working.

It usually starts with the baseline, because that’s where most of the framing lives. Pick a skewed baseline and you’ve picked the conclusion before the analysis starts. Show a trend with no baseline and every uptick looks like momentum. Compare against a weak season and call the rebound growth. You haven’t touched the data. You’ve touched the frame around it.

Then there’s the experiment that was never a fair fight. Two versions of this: set it up to fail, or set it up to prove a point. Either way the outcome was decided before the first number landed. The dashboard just makes it look objective. Throw in the external factor nobody mentions, the thing outside anyone’s control that explains the whole move, and you’ve got a story that’s technically accurate and completely misleading.

What gets left out matters as much as what gets put in. Anything that can’t be quantified quietly disappears, as if unmeasurable means unimportant. Inputs vanish while outputs get celebrated. More spend, more people, more effort, reported with no counter-metric anywhere. Nobody shows the before, only the after. And channels get judged on the narrowest definition of performance. Few direct conversions? Written off. The awareness it built upstream, the part attribution never sees, goes uncounted. For complex products especially, the thing that looks like waste on the dashboard is often doing quiet work the dashboard wasn’t built to see.

None of this requires lying. Every number on the screen can be correct.

Sit with that for a second, because it breaks how most people think about data. “The numbers are right” feels like the end of scrutiny. It’s the beginning.

Is it deliberate? Sometimes. Sometimes it’s tunnel vision. People see the metric they own and stop there, never noticing the frame they’re standing inside. And sometimes it’s sharper than that: the metric is good for someone’s career, so the framing follows the incentive. Telling those apart takes investigation, not instinct. Why this metric. Why this window. Why this segment. Keep asking until the answers run out. Most people stop at the first answer that sounds reasonable. That’s exactly where framing survives.

My checklist changes with the metric, but the shape of it doesn’t. First: what’s new? What changed, what’s being tried, where did resources move, what got optimized? A number never moves for no reason, and “new” is usually the reason. Second: check the third parties. If something upstream feeds you, or you feed something downstream, what changed on their end? Some of the most confusing dashboard mysteries I’ve seen were just a vendor quietly changing something. Third, the part most people skip: trace the subsequences downstream. Find the core indicators where the biggest impact should show up. Nobody watches these metrics. They carry no business value on their own, so they’re not on any dashboard and nobody gets promoted for moving them. But they’re where the real trend shows up first, before it’s been dressed for presentation. Headline metric says everything’s fine and the subsequence disagrees? Trust the subsequence.

Now the AI part. I’m an early adopter, not a skeptic of the technology. I use these tools every day. But on this specific problem, AI summaries make selective framing worse.

It’s structural. Sycophancy is built in, and it persists no matter how hard the frontier companies try to train it out. I checked; the research backs this up through 2026. The model reads how you framed the question and tells you what you wanted to hear. It assists. It doesn’t push back. Plus its own biases stacked on yours: training data, recency, a kind of context aphagia where it gorges on whatever was said most recently and treats it as the whole picture.

So you ask it to summarize a dashboard and it summarizes the frame. It doesn’t interrogate it. The selective story comes back cleaner, more confident, more authoritative. Two sides, same data, opposite stories, and now each side has a tireless assistant that never asks what was left out. The odds of that happening don’t go down. They go up.

People keep asking me for the structural fix. The one dashboard feature that would make selective framing harder. I don’t think it exists. No visualization, no alert, no auto-insight. What works is older and less convenient: people holding each other accountable, real transparency, and hard questions for anyone driving a narrative. Ask what was excluded. Ask who benefits if this version wins. Details kill framing. It can’t survive specificity.

The data is real. The story is a choice. Check what’s outside the frame.

Originally published at https://ketushah.com/writing/notes-4/

The selectivity audit

Ketu Shah — Sat, 14 Nov 2026 21:39:53 +0000

Originally published at https://ketushah.com/writing/the-selectivity-audit/

The selectivity audit

A while back I wrote Dashboards don’t lie, but selection does. The argument there was simple: nobody has to fake a number for a story to mislead. Somebody just chooses what goes in the frame. Fine. But a diagnosis is not a practice, and complaining about framing is easy. This is the other half. What I actually do when a clean story lands in front of me.

The number is the last thing I look at. First, the story. What is this trying to tell me? Every update and readout is built to land somewhere. Give me the claim in one sentence, the sentence I am actually being asked to believe, and now we have something that can be audited.

Then motive. It matters what room the story was built for. An OKR getting relayed is one thing. An achievement being claimed is another. Business health, another again. The pressures are different in each, and so is the framing. Underneath that there is a plainer question: what is tied to this number? Whose performance, whose gain, gets better if the number lands a certain way? I am not accusing anybody of anything. Framing pressure does not need intent. It just needs something riding on the outcome.

Then the notes. If there are any. A chart gives you a message. A summary gives you a message. Neither gives you the story behind the message, and the story is where the gotchas are, the nuances, the caveats, the whole fine print the message was built on. That is what I read the notes for. When the notes confess the limits, I start trusting the story a little. When there are no notes, the story is asking to be believed at full price, and I notice that.

The questions come after all that, and by then they are not generic. They are aimed at this story. Usually five. What led to it, because numbers do not move by themselves; something got spent or shipped or changed, and a story with the input missing is incomplete in a way that is rarely an accident. What are we comparing against, since every result is a comparison in disguise and the baseline is where conclusions get decided without a fight. Where are the counter-metrics, the numbers that paid for this gain, because there is always a metric that absorbed the cost and its absence from the story is itself a fact. How did the competition do, if that exists. And was there a headwind or a tailwind, because markets rise and seasons turn and algorithms change, and until the outside world is priced in, credit and blame are both fiction.

If this sounds abstract, it should not. The most expensive framing failure of the last decade collapsed in about six weeks, and not because new facts showed up. The facts had been filed in public the whole time. Somebody finally read the definitions.

In August 2019, WeWork filed its paperwork to sell shares to the public. The business was renting desks: long leases on buildings, short memberships for the people at the desks, and a story that valued the arrangement at 47 billion dollars. The filing told a second story to anyone who read it closely. WeWork’s favorite measure of profit was a number of its own invention called Community Adjusted EBITDA. It started with the revenue, subtracted the costs of running the buildings, and left out general and administrative costs, sales and marketing, and much of what it costs to run the company doing the reporting. On that measure, 2018 was a profitable year, hundreds of millions in the black. On the ordinary measure printed in the same document, WeWork had lost 1.9 billion dollars on 1.8 billion of revenue. Nothing was faked. Both numbers were real. The frame was a definition, and the definition sat in the notes, where definitions live, disclosed to anyone willing to read it.

Run the five questions over it and watch how fast they bite. What was tied to the number? A public offering and a 47 billion dollar valuation. Motive does not come louder. What led to the profit? A metric that excluded the cost of being the company reporting the profit. Compared against what? The older office landlord, IWG, had taken in nearly double the revenue in 2018 and earned an actual profit, and the market valued it at a small fraction of WeWork. The story never volunteered that comparison. Where is the counter-metric? A 1.9 billion dollar loss, in the same filing, one definition away. Headwind or tailwind? A decade of cheap money, lifting every growth story in sight whether it deserved lifting or not. Six weeks after the filing, the offering was withdrawn, the valuation was being cut by tens of billions, and the chief executive was gone. WeWork was rescued that October at a valuation of roughly 8 billion, and it filed for bankruptcy four years later. That is what a framed story looks like when the questions get asked early. The tragedy in most cases is not that the questions are hard. It is how long they wait to be asked.

Last part, and it is the part that keeps the rest honest. I run this sequence on my own work before it goes out. Same order. Same questions. My stories first, while fixing them is still cheap. I am not a hypocrite about it. I do not frame the story; I tell it as it is, and when I have selected something out, the notes say what and why. An audit you only aim at other people is not an audit. It is a weapon with good branding.

A story that survives these questions has earned its clean edges. Most have not. That is what the questions are for.

Originally published at https://ketushah.com/writing/the-selectivity-audit/