DEV Community: Amrishkhan Sheik Abdullah

200 Broken Watches Couldn't Solve It. A Bouncing Ball Did.

Amrishkhan Sheik Abdullah — Sat, 05 Sep 2026 07:58:15 +0000

Who Saw the Connection? #05 — G-SHOCK × A Bouncing Ball × Engineering Breakthroughs

In the early 1980s, Kikuo Ibe and a small team at Casio spent roughly two years trying to make a watch survive something watches weren't supposed to survive.

They built more than 200 prototypes. Ibe dropped them from a third-floor bathroom window at Casio's research facility. Components broke, designs changed, more prototypes followed, and eventually the team reached that particularly unpleasant stage of engineering where you're no longer sure whether the next iteration is progress or simply a more expensive way of confirming that the previous one failed.

The breakthrough didn't come from another watch.

It came when Ibe saw children playing with a rubber ball.

That observation helped produce G-SHOCK.

And strangely enough, solving the engineering problem wasn't even the end of the story. Casio still had to convince people that they wanted the thing.

It Started With a Broken Watch

The story begins in 1981 with an accident that was completely ordinary at the time.

Ibe owned a watch given to him by his father. One day, it fell from his wrist and broke.

Today, that sounds like a product failure. At the time, it wasn't necessarily seen that way. Wristwatches were precision instruments. You were expected to treat them carefully. Thinness and elegance were desirable; surviving serious physical punishment wasn't generally part of the bargain.

There is an interesting habit in engineering where, if a limitation survives long enough, we eventually stop describing it as a limitation.

We start describing it as proper usage.

Don't drop the watch. Don't send requests that quickly. Don't run that query during peak hours. Don't let the payload get that large. Don't deploy on Friday.

Some of those rules exist for excellent reasons. Others are historical compromises that quietly acquired the status of natural law.

Ibe questioned one of them.

He submitted a proposal internally at Casio with a remarkably simple objective: a watch that wouldn't break when dropped.

Casio approved the idea, and Ibe joined a small development group called Project Team Tough.

Then they made the requirement considerably less reasonable.

Triple 10 Was an Almost Rude Engineering Requirement

The development philosophy became known as Triple 10:

survive a 10-metre drop
withstand 10 bar of water pressure
target a 10-year battery life

That combination matters.

Casio wasn't merely asking engineers to improve one specification. A wristwatch contains a quartz timing mechanism, electronics, an LCD, a battery and multiple connections, all packed into something small enough to wear comfortably.

Shock resistance alone is difficult.

Water resistance alone is difficult.

Long battery life imposes another set of constraints.

Now solve all of them without producing something approximately the size of a microwave.

The original production G-SHOCK wouldn't literally achieve every Triple 10 number. The DW-5000C actually exceeded the water target with 20-bar resistance, while its specified battery life was seven years. Triple 10 was the development ambition that shaped the project rather than a literal specification sheet for the first model.

That distinction is important because the mythology around successful products tends to become cleaner than the engineering that created them.

The real story is better.

The requirement was aggressive enough that the existing idea of a wristwatch stopped being particularly useful.

Then They Started Throwing Watches Out of a Window

Casio says Project Team Tough produced more than 200 prototypes over roughly two years.

Testing shock resistance initially involved a beautifully direct piece of engineering equipment:

a third-floor bathroom window.

Ibe repeatedly dropped prototypes from roughly ten metres above the ground at Casio's research facility.

Something would break. The team would examine it, reinforce it and build another version. That prototype would survive one failure mode and reveal another.

They experimented with cushioning. They added rubber. They strengthened vulnerable components. At one stage, the protective structure became so large that Casio describes a prototype approaching the size of a softball.

This is where the number “200 prototypes” is often turned into a motivational story.

Keep trying.

Never give up.

Prototype number 201 might change your life.

That's pleasant advice. It also misses the most interesting part of the engineering.

The prototypes weren't valuable because there were 200 of them.

They were valuable because the failures were accumulating information.

Strengthen one component and the impact finds another path.

Protect that component and something else becomes the weak point.

Add enough protection and you've technically saved the watch by turning it into an object nobody wants to wear.

Eventually, the failures were telling Ibe that he might be solving the wrong problem.

Then He Saw a Child Bouncing a Ball

According to Casio's account, development eventually reached the point where Ibe gave himself one final week to find a solution.

On the final day, a Sunday, he went to a park near the research facility.

Children were playing with a rubber ball.

Nothing about that scene had anything obvious to do with watches. Nobody was conducting an experiment. The children were not unknowingly participating in Casio's R&D program.

But Ibe noticed something.

When the ball hit the ground, its exterior absorbed and deformed around the impact while its centre remained comparatively isolated from the collision.

That changed the question.

Until then, much of the effort had been directed toward making the watch's individual components capable of surviving shock.

The ball suggested another approach:

What if the important components didn't have to take the impact directly at all?

Instead of trying to make the mechanism indestructible, isolate it.

That idea contributed to the hollow, shock-resistant structure behind G-SHOCK. The watch's critical module could effectively float inside the protective case, reducing the amount of impact energy transmitted directly into the electronics.

The breakthrough wasn't stronger components.

It was a different relationship between the components and the force trying to destroy them.

Millions of people had watched children bounce rubber balls.

Ibe saw shock isolation.

That is the moment that makes this a Who Saw the Connection? story.

The Best Debugging Sometimes Changes the Question

There is a software version of the 200 broken watches.

A service starts timing out, so we increase the timeout.

Now requests survive longer but consume connections, so we increase the connection pool.

That increases pressure on the database, so we scale the database.

Retries begin amplifying traffic, so we add backoff.

Then we add caching.

Then somebody proposes another queue.

Six weeks later, the architecture diagram looks like it has developed a stress disorder.

Every individual change may be reasonable. The danger is that we can become extremely good at solving the latest symptom without reconsidering the model generating those symptoms.

Ibe could have continued reinforcing components.

Instead, the bouncing ball gave him another abstraction.

The question changed from:

How do I make every component survive impact?

to:

How do I stop the impact reaching the components?

That distinction appears everywhere in serious engineering.

Don't make the downstream service infinitely tolerant of duplicate requests if you can make the operation idempotent.

Don't keep increasing worker capacity if the real problem is uncontrolled upstream demand.

Don't repeatedly repair corrupted state if the architecture allows invalid state to exist unnecessarily.

Sometimes the breakthrough isn't a better answer.

It's discovering that you've been asking the wrong question.

In 1983, They Finally Built It

The first G-SHOCK, the DW-5000C, launched in April 1983.

It looked like the engineering problem that had created it: thick, protected, functional and unapologetically different from the slimmer watches popular at the time.

Two years of work had produced something genuinely unusual. The watch could tolerate punishment that would have destroyed conventional watches, and the structural thinking behind it would become foundational to the G-SHOCK family.

This is where the movie version ends.

The engineer refuses to give up. The impossible product finally works. Customers immediately recognize its brilliance. Inspirational music arrives precisely on schedule.

Actual markets have a crueler sense of humour.

The G-SHOCK initially struggled in Japan.

Casio's own retrospective describes a market where thin watches were mainstream and the bulky G-SHOCK wasn't immediately embraced. One person involved with the project later recalled that there was “almost no reaction at all” in Japan initially.

Think about that.

Two years.

More than 200 prototypes.

A genuinely different architecture.

A product that solved something people had largely accepted as unavoidable.

And the market essentially responded:

That's quite a large watch.

Technical achievement and market acceptance, it turns out, are completely different engineering problems.

America Saw Something Japan Didn't

The G-SHOCK story became stranger when the watch reached the United States.

Its ruggedness made immediate sense to people whose jobs or lifestyles were hard on equipment. Casio later identified early adoption among firefighters, police officers, military personnel and other professionals working in demanding environments.

The same characteristic that made the watch seem unnecessarily bulky to one audience made it useful to another.

Then Casio's American marketing operation found an exceptionally efficient way of explaining the product.

They used a G-SHOCK as an ice-hockey puck.

A 1984 television advertisement showed a player taking a shot with the watch. The goalkeeper caught it. The G-SHOCK survived.

This was a fantastic advertisement partly because some viewers thought it was nonsense.

A precision electronic instrument surviving a hockey slapshot sounded suspiciously convenient for a commercial.

Questions about the claim reached an American television program that investigated potentially misleading advertising. The program recreated the test.

The watch survived.

Casio's account says they went further and placed a watch on a road before running over it with a large truck.

Still working.

Casio had spent years engineering a difficult-to-believe property into the product.

Skeptical television producers accidentally gave them something even better than advertising:

independent proof.

If Your Claim Sounds Extraordinary, Stop Describing It

There is a wonderfully practical lesson here for anyone trying to build anything.

Imagine Casio advertising the G-SHOCK like this:

Featuring an innovative hollow-case architecture and advanced shock-resistant construction for exceptional durability.

Perfectly accurate.

Completely forgettable.

Now compare it with:

Here's our watch. We're going to use it as a hockey puck.

You understand the product in approximately three seconds.

Technology is full of products whose strongest characteristics are buried under adjectives.

High-performance.

Enterprise-grade.

Privacy-first.

AI-powered.

Scalable.

Resilient.

After enough years in software, these words develop a remarkable ability to pass through the human brain without disturbing anything on the way.

If the thing you built is genuinely unusual, demonstrate the unusual part.

If you built a privacy-first developer tool that processes data locally, show the network tab while somebody processes a sensitive file.

If your coding-agent plugin claims to prevent bloated AI changes, show a 600-line patch becoming 40 lines while preserving the requirement.

If your architecture dramatically reduces latency, run the workload.

If you're interviewing for an architecture role, don't spend fifteen minutes saying you're good at architecture. Show a difficult decision, the alternatives you rejected, the failure mode you were protecting against and what happened in production.

Proof travels farther than adjectives.

G-SHOCK happened to have one of the greatest product demonstrations imaginable.

Hit it.

Still works.

No architecture diagram required.

Then Japan Imported Its Own Idea Back

By the late 1980s, something unexpected was happening.

G-SHOCK was gaining an identity in the United States that it hadn't originally possessed in Japan.

Its toughness had practical value for people in demanding professions, but the watch also began spreading through sports and youth culture. Skateboarders, snowboarders, surfers and other communities found something appealing in the same rugged design that had once worked against it.

Then that image crossed the Pacific.

Casio executives later described noticing Japanese G-SHOCK sales increasing around 1989–1990 and investigating what had changed.

Their conclusion was unusually fascinating.

It was a reverse import.

The watch had been designed in Japan, struggled to find enthusiasm there, acquired popularity and cultural meaning in America, and then returned to Japanese consumers through magazines and American cultural influence.

Casio eventually leaned into this deliberately. Its Japanese marketing emphasized the watch's American popularity, including campaigns built around the message that America loved G-SHOCK.

The engineering hadn't suddenly improved during the flight back to Japan.

The product was fundamentally the same.

What changed was what the product meant.

The Same Product Can Be Wrong in One Room and Right in Another

This may be the part of the G-SHOCK story I find most useful outside product development.

We tend to interpret rejection as a verdict on quality.

Sometimes that's exactly what it is.

But rejection can also contain another message:

wrong audience.

G-SHOCK's toughness mattered immediately to people whose watches were exposed to physical punishment. Later, that toughness acquired another meaning among sports and youth communities. Eventually, the chunky design itself became part of the fashion.

Same watch.

Different people saw different value in it.

That happens in careers constantly.

An engineer can work somewhere that values ticket throughput above everything else. Their habit of thinking about architecture, failure modes and long-term maintainability is treated as unnecessary complication.

Move that same engineer into an organization struggling with distributed systems, platform reliability or technical strategy and the trait that previously looked like overthinking may become exactly why they were hired.

The engineer didn't suddenly become smarter on Monday.

The room changed.

The same thing happens to ideas. An open-source project can receive almost no attention because it was shown to the wrong developer community. An article can disappear on one publishing platform and spread rapidly on another. A product aimed at enterprises can fail while individual developers quietly love it.

The dangerous conclusion is:

Nobody wants this.

Sometimes the more accurate conclusion is:

Nobody here wants this.

Those sentences are separated by one word.

Careers and companies can be separated by the same distance.

But “Wrong Audience” Can Become a Very Comfortable Lie

There is an obvious trap here.

If nobody likes what you've built, you cannot spend the next five years declaring that society has simply failed to recognize your genius.

Sometimes the product is bad.

Sometimes the feature isn't useful.

Sometimes your résumé isn't communicating enough evidence.

Sometimes the article is boring.

Sometimes the side project solves a problem nobody actually has.

The G-SHOCK lesson isn't to ignore rejection.

It is to diagnose rejection correctly.

Ask what the evidence says.

Does the underlying thing work?

Can you demonstrate the value?

Has anyone independently found it useful?

Does another audience respond differently?

Are people rejecting the capability itself, or the way you've positioned it?

G-SHOCK accumulated evidence.

The watch survived punishment. Professionals bought it. American sales grew. Different communities independently found reasons to wear it.

That evidence justified reconsidering the market.

Without evidence, “wrong audience” is merely a pleasant story we tell ourselves.

This is why careers, products and engineering failures all require roughly the same uncomfortable skill:

separating persistence from denial.

Triple 10 Was Really a Bet Against Incremental Thinking

There is another reason I like the Triple 10 story.

Imagine Casio beginning with the existing watch architecture and asking what improvement seemed reasonable.

Perhaps they could make it 20% more resistant to shock.

Add some cushioning.

Improve the case.

Strengthen a few components.

That could have produced a perfectly respectable watch.

It probably wouldn't have produced G-SHOCK.

Triple 10 approached the problem from the other direction. It described the desired outcome first and allowed that outcome to make the existing architecture uncomfortable.

Software requirements often get negotiated in reverse.

We quietly shape the requirement around whatever system already exists.

“The database can't support that.”

“Our deployment model doesn't work that way.”

“The framework makes that difficult.”

Those are useful engineering constraints. Ignoring reality is not architecture.

But occasionally the requirement is telling us something else:

the architecture is answering an old question.

Good engineering judgment is knowing when a constraint should be respected and when the constraint itself deserves investigation.

Ibe could have produced a stronger conventional watch.

Instead, the requirement eventually forced him to reconsider what “protecting a watch” actually meant.

Two Hundred Attempts Are Useless If Attempt 201 Learns Nothing

This is where I would resist turning the G-SHOCK story into motivational wallpaper.

“Never give up” is incomplete advice.

If prototype 198 teaches you exactly what prototype 12 taught you, you're not demonstrating resilience. You've developed a remarkably expensive hobby.

The useful part of failure is the information it produces.

One component breaks. Strengthening it exposes another weak point. More cushioning makes the product impractically large. Eventually enough evidence accumulates that the original model becomes difficult to defend.

Then something has to change.

That applies far beyond engineering.

Twenty rejected job applications may mean you need to send twenty more. They may also mean your positioning isn't working.

Six months of low product adoption may mean marketing needs more time. It may also mean you've misunderstood the problem.

Five architectural fixes may mean the sixth needs more effort. They may also mean you've been fixing the same symptom five different ways.

Persistence matters enormously.

But persistence without model correction is repetition.

The difficult skill isn't refusing to quit.

It's understanding what the failure is trying to tell you before deciding what to do next.

The Ball Is Still My Favourite Part

For all the marketing, cultural and career lessons hiding inside G-SHOCK, I keep coming back to the rubber ball.

Because Ibe wasn't studying shock-resistant products when the breakthrough appeared.

He was watching children play.

A bouncing ball and a wristwatch share almost nothing at the level of appearance.

The connection exists one level deeper.

The exterior experiences the collision.

The centre is protected.

That's transferable.

This ability to recognize the structure of a problem somewhere that uses completely different objects, language and assumptions is one of the most valuable forms of thinking I know.

Toyota engineers could look at supermarkets and think about manufacturing flow.

Hospital teams could look at Formula 1 pit stops and think about patient handovers.

Japanese railway operators could turn a simple physical gesture into a mechanism for reducing human error.

And a watch engineer could look at a child bouncing a ball and stop thinking about stronger watch components.

The world is full of solved problems wearing the wrong uniform.

Sometimes expertise means knowing your field extremely well.

Sometimes it means looking away from your field long enough to notice that somebody else has already solved the shape of your problem.

Who Saw the Connection?

G-SHOCK eventually became one of Casio's defining products. By 2017, cumulative shipments had passed 100 million units, and Casio announced that the brand surpassed 150 million watches shipped worldwide in 2024.

Looking backward, success makes the path appear almost inevitable.

It wasn't.

There was a broken watch.

An unreasonable requirement.

A bathroom window.

More than 200 prototypes.

A design that kept failing.

A child bouncing a ball.

A product Japan initially didn't particularly want.

An American hockey commercial people thought was fake.

A television program that accidentally became product validation.

And eventually, Japan rediscovering a Japanese watch partly because America had taught it to look at the product differently.

That isn't a clean story about perseverance.

It's a much more useful story about seeing differently.

When the prototypes kept breaking, Ibe eventually changed how he saw the engineering problem.

When the Japanese market didn't respond, America saw a different use and identity for the product.

When people didn't believe the advertising, demonstration replaced description.

And when the product returned to Japan, consumers saw something in it they hadn't seen the first time.

That's why the rubber ball matters.

Hundreds of prototypes had already told Ibe that his current answers weren't enough.

The ball helped him ask a different question.

The next time something you've built isn't working—whether it's software, a product, an idea or even your career—the useful question may not immediately be:

How do I try harder?

It might be:

What am I still looking at the same way?

Because sometimes the breakthrough isn't hidden in another hundred attempts.

Sometimes it's bouncing around in a park.

Sources & Further Reading

Casio's official G-SHOCK history documents Project Team Tough, the approximately two-year development effort, more than 200 prototypes and the shock-resistant structure that emerged from the project.

Casio's history of the original 5000/5600 family describes Ibe's repeated drop tests from the third-floor bathroom window, the increasingly large prototypes and the observation of children playing with a ball that helped lead toward the floating-module structure.

Casio's material on Triple 10 documents the original development ambition and helps distinguish that philosophy from the literal specifications of the first DW-5000C.

The Japanese Patent Office has also documented the development of G-SHOCK, including the hollow structural concept associated with Ibe's observation of a bouncing ball.

Casio's corporate retrospective and integrated reporting provide the particularly interesting second half of the story: weak initial reaction in Japan, American adoption, the hockey-puck demonstration, growth through American sports and youth culture, and the later reverse-import effect back into Japan.

About the Person Who Somehow Ended Up Throwing Software Out of a Third-Floor Window

I'm Amrishkhan Sheik Abdullah, a Technical Lead and software architect based in Dubai. I build backend systems, microservices and distributed architectures for a living — which mostly means discovering that when the fifth patch doesn't fix something, the architecture may be trying to tell us something.

I write about software architecture, distributed systems, engineering decisions, AI developer tooling and, increasingly, ordinary products and behaviours that turn out to have surprisingly serious engineering hiding underneath them.

Who Saw the Connection? is my series about exactly that: true stories where somebody looked at an apparently unrelated problem and saw an idea everyone else was walking past.

Elsewhere on the Internet

Portfolio: amrishkhan.dev
GitHub: github.com/amrishkhan05
DEV Community: dev.to/amrishkhan05
LinkedIn: linkedin.com/in/amrishkhan

If you enjoyed this one, stick around. Apparently my current research strategy is to keep staring at ordinary things until somebody's strange decision from forty years ago starts explaining software engineering.

Why Do Japanese Train Drivers Keep Pointing at Things?

Amrishkhan Sheik Abdullah — Sat, 05 Sep 2026 05:05:00 +0000

Who Saw the Connection? #04 — Railways × Human Factors × Software

If you spend enough time on a railway platform in Japan, you may notice something that looks oddly theatrical.

A conductor checks the platform, points down it, says something aloud, looks toward another reference point, and repeats the process. Inside the driver's cab, the same pattern appears around signals, instruments and track conditions.

At first glance, it can look ceremonial. Maybe it's part of the formality people associate with Japanese railways. Maybe the employee is communicating with somebody farther down the platform.

The real explanation is more interesting.

They're performing a safety technique called shisa kanko, usually translated as pointing and calling. Instead of merely looking at an important signal or instrument, the worker points toward it and says its status aloud.

The practice has existed in Japanese railways in some form for more than a century. Calling out signal information appeared in railway instruction manuals by the 1910s; pointing was added later, during the late 1920s, and Japanese National Railways formally required pointing while calling signals in its regulations in 1970. (Nippon)

That history is interesting on its own.

What makes it worth writing about is why the ritual survived.

Because apparently looking at something and actually noticing it are not the same operation.

A Railway Problem That Was Really a Human Problem

Railways are full of repetitive decisions.

Signals are checked again and again. Instruments are read again and again. Doors are opened and closed hundreds of times. Routes become familiar. Stations become familiar. Procedures become familiar.

That familiarity is useful because humans would be painfully inefficient if every routine action demanded our full concentration.

Unfortunately, the same automaticity that makes experienced people fast also creates a peculiar failure mode: sometimes the brain becomes very good at seeing what it expects to see.

Most signals behave exactly as expected.

Until one doesn't.

Most gauges remain within their normal range.

Until one doesn't.

A person can therefore perform the physical act of looking without necessarily performing the cognitive act of registering what changed.

Railway operators discovered that simply telling people to “pay attention” was not a particularly sophisticated safety system.

Pointing and calling forces the check out of the worker's head.

The operator looks at the target, points at it, identifies it verbally and hears their own confirmation. Instead of allowing the brain to glide over a familiar object, the procedure turns observation into a small physical event.

It feels almost comically simple.

That simplicity turns out to be important.

Researchers Eventually Tested It

In 1996, researchers from Japan's Railway Technical Research Institute published a laboratory experiment designed to measure whether pointing and calling genuinely reduced errors rather than merely looking reassuring.

Participants performed a choice-reaction task under four conditions: no special confirmation, pointing only, calling only, and pointing while calling.

The reported error rates were:

Confirmation method	Error rate
No pointing or calling	2.38%
Pointing only	1.00%
Calling only	0.75%
Pointing and calling	0.38%

The combined technique reduced mistakes in that experiment to roughly one-sixth of the baseline rate. (J-STAGE)

There is an important caveat here.

That does not mean Japanese train drivers have 84% fewer accidents because they point at signals. The study was a controlled cognitive experiment, not a railway-wide accident trial.

But it does establish something more useful: forcing a person to externalize a confirmation can materially change how reliably they perform that confirmation.

That is a surprisingly deep result hiding inside what looks like a man pointing at a light.

The Finger Is Not the Important Part

It would be easy to turn this into a quirky piece of advice:

Software engineers should point at production servers.

I would prefer not to be responsible for that management initiative.

The gesture itself is not the transferable idea.

The interesting mechanism is that an important internal assumption is converted into an observable action before something consequential happens.

Imagine a train driver approaching a signal.

A purely mental workflow might be:

I saw it. Green. Continue.

Pointing and calling adds another step between perception and action.

The driver has to identify the actual object, direct attention toward it, verbalize what it says, and only then continue.

That is essentially a tiny piece of human-factors engineering inserted between “I think I know” and “execute.”

Software systems desperately need more thinking in that space.

Your Terminal Has the Same Problem

Consider a production database migration.

You have already tested it in development. Then staging. The pull request is reviewed, the change ticket is approved, and everything looks routine.

You have also done this sort of thing hundreds of times.

That sounds reassuring until you realize familiarity is precisely what makes the situation dangerous.

Beginners tend to move slowly around dangerous systems. Experienced engineers become comfortable around them. Usually that is what expertise is supposed to achieve.

Then one day somebody runs a perfectly valid command against the perfectly wrong database.

Nothing crashes because of a software defect.

Authentication works.

DNS works.

The database receives the command and performs it faithfully.

From the database's perspective, this is excellent service.

The human definition of success tends to differ.

A surprising number of operational interfaces ask humans to distinguish dangerous contexts using extremely weak visual cues. Perhaps the shell prompt contains prod instead of preprod. Perhaps one browser tab belongs to a different AWS account. Perhaps the Kubernetes context differs by three characters.

Then an engineer is expected to notice the distinction during an incident at two in the morning.

When they don't, the postmortem eventually contains the phrase human error, as though we have discovered an exotic new failure mode.

What Software Can Steal From the Railway

A better system doesn't merely ask the engineer whether they checked the environment.

It makes the environment itself part of the confirmation.

Suppose a migration tool displays this immediately before executing:

TARGET ENVIRONMENT : PRODUCTION
ACCOUNT            : 847291
REGION             : eu-west-1
DATABASE           : payments-primary

Type the database name to continue:

That is quite different from:

Are you sure? [y/N]

Anyone who has used computers for more than a few years has developed world-class muscle memory for defeating “Are you sure?” dialogs.

The first version requires the engineer to inspect something that exists in the actual execution context and feed that information back into the action.

It is a software equivalent of pointing at the signal and calling its state.

This idea already appears in good operational tooling. Destructive interfaces sometimes require users to type the resource name. Production deployment systems prominently display environment, commit and target information immediately before promotion. Safety-critical workflows may require a second operator to independently verify an action.

The common principle is simple:

don't merely ask whether someone checked; design the workflow so the check leaves evidence.

That is much stronger than another sentence in a runbook.

We Keep Trying to Solve Human Error With Documentation

A familiar incident-response pattern goes like this.

Something goes wrong. The team fixes the immediate problem, conducts a postmortem and eventually adds a new instruction:

Engineers must verify that the production environment is selected before running migrations.

There.

Safety has been upgraded.

Until somebody selects the wrong environment again.

The problem is not that the instruction is incorrect. Obviously the engineer should verify the environment.

The problem is that the instruction does nothing to change the circumstances under which the previous person failed to verify it.

Japanese railway workers did not spend decades developing pointing and calling because nobody had previously realized signals were important.

They developed it because knowing that something should be checked and reliably performing that check are different engineering problems.

Software organizations confuse those two surprisingly often.

We write documentation describing ideal human behavior and then treat deviations as individual mistakes.

Sometimes they are.

But sometimes the interface itself quietly assumes humans will remain perfectly attentive while performing repetitive work under time pressure.

That is an optimistic architecture.

Friction Is Not Always the Enemy

Modern product engineering has spent years removing friction.

Fewer clicks. Faster flows. Automatic defaults. Remember the previous value. Skip the confirmation. Infer the next step.

For routine actions, that is usually excellent design.

Nobody wants a three-person approval ceremony to change a CSS variable.

But we've become so accustomed to treating friction as inherently bad that we occasionally remove it from places where friction was doing useful work.

A railway driver pointing at a signal takes slightly longer than glancing at it.

That extra moment is not inefficiency accidentally left in the process.

It is part of the process.

The same distinction matters in software. A confirmation step that appears only before deleting a production database is not the same thing as an unnecessary modal appearing every time somebody saves a preference.

One is bureaucracy.

The other is a cognitive speed bump placed where moving too quickly is expensive.

Good engineering is not about removing all friction.

It is about knowing where friction costs more than it protects, and where a small inconvenience is extraordinarily cheap insurance.

There Is a Catch: Humans Can Automate the Ritual Too

Pointing and calling has its own uncomfortable failure mode.

Any procedure repeated often enough can become automatic.

A worker can eventually point, say the required words and continue while giving the confirmation very little thought. Japanese railway safety work has therefore also examined how to keep the practice from becoming an empty ritual and how to demonstrate its actual error-prevention effect during training. (Nippon)

Software engineers will recognize this immediately.

We are exceptionally good at turning meaningful safeguards into ceremonial checkboxes.

Did you review the logs? Yes.

Rollback tested? Yes.

Security impact considered? Yes.

Did anybody meaningfully perform those checks?

That question tends not to fit inside the form.

This is why the strongest confirmations interact with evidence rather than merely requesting agreement.

Don't ask:

Is this the correct database?

Show:

payments-primary

and require the operator to act on that identity.

Don't ask:

Are you sure this is production?

Make the production account visually and operationally different enough that the person has to acknowledge the actual context.

And wherever a machine can verify something reliably, let the machine verify it.

Pointing and calling is not an excuse for poor automation.

It is useful precisely where human judgment remains unavoidable.

Automation Makes the Remaining Human Actions Stranger, Not Safer

There is another reason this idea matters more as systems become increasingly automated.

Automation removes routine human intervention.

That is good.

But it means the actions humans still perform tend to happen when something unusual has occurred.

Think about an automated deployment platform that works perfectly 99.9% of the time. The operator no longer manually performs dozens of steps every day. They intervene only when the automation cannot determine what to do.

So the human is now performing a rarely practised task during an abnormal situation, probably while somebody is asking for an ETA in Teams.

That is exactly when memory and attention are least deserving of architectural faith.

The more sophisticated our automation becomes, the more carefully we should design the remaining moments where a human must make an irreversible decision.

“An experienced engineer will notice” is not a control mechanism.

It is hope wearing an access badge.

The Practice Escaped the Railway

Pointing and calling did not remain confined to train operations. It spread into Japanese factories, construction sites and other industries where repetitive work and human error can combine badly. (Nippon)

That spread makes sense because the underlying problem was never really about trains.

Railways simply happened to encounter it early and frequently.

Any environment containing repetitive operations, dangerous exceptions and humans working under time pressure eventually faces the same question:

How do you make sure somebody notices the one thing that changed?

Different industries answer differently.

Checklists.

Two-person verification.

Physical interlocks.

Audible alarms.

Typed confirmations.

Pointing and calling.

The implementations differ, but the systems thinking underneath them is remarkably similar.

You do not solve human fallibility by demanding less fallible humans.

You redesign the interaction around the human you actually have.

Maybe the Pointing Isn't the Strange Part

Once you understand what Japanese railway workers are doing, the pointing stops looking particularly strange.

What starts looking strange is how many software systems rely on much weaker safeguards.

We put staging and production behind nearly identical screens.

We distinguish destructive commands from harmless ones with a flag.

We hide critical context inside terminal prompts that experienced engineers barely consciously read anymore.

Then somebody misses something and we add another paragraph to the runbook.

Railway workers arrived at a more practical conclusion more than a century ago: if a particular observation matters enough, don't leave it entirely inside somebody's head.

Make them identify it.

Make them interact with it.

Make the confirmation part of the operation itself.

The laboratory evidence suggests that this apparently simple change can substantially reduce mistakes during confirmation tasks. (J-STAGE)

And there is something wonderfully unglamorous about that.

No artificial intelligence.

No predictive analytics.

No billion-dollar control system.

Just a small piece of behavior designed around an inconvenient fact about humans.

Sometimes we look directly at things and don't really see them.

Who Saw the Connection?

This series keeps pulling me toward engineering decisions that are sitting in public view while almost everyone walks past them.

A UPS truck avoids a left turn and most of us see traffic.

A railway worker points at a signal and most of us see a strange gesture.

But underneath both behaviors is deliberate engineering: somebody studied a mundane action closely enough to discover that changing it could make an entire system behave better.

That is the part I find fascinating.

Software engineers tend to search for solutions inside software. We look for a new framework, architecture pattern, database or protocol because our vocabulary makes our problems feel unusually technical.

But reliability problems existed long before Kubernetes decided to give them YAML.

Railway operators have spent more than a century dealing with repetitive work, incomplete attention, dangerous exceptions and humans who occasionally make mistakes while doing something they have successfully done thousands of times.

Their solution to one part of that problem looks almost absurdly simple.

Point at the thing.

Say what the thing actually is.

Then act.

Perhaps the next time a production procedure depends on somebody “just checking carefully,” we should ask a slightly different question:

What would make the check impossible to perform without actually noticing what you're checking?

Japanese railway workers have been answering that question with one finger for generations.

Most passengers never realize why.

And that's exactly what makes the story worth noticing.

Sources & Further Reading

For the history and modern use of pointing and calling, Nippon.com provides a useful overview covering its development from early call-and-response railway practices, the addition of pointing in the late 1920s, formalization by Japanese National Railways and adoption beyond railways. (Nippon)

The experimental evidence comes from Shigeru Haga, Hajime Akatsuka and Hiroaki Shiroto's 1996 laboratory study, Laboratory experiment for verifying the effectiveness of “finger-pointing and call” as a practical tool of human error prevention. (J-STAGE)

The older origin story around locomotive engineer Yasoichi Hori is widely repeated, but contemporary reporting notes that the exact historical origin remains uncertain, which is why I have not treated that anecdote as established fact here. (The Japan Times)

About the Person Who Somehow Ended Up Pointing at Production Servers

I'm Amrishkhan Sheik Abdullah, a Technical Lead and software architect based in Dubai. I build backend systems, microservices and distributed architectures for a living — which mostly means trying to design systems where humans don't have to remember twelve critically important things at exactly the worst possible moment.

I write about software architecture, distributed systems, engineering decisions, AI developer tooling and, increasingly, ordinary-looking things that turn out to have surprisingly serious engineering hiding underneath them.

Who Saw the Connection? is my series about exactly that: true stories where everyday behaviors, objects or decisions reveal an engineering idea worth stealing.

Elsewhere on the Internet

Portfolio: amrishkhan.dev

GitHub: github.com/amrishkhan05

DEV Community: dev.to/amrishkhan05

LinkedIn: linkedin.com/in/amrishkhan

If you enjoyed this one, stick around. Apparently my current research strategy is to stare suspiciously at ordinary things until they confess why they were designed that way.

What Did Heart Surgeons See in a Formula 1 Pit Stop?

Amrishkhan Sheik Abdullah — Fri, 04 Sep 2026 06:11:00 +0000

Who Saw the Connection? #03 — Formula 1 × Pediatric Heart Surgery × Software

In 2003, a group of doctors at Great Ormond Street Hospital in London found themselves watching Formula 1.

They had spent the morning performing heart surgery on children, which is an important detail because what caught their attention wasn't the racing. It was what happened when the racing stopped.

A car came into the pits and was immediately surrounded by mechanics. Different people performed different jobs, some simultaneously and others in a deliberate sequence. Nobody appeared to be negotiating responsibilities. Nobody wandered over to the front-left wheel halfway through and asked whether somebody had taken care of it.

Seconds later, the car was gone.

The doctors had spent their morning doing something obviously much more serious, but there was something familiar in what they were watching. Their patients also had to make a stop after surgery.

And that stop was causing problems.

The Operation Was Over. The Risk Wasn't.

A child coming out of cardiac surgery at Great Ormond Street had to be transferred from the operating theatre to intensive care.

Calling this a “transfer” makes it sound considerably simpler than it was.

The patient could be connected to a ventilator, monitoring equipment, intravenous lines and several drug infusions. During the move, some equipment had to be disconnected from theatre systems, transferred to portable equipment and connected again once the child reached intensive care.

At roughly the same time, something less visible had to move: information.

The ICU team needed to know what had happened during surgery, what medication the child was receiving, whether there had been complications, what the surgeons were concerned about and what needed particular attention during recovery. Somewhere during all of this, responsibility for the patient changed as well. The operating team had owned the case; now the intensive-care team did.

So several highly trained people were moving equipment, information and responsibility around the same vulnerable patient under time pressure.

The individual clinicians knew their jobs.

The transition between them was less reliable.

That was what the doctors recognized while watching Formula 1.

So They Went to See How the Pit Crew Did It

The story is often compressed into a much better headline:

Doctors copied Ferrari pit stops.

It's memorable. It also skips most of what makes the story useful.

The Great Ormond Street team didn't watch a race, return to the hospital on Monday and arrange the nurses around the bed like tyre mechanics. They began investigating how Formula 1 teams actually organized pit stops.

The medical team interacted with people from Formula 1, including Ferrari. Contemporary accounts describe clinicians observing pit-stop practice, and the academic paper that eventually came from the work acknowledges Ferrari technical manager Nigel Stepney for his contribution.

What interested the doctors wasn't how quickly somebody could remove a wheel. They were interested in everything that made that speed possible.

Responsibilities had been decided before the car arrived. People knew where they were supposed to be. Tasks followed a sequence, with some happening in parallel without turning into random parallelism. There was clear coordination, and the same transition had been practiced repeatedly. Crucially, somebody knew when the operation was complete and the car could safely leave.

The doctors compared that with their own handovers.

Several conversations might be happening while equipment was being moved. Information transfer and technical work could overlap. Leadership wasn't always obvious. People interrupted one another because the workflow itself required a surprising amount of live coordination.

Nobody involved was incompetent.

That was precisely the interesting part.

The process was making competent people solve coordination problems while they were already doing critical work.

Ferrari Wasn't the Only Place They Looked

This part tends to disappear from the popular version of the story.

The redesigned handover didn't come purely from Formula 1. The Great Ormond Street team also worked with aviation training specialists.

That makes sense once you stop thinking about industries and start thinking about failure modes.

Aviation had spent decades dealing with highly trained people operating complicated systems where unclear communication, ambiguous responsibility and small human mistakes could become disproportionately expensive. Formula 1 had developed an extraordinary approach to choreographing parallel work around an expensive machine under severe time pressure.

The hospital had to decide which parts of either world made sense around a child recovering from heart surgery.

The result wasn't a Ferrari pit stop transplanted into an ICU. It was a hospital handover protocol informed by Formula 1, aviation and clinical experience.

That distinction matters because cross-industry innovation is often described as copying an idea from somewhere else. In practice, figuring out what not to copy may require more judgment.

At one point, for example, the team considered having clinicians stand in predefined positions around the patient, similar to mechanics around a Formula 1 car.

They didn't keep that part.

Apparently there is a limit to how much Ferrari an ICU requires.

What Actually Changed?

The handover became more structured, but not dramatically more complicated.

Rather than allowing technical work and information exchange to occur as one overlapping activity, the redesigned process separated the transition into clearer phases. Immediate technical work around connecting and stabilizing the patient happened first. Information transfer followed once that activity had settled.

Roles became more explicit, as did leadership. The anesthetist took a coordinating role during the transfer, giving the process a clearer point of control instead of leaving several specialists to independently decide when the transition had reached a safe state.

This is an easy detail to overlook.

The intervention wasn't trying to make clinicians work harder or faster. It was removing decisions that didn't need to be made in the middle of the handover.

Then the team did something that separates this story from the large pile of business anecdotes that survive primarily because they make good conference slides.

They measured it.

Fifty Handovers

Researchers observed 50 postoperative handovers: 23 before the new protocol and 27 after it was introduced.

They recorded technical errors, information omissions, teamwork and the time required for the handover.

Before the intervention, the mean number of technical errors per handover was 5.42. Afterward, it was 3.15.

Information omissions fell from an average of 2.09 to 1.07.

Average handover duration also went from 10.8 minutes to 9.4 minutes.

There was another result I find particularly useful. Before the intervention, 9 of the 23 observed patients experienced more than one technical error and more than one information error during handover. After the intervention, that happened in 3 of 27 cases.

Those numbers need some restraint.

This was a relatively small prospective intervention study. It didn't demonstrate that Formula 1 techniques reduced mortality, and it certainly didn't establish that every hospital should redesign itself around motorsport.

What it showed was narrower and more useful: in this setting, a deliberately redesigned handover reduced measured technical errors and information omissions without making the transfer slower.

That's already interesting enough.

The story doesn't need us to upgrade the evidence.

The Pit Stop Wasn't Really About Speed

Formula 1 makes this story easy to misunderstand because pit stops are famous for being fast.

But speed wasn't the useful thing the hospital borrowed.

A pit stop is fast partly because very little needs to be decided after the car arrives.

Imagine doing it differently. The car stops and a dozen people walk over. Someone asks who wants tyres. Two people reach for the same wheel. Someone discovers nobody brought the replacement front wing.

Eventually the driver asks for an update in Teams.

It would be a terrible pit stop, although a surprisingly recognizable software project.

The choreography removes decisions from the moment when those decisions are most expensive.

That's what makes the hospital story interesting to me. The team wasn't simply standardizing tasks; it was moving coordination work out of the critical path.

Once I started looking at it that way, the connection to software became difficult to ignore.

Not because an API is somehow comparable to a child leaving heart surgery. Obviously it isn't.

But our systems contain plenty of transitions where several facts need to remain true while responsibility moves from one component to another. Those transitions have a habit of producing failures even when the components on either side are behaving correctly.

Payments are particularly good at demonstrating this.

Your Customer Has Been Charged. Your System Thinks They Haven't.

Imagine an order service sending a payment request to a provider.

The provider receives it, processes the transaction and charges the card. Before the response reaches us, the connection dies.

Maybe there was a brief network interruption. Maybe our timeout was slightly aggressive. It doesn't matter yet.

Our service sees:

ETIMEDOUT

The provider sees:

SUCCESS

The payment provider did its job, and our service behaved exactly according to its timeout policy. Unfortunately, the customer has now been charged while our application is preparing to tell them that the payment failed.

This is the sort of failure that makes distributed systems interesting in ways nobody requested.

The obvious response is to retry.

But retry what?

If we send another payment request and the provider treats it as a new transaction, the customer may be charged twice. The retry therefore has to represent the same business operation rather than “please charge this card again.”

Now idempotency matters.

We also need some identifier that both systems understand so we can ask the provider what happened. Perhaps the provider sends a webhook later confirming the transaction. That webhook needs to correlate with the order we created earlier.

Then there is the uncomfortable question of what state our order occupies while we don't know the outcome.

Not FAILED.

We don't know that.

Not PAID.

We don't know that either.

The truthful state might be PAYMENT_PENDING_CONFIRMATION, or whatever equally unattractive name your domain model eventually settles on.

“Unknown” is irritating, but it is a legitimate distributed-systems state. Pretending otherwise doesn't remove the uncertainty.

It usually just moves it into customer support.

The Handoff Is Where Things Become Expensive

Look closely at what failed in that payment example.

Neither participant necessarily malfunctioned.

The trouble appeared while the outcome of an operation was being transferred between them.

That's why the boring mechanisms around integrations matter.

Idempotency keys aren't there because architects enjoy adding headers. Correlation IDs shouldn't exist merely to make logs look enterprise-ready. Status endpoints, webhooks and reconciliation jobs can look like several ways of solving the same problem until a synchronous response disappears after the provider has already changed the real world.

At that point the question becomes very simple:

Who knows what happened, and how do we prove it?

That's surprisingly close to the problem Great Ormond Street was working on.

During a clinical handover, it wasn't enough for the operating team to know what had happened. The receiving team needed the relevant information in a form they could act on while responsibility for the patient changed.

In software we tend to hide all of this behind an arrow:

Order Service → Payment Provider

That arrow looks remarkably confident.

It doesn't show the timeout, the retry or the possibility that the first request actually succeeded. It doesn't show the webhook arriving three minutes later.

And it definitely doesn't show the engineer searching production logs using a transaction ID copied from a customer-support screenshot.

Architecture diagrams are optimistic documents.

Production tends to add the footnotes.

We Spend Much More Time Designing the Boxes

I've noticed this repeatedly in architecture discussions.

We'll spend a considerable amount of time deciding whether something deserves to be its own service. We'll discuss what owns the data, which database it should use, whether it needs a cache and how it will scale.

All reasonable questions.

Then somebody draws an arrow to another service and labels it HTTP.

Done.

Except the arrow represents a protocol between two systems capable of failing independently.

What happens when the call times out? Can we retry safely? What if execution succeeded but the acknowledgement disappeared? How do both sides identify the same business operation? Who owns recovery if the outcome remains uncertain for ten minutes?

And eventually there is the question most diagrams politely avoid: when does the machine stop trying and hand the problem to a human?

Those aren't edge details around the architecture.

They are the handoff design.

A beautifully engineered Order Service and an equally beautiful Payment Service can still disagree about whether somebody has paid.

The boxes aren't always where the interesting failure lives.

The Lesson Isn't “Run Your Team Like Ferrari”

I should probably say this explicitly before someone introduces racing helmets to sprint planning.

Formula 1 isn't a universal operating model, and neither aviation nor a pediatric ICU should be treated as one.

Great Ormond Street didn't do that.

The team had a specific problem: a complicated transition between groups where technical work, information and responsibility all had to move safely.

They found another environment that had become unusually good at coordinating a complicated transition under pressure and studied how it worked. Some ideas transferred and others didn't. Aviation contributed additional thinking. Clinical reality determined what survived.

Then they measured the result.

That sequence is much more valuable than the generic instruction to “look outside your industry.”

Looking outside your industry is easy. Recognizing what you're actually looking at is harder.

A Formula 1 pit stop and a postoperative cardiac handover don't resemble each other if you classify them by industry. On the surface, a racetrack full of mechanics has almost nothing in common with clinicians transferring a child into intensive care.

They become related only when you stop describing the setting and describe the problem.

Both involve something valuable arriving in the middle of a specialist team while several tasks need to happen around it. Some work can happen concurrently, other work depends on a particular sequence, information has to survive the transition, and responsibility eventually needs to become unambiguous.

Once you describe the problem that way, the connection isn't particularly strange.

What's remarkable is that somebody saw it.

Who Saw the Connection?

That's the part of this story that stays with me.

What interests me most isn't Ferrari or even the improvement in the numbers. It's the moment before any of that happened: a group of clinicians watched a pit stop and recognized their own problem inside it.

On the surface, almost everything was wrong for the analogy. One involved mechanics trying to return a racing car to a circuit; the other involved clinicians transferring a child who had just undergone heart surgery into intensive care.

But the coordination problem underneath was familiar.

Once they saw that, they could ask a much better question:

What has Formula 1 learned about this problem that we haven't?

Software engineering could probably ask that question more often.

Our terminology has a habit of making our problems feel unique. Distributed transactions, service boundaries and event-driven workflows sound like subjects that belong exclusively inside architecture meetings.

Other industries won't necessarily use those words. That doesn't mean they haven't spent decades dealing with unreliable handoffs, incomplete information, ambiguous ownership, time pressure and humans coordinating around complicated systems.

Sometimes the useful engineering paper isn't an engineering paper.

Sometimes it's a hospital studying Ferrari.

So the next time I'm staring at an architecture diagram full of carefully designed boxes and suspiciously innocent arrows, I'll probably remember that a group of heart surgeons once watched a Formula 1 pit stop and saw something everyone else was watching straight through.

The interesting question was never what they were watching. It was what they saw.

The Shortest Route Is Sometimes the Slowest: What UPS Can Teach Software Engineers About Optimization

Amrishkhan Sheik Abdullah — Thu, 03 Sep 2026 06:10:00 +0000

Three right turns can be faster than one left turn.

That sentence annoyed me the first time I properly thought about it.

Not because it's difficult to understand. Quite the opposite. It sounds suspiciously like one of those business anecdotes that survives because it's memorable and nobody bothers checking whether it's actually true.

UPS trucks don't turn left.

Great story.

Also not true.

UPS trucks absolutely turn left. What UPS has done for years is try to reduce unnecessary left turns in countries where vehicles drive on the right.

The distinction is important.

A left turn across opposing traffic can leave a truck sitting at an intersection waiting for a gap. Sometimes that's a few seconds. Sometimes it means waiting through a signal cycle. The vehicle is consuming time and fuel while making precisely zero progress, and crossing traffic adds another source of accident risk.

For one truck, on one Tuesday afternoon, this is almost boring.

UPS doesn't operate one truck.

And that's where the arithmetic gets interesting.

The Shortest Route Has a Few Missing Columns

Imagine two routes to the same delivery.

One is 2.8 miles.

The other is 3.1.

If those are the only numbers available, there isn't much of a decision to make. Take the 2.8-mile route.

Now add two awkward intersections to the shorter route. Perhaps the truck needs to cross several lanes of opposing traffic. Perhaps one turn regularly takes 40 seconds during the delivery window.

The extra 0.3 miles no longer looks obviously wasteful.

Nothing about the distance changed. We simply started accounting for more of the cost.

UPS eventually turned this class of problem into ORION: On-Road Integrated Optimization and Navigation.

Calling ORION a route-planning system is accurate, although slightly underselling the problem. UPS needed to calculate useful routes for tens of thousands of drivers making large numbers of stops while respecting delivery commitments, pickup windows, road conditions and the awkward realities of getting a large brown truck to the correct side of a building.

The mathematics gets ugly quickly.

A driver with around 120 stops doesn't have 120 possible routes. The number of possible sequences is astronomically larger. You can't enumerate everything, find the perfect route and go home early.

And even a mathematically attractive route can be operationally ridiculous.

An address might technically be on a road while the receiving dock is around the back. A route that works nicely at 9 AM may become painful when school traffic appears. A particular sequence may force the driver to cross the same busy road repeatedly.

This is what makes ORION more interesting than the left-turn trivia.

UPS wasn't solving geometry.

It was trying to model reality closely enough to make better decisions inside it.

Six Miles Doesn't Sound Like Much

During ORION's rollout, UPS reported reducing routes by roughly six to eight miles in areas where the system had been deployed.

Six miles.

If somebody told you they had spent years developing algorithms, collecting location data, changing driver workflows and deploying a large optimization platform so one vehicle could drive six fewer miles, you might reasonably wonder whether the project had escaped from a PowerPoint presentation.

Then multiply it.

UPS projected that full deployment could eliminate roughly 100 million miles of driving and save around 10 million gallons of fuel annually.

That's the part of the story I like.

There isn't one spectacular optimization hiding somewhere.

One truck saves a little.

Then another does.

Then tens of thousands of trucks do it again tomorrow.

Software engineering has a weakness for dramatic optimization. Replace the database. Rewrite the service in Rust. Introduce a new platform. Split the monolith. Announce an architecture initiative with a name that sounds faintly military.

Those things are occasionally exactly what a system needs.

But high-volume systems also contain much less glamorous opportunities.

Remove a small allocation from a path executed hundreds of millions of times. Avoid serializing data nobody reads. Eliminate one unnecessary network round trip from a hot request. Improve a cache hit rate slightly on something called constantly.

None of these makes an exciting conference keynote.

At sufficient scale, they don't need to.

And this was roughly where the UPS story started bothering the software engineer in me.

We Do This Constantly

Suppose I put two backend designs in front of you.

The first makes three network calls.

The second makes five.

Which one is faster?

Three feels like the sensible answer.

But suppose those three calls are sequential and one goes to an external provider. The five calls are internal, independent and executed concurrently.

Now I haven't given you enough information.

We need to know actual latency, whether connections are reused, failure rates, retry behaviour, rate limits, payload sizes and what those dependencies look like when they're having a bad afternoon.

Counting calls wasn't useless. It just wasn't enough.

Microservice diagrams make this particularly easy to forget because every dependency becomes an arrow.

An API calling Redis and an API calling a third-party payment provider may each occupy roughly the same amount of ink.

Operationally, they're very different dependencies.

The external call crosses infrastructure you don't own. It can time out. It may be rate-limited. Its latency distribution can change without your deployment changing at all. When it fails, retries have to be designed carefully because an unhealthy dependency doesn't usually recover faster when all of its customers enthusiastically send it additional traffic.

“Three calls instead of five” tells me almost nothing about that.

Yet architecture discussions regularly treat these counts as evidence.

We do the same thing with services, queues, database queries and deployment steps because they're visible and easy to count.

Easy to count is not the same thing as expensive.

My Favourite Version Is “We Made It One Query”

Database optimization produces a particularly good version of this mistake.

An endpoint makes six queries.

Someone notices.

“We can make this one query.”

Possibly.

And sometimes that's a very good change.

Other times the result is a magnificent SQL statement with enough joins, CTEs, aggregations and conditional logic that EXPLAIN ANALYZE starts looking like a genealogy project.

Still, six became one.

The dashboard in our heads turns green.

The problem is that databases don't invoice us by query count.

Six small indexed lookups can be perfectly reasonable. One badly planned query over large tables can be disastrous. The opposite can also be true: six network round trips may be unnecessary when the database could efficiently do the work once.

You have to look.

Execution plans matter. Cardinality matters. Index selectivity matters. Data volume matters. Locking matters. Round-trip latency matters. What happens when twenty requests execute the same query concurrently matters quite a lot.

This sounds obvious when written down.

It becomes surprisingly non-obvious during optimization work because reducing a visible number feels like progress.

I've become suspicious of improvements whose strongest argument is that one number got smaller.

Not dismissive. Suspicious.

There is a difference.

The One-Arrow Architecture

The same problem gets more expensive once services start depending on each other.

Service A needs information owned by Service B.

The straightforward implementation is:

Service A → Service B

Done.

One dependency. Easy to explain. Easy to draw.

Then B becomes slow.

Now A becomes slow.

B occasionally becomes unavailable, so A starts failing too. We add retries because optimism is apparently an infrastructure pattern.

The retries help with transient failures until B experiences a larger incident, at which point several callers simultaneously decide that the struggling service would benefit from additional requests.

So we add exponential backoff. Maybe circuit breaking. Perhaps caching. We have to think about timeouts properly. Suddenly our one-arrow architecture contains quite a lot of behaviour that isn't visible in the arrow.

There may be another design.

B publishes relevant changes asynchronously. A consumes them and keeps the small piece of state it needs locally.

The request path becomes independent of B, but we've paid for that independence elsewhere. There is a broker now. Event schemas need ownership. Consumers need monitoring. Replay needs to work. Someone has to decide what eventual consistency means for the business operation rather than merely writing the phrase in an architecture document and hoping everybody finds it reassuring.

This is not automatically better.

If the information must be strongly consistent, the asynchronous design may be wrong. If A requests the data twice a week, it is almost certainly unnecessary.

We should not deploy Kafka every time two services make eye contact.

But there are workloads where the design containing more components produces a simpler runtime failure model.

That's a difficult trade-off to see if “fewer boxes” is being used as a proxy for simplicity.

Some complexity comes from our architecture.

Some complexity comes from the problem.

Deleting the former is useful. Pretending the latter disappeared because we removed a box from a diagram tends to end less well.

ORION Had Another Input: Drivers

One part of the ORION story tends to disappear when it gets reduced to “UPS avoids left turns.”

The drivers already knew a lot.

A driver who has served the same area for years knows things that are difficult to infer from an address database. Which entrance actually works. Where the receiving dock is. Which road becomes unpleasant at a particular hour. Where turning around a large vehicle is technically possible but strategically regrettable.

Meanwhile, an optimization system can evaluate combinations and patterns that no driver could reasonably calculate during a shift.

UPS spent years field-testing ORION rather than simply dropping an algorithm into trucks and declaring mathematics victorious. Its navigation systems became increasingly precise about actual delivery and pickup locations.

That feels familiar.

The first version of an architecture is based on what we know about the system before operating it.

Then production contributes its review comments.

A retry policy behaves differently under a real outage. A cache introduces an invalidation problem nobody cared about in the prototype. A dependency with perfectly acceptable average latency develops a horrible tail. A queue absorbs traffic beautifully until the consumer falls behind for six hours and recovery becomes the real problem.

This doesn't necessarily mean the design was bad.

It means production supplied information the original model didn't contain.

The important part is what happens next.

If measurements disagree with our assumptions, the assumptions don't get seniority.

The Fast Service That Takes Four Seconds

Latency is probably where software engineers encounter this problem most often.

Imagine two services.

The first usually responds in 80 ms but occasionally takes four seconds.

The second tends to respond around 130 ms and rarely exceeds 220 ms.

Which one would you rather put in a checkout path?

There isn't a universal answer, but I know which set of graphs I'd want to inspect before choosing.

Average latency can make the first service look fantastic. Median latency might too.

Customers eventually meet the tail.

It gets worse when requests compose several dependencies. Your endpoint can be individually well behaved while inheriting somebody else's P99, and enough downstream calls give a request several opportunities to encounter an outlier.

This is why I find optimizations focused entirely on averages increasingly difficult to trust.

Sometimes predictability is worth paying a little latency for.

The requirement isn't always “make the fastest request faster.”

Quite often it's “make the bad requests stop being terrible.”

Those are different engineering projects.

So What Are We Actually Optimizing?

This question should probably appear earlier in more performance discussions.

Someone says:

“We need to optimize this.”

Okay.

What is expensive?

CPU? Database time? Network latency? Cloud spend? Failure rate? Human intervention? Customer waiting time?

Those can point toward completely different changes, and improving one can make another worse.

A cache might reduce latency while introducing stale-data behaviour.

Parallel calls can reduce response time while increasing instantaneous pressure on downstream systems.

More replicas improve capacity and resilience while costing money.

A queue can isolate failures but adds another operational system and changes when work becomes visible to the caller.

None of this is a reason to avoid optimization. It's a reason to be precise about what success means before changing the architecture.

UPS could have optimized purely for mileage.

It didn't, because a delivery operation doesn't make money by winning a shortest-distance competition.

Software systems don't either.

Sometimes the Longer Route Is Just Longer

There is a danger in getting too enthusiastic about this idea.

Once engineers discover that the obvious optimization can be wrong, we occasionally become suspicious of anything simple.

That's how a service that needs to store three settings ends up with Kafka, Redis, CQRS and an architectural decision record explaining its “future scalability posture.”

Sometimes the shortest path really is best.

Sometimes one query is better than six.

Sometimes removing a service reduces both latency and operational burden.

And sometimes the left turn is completely empty and you should just turn left.

The point isn't to prefer complicated routes.

It's to stop assuming that distance and cost are interchangeable.

Before optimizing something, I now want to understand three things reasonably well: what we're paying for, how often we're paying for it, and what the proposed improvement makes worse.

That usually gets us much further than starting with “How do we reduce the number of X?”

Especially when X happens to be the easiest thing on the dashboard to count.

The Part Worth Stealing From UPS

The famous version of the UPS story is wonderfully simple:

Avoid left turns.

I don't think that's the useful version.

The useful version is that UPS spent years figuring out what a mile actually costs inside its operation.

Not theoretically. Operationally.

A mile driven at speed isn't equivalent to a mile spent crawling through traffic. A short route with awkward intersections isn't necessarily cheaper than a slightly longer one. A mathematically elegant sequence isn't particularly valuable if the driver can't execute it efficiently.

Once you see the problem that way, the left-turn rule stops looking quirky. It's simply one consequence of having a better understanding of cost.

That's the part worth carrying into software.

We spend a lot of time making systems smaller on paper.

Fewer calls. Fewer queries. Fewer services. Fewer components.

Often that's exactly the right thing to do. Simplicity deserves a strong default.

But the number getting smaller isn't proof that the system got better.

A network call disappears, but perhaps the data now has to be synchronized somewhere else. Six queries become one, but perhaps that query behaves terribly once the table has 200 million rows. A service gets removed, but its responsibilities haven't vanished; they've just moved into another service that nobody wanted to make larger.

The cost didn't disappear because the architecture diagram became prettier.

And I think that's why the UPS story has stuck with me.

It isn't really a story about left turns.

It's about choosing the right thing to optimize.

The shortest route is obvious when you're looking at a map. The better route only becomes obvious when you understand what actually happens while driving it.

UPS figured out that driving a little farther can sometimes get you there faster.

In software, we keep learning the same lesson.
Usually right after deploying the shorter route to production!

Sometimes the Fastest System Is the One Willing to Stop

Amrishkhan Sheik Abdullah — Wed, 02 Sep 2026 07:12:22 +0000

There is a particular kind of meeting every engineer eventually attends.

A metric is projected onto a screen. It has moved in the wrong direction. Someone asks why. Another person suggests we should make sure it never moves in that direction again.

And, somewhere in the organization, a perfectly sensible system is about to be made slightly worse.

Manufacturing has units per hour. Customer support has tickets closed. Software teams have deployment frequency, utilization, velocity and, for reasons historians will struggle to explain, story points.

The attraction is obvious. Movement looks productive.

A factory line that keeps moving feels efficient. A support queue that keeps shrinking looks healthy. A deployment pipeline that keeps shipping looks fast.

Toyota built one of the world's most influential production systems partly by accepting something rather uncomfortable:

sometimes the correct thing for a production line to do is stop.

Decades later, Amazon borrowed that idea.

Except Amazon didn't have a car assembly line to stop.

It had a website.

And that small jump — from stopping defective cars to stopping defective products from being sold online — is far more interesting to me than either story on its own.

Because this isn't really an article about Toyota.

It isn't really about Amazon either.

It's about recognizing when another industry has already solved your engineering problem, only with completely different nouns.

Toyota's strange idea: give people permission to interrupt production

One of the foundations of the Toyota Production System is jidoka, often translated as “automation with a human touch.”

The idea predates Toyota's automobile factories.

Toyota traces it back to Sakichi Toyoda and his automatic looms. One of the important innovations was deceptively simple: when something abnormal happened, such as a thread breaking, the machine could stop rather than continuing to produce defective fabric.

That distinction matters.

A machine can have magnificent uptime while producing absolute rubbish.

If you've worked with distributed systems, you have probably met its descendants.

Toyota carried this principle into automobile manufacturing. Its historical documentation records the adoption of an Andon visual-control system in 1950 and stop-button-linked Andons on the Crown assembly line at the Honsha Plant in 1955.

The Andon made abnormalities visible. Workers could signal for assistance when something wasn't right, and if the issue could not be resolved within the work cycle, production could be stopped rather than allowing the defect to continue downstream.

Toyota's own description of the Toyota Production System still emphasizes this principle: equipment stops when abnormalities are detected, and operators can stop production when necessary so defective products are not simply passed to the next process.

That's quite a bit more radical than it sounds today.

Imagine explaining the idea purely through a utilization dashboard.

“We installed a system that allows employees to interrupt our extremely expensive production machinery.”

Wonderful.

Did Finance approve this?

The internet version of Andon misses the clever bit

The story is often compressed into:

Every Toyota worker can pull a cord and stop the entire factory.

It's memorable. It also makes Toyota sound slightly more theatrical than it actually is.

Toyota's implementation evolved over time, and one particularly interesting design is the fixed-position line-stop system.

When a worker detects an abnormality, the first action isn't necessarily an immediate emergency stop. The Andon signals the problem and alerts the team leader while the vehicle continues toward a designated position.

That creates a small intervention window.

If the team leader reaches the worker and the problem can be resolved within the cycle, production continues. If it can't, the affected line stops at the fixed position.

So the mechanism isn't simply:

problem → STOP EVERYTHING

It's closer to:

abnormality detected
        ↓
make it visible
        ↓
bring help to the problem
        ↓
can we resolve it safely within the cycle?
       / \
     yes  no
      ↓    ↓
continue  stop
           ↓
      resolve cause
           ↓
         resume

I like this much more than the viral version because it's recognizably good systems engineering.

There is detection, escalation, a bounded recovery opportunity and a controlled stop.

More importantly, the abnormality isn't allowed to quietly travel downstream just because stopping would make today's production chart look unpleasant.

A factory can be extremely efficient at producing defects

Consider two imaginary factories.

Factory A runs for an entire shift without interruption. Management sees beautiful utilization numbers.

Unfortunately, several defects occur early in the process and aren't caught until much later. Components have already moved through additional stations. Labor has been added. Other parts have been attached. Some products require rework.

Factory B stops several times during the same shift. Its utilization graph is uglier, but abnormalities are caught close to where they originate.

Which one is more productive?

You can't answer that until you decide where the system boundary is.

If you're measuring minutes the assembly line moved, Factory A looks fantastic.

If you're measuring good products delivered with minimum total waste, the answer may reverse completely.

Software teams make this mistake all the time because local metrics are wonderfully easy to put on dashboards.

A developer closes more tickets. Good.

Deployment frequency increases. Good.

CPU utilization improves. Good.

Except perhaps the tickets are being reopened, the deployments are generating incidents and the service now has no headroom during traffic spikes.

None of those metrics are useless. The problem starts when the metric becomes the objective instead of evidence about the objective.

And defects have an especially nasty property: they become more expensive as they travel.

A bug noticed while writing the code might cost minutes.

The same bug found by a unit test costs a little more.

Found during integration testing, more again.

Found after deployment, now we may have logs, rollback decisions, customer impact, corrupted state, support tickets and several engineers trying to remember who changed what on Tuesday.

By the time a customer discovers it, the bug has acquired colleagues.

Toyota's production philosophy attacks that propagation.

Which brings us to Amazon.

Amazon looked at a car factory and saw a website

In December 2006, Marc Onetto joined Amazon as Senior Vice President of Worldwide Operations. Amazon's SEC filings from the period confirm the role, and Onetto brought extensive manufacturing and operations experience from companies including GE and Solectron.

Soon after arriving, he discussed applying Toyota's Andon concept at Amazon.

He later described the story in an interview with McKinsey.

Jeff Bezos supported the idea.

Amazon implemented it in roughly six months.

But there was an obvious problem.

There was no assembly-line worker standing beside the Amazon product page.

There was, however, someone unusually well positioned to detect defects:

the customer.

And there was another group receiving those defect reports all day:

customer service.

Suppose customers repeatedly contact Amazon because a particular product arrives with the wrong accessory.

A conventional support system is very good at treating these as individual transactions.

Refund the customer. Replace the item. Apologize. Close the ticket.

Then sell another one.

The next customer reports exactly the same defect.

Refund. Replace.

Another customer buys it.

Same problem.

At some point you no longer have a customer-service problem. You have built a defect subscription service.

Amazon changed the feedback loop.

According to Onetto, when customer-service agents identified a repetitive defect, they were empowered to “stop the line” by removing the product from the website until the underlying problem could be investigated and corrected.

Amazon also built a background process around identifying, tracking and resolving those defects.

Onetto reported that the approach eliminated tens of thousands of defects per year.

That claim is worth dwelling on because this story sometimes gets repeated online as management folklore. In this case, we have the Amazon executive responsible for the implementation describing what happened, contemporary corporate records establishing his role, and lean-management organizations documenting the practice.

Amazon really did build a digital Andon.

What exactly did Amazon copy?

This is the part I find most useful as an engineer.

Amazon obviously didn't copy Toyota literally.

There were no ropes hanging beside customer-service desks. Nobody in Seattle needed to understand how to assemble a Corolla.

Instead, somebody stripped away the domain-specific implementation and noticed the underlying structure.

Toyota had something like this:

worker observes abnormality
        ↓
abnormality becomes visible
        ↓
team responds
        ↓
unresolved defect stops flow
        ↓
cause is corrected
        ↓
flow resumes

Amazon could map its own system onto the same structure:

customer experiences abnormality
        ↓
customer service sees the signal
        ↓
repeated defect is identified
        ↓
product availability is stopped
        ↓
cause is investigated/corrected
        ↓
selling resumes

The interesting object isn't the Andon cord.

It's the feedback architecture.

Detection happens near the place where evidence first appears. The signal is made visible. Someone has authority to act on it. Flow can be interrupted before more defective output is created. Then the system has a mechanism for recovery.

Once you describe it that way, the automobile disappears from the idea completely.

Toyota had solved a defect-propagation problem.

Amazon also had a defect-propagation problem.

The physical implementation was almost irrelevant.

This is one of my favourite forms of innovation because it requires less invention and more abstraction.

You look at something that works elsewhere and ask:

Which parts of this solution are accidental details of that industry, and which parts are fundamental?

That's a very engineering way of stealing ideas.

We've already built Andon cords into software

Once I started looking at Andon this way, it became difficult not to see it everywhere in software architecture.

A CI pipeline failing because a test breaks is an Andon-like mechanism.

So is a circuit breaker.

A canary deployment that detects an increase in error rate and automatically halts rollout is probably an even cleaner example.

Suppose we're deploying version v2 across 200 instances.

Instead of replacing all 200:

v1 → v2 everywhere

we deliberately expose a small portion of traffic first:

5% traffic → v2
95% traffic → v1

observe:
  error rate
  p95/p99 latency
  saturation
  business failures

If the error budget starts burning unusually fast, rollout stops.

From a narrow deployment-throughput perspective, this is slower. We've added monitoring periods, thresholds and rollback logic to something Kubernetes could otherwise execute quite enthusiastically.

From a system perspective, we're paying a small interruption cost to limit the blast radius of a defect.

The same reasoning appears in message processing.

If an event fails schema validation, you can reject or quarantine it. Or you can be “resilient,” accept it anyway and discover six services later that one malformed event has been enthusiastically replicated across Kafka topics, databases, caches and analytics pipelines.

The second architecture has excellent throughput right up until everyone opens Slack.

Circuit breakers make the same trade.

When a dependency is failing, continuing to send requests may feel like maintaining availability. In practice, you're often amplifying the failure: consuming threads, sockets, connection pools and retry budgets while increasing load on a service already struggling to recover.

Opening the circuit deliberately refuses work for a period.

Locally, that looks like failure.

Globally, it may be the thing preventing one unhealthy service from dragging five healthy services down with it.

These mechanisms don't exist because engineers dislike throughput.

They exist because uncontrolled propagation is usually more expensive than controlled interruption.

The uncomfortable part isn't stopping. It's authority.

This is where Andon becomes less of a technical pattern and more of an organizational one.

Installing an alert is easy.

Giving the person who sees the alert authority to interrupt something important is considerably harder.

Toyota's worker isn't merely reporting an abnormality into a system where somebody might eventually create an action item.

The signal demands a response.

Amazon's customer-service agents weren't limited to apologizing more elegantly. The system gave frontline employees a path that could result in a product being removed from sale.

That has commercial consequences.

Which is why copying the visible mechanics of systems like Andon usually isn't enough.

An organization can build a beautiful escalation workflow:

Problem detected
      ↓
Ticket created
      ↓
Assigned to team
      ↓
Prioritized next sprint
      ↓
Moved to backlog
      ↓
Renamed "Known Limitation"

JIRA archaeology begins surprisingly quickly.

Marc Onetto later discussed another revealing detail: unresolved Andon issues could eventually reach senior leadership. In an interview published by Planet Lean, he described personally reviewing issues that had remained unresolved for two weeks.

That's the other half of the design.

Escalation has to lead somewhere.

Otherwise, you've built an expensive notification system for problems everybody already knows about.

The pattern I would steal

If I were borrowing the Toyota/Amazon idea for a software platform, I wouldn't create something called an “Andon Service” and start adding Toyota terminology to architecture diagrams.

That's how useful ideas end up becoming internal frameworks with 43 configuration options.

I'd steal the constraints.

For a critical flow, ask:

1. Where is the earliest reliable signal that something is abnormal?

Don't wait for the final customer-facing failure if the system can identify the condition three stages earlier.

2. Can that signal automatically prevent propagation?

Validation, circuit breaking, deployment gates, quarantines and feature flags are all useful here.

3. Is the interruption scoped appropriately?

Toyota's fixed-position approach is instructive. Don't shut down the planet because one request failed. Stop the smallest safe unit of flow.

In distributed systems that could mean a tenant, partition, feature, deployment cohort, consumer group or dependency route.

4. What is the recovery path?

A stop without recovery is just an outage.

Who investigates? What proves the condition is safe again? Is recovery automatic or manual? What state needs reconciliation?

5. Does the organization actually permit the stop?

This one isn't technical.

You can have flawless automated rollback logic, but if every failed deployment threshold gets manually overridden because “the release was promised for Thursday,” your Andon cord is decorative.

That last failure mode is more common than I'd like.

There is also a danger in romanticizing this

“Stop the line” sounds so sensible that it can become another slogan.

Stopping has a cost.

False positives have a cost.

An overly sensitive circuit breaker can create an outage where none existed. Aggressive fraud controls can block legitimate customers. A noisy deployment gate can train engineers to override it. A marketplace that removes products based on weak signals can hurt sellers and customers.

Toyota's system itself is evidence that the answer isn't simply “stop more often.”

The fixed-position line-stop design introduced a bounded opportunity to resolve the issue before stopping became necessary.

That's the sophisticated part.

The design problem is deciding:

When is the confidence and expected downstream cost
high enough that interruption is cheaper than continuation?

That's a threshold problem.

And thresholds need evidence.

In software, I would want hysteresis, minimum sample sizes, error-budget context and carefully chosen recovery criteria rather than:

if errors > 0:
    shut_down_company()

Tempting though that implementation occasionally feels.

The Toyota lesson isn't that stopping is good.

It's that continuing is not automatically good.

Those are very different ideas.

The abstraction is the innovation

What stays with me about the Amazon story isn't really that Amazon learned something from Toyota.

Companies benchmark one another constantly.

The clever move was recognizing what not to copy.

Toyota had factories, assembly workers, Andon boards, production cycles and physical vehicles.

Amazon had warehouses, customer-service agents, web pages, suppliers and orders.

At the surface, they're different systems.

Underneath, both had the same uncomfortable question:

What should happen when we know the output is defective but the system is perfectly capable of continuing anyway?

Toyota's answer was to make abnormality visible and allow production to stop before the defect travelled further.

Amazon translated that principle into e-commerce and gave customer service a way to interrupt the source of repeated defects rather than endlessly cleaning up the consequences.

Software architecture has been rediscovering versions of the same idea ever since.

Circuit breakers.

Canary rollouts.

Automated rollback.

Dead-letter queues.

Schema gates.

Kill switches.

Feature flags.

Different industries. Different vocabulary. Same architecture.

And that's probably the part worth carrying into our own engineering work.

When I'm looking at a difficult systems problem, the useful question may not always be:

“Which new technology solves this?”

Sometimes it's:

“Who has already had this problem in a completely different industry?”

The answer might be buried in manufacturing, aviation, logistics, medicine or finance rather than another engineering blog.

So perhaps before adding another framework to package.json, we should occasionally ask a stranger question:

Have you tried looking at a factory?

CocoIndex vs. code-review-graph: Semantic Search vs. Structural Code Intelligence

Amrishkhan Sheik Abdullah — Wed, 26 Aug 2026 09:31:41 +0000

Your Coding Agent Doesn't Need More Context. It Needs the Right Kind.

AI coding agents have developed an expensive habit.

When they don't understand a codebase, they read more of it.

That works surprisingly well on a 30-file side project.

On a mature repository, it's the software equivalent of solving a library question by reading every book.

The obvious fix is indexing.

Parse the repository once, keep the index fresh, and give the agent only the context relevant to its current task.

Two open-source projects approaching this problem caught my attention recently: CocoIndex Code and code-review-graph.

At first glance they look remarkably similar.

Both parse code with Tree-sitter.

Both update incrementally.

Both integrate with coding agents through MCP.

Both are trying to stop agents from stuffing unnecessary source files into already-expensive context windows.

But after digging into how they work, I don't think they're really solving the same problem.

My mental model is simpler:

CocoIndex is better positioned to answer “Where is the code related to this concept?”

code-review-graph is designed to answer “What does this change affect?”

That distinction becomes important the moment your coding agent graduates from finding code to modifying it.

Here are the official GitHub repositories for both:

CocoIndex Code: github.com/cocoindex-io/cocoindex-code — AST-based semantic code search for coding agents. (GitHub)
code-review-graph: github.com/tirth8205/code-review-graph — structural code intelligence and blast-radius analysis for coding agents. (GitHub)

The Context Window Isn't a Codebase

Suppose I ask an agent:

Find where session expiration is handled.

A naive agent can search filenames, grep for session, inspect likely files and progressively build context.

That's reasonable.

Now ask:

I'm changing validateSession(). What else could this break?

That is a different problem.

Semantic similarity can help locate authentication-related code.

But the most semantically similar function isn't necessarily the function that calls validateSession(), implements the same interface, depends on its return contract, or contains the test that catches the regression.

This is where code intelligence starts splitting into two different retrieval models.

                    CODEBASE
                       │
              ┌────────┴────────┐
              │                 │
              ▼                 ▼
       SEMANTIC VIEW      STRUCTURAL VIEW
              │                 │
        "What looks        "What depends
         relevant?"         on what?"
              │                 │
              ▼                 ▼
         CocoIndex        code-review-graph

That diagram is deliberately simplified. CocoIndex is AST-aware, and code-review-graph also has search capabilities.

The difference is what each system makes first-class.

CocoIndex Treats Code Retrieval as an Indexing Problem

CocoIndex itself is an incremental data-processing engine for AI workloads. Its code-indexing pipeline walks a repository, uses Tree-sitter to split source along syntax boundaries, embeds those chunks and maintains the resulting index as files change. The underlying incremental processing engine is implemented in Rust.

The packaged developer tool is CocoIndex Code, exposed through the ccc CLI.

Its core operation looks refreshingly unsurprising:

ccc index
ccc search "where is session expiration handled?"

Through MCP, the primary search operation accepts a natural-language query or code snippet and returns matching chunks with their file paths, languages, line numbers and similarity scores. It can also filter by language and path.

Architecturally, the important path looks roughly like this:

Repository
    │
    ▼
Tree-sitter
    │
    ▼
AST-aware chunks
    │
    ▼
Embedding model
    │
    ▼
Vector index
    │
    ▼
Semantic query
    │
    ▼
Top-K relevant code
    │
    ▼
Coding agent

This is a very useful abstraction because code search is often conceptual.

I may not know whether the repository calls something:

SessionManager
AuthContext
TokenValidator
IdentityService
AccessPolicy

I only know what I'm trying to find:

Where do we decide whether a user is still authenticated?

Semantic retrieval is excellent at bridging that vocabulary gap.

Traditional grep isn't.

CocoIndex Code also includes a structural ccc grep mode that matches syntax-tree patterns without requiring the embedding index, so the tool isn't exclusively vector retrieval.

Still, semantic retrieval is the center of gravity.

And for exploration, that's a very sensible center.

code-review-graph Starts Somewhere Else

code-review-graph parses the repository with Tree-sitter too.

But instead of primarily turning syntax units into searchable embeddings, it persists the repository as a structural graph.

Its nodes represent things such as:

Files
Functions
Classes
Methods
Tests

and its edges capture relationships such as:

imports
calls
inheritance
test coverage
dependencies

At review time, the system can traverse those relationships to calculate the blast radius of a change.

Conceptually:

                         ┌──────────────┐
                         │ API Handler  │
                         └──────┬───────┘
                                │ calls
                                ▼
                       ┌────────────────┐
                       │ validateSession│
                       └───────┬────────┘
                               │
                   ┌───────────┼────────────┐
                   │           │            │
                called by   tested by    imports
                   │           │            │
                   ▼           ▼            ▼
             AuthGuard    SessionTests   TokenStore
                   │
                   ▼
             Route Group

Now changing validateSession() isn't merely a search query.

It's a graph traversal.

That's a materially different operation.

Similar Code and Dependent Code Are Not the Same Thing

This is the part I think matters most.

Imagine this TypeScript service:

export async function calculateFare(
  itinerary: Itinerary,
  pricingContext: PricingContext
): Promise<Fare> {
  // pricing logic
}

A semantic index queried with:

Find fare calculation logic.

should do very well.

It can retrieve calculateFare, adjacent pricing functions, perhaps tax calculations and discount logic.

Now change the question:

If I modify the return contract of calculateFare, what needs review?

The answer might include code that isn't semantically about fare calculation at all:

BookingController
CheckoutMapper
PaymentRequestBuilder
AncillaryPricingAdapter
AnalyticsPublisher
FareContractTests

AnalyticsPublisher may have almost no semantic resemblance to calculateFare.

But if it consumes the output, I care.

A lot.

That's why I wouldn't frame this comparison as:

vector database vs. graph database.

The more useful distinction is:

similarity retrieval vs. dependency retrieval.

One answers relevance.

The other answers consequence.

CocoIndex Is the Tool I'd Reach for During Exploration

Suppose I've just opened an unfamiliar repository.

My questions are usually fuzzy:

Where is authorization enforced?

How are payment retries implemented?

Where does this service publish events?

Find the logic responsible for refreshing tokens.

I don't yet know the symbols.

I don't know the architecture.

Sometimes I don't even know which package owns the behavior.

This is exactly where semantic search earns its keep.

CocoIndex Code's index is built from syntax-aware chunks rather than arbitrary windows of text, and incremental updates mean changed files can be reprocessed without rebuilding the whole repository index. CocoIndex's official example describes live filesystem indexing where only changed chunks are re-embedded and upserted.

For an agent doing exploration, that produces a clean loop:

Question
   │
   ▼
Semantic search
   │
   ▼
5 relevant chunks
   │
   ▼
Agent reads them
   │
   ├── enough context? ──► reason
   │
   └── not enough? ─────► search again

That is much healthier than:

Read src/
Read services/
Read auth/
Read utils/
Read another 14 files because we're here anyway

The CocoIndex Code repository currently advertises a 70% token-saving figure. I would treat that as a project-reported result, not a universal constant: retrieval efficiency depends heavily on repository shape, query quality, embedding model and what the agent would otherwise have read.

The architecture is more interesting than the headline percentage anyway.

code-review-graph Is the Tool I'd Reach for Before Changing Something Dangerous

Now suppose I've found the code.

The ticket says:

Change session validation so revoked device tokens are rejected immediately.

Finding validateSession() isn't the difficult part anymore.

The difficult questions are:

Who calls it?

Which execution flows pass through it?

What tests exercise those callers?

Is another module depending on behavior that isn't obvious from the function signature?

Does this change cross a community or subsystem boundary?

That's where code-review-graph becomes interesting.

Its detect_changes and review workflows map changed code to affected functions, flows and tests. The project also exposes architecture maps, execution-flow tracing, community detection, refactoring tools and risk-scored reviews.

Instead of:

Changed file
    │
    ▼
Search for similar files

the reasoning becomes:

Changed symbol
      │
      ▼
Direct callers
      │
      ▼
Transitive dependents
      │
      ├──────────► Tests
      │
      ├──────────► Entry points
      │
      └──────────► Cross-module edges
      │
      ▼
Impact radius
      │
      ▼
Agent review context

For code review, I prefer that mental model.

A pull request isn't asking:

What code resembles this diff?

It's asking:

What assumptions did this diff disturb?

The Token Benchmarks Need a Footnote the Size of a Small Service

code-review-graph publishes some dramatic context-reduction numbers.

Its current README reports roughly 82× median per-question reduction across six benchmark repositories when comparing graph query context against reading the entire source corpus, with results ranging from 38× to 528×. The repository itself explicitly warns that the whole-corpus baseline is an upper bound because a competent coding agent wouldn't normally read every source file.

I appreciate that qualification.

The project maintains several benchmarks precisely because “tokens saved” changes meaning depending on what baseline you choose.

Its documentation distinguishes:

whole-repository reading,
a more realistic grep-and-read agent,
changed-file context for reviews,
and complete MCP workflow cost.

That matters enormously.

If I change three lines in a 90-line file, a graph response containing impact edges, source snippets and test relationships could actually be larger than simply handing the changed file to the model.

The project documents this too: its formal review-context benchmark can produce ratios below 1 for small commits because the structural metadata itself has a cost.

That's exactly the kind of boring benchmark detail I trust more than another giant “500×” badge.

Context optimization isn't:

graph = fewer tokens.

It's:

graph = spend tokens describing relationships when those relationships are worth more than their serialization cost.

At 10× Repository Size, the Difference Gets More Interesting

Take a repository that grows from 300 files to 3,000.

Then 30,000.

Semantic retrieval has a pleasant property: the agent can still ask for the top few relevant chunks.

The corpus becomes larger, but the final context doesn't necessarily grow linearly.

That's good.

A structural graph has a different scaling challenge.

More code creates more:

nodes
edges
communities
call paths
cross-module dependencies
test relationships

But that additional structure is precisely what becomes valuable in large systems.

code-review-graph reports incremental updates based on changed-file hashes and graph relationships. Its current documentation describes a roughly 3,000-file Django repository where a two-file edit re-indexes in about 2.5 seconds on the hook path, with a substantial portion of that being process startup.

CocoIndex attacks the same freshness problem from the indexing side: only changed chunks need to be reprocessed and re-embedded rather than rebuilding the corpus.

So both systems understand the same production truth:

A code index that becomes stale every time somebody presses Save isn't code intelligence. It's documentation with impressive latency.

Incrementality isn't an optimization here.

It's part of correctness.

Freshness Is a Consistency Problem

This deserves more attention.

Imagine an agent retrieves an index saying:

PaymentService → LegacyFraudClient

but three minutes ago another developer changed the code to:

PaymentService → RiskGateway

The index is now lying.

That can be worse than having no index because the agent doesn't know its context is stale.

Both tools address this through incremental updates.

CocoIndex Code can refresh the semantic index and its agent integration can keep indexing current as the repository changes.

code-review-graph offers watch mode and hooks that update the structural graph after file changes or commits.

Architecturally, I think this should be treated like cache consistency:

Source code
    │
    │ change
    ▼
Change detector
    │
    ├────────► Semantic index update
    │
    └────────► Graph update
                    │
                    ▼
              Agent query

The agent should never have to wonder whether yesterday's architecture is answering today's question.

Where CocoIndex Starts Hurting

Semantic search is not dependency analysis.

That's the boundary I'd keep in mind.

If I ask:

What code is related to payment retries?

CocoIndex is a natural fit.

If I ask:

Can changing this retry function alter checkout behavior for callers that never mention retries?

semantic similarity alone isn't enough.

You can compensate by retrieving more context, searching symbols, letting the agent inspect imports and iterating.

But at some point you're asking the LLM to reconstruct a dependency graph on demand.

That's expensive.

And slightly absurd when a parser can build one deterministically.

The other operational cost is embeddings.

CocoIndex Code supports local SentenceTransformers or cloud embedding providers through LiteLLM. Its full local installation brings heavier dependencies, while the slim installation expects a cloud embedding provider.

That's not a criticism.

Semantic retrieval requires a semantic representation somewhere.

But it means your architecture has another dimension:

Local embeddings
    ├── privacy friendly
    ├── offline capable
    └── model/runtime footprint

Cloud embeddings
    ├── lighter local setup
    ├── potentially stronger model choice
    └── code/privacy/network boundary

For enterprise repositories, that trust boundary isn't a footnote.

It's architecture.

Where code-review-graph Starts Hurting

Graphs aren't free either.

A static graph can tell me:

A calls B
B imports C
D tests B

But production software contains relationships that static analysis struggles to prove.

Dependency injection.

Runtime reflection.

Dynamic imports.

Framework magic.

Configuration-selected implementations.

Message brokers.

Database-driven workflows.

HTTP calls whose relationship exists in configuration rather than source syntax.

A TypeScript service can publish:

eventBus.publish("payment.completed", payload);

while the consumer lives in another repository.

Your local AST doesn't magically know that relationship.

A graph can only be as correct as the relationships it can observe or infer.

This is why I would never interpret blast-radius analysis as:

These are all the things this change can affect.

I would interpret it as:

These are the dependencies the graph can prove or reasonably model.

That's still extremely useful.

It's just not omniscience.

Security Changes the Recommendation

For sensitive repositories, both tools have an attractive property: they can operate locally.

code-review-graph stores its core graph in a local SQLite file under .code-review-graph/, and its documentation warns that exported graph data can contain absolute paths and structural metadata that should be sanitized before publishing.

CocoIndex Code can use local embeddings, meaning source doesn't need to leave the machine for embedding generation. It also supports cloud providers when teams prefer them.

If I were deploying either in a regulated environment, I'd still review:

Source access
     │
     ├── Parser process
     ├── Embedding provider
     ├── Local index
     ├── MCP transport
     ├── Agent process
     └── Export / telemetry behavior

“Local-first” is a useful property.

It isn't a substitute for a data-flow review.

I Wouldn't Actually Choose One for Every Workflow

After comparing them, I think asking:

CocoIndex or code-review-graph?

is slightly the wrong architecture question.

For an agent that spends most of its time discovering implementations, I'd start with CocoIndex Code.

For an agent heavily involved in PR review, refactoring and impact analysis, I'd start with code-review-graph.

For a serious engineering agent, the architecture I find most interesting is actually:

                         Developer Question
                                │
                                ▼
                         Coding Agent
                                │
              ┌─────────────────┴─────────────────┐
              │                                   │
              ▼                                   ▼
     Semantic Retrieval                  Structural Retrieval
              │                                   │
         CocoIndex                         Code Graph
              │                                   │
       "What code is                       "What code is
        relevant?"                          connected?"
              │                                   │
              └─────────────────┬─────────────────┘
                                ▼
                       Context Composer
                                │
                                ▼
                       Minimal useful context
                                │
                                ▼
                         Agent reasoning

Now we have two independent signals.

Semantic similarity gives us meaning.

Graph traversal gives us relationships.

Those signals can disagree.

That's useful.

If semantic search says a function is relevant but the graph shows no dependency path from the changed code, maybe it's conceptual context rather than blast radius.

If the graph says a distant analytics module is affected but semantic similarity ranks it near zero, that's exactly the kind of non-obvious dependency I want surfaced during review.

The disagreement is information.

The Best Retrieval System Might Be a Router

I wouldn't blindly query both systems for every prompt.

That's how an optimization becomes another token bill.

I'd route based on intent.

"Where is X implemented?"
            │
            ▼
       Semantic Search


"What calls X?"
            │
            ▼
       Graph Traversal


"What breaks if I change X?"
            │
            ▼
       Graph + Tests


"How does authentication work?"
            │
            ▼
   Semantic + Execution Flow


"Find code similar to this pattern"
            │
            ▼
   Semantic / Structural Search


"Review this PR"
            │
            ▼
   Diff → Graph Blast Radius
            │
            ▼
   Semantic Search for Missing Context

This is the architecture I'd want in an agent.

Not one enormous context engine.

A context router that knows what kind of evidence the current question requires.

Because “relevant” is not one thing.

When I'd Choose Something Simpler

Neither tool is automatically necessary.

If your repository has 80 files, clear module boundaries and predictable naming, rg, language-server references and an agent that knows how to search may already be enough.

Adding embeddings, an indexing daemon, MCP configuration and persistent graph state just to save 900 tokens is infrastructure cosplay.

I'd introduce code intelligence when I can identify an actual retrieval problem:

agents repeatedly reading irrelevant files,
large monorepos exhausting context,
developers struggling to discover implementations,
PR reviews missing transitive impact,
cross-module refactors becoming dangerous,
onboarding requiring archaeological expeditions through the repository.

The architecture should pay rent.

Otherwise, grep remains one of the best pieces of developer infrastructure ever shipped.

What I'd Deploy

If my primary problem were agent exploration, I'd deploy CocoIndex Code first.

Its semantic model maps naturally to the fuzzy questions developers ask when they don't yet understand a repository.

If my primary problem were change safety, I'd deploy code-review-graph first.

Call relationships, tests and blast-radius analysis are closer to the evidence I want before modifying production behavior.

For a mature platform with heavy AI-assisted development, I'd eventually want both capabilities behind one retrieval layer.

Something like:

                    ┌─────────────────────┐
                    │     Agent Query     │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │   Intent Classifier │
                    └──────────┬──────────┘
                               │
             ┌─────────────────┼─────────────────┐
             │                 │                 │
             ▼                 ▼                 ▼
         Semantic          Structural          Hybrid
          Search             Graph              Query
             │                 │                 │
             └─────────────────┼─────────────────┘
                               ▼
                    ┌─────────────────────┐
                    │ Context Budgeter    │
                    │ rank · dedupe · cap │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │   Coding Agent      │
                    └─────────────────────┘

The interesting engineering problem then stops being indexing.

It becomes context selection.

Which evidence deserves 500 tokens?

Which dependency edge is important enough to include?

When should semantic similarity override structural distance?

When should the agent expand another hop?

When does the retrieval cost exceed the cost of simply reading the file?

Those are the questions I expect code-agent infrastructure to spend a lot more time solving.

Because a larger context window doesn't eliminate retrieval architecture.

It just makes bad retrieval more expensive before anyone notices.

CocoIndex and code-review-graph approach the problem from different directions.

One asks:

What code means something similar to what you're looking for?

The other asks:

What code is structurally connected to what you're touching?

For exploration, meaning often wins.

For change safety, relationships often win.

And for serious codebase intelligence, I don't think the future is choosing between them.

It's knowing which kind of context to ask for before the agent starts reading.

Ponytail: Teaching AI Coding Agents the Most Expensive Senior Engineering Skill — Restraint

Amrishkhan Sheik Abdullah — Wed, 26 Aug 2026 06:19:41 +0000

Give an AI coding agent a small problem and, occasionally, you'll get the software equivalent of hiring a construction crew to hang a picture frame.

Need a date input?

The agent can absolutely build one.

A component. State management. Validation. Styling. A date-picker dependency. Maybe a wrapper around that dependency because apparently we have architectural ambitions now.

Meanwhile, the browser has been sitting quietly in the corner with:

<input type="date">

This is one of the more interesting failure modes of AI-assisted development.

The models have become remarkably capable at writing code.

They aren't always equally good at deciding whether the code should exist.

That distinction is exactly why Ponytail caught my attention.

Ponytail is an open-source ruleset/plugin by Dietrich Gebert that describes itself as putting a "lazy senior developer" inside your AI coding agent. It works across a surprisingly broad set of coding-agent environments, including Claude Code, Codex, GitHub Copilot CLI, Gemini CLI, OpenCode and several instruction-file-based integrations.

The joke is good.

The engineering idea underneath it is better.

Ponytail is essentially trying to teach an AI agent something experienced engineers eventually learn the expensive way:

Every line you don't write is a line you don't have to debug, secure, test, review, upgrade and explain three years later.

AI Agents Have an Over-Engineering Problem

The problem isn't that AI-generated code is always bad.

That would actually be easier.

The uncomfortable problem is that the code can be perfectly reasonable while the decision to create it was unnecessary.

Ask for a small feature and the agent has an enormous solution space available:

Requirement
    │
    ▼
┌─────────────────────────────┐
│       AI Coding Agent       │
└─────────────────────────────┘
    │
    ├── Create abstraction
    ├── Add dependency
    ├── Create helper
    ├── Create component
    ├── Add configuration
    ├── Introduce interface
    └── Write custom solution

Most of those options can produce valid software.

That's precisely the problem.

A compiler can tell you whether your TypeScript is valid. A test can tell you whether calculatePrice() returned the expected value.

Neither tells you:

Why did we create AbstractPriceCalculationStrategyFactory in the first place?

Senior engineering is full of these negative decisions.

Don't create the service.

Don't introduce Kafka yet.

Don't add Redis because one query is slow.

Don't write a custom retry framework when the client already supports retries.

Don't introduce another abstraction because two functions happen to contain four similar lines.

And please don't install 38 KB of JavaScript because HTML already solved the problem.

Ponytail tries to move that decision before code generation.

That ordering matters.

The Seven-Rung Ladder Is the Real Product

Strip away the branding and Ponytail's core mechanism is remarkably small.

Before generating a solution, the agent walks through a hierarchy:

The actual rules add an important qualification: the ladder runs after the agent understands the problem. Ponytail tells the agent to read the affected code and trace the real flow before choosing the smallest solution. It also explicitly favors fixing a shared root cause over scattering symptom patches across callers.

That's an important distinction.

Otherwise "write less code" becomes code golf wearing an architecture badge.

Ponytail isn't really optimizing for minimum LOC.

It's optimizing for minimum unnecessary ownership.

Those are very different objectives.

The Best Rung Might Be Rung Zero

The first question is the one engineering teams routinely skip:

Does this need to exist?

This is YAGNI, but AI makes YAGNI more important than it used to be.

Historically, unnecessary software had friction.

Someone had to design it. Someone had to type it. Someone had to get tired halfway through implementing it.

AI has dramatically reduced that friction.

A developer can now generate an abstraction, tests, DTOs, documentation and adapters before their coffee has reached a drinkable temperature.

That's useful when the abstraction is necessary.

It's dangerous when generation cost gets confused with ownership cost.

Suppose an agent can generate 500 lines in 30 seconds.

Those 500 lines still enter your system.

They still participate in:

Generated code
     │
     ├──► Code review
     │
     ├──► Tests
     │
     ├──► Security surface
     │
     ├──► Dependency upgrades
     │
     ├──► Refactoring
     │
     ├──► Debugging
     │
     ├──► Observability
     │
     └──► Future developer comprehension

Generation became cheap.

Maintenance didn't.

That's the architectural consequence I find more interesting than Ponytail itself.

As coding agents become faster, the scarce engineering resource shifts from implementation capacity toward judgment.

We don't need agents merely capable of producing more software.

We need agents capable of refusing to produce software when the existing system already contains the answer.

Reuse Before Reinvention Is Harder Than It Sounds

The second rung asks whether the codebase already contains the solution.

This sounds obvious.It isn't.

In a mature backend, the same conceptual operation may already exist behind:

a shared utility,
an internal SDK,
middleware,
a domain service,
a framework extension,
an infrastructure adapter,
or an implementation whose name doesn't quite match the new ticket.

An agent working too locally can easily generate:

function normalizeEmail(email: string): string {
  return email.trim().toLowerCase();
}

Perfectly harmless.

Until normalizeUserEmail() already exists three directories away with additional Unicode handling required by the system.

Now we don't have one simple function.

We have two definitions of what a normalized email means.

That's how small duplication becomes domain drift.

Ponytail's rules explicitly instruct the agent to inspect the code it touches and trace callers when fixing bugs. The smallest correct diff may therefore involve changing a shared function rather than inserting a guard into the exact path named by the ticket.

That is much closer to how an experienced engineer approaches maintenance work.

The ticket tells you where somebody observed the problem.

It doesn't necessarily tell you where the problem lives.

Native Platform Features Are Criminally Underrated

This is where Ponytail's benchmark gets interesting.

The project tested the skill using real headless Claude Code sessions against a pinned version of the open-source FastAPI + React full-stack template. Twelve feature tasks were run four times per test arm using Haiku 4.5. Instead of counting prose emitted by the model, the newer benchmark measures added lines left in the actual git diff.

The biggest reductions appeared where the platform already provided the feature.

For the date-picker task, the baseline averaged 404 added lines. Ponytail averaged 23.

For the color picker: 287 versus 23.

For file upload/drop behavior: 251 versus 95.

Meanwhile, straightforward backend tasks such as searching by title were effectively identical across approaches: 44 lines for the baseline and 44 for Ponytail.

That last result matters.

If Ponytail simply forced everything into absurd one-liners, I'd consider it a fun prompt rather than an engineering tool.

Instead, the benchmark suggests its largest effect occurs where there is genuinely something unnecessary to remove.

Conceptually:

                 ROOM TO OVER-ENGINEER
                         ▲
                         │
 Native UI feature       │ ███████████████████
 Custom UI component     │ █████████████████
                         │
 File handling           │ ███████████
                         │
 CRUD endpoint           │ ██
                         │
 Simple query            │ █
                         └────────────────────────►
                           Potential Ponytail Gain

The less irreducible code a task requires, the more useful restraint becomes.

That's exactly where I'd expect a senior engineer to save time too.

Not by typing the same architecture faster.

By noticing that half the architecture isn't necessary.

Less Code Is Not Automatically Better Code

This is where any "lazy developer" philosophy can become dangerous.

Take an untrusted filename:

const path = join(uploadDirectory, userProvidedFilename);

Beautiful.

Short.

Potentially terrible.

If the filename contains traversal sequences, brevity just bought you a security bug.

Ponytail's rules explicitly exclude trust-boundary validation, security, accessibility and error handling that prevents data loss from simplification. Non-trivial logic is also expected to leave behind a small runnable check.

The project's safety benchmark is small, so I wouldn't turn it into a universal security claim. The maintainers don't either.

Across five security-oriented tasks with four runs per arm, Ponytail's outputs passed all 20 adversarial checks. A simple "YAGNI + prefer one-liners" control passed 19/20; one generated path-handling implementation allowed directory traversal. The benchmark itself explicitly says these deterministic checks are a floor, not proof that generated code is secure.

That's the correct interpretation.

There is a boundary between:

             DELETE / SIMPLIFY
                    │
                    ▼
        ┌───────────────────────┐
        │ Boilerplate           │
        │ Duplicate abstraction │
        │ Reinvented helpers    │
        │ Unneeded dependencies │
        │ Speculative features  │
        └───────────────────────┘


               DO NOT CUT
                    │
                    ▼
        ┌───────────────────────┐
        │ Trust validation      │
        │ Authorization         │
        │ Data-loss protection  │
        │ Required error paths  │
        │ Accessibility         │
        └───────────────────────┘

The difference between minimal engineering and careless engineering is knowing which side of that line you're standing on.

The Benchmark Story Is Better Because the First One Was Wrong

This part deserves more attention than the headline percentages.

Ponytail originally reported much larger reductions from a single-shot benchmark: roughly 80–94% less code.

That benchmark was criticized because the baseline model produced conversational prose and multiple options, while the measurement counted lines of the whole response. In other words, some of the apparent reduction wasn't less implementation. It was simply less talking.

The maintainer agreed with the criticism.

Then rebuilt the benchmark around actual coding-agent sessions, fresh repository copies, isolated contexts and git diff output. During that process, they also found a contamination bug where Ponytail's lifecycle hook had accidentally fired in baseline runs, making the baseline secretly use Ponytail. That run was discarded and the harness was fixed to isolate each test arm.

I like this considerably more than pretending the original number never existed.

The corrected feature-task aggregate was:

Approach	LOC vs. Baseline	Tokens	Cost	Time	Safety Check
Ponytail	-54%	-22%	-20%	-27%	100%
Terse "caveman" control	-20%	+7%	+3%	+2%	100%
YAGNI + one-liner prompt	-33%	-14%	-21%	-30%	95%

These are project-reported results from one model, one repository family and a relatively small task set, not evidence that installing Ponytail will cut every engineering team's LOC bill in half. The project's own limitations call out the single-model test, n=4 nondeterminism, limited security tests and several timed-out benchmark cells.

That's how I'd read the numbers.

Not:

"Ponytail makes AI coding 54% better."

Rather:

"There is measurable evidence that explicitly teaching an agent to search for simpler solution classes can materially reduce over-building on tasks where simpler solution classes exist."

Less exciting headline.

Much more useful engineering claim.

Why Not Just Put "Use YAGNI" in Your Prompt?

That was my first question.

We don't exactly need another repository to tell a language model:

Keep the solution simple.
Follow YAGNI.
Prefer existing functionality.

The benchmark tests roughly that idea with a short "YAGNI + one-liners" control.

Sometimes it works extremely well. On the color-picker task it came close to Ponytail.

Other times it didn't. The date-picker implementation averaged 162 lines against Ponytail's 23, and the command-palette result was actually larger than the baseline.

This points to the useful part of packaging engineering behavior as an agent skill.

A casual instruction is a preference.

A repeated decision ladder, injected into the working context and supported by review/audit commands, is closer to a policy.

Ponytail exposes intensity levels (lite, full, ultra, off) plus commands for reviewing a current diff, auditing a repository, tracking deferred shortcuts and inspecting benchmark gains. Its default mode is full. Depending on the host, the rules can be injected through plugins/hooks or loaded through files such as AGENTS.md.

That gives the idea persistence.

And persistence matters with coding agents because every fresh context is another opportunity for your architectural preferences to mysteriously develop amnesia.

What I'd Actually Use Ponytail For

I wouldn't treat Ponytail as an architecture oracle.

That's not what it is.

I'd use it as a counterweight.

AI coding agents have a natural advantage in construction. Give them a sufficiently precise requirement and they can generate implementation machinery very quickly.

Ponytail introduces pressure in the opposite direction:

                    USER REQUIREMENT
                           │
                           ▼
                 ┌───────────────────┐
                 │ Understand Flow   │
                 └─────────┬─────────┘
                           │
                           ▼
                 ┌───────────────────┐
                 │ Ponytail Ladder   │
                 │ "Can we do less?" │
                 └─────────┬─────────┘
                           │
               ┌───────────┴───────────┐
               ▼                       ▼
       Existing capability      New code required
               │                       │
               ▼                       ▼
          Reuse / native         Minimum correct
             solution              implementation
               │                       │
               └───────────┬───────────┘
                           ▼
                 ┌───────────────────┐
                 │ Safety / Tests    │
                 └─────────┬─────────┘
                           ▼
                        DIFF

I especially like the idea for:

mature repositories with established utilities and conventions,
frontend work where native browser capabilities are easily overlooked,
maintenance tickets where agents tend to widen the change unnecessarily,
teams using agents heavily enough that generated-code volume is becoming a review burden,
and repositories where dependency creep has started becoming noticeable.

The /ponytail-review concept is arguably as interesting as generation itself.

Generating unnecessary code is one problem.

Finding the unnecessary code already sitting in your diff before it becomes permanent is another.

Where I'd Turn It Down

Minimalism isn't universally correct.

There are situations where writing more code today deliberately buys optionality, isolation or operational safety tomorrow.

A payment integration may justify an adapter around a provider SDK because provider substitution and contract isolation are real architectural requirements.

A distributed workflow may need explicit idempotency state even though the happy-path implementation works perfectly without it.

A service boundary can be justified by deployment ownership and security isolation even when keeping everything in one process would require fewer lines.

And sometimes a boring abstraction is valuable because twelve teams need to obey the same contract.

This is where I'd be careful with ultra mode.

The smallest local diff and the lowest long-term system cost are not guaranteed to be the same thing.

Consider:

                 LOCAL OPTIMUM
                 ─────────────
                 20-line patch
                       │
                       ▼
              Lowest code today


                 SYSTEM OPTIMUM
                 ──────────────
                 60-line adapter
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
      Provider isolation    Stable contract
             │                   │
             └─────────┬─────────┘
                       ▼
              Lower change cost
                 over 3 years

Ponytail's own rules partially protect against this by requiring the agent to understand the real flow first and by allowing deliberate simplifications to document their known ceiling and upgrade path.

But the tool cannot know your future business constraints unless those constraints are represented in the codebase, instructions or task context.

No prompt can.

That's still our job.

At 10× Scale, LOC Probably Isn't What Breaks First

There's another reason I wouldn't turn minimalism into dogma.

When traffic jumps by an order of magnitude, the first failure usually isn't:

We have too many classes.

It's something operational.

Connection-pool exhaustion.

Queue depth.

Lock contention.

A downstream rate limit.

An unbounded retry loop.

Memory pressure.

Hot partitions.

Timeout budgets that looked generous until every upstream service consumed all of them simultaneously.

Ponytail can reduce unnecessary implementation surface, but it doesn't remove the need for production reasoning.

In fact, minimal code makes that reasoning more important, not less.

If you replace a custom subsystem with a native or standard-library capability, you need to understand that capability's operational ceiling.

The right question isn't:

Which solution has fewer lines?

It's:

Which solution is the smallest one that still satisfies the system's actual invariants?

That's a much harder optimization target.

And a much better one.

Installing It Is Appropriately Uneventful

The project supports multiple agent environments. For example, GitHub Copilot CLI currently uses:

copilot plugin marketplace add DietrichGebert/ponytail
copilot plugin install ponytail@ponytail

Claude Code uses its plugin marketplace flow, Codex has a plugin installation path, Gemini CLI can install the repository as an extension, and several editor agents can consume the provided rules files directly. Check the repository's current installation section before installing because agent plugin interfaces are still moving quickly.

Once active, the useful commands include:

/ponytail lite
/ponytail full
/ponytail ultra
/ponytail off

/ponytail-review
/ponytail-audit
/ponytail-debt
/ponytail-gain

I would start with full.

ultra sounds entertaining.

Production has cured me of selecting modes based on how entertaining their names are.

The Bigger Idea Isn't Ponytail

Ponytail is a small project built around a small idea.

That's why I think it's worth paying attention to.

We've spent much of the AI coding conversation measuring what models can generate: bigger features, longer autonomous runs, more tool calls, larger repository changes.

But increasing generation capacity changes the economics of software development.

When producing code becomes nearly free, saying no to code becomes more valuable.

The next generation of useful agent tooling may therefore look less like:

"Here are more tools the agent can use."

and more like:

"Here is the engineering judgment that constrains when
the agent should use them."

Ponytail's ladder is one version of that.

Does the feature need to exist?

Does the solution already exist?

Can the language do it?

Can the platform do it?

Can something we already depend on do it?

Only then do we start creating new machinery.

That's not revolutionary architecture advice.

It's almost aggressively boring.

Which is exactly why it resembles senior engineering.

The codebase doesn't care how impressive the generated solution looked in the terminal.

Six months later, it only cares how much software we decided to own.

And the cheapest component to operate, patch, secure, observe and eventually delete remains the one we had enough judgment not to build.

Article Metadata

Recommended title: Ponytail: Teaching AI Coding Agents the Most Expensive Senior Engineering Skill — Restraint

Direct technical: How Ponytail Reduces Over-Engineering in AI Coding Agents

Opinionated: Your AI Coding Agent Writes Too Much Code. Ponytail Has a Point.

Curiosity-driven: What Happens When You Teach an AI Coding Agent to Be Lazy?

Meta description: Ponytail teaches AI coding agents to prefer reuse, native features and smaller diffs. Here's what its architecture, benchmarks and trade-offs reveal.

Canonical slug: ponytail-ai-coding-agent-over-engineering

OpenGraph summary: AI agents are getting better at writing code. Ponytail asks a more valuable question: should that code exist at all? A production-minded look at AI-assisted minimalism.

Primary topics: AI coding agents, Ponytail, YAGNI, software architecture, code quality, developer productivity

References

Ponytail — Official Site
DietrichGebert/ponytail — GitHub
Ponytail AGENTS.md
Ponytail Agentic Benchmark, June 18, 2026

Fix Next.js "params should be awaited" Error in Next.js 15+

Amrishkhan Sheik Abdullah — Thu, 20 Aug 2026 06:16:41 +0000

Fix Next.js "params should be awaited" Error in Next.js 15+

If you are seeing the params should be awaited Next.js error after upgrading to Next.js 15 or following an older App Router tutorial, you are not alone.

The error usually looks something like this:

Route "/blog/[slug]" used params.slug. params should be awaited before using its properties.

Sometimes it appears with searchParams.

Sometimes it appears with cookies() or headers().

And sometimes the page still seems to work, but your terminal keeps shouting at you.

This article will slow it down and explain the fix in a beginner-friendly way.

No deep framework lecture first. Just the actual problem, the broken code, the fixed code, and the reason it works.

What This Error Means in Plain English

In older Next.js code, you may have treated params like a normal JavaScript object.

Something like this:

const slug = params.slug;

That used to feel natural.

If your route was:

/blog/[slug]

and the user opened:

/blog/my-first-post

you expected:

params.slug; // "my-first-post"

In newer Next.js versions, especially Next.js 15+, some request-based values became asynchronous. That means you should treat them like values that need to be waited for before you read from them.

So instead of reading params.slug directly, you do this:

const { slug } = await params;

That is the heart of the fix.

The error is not saying your route is missing.

It is not saying your [slug] folder is wrong.

It is saying:

You are trying to read route data before awaiting it.

The common flow: the page loads, the code reads params.slug directly, Next.js expects params to be awaited, and the error appears.

Why This Changed

Next.js has a group of features called Dynamic APIs.

That sounds more complicated than it is.

In simple terms, Dynamic APIs are values that depend on the current request.

For example:

What route did the user open?
What query string is in the URL?
What cookies came with this request?
What headers came with this request?
Is draft mode enabled?

Those values are not the same for every user.

They depend on the incoming request, so Next.js treats them differently from static code.

In Next.js 15+, these Dynamic APIs are asynchronous:

params
searchParams
cookies()
headers()
draftMode()

That is why old examples from Next.js 13 or 14 can suddenly throw warnings or errors after an upgrade.

The same async idea applies to more than just params. If it depends on the request, check whether it needs await.

The Broken Dynamic Route Example

Let's start with the most common case: a blog post page.

You have this route:

app/blog/[slug]/page.js

The [slug] part means the route is dynamic.

So these URLs can all use the same file:

/blog/nextjs-routing
/blog/react-state
/blog/my-first-post

In older tutorials, you may see code like this:

export default function BlogPostPage({ params }) {
  const slug = params.slug;

  return <h1>Blog post: {slug}</h1>;
}

This is the broken pattern in Next.js 15+.

The problem is this line:

const slug = params.slug;

You are reading slug directly from params.

But Next.js expects you to wait for params first.

The Fixed Dynamic Route Example

Here is the fixed version:

export default async function BlogPostPage({ params }) {
  const { slug } = await params;

  return <h1>Blog post: {slug}</h1>;
}

Two things changed:

The component became async.
We used await params before reading slug.

That is it.

Small code change, big difference.

Before: read params.slug directly. After: make the function async, await params, then read slug.

Why Adding `async` Matters

This part is important for beginners.

You cannot use await inside a normal function.

This will not work:

export default function BlogPostPage({ params }) {
  const { slug } = await params;

  return <h1>{slug}</h1>;
}

JavaScript will complain because await needs an async function.

So you need this:

export default async function BlogPostPage({ params }) {
  const { slug } = await params;

  return <h1>{slug}</h1>;
}

Think of async as telling JavaScript:

This function may need to wait for something.

And await means:

Wait here until this value is ready, then continue.

So the pattern is:

async function Page({ params }) {
  const { slug } = await params;
}

Do Not Await `params.slug`

This is a very common mistake.

You might try this:

export default async function BlogPostPage({ params }) {
  const slug = await params.slug;

  return <h1>{slug}</h1>;
}

That looks close, but it is not the right fix.

You are still trying to access .slug before awaiting params.

The correct order is:

const { slug } = await params;

Wait for the whole params object first.

Then read slug.

Fixing `generateMetadata`

This error often appears on blog sites because the page is not the only place where you read params.

You may also use params inside generateMetadata.

For example, this is broken:

export function generateMetadata({ params }) {
  const slug = params.slug;

  return {
    title: `Post: ${slug}`,
  };
}

The fixed version is:

export async function generateMetadata({ params }) {
  const { slug } = await params;

  return {
    title: `Post: ${slug}`,
  };
}

This matters because a lot of developers fix page.js, refresh the app, and still see the same warning.

Then they feel stuck.

The reason is simple:

The same broken params.slug access may also exist in generateMetadata.

So when you fix a dynamic route, search the whole file for:

params.

Do not only check the page component.

Fixing `searchParams`

Now let's talk about searchParams.

params usually comes from the path.

Example:

/blog/[slug]

searchParams comes from the query string.

Example:

/products?page=2&sort=new

In older code, you might write:

export default function ProductsPage({ searchParams }) {
  const page = searchParams.page || "1";

  return <p>Page: {page}</p>;
}

In Next.js 15+, use the async pattern:

export default async function ProductsPage({ searchParams }) {
  const { page = "1", sort = "new" } = await searchParams;

  return (
    <div>
      <p>Page: {page}</p>
      <p>Sort: {sort}</p>
    </div>
  );
}

The idea is the same.

Do not read from searchParams first.

Await it first.

Then read from it.

Fixing Both `params` and `searchParams` Together

Sometimes a page needs both.

Example:

/category/shoes?page=2

Your route might be:

app/category/[slug]/page.js

Here is the fixed version:

export default async function CategoryPage({ params, searchParams }) {
  const { slug } = await params;
  const { page = "1" } = await searchParams;

  return (
    <main>
      <h1>Category: {slug}</h1>
      <p>Current page: {page}</p>
    </main>
  );
}

You can await them separately.

That keeps the code easy to read.

Fixing `cookies()`

The same migration affects cookies() from next/headers.

Here is the broken version:

import { cookies } from "next/headers";

export default function DashboardPage() {
  const token = cookies().get("token");

  return <pre>{JSON.stringify(token, null, 2)}</pre>;
}

The fixed version:

import { cookies } from "next/headers";

export default async function DashboardPage() {
  const cookieStore = await cookies();
  const token = cookieStore.get("token");

  return <pre>{JSON.stringify(token, null, 2)}</pre>;
}

Again, the order matters.

Do not do this:

const token = await cookies().get("token");

Do this:

const cookieStore = await cookies();
const token = cookieStore.get("token");

You wait for the cookie store first.

Then you read from it.

Fixing `headers()`

headers() follows the same idea.

Broken:

import { headers } from "next/headers";

export default function Page() {
  const userAgent = headers().get("user-agent");

  return <p>{userAgent}</p>;
}

Fixed:

import { headers } from "next/headers";

export default async function Page() {
  const headerStore = await headers();
  const userAgent = headerStore.get("user-agent");

  return <p>{userAgent}</p>;
}

If you remember the cookie example, this should feel familiar.

Await the store.

Then read from the store.

Route Handler Example

This error can also show up in route handlers.

For example:

app/api/users/[id]/route.js

Broken:

export async function GET(request, { params }) {
  const id = params.id;

  return Response.json({
    userId: id,
  });
}

Fixed:

export async function GET(request, { params }) {
  const { id } = await params;

  return Response.json({
    userId: id,
  });
}

The file is different, but the fix is the same.

If the handler receives params, await it before reading id, slug, or any other dynamic value.

TypeScript Version

If you use TypeScript, update the type too.

Broken:

type PageProps = {
  params: {
    slug: string;
  };
};

export default function BlogPostPage({ params }: PageProps) {
  return <h1>{params.slug}</h1>;
}

Fixed:

type PageProps = {
  params: Promise<{
    slug: string;
  }>;
};

export default async function BlogPostPage({ params }: PageProps) {
  const { slug } = await params;

  return <h1>{slug}</h1>;
}

This is where many build errors come from.

The code may look logically correct, but the type still says params is a plain object.

In Next.js 15+, type it like a Promise when you are using this async pattern.

The Codemod Can Help

Next.js provides a codemod for this migration.

A codemod is a script that updates code automatically.

You can run:

npx @next/codemod@latest next-async-request-api .

This can update many params, searchParams, cookies(), and headers() usages for you.

But do not run it and assume everything is perfect.

After the codemod, review the changed files.

Pay extra attention to:

custom TypeScript types
helper functions
auth utilities
Supabase or other server clients
generateMetadata
route handlers

The codemod is useful, but your app still needs a human review.

Common Mistakes

Mistake 1: Awaiting the Property Instead of `params`

Avoid this:

const slug = await params.slug;

Use this:

const { slug } = await params;

Await the object first.

Then read the property.

Mistake 2: Forgetting `async`

Avoid this:

export default function Page({ params }) {
  const { slug } = await params;
}

Use this:

export default async function Page({ params }) {
  const { slug } = await params;
}

If there is await, the function needs async.

Mistake 3: Fixing the Page but Forgetting Metadata

You may fix this:

export default async function Page({ params }) {
  const { slug } = await params;
}

But forget this:

export function generateMetadata({ params }) {
  const title = params.slug;
}

If the warning still appears, search the file for:

params.

Mistake 4: Copying Old Tutorial Code

This one is common.

You follow a tutorial.

The tutorial uses:

params.slug

Your app uses Next.js 15 or newer.

Now you get an error.

That does not mean the tutorial is completely useless. It may just be written for an older version of Next.js.

Update the request-data parts to the newer async pattern.

Mistake 5: Thinking the Route Folder Is Wrong

When beginners see this error, they sometimes rename folders or move files around.

Usually, that is not needed.

If your route is:

app/blog/[slug]/page.js

that folder structure is fine.

The issue is usually inside the code:

params.slug

not the route folder itself.

Beginner Debugging Checklist

Use this when the error will not go away.

A quick checklist for finding the common places where this error hides.

Ask these questions:

Is this a dynamic route like [slug], [id], or [category]?
Am I reading params.slug, params.id, or another property directly?
Did I make the function async?
Did I use const { slug } = await params?
Did I also check generateMetadata?
Am I using searchParams?
Am I using cookies() or headers()?
Do my TypeScript types say params is a Promise?
Did I run the codemod and review the result?

Most of the time, one of these checks will reveal the issue.

Quick Fix Cheatsheet

Here are the patterns to remember.

For params:

const { slug } = await params;

For searchParams:

const { page = "1" } = await searchParams;

For cookies():

const cookieStore = await cookies();
const token = cookieStore.get("token");

For headers():

const headerStore = await headers();
const userAgent = headerStore.get("user-agent");

For TypeScript:

type PageProps = {
  params: Promise<{ slug: string }>;
};

How This Relates to Other Next.js Bugs

This error is part of a bigger pattern in Next.js.

Some bugs happen because code runs in a different place than you expected.

For example, in my previous article on fixing Next.js hydration errors with localStorage, the problem was server HTML not matching the first client render:

Server render and browser render did not match.

In the middleware redirect loop article, the problem was request flow:

/dashboard redirects to /login
/login also triggers middleware
the app loops

This article is also about flow.

But this time, the flow is about when request data is ready:

Wait for request data
then read from it

Once you see it that way, the error becomes much less scary.

Final Takeaway

The Next.js params should be awaited error usually means you are reading request-based data too early.

In older code, this looked normal:

const slug = params.slug;

In Next.js 15+, use this:

const { slug } = await params;

And remember the simple rule:

Treat route and request data as async. Await it first, then read from it.

That rule applies to:

params
searchParams
cookies()
headers()
draftMode()

If you are upgrading a project, run the codemod, then review your dynamic routes, metadata functions, route handlers, and TypeScript types.

I write practical debugging guides and build full-stack products. You can find more of my work at amrishkhan.dev.

References

Next.js docs: Dynamic APIs are Asynchronous
Next.js docs: Upgrading to Version 15
Next.js docs: cookies
Next.js docs: Dynamic Segments

Fix Next.js Middleware Redirect Loop: Why Your Login Page Keeps Redirecting

Amrishkhan Sheik Abdullah — Wed, 19 Aug 2026 07:14:44 +0000

This is the third article in Real Coding Problems, Simple Fixes.

This one is about a problem that can waste hours because the app looks like it is doing the right thing:

Protect private pages and redirect logged-out users to /login.

But then the browser shows:

ERR_TOO_MANY_REDIRECTS

or your Next.js app keeps sending users back to the login page again and again.

That is usually a Next.js middleware redirect loop.

The cause is often simple:

Your middleware is protecting the login page too.

Let's fix it with clear code.

What We Are Fixing

If you are dealing with this bug, it usually looks like one of these problems:

Next.js middleware redirect loop
Next.js login page keeps redirecting
ERR_TOO_MANY_REDIRECTS Next.js
NextAuth middleware redirect loop
Next.js middleware matcher exclude login page
Next.js protected routes middleware

We will use a simple auth-token example first, then talk about NextAuth/Auth.js after.

The Real Problem

Imagine your app has these routes:

/login
/dashboard
/settings
/api/auth/login

You want this behavior:

Logged out user opens /dashboard
Redirect to /login

Logged out user opens /login
Show the login page

Logged in user opens /dashboard
Show the dashboard

That sounds straightforward.

So you write middleware to check whether the user has an auth token.

The Broken Middleware

Here is a common broken version:

import { NextResponse } from "next/server";

export function middleware(request) {
  const token = request.cookies.get("auth_token")?.value;

  if (!token) {
    return NextResponse.redirect(new URL("/login", request.url));
  }

  return NextResponse.next();
}

This looks reasonable at first.

If there is no token, send the user to /login.

But there is a hidden problem.

This middleware runs for more routes than you think.

So the request flow becomes:

User opens /dashboard
No token found
Redirect to /login

Browser opens /login
Middleware runs again
Still no token found
Redirect to /login again

Browser opens /login again
Middleware runs again
Still no token found
Redirect to /login again

That is the loop.

The login page can never load because the middleware keeps redirecting it to itself.

The Plain-English Explanation

Middleware runs before the page is rendered.

That makes it useful for auth checks, redirects, headers, rewrites, and request decisions.

But it also means you need to be specific.

If your middleware says:

Every logged-out request must go to /login

then /login itself also becomes a logged-out request.

So the app gets stuck.

The fix is not complicated:

Do not run auth protection on public routes.

Your public routes usually include:

/login
/register
/forgot-password
/api/auth
static files
images
favicon
auth callback routes

Fix 1: Add Public Route Checks

The first fix is to define which routes are public.

import { NextResponse } from "next/server";

const publicRoutes = ["/login", "/register", "/forgot-password"];

function isPublicRoute(pathname) {
  return publicRoutes.some((route) => pathname.startsWith(route));
}

export function middleware(request) {
  const { pathname } = request.nextUrl;
  const token = request.cookies.get("auth_token")?.value;

  if (isPublicRoute(pathname)) {
    return NextResponse.next();
  }

  if (!token) {
    const loginUrl = new URL("/login", request.url);
    loginUrl.searchParams.set("from", pathname);

    return NextResponse.redirect(loginUrl);
  }

  return NextResponse.next();
}

Now the request flow is different:

User opens /dashboard
No token found
Redirect to /login?from=/dashboard

Browser opens /login
/login is public
Show login page

That small public route check breaks the loop.

Fix 2: Use `matcher` to Protect Only Private Routes

Another good fix is to tell Next.js exactly where the middleware should run.

If only /dashboard and /settings need protection, the middleware should not run everywhere.

import { NextResponse } from "next/server";

export function middleware(request) {
  const token = request.cookies.get("auth_token")?.value;

  if (!token) {
    const loginUrl = new URL("/login", request.url);
    loginUrl.searchParams.set("from", request.nextUrl.pathname);

    return NextResponse.redirect(loginUrl);
  }

  return NextResponse.next();
}

export const config = {
  matcher: ["/dashboard/:path*", "/settings/:path*"],
};

This is often cleaner.

With this setup:

/dashboard      -> middleware runs
/dashboard/team -> middleware runs
/settings       -> middleware runs
/login          -> middleware does not run
/api/auth/login -> middleware does not run

For many apps, this is the simplest and safest solution.

Which Fix Should You Use?

Use matcher when you have a clear list of private route groups.

Example:

/dashboard
/account
/admin
/settings

Use public route checks when your app has more complex rules.

Example:

Some pages are public
Some pages are private
Some pages depend on role
Some pages depend on onboarding status

You can also combine both:

use matcher to avoid static files and unrelated routes
use route checks inside middleware for auth-specific decisions

Fix 3: Avoid Redirecting Logged-In Users Back to Login

There is another common issue.

After login, the user visits /login again and still sees the login page.

Usually, logged-in users should be redirected away from auth pages.

You can handle that like this:

import { NextResponse } from "next/server";

const authRoutes = ["/login", "/register"];
const privateRoutes = ["/dashboard", "/settings"];

function matchesRoute(pathname, routes) {
  return routes.some((route) => pathname.startsWith(route));
}

export function middleware(request) {
  const { pathname } = request.nextUrl;
  const token = request.cookies.get("auth_token")?.value;

  const isAuthRoute = matchesRoute(pathname, authRoutes);
  const isPrivateRoute = matchesRoute(pathname, privateRoutes);

  if (isAuthRoute && token) {
    return NextResponse.redirect(new URL("/dashboard", request.url));
  }

  if (isPrivateRoute && !token) {
    const loginUrl = new URL("/login", request.url);
    loginUrl.searchParams.set("from", pathname);

    return NextResponse.redirect(loginUrl);
  }

  return NextResponse.next();
}

Now the rules are clear:

Logged-out user on /dashboard -> send to /login
Logged-out user on /login -> allow
Logged-in user on /login -> send to /dashboard
Logged-in user on /dashboard -> allow

That is the mental model you want.

Fix 4: Exclude API Routes and Static Files

If your middleware runs too broadly, it can also touch routes it should ignore.

For example:

/_next/static
/_next/image
/favicon.ico
/api/auth/login
/api/auth/callback

You usually do not want your page-auth middleware interfering with those.

Here is a common matcher pattern:

export const config = {
  matcher: [
    "/((?!api|_next/static|_next/image|favicon.ico).*)",
  ],
};

This tells middleware to run on most pages, but skip API routes and common Next.js asset paths.

Be careful with this pattern though.

If your login page is still inside that matched group, you still need a public route check inside middleware.

That is why I prefer being explicit for beginner-friendly auth:

export const config = {
  matcher: ["/dashboard/:path*", "/settings/:path*", "/account/:path*"],
};

It is easier to reason about.

NextAuth/Auth.js Redirect Loop Note

If you use NextAuth or Auth.js, redirect loops usually come from one of these problems:

Your custom sign-in page is also protected by middleware.
Your middleware does not know about your custom signIn page.
Your callback or auth API route is being matched accidentally.
The user is redirected after login to a route that immediately fails another middleware rule.

For a custom sign-in page, make sure your auth config and middleware agree about the sign-in route.

Example idea:

export const authOptions = {
  pages: {
    signIn: "/login",
  },
};

And make sure /login is not treated like a protected page.

If your app uses a matcher like this:

export const config = {
  matcher: ["/dashboard/:path*"],
};

then /login will not be protected by that middleware.

That alone fixes many redirect loops.

Common Mistakes

Mistake 1: Matching Every Route

This is risky:

export const config = {
  matcher: ["/:path*"],
};

It can work, but only if your internal route logic is careful.

If you are new to middleware, start by matching only private sections.

Mistake 2: Redirecting to the Same URL

Before redirecting, think:

Am I already on the page I am redirecting to?

If the user is already on /login, do not redirect them to /login.

Mistake 3: Protecting Auth Callback Routes

Auth libraries often need callback routes to finish login.

If middleware blocks those routes, login may succeed with the provider but fail when the app receives the callback.

Watch routes like:

/api/auth/callback
/api/auth/session
/auth/callback

The exact paths depend on your auth library.

Mistake 4: Forgetting Role-Based Redirect Loops

Redirect loops are not only about login.

They can also happen with roles.

Example:

User opens /admin
Middleware says user is not admin
Redirect to /dashboard

Middleware checks /dashboard
Onboarding is incomplete
Redirect to /onboarding

Middleware checks /onboarding
Some rule sends user back to /dashboard

The fix is the same:

Make every redirect rule clear, and make sure the destination route is allowed for that user state.

A Simple Debugging Checklist

When your Next.js login page keeps redirecting, check this:

Does middleware run on /login?
Does middleware run on /api/auth?
Does your matcher include too many routes?
Are you redirecting to the same route you are already on?
Does your custom sign-in page match your auth config?
Are callback routes excluded?
After login, are you redirecting to a route the user is actually allowed to access?

If you are stuck, add temporary logging:

export function middleware(request) {
  console.log("middleware path:", request.nextUrl.pathname);

  return NextResponse.next();
}

Then reload the page and watch which routes middleware is touching.

The path list usually reveals the problem quickly.

The Mental Model

Middleware redirect bugs become easier when you think in flows.

Do not only ask:

Is the user logged in?

Also ask:

Which route is the user trying to open?
Is that route public or private?
If I redirect, will the destination route be allowed?
Will middleware run again on the destination route?

That last question is the important one.

Because every redirect creates another request.

And middleware can run again on that new request.

Final Takeaway

A Next.js middleware redirect loop usually happens because the middleware is too broad.

Most commonly:

/dashboard redirects to /login
/login also runs middleware
/login redirects to /login again

The practical fix:

Protect only private routes
+ Keep login/register pages public
+ Exclude auth API and callback routes
+ Use matcher carefully
+ Never redirect a route to itself

Once your routes are separated into public, auth-only, and private groups, the loop becomes much easier to avoid.

Publishing SEO Notes

SEO title:
Fix Next.js Middleware Redirect Loop: Protect Routes Without Breaking Login

Meta description:
Learn why a Next.js middleware redirect loop happens, why login pages get accidentally protected, and how to fix protected routes with matcher and public route checks.

Suggested slug:
fix-nextjs-middleware-redirect-loop-protected-routes

Primary keyword:
Next.js middleware redirect loop

Secondary keywords:
Next.js auth redirect loop
Next.js login page keeps redirecting
ERR_TOO_MANY_REDIRECTS Next.js
NextAuth middleware redirect loop
Next.js middleware matcher exclude login page
Next.js protected routes middleware

These notes are for publishing. You can keep them out of the visible article body if your CMS already has dedicated SEO fields.

References

Next.js docs: Middleware
Next.js docs: File-system convention: middleware
NextAuth docs: Next.js middleware configuration
Next.js docs: Redirecting

The Next.js Hydration Error Everyone Hits and How to Actually Fix It

Amrishkhan Sheik Abdullah — Tue, 18 Aug 2026 05:21:30 +0000

This is the second article in Real Coding Problems, Simple Fixes.

The first article was about a React search box showing old results because API responses came back in the wrong order.

This one is a little harder.

We are going to talk about one of the most confusing errors people hit when moving from normal React to Next.js:

Hydration failed because the server rendered HTML didn't match the client.

That error sounds serious. It also sounds like React is angry about something deep inside the framework.

Most of the time, the problem is much simpler:

The server rendered one version of the page, but the browser rendered a different version when React started.

Let's break that down with a real example.

The Real Problem

Imagine you are building a dashboard.

The dashboard has a theme toggle. If the user selected dark mode earlier, you store that choice in localStorage.

That feels normal:

theme = "dark"

So you write a component that reads the saved theme and renders the button text.

"use client";

import { useState } from "react";

export default function ThemeToggle() {
  const [theme, setTheme] = useState(() => {
    return localStorage.getItem("theme") || "light";
  });

  function toggleTheme() {
    const nextTheme = theme === "light" ? "dark" : "light";
    setTheme(nextTheme);
    localStorage.setItem("theme", nextTheme);
  }

  return (
    <button onClick={toggleTheme}>
      Current theme: {theme}
    </button>
  );
}

If you are coming from a client-only React app, this may look fine.

But in Next.js, this can break.

You may see:

ReferenceError: localStorage is not defined

or:

Hydration failed because the server rendered HTML didn't match the client.

The confusing part is that the file already says:

"use client";

So why is this still a problem?

Why `"use client"` Does Not Mean "Only Runs in the Browser"

This is the part that trips up a lot of developers.

In Next.js App Router, "use client" means the component is allowed to use browser-side React features like:

useState
useEffect
event handlers
browser interactions after the page loads

But the first HTML can still be prepared before the browser fully takes over.

That means your first render must be safe for both worlds:

the server
the browser

The server does not have:

window
document
localStorage
sessionStorage
matchMedia

Those belong to the browser.

So if your render logic depends on them too early, Next.js can end up with different output on the server and the client.

That difference is what causes the hydration error.

What Hydration Means in Plain English

Think of Next.js like a restaurant preparing your table before you arrive.

The server sends ready-made HTML so the page appears quickly.

Then React loads in the browser and attaches all the interactive behavior:

clicks
state
event handlers
updates

That process is hydration.

For hydration to work cleanly, React expects the first browser render to match the HTML that came from the server.

If the server rendered:

<button>Current theme: light</button>

but the browser immediately renders:

<button>Current theme: dark</button>

React sees a mismatch.

That is the error.

The Bad Pattern

Here is the common mistake:

const [theme, setTheme] = useState(() => {
  return localStorage.getItem("theme") || "light";
});

This reads localStorage while React is calculating the initial render.

That is too early.

The server cannot read localStorage, and even if you add checks like this:

const savedTheme =
  typeof window !== "undefined"
    ? localStorage.getItem("theme")
    : "light";

you can still create different server and client output.

The server may render light.

The browser may instantly render dark.

Now the first HTML does not match.

Fix 1: Start With a Safe Default, Then Read Browser Storage

The safest beginner-friendly fix is:

Render the same default value on the server and first client render.
Read localStorage inside useEffect.
Update the UI after the component has mounted.

Here is the fixed version.

"use client";

import { useEffect, useState } from "react";

export default function ThemeToggle() {
  const [theme, setTheme] = useState("light");
  const [isReady, setIsReady] = useState(false);

  useEffect(() => {
    const savedTheme = localStorage.getItem("theme");

    if (savedTheme === "light" || savedTheme === "dark") {
      setTheme(savedTheme);
    }

    setIsReady(true);
  }, []);

  function toggleTheme() {
    const nextTheme = theme === "light" ? "dark" : "light";
    setTheme(nextTheme);
    localStorage.setItem("theme", nextTheme);
  }

  if (!isReady) {
    return <button disabled>Loading theme...</button>;
  }

  return (
    <button onClick={toggleTheme}>
      Current theme: {theme}
    </button>
  );
}

Now the first render is predictable.

The component starts with:

const [theme, setTheme] = useState("light");

Then, after the browser is ready, useEffect reads localStorage.

Why does this work?

Because useEffect only runs in the browser after React has rendered.

So the server is no longer asked to read something it does not have.

Fix 2: Create a Reusable `useMounted` Hook

If you do this in multiple components, repeating isReady everywhere gets boring.

You can create a small hook:

import { useEffect, useState } from "react";

export function useMounted() {
  const [mounted, setMounted] = useState(false);

  useEffect(() => {
    setMounted(true);
  }, []);

  return mounted;
}

Then use it like this:

"use client";

import { useEffect, useState } from "react";
import { useMounted } from "./useMounted";

export default function ThemeToggle() {
  const mounted = useMounted();
  const [theme, setTheme] = useState("light");

  useEffect(() => {
    if (!mounted) return;

    const savedTheme = localStorage.getItem("theme");

    if (savedTheme === "light" || savedTheme === "dark") {
      setTheme(savedTheme);
    }
  }, [mounted]);

  function toggleTheme() {
    const nextTheme = theme === "light" ? "dark" : "light";
    setTheme(nextTheme);
    localStorage.setItem("theme", nextTheme);
  }

  if (!mounted) {
    return <button disabled>Loading theme...</button>;
  }

  return (
    <button onClick={toggleTheme}>
      Current theme: {theme}
    </button>
  );
}

The hook makes the intent clear:

Do not show browser-dependent UI until the component is mounted in the browser.

This is useful for theme, cart count, persisted filters, saved sidebar state, or anything else stored in the browser.

Fix 3: Build a Small `useLocalStorageValue` Hook

For a real project, I prefer making storage access explicit and reusable.

Here is a simple hook:

import { useEffect, useState } from "react";

export function useLocalStorageValue(key, defaultValue) {
  const [value, setValue] = useState(defaultValue);
  const [ready, setReady] = useState(false);

  useEffect(() => {
    const storedValue = localStorage.getItem(key);

    if (storedValue !== null) {
      setValue(storedValue);
    }

    setReady(true);
  }, [key]);

  function updateValue(nextValue) {
    setValue(nextValue);
    localStorage.setItem(key, nextValue);
  }

  return { value, setValue: updateValue, ready };
}

Now the component becomes smaller:

"use client";

import { useLocalStorageValue } from "./useLocalStorageValue";

export default function ThemeToggle() {
  const {
    value: theme,
    setValue: setTheme,
    ready,
  } = useLocalStorageValue("theme", "light");

  function toggleTheme() {
    const nextTheme = theme === "light" ? "dark" : "light";
    setTheme(nextTheme);
  }

  if (!ready) {
    return <button disabled>Loading theme...</button>;
  }

  return (
    <button onClick={toggleTheme}>
      Current theme: {theme}
    </button>
  );
}

This is not a perfect hook for every storage case. For example, it only stores strings.

But the pattern is the important part:

use a safe default
read browser storage after mount
avoid changing the first render unexpectedly

Fix 4: Use `dynamic` With `ssr: false` for Browser-Only Widgets

Sometimes a component is completely browser-only.

Examples:

a chart library that needs window
a map widget
a rich text editor
a component that depends heavily on browser APIs

For those cases, you can tell Next.js not to server-render that component.

"use client";

import dynamic from "next/dynamic";

const BrowserOnlyChart = dynamic(() => import("./BrowserOnlyChart"), {
  ssr: false,
});

export default function AnalyticsPage() {
  return (
    <main>
      <h1>Analytics</h1>
      <BrowserOnlyChart />
    </main>
  );
}

This can be the right fix when the component simply cannot produce meaningful HTML on the server.

In the App Router, keep this dynamic usage inside a Client Component file. If you try to use ssr: false directly inside a Server Component, Next.js will complain.

But do not use it everywhere.

If you disable server rendering for too much of your page, you lose some of the benefits of Next.js:

faster first HTML
better SEO for meaningful content
less useful server-rendered output

Use it when the component is truly browser-only.

What About `suppressHydrationWarning`?

Next.js also gives you an escape hatch:

<span suppressHydrationWarning>
  {new Date().toLocaleTimeString()}
</span>

This tells React:

I know this content may be different. Do not warn me for this specific element.

This can be fine for small unavoidable differences, like a timestamp.

But it should not be your first fix for normal app state.

If your cart count, theme, auth UI, or dashboard content is mismatching, hiding the warning does not fix the actual problem. It only makes the warning quieter.

Most of the time, you should fix the render flow instead.

A Simple Debugging Checklist

When you see a hydration error, ask these questions:

Am I reading window, document, or localStorage during render?
Am I rendering Date.now() or Math.random() directly in JSX?
Does the server render one value while the browser immediately renders another?
Is my first render dependent on auth, theme, cart, or browser storage?
Is a third-party component using browser APIs before mount?
Is my HTML valid, or do I have tags nested incorrectly?

The goal is simple:

The server HTML and the first client render should match.

After that first render, React can update the UI normally.

The Mental Model

Here is the easiest way to remember it:

Server render:
Should not depend on browser-only data.

First client render:
Should match what the server rendered.

After mount:
Safe to read localStorage, window, document, and other browser APIs.

If you keep that order in your head, hydration errors become much less mysterious.

Final Takeaway

Hydration errors are not random.

They usually happen because your first browser render does not match the HTML that came from the server.

For localStorage, the practical fix is:

Start with a safe default
+ Read localStorage inside useEffect
+ Render browser-only UI after mount
+ Use dynamic imports only for truly browser-only components
+ Avoid suppressHydrationWarning unless you really mean it

Once you understand that, the error message becomes less scary.

It is just React saying:

I expected the first client render to match the server HTML, but it did not.

And now you know where to look.

References

Next.js docs: Text content does not match server-rendered HTML
React docs: hydrateRoot
React docs: useEffect
Next.js docs: Lazy loading Client Components and libraries

The React Search Bug That Shows Old Results and How to Fix It

Amrishkhan Sheik Abdullah — Mon, 17 Aug 2026 13:18:00 +0000

This is part of a series I like to think of as Real Coding Problems, Simple Fixes.

No fancy architecture. No interview puzzle. Just the kind of bug that shows up while building a normal app.

Today we are fixing a very common search bug:

The user types a new search term, but the UI sometimes shows results from an older search.

If you have built a search box, autocomplete field, customer lookup, product filter, user picker, or admin dashboard table, you have probably been close to this bug.

It usually does not happen every time. That is what makes it annoying. On your machine, with fast internet, everything may look fine. Then someone uses the app on a slower network and suddenly the search results feel haunted.

Let's make that bug visible, then fix it properly.

The Real Problem

Imagine you are building a customer lookup for a support dashboard.

The support agent starts typing a customer name:

a
am
amy

Your React component sends a request for each value:

GET /api/customers?search=a
GET /api/customers?search=am
GET /api/customers?search=amy

You probably expect the responses to come back in the same order.

But the internet does not promise that.

The request for amy might finish first. Then the older request for a might finish last. If your component blindly updates state whenever any response returns, the screen can end up showing results for a even though the input says amy.

That is the bug.

In simple words:

The latest thing the user typed is not always the latest response your app receives.

The Broken Version

Here is a small version of the problem.

import { useEffect, useState } from "react";

export default function CustomerSearch() {
  const [query, setQuery] = useState("");
  const [customers, setCustomers] = useState([]);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState("");

  useEffect(() => {
    if (!query.trim()) {
      setCustomers([]);
      return;
    }

    async function searchCustomers() {
      setLoading(true);
      setError("");

      try {
        const response = await fetch(`/api/customers?search=${query}`);
        const data = await response.json();
        setCustomers(data.customers);
      } catch (err) {
        setError("Could not load customers.");
      } finally {
        setLoading(false);
      }
    }

    searchCustomers();
  }, [query]);

  return (
    <section>
      <label htmlFor="customer-search">Search customers</label>
      <input
        id="customer-search"
        value={query}
        onChange={(event) => setQuery(event.target.value)}
        placeholder="Type a customer name..."
      />

      {loading && <p>Searching...</p>}
      {error && <p>{error}</p>}

      <ul>
        {customers.map((customer) => (
          <li key={customer.id}>{customer.name}</li>
        ))}
      </ul>
    </section>
  );
}

At first glance, this looks reasonable.

When query changes, we fetch customers. When the response arrives, we put the customers into state.

The hidden problem is that every request is allowed to update the same customers state, even if that request is old news by the time it finishes.

Why This Happens

React is not doing anything weird here.

The bug comes from timing.

Let's say the user types quickly:

Search 1: "a"   -> slow response
Search 2: "am"  -> medium response
Search 3: "amy" -> fast response

The response order might be:

"amy" returns first
"am" returns second
"a" returns last

If all three responses call setCustomers, the oldest search can win just because it finished last.

That kind of bug is called a race condition.

You do not need to memorize the term. Just remember the practical version:

Multiple async tasks are trying to update the same screen, and the wrong one finishes last.

First Fix: Debounce the Input

Before we cancel old requests, we should also avoid creating too many requests.

If the user types a, am, amy quickly, we probably do not need to search after every single keypress. We can wait until the user pauses for a moment.

That is called debouncing.

Here is a small reusable hook:

import { useEffect, useState } from "react";

export function useDebounce(value, delay = 400) {
  const [debouncedValue, setDebouncedValue] = useState(value);

  useEffect(() => {
    const timerId = setTimeout(() => {
      setDebouncedValue(value);
    }, delay);

    return () => {
      clearTimeout(timerId);
    };
  }, [value, delay]);

  return debouncedValue;
}

Now React will wait 400 milliseconds after the latest change before updating debouncedValue.

That means this:

a
am
amy

can become one actual search request instead of three.

But there is an important detail:

Debounce reduces the number of requests. It does not guarantee old requests can never update your UI.

If a request has already started, debouncing will not magically cancel it.

So debounce is helpful, but it is not the complete fix.

The Proper Fix: Cancel the Old Request

The browser gives us a built-in way to cancel a fetch request: AbortController.

The idea is simple:

Create an AbortController for the current request.
Pass its signal to fetch.
When the effect cleans up, call abort().
Ignore the abort error because it is expected.

Here is the fixed component.

import { useEffect, useState } from "react";
import { useDebounce } from "./useDebounce";

export default function CustomerSearch() {
  const [query, setQuery] = useState("");
  const [customers, setCustomers] = useState([]);
  const [status, setStatus] = useState("idle");
  const [error, setError] = useState("");

  const debouncedQuery = useDebounce(query, 400);

  useEffect(() => {
    const searchTerm = debouncedQuery.trim();

    if (!searchTerm) {
      setCustomers([]);
      setStatus("idle");
      setError("");
      return;
    }

    const controller = new AbortController();

    async function searchCustomers() {
      setStatus("loading");
      setError("");

      try {
        const response = await fetch(
          `/api/customers?search=${encodeURIComponent(searchTerm)}`,
          {
            signal: controller.signal,
          }
        );

        if (!response.ok) {
          throw new Error("Search request failed");
        }

        const data = await response.json();

        setCustomers(data.customers ?? []);
        setStatus("success");
      } catch (err) {
        if (err.name === "AbortError") {
          return;
        }

        setCustomers([]);
        setStatus("error");
        setError("Could not load customers. Please try again.");
      }
    }

    searchCustomers();

    return () => {
      controller.abort();
    };
  }, [debouncedQuery]);

  return (
    <section>
      <label htmlFor="customer-search">Search customers</label>

      <input
        id="customer-search"
        value={query}
        onChange={(event) => setQuery(event.target.value)}
        placeholder="Type a customer name..."
      />

      {status === "loading" && <p>Searching...</p>}
      {status === "error" && <p>{error}</p>}

      {status === "success" && customers.length === 0 && (
        <p>No customers found.</p>
      )}

      <ul>
        {customers.map((customer) => (
          <li key={customer.id}>
            <strong>{customer.name}</strong>
            <span> - {customer.email}</span>
          </li>
        ))}
      </ul>
    </section>
  );
}

Now when debouncedQuery changes, React runs the cleanup function for the previous effect.

That cleanup calls:

controller.abort();

So the older request is told:

You are no longer needed.

When fetch is aborted, it throws an error with the name AbortError. That is why we check this:

if (err.name === "AbortError") {
  return;
}

An aborted request is not a real failure from the user's point of view. It just means the user typed something newer.

What the Fixed Flow Looks Like

With the fixed version, the flow is cleaner:

User types "a"
Wait briefly
Start request for "a"

User changes input to "amy"
React cleans up the old effect
Request for "a" is aborted
Wait briefly
Start request for "amy"

Only the useful result updates the UI

This gives you two benefits:

You send fewer requests because of debounce.
You stop outdated requests because of AbortController.

That combination is much stronger than debounce alone.

A Tiny Mock API for Testing

If you want to test this locally, you can create an endpoint that responds after a random delay.

This makes the bug easier to reproduce.

Here is a tiny Express-style example:

const express = require("express");

const app = express();

const customers = [
  { id: 1, name: "Amy Chen", email: "amy@example.com" },
  { id: 2, name: "Amir Khan", email: "amir@example.com" },
  { id: 3, name: "Ana Lopez", email: "ana@example.com" },
  { id: 4, name: "Brian Stone", email: "brian@example.com" },
];

app.get("/api/customers", async (req, res) => {
  const search = String(req.query.search || "").toLowerCase();
  const delay = Math.floor(Math.random() * 1500);

  await new Promise((resolve) => setTimeout(resolve, delay));

  const matches = customers.filter((customer) =>
    customer.name.toLowerCase().includes(search)
  );

  res.json({ customers: matches });
});

app.listen(3000, () => {
  console.log("API running on http://localhost:3000");
});

The random delay is useful because real networks are not perfectly predictable either.

Try the broken component with this API and type quickly. You may see old results appear. Then switch to the fixed version and test again.

Common Mistakes to Avoid

Mistake 1: Only using debounce

Debounce is good for reducing noise.

But if a request has already started, debounce does not stop it.

Use debounce to avoid unnecessary requests. Use AbortController to cancel requests that are no longer relevant.

Mistake 2: Showing an error when a request was canceled

If the user types a new search term, canceling the old request is normal.

Do not show:

Something went wrong.

just because an old request was aborted.

Check for AbortError and quietly return.

Mistake 3: Forgetting empty input

If the user clears the search box, reset the results.

if (!searchTerm) {
  setCustomers([]);
  setStatus("idle");
  setError("");
  return;
}

This avoids showing old results when the input is empty.

Mistake 4: Not encoding the search term

Search text can contain spaces, symbols, and special characters.

Use encodeURIComponent before putting user input into a query string:

`/api/customers?search=${encodeURIComponent(searchTerm)}`

Could You Use a Library Instead?

Yes.

In a bigger app, you might use something like TanStack Query, SWR, or another data-fetching library. Those tools can help with caching, request state, retries, background refreshes, and more.

But even if you use a library later, it is still worth understanding this bug.

Because the root idea appears everywhere:

When async work finishes, make sure it is still relevant before updating the UI.

That applies to search boxes, file uploads, page filters, route changes, modal forms, and many other everyday features.

Final Takeaway

The bug is not that React is slow.

The bug is not that useEffect is broken.

The bug is that network responses can arrive in a different order than the requests were sent.

For search inputs, a practical fix is:

Debounce the input
+ Cancel outdated requests
+ Ignore expected abort errors
+ Reset state for empty input

That gives users a search box that feels stable, even when the network is not.

References

React docs: useEffect and race conditions
React docs: You Might Not Need an Effect
MDN: AbortController
MDN: AbortSignal

The Hidden Cost of Object Spread

Amrishkhan Sheik Abdullah — Thu, 09 Jul 2026 01:40:00 +0000

One of the most common pieces of JavaScript you'll find in modern codebases looks like this:

const nextState = {
  ...state,
  count: state.count + 1
}

React developers write it.

Redux developers write it.

Vue developers write it.

Angular developers write it.

Nearly every frontend framework encourages some variation of it.

It feels clean.

It feels immutable.

It feels modern.

It feels almost free.

But here's the uncomfortable truth:

Object spread is often far more expensive than most developers realize.

This doesn't mean object spread is bad.

It doesn't mean immutability is bad.

It doesn't mean you should stop using it.

But it does mean you should understand what it's actually doing.

Because once you understand the cost, you start making much better engineering decisions.

The Illusion

When developers see:

const nextState = {
  ...state,
  count: state.count + 1
}

they often mentally model it as:

Take state
Change count
Done

But that's not what happens.

JavaScript doesn't magically update the object.

Instead it performs something conceptually closer to:

const nextState = {}

for (const key in state) {
  nextState[key] = state[key]
}

nextState.count =
  state.count + 1

Notice the difference.

We're not updating.

We're copying.

Every property.

Every time.

What Actually Happens

Consider:

const user = {
  id: 1,
  name: "John",
  email: "john@example.com"
}

const updatedUser = {
  ...user,
  active: true
}

Internally:

Allocate New Object
↓
Copy id
↓
Copy name
↓
Copy email
↓
Add active

For three properties?

Nobody cares.

For thousands of properties?

You probably should.

The Cost Grows With Size

Imagine:

const hugeObject = {
  ...
}

containing:

10,000 properties

Now:

{
  ...hugeObject,
  updated: true
}

must:

Allocate New Object
+
Copy 10,000 Properties
+
Add One Property

Just to change one value.

That's not free.

The Famous Reduce Trap

This is one of the most common performance issues I see.

const usersById =
  users.reduce(
    (acc, user) => ({
      ...acc,
      [user.id]: user
    }),
    {}
  )

Looks elegant.

Looks immutable.

Looks functional.

Looks expensive.

Let's see why.

Iteration 1

{}

Copy:

0 properties

Iteration 2

{
  1: user1
}

Copy:

1 property

Iteration 3

{
  1: user1,
  2: user2
}

Copy:

2 properties

Iteration 1000

Copy 999 properties

Total Work

What initially looks like:

O(n)

can become closer to:

O(n²)

because every iteration copies everything accumulated so far.

That is a very different performance profile.

The Loop Equivalent

Compare:

const usersById =
  users.reduce(
    (acc, user) => ({
      ...acc,
      [user.id]: user
    }),
    {}
  )

with:

const usersById = {}

for (const user of users) {
  usersById[user.id] = user
}

The loop:

Creates One Object
Mutates One Object
Performs One Pass

No repeated copies.

No repeated allocations.

No repeated garbage collection.

Why React Popularized Object Spread

This is where things get interesting.

React didn't make object spread popular by accident.

React relies heavily on:

Reference Equality

Example:

if (
  previousState !== nextState
) {
  rerender()
}

This works beautifully when:

const nextState = {
  ...state,
  count: 1
}

because:

New Object
New Reference

React can instantly detect the change.

This is a great reason to use object spread.

But:

Useful
≠
Free

Deeply Nested Objects

This is where things get ugly.

Suppose:

const nextState = {
  ...state,
  user: {
    ...state.user,
    address: {
      ...state.user.address,
      city: "Dubai"
    }
  }
}

You've probably written something similar.

Maybe many times.

Let's count.

Copy state
Copy user
Copy address
Update city

Multiple allocations.

Multiple copies.

For one change.

Why Immer Became Popular

This problem became so common that libraries emerged specifically to solve it.

One of the most popular is Immer.

Instead of:

const nextState = {
  ...state,
  user: {
    ...state.user,
    address: {
      ...state.user.address,
      city: "Dubai"
    }
  }
}

Immer allows:

draft.user.address.city =
  "Dubai"

while still producing an immutable result.

This dramatically improves readability.

Garbage Collection Matters

Most developers focus on CPU.

But memory matters too.

Every spread operation creates:

Temporary Objects

Those objects eventually become:

Garbage

Which means:

Garbage Collection

must clean them.

The larger the application becomes:

More Allocations
↓
More Garbage
↓
More GC Work

Sometimes the bottleneck isn't computation.

It's memory churn.

The Hidden Cost In State Management

Consider:

return {
  ...state,
  loading: true
}

Looks harmless.

Now imagine:

100 updates per second

across:

Multiple Stores
Multiple Components
Large State Trees

Suddenly those allocations become measurable.

Not catastrophic.

Just measurable.

And that's the point.

When Object Spread Is Perfect

Let's be fair.

Object spread solves real problems.

Example:

const updatedUser = {
  ...user,
  active: true
}

Clear.

Readable.

Predictable.

For small objects:

Use it.

Without hesitation.

When You Should Be Careful

Pay attention when you see:

Large Collections

Large State Trees

Reducers Executing Frequently

Performance-Critical Loops

This is where spread begins to matter.

Benchmark Mentality

One of the biggest mistakes developers make is assuming:

Spread Is Slow

Spread Is Fast

Both are wrong.

The correct answer is:

It Depends

How many properties?

How often?

How frequently is the code executed?

How large are the objects?

Engineering is always contextual.

Real World Example: API Processing

Bad:

const result =
  users.reduce(
    (acc, user) => ({
      ...acc,
      [user.id]: user
    }),
    {}
  )

Better:

const result = {}

for (const user of users) {
  result[user.id] = user
}

The second version scales significantly better.

Real World Example: React State

Good:

setUser({
  ...user,
  name: "John"
})

The object is small.

Readability wins.

Optimization would be pointless.

Real World Example: Deep Updates

Instead of:

{
  ...state,
  a: {
    ...state.a,
    b: {
      ...state.a.b,
      c: value
    }
  }
}

consider:

State normalization
Immer
Better state structure

Architecture often beats optimization.

Pros Of Object Spread

1. Readable

Intent is obvious.

2. Immutable

Reduces accidental mutations.

3. React-Friendly

Works perfectly with reference equality.

4. Predictable

Creates explicit state transitions.

5. Easy To Learn

Minimal cognitive overhead.

Cons Of Object Spread

1. Allocations

Every spread creates a new object.

2. Property Copying

The larger the object, the more expensive the copy.

3. Garbage Collection Pressure

Temporary objects accumulate.

4. Easy To Abuse In Reducers

Repeated spreads can become surprisingly expensive.

5. Deep Updates Become Ugly

Nested spreads quickly reduce readability.

The Real Lesson

The biggest mistake developers make with object spread is treating it as a free operation.

It isn't.

The second biggest mistake is treating it as a bad operation.

It isn't.

Object spread is a tool.

A very useful tool.

A very readable tool.

A very common tool.

But every immutable update has a cost.

The goal isn't avoiding object spread.

The goal is understanding when its benefits outweigh its costs.

Because once you understand the tradeoff, you stop blindly copying objects.

And start making deliberate engineering decisions.

What's Next?

In the next article we'll discuss:

Composability Is The Real Superpower

Because after exploring:

Reduce
Transducers
Functors
FlatMap
Monads
RxJS
Event Sourcing

you'll discover that all of them ultimately revolve around a single idea:

Composition

And that idea is far more important than any individual function.

About The Author

Hi, I'm Amrish Khan.

I enjoy building developer tools, exploring software architecture, and writing about the deeper ideas behind everyday programming concepts.

I'm also building Aruvix — a growing ecosystem of local-first developer tools designed to process data directly in the browser without unnecessary uploads.

Here's a detailed blog on Aruvix:

https://dev.to/amrishkhan05/aruvix-the-ultimate-offline-first-developer-toolkit-e0i

You can follow my work and thoughts here:

Portfolio:
https://www.amrishkhan.dev

LinkedIn:
https://www.linkedin.com/in/amrishkhan

GitHub:
https://www.github.com/amrishkhan05

If you enjoyed this article, consider following for more deep dives into JavaScript, architecture, local-first software, and performance engineering.

DEV Community: Amrishkhan Sheik Abdullah

200 Broken Watches Couldn't Solve It. A Bouncing Ball Did.

Who Saw the Connection? #05 — G-SHOCK × A Bouncing Ball × Engineering Breakthroughs

It Started With a Broken Watch

Triple 10 Was an Almost Rude Engineering Requirement

Then They Started Throwing Watches Out of a Window

Then He Saw a Child Bouncing a Ball

The Best Debugging Sometimes Changes the Question

In 1983, They Finally Built It

America Saw Something Japan Didn't

If Your Claim Sounds Extraordinary, Stop Describing It

Then Japan Imported Its Own Idea Back

The Same Product Can Be Wrong in One Room and Right in Another

But “Wrong Audience” Can Become a Very Comfortable Lie

Triple 10 Was Really a Bet Against Incremental Thinking

Two Hundred Attempts Are Useless If Attempt 201 Learns Nothing

The Ball Is Still My Favourite Part

Who Saw the Connection?

Sources & Further Reading

About the Person Who Somehow Ended Up Throwing Software Out of a Third-Floor Window

Read More of My Writing

Elsewhere on the Internet

Why Do Japanese Train Drivers Keep Pointing at Things?

Who Saw the Connection? #04 — Railways × Human Factors × Software

A Railway Problem That Was Really a Human Problem

Researchers Eventually Tested It

The Finger Is Not the Important Part

Your Terminal Has the Same Problem

What Software Can Steal From the Railway

We Keep Trying to Solve Human Error With Documentation

Friction Is Not Always the Enemy

There Is a Catch: Humans Can Automate the Ritual Too

Automation Makes the Remaining Human Actions Stranger, Not Safer

The Practice Escaped the Railway

Maybe the Pointing Isn't the Strange Part

Who Saw the Connection?

Sources & Further Reading

About the Person Who Somehow Ended Up Pointing at Production Servers

Read More of My Writing

Elsewhere on the Internet

What Did Heart Surgeons See in a Formula 1 Pit Stop?

Who Saw the Connection? #03 — Formula 1 × Pediatric Heart Surgery × Software

The Operation Was Over. The Risk Wasn't.

So They Went to See How the Pit Crew Did It

Ferrari Wasn't the Only Place They Looked

What Actually Changed?

Fifty Handovers

The Pit Stop Wasn't Really About Speed

Your Customer Has Been Charged. Your System Thinks They Haven't.

The Handoff Is Where Things Become Expensive

We Spend Much More Time Designing the Boxes

The Lesson Isn't “Run Your Team Like Ferrari”

Who Saw the Connection?

The Shortest Route Is Sometimes the Slowest: What UPS Can Teach Software Engineers About Optimization

The Shortest Route Has a Few Missing Columns

Six Miles Doesn't Sound Like Much

We Do This Constantly

My Favourite Version Is “We Made It One Query”

The One-Arrow Architecture

ORION Had Another Input: Drivers

The Fast Service That Takes Four Seconds

So What Are We Actually Optimizing?

Sometimes the Longer Route Is Just Longer

The Part Worth Stealing From UPS

Sometimes the Fastest System Is the One Willing to Stop

Toyota's strange idea: give people permission to interrupt production

The internet version of Andon misses the clever bit

A factory can be extremely efficient at producing defects

Amazon looked at a car factory and saw a website

What exactly did Amazon copy?

We've already built Andon cords into software

The uncomfortable part isn't stopping. It's authority.

The pattern I would steal

There is also a danger in romanticizing this

The abstraction is the innovation

CocoIndex vs. code-review-graph: Semantic Search vs. Structural Code Intelligence

The Context Window Isn't a Codebase

CocoIndex Treats Code Retrieval as an Indexing Problem

code-review-graph Starts Somewhere Else

Similar Code and Dependent Code Are Not the Same Thing

Why Adding `async` Matters

Do Not Await `params.slug`

Fixing `generateMetadata`

Fixing `searchParams`

Fixing Both `params` and `searchParams` Together

Fixing `cookies()`

Fixing `headers()`

Mistake 1: Awaiting the Property Instead of `params`

Mistake 2: Forgetting `async`

Fix 2: Use `matcher` to Protect Only Private Routes