DEV Community:

I replaced GitHub Copilot with 3 tools. My team noticed within a week.

Thu, 21 May 2026 06:27:02 +0000

Cursor, Claude Code, and Windsurf didn’t just change how I code they changed what my PRs look like.

The Copilot breakup nobody talks about

I didn’t plan to replace GitHub Copilot. It was fine. It’s still fine. But “fine” stopped being good enough somewhere around month three of watching teammates ship faster than me while I was still waiting for an autocomplete suggestion that missed the point.

So I started experimenting. Not with one tool. With everything. Forty-something IDEs, agents, plugins, and CLI tools over four months, run against real work not demos, not tutorials, actual PRs that had to pass review. Most of them lasted a week. A few lasted a day. One deleted a file I needed and I don’t want to talk about it.

Three survived.

Cursor for multi-file feature work. Claude Code for everything that lives in the terminal. Windsurf for the days when I need to stay in flow without managing the AI every five minutes. By the end of week one my team lead asked what I’d changed. By week three two teammates had switched.

This isn’t a sponsored comparison post. None of these companies know I exist. It’s just what happened when I stopped treating AI coding tools as a category and started treating them as team members with different strengths.

TL;DR: Copilot is fine for autocomplete. If that’s still your whole AI coding stack in 2026 you’re leaving serious velocity on the table. Cursor, Claude Code, and Windsurf each own a different slot in the workflow and together they cover everything Copilot never could.

Why Copilot stopped being enough

Let me be fair to Copilot first. It’s not bad. For a dev who mostly works in one file at a time, writes straightforward code, and doesn’t need much more than smart autocomplete it does the job. I used it for over a year and I was mostly happy.

The problem isn’t what Copilot does. It’s what it doesn’t do.

Copilot watches what you’re typing and tries to finish the sentence. That’s the whole model. It doesn’t know why you’re writing what you’re writing. It doesn’t know what the rest of the codebase looks like. It doesn’t know that the function you’re building has to fit into an auth system three files away or that your team has a convention for error handling that isn’t in any docs. It makes educated guesses based on what’s visible in the current file and what it saw during training.

For a while that’s enough. Then your codebase grows. Your features get more interconnected. A change to one interface ripples through six files. A refactor touches the API layer, the service layer, and the tests simultaneously. And suddenly you’re doing all the thinking that the AI should be helping with, while it cheerfully suggests the wrong variable name.

The real cost isn’t the bad suggestions. It’s what you do with them. You stop, evaluate, reject, retype. You context switch out of the problem you were solving to manage the tool that was supposed to help you solve it. That friction is small per instance and significant over a day.

I started noticing it when I’d spend twenty minutes on something I expected to take five. Not because the problem was hard. Because I was fighting my tools to get there.

That’s when I started looking for something different. Not a better autocomplete. A different category of help entirely.

Cursor the IDE that started arguing back

I thought AI-assisted coding meant better autocomplete. Then Cursor refactored a function I didn’t ask it to touch, the PR passed review without a single comment, and I had to sit with that for a minute.

Cursor isn’t a smarter Copilot. It’s a different thing entirely. Where Copilot watches what you’re typing and tries to finish the sentence, Cursor reads your whole codebase and forms opinions about it. You can ask it to build a feature and it’ll touch six files, write the tests, and explain what it changed and why.

What I actually use it for:

Multi-file edits This is where it earns its keep. I don’t use Cursor for writing single functions. I use it when a change needs to ripple across the codebase updating an interface, migrating an API, refactoring auth logic. It plans the changes, shows you the diff across every affected file, and lets you approve before anything gets written.
.cursorrules Drop this file in the root of your project. Cursor reads it at the start of every session preferred patterns, things to avoid, naming conventions and actually respects that context every time.

# .cursorrules
You are working on a Node.js REST API.
Always use async/await. Never use callbacks.
Prefer explicit error handling. Never swallow errors silently.
Add JSDoc comments to all exported functions.

This alone cuts the back-and-forth in half.

Cmd+K inline editing Highlight a block, hit Cmd+K, describe what you want. Rewrites in place. No sidebar, no context switching, no copy-paste. Fastest way to refactor something small without losing your train of thought.
Agent window Cursor 3 April 2026, Cursor shipped a new interface built from scratch around agents. Multiple agents running in parallel across different repos, all visible in one sidebar. You dispatch tasks, review diffs, approve changes. It looks less like a code editor and more like an engineering manager’s dashboard. In a good way.

I still write code manually. Cursor doesn’t replace that. But for anything involving more than one file, more than one decision, or more than five minutes of thinking I hand it off and review the output. That’s a different relationship with your tools than most devs are used to, and it takes about a week to actually trust it.

Worth it.

cursor.com

Claude Code the terminal agent I didn’t know I needed

Most AI coding tools live inside your editor. Claude Code lives in your terminal. No IDE. No sidebar. No chat window. You talk to it like a senior engineer who already read the repo, and it writes files, runs commands, fixes test failures, and ships. It’s not a chatbot with code awareness. It’s a collaborator.

The first time it ran a full test suite, found a failing edge case I hadn’t noticed, fixed it, and committed while I was reviewing a completely different PR I had to close my laptop and think about my life choices for a moment.

What I actually use it for:

Natural language task execution You describe what you want in plain English. It reads the repo, makes a plan, executes it, and tells you what it did. No hand-holding, no step-by-step prompting.

$ claude "refactor the auth middleware to use JWT RS256,
run the tests, and fix anything that breaks"

It reads the codebase, plans the changes, runs the tests, iterates on failures without you touching anything.

Terminal-native workflow Claude Code lives where backend and DevOps work actually happens. No tab switching, no copy-pasting output into a chat window, no losing context. You stay in the terminal, it stays in the terminal, and the whole session feels like pairing with someone who never gets tired or distracted.
MCP tool integrations Claude Code connects to external tools via MCP GitHub, databases, deployment pipelines. You can give it real reach beyond just the codebase and it handles multi-step workflows that would normally take you 20 manual commands.
Computer use On Mac, Claude Code can open apps, click through UI, take a screenshot, and verify the result. Build a feature, launch it, test it visually from one terminal session. That one still gets me every time.

I still keep human oversight on anything production-critical. But for local dev, test environments, and deploy pipelines it runs, I review. That’s the whole loop.

docs.anthropic.com/claude-code

Windsurf the dark horse nobody warned me about

Everyone I know is on Cursor. Windsurf kept coming up in my Discord as the tool people switched to and then got annoyingly quiet about like they’d found something they didn’t want to share yet.

I tried it out of mild spite. They were right and I was annoyed about it.

Windsurf isn’t trying to beat Cursor on features. It’s trying to beat it on feel. The whole thing is built around Cascade an agentic AI that doesn’t wait for you to ask it something. It watches what you’re doing, understands the context, and acts. The difference between using Cursor and using Windsurf is the difference between having a very capable assistant and having a very capable assistant who also pays attention.

What I actually use it for:

Cascade agent Multi-file edits, terminal commands, test runs all without you directing every step. You describe the goal, Cascade figures out the path. It feels less like “here’s what I’m going to do, approve?” and more like “here’s what I did, check it.” That distinction matters more than it sounds when you’re deep in a feature and don’t want to break flow.
Codemaps Windsurf indexes your repo and builds a visual map of your architecture how files relate, where the entry points are, what connects to what. Genuinely useful when you’re jumping into a codebase you didn’t write, and it gives the AI accurate context on large projects without you having to explain the structure manually.
Drag-and-drop screenshot → UI generation Drop a screenshot of a design into Cascade and it generates the frontend code. For anyone doing UI work this is the kind of feature that makes you feel like you skipped two steps.
$15/month Cursor Pro is $20. Windsurf Pro is $15. Same capability tier, lower price. Not the reason to pick it, but not nothing either.

As of February 2026 Windsurf sits at number one in the LogRocket AI Dev Tool Power Rankings ahead of Cursor and GitHub Copilot. And with the Cognition AI acquisition bringing Devin integration into the roadmap, it’s about to get significantly more powerful.

I run Windsurf when I want to stay in flow on a feature without stopping to manage the AI. It gets out of the way in a way that Cursor, for all its power, sometimes doesn’t.

windsurf.com

How Cursor, Claude Code, and Windsurf work together

Each tool on its own is solid. Stacked right they cover every layer of the workflow without overlap and without gaps. It’s not about having three tools open at once it’s about each one owning a different job.

Here’s how they actually fit into my day:

Windsurf for active feature work When I’m building something new and want to stay in flow, Windsurf is open. Cascade handles the context, suggests the next move, runs the changes. I steer, it executes. I don’t stop to manage it and it doesn’t ask me to.
Cursor for multi-file refactors and reviews When a change is complex, touches multiple services, or needs careful diff review before anything gets committed Cursor. The agent window and .cursorrules context make it the right tool for surgical, deliberate work where I want to see exactly what's changing before it changes.
Claude Code for everything terminal Tests, deploys, migrations, environment setup, CI debugging. Anything that lives in the command line. Claude Code handles the full sequence while I’m doing something else, then tells me what it did.

The real-world flow looks like this:

Feature branch
→ Windsurf/Cascade writes the feature
→ Cursor agent reviews multi-file diffs
→ Claude Code runs tests + deploys to staging
→ Push PR

Three tools, zero browser tabs open to paste errors into

The week my team lead asked what I’d changed, this was the answer. Not one tool. Not a new IDE. A stack where every part of the workflow had the right tool behind it. Windsurf for flow, Cursor for precision, Claude Code for the terminal. Nothing falling through the cracks.

That’s the whole thing. It took four months and forty tools to figure out. Now it just runs.

Final thoughts: you don’t need 40 tools

I didn’t set out to replace Copilot. I set out to stop feeling like my tools were slowing me down. Those are different problems and they lead to different solutions.

Copilot isn’t the villain here. It’s a solid tool that does exactly what it promises. The issue is that what it promises stopped being enough once the rest of the ecosystem caught up and then kept going. Standing still while everything around you accelerates is its own kind of falling behind.

After four months and forty experiments the three tools that actually changed my output were:

Cursor for multi-file feature work and agentic refactors
Claude Code for terminal tasks, deploys, and anything CLI
Windsurf for staying in flow with Cascade running alongside you

None of them require you to change how you think about code. They just remove the parts that were slowing you down the context switching, the tab juggling, the manual command sequences, the back-and-forth with a tool that doesn’t know what you’re building or why.

My PRs got cleaner. My review cycles got shorter. My team noticed before I even said anything. That’s the only metric that matters.

If you’re still on vanilla Copilot and shipping just fine genuinely, keep going. But if you’ve been feeling the friction and just assumed that was normal, it isn’t. The gap between what Copilot does and what this stack does is real and it’s only getting wider.

Start with one. Cursor if you spend most of your time in the editor. Claude Code if you live in the terminal. Windsurf if you keep getting pulled out of flow. Run it for two weeks on real work. You’ll know by then.

Helpful resources and links

Cursor multi-file AI editor with agent window and .cursorrules support
Cursor 3 announcement the April 2026 agent-first rebuild
Claude Code terminal-native AI coding agent
Windsurf agentic IDE with Cascade and Codemaps
GitHub Copilot still worth knowing what you’re moving away from
LogRocket AI Dev Tool Power Rankings February 2026 edition
r/cursor real-world Cursor workflows and community feedback

Cursor, Claude Code, Windsurf?! My AI coding stack after 40 dev experiments

Tue, 19 May 2026 03:22:53 +0000

For devs drowning in AI tool hype who just want to know what actually stuck

40 tools, 3 survivors

If you’ve ever installed a new AI coding tool on a Monday, spent the whole evening configuring it, been genuinely impressed for about three days, and then quietly gone back to your old setup by Thursday you’re not alone. Some devs binge Netflix. I install AI tools.

I went through a phase (fine, a four-month spiral) where I tested nearly every tool promising to make me ship faster, code smarter, or finally stop copy-pasting stack traces into a browser tab. Most of them were fine. A few were actually good. One deleted a file I needed. But out of the 40+ IDEs, agents, plugins, extensions, and CLI tools I ran through real work projects not toy repos, actual PRs three made it through.

Cursor. Claude Code. Windsurf.

This isn’t a “best of 2026” roundup written by someone who ran each tool for 20 minutes. It’s what survived four months of daily use across real backend work, DevOps tasks, and the occasional frontend emergency. Tools I open every morning without thinking about it. The ones that made my workflow cleaner, faster, and way less frustrating.

If you’re still on vanilla Copilot wondering why things still feel slow, or you’re deep in comparison hell and just want someone to cut through it this one’s for you.

TL;DR: Cursor handles multi-file feature work and agentic refactors. Claude Code owns the terminal tests, deploys, migrations, anything CLI. Windsurf runs Cascade in the background while you stay in flow. Together they cover every slot in a serious dev workflow. Separately, each one is still better than most of the 37 tools I’m not writing about.

Let’s get into it.

Why your AI tool stack actually matters now
Cursor the IDE that started arguing back
Claude Code the terminal agent I didn’t know I needed
Windsurf the dark horse nobody warned me about
How Cursor, Claude Code, and Windsurf work together
Final thoughts: you don’t need 40 tools
Helpful resources and links

Why your AI tool stack actually matters now

Here’s the thing nobody tells you when you first install an AI coding tool: the tool isn’t the hard part. The hard part is figuring out which problems you actually have, and whether what you just installed solves any of them.

Most devs treat AI tooling like a plugin they’ll figure out later. They install Copilot, use it for autocomplete, occasionally ask it to explain a regex, and call it done. That worked fine in 2024. In 2026 it’s the equivalent of using a GPS only to check if it’s raining.

The conversation has moved. Agents are the standard now, not the novelty. The tools worth your time aren’t the ones that finish your line of code they’re the ones that read your whole codebase, plan a multi-step task, execute it, run your tests, fix the failures, and come back when it’s done. That’s not autocomplete. That’s a different category of tool entirely.

And the cost of getting this wrong isn’t just a wasted subscription. It’s the context-switching penalty. Every time you break flow to copy an error into a chat window, switch tabs to ask a question you should be able to ask inside your editor, or manually run a command sequence an agent could handle that’s compounding friction. The DORA 2025 report found that high-performing engineering teams are pulling significantly ahead of the rest, and tooling decisions are a real part of that gap.

The developers figuring out the right AI stack right now aren’t the ones with the most tools installed. They’re the ones who stopped treating AI like a fancy tab completion and started treating it like a collaborator with a job description. You wouldn’t hire one person to do your backend, frontend, DevOps, and code review. Same logic applies here.

That’s the frame for everything that follows not which AI tool is best, but which tool owns which job, and how you stack them so nothing falls through the cracks.

Cursor the IDE that started arguing back

What I actually use it for:

Multi-file edits This is where it earns its keep. I don’t use Cursor for writing single functions. I use it when a change needs to ripple across the codebase updating an interface, migrating an API, refactoring auth logic. It plans the changes, shows you the diff across every affected file, and lets you approve before anything gets written.
cursorrules Drop this file in the root of your project. Cursor reads it at the start of every session preferred patterns, things to avoid, naming conventions and actually respects that context every time.

# .cursorrules
You are working on a Node.js REST API.
Always use async/await. Never use callbacks.
Prefer explicit error handling. Never swallow errors silently.
Add JSDoc comments to all exported functions.

This alone cuts the back-and-forth in half.

Cmd+K inline editing Highlight a block, hit Cmd+K, describe what you want. Rewrites in place. No sidebar, no context switching, no copy-paste. Fastest way to refactor something small without losing your train of thought.
Agent window Cursor 3 April 2026, Cursor shipped a new interface built from scratch around agents. Multiple agents running in parallel across different repos, all visible in one sidebar. You dispatch tasks, review diffs, approve changes. It looks less like a code editor and more like an engineering manager’s dashboard. In a good way.

Worth it.

cursor.com

Claude Code the terminal agent I didn’t know I needed

Why I stuck with it:

Terminal-native workflow Claude Code lives where backend and DevOps work actually happens. No tab switching, no copy-pasting output into a chat window, no losing context. You stay in the terminal, it stays in the terminal, and the whole session feels like pairing with someone who never gets tired or distracted.
Natural language task execution You describe what you want in plain English. It reads the repo, makes a plan, executes it, and tells you what it did.

$ claude "refactor the auth middleware to use JWT RS256,
run the tests, and fix anything that breaks"

It reads the codebase, plans the changes, runs the tests, iterates on failures without you touching anything.

MCP tool integrations Claude Code connects to external tools via MCP GitHub, databases, deployment pipelines. You can give it real reach beyond just the codebase and it handles multi-step workflows that would normally take you 20 manual commands.
Computer use On Mac, Claude Code can open apps, click through UI, screenshot the result, and verify it worked. Build a feature, launch it, test it visually from one terminal session. That one still gets me every time.

I still use it with human oversight on anything production-critical. But for local dev, test environments, and deploy pipelines it runs. I review. That’s the whole loop.

docs.anthropic.com/claude-code

Windsurf the dark horse nobody warned me about

Everyone I know is on Cursor. Windsurf kept coming up in my Discord as the tool that people switched to and then got annoyingly quiet about like they’d found something they didn’t want to share yet.

I tried it out of mild spite. They were right and I was annoyed about it.

Windsurf isn’t trying to beat Cursor on features. It’s trying to beat it on feel. The whole thing is built around Cascade an agentic AI that doesn’t wait for you to ask it something. It watches what you’re doing, understands the context, and acts. The difference between using Cursor and using Windsurf is the difference between having a very capable assistant and having a very capable assistant who also pays attention.

What makes it essential:

Cascade agent Multi-file edits, terminal commands, test runs all without you directing every step. You describe the goal, Cascade figures out the path. It’s similar to Cursor’s agent mode but the flow feels less interrupted. Less “here’s what I’m going to do, approve?” and more “here’s what I did, check it.”
Codemaps Windsurf indexes your repo and builds a visual map of your architecture how files relate, where the entry points are, what connects to what. Useful when you’re jumping into a codebase you didn’t write, and genuinely helpful for giving the AI accurate context on large projects.
Drag-and-drop screenshot → UI generation Drop a screenshot of a design into Cascade and it generates the frontend code. For anyone doing UI work this is the kind of feature that makes you feel like you skipped two steps.
$15/month Cursor Pro is $20. Windsurf Pro is $15. Same tier, same power level, lower price. Not the reason to pick it, but not nothing either.

As of February 2026, Windsurf sits at number one in the LogRocket AI Dev Tool Power Rankings ahead of Cursor and GitHub Copilot. And with the Cognition AI acquisition bringing Devin integration into the roadmap, it’s about to get significantly more powerful.

I run Windsurf when I want to stay in flow on a feature without stopping to manage the AI. It gets out of the way in a way that Cursor, for all its power, sometimes doesn’t.

windsurf.com

How Cursor, Claude Code, and Windsurf work together

Each tool on its own is solid. Stacked right, they cover every layer of the workflow without overlap and without gaps. It’s not about having three tools open at once it’s about each one owning a different job.

Here’s how they fit into my actual day:

Windsurf for active feature work When I’m building something new and want to stay in flow, Windsurf is open. Cascade handles the context, suggests the next move, runs the changes. I steer, it executes.
Cursor for multi-file refactors and reviews When a change is complex, touches multiple services, or needs careful diff review before anything gets committed Cursor. The agent window and .cursorrules context make it the right tool for surgical, deliberate work.
Claude Code for everything terminal Tests, deploys, migrations, environment setup, CI debugging. Anything that lives in the command line. Claude Code handles the full sequence while I’m doing something else.

The real-world flow looks like this:

Feature branch
→ Windsurf/Cascade writes the feature
→ Cursor agent reviews multi-file diffs
→ Claude Code runs tests + deploys to staging
→ Push PR

Three tools, zero browser tabs open to paste errors into.

I stopped thinking of these as “AI tools” and started thinking of them as team members with specializations. Different strengths, different contexts, different jobs. Once you frame it that way the stack stops feeling like overkill and starts feeling obvious.

Final thoughts: you don’t need 40 tools

I didn’t set out to build some ultimate AI coding stack. I just wanted something that worked without me having to think about it every week.

After testing more than 40 tools, the three that actually made my workflow better were:

Cursor for multi-file feature work and agentic refactors
Claude Code for terminal tasks, deploys, and anything CLI
Windsurf for staying in flow with Cascade running alongside you

They’re not trying to do the same thing. They don’t step on each other. And none of them require you to change how you code they just remove the parts that were slowing you down.

If you’re still running vanilla Copilot and calling it your AI stack that’s fine. But you’re leaving a lot on the table. The tooling gap between devs who’ve figured this out and devs who haven’t is real, and it’s only getting wider.

Start with one. Get comfortable. Add the next. By the time you’ve run all three for a month you won’t remember what the friction felt like.

Helpful resources and links

Cursor multi-file AI editor with agent window and .cursorrules support
Cursor 3 announcement the April 2026 agent-first interface rebuild
Claude Code terminal-native AI coding agent
Windsurf agentic IDE with Cascade, Codemaps, and Devin integration incoming
DORA 2025 report state of DevOps and AI-assisted development
LogRocket AI Dev Tool Power Rankings February 2026 rankings
r/cursor community discussion and real-world Cursor workflows

You’re the reason your React app is slow

Mon, 18 May 2026 05:09:41 +0000

You didn’t hit a framework limit. You wrote the bottleneck yourself and it’s been quietly billing you in FPS ever since.

There’s a specific kind of suffering that happens when you open the React DevTools Profiler for the first time on a project that’s been “running fine.” You hit record. You click a button. You stop recording. And then you just sit there, staring at a flame graph that looks like a city on fire, wondering how a todo app is re-rendering 47 components when you clicked “add item.”

That was me, about three years into thinking I was pretty decent at React.

I wasn’t bad. My components looked clean. My PRs got approved. The app shipped. But under the hood, I was doing roughly eight things wrong simultaneously, and the only reason nobody noticed was that our user base was small enough that the jank felt like a “network thing.” It was not a network thing.

The React ecosystem has an interesting culture around performance: everyone knows it matters, most articles cover the same four hooks, and almost nobody talks about the architectural decisions that create the problem in the first place. The React Compiler landing in React 19 is going to paper over some of this it automatically memoizes components and values, essentially applying useMemo and useCallback everywhere it's safe to do so but here's the honest truth: it won't save you from architectural issues like overly broad context providers or massive component trees. You can't compile your way out of a bad design. DEV CommunityDEV Community

This article is the one I wish someone had thrown at me back then. No cargo-cult hooks. No “just add React.memo" advice. Just the actual mistakes, why they happen, and what they cost you in the real world.

TL;DR: React is fast by default. You are often the problem. These 10 mistakes are the most common ways engineers including experienced ones quietly murder their own app’s performance, and most of them have nothing to do with the hooks you’ve been memorizing.

The re-render killers

Let’s start with the category that accounts for maybe 60% of React performance complaints I’ve seen in the wild. Not slow APIs. Not bad algorithms. Just components re-rendering when they absolutely did not need to, because of decisions made in the five seconds it took to write a JSX prop.

React’s re-render model is simple enough that it’s easy to underestimate. React re-renders when state or props change by reference, not by value. That one sentence is responsible for more production slowdowns than any framework bug ever was. It sounds obvious until you realize how many ways you’re accidentally creating new references on every render without thinking about it. DEV Community

Mistake 1: Inline functions in JSX

This is the one that gets everyone eventually, usually when they’re moving fast and the code looks clean.

// you write this and it feels fine
<Button onClick={() => handleDelete(item.id)} label="Delete" />

Here’s what’s actually happening: every time the parent component renders, that arrow function is a brand new function object in memory. React does a shallow comparison on props. New reference equals “props changed” equals re-render even if item.id hasn't moved an inch. JavaScript creates a new object or function reference on every render. React does a shallow comparison when deciding whether to re-render a child, and since the reference is always new, the child always re-renders, even when nothing meaningful has changed. DEV Community

The fix is boring and correct: move static handlers outside the component, and for dynamic ones that depend on state or props, reach for useCallback. But there's a catch which brings us directly to mistake number two.

Mistake 2: Using `useCallback` as a good luck charm

So you read about inline functions, you start wrapping everything in useCallback, and you feel like you've leveled up. You haven't. You've just moved the problem around and added overhead.

useCallback only does anything useful when the component receiving that function is actually memoized wrapped in React.memo. Without that, you're paying the cost of memoization (React has to store the previous function, compare dependencies, and make a decision) while getting zero benefit, because the child rerenders anyway. useCallback only helps if the child component is memoized (React.memo) or uses the callback in its own dependency arrays. Otherwise, you're adding overhead for no benefit. DEV Community

I’ve seen codebases where someone went on a useCallback spree across the entire app, felt productive for a day, and then wondered why nothing got faster. There is a cost to memoization. React must store the previous props, compare them, and make a decision this adds overhead. If your component is fast to render and frequently changing, this comparison step may become more expensive than the render itself. Growin

The actual rule: useCallback is a tool for reference stability, not a performance incantation. Use it when you have a memoized child that receives the function as a prop, or when the function is in a useEffect dependency array and you want control over when the effect fires. That's basically it. Profile first, reach for it second.

Mistake 3: Using array index as `key` in lists

This one feels harmless until you have a list that changes items get added, removed, or reordered and suddenly your UI starts doing weird things. State ends up in the wrong component. Inputs keep the wrong value. Animations fire on the wrong element. You spend an hour blaming a library that did nothing wrong.

The key prop is React's identity system for list items. When you use the array index, you're telling React "the first item is always the first item, regardless of what it actually is." Reorder the list, and React thinks all the same items are still there just with different content. It patches the DOM in place instead of remounting, which is fast but wrong.

// this looks fine and is not fine
{items.map((item, i) => <Card key={i} data={item} />)}

// this is fine
{items.map(item => <Card key={item.id} data={item} />)}

If your data genuinely has no stable IDs which happens more than it should generate them when the data is created, not at render time. A crypto.randomUUID() call in your fetch handler costs nothing. A Math.random() call inside map gives every item a new key on every render, which tells React to unmount and remount the entire list. That costs a lot.

All three of these mistakes share the same root: React’s rendering model is predictable once you understand it, but it punishes you quietly. No errors. No warnings. Just a Profiler graph that looks increasingly unwell.

The good news is that the React DevTools Profiler will catch all three almost immediately. Record a session, look for components highlighted in yellow or red, and ask yourself: “did this actually need to re-render?” Usually the answer is no, and usually one of these three is why.

State architecture sins

Re-renders from inline functions are annoying. State architecture mistakes are a different category of problem entirely. They’re the ones that survive a code review, pass all your tests, and then slowly make your app feel like it’s running through wet concrete as the feature count grows. They’re structural. And they’re almost always invisible until you’re already in pain.

The pattern is consistent across every codebase I’ve seen it in: someone makes a reasonable decision early, the app grows around that decision, and by the time the jank is obvious there are forty components depending on the thing that’s wrong. Refactoring it feels like surgery on a patient who’s still running a marathon.

Understanding why these happen is more useful than just memorizing the fix.

Mistake 4: Putting state too high up the tree

This is the most common architectural mistake in React, and it’s almost always made with good intentions. You want state to be accessible from multiple places, so you lift it up to a common ancestor. Reasonable. Except that ancestor is now App, and every time a checkbox toggles in a deeply nested form, your entire component tree re-renders.

If your App component’s state changes, every child component re-renders even if their props didn’t change. This cascading effect kills performance at scale. The mental model that helps here: state should live as close as possible to the components that actually use it. Not one level above. Not in a global provider “just in case something else needs it later.” Right next to the thing that needs it. This is called state colocation, and it’s one of those ideas that sounds obvious until you see how rarely it’s actually practiced.

If only two components in a subtree share a piece of state, their nearest common ancestor is the right home for it not the root. If state is only used by one component, it belongs inside that component, full stop. Split your components into two types: container components that handle the logic, and presentational components that are pure display. Because they have no local state, presentational components only re-render when their props actually change.

The performance difference between “state at the root” and “state colocated” can be dramatic in a large component tree. It’s also the kind of change that makes every subsequent optimization easier, because you’re no longer fighting against a re-render cascade every time you try to fix something specific.

Mistake 5: Context without memoization

React Context is one of those features that feels like a complete solution right up until it isn’t. You set up a provider, everything can access your global state, PRs get merged, life is good. Then someone notices that updating the user’s theme preference is somehow causing the entire dashboard to re-render, including the charts, the sidebar, and the table that has nothing to do with theming.

Here’s why. When you pass an object as the context value which almost everyone does that object is recreated on every render of the provider component. Every consumer sees a new reference. Every consumer re-renders. Even the ones that only care about one field in that object that didn’t change.

// every render of AppProvider creates a new value object
// every consumer re-renders every time anything changes
function AppProvider({ children }) {
  const [user, setUser] = useState(null);
  const [theme, setTheme] = useState('dark');
  const value = { user, setUser, theme, setTheme };
  return <AppContext.Provider value={value}>{children}</AppContext.Provider>;
}

Every state update created a new value object. Every context consumer re-rendered. Including components that only cared about the theme, not the user.

Two fixes, and you’ll usually want both. First, wrap the context value in useMemo so the reference only changes when the actual data changes. Second, split large contexts into smaller ones by concern a UserContext and a ThemeContext rather than one AppContext that holds everything. A component that only reads theme should never re-render because the user object updated.

This one is particularly brutal in larger apps because context is often set up early, before the component tree is complex enough to make the problem visible. By the time you feel it, the context is load-bearing and everything’s wired to it.

Mistake 6: `useEffect` doing too much

useEffect is the Swiss Army knife of React hooks, which is both its strength and the reason developers use it to solve problems it was never designed for. The classic version of this mistake: a useEffect with a dependency array that's either wrong, empty when it shouldn't be, or so long it fires on basically every render anyway.

The subtler version is using useEffect for things that are actually derived state or event-driven logic. If you're running an effect to compute a value from existing state, that should probably be useMemo. If you're running an effect in response to a user action, that logic belongs in the event handler not in a side effect that triggers after the render. Using useEffect without proper dependencies, or too many, can trigger unnecessary logic or cause infinite loops.

The infinite loop variant is a rite of passage. You set state inside a useEffect, that state triggers a re-render, the effect fires again, sets state again, render again, and so on until your browser tab starts sweating. It happens to everyone. The less dramatic version an effect that fires twice as often as it should because the dependency array includes an object that gets recreated on every render is more insidious because it's harder to notice and easy to shrug off as "just how React works."

It’s not just how React works. It’s a dependency array that needs fixing.

A useful heuristic: if you can’t explain in one sentence what your useEffect is synchronizing with the outside world, there's a decent chance it shouldn't be a useEffect at all.

These three mistakes compound each other in particularly ugly ways. State too high up means more components consuming context. Context without memoization means all those components re-render together. Overloaded effects means those re-renders trigger more side effects. By the time you feel it, the performance problem isn’t one thing it’s a system of bad decisions that arrived gradually and now all need untangling at once.

The good news: fixing any one of them improves things measurably. Fixing all three feels like discovering your app had been running with the handbrake on.

Bundle crimes

Everything so far has been about what happens at runtime components re-rendering when they shouldn’t, state living in the wrong place, effects misfiring. These next two mistakes happen before a single line of your React code executes. They happen at load time, and they’re the reason some apps feel slow before the user has even done anything.

The bundle is the thing you’re shipping. It’s the JavaScript your users have to download, parse, and execute before they can interact with your product. Most developers think about it roughly once during the initial setup and then stop thinking about it entirely as the codebase grows. Features get added, dependencies get installed, and the bundle gets quietly heavier with every sprint until your Lighthouse score starts looking embarrassing.

According to HTTP Archive data, the median JavaScript payload for desktop users is over 500 KB, with mobile users often downloading significantly more. That’s the median. Plenty of production React apps are shipping multiples of that. On a mid-range phone on a decent connection, that’s a real wait before anything is interactive and most users won’t stick around for it. Growin

Mistake 7: Not using `React.lazy` and `Suspense`

The default behavior in a React app without code splitting is simple: everything ships together. Your landing page bundle includes the code for your settings panel, your admin dashboard, your onboarding flow, and every modal you’ve ever built. The user visiting your homepage for the first time downloads all of it, even though they haven’t navigated anywhere yet and statistically might never open half of those routes.

React.lazy and Suspense exist specifically for this. They let you split your bundle at the route or component level, loading chunks only when they're actually needed.

const AdminDashboard = React.lazy(() => import('./AdminDashboard'));

function App() {
  return (
    <Suspense fallback={<Spinner />}>
      <AdminDashboard />
    </Suspense>
  );
}

React.lazy() and Suspense, when combined with route-level code splitting or dynamic imports, offer a reliable and modern solution to optimize loading behavior. Growin

The gains here can be significant. A route-level split on a mid-sized app commonly cuts the initial bundle by 30–50%, which translates directly into faster time-to-interactive. The user landing on your homepage gets only what they need to render that page. Everything else loads on demand, in the background, when it becomes relevant.

The reason developers skip this is usually that the app worked fine without it during development. Local development has no network latency and no cold cache, so a 2MB bundle feels instantaneous. Your users on mobile networks in the real world are having a measurably worse time, and the Profiler won’t show you that you have to look at your bundle analyzer and your Core Web Vitals to see it.

If you’re on Next.js or Remix, a lot of this is handled for you at the framework level. If you’re on a custom Vite or Webpack setup, route-level lazy loading is one of the highest-leverage changes you can make with the least amount of code.

Mistake 8: Importing entire libraries when you need one function

This one has been a known issue for years and it still shows up constantly. The pattern looks like this:

import _ from 'lodash';

const result = _.groupBy(data, 'category');

You needed groupBy. You imported all of lodash. Lodash is around 70KB minified and gzipped which is not catastrophic on its own, but it's also not free, and it compounds with every other library you're doing the same thing with.

Importing an entire library rather than just one or more components can dramatically increase your bundle size. A large bundle can slow download times, thereby negatively affecting the user experience. Use named imports rather than default imports, and use code-splitting so you can load only the code you need. TFTUS Official Blog

The fix is named imports, which tree-shake correctly:

import groupBy from 'lodash/groupBy';
// or
import { groupBy } from 'lodash-es';

The lodash-es variant is the ESM version of lodash, which plays nicely with modern bundlers and tree-shaking. Only the functions you actually import end up in your bundle.

Lodash is just the most common example. The same mistake gets made with moment.js a library that is famously large and should almost always be replaced with date-fns or the native Intl API at this point. It gets made with UI component libraries that export everything from a single entry point. It gets made with icon packs where someone imports the entire icon set to use three icons.

The way to catch this at scale is the Webpack Bundle Analyzer or the equivalent for Vite. Run it once on your production build and look at what’s actually inside your bundle. The results are often surprising in ways that are equal parts educational and mortifying. You will almost certainly find at least one dependency in there that you forgot you installed, and at least one that you installed for something you ended up not shipping.

This is worth doing before you spend time on any runtime optimization. It doesn’t matter how well-memoized your components are if you’re making users wait three seconds to download the JavaScript before any of that code runs.

These two mistakes live in a different category from the previous six because they’re invisible during development and only show up in production metrics. Your app feels fast locally. Your users experience something different. The fix for both requires adding a step to your workflow looking at bundle output, running Lighthouse against a production build, checking Core Web Vitals in Search Console rather than changing how you write components.

That habit of looking at what you’re actually shipping is one of the things that separates engineers who ship fast apps from engineers who ship apps that feel fast on their own machine.

The tools you’ve been ignoring

The previous eight mistakes were all things you did. These last two are things you didn’t do which somehow makes them worse. There’s a special category of performance problem that exists not because you wrote something wrong, but because you never reached for the tool that would have either prevented the problem or shown you it existed.

These aren’t obscure. They’re not advanced. They ship with React or have been in the ecosystem for years. They just require you to stop and think about scale before scale becomes the problem, and most of us don’t do that until a user complains or a Lighthouse score goes red.

Mistake 9: Rendering a thousand items when you could render twelve

At some point in almost every data-heavy React app, someone builds a list. The list works great with twenty items in development. It goes to production. The list now has two thousand items. Scrolling feels like dragging furniture across carpet. The component that renders each row is perfectly optimized memoized, no inline functions, stable keys and it doesn’t matter at all, because you’re rendering all two thousand of them simultaneously into the DOM whether the user can see them or not.

This is the problem that virtualization solves, and it’s one of those solutions that feels almost too simple once you understand it. Instead of rendering every item in the list, you render only the ones currently visible in the viewport plus a small buffer above and below for smooth scrolling. As the user scrolls, items that leave the viewport get unmounted, new ones get mounted. The DOM stays small. Performance stays flat regardless of dataset size.

import { FixedSizeList as List } from 'react-window';

<List
  height={600}
  itemCount={items.length}
  itemSize={72}
  width="100%"
>
  {({ index, style }) => (
    <div style={style}>
      <UserCard user={items[index]} />
    </div>
  )}
</List>

[react-window] only renders visible items, making scrolling smooth. Rendering thousands of DOM nodes at once kills performance. DEV Community

react-window is the standard library for this, maintained by Brian Vaughn who also built the React DevTools. It's small, fast, and well-documented. react-virtuoso is a newer alternative with more built-in features if you need variable item heights or grouped lists. Both work on the same principle.

The reason developers skip virtualization is usually that they don’t anticipate scale. The list has twenty items, the list works, ship it. Then the list grows. The performance cost of rendering a DOM node for every item in a list scales linearly double the items, roughly double the render time, double the memory usage, double the layout work the browser has to do on every scroll event. By the time you feel it, you often have a list that’s expensive to refactor because everything downstream depends on how it currently works.

The rule of thumb: if a list could plausibly ever exceed fifty items in production, plan for virtualization from the start. It’s much easier to set up on a new component than to retrofit it onto a complex one that’s been in production for six months.

Mistake 10: Never opening the React DevTools Profiler

This is the one that quietly enables every other mistake on this list to survive as long as it does. The Profiler has been part of React DevTools since React 16.5. It shows you exactly which components re-rendered, how long each render took, and why the render was triggered. It is, genuinely, one of the most useful debugging tools in the frontend ecosystem. Most developers have it installed and have never clicked the record button.

The workflow is straightforward. Open DevTools. Go to the Profiler tab. Hit record. Interact with the part of your app that feels slow. Stop recording. Look at the flame graph.

What you’re looking for: components that re-render more often than they should, and components whose render time is disproportionately long. The Profiler color-codes this for you gray means fast, yellow means moderate, orange and red mean you should probably look at this. You can click any bar in the graph to see exactly why that component rendered, including which prop or state value changed to trigger it.

React DevTools Profiler will show you whether a component re-render is expensive enough to justify memoization. In React development, especially as applications grow more interactive and component-driven, it’s easy to introduce performance issues without realizing it. Growin

This matters because performance optimization without measurement is just guessing. You add useCallback somewhere because it seems like the right area. You split a component because it feels too big. You might be right. You might be optimizing something that was already fast while the actual bottleneck sits three components over, untouched. No cargo-cult programming just understanding the system and applying targeted fixes. DEV Community

The Profiler removes the guessing. It tells you where the fire actually is, not where you think it might be. Every fix on this list inline functions, context memoization, colocated state, code splitting lands differently depending on your specific app. The Profiler is how you know which ones to prioritize and whether your changes actually did anything.

A practical habit: run the Profiler on any feature before you ship it, not after someone complains. It takes three minutes and it has saved me from shipping performance regressions more than once. It’s also genuinely interesting seeing exactly how React thinks about your component tree teaches you things about the framework that no article can.

These two mistakes share the same underlying cause: not thinking about scale and measurement until after the problem exists. Virtualization is what you add when you think ahead about large datasets. The Profiler is what you use when you want to stop guessing and start knowing.

Together, they close the loop on the entire list. The first eight mistakes give you specific patterns to avoid. These two give you the habit of catching what slips through anyway.

You’re not off the hook just because React 19 exists

Here’s the take that’s going to age either very well or very poorly: most of what’s on this list is going to become less relevant over the next two years, and that should make you more careful, not less.

React 19’s biggest change is the React Compiler, which automatically optimizes components without manual useMemo or useCallback wrappers. It analyzes your code at build time and applies memoization where it's safe. The inline function problem, the useCallback cargo-culting, a chunk of the re-render chaos the compiler handles a meaningful portion of that automatically. That's genuinely good news and the React team deserves credit for it. DEV Community

But I keep coming back to something that doesn’t change regardless of what the compiler does. Architecture isn’t a compiler problem. State living in the wrong place, context providers that re-render everything downstream, useEffect being used as a catch-all for logic that should live elsewhere none of that gets fixed at build time. The compiler won't save you from architectural issues like overly broad context providers or massive component trees. You still have to understand the system well enough to design it correctly. DEV Community

And here’s the uncomfortable part: if you learned React by reaching for useMemo and useCallback without understanding why, the compiler bailing you out doesn't mean you understood what it fixed. It means you got lucky. The next framework, the next abstraction, the next performance problem that falls outside what the compiler covers that's where the gap shows up.

Performance isn’t a checklist you run through once before a release. It’s a design skill. It lives in the decisions you make about where state goes, how components are composed, what you ship in the initial bundle, and whether you ever actually look at what your app is doing under load. The engineers I’ve seen consistently ship fast products aren’t the ones who know the most hooks. They’re the ones who profile before they optimize, think about scale before it’s a crisis, and treat the bundle as something they’re responsible for not something that happens automatically.

The React compiler is a great tool. So is the Profiler. So is react-window. None of them replace the judgment call about whether your component tree makes sense.

If this list had a single takeaway it’d be this: open the Profiler on your current project today, before you read another article, before you install another dependency. Record thirty seconds of normal usage. See what’s actually happening. You might be surprised. You might be horrified. Either way, you’ll know something real and that’s where every fix on this list actually starts.

Drop your worst React performance horror story in the comments. Bonus points if the root cause was on this list and you didn’t know it until a user complained.

Helpful resources

React DevTools Profiler docs official guide to actually using the tool
React.lazy and Suspense React docs on code splitting
React-window on GitHub list virtualization by Brian Vaughn
Webpack Bundle Analyzer see what you’re actually shipping
React Compiler docs what it does and doesn’t do
web.dev Core Web Vitals the metrics that actually matter to users
Kent C. Dodds on state colocation the best deep-dive on mistake #4

The only AI agents article you’ll ever need

Fri, 15 May 2026 00:47:34 +0000

From ReAct loops to production multi-agent systems everything in one place, nothing left out.

Somewhere between the fifteenth “AI agent” LinkedIn post and the third vendor announcing their autonomous workflow platform, I stopped nodding along and started asking the obvious question nobody seemed to want to answer:

What is actually running here?

Because the word “agent” has been doing a lot of heavy lifting lately. It’s been stretched over chatbots with a search button, over multi-step pipelines glued together with vibes, and over genuinely sophisticated systems that can decompose a task, call external APIs, reflect on their own output, and course-correct all without you touching a keyboard. Those are not the same thing. Not even close.

I’ve spent the last few months going deep on this. Built agents that failed in embarrassing ways. Read the research, the docs, the Reddit threads where someone’s agent looped itself into a $600 API bill at midnight. Took the courses. Talked to people shipping this stuff in production at scale. And I distilled all of it down into this article.

This is not a hype piece. There are no screenshots of a ChatGPT conversation doing something mildly impressive. This is the full picture from the first-principles question of what an agent actually is, through the design patterns that separate demos from real systems, all the way to the production concerns that nobody tweets about: evaluation, latency, cost, observability, and security.

Whether you’re just trying to understand what your team keeps talking about in standups, or you’re actively building agent systems and hitting walls, this is the article you keep open in a tab and come back to.

TL;DR: An AI agent is an LLM inside a loop, equipped with tools, memory, and decision-making logic. Building one that demos well takes an afternoon. Building one you’d actually trust with real work takes a different mindset closer to distributed systems design than prompt engineering. This article covers both ends of that spectrum and everything in between.

What an agent actually is

If you’ve used ChatGPT to write an email, you’ve used an LLM. You gave it a prompt, it gave you an output, done. One shot. Linear. The model doesn’t remember what it did, doesn’t check its own work, doesn’t go looking for missing information. It just generates the next most likely tokens until it hits the end and stops.

An agent is what happens when you take that same model and put it inside a loop.

Instead of prompt-in, response-out, you get something closer to how a human actually tackles a non-trivial task. You plan a little. You gather some information. You do a first pass. You read it back, notice what’s wrong, and fix it. You check one more thing. You finish. That back-and-forth, that iterative reasoning over multiple steps that’s the core of what makes something an agent rather than a fancy autocomplete.

The technical name for this loop is the ReAct pattern: Reason, Act, Observe, repeat. The model reasons about what to do next. It acts usually by calling a tool, running a search, querying a database, executing some code. It observes the result. Then it either gives you a final answer or loops back to reason again based on what it just learned. That cycle is the engine underneath almost every agent system you’ll encounter, from the simplest LangChain pipeline to Claude Code rewriting your entire test suite.

Here’s what makes this more powerful than it sounds. Each pass through the loop adds depth. The model isn’t trying to solve everything in one shot under the pressure of a single context window it’s allowed to work iteratively, to gather information it didn’t have at step one, to catch mistakes it made in step two. The output at the end of three loops is almost always better than the output of one. Not because the model got smarter, but because the architecture gave it room to think.

The practical upside of this shows up immediately in tasks that need accuracy and sourcing. Legal research where you have to cite specific cases. Customer support that requires pulling account details before responding. Code generation that needs to run the code, read the error, and try again. Any domain where a single-pass answer is almost certainly incomplete or wrong that’s where agents earn their keep.

Here’s the mental model I use: a regular LLM call is a consultant who reads your brief once and writes a report on the plane. An agent is that same consultant who actually does the research, drafts something, reads it back, realizes they missed a key detail, goes and finds it, then rewrites the section. Same intelligence, different process. The process is what changes the output quality.

One thing worth clearing up early: the model itself isn’t what makes something an agent. You can build a mediocre agent on GPT-4 and a great one on a smaller, faster model with a well-designed loop and the right tools. The architecture and the task decomposition matter more than the leaderboard position of the underlying LLM. Remember that when someone tries to sell you on “the most agentic model” the model is one part of the system, not the whole thing.

The core building blocks

Before you write a single line of agent code, you need to understand four things. Get these right and everything else becomes easier to reason about. Get them wrong and you’ll spend weeks debugging behavior that feels random but isn’t.

Context is the agent’s entire world. Whatever isn’t in the context window doesn’t exist as far as the model is concerned. Context engineering deciding what goes in there is one of the most underrated skills in agent development. It includes the task description, the agent’s role, any memory from previous steps, available tools, and relevant background knowledge. A poorly engineered context produces an agent that hallucinates, repeats itself, or completely ignores the instructions you thought were obvious. Most agent bugs aren’t model bugs. They’re context bugs.

Memory comes in two flavors. Short-term memory is what the agent writes down as it works intermediate results, tool outputs, notes to itself. Long-term memory is lessons from previous runs, stored and loaded at the start of each new task. The combination is what lets an agent improve over time rather than starting from zero on every execution. Knowledge is different from both it’s static reference material you load upfront. Documentation, PDFs, database access. The agent reads from it but doesn’t update it.

Task decomposition is the part nobody talks about enough. The rule is simple: break each step down until a single LLM call or a single tool can handle it cleanly. If a step is too big, the output gets sloppy. The exercise is to think about how you’d do the task yourself what are the actual discrete steps then figure out which of those steps map to an LLM call, which map to a tool call, and which map to a bit of regular code. When something isn’t working, nine times out of ten a step is too coarse.

Guardrails are the bouncer at the door. Because LLMs are non-deterministic, you can’t assume the output will always be in the right format, the right length, or factually consistent with the sources the agent just retrieved. Guardrails are the layer that catches these failures before they reach the user or before they get passed to the next step in the pipeline and silently corrupt everything downstream. Some guardrails are just code: check the output format, validate the schema, enforce length limits. Others use a second LLM to judge quality. And sometimes the right guardrail is a human checkpoint especially for anything irreversible.

Four concepts. Everything else in agent design is built on top of them.

Four design patterns that actually work

Once your building blocks are solid, the next question is how you structure the actual behavior of the agent. There are four patterns that show up in almost every serious agent system. You don’t always need all four, but you need to know all four.

Reflection

The simplest and most effective upgrade you can make to any agent. Instead of shipping the first output, the agent critiques it and rewrites it.

The model produces something, reads it back with a prompt like “what’s wrong with this and how would you fix it,” then revises. That second pass almost always improves the result not because the model is smarter on round two, but because reviewing is an easier cognitive task than generating from scratch. You’re offloading the hard part across two steps instead of cramming it into one.

Reflection is especially powerful when you can add external feedback to the loop. Write code, run it, feed the error back, try again. Generate JSON, validate it against a schema, send the validation errors back if it fails. That concrete feedback signal is what separates reflection from just asking the model to “try harder.”

The tradeoff is latency and cost you’re doing multiple passes. Test with and without it before you commit.

Tool use

An LLM by itself is a text generator. It doesn’t know what time it is, can’t query your database, can’t run code, and has no idea what’s in your company’s internal docs. Tools fix that.

You define a set of functions web search, database query, code execution, calendar access, whatever your use case needs and the model decides when and which ones to call. Under the hood, the model doesn’t actually execute anything. It outputs a structured request, your code runs the function, and the result gets fed back into the context. The model uses that result to continue.

Well-designed tools have clear names, plain-English descriptions of when to use them, typed input schemas, and clean error handling. Think of them as an API your agent uses. Document them like one.

Planning

Instead of following a hardcoded sequence of steps, the agent decides what to do and in what order.

You give it a toolkit, prompt it to create a step-by-step plan, and execute that plan running each tool, feeding results back, and repeating until the task is done. The model acts as its own project manager. This is powerful for tasks where you can’t anticipate every possible path upfront, like a customer service agent handling wildly different request types.

The catch: more autonomy means more unpredictability. Planning agents need tight guardrails, permission checks, and good logging. The strongest current use case is agentic coding systems where the task space is well-defined even if the exact steps aren’t.

Multi-agent collaboration

Some tasks are too complex, too long, or too varied for one agent to handle well. The answer is the same one humans figured out a long time ago: build a team.

Each agent gets a specific role and only the tools that role needs. A researcher agent does web search and retrieval. A writer agent handles drafting. A reviewer agent checks quality. A manager agent coordinates the others. Specialization produces better output than one generalist trying to do everything inside a single sprawling context window.

The coordination patterns range from simple sequential handoffs researcher finishes, passes to writer, writer passes to reviewer to parallel execution where independent agents run simultaneously and merge results. Most production systems start sequential and add parallelism only where latency actually matters.

Multi-agent systems are not the default answer. They add real complexity: agents can conflict, communication overhead adds up, and debugging a failure that happened three agents deep is genuinely painful. Start with one agent. Add a second only when the first one has a clear ceiling it can’t break through.

Shipping to production

This is the section that doesn’t make it into the demo videos. Everything up to this point gets you a working agent. This is what gets you a trustworthy one.

Evaluate before you optimize

The most common mistake people make with agents is trying to improve something they haven’t measured. Before you touch a prompt, swap a model, or restructure a pipeline, you need to know what’s actually failing and how often.

Some evals are simple.

Does the customer service agent correctly identify whether an item is in stock?

That’s a pass/fail check you can automate. Others are harder is this research report actually good? For those, use a second LLM as a judge. Give it a consistent rubric, have it score outputs on a 1–5 scale, and track that score across runs.

Evaluate at two levels. Component-level tells you which specific step is underperforming. End-to-end tells you whether the final output is actually good. If end-to-end scores are low but every component scores fine, the problem is in the handoffs between steps that’s a different fix than a bad prompt.

Start evaluating on day one. An imperfect eval that exists beats a perfect eval you’re still designing.

Latency and cost are the same problem

Every extra LLM call costs time and money. In agent systems, those calls stack up fast.

The fix is the same for both: measure each step, then attack the biggest buckets. Parallelize anything that doesn’t depend on the step before it multiple web searches, multiple document fetches, multiple sub-tasks that can run simultaneously. Right-size your models use a smaller, faster model for simple steps like keyword extraction or format validation, and reserve the expensive one for actual reasoning. Cache aggressively search results, embeddings, intermediate summaries. If the input is identical, don’t recompute.

One research agent run might cost a few cents. At a thousand runs a day that’s hundreds of dollars a month. Know your per-run cost before you scale.

Log everything, assume nothing

Traditional software fails with stack traces. Agent systems fail silently the output looks plausible, the logs show no errors, and something is still wrong.

Observability for agents means tracing every decision: what did the agent plan to do, what tool did it call, what came back, what did it decide next. Tools like LangSmith and Weights & Biases are built for exactly this. When something breaks and it will you want to be able to replay the exact sequence of steps that produced the bad output and see precisely where it went sideways.

Beyond individual traces, track aggregate metrics over time. Hallucination rate. Task success rate. Average cost per run. These trend lines tell you whether your changes are actually helping or just moving the problem around.

Sandbox your code execution

If your agent can write and run code and the useful ones usually can you need to treat that capability like a loaded gun. Run all code in an isolated container that gets destroyed after each execution. Set hard timeouts and memory limits. Whitelist the libraries it’s allowed to use. Never let agent-generated code write to anywhere that matters or reach the network unless you explicitly decided it should.

The failure mode here isn’t theoretical. An agent with unrestricted code execution and a bad prompt is a very expensive, very fast way to ruin your afternoon.

Production isn’t a different version of your demo. It’s a different discipline entirely.

The job changed. Most people haven’t caught up yet.

Here’s the take I’ll leave you with, and you can disagree with me in the comments: the bottleneck in AI development right now isn’t the models. The models are good. The bottleneck is engineers who understand how to build reliable systems around them.

Prompting was never the skill. It was always the entry point. The actual work designing context, decomposing tasks, wiring tools, evaluating outputs, controlling costs, tracing failures that’s systems design. It always was. The wrapper just changed.

The developers who are going to do interesting things with agents in the next few years aren’t the ones who found the best jailbreak or the cleverest chain-of-thought trick. They’re the ones who treat agents the way they treat any other distributed system: with logging, with testing, with failure modes they planned for, with an understanding of what happens when one component does something unexpected.

That’s not a pessimistic take. If anything it’s the opposite. It means the skills you already have debugging, systems thinking, knowing when to add complexity and when not to transfer directly. You’re not starting from zero. You’re applying what you know to a new kind of component.

Agents are going to keep getting more capable, more autonomous, and more embedded in real workflows. The tooling is improving fast. The patterns are stabilizing. This is a good time to actually understand the stack rather than just use the abstraction on top of it.

Build something small. Evaluate it honestly. Add one pattern at a time. Log everything from day one. That’s the whole playbook.

Helpful resources

Anthropic’s guide to building effective agents the clearest first-principles breakdown of agent design patterns available
LangChain documentation practical starting point for building agent pipelines in Python
LangSmith tracing and evaluation tooling built specifically for LLM applications
Building an agentic system deep technical breakdown of how tools like Claude Code are architected under the hood
Weights & Biases production monitoring and experiment tracking for ML systems
OpenAI Cookbook agent examples real code examples for tool use, multi-agent patterns, and evals
r/LocalLLaMA where practitioners actually talk about what’s working and what isn’t

Go vs Rust: the only backend language debate that actually matters in 2026

Thu, 14 May 2026 08:02:23 +0000

You don’t need to pick one. You need to know which fight each one was built for.

There’s a certain kind of developer who treats language choice like a religion.

Go devs will tell you Rust is overkill.

Rust devs will tell you Go is for people who don’t understand memory.

Both are partially right. Both are completely missing the point.

Here’s the thing: choosing between Go and Rust in 2026 isn’t really a language debate anymore. It’s a system design decision. And most of the hot takes you’ll find online are arguing about the wrong thing comparing raw benchmarks and borrow checker frustration instead of asking the actual question:

Where in your architecture does this choice matter, and when does it stop mattering?

I’ve watched teams rewrite entire Go services in Rust because a benchmarks blog post made someone nervous. I’ve also watched Rust codebases grind sprint velocity to a halt because the team picked the wrong tool for a job that really just needed a couple of goroutines and a coffee break.

Neither is winning. Both are expensive.

The honest answer in 2026 is boring:

Go and Rust aren’t competing. They’re slotting into different layers of the same system. One builds the thing. The other makes the thing survive.

Understanding which layer needs which tool is the only skill that actually pays out.

TL;DR: Go is your default. Fast enough, ships fast, scales horizontally, and your team won’t hate you for picking it. Rust enters the picture when specific parts of your system hit a wall that Go can’t resolve without throwing more money at AWS. The debate isn’t Go or Rust it’s Go first, Rust where it earns its keep.

Go: the default loadout every backend team starts with

There’s a reason Go became the lingua franca of cloud-native backend development.

It’s not because Google has a good marketing department.

It’s because Go made a specific, opinionated bet:

Developer velocity and system predictability matter more than raw performance.

For most production systems, that bet pays out every single time.

Think of it like picking your starter in an RPG. Go is balanced stats across the board. Not the highest damage output, not the tankiest build but the one that gets you through 80% of the game without hitting a wall. You pick it, you learn the basics, and you’re shipping features by the end of the week.

The concurrency model is where Go genuinely earns its reputation.

Goroutines are cheap enough to spin up thousands without sweating memory. The channel model gives you a mental framework for concurrent work that doesn’t require a degree in threading theory to reason about.

Building an SQS consumer that fans out to 20 parallel workers?

That’s an afternoon in Go.

Writing a gRPC service sitting behind an ALB on ECS?

Go makes that boring in the best possible way. And boring is exactly what you want in a production system .

The AWS ecosystem leans into this hard:

Fast startup times mean tighter Lambda cold starts
Low memory footprint means cheaper ECS containers
The AWS SDK for Go v2 covers everything from S3 to EventBridge without fighting you

The broader ecosystem is settled too. Gin and Chi for HTTP routing, sqlc for type-safe queries, Wire for dependency injection if that’s your thing. The compiler errors are readable. Onboarding a new engineer onto a Go codebase takes days, not weeks.

That last point matters more than most architecture blogs admit.

The best language for your system is the one your team can debug at midnight without wanting to quit their job. Go clears that bar repeatedly.

Where Go starts showing cracks is predictable if you know where to look.

The garbage collector has improved dramatically but GC pauses are still a reality in latency-sensitive workloads. Memory efficiency plateaus when you’re doing genuinely CPU-heavy work. And if you’re processing high-throughput data streams where every microsecond of predictability matters, Go’s runtime is making decisions for you that you might not agree with.

That’s not a criticism. That’s Go doing exactly what it was designed to do.

The tradeoff is explicit. Most teams never hit the ceiling.

The ones that do need a different tool for that specific corner of the system.

That tool has a crab mascot and a notoriously opinionated compiler.

Rust: the late-game unlock you didn’t know you needed

Nobody picks up Rust on day one.

You come to Rust after you’ve shipped something. After you’ve scaled something. After you’ve sat in an incident review trying to explain why your Go service started spiking p99 latency at a completely unpredictable interval and the only honest answer was “the GC decided it was time.”

That’s the origin story for most Rust adoption in production. Not ideology. Not a rewrite-everything agenda. Just a specific part of the system that stopped behaving and needed a tool with harder guarantees.

Think of it like unlocking a late-game weapon in an RPG. You don’t get it at the start. You earn it. And once you have it, you don’t use it on every enemy you save it for the boss fights.

The core promise of Rust is control.

No garbage collector means no GC pauses. No runtime surprises. What you write is what runs, with predictable memory behavior from the first request to the millionth. The borrow checker the thing everyone complains about until they don’t is just the compiler enforcing that promise at build time instead of letting it blow up in production at 3am.

In AWS terms, this matters in very specific places:

High-throughput Kinesis or Kafka consumers where processing latency compounds
Lambda functions where cold start time and memory ceiling are both constrained
ECS services doing CPU-heavy transformation work on tight instance budgets
Real-time fraud detection or risk scoring pipelines where a GC pause is a business problem

The async ecosystem has matured to the point where this is actually enjoyable to build now. Tokio is the async runtime most production Rust services are built on, and Axum gives you an ergonomic HTTP layer that won’t make you miss Go’s simplicity quite as much as you’d expect.

Crates.io has filled in the gaps that used to make Rust feel incomplete for backend work. There’s a crate for almost everything now serialization, database access, observability, AWS integrations. It’s not Go’s ecosystem in terms of breadth, but it’s not the frontier territory it was three years ago either.

The WASM angle is worth mentioning too.

Rust compiles to WebAssembly better than almost anything else, which opens up edge compute scenarios Cloudflare Workers, Fastly Compute where you want near-native performance in a serverless container with a sub-millisecond cold start. Go can do this too, but Rust’s output is leaner and the toolchain support is stronger.

The honest cost of Rust is team velocity at least upfront.

The borrow checker has opinions. Strong ones. Loudly expressed. Onboarding an engineer who hasn’t written Rust before is a multi-week investment, not a multi-day one. Code review takes longer. Simple things that would take an hour in Go can take a morning in Rust while you negotiate with the compiler about who owns what.

But here’s the flip side nobody puts in the benchmarks post:

Once the code compiles, it tends to just work.

Not “works in staging” work. Not “works until load testing” work. The class of bugs that Rust’s type system eliminates at compile time null pointer dereferences, data races, use-after-free are the exact bugs that cause 2am incidents in Go and every other language that trusts you more than the compiler does.

The tradeoff isn’t really speed vs safety. It’s upfront cost vs long-term stability.

For the right part of your system, that’s an easy trade.

Real architecture patterns: how teams actually use both

Here’s what nobody tells you in the language comparison posts:

Most production systems that use Rust don’t replace Go.

They add Rust to specific coordinates in the architecture where Go stopped being enough. The rest stays Go. The team keeps shipping. Nobody rewrites everything and nobody has an existential crisis about the stack.

These are the three patterns that show up repeatedly in real systems.

Pattern 1: Go orchestrates, Rust crunches

This is the most common one and the cleanest.

Go handles everything that talks to other things the API layer, the service-to-service communication, the SQS consumers that fan out work, the schedulers that trigger jobs. It’s the connective tissue of the system. Fast enough, easy to reason about, easy to change.

Rust handles the part that actually does the heavy lifting.

A real example: a data pipeline that ingests raw events from Kinesis, runs enrichment and scoring logic, and writes results to DynamoDB. The Go service manages the consumer group, handles retries, and pushes work into a processing queue. The Rust binary does the actual scoring CPU-bound, memory-intensive, latency-sensitive. Two languages, one coherent system, each doing the job it was built for.

Pattern 2: Cost optimization at scale

This one sneaks up on you.

Your Go services are fast enough. Response times are fine. Users aren’t complaining. But the AWS bill is climbing faster than your traffic is growing, and when you dig in, you find a handful of services that are just eating CPU and memory disproportionately.

That’s the cost optimization signal.

Rewriting those specific services in Rust not everything, just the ones burning resources can drop memory footprint significantly and increase throughput per instance. Fewer instances needed. Smaller instance sizes. Same traffic handled for less money.

At low scale this is a rounding error. At high scale this is a conversation your CFO notices.

The Benchmarks Game numbers aren’t perfectly representative of production workloads, but the directional signal is real: Rust is consistently 2–5x more efficient than Go on CPU-bound work, and the memory story is even more pronounced.

Pattern 3: Latency-sensitive systems where GC pauses are a product problem

Some systems can’t tolerate unpredictability at the tail.

Financial systems where a p99 spike causes a missed execution window. Voice and video processing where a pause means a glitch the user hears. Real-time analytics dashboards where a stall breaks the illusion of live data.

In these cases Go’s GC even with tuning introduces a variance floor that you can’t engineer away. You can shrink it. You can schedule around it. You can’t eliminate it.

Rust eliminates it.

No runtime. No collector. The memory behavior of a Rust service under load is the same as the memory behavior of a Rust service at rest deterministic, predictable, boring in exactly the way your SLA needs it to be.

This isn’t about raw speed. It’s about the shape of the latency distribution. Go might average faster on a given workload and still lose this comparison because the tail is worse.

The common thread across all three patterns is the same:

You don’t choose Rust instead of Go. You choose Rust for the specific part of the system where Go’s tradeoffs stop working in your favor. Everything else stays exactly as it was.

Build with Go. Optimize with Rust. Ship both.

The wall: when Go stops being enough

Every backend system hits a wall.

At first, everything is fine.

You spin up services, deploy to ECS, wire up queues, add a scheduler, scale horizontally things just work. APIs respond fast enough. The team ships features. The infra bill is acceptable.

Then growth happens.

Traffic increases
Latency spikes in weird places
Costs start climbing faster than usage
Debugging gets harder
Small inefficiencies compound into real problems

And suddenly the question changes.

It’s no longer:

“How fast can we build this?”

It becomes:

“How do we keep this system predictable, efficient, and scalable without slowing the team down?”

That’s where the Go vs Rust decision actually starts to matter. Not at the beginning. Right when you hit that wall.

How to know if you’ve actually hit it

Before you start a Rust migration conversation, run this check first.

Most systems that feel slow aren’t CPU-bound. They’re waiting on databases, on network calls, on downstream services that are slower than they should be. Rewriting a Go service in Rust doesn’t fix a Postgres query that’s missing an index. It doesn’t fix an N+1 problem in your ORM. It doesn’t fix a Lambda that’s cold-starting because you gave it 128MB of memory and called it done.

Profile before you decide.

If your bottleneck is I/O and it usually is Go is not your problem. Fix the query. Add the cache. Right-size the instance. Go home.

If your bottleneck is genuinely CPU sustained high utilization, processing-bound work, memory pressure that doesn’t resolve with horizontal scalingthen you have a real signal. That’s the wall. That’s where Rust earns the conversation.

The team cost is real too

Here’s the part that gets skipped in every benchmark post.

Introducing Rust into a Go shop isn’t free. You’re adding a second language to your codebase, which means:

Two build pipelines to maintain
Two sets of idioms for new engineers to learn
Code review that requires Rust-literate reviewers
A slower ramp for anyone joining the team fresh

None of that is disqualifying. All of it is real. The question is whether the performance gain in the specific component you’re optimizing is worth the ongoing operational tax across the whole team.

For most teams, the answer is: yes, but only for a small slice of the system.

The quick reference

The actual decision

Use Go to build systems.

Use Rust to optimize systems.

Start in Go. Ship in Go. Run in Go. When a specific component starts costing you more than it should in latency, in money, in stability isolate it. Profile it. And if the data points at a genuine CPU or memory problem that Go can’t resolve without throwing more hardware at it, that’s your Rust entry point.

Don’t migrate the whole system. Rewrite the one service that earned it.

Then go back to shipping in Go.

Conclusion: Go builds companies, Rust survives them

Here’s the take nobody wants to say out loud:
Most teams that argue about Go vs Rust don’t have a language problem. They have a prioritization problem. The system isn’t slow because of the runtime. It’s slow because of the decisions made three sprints ago that nobody had time to revisit.

Language choice is downstream of system design. Always.

Go became the default backend language of the cloud-native era not because it’s the fastest or the most elegant or the most technically impressive thing you can put on a resume. It became the default because it consistently produces systems that teams can build quickly, reason about clearly, and operate without a dedicated platform engineering team just to keep the lights on.

That’s genuinely hard to beat.

Rust earns its place in the stack the same way any good tool earns its place by solving a specific problem better than everything else available. Not because it’s newer. Not because the crab mascot is charming. Because when you need deterministic memory behavior, zero GC overhead, and the kind of compile-time guarantees that let you sleep through the night without a PagerDuty notification, nothing else in the backend ecosystem comes close.

The developers who will win in the next few years aren’t the ones who picked a side in this debate. They’re the ones who got comfortable moving between both who can ship a Go service on a Tuesday and drop into a Rust codebase on a Thursday without losing a step.

Polyglot isn’t a buzzword anymore. It’s a survival skill.

The question was never Go or Rust. It was always Go and Rust, applied with enough discipline to know which one the problem actually needs.

Pick the tool that fits the job. Ship the thing. Move on.

And if someone on your team is still writing LinkedIn posts about which language is objectively better in 2026 send them this article and go touch grass.

What do you think? Is your team running both in production, or did you go all-in on one? Drop your take in the comments I read every one.

Helpful resources

A Tour of Go best starting point if you’re new to Go
The Rust Book the official, actually-readable Rust guide
Tokio async runtime where most production Rust backend work lives
Axum web framework ergonomic HTTP for Rust services
AWS SDK for Go v2 the one you actually want to use
The Benchmarks Game real language performance comparisons, not blog post numbers
Crates.io Rust package registry, wa more mature than it used to be
Gin HTTP framework Go’s most popular web framework

Junior devs catch exceptions. Senior devs prevent them. Here’s the difference.

Wed, 13 May 2026 03:09:14 +0000

Four patterns that replace most of the try-catch in your codebase and the mental model that makes the right choice obvious.

Every developer has written this code. You’re building something, you get a compiler warning, or maybe a runtime blow-up in testing, and the fastest fix feels obvious: wrap it. Slap a try-catch around it, log the message, return null, move on. Done. Safe.

Except it isn’t safe. You’ve just installed a smoke detector that beeps once and then goes quiet forever.

I’ve been inside codebases where try-catch was the default answer to every uncomfortable question the compiler asked. Controllers wrapped in it. Services wrapped in it. Utility methods wrapped in it. The team described it as “defensive programming.” What it actually was: a distributed system of silence. Errors happened. Nobody knew. The first sign of a problem was usually a customer email.

There’s a reason tutorials teach try-catch first. It works, in the narrowest possible sense. The app doesn’t crash. The stack trace disappears. The demo runs clean. What the tutorial doesn’t show you is the 3am incident six months later when a payment silently failed for two hundred users and your logs say "Something went wrong: null".

Senior devs don’t write more try-catch than juniors. They write significantly less. Not because they’re reckless, but because they’ve learned to ask a different question before reaching for it: can I stop this from going wrong in the first place?

This article is about the four patterns that replace most of the try-catch you’re writing. Each one handles a real category of failure invalid input, unclear errors, duplicated handling logic, and expected outcomes that aren’t actually exceptional. Together they form a mental model that changes how you read code, not just how you write it.

TL;DR: try-catch is a reactive tool. Most of the time, the right move is to prevent, name, centralize, or model the failure not catch it in silence.

Why junior devs default to try-catch

Here’s the thing: defaulting to try-catch isn’t stupidity. It’s pattern recognition from incomplete data.

You learn Java, Python, or whatever backend language the bootcamp or degree threw at you. The first exception handling example looks like this: something might fail, so you wrap it, you catch it, you print the error, you move on. The tutorial works. You internalize the shape of the solution. From that point forward, any time the compiler complains or the runtime blows up, your brain pattern-matches straight to the familiar shape.

Stack Overflow reinforces it. You search for your error, the accepted answer has a try-catch, it has 847 upvotes, someone accepted it in 2014, and it fixes your immediate problem. You copy it. You ship it. Repeat a few hundred times across a year of coding and you’ve built a reflex.

The problem isn’t the reflex. The problem is what it actually does to your codebase.

Every time you wrap code in try-catch, you’re making an implicit declaration: I don’t know what’s going to go wrong here, so I’ll just intercept whatever happens and deal with it. That sounds humble. It isn’t. It’s actually abdication. You’ve handed the decision to runtime and told it to figure things out while you look away.

Think of it this way: try-catch is a bucket under a leaky pipe. It catches the drip. It keeps the floor dry for now. But the pipe is still leaking, the water is still going somewhere it shouldn’t, and the leak is getting slightly worse every week. The senior move is to call a plumber. The junior move is to empty the bucket on a cron job and call it fixed.

Senior developers ask a different question before they reach for try-catch:

should this be able to fail at all?

If invalid input causes the failure is the input even allowed in?

If a null reference causes the crash why is null a valid state here?

The goal isn’t to catch the explosion. It’s to make the explosion impossible, or at least to make it impossible to ignore.

That reframe is where the four patterns come from. Each one answers a version of that question for a different category of failure.

Pattern 1: validate first, catch never

The single most common reason junior devs reach for try-catch is bad input. A null slips through. A string arrives where a number was expected. An email field is missing. Something downstream blows up, the catch block fires, and the problem disappears into a log message nobody reads.

Here’s the thing though: none of that is an exception handling problem. It’s a validation problem. The failure didn’t happen because the system encountered something genuinely unexpected it happened because you let garbage through the front door and were surprised when it broke something in the kitchen.

I spent three hours debugging a production issue once that turned out to be a null email field on a user registration request. Three hours. The stack trace was unhelpful, the catch block had swallowed the context, and the only log entry was something like "User creation failed". The fix itself took forty seconds once I found it. The try-catch didn't protect anything it just made the debugging slower and more painful.

The senior pattern here is straightforward: validate at the boundary. Nothing invalid gets past the entry point, so nothing downstream needs to defend against it.

In a Spring application this looks like replacing a defensive catch block with a @Valid annotation and letting the framework enforce the contract:

// what fear-based programming looks like
@PostMapping("/users")
public ResponseEntity<User> createUser(@RequestBody UserRequest request) {
    try {
        User user = userService.create(request);
        return ResponseEntity.ok(user);
    } catch (NullPointerException e) {
        return ResponseEntity.badRequest().body(null);
    } catch (IllegalArgumentException e) {
        return ResponseEntity.badRequest().body(null);
    }
}

// what confidence looks like
@PostMapping("/users")
public ResponseEntity<User> createUser(@valid @RequestBody UserRequest request) {
    User user = userService.create(request);
    return ResponseEntity.status(HttpStatus.CREATED).body(user);
}

public class UserRequest {
    @notblank(message = "Name is required")
    private String name;
    @Email(message = "Must be a valid email")
    @NotNull(message = "Email is required")
    private String email;
    @min(value = 18, message = "Must be at least 18")
    private int age;
}

The second version is shorter, cleaner, and more honest. The constraints live on the data object itself, which means any developer who opens UserRequest immediately knows the rules. There's no hidden catch block somewhere else in the call stack silently absorbing violations. If the input is bad, the framework rejects it immediately with a clear message before it touches a single line of your business logic.

This pattern isn’t Spring-specific either. Django has form and serializer validation. Express has middleware validators. Laravel has request classes. Every mature backend framework has a first-class answer to this problem, and that answer is never “catch the NullPointerException deeper in the stack.”

The rule worth internalizing: if you’re catching an exception caused by bad input, you don’t have an exception handling problem. You have a validation problem. Fix it upstream, not downstream.

Pattern 2: build a custom exception hierarchy

So you’ve validated your input. Clean data is flowing through the system. But things still go wrong because they always do. A record doesn’t exist. A business rule gets violated. An order is already processed. These are real failures, and they need real handling.

The junior response to this is catch (Exception e). Log the message. Return a 500. Hope the message is descriptive enough to debug later. It never is.

I’ve been in that 3am incident call. The alert fires, you pull up the logs, and staring back at you is: "Something failed: null". That's it. No context. No indication of which service, which record, which rule. Just a generic catch block somewhere in the stack that swallowed everything meaningful and handed you nothing. You start adding log statements, redeploying, waiting for it to happen again. It is not a fun way to spend a night.

The problem is that Exception is the nuclear option. It catches everything the failure you expected, the one you didn't, the one that's actually a bug in a completely different part of the system. When you catch everything, you lose the ability to distinguish between any of it.

Senior devs build a hierarchy instead. You start with a base application exception, then extend it into specific types that are self-describing:

// base — everything app-specific extends this
public class AppException extends RuntimeException {
    private final HttpStatus status;
    private final String errorCode;

public AppException(String message, HttpStatus status, String errorCode) {
        super(message);
        this.status = status;
        this.errorCode = errorCode;
    }
}
// specific types that speak for themselves
public class NotFoundException extends AppException {
    public NotFoundException(String resource, Long id) {
        super(resource + " not found with id: " + id,
              HttpStatus.NOT_FOUND, "NOT_FOUND");
    }
}
public class BusinessRuleViolatedException extends AppException {
    public BusinessRuleViolatedException(String rule) {
        super("Business rule violated: " + rule,
              HttpStatus.CONFLICT, "BUSINESS_RULE_VIOLATED");
    }
}

Now your service code becomes readable without a single try-catch in sight:

public Order processOrder(OrderRequest request) {
    Order order = orderRepository.findById(request.getOrderId())
        .orElseThrow(() -> new NotFoundException("Order", request.getOrderId()));

if (order.isAlreadyProcessed()) {
        throw new BusinessRuleViolatedException("Order already processed");
    }
    return orderRepository.save(order);
}

When NotFoundException gets thrown anywhere in the system, you already know what failed, why it failed, and what HTTP status to return. You don't need to grep through logs trying to reconstruct context. The exception is the context.

Think of it like hospital triage codes versus a nurse shouting “something’s wrong with someone.” Triage codes exist because specificity saves time when time costs lives. Your exception hierarchy exists for the same reason specificity saves debugging time when production is on fire.

There’s also a less obvious benefit here: a good exception hierarchy forces you to think about your failure modes upfront. When you’re defining BusinessRuleViolatedException, you're implicitly cataloguing what your business rules actually are. That clarity leaks into your design in good ways.

The rule: if you’re catching a generic exception and then trying to figure out what actually went wrong from the message string, your exceptions aren’t specific enough. Name the failure. Make it impossible to confuse with anything else.

Pattern 3: handle everything in one place

You’ve got clean input coming in. You’ve got specific, named exceptions being thrown. Now the question is: where do you actually handle them?

The junior answer is in every controller. You write a try-catch in the first endpoint, it works, you copy the pattern to the second endpoint, and the third, and the fifteenth. Six months later you have a codebase where every controller method is forty lines long and thirty of those lines are catch blocks doing nearly identical things. Changing the error response format means touching fifteen files. Adding a new exception type means hunting through every controller to add another catch branch. It compounds fast.

This is the copy-paste catch block epidemic, and almost every team above a certain age has lived through it.

The senior pattern collapses all of that into one place. In Spring, that’s @RestControllerAdvice a single global handler that intercepts every unhandled exception from every controller in the application:

@RestControllerAdvice
public class GlobalExceptionHandler {
@ExceptionHandler(NotFoundException.class)
    public ResponseEntity<ErrorResponse> handleNotFound(NotFoundException e) {
        return ResponseEntity.status(HttpStatus.NOT_FOUND)
            .body(new ErrorResponse(e.getErrorCode(), e.getMessage()));
    }

    @ExceptionHandler(BusinessRuleViolatedException.class)
    public ResponseEntity<ErrorResponse> handleBusinessRule(
            BusinessRuleViolatedException e) {
        return ResponseEntity.status(HttpStatus.CONFLICT)
            .body(new ErrorResponse(e.getErrorCode(), e.getMessage()));
    }

    @ExceptionHandler(MethodArgumentNotValidException.class)
    public ResponseEntity<ErrorResponse> handleValidation(
            MethodArgumentNotValidException e) {
        String details = e.getBindingResult().getFieldErrors().stream()
            .map(error -> error.getField() + ": " + error.getDefaultMessage())
            .collect(Collectors.joining(", "));
        return ResponseEntity.status(HttpStatus.BAD_REQUEST)
            .body(new ErrorResponse("VALIDATION_FAILED", details));
    }

    @ExceptionHandler(Exception.class)
    public ResponseEntity<ErrorResponse> handleUnexpected(Exception e) {
        logger.error("Unexpected error", e);
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
            .body(new ErrorResponse("INTERNAL_ERROR", "Something went wrong"));
    }
}

And here’s what your controllers look like after:

@GetMapping("/orders/{id}")
public Order getOrder(@PathVariable Long id) {
    return orderService.findById(id);
}

That’s it. Three lines. The controller does exactly one thing: call the service and return the result. If NotFoundException gets thrown, the global handler catches it, formats it correctly, and returns the right HTTP status. The controller never needed to know about any of that.

This is the bouncer analogy applied to architecture. You don’t put a bouncer at every table in a restaurant. You put one at the door. One point of entry, one set of rules, consistently enforced. Your GlobalExceptionHandler is the door. Every exception has to pass through it, which means every exception gets handled the same way, every time, from one file you can actually reason about.

The practical benefits stack up quickly. Want to change your error response format? One file. Adding a new custom exception type? One new method in the handler. Need to add request ID tracking to every error response for distributed tracing? One change, instantly applied everywhere. What used to be a fifteen-file refactor becomes a three-line addition.

The framework doesn’t matter much here either the pattern exists everywhere. Django has exception middleware. Express has error-handling middleware with the four-argument signature. Laravel has the Handler class in app/Exceptions. The idea is universal: centralize your exception handling, stop duplicating it.

The rule: if you’re writing the same catch logic in more than one place, you need a centralized handler. Every duplicate catch block is future maintenance debt you’re taking out a loan on right now.

Pattern 4: result objects for expected failures

This is the pattern that tends to produce the most argument in code reviews, which is usually a sign it’s worth understanding properly.

The premise is simple: not every failure is exceptional. “User not found” isn’t a system error it’s a Tuesday. “Payment declined” isn’t a catastrophe it’s an expected outcome of the payment flow. “Item out of stock” isn’t a bug it’s a valid state your business logic needs to handle gracefully.

When you model these as exceptions, you’re using the wrong tool. Exceptions exist for genuinely unexpected conditions things that shouldn’t happen under normal operation. Using them for routine business outcomes is like calling an ambulance because you got a parking ticket. Technically it gets the job done, but it’s expensive, it misleads everyone involved, and it trains the system to treat normal outcomes as emergencies.

The result object pattern makes expected outcomes explicit in the return type instead:

public class Result<T> {
    private final T value;
    private final String error;
    private final boolean success;

private Result(T value, String error, boolean success) {
        this.value = value;
        this.error = error;
        this.success = success;
    }

    public static <T> Result<T> ok(T value) {
        return new Result<>(value, null, true);
    }

    public static <T> Result<T> failure(String error) {
        return new Result<>(null, error, false);
    }

    public boolean isSuccess() { return success; }
    public T getValue() { return value; }
    public String getError() { return error; }
}

Now your service communicates both possible outcomes in its signature:

// before — the caller has no idea this can fail
// until the exception surprises them at runtime
public User findUser(Long id) {
    return userRepository.findById(id)
        .orElseThrow(() -> new NotFoundException("User", id));
}

// after - the return type tells the whole story upfront
public Result<User> findUser(Long id) {
    return userRepository.findById(id)
        .map(Result::ok)
        .orElse(Result.failure("User not found with id: " + id));
}

And the caller handles both paths without needing a try-catch anywhere:

Result<User> result = userService.findUser(123L);
if (result.isSuccess()) {
    return ResponseEntity.ok(result.getValue());
}

return ResponseEntity.notFound().build();

The difference in readability is significant. In the exception-throwing version, a developer reading the method signature sees User findUser(Long id) and assumes it always returns a user. The failure mode is invisible until it bites them. In the result version, Result<User> findUser(Long id) makes the contract obvious this operation produces either a user or an explanation of why it didn't. The caller is forced to acknowledge both possibilities.

This isn’t a new idea and it definitely isn’t Java-specific. Rust builds this directly into the language with Result<T, E> if your function can fail, the type system requires you to say so and requires the caller to handle it. Go uses the idiomatic (value, error) return pair for the same reason. Haskell has Maybe. These languages treat explicit failure modeling as a first-class concern, not an afterthought.

The Java ecosystem is slowly catching up. Libraries like Vavr bring proper Either and Try types to the JVM. The direction the industry is moving is clear: make failure visible in the type signature, not hidden behind runtime surprises.

The rule: if a scenario is expected not a bug, not a system fault, just a normal outcome your business logic needs to handle don’t throw an exception. Return a result. Make the failure a first-class citizen of your API, not a trap waiting for the next developer.

The mental model that ties it all together

Four patterns is a lot to hold in your head at once. The good news is there’s a single question underneath all of them that makes the right choice obvious once you’ve internalized it:

Is this failure exceptional, or is it expected?

That’s it. Two buckets. Everything goes in one of them.

Exceptional failures are things that shouldn’t happen under normal operating conditions. The database connection drops mid-request. A third-party API times out after working fine for months. The server runs out of memory. A file that should always exist gets deleted by something outside your application. These are infrastructure-level surprises conditions your code didn’t cause and can’t reasonably prevent. They deserve try-catch, proper logging, alerting, and a human looking at them.

Expected failures are outcomes that are entirely normal within your business domain. A user searches for a record that doesn’t exist. A payment gets declined because the card is expired. Someone tries to book a seat that just got taken. An order is submitted twice. These aren’t bugs. They’re valid states your system needs to navigate gracefully. They deserve validation, result objects, and clean control flow not try-catch, not stack traces, not PagerDuty alerts at midnight.

Once you start sorting failures into these two buckets automatically, the right pattern follows almost without thinking. Bad input coming in? That’s preventable validate it at the boundary. A business rule getting violated? Name it explicitly with a custom exception. Error handling logic spreading across controllers? Centralize it. A routine “not found” outcome? Model it as a result, not an emergency.

The reason most codebases end up with try-catch everywhere is that nobody ever drew this line. Everything got treated as equally unpredictable, equally dangerous, equally worthy of the same blunt instrument. The bucket metaphor is simple, but the discipline of actually applying it consistently is what separates a codebase you can debug at 3am from one that makes you want to update your resume at 3am.

Think of it like your notification system. A fire alarm and a calendar reminder are both alerts. But they belong to completely different categories of urgency, and you’d never design a system that treated them the same way. Your exception handling is your application’s notification system. Design it with the same intentionality.

Conclusion

Here’s the uncomfortable truth the tutorials won’t tell you: most of the try-catch in production codebases today exists because the people who wrote it were taught a pattern without being taught the reasoning behind it. Copy, paste, ship. It works until it doesn’t, and when it doesn’t, it fails in the worst possible way silently, slowly, invisibly.

The four patterns in this article aren’t advanced techniques. They’re not senior-level secrets. They’re just what happens when you start asking “why is this failing?” instead of “how do I make this stop crashing?” Validate the input. Name the failure. Centralize the handling. Model the expected outcomes. None of it is complicated. All of it compounds.

The industry is already moving in this direction. Languages like Rust make explicit error modeling non-negotiable at the compiler level. TypeScript’s ecosystem is increasingly leaning on discriminated unions for the same purpose. The direction is clear: failures belong in the type signature, not hiding behind runtime surprises in catch blocks nobody reads.

Do this tomorrow one change at a time:

Find one try-catch in your codebase that’s catching bad input. Replace it with proper validation a @Valid annotation, a schema check, a guard clause. Just one. Find one catch (Exception e) that's swallowing everything. Replace it with a specific custom exception that names the actual failure. If you have catch blocks duplicated across multiple controllers or handlers, build a centralized exception handler and delete the copies. Find one place where you're throwing an exception for an outcome that's entirely expected. Replace it with a result object.

Four changes. One codebase that’s measurably more honest about what it does and what can go wrong.

The worst try-catch I ever saw in production wrapped an entire service class constructor and returned null on failure. The service would silently initialize as null, work fine for a while, then NullPointerException somewhere completely unrelated, no trail, no context, nothing. Three days of debugging. The fix was six lines.

What’s yours? Drop the worst try-catch you’ve ever shipped in the comments. No judgment we’ve all got one.

Helpful resources

Jakarta Bean Validation specification the standard behind @Valid and constraint annotations
Spring @ControllerAdvice documentation official reference for global exception handling
Vavr library brings Either, Try, and Option types to Java
Rust Book: Recoverable errors with Result the gold standard for how explicit error modeling should work
Error Handling in Go Google’s official take on the (value, error) pattern

10 SQL changes. One took 30 seconds. It cut query time by 85%.

Tue, 12 May 2026 06:38:34 +0000

A 2015 research paper tested every tip against real data. Most developers have never seen it. The numbers are hard to ignore.

You wrote a query. It returned the right data. You moved on.

That’s the whole story for most developers. The query works, the feature ships, and nobody looks back. Until a senior engineer pulls up the execution plan in a prod review and asks why you’re doing a full table scan on 2 million rows to return twelve records.

That moment has happened to more engineers than will ever admit it publicly.

Here’s the thing: most SQL slowness isn’t mysterious. It’s not a missing index, a misconfigured database, or a vendor problem. It’s patterns habits that looked fine when the table had 500 rows and became quietly catastrophic when it hit 5 million.

In 2015, a researcher named Jean Habimana published a paper through IJSTR titled “Query Optimization Techniques: Tips For Writing Efficient And Faster SQL Queries.” Five pages. Tested against Oracle’s sample Sales database. Every tip benchmarked with actual time reductions. It’s been sitting in academic obscurity ever since, which is a shame, because some of these changes take thirty seconds to make and show query time reductions above 80%.

This article is that paper, translated into something you’ll actually read.

TL;DR: 10 SQL habits ranked by impact. Time reductions range from 11% to 85%. Most fixes take under five minutes. A few will make you retroactively embarrassed about queries you shipped last year.

Why your query isn’t just “running”

Most SQL tutorials skip this part entirely. They go straight to syntax, joins, and GROUP BY, and leave you with a model of SQL that looks roughly like: you write it, the database runs it, data comes back. Clean. Simple. Wrong.

When you submit a query, it doesn’t execute directly. It goes through a query optimizer first a component that reads your SQL, estimates multiple ways to fetch the result, and picks the one it thinks is cheapest. Cheapest meaning least I/O, least memory, least CPU. The optimizer is making decisions you never see, and those decisions are heavily influenced by how you wrote the query.

Think of it like a GPS. You give it a destination and it figures out the route. But if you give it bad inputs a vague address, a restricted road it doesn’t know about it picks a worse path. The database optimizer works the same way. Write the query clearly and it finds the fast route. Write it carelessly and it takes the scenic route through every row in your table.

The execution path looks roughly like this:

Your SQL
    ↓
Query Parser — checks syntax
    ↓
Query Optimizer — estimates cheapest execution path
    ↓
Execution Plan — the actual instruction set
    ↓
Data retrieval → Result

Every tip in this article targets one of two things: either helping the optimizer make a better decision, or removing work it shouldn’t have to do in the first place. That’s the whole framework. Keep it in mind as you read the rest.

The lazy habits costing you the most

Let’s start with the number that stopped me mid-scroll when I first read this paper: 85% query time reduction. Not from adding an index. Not from upgrading hardware. From removing one word.

Tip 3 first because 85% deserves the spotlight

When you write SELECT DISTINCT on a join where the primary key is already in the result, duplicates are mathematically impossible. There are no duplicates to remove. But the database doesn't know that so it sorts the entire result set and checks anyway. You're paying full deduplication cost for zero benefit.

-- Slow: DISTINCT is doing nothing here (primary key present)
SELECT DISTINCT  FROM sales s
JOIN customers c ON s.cust_id = c.cust_id;

-- Fast: just don't
SELECT  FROM sales s
JOIN customers c ON s.cust_id = c.cust_id;

85% faster. One word removed.

Tip 1: SELECT columns, not SELECT * 27% reduction

The most universal bad habit in SQL. When you write SELECT *, the database fetches every column and ships it across the network. If you need two columns from a table with twenty, you're paying for eighteen you'll never use. If any of those columns are large types TEXT, BLOB, JSON you're paying a lot.

-- Slow
SELECT  FROM sales;

-- Fast
SELECT prod_id, cust_id FROM sales;

It feels pedantic until your table has 40 columns and half of them are storing documents.

Tip 2: WHERE before GROUP BY, not HAVING 31% reduction

HAVING filters rows after grouping. WHERE filters rows before grouping. If your condition doesn't involve an aggregate function, it has no business being in HAVING. Putting it there means the database groups every row first, then throws away the ones you didn't want work it never needed to do.

-- Slow: groups everything, then filters
SELECT cust_id, COUNT(cust_id)
FROM sales
GROUP BY cust_id
HAVING cust_id != '1660';

-- Fast: filters first, groups less
SELECT cust_id, COUNT(cust_id)
FROM sales
WHERE cust_id != '1660'
GROUP BY cust_id;

Tip 3: Drop unnecessary DISTINCT 85% reduction

Here it is. The biggest number in the paper.

-- Slow: DISTINCT is doing nothing here (primary key present)
SELECT DISTINCT  FROM sales s
JOIN customers c ON s.cust_id = c.cust_id;

-- Fast: just don't
SELECT  FROM sales s
JOIN customers c ON s.cust_id = c.cust_id;

85% faster. One word removed.

Tip 4: Un-nest your subqueries 61% reduction

Correlated subqueries are the silent performance killer most junior devs don’t recognize until it’s too late. A subquery inside a WHERE clause that references the outer query runs once per row. Not once total once per row. On a table with 100,000 rows, that's 100,000 executions of your inner query.

A JOIN runs once.

-- Slow: subquery executes for every row in products
SELECT  FROM products p
WHERE p.prod_id = (
    SELECT s.prod_id FROM sales s
    WHERE s.cust_id = 100996
);

-- Fast: single join operation
SELECT p. FROM products p
JOIN sales s ON p.prod_id = s.prod_id
WHERE s.cust_id = 100996;

If you’ve ever watched a query go from instant to “still running after 40 seconds” as the table grew, a correlated subquery somewhere is usually the culprit.

Four tips. Average time reduction across all four: 51%. And none of them required touching your schema, your indexes, or your infrastructure. Just the query.

The optimizer killers

These four are sneakier. The previous section was mostly about obvious waste fetching columns you don’t need, grouping rows you’re about to throw away. This section is about patterns that look perfectly reasonable but quietly prevent the optimizer from using your indexes. That distinction matters a lot at scale.

Tip 5: IN instead of multiple ORs 73% reduction

This one surprises people. Both IN and OR feel like they're doing the same thing, and syntactically they kind of are. The difference is what the optimizer can do with them.

With an IN list, the optimizer can sort the values and match them against the index in order. With chained OR conditions, it can't apply that optimization it evaluates each condition independently, often falling back to a full scan.

-- Slow
WHERE prod_id = 14 OR prod_id = 17;

-- Fast
WHERE prod_id IN (14, 17);

Small change. The optimizer sees a completely different instruction. 73% faster.

Tip 6: EXISTS over DISTINCT on one-to-many joins 61% reduction

When you join a parent table to a child table in a one-to-many relationship and slap DISTINCT on it to collapse the duplicates, the database fetches every matching row from both tables and then deduplicates the whole thing. That's a lot of rows moved around just to throw most of them away.

EXISTS short-circuits. It checks whether a match exists and stops the moment it finds one. It never fetches the duplicates in the first place.

-- Slow: fetches everything, then deduplicates
SELECT DISTINCT c.country_id, c.country_name
FROM countries c
JOIN customers e ON e.country_id = c.country_id;

-- Fast: stops at first match per country
SELECT c.country_id, c.country_name
FROM countries c
WHERE EXISTS (
    SELECT 1 FROM customers e
    WHERE e.country_id = c.country_id
);

Once you understand the short-circuit behavior, you’ll see DISTINCT on joins differently. It's not wrong it's just usually the expensive way to ask a simple question.

Tip 7: UNION ALL over UNION 81% reduction

UNION deduplicates. UNION ALL doesn't. That's the entire difference, and it costs 81% of your query time when the data can't have duplicates anyway or when you simply don't care.

-- Slow: scans combined result for duplicates
SELECT cust_id FROM sales
UNION
SELECT cust_id FROM customers;

-- Fast: skips deduplication entirely
SELECT cust_id FROM sales
UNION ALL
SELECT cust_id FROM customers;

The rule is simple: if your data sources can’t produce duplicates by definition, or if downstream logic handles it, UNION ALL is strictly faster. Using UNION by default because it feels safer is leaving 81% on the table.

Tip 8: Split OR in JOIN conditions into UNION ALL 70% reduction

This is the least obvious one in the entire paper, and probably the most common silent killer in production codebases. An OR condition inside a JOIN prevents index usage on both sides. The optimizer sees it and gives up on the index entirely, falling back to a full scan of both tables.

The fix is to split it into two separate joins and combine them with UNION ALL.

-- Slow: OR blocks index usage on both columns
SELECT  FROM costs c
INNER JOIN products p
ON c.unit_price = p.prod_min_price
OR c.unit_price = p.prod_list_price;

-- Fast: two indexed joins, combined
SELECT  FROM costs c
INNER JOIN products p ON c.unit_price = p.prod_min_price
UNION ALL
SELECT  FROM costs c
INNER JOIN products p ON c.unit_price = p.prod_list_price;

It looks more verbose. It is more verbose. It’s also 70% faster because both joins can now use their indexes cleanly. Verbose and fast beats clean and slow every time a product manager asks why the report takes three minutes to load.

If you want to see exactly what your optimizer is doing with any of these patterns, MySQL’s optimizer trace and PostgreSQL’s EXPLAIN ANALYZE will show you the execution plan in detail and make the index abandonment painfully visible.

Four tips. Reductions of 73%, 61%, 81%, and 70%. All from patterns that look harmless in a code review because the query returns the right data. Correctness and performance are different problems, and SQL will let you solve one while completely ignoring the other.

The silent ones

These two don’t announce themselves. No error, no warning, no slow query log entry when the table is small. You write them, they work, and then six months later when the data has grown and something is mysteriously sluggish, nobody connects it back to these lines.

Tip 9: No functions on indexed columns in WHERE 70% reduction

This one is responsible for more silent performance regressions than almost anything else on this list. The logic feels completely reasonable when you write it: you need to filter by year, the column stores full dates, so you wrap it in EXTRACT() and move on.

-- Slow: function call prevents index usage
WHERE EXTRACT(YEAR FROM time_id) = 2001;

-- Fast: BETWEEN works with the index
WHERE time_id BETWEEN '01-JAN-2001' AND '31-DEC-2001';

Here’s what actually happens: the moment you wrap a column in a function inside a WHERE clause, the optimizer can't use the index on that column anymore. The index is organized around the raw column values. Once you transform those values with a function, the index has no idea how to help so it doesn't. The database computes EXTRACT() for every single row, then filters. Full scan. Every time.

The fix is to move the transformation to the value side, not the column side. Instead of asking “what year does this date belong to,” ask “does this date fall inside this year.” BETWEEN two known date boundaries does exactly that, leaves the column untouched, and lets the index do its job.

This applies beyond EXTRACT(). Any function wrapping an indexed column in a WHERE clause UPPER(), LOWER(), CAST(), DATE_TRUNC(), TO_CHAR() breaks index usage the same way. It's a category of mistake, not just one specific function.

70% reduction. Zero schema changes required.

Tip 10: Pre-calculate your math 11% reduction

This one is smaller 11% but it might be the most embarrassing item on the list once you understand what’s happening.

-- Slow: recalculates for every row
WHERE cust_id + 10000 < 35000;

-- Fast: constant evaluated once
WHERE cust_id < 25000;

When your WHERE clause contains arithmetic on a column, the database evaluates that expression for every row it scans. Not once every row. The value 10000 isn't changing between rows. You already know the answer is 25000. But you made the database do the same addition hundreds of thousands of times because you didn't do it yourself first.

Do the math before the query runs. It costs you nothing and saves the database from doing redundant arithmetic at scale.

11% might sound modest compared to the numbers earlier in this article. But this is also the change that takes literally ten seconds. There’s no tradeoff to weigh, no refactor to plan. Just move the arithmetic outside the query. If you’re leaving 11% on the table because the fix felt too small to bother with, that’s a habit worth breaking.

Both of these fall into the same category: the optimizer wanted to help, and the way the query was written made that impossible. The database didn’t fail it did exactly what you asked. You just didn’t realize what you were asking.

The pre-ship checklist

Before you push that query whether it’s going into an ORM, a stored procedure, a data pipeline, or directly into prod run it through this. Seven questions. Takes thirty seconds.

SELECT * anywhere?          → Specify only the columns you need
Filtering in HAVING?        → Move it to WHERE if no aggregate involved
Any DISTINCT?               → Do you actually need it, or is the key already there?
Nested subquery?            → Can it be rewritten as a JOIN?
OR in WHERE or JOIN?        → Try IN or UNION ALL instead
Function wrapping a column? → Move the transformation to the value side
Math on a column?           → Pre-calculate it before the query runs

None of these require a DBA, a schema change, or a ticket. They’re query-level habits. The kind that separate code that works from code that scales.

Bookmark this. Drop it in your team’s wiki. Put it in your CLAUDE.md if you're using Claude Code. Stick it somewhere you'll actually see it before the slow query alert fires at an inconvenient hour.

For deeper reading on index behavior and execution plans, Use The Index, Luke is the best free resource on the internet for this topic. No fluff, just mechanics.

The queries you wrote last year are probably still running

The paper this article is based on was published in 2015. The database you’re querying today almost certainly has rows that didn’t exist then. The query you wrote last year, maybe last month, is probably still executing exactly as you wrote it habits intact, DISTINCT in place, SELECT * quietly fetching columns nobody asked for.

That’s the uncomfortable part. These aren’t exotic edge cases. They’re patterns that pass code review because the query returns correct results, and correctness is usually all anyone checks. Performance is someone else’s problem until it becomes everyone’s problem at 11pm on a Tuesday.

The slightly spicy take: most slow SQL isn’t a database problem. It’s a habits problem. The optimizer is genuinely trying to help you it’s building execution plans, estimating costs, looking for index paths. What it can’t do is save you from instructions that actively prevent it from doing its job. Functions on indexed columns, OR conditions in joins, DISTINCT on results that can’t have duplicates these aren’t bugs the database can route around. They’re the query working exactly as written.

The good news is that optimizers are getting smarter. PostgreSQL 16 brought improvements to parallel query execution and partition pruning. AI-assisted query analysis tools are starting to surface execution plan issues automatically. The tooling is moving in the right direction.

But none of it will save you from SELECT *.

The changes in this article aren’t framework upgrades or infrastructure investments. They’re rewrites. Most take under five minutes. Some show reductions above 80%. The paper tested them on Oracle but the underlying optimizer logic holds across PostgreSQL, MySQL, and SQL Server the principles are the same.

Go check what your slowest queries are doing. There’s a reasonable chance one of these seven checklist items is in there.

And if you’ve got a SQL habit that you’re mildly ashamed of a correlated subquery you know you should rewrite, a SELECT * you keep meaning to fix

drop it in the comments. No judgment. We’ve all shipped something we’d rather not explain.

Resources

Original paper: Jean Habimana, “Query Optimization Techniques: Tips For Writing Efficient And Faster SQL Queries”, IJSTR Vol. 4, Issue 10, October 2015
Use The Index, Luke index mechanics explained without the academic fog
PostgreSQL query planning docs how PG’s optimizer actually works
MySQL optimizer trace see exactly what MySQL is doing with your query
explain.dalibo.com visual EXPLAIN ANALYZE for Postgres, free and genuinely useful

I built an AI agent in 50 lines of Python. Here’s what everyone gets wrong about them.

Mon, 11 May 2026 01:42:36 +0000

You’ve been calling things agents for months. Most of them are just chatbots with extra steps.

Every senior developer I know has done this at least once: you’re mid-review, someone asks how the agent loop actually works under the hood, and your brain quietly blue-screens. You’ve been using Claude, Cursor, Copilot, and Junie every single day. You’ve shipped features on top of them. You’ve nodded confidently in sprint planning while someone called the autocomplete a “multi-agent orchestration layer.” And yet, if someone put a gun to your head and asked you to draw the architecture the actual mechanics of what makes something an agent instead of a really fast autocomplete you’d stall.

That was me about six weeks ago.

I got fed up with not knowing, so I did what I always do when a concept won’t click: I built the stupidest possible version of it from scratch. No LangChain. No LangGraph. No CrewAI. No framework scaffolding to hide behind. Just Python, an API key, and a while loop. Fifty lines. And somewhere around line thirty-two, the thing that’s been fuzzy for a year finally came into focus.

Turns out an agent isn’t magic. It’s not even particularly clever. It’s a deterministic loop with a short memory and a decision point. The model doesn’t “think.” It reads a conversation history, decides whether it has enough to answer or needs to call a tool, and either exits or loops. That’s it. Everything else the retry logic, the subagents, the memory persistence, the human-in-the-loop approvals is infrastructure layered on top of that core loop.

TL;DR: This article walks through building a real AI agent from first principle cloud API brain, then local models via Ollama, then mixed-mode orchestration, then MCP tool sharing, then a straight talk about where frameworks actually earn their keep. By the end, you’ll be able to explain exactly what’s happening when Claude Code spins up a subagent, and you’ll have the code to prove you understand it.

What an agent actually is (and why most “agents” aren’t)

Here’s the real definition, stripped of the marketing: a regular LLM call is a one-shot operation. You send a prompt, you get a response, done. The model has no memory of what just happened. It doesn’t know if it got it right. It doesn’t retry. It just answers and exits.

An agent is different because it loops. It takes a high-level task, reasons about what to do next, takes an action, observes the result, and keeps going until it decides it’s finished. That repeating cycle think, act, observe, decide is the thing that turns a language model into something that behaves like an autonomous system. Remove the loop and you just have a chatbot that called a function once.

Most things being marketed as “agents” right now are closer to the chatbot end of that spectrum. A RAG pipeline that retrieves documents and summarizes them isn’t an agent. A Slack bot that calls an API when you type a command isn’t an agent. They’re useful tools, but they don’t loop, they don’t observe results and adjust, and they don’t decide when to stop. Calling them agents is like calling a calculator a mathematician because it can add numbers.

The pattern that most real agents follow is called ReAct Reasoning and Acting, introduced by Yao et al. in a 2022 paper that’s worth at least skimming. The idea is simple: the model doesn’t jump straight to a final answer. It produces a thought about what to do, then an action (a tool call), then waits to observe the result before deciding what to do next. The loop continues until the model has enough context to answer directly without calling any more tools.

Think of it like a developer assigned a Jira ticket. They don’t read it once and immediately output the solution. They read it, try something, hit an error, read the error, Google something, try again, check if the tests pass, then mark it done. The agent does the same thing it’s just that the conversation history is the Jira thread, the tools are the shell commands, and the LLM is the developer who never sleeps and never complains about the sprint velocity.

Here’s what every single cycle of the loop looks like under the hood:

Send the current conversation to the LLM system prompt, user message, any prior tool results
The LLM returns either a final answer or a list of tool calls it wants to make
If it’s a final answer, you’re done
If it’s tool calls, execute them, append the results to the conversation, go back to step 1

That’s the entire architecture. The model has no consciousness and no genuine self-reflection what it has is the full conversation history sitting in its context window, and a system prompt telling it what tools exist and when to stop. The ReAct pattern turns that into something that looks like self-correction. And it works surprisingly well.

The part people consistently underestimate is the system prompt. It’s not a formality. It’s the steering wheel. It tells the model when to use tools, when a task is complete, what a final answer should look like, and what it should never do. When leaked system prompts from production agents at Anthropic and Apple showed up online, the striking thing wasn’t the model sophistication it was how long and careful the system prompts were. Hundreds of lines of plain English, constraining behavior with precision. The loop is simple. The steering is where the craft lives.

Points:

Agent = a loop with a decision point, not a single LLM call with extra marketing
ReAct (Reason + Act) is the foundational pattern think, act, observe, repeat until done
The model “self-corrects” because the conversation history accumulates not because it’s actually reflecting
The system prompt is the real logic layer underestimate it and your agent does whatever it wants

“The model doesn’t think. It reads a very long chat history and decides whether to answer or loop. That’s it.”

The 50-line implementation cloud brain, zero framework

Let’s build the thing. The core of every agent ever written is a function that looks roughly like this:

def run_agent(task: str, client: OpenAI, model: str = "gpt-4o-mini") -> str:
    messages = [
        {
            "role": "system",
            "content": (
                "You are a helpful assistant. Use tools when needed. "
                "When you have a final answer, respond without calling any tools."
            ),
        },
        {"role": "user", "content": task},
    ]

while True:
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            tools=TOOLS,
            tool_choice="auto",
        )

        message = response.choices[0].message
        messages.append(message)

        if not message.tool_calls:
            return message.content

        for tool_call in message.tool_calls:
            name = tool_call.function.name
            args = json.loads(tool_call.function.arguments)
            fn = TOOL_FUNCTIONS.get(name)
            result = fn(**args) if fn else f"Unknown tool: {name}"
            messages.append({
                "role": "tool",
                "tool_call_id": tool_call.id,
                "content": result,
        })

That’s the whole architecture. Everything else LangGraph, CrewAI, AutoGen, the Claude Agents SDK is infrastructure layered on top of that pattern. Strip any of them down far enough and you’ll find a while loop, a messages list, and an if not tool_calls check hiding somewhere inside.

The critical line is if not message.tool_calls. If the model returns text without requesting any tools, it's signaling it has everything it needs to answer. The agent exits and returns that response. If the model requests tools, the agent executes them, appends the results to messages, and sends the whole conversation back for another round. The messages list is the agent's short-term memory every tool call and every result gets appended to it, so by the time the LLM decides it's done, it has seen everything it did and everything it learned from doing it.

To make this concrete, three simple tools: current date/time, a calculator, and a weather stub you’d replace with a real API call in production.

def get_current_date() -> str:
    return datetime.now().strftime("%Y-%m-%d %H:%M:%S")

def calculate(expression: str) -> str:
    try:
        result = eval(expression, {"builtins": {}}, {})
        return str(result)
    except Exception as e:
        return f"Error: {e}"

def get_weather(city: str) -> str:
    return f"Weather in {city}: 72°F, partly cloudy"

Each tool also needs a JSON schema that tells the LLM what’s available, what arguments it takes, and what they mean. This is what the model actually reads when it decides which tool to call. Sloppy descriptions here = wrong tool calls later. The schema is documentation the model acts on, not just metadata you write once and forget.

Run it against a task that needs all three tools at once:

Task: What's today's date? Also, what is 15% of 847? And what's the weather in Tokyo?

> calling get_current_date({})
  > calling calculate({'expression': '847 * 0.15'})
  > calling get_weather({'city': 'Tokyo'})

Answer: Today is 2026-05-11 09:14:22. 15% of 847 is 127.05.
The weather in Tokyo is 72°F and partly cloudy.

On the first turn the LLM identified all three tools, called them, got the results, and assembled the final answer. No orchestration layer. No dependency to install. The model just read the tool schemas, figured out it needed all three, executed them in parallel, and exited cleanly.

What you’re looking at is a complete mental model for every production agent you’ll ever use. Claude Code starting a subagent is this loop calling another loop. Cursor retrying a failed file write is this loop with error handling bolted on. GitHub Copilot Workspace planning a multi-step refactor is this loop running longer with a more complex system prompt. The shape is always the same.

One thing worth calling out: I used OpenAI here because it has the cleanest tool-calling interface for a tutorial, but this works with any OpenAI-compatible API Anthropic, Gemini, and most local inference servers all support the same pattern. The agent code has no opinion about which model is on the other end, as long as it speaks the protocol.

The full working version with all tool schemas is in sergenes/mini_agent on GitHub if you want to run it directly. Worth pulling down and stepping through with a debugger once watching the messages list grow in real time makes the memory model click in a way that reading about it doesn’t.

The ReAct loop the actual architecture underneath every agent you’ve used

Points:

The agent is a while loop, a messages list, and one if-statement everything else is scaffolding
messages is the memory every tool result gets appended before the next LLM call
Tool schemas are documentation the model acts on write them carefully
Every production agent you’ve used is this pattern with error handling, retry logic, and a longer system prompt

“LangGraph, CrewAI, AutoGen strip any of them down far enough and you’ll find a while loop and an if not tool_calls check hiding inside.”

The local model trap why Mistral 7B silently broke everything

Ollama exposes an OpenAI-compatible API, which means the agent code runs on a local model with exactly one change:

ollama_client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",
)

answer = run_agent(task, ollama_client, model="qwen2.5")

That’s it. The agent has no idea whether it’s talking to OpenAI’s servers or a model running on your laptop. Same loop, same tool schemas, same exit condition. To get Ollama running: install from ollama.com, then ollama pull qwen2.5 and ollama serve. After that, the whole thing runs offline. No API costs, no data leaving your machine, no rate limits at two in the morning when you're debugging something you broke before bed.

Except here’s what actually happened when I tried it the first time.

I pulled Mistral 7B. Widely recommended, solid benchmark numbers, comes up in every “best local models” thread on Hacker News. Ran the same three-tool task. No errors. Clean output. And then I read it:

Answer: I need to call get_current_date() to find today's date.
Let me use the calculate tool: calculate(expression="847 * 0.15")...
The weather in Tokyo is probably warm this time of year.

Plain text. Describing tool calls in prose. Guessing the weather. response.tool_calls was empty on every turn, so the agent hit the if not message.tool_calls check, found nothing, and exited immediately with whatever the model had written.

My first instinct was that I’d broken the agent code. Spent a solid chunk of time staring at the while loop, checking the tool schemas, wondering if Ollama’s API response format was slightly different. It wasn’t. The code worked exactly as written. The problem was the model.

Mistral 7B doesn’t support OpenAI-style structured function calling. It was trained to describe actions in prose, not emit them as structured JSON. When it saw the tool schemas in the request, it understood conceptually what tools were available so it started narrating what it would call. But it never actually emitted the structured tool_calls object the agent was waiting for. The model hallucinated the syntax it thought I expected, and the agent politely returned that hallucination as a final answer.

This is the trap: the code gives you no error. No exception. No warning. The agent just exits on the first turn and you get back a paragraph of the model describing what it wishes it could do. If you’re not watching closely, you might not even notice the tools never fired.

The fix is model selection. Not all local models support structured function calling through Ollama, and the ones that don’t fail silently in exactly this way.
The models that reliably work:

qwen2.5          ✓  Strong tool calling, good reasoning
llama3.1         ✓  Solid across most tool schemas
mistral-nemo     ✓  Works, occasionally verbose in reasoning
phi4             ✓  Surprisingly capable for its size
mistral 7B       ✗  Prose descriptions, no structured calls
gemma2           ✗  Inconsistent — sometimes works, often doesn't

If your agent returns immediately without calling any tools, suspect the model before the code. Swap to qwen2.5 and see if the behavior changes. Nine times out of ten that's it.

There’s a deeper lesson here about the gap between benchmark performance and production behavior. Mistral 7B scores well on reasoning benchmarks. It generates coherent, useful text. But “coherent text” and “emits structured JSON tool calls” are different capabilities, and the benchmarks that get cited in model announcements don’t always test the latter. Before you commit to a local model for anything agentic, run the simplest possible tool-calling task first. Date, calculator, weather stub. If those three work, you’re probably fine. If the model narrates them instead of calling them, move on.

The mixed-mode pattern handles this more elegantly run the loop locally, delegate to a cloud model when the task genuinely needs it but that’s the next section.

Points:

Ollama’s OpenAI-compatible API means zero code changes to run locally one base_url swap
Not all local models support structured function calling silent failure is the default behavior
Mistral 7B describes tool calls in prose instead of emitting JSON; the agent exits cleanly on turn one
Test with a simple three-tool task before committing to any local model for agentic work
qwen2.5 and llama3.1 are the reliable starting points as of mid-2026

“The code gives you no error. No exception. No warning. The agent just exits and returns a paragraph of the model describing what it wishes it could do.”

MCP the protocol that finally fixes hardcoded tool hell

Here’s the problem nobody talks about when they show you the 50-line agent: every tool is hardcoded into the script. get_current_date, calculate, get_weather all defined in the same file, all manually added to the TOOLS list, all maintained by you. If you want another agent to use the same tools, you copy and paste. If you want to use tools someone else built, you rewrite their implementation into your format. If you update a tool, you update every script that hardcoded it.

This is fine for a tutorial. It becomes genuinely painful around the time you have three agents across two projects and you’re maintaining six copies of the same file read utility with slightly different signatures.

MCP Model Context Protocol is the standard Anthropic shipped in November 2024 specifically to fix this. The spec lives at modelcontextprotocol.io and the idea is straightforward: instead of hardcoding tool definitions into your agent, you point the agent at a server. The server advertises what tools it has. The agent discovers them, gets their schemas, and calls them exactly the same way it calls local functions. The agent doesn’t care whether a tool is a Python function in the same file or a service running on the other side of the internetas long as it speaks MCP, it works.

Think USB-C. Before USB-C, every device had its own connector and you maintained a drawer full of incompatible cables. MCP is the USB-C for AI tools one protocol, anything plugs in. GitHub has an MCP server. Slack has one. Postgres has one. Google Drive has one. You point your agent at any of them and the tools just appear, already described, ready to use. Someone else writes the implementation once; everyone else benefits.

Building an MCP server is almost comically simple with FastMCP:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("mini-tools")

@mcp.tool()
def to_uppercase(text: str) -> str:
    """Convert text to uppercase."""
    return text.upper()

@mcp.tool()
def count_words(text: str) -> int:
    """Count the number of words in a string."""
    return len(text.split())

if name == "main":
    mcp.run()

That’s a complete, working MCP server. Ten lines. The decorator handles schema generation from the type hints and docstring no manual JSON. Run it as a subprocess and any MCP-compatible client can discover those tools via a JSON-RPC handshake. Claude Desktop, Cursor, your DIY agent, anyone. Publish the server, point a config at it, and it just works.

From the agent’s side, the MCP client starts the server as a subprocess and calls tools via JSON-RPC. The agent receives the tool list exactly the same way it would receive hardcoded definitions they show up in TOOLS, get passed to the LLM, get called the same way, return results the same way. The companion repo at github.com/sergenes/mini_agent includes a working mcp_client.py that demonstrates the full handshake if you want to see the plumbing.

The architectural shift here is bigger than it sounds. Before MCP, every team building agents was maintaining their own tool library, reinventing the same integrations “call GitHub API”, “query Postgres”, “read a Slack channel” across dozens of repos with incompatible interfaces. MCP turns those into shared infrastructure. One well-maintained server replaces hundreds of slightly-wrong copies.

It also changes what “building an agent” means in practice. The interesting work stops being “how do I wire up this API” and starts being “which servers do I need and how do I write a system prompt that uses them well.” The loop stays exactly the same. What changes is that the tool ecosystem is now the internet instead of whatever you’ve personally had time to implement.

The FastMCP library is worth bookmarking it’s the fastest way to expose your own tools to any agent in the ecosystem, and the decorator pattern makes the schema definition essentially free. Write the function, add the decorator, document the arguments, done.

Points:

Hardcoded tools don’t scale you end up maintaining copies across every project and agent
MCP is the standard protocol for tool discovery and calling agent points at server, server advertises tools, agent uses them
Building an MCP server with FastMCP is ten lines decorator handles schema generation from type hints
The ecosystem already has MCP servers for GitHub, Slack, Postgres, Google Drive, and hundreds more
MCP shifts the interesting work from “wiring APIs” to “writing system prompts that use shared tools well”

“Before MCP I was copy-pasting tool definitions across three different repos. Not great. Not sustainable. Definitely the kind of thing you notice at code review.”

Frameworks aren’t magic they’re load-bearing walls

Let’s be honest about what the 50-line agent is missing, because pretending it’s production-ready would be doing you a disservice.

No error handling. If a tool throws an exception, the agent crashes. No retry logic a flaky API call takes down the whole run. No way to pause for human approval before the agent does something destructive like deleting files or sending emails. No memory beyond the current conversation restart the process and it has no idea what it did ten minutes ago. No ability to spawn parallel subagents when a task is large enough to benefit from splitting. No context compression when the messages list grows long enough to hit the model’s limit. No observability into what the agent decided at each step and why.

That list is not a criticism of the 50-line version. It’s a description of what frameworks exist to solve.

LangGraph models the agent as a state machine with explicit nodes and edges. You define what happens at each step and what conditions trigger the next one. More setup upfront, but you get checkpointing the agent can pause mid-run and resume from exactly where it stopped. You get structured error handling at each node. You get human-in-the-loop steps where execution blocks until a human approves the next action. You get full observability into the graph traversal. If you’re building something where a failed tool call means a corrupted database or a mistaken API call means actual money spent, LangGraph is where you want to be. The docs are dense but the core concept clicks fast once you’ve built the naive loop yourself.

CrewAI and AutoGen go a layer higher multi-agent coordination. Instead of one agent with many tools, you define multiple agents with specialized roles: a researcher, a writer, a critic. Each has its own system prompt, its own tool access, its own model if you want. The orchestrator decides who talks to whom and in what order. Useful for complex tasks where different phases genuinely need different prompts a research phase that needs web search and summarization, followed by a writing phase that needs a totally different voice, followed by a review phase that needs to be adversarial. Trying to do all of that with one system prompt and one tool set gets messy fast. CrewAI and AutoGen are the frameworks worth reaching for when you hit that wall.

The managed runtimes Claude Agents SDK and OpenAI Assistants API trade control for speed. You hand off state management, tool routing, threading, and context compression to the platform. Less visibility into what’s happening, but dramatically less code to write and maintain. Worth it when you need to ship something in a week and the task doesn’t require custom orchestration logic. The Claude Agents SDK docs in particular are worth reading even if you don’t use the SDK the mental model they describe maps directly onto everything we’ve covered here.

Claude Code is the honest comparison point for the DIY version. It does things the 50-line agent can’t touch: starts subagents with isolated context windows when a task is too large for one loop, prompts for confirmation before running destructive shell commands, maintains persistent memory across sessions, retries failed tool calls with adjusted parameters, compresses prior messages when approaching context limits. My agent has six tools, one messages list, and no safety net. Claude Code bills per message. My agent costs nothing until I tell it to hit GPT-4.

If I need to ship something reliable and I don’t want to think about orchestration, I use Claude Code. If I’m prototyping something I’d spend a week fighting a framework to implement, I start from the loop. That tension is actually healthy. Knowing what frameworks abstract means you can make the decision deliberately instead of defaulting to LangGraph because it’s what the tutorial used.

The 50-line version is a sketch. LangGraph is that sketch turned into a building with proper load-bearing walls. You need to understand the sketch before you can reason about the building. Now you do.

Points:

The naive loop is missing error handling, retries, human-in-the-loop, persistent memory, parallel subagents, and context compression frameworks exist to solve exactly these problems
LangGraph: state machine model, checkpointing, structured error handling, best for production reliability
CrewAI / AutoGen: multi-agent coordination, specialized roles, best when different phases need genuinely different prompts
Managed runtimes (Claude Agents SDK, OpenAI Assistants): fastest to ship, least control, worth it for tight deadlines
Use frameworks when you’ve hit the problem they solve not before

“The 50-line loop is a sketch. LangGraph is that sketch turned into a building with proper load-bearing walls. Understand the sketch first.”

What building this actually taught me

I started this because I couldn’t answer a question in a code review. I ended up with something more useful than the answer a complete mental model I couldn’t get from using production tools, no matter how much I used them.

That’s the thing about abstractions. They’re designed to hide complexity, and they’re good at it. Cursor doesn’t show you the while loop. Claude Code doesn’t surface the messages list growing with each tool call. LangGraph definitely doesn’t make you think about the if not tool_calls check at the center of everything. The abstraction works, which means you can ship fast without understanding the foundation. Until you can't. Until something breaks in a way the framework doesn't have a handler for, or you need behavior the abstraction actively fights against, or someone asks you in a review to explain what's actually happening under the hood.

Building the naive version closed that gap for me. I can see now exactly where an agent gets stuck when the system prompt is underspecified and the model doesn’t know when to stop. I can see why it picks one tool over another because the schema description was more precise. I can see when adding more tools actually makes things worse because the model starts hallucinating tool calls for tools that don’t quite fit the task, instead of admitting it can’t do it. That visibility is worth more than any framework feature.

Some of my projects will use LangGraph or the Claude Agents SDK going forward. Those frameworks solve real problems I genuinely don’t want to reimplement retry logic, checkpointing, human-in-the-loop approvals. But some will start from this 50-line loop, because I know exactly what it does and I can modify it without fighting abstractions I don’t fully understand. That optionality is the real output of the exercise.

Here’s the slightly uncomfortable opinion to end on: too many developers are reaching for agent frameworks before they understand what’s being abstracted. The ecosystem moved fast LangChain appeared, then LangGraph, then CrewAI, then a dozen managed runtimes, all within about eighteen months. The tutorials went straight to the framework. A lot of people never saw the loop. That’s going to become a problem as agents get more embedded in production systems and the failure modes get more consequential. You can’t debug what you don’t understand, and “the framework handled it” stops being a satisfying answer when the thing the framework handled was your customer’s data.

Build the naive version first. Then decide what infrastructure you actually need. The loop is twenty minutes of work. The clarity it gives you is permanent.

What did you build to understand something you were already using? Drop it in the comments I read every one.

Helpful resources

GitHub is dead to the Netherlands and they built something better

Sun, 10 May 2026 07:31:28 +0000

The Dutch government quietly replaced GitHub with a self-hosted FOSS forge. No press release. Just commits.

Most developers have had the self-hosting conversation at least once. Someone on the team floats it, everyone nods, someone opens a Hetzner tab, and then a Jira ticket appears and the whole thing dies quietly on a Friday afternoon. We talk about owning our stack the way people talk about going to the gym constantly, convincingly, and without follow-through.

The Netherlands apparently skipped that part.

In November 2025, a software engineer named Jan Vlug published a detailed evaluation on the Dutch government’s developer portal asking one very reasonable question: which Git forge should the Netherlands actually use for its governmental source code? Not which one is most popular. Not which one has the slickest UI. Which one is appropriate for a government that has an actual policy about open source software.

That blog post wasn’t a thought experiment. The Ministry of the Interior was already spinning up a dedicated Git instance. The platform decision was live and open. And the Dutch government’s code at that point was scattered across GitHub and GitLab neither of which is under any form of government oversight or control.

So they actually did it. On April 24, 2026, code.overheid.nl went live quietly, without fanfare, in pilot phase as a fully self-hosted, FOSS-only government code forge. No press release. No launch party. Just a blog post from developer advocate Tom Ootes saying, essentially,

“we’re building this together, come help.”

TL;DR: The Dutch government evaluated GitHub, evaluated GitLab, rejected both on legitimate technical and policy grounds, and shipped a self-hosted Forgejo instance with actual ministries and municipalities already committing code to it. This is what digital sovereignty looks like when someone actually does the work instead of just writing about it.

Why GitHub got killed first

Let’s be honest GitHub losing a government contract isn’t exactly a scandal. It’s more like getting eliminated from a cooking show because you showed up with a microwave. The outcome was obvious the moment someone read the rules.

The Dutch government operates under a policy that says: prefer open source software when options are equally suitable. That’s not a vague suggestion buried in a footnote. It’s the kind of thing that shows up in procurement checklists and makes lawyers nervous when ignored. GitHub is proprietary software. Full stop. It didn’t matter how good the UI is, how many integrations exist, or how many developers already live there. Proprietary meant disqualified, and that was the end of that conversation.

What’s interesting though isn’t that GitHub lost it’s why that reasoning is actually correct. When your government’s source code lives on someone else’s SaaS platform, you don’t control it. You control a subscription. There’s a difference between storing your nation’s infrastructure code on a platform and owning where that code lives. One of those is a business arrangement. The other is sovereignty.

The Microsoft acquisition in 2018 didn’t help GitHub’s case either. That’s not FUD it’s a real procurement consideration for any government evaluating long-term vendor risk. When a $26 billion acquisition happens, the platform’s priorities shift. Governments move slowly enough that they’re still feeling that shift when everyone else has moved on.

GitLab made it further, which is fair. GitLab’s Community Edition is genuinely solid software it’s what a lot of self-hosting teams reach for when they want the GitHub experience without the GitHub dependency. But GitLab runs an open-core model, and that model has a very specific problem: the free version is real, but the version with everything your team eventually needs is not.

Every team I’ve been on has hit this wall. You start on GitLab CE, everything’s fine, and then six months later someone needs a feature advanced CI analytics, security dashboards, dependency scanning at scale and it’s sitting right there in the UI with a little lock icon on it. The open-core model is free-to-play with a battle pass. The base game works. The content you actually want is in the enterprise tier.

For a government platform that needs to stay fully open and auditable indefinitely, “free until it isn’t” isn’t good enough. GitLab CE is free software. GitLab EE is not. That split was enough to end the evaluation.

Enter Forgejo: the underdog with a squirrel

If you haven’t heard of Forgejo, you’re not alone. It doesn’t have a $7 billion valuation, a Super Bowl ad, or a developer relations team sending you swag. What it has is a GPLv3+ license, a democratic nonprofit governing it, and a squirrel mascot that somehow makes the whole thing feel more trustworthy than it has any right to.

Forgejo is a fork of Gitea, which itself was a fork of Gogs, which means it has the kind of lineage that open source projects accumulate when the community keeps deciding the current maintainers aren’t moving fast enough. The fork happened in 2022 when a subset of the Gitea community felt the project was drifting toward commercial interests. They took the code, stood up Codeberg e.V. as a governing body, and built something with an explicit commitment to staying fully free no enterprise edition, no proprietary upsell, no acquisition exit ramp.

Codeberg e.V. is a registered democratic nonprofit based in Germany. That structure matters more than it sounds. There’s no board of investors with a liquidation preference waiting for a strategic exit. No sales team trying to land an enterprise deal that quietly shapes the product roadmap. The governance model is the moat, and for a government platform that needs to be auditable and independent indefinitely, that moat is exactly what the evaluation was looking for.

Feature-wise, Forgejo covers the 90% of what most teams actually use on GitHub. Pull requests, issue tracking, code review, CI integration hooks, webhooks, org management it’s all there. It’s not trying to be GitHub. It’s trying to be a solid, self-hostable Git forge that doesn’t ask you to trust a vendor with your data, your uptime, or your roadmap.

The analogy that keeps coming to mind is Nextcloud. Nextcloud isn’t sexier than Dropbox. The setup takes longer, the UI isn’t as polished, and you have to think about your own backups. But it’s yours. When Dropbox changes its pricing, raises its limits, or gets acquired, your data isn’t caught in the crossfire. Forgejo is that, but for code hosting. Less glamorous than the SaaS alternative. Infinitely more yours.

Jan Vlug’s evaluation post laid all of this out methodically license model, governance structure, hosting requirements, community health. It wasn’t a hot take. It was the kind of calm, thorough technical writeup that procurement decisions should be based on and almost never are. Forgejo cleared every bar. So Forgejo won.

What’s already live and why it matters more than you think

Here’s where it stops being a policy story and starts being an engineering story.

code.overheid.nl had its soft launch on April 24, 2026. Developer advocate Tom Ootes wrote about it on the government’s developer portal, framing it not as a finished product being handed down but as a collective build early adopters were explicitly encouraged to file issues and open pull requests on the platform itself. Eat your own dog food, government edition.

The platform is a self-hosted Forgejo instance running on Dutch government infrastructure managed by SSC-ICT, the government’s shared IT services organization. It’s free for all government organizations to join. No licensing fees, no per-seat pricing, no surprise invoice when you add your fifteenth ministry.

Now here’s the part that made me stop scrolling.

The most prominent presence on the platform right now is Kiesraad the Dutch Electoral Council. They’ve pushed several repositories including Abacus, the software used for vote counting and seat distribution in Dutch elections, and e-KS, an electronic candidate nomination system.

Let that sit for a second. The software that counts votes in a national election is publicly hosted, publicly auditable, open source, and sitting on a government-controlled server. No black box. No “trust us.” Anyone can read the code, file an issue, or submit a pull request. That’s not just good open source practice that’s a fundamentally different relationship between a government and the people it serves.

Think about how many countries have their election-adjacent infrastructure running on proprietary systems or third-party SaaS platforms where independent audit is somewhere between difficult and impossible. The bar for public trust in election software is already low. Making it open and auditable doesn’t fix every problem, but it’s a meaningful step that most governments haven’t taken.

The Ministry of the Interior has the DAWO project on there their digital autonomous workplace initiative along with a DigiD source code release that was published under a freedom of information ruling. That last one is particularly interesting. The code went public not because the government volunteered it, but because a legal ruling required it. And now it lives on infrastructure the government actually controls.

On the organization side, the roster for a platform still in pilot is genuinely surprising. National ministries already signed up include Finance, Foreign Affairs, Agriculture, and Interior. Municipalities include The Hague, Utrecht, Leiden, and Arnhem. That’s not a proof of concept anymore. That’s adoption.

Tom Ootes framed the whole thing as building alongside the people who will use it rather than shipping something finished. That framing is quietly smart. Government software projects fail constantly because they get designed in isolation and handed to users who had no input. Starting in public, in pilot, with issues open that’s the right instinct even if it’s a slower path.

Europe is waking up slowly, quietly, correctly

The Netherlands didn’t invent this idea. They just executed it better than most.

France has been running code.gouv.fr for a while now. It’s a public inventory of open source software published by French public agencies. Germany has opencode.de, a similar initiative pushing government code into the open on self-hosted infrastructure. The pattern is consistent across every country doing this:

US cloud platform → data sovereignty concern → FOSS forge.

It’s not a coincidence. It’s GDPR doing its job.

When your government code lives on servers in Virginia, you have a problem. Not a theoretical one. A legal one. GDPR has real teeth, and “we use GitHub” is not a compliance strategy when the data in question belongs to citizens of a member state. The move toward self-hosted, EU-controlled infrastructure isn’t ideological it’s a legal and operational requirement that some governments are finally taking seriously.

Here’s the thing though. Nobody’s making noise about this.

If a VC-backed startup pulled off what the Netherlands just did evaluated the market, rejected the dominant players on principled technical grounds, shipped a self-hosted alternative with real adoption in under six months it would be a Product Hunt #1, a TechCrunch piece, and seventeen LinkedIn posts about “disruption.”

Instead it’s a quiet blog post from a developer advocate and a Forgejo instance that most devs outside the Netherlands have never heard of.

That asymmetry says something about where the dev community puts its attention. We celebrate funded startups shipping fast. We ignore governments shipping slowly and correctly. But slow and correct compounds. The Netherlands isn’t going to wake up one day and discover their code platform got acquired, deprecated, or pivoted into an enterprise product. They own it.

The “Europe buying a NAS instead of paying for iCloud” energy is real here. Yes, the NAS takes longer to set up. Yes, you have to think about your own redundancy. But when Apple changes its storage tiers, your files aren’t hostage to a pricing decision made in Cupertino.

Same logic. Different scale. Higher stakes.

GitHub won the developer market on network effects, not on being the objectively correct choice for every use case. For personal projects, open source collaboration, and public repos GitHub is great. For government infrastructure that needs to stay auditable, sovereign, and free from vendor risk indefinitely? The Netherlands just proved there’s a better answer.

And it has a squirrel mascot.

The quiet ones ship

There’s a version of this story where the Netherlands holds a press conference, brings in a minister to cut a ribbon on a server rack, and publishes a forty-page digital sovereignty strategy document that nobody reads.

They didn’t do that.

They hired an engineer to write an honest evaluation. They picked the right tool. They stood up the infrastructure. They opened it in pilot with real organizations committing real code. And then they wrote a blog post about it.

That’s it. That’s the whole move.

There’s something genuinely refreshing about watching an institution do the unglamorous work without performing it. No roadmap deck. No keynote. No waitlist with a referral bonus. Just a Forgejo instance, a squirrel mascot, and government ministries quietly pushing commits.

The part that sticks with me is Abacus. Election software. Publicly hosted. Openly auditable. In a world where public trust in institutions is running low and the black-box nature of critical infrastructure is a legitimate concern making your vote-counting code readable by anyone is a statement. Not a loud one. But a real one.

GitHub isn’t going anywhere. It’s still where most of the world’s open source code lives and where most developers spend their days. But the assumption that GitHub is the default, obvious, unquestioned choice for every organization is starting to crack. Slowly. One government forge at a time.

The Netherlands killed that assumption for themselves. France and Germany are doing the same. More will follow not because FOSS forges are trendy, but because the alternative is permanent dependency on platforms you don’t control, running on infrastructure outside your jurisdiction, governed by priorities that aren’t yours.

If you’ve been meaning to look into self-hosting your team’s code, this is a decent nudge. And if you want to see what a government actually shipping digital sovereignty looks like, code.overheid.nl is live right now.

Go look at it. The squirrel is worth it.

Are you self-hosting your team’s Git infrastructure? Is your organization still on GitHub by default or by choice? Drop your take in the comments.

Helpful resources

code.overheid.nl the live platform
Forgejo official site what it is and how to self-host
Codeberg e.V. the nonprofit governing Forgejo
Jan Vlug’s Git forge evaluation the blog post that started it
Tom Ootes soft launch post the April 24 announcement
code.gouv.fr France’s government FOSS platform
opencode.de Germany’s equivalent
Forgejo FAQ and governance how the project is actually run

30+ Linux environment variables hackers memorize on day one (and most devs never bother with

Thu, 07 May 2026 14:08:40 +0000

Alt: “Your Linux setup is leaking. These 30+ environment variables are why.”

You’ve been writing code on Linux for years. Maybe you run it in Docker, on a VPS, on your actual machine like a person with good taste. You know your way around a terminal. You feel comfortable there.

And yet.

Somewhere between your first apt install and your current career, you quietly decided that environment variables were a "good to know someday" topic. You learned PATH. You learned HOME. You maybe, on a heroic day, looked up what SHELL does. Then you moved on.

That’s fine. Until you’re debugging a production issue at an hour you’d rather not name, and the answer is buried in a variable you’ve never heard of. Or worse until someone who has done their homework walks into a system and does things you can’t explain because you never learned the control plane sitting right under your nose.

Here’s the uncomfortable truth: environment variables aren’t a Linux curiosity. They’re the configuration layer for every process, every shell session, every tool you run. Hackers the ethical, CTF-grinding, red-team-report-writing kind treat them like first-class knowledge. Most developers treat them like a footnote.

This article fixes that. We’re going through 30+ of them across three tiers: the essentials you should already know cold, the power-user variables most people skip entirely, and the ones that show up in every serious security engagement. By the end, you’ll have a mental model for this stuff that actually sticks.

TL;DR: There are dozens of Linux environment variables that shape how your system behaves. Most developers know three. This guide covers 30+, split into essentials, power-user config, and security-critical variables with real examples and the context to actually use them.

Tier 1: The essentials the 10 you should know cold

Most developers exist in a comfortable relationship with about three environment variables. PATH gets them where they need to go. HOME tells their tools where to store things. USER shows up occasionally in a script. That’s the whole map.

The problem is that “knowing” PATH and actually understanding what it does are two very different things. And when something breaks a command not found, a tool opening the wrong editor, logs full of encoding garbage the answer is almost always in one of these ten variables. You just didn’t know to look there.

Let’s close that gap.

PATH

The most consequential variable on your system. When you type any command, Linux doesn’t search every directory it walks through PATH left to right and stops at the first match.

echo $PATH
# /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

export PATH=/usr/local/bin:$PATH

Order matters more than most people realize. If you’ve ever installed a tool and gotten an older version back, PATH ordering is your suspect. The directory that appears first wins, every time.

HOME

The absolute path to your current user’s home directory. Nearly every application uses this to figure out where to read config, write cache, and store data.

echo $HOME
# /home/devtips

Change HOME and you change where everything lands. Useful to know when you’re scripting something that needs to behave differently across users.

USER

The username of whoever is running the current session. Simple on the surface, essential in any script that needs to branch based on who’s executing it.

echo $USER
# devtips

Don’t hardcode usernames in scripts. Read USER instead. Your future self will thank you when someone else runs it.

SHELL

The path to your active shell binary. This determines your scripting behavior, your tab completion, and your default prompt.

echo $SHELL
# /bin/bash

Never assume bash. In a lot of modern setups especially containers and minimal server installs you’re on dash, sh, or zsh. SHELL tells you what you’re actually dealing with.

PWD

Your current working directory, updated automatically every time you cd. More useful in scripts than in interactive use.

echo $PWD
# /home/devtips/projects

Use it to build absolute paths inside scripts instead of relying on relative paths that break the moment someone runs the script from a different directory.

HOSTNAME

The network name of the machine you’re on. The one you actually want in scripts that need to behave differently across dev, staging, and prod.

echo $HOSTNAME
# ubuntu-server

I’ve seen scripts that hardcode machine names. I’ve seen what happens when those machines get renamed. Set HOSTNAME checks in your scripts and stop praying.

LANG

Controls language, character encoding, and locale behavior across the entire system.

echo $LANG
# en_US.UTF-8

export LANG=en_US.UTF-8

Mismatched locales are responsible for some of the most baffling, hard-to-reproduce bugs in production. Logs showing up as question marks. Sorting behaving wrong. String comparisons failing in ways that make no sense. Always set this explicitly in production scripts don’t inherit whatever the system happens to have.

TERM

Tells applications what kind of terminal they’re dealing with, which determines what escape codes they use for colors, cursor movement, and formatting.

echo $TERM
# xterm-256color

If your terminal output looks garbled, colors aren’t rendering, or a tool is behaving like it’s running blind TERM is your first check. A lot of SSH sessions inherit a wrong TERM value and everything downstream breaks quietly.

EDITOR

Defines which text editor programs open when they need you to write something git commit messages, crontab entries, anything that spawns an interactive editor.

export EDITOR=vim

Set this once in your ~/.bashrc and every tool that respects it falls in line. Not setting it means you're at the mercy of whatever the system default is. On a lot of servers, that's ed. You do not want to meet ed unprepared.

TZ

Sets the timezone for the current session and any process that inherits the environment.

export TZ=America/New_York
# or
export TZ=Asia/Karachi

This one bites teams constantly. Two servers, same codebase, logs showing timestamps an hour apart someone forgot to standardize TZ across environments. In containerized systems especially, never assume the timezone is what you think it is. Set it explicitly and move on.

Those are your ten. If you can explain what each one does without looking it up, you’re already ahead of most people running Linux daily. If two or three were new good, now they’re not.

Tier 2: Power user variables most devs sleep on

Here’s where the gap actually opens up. Tier 1 variables are the ones you stumble into eventually they show up in error messages, Stack Overflow answers, and “getting started with Linux” guides. You learn them by accident.

Tier 2 doesn’t work that way. Nobody’s going to hand you these. They live in the corners of documentation pages, in experienced engineers’ dotfiles, in the kind of config that makes you think “wait, you can just do that?” the first time you see it. Most developers go years without touching them. Power users configure them on day one.

PS1

Your bash prompt is programmable. PS1 is the variable that controls exactly what it displays.

echo $PS1

export PS1="[\u@\h \W]\$ "
# Output: [devtips@ubuntu projects]$

\u is your username, \h is hostname, \W is the current directory. You can layer in colors, git branch names, exit codes from the last command, timestamps basically anything. Think of the default prompt like a stock game HUD. PS1 is where you remap everything to actually make sense for how you work. If you're still on the default, you're leaving information on the table every time you open a terminal.

HISTSIZE

Controls how many commands are kept in memory during your active session.

bash

echo $HISTSIZE
# 1000

export HISTSIZE=10000

The default on most systems is 1000. That sounds like a lot until you’re trying to find a command you ran three days ago and it’s gone. Power users set this to something large ten thousand, fifty thousand and treat their history like a searchable log of everything they’ve done. Pair it with Ctrl+R for reverse history search and you've got a lightweight audit trail of your own work.

We’ll revisit HISTSIZE in Tier 3 for very different reasons.

HISTFILESIZE

The on-disk companion to HISTSIZE. Controls how many commands get saved to ~/.bash_history when your session ends.

bash

echo $HISTFILESIZE
# 2000

export HISTFILESIZE=20000

These two variables are separate knobs and that trips people up. HISTSIZE is what’s held in memory during your session. HISTFILESIZE is what gets written to disk afterward. You can have a large in-session history and a small file, or vice versa. Set both deliberately instead of inheriting whatever your distro shipped with.

MANPATH

Defines where the man command looks for manual pages.

export MANPATH=/usr/local/share/man:$MANPATH

This one matters the moment you start installing tools to non-standard locations custom builds, things in /opt, tools you've compiled yourself. If man yourtool returns nothing, the manual exists somewhere your system doesn't know to look. Add the right path to MANPATH and it works immediately. Most people just Google the man page instead. Setting MANPATH is faster and works offline.

DISPLAY

Used by the X Window System to specify which display server to connect to.

echo $DISPLAY
# :0.0

export DISPLAY=:0.0

You won’t think about this variable until the exact moment you need it. That moment is usually: you SSH into a remote machine, try to open something with a GUI, and get a cryptic error about not being able to connect to a display. DISPLAY is what tells the application where to render. Get it right and the GUI opens on your local screen. Get it wrong and you’re reading X11 error messages at a time of day that tests your patience. Enable X11 forwarding in your SSH config and set DISPLAY accordingly.

MAIL

Points to the location of the current user’s mail spool where local system mail lands.

echo $MAIL
# /var/mail/devlink

Nobody thinks about this one. Cron jobs do. When a scheduled task fails or produces output, it doesn’t throw an error into the void it sends local mail. If MAIL isn’t set or nobody’s checking it, those failure messages have been quietly piling up unseen. This is genuinely how some cron jobs silently die for months before anyone notices. Check your mail spool occasionally. You might find a graveyard.

OSTYPE

Tells you what operating system you’re running on at the shell level.

echo $OSTYPE
# linux-gnu

The place this earns its keep is cross-platform shell scripting. If you’re writing a script that needs to run on both Linux and macOS and eventually you will be OSTYPE lets you branch cleanly without spawning a uname subprocess. Linux returns linux-gnu, macOS returns darwin, BSD variants have their own values. One variable, clean conditional logic, no forks.

COLORTERM

Signals to applications that your terminal supports true color full 24-bit RGB rather than the 256-color or 8-color fallback modes.

echo $COLORTERM
# truecolor

export COLORTERM=truecolor

This is one of those variables where not setting it correctly causes problems that look like something completely different. Your terminal supports full color. Your tool supports full color. But COLORTERM isn’t set, so the tool falls back to 256 colors and everything looks slightly off in a way you can’t quite name. Set it explicitly in your dotfiles and stop wondering why your color scheme looks duller on one machine than another.

Those eight variables separate the people who use Linux from the people who actually configure it. None of them are secrets they’re all in the documentation. But documentation doesn’t tell you why they matter or when you’ll need them. That’s what experience does, and now you’ve got a head start.

Tier 3: The hacker toolkit

Disclaimer: Everything in this section is for ethical hacking, penetration testing, CTF challenges, and defensive security awareness. Understanding what attackers use is how defenders build better detection. Use these only on systems you own or have explicit written permission to test.

This is where the article gets interesting.

Tiers 1 and 2 are about configuration. Tier 3 is about control. These variables don’t just shape how your shell looks or which directories get searched they touch process loading, traffic routing, credential handling, library injection, and session forensics. They’re the reason experienced security engineers audit environment variables on any box they’re responsible for, and why attackers learn them before almost anything else.

None of this is exotic. It’s all documented. It’s all standard Linux. That’s precisely what makes it dangerous.

HISTSIZE=0 and HISTFILESIZE=0

export HISTSIZE=0
export HISTFILESIZE=0

Set both to zero at the start of a session and you get complete command history suppression. Nothing gets stored in memory, nothing gets written to disk when the session ends. Standard OPSEC in any authorized red team engagement.

Why defenders care: if you’re doing incident response on a compromised machine and find these set early in a session especially in .bashrc or /etc/profile someone was thinking about logging before they started working. That's not accidental configuration. That's a signal.

http_proxy and https_proxy

export http_proxy="http://10.10.10.10:8080"
export https_proxy="http://10.10.10.10:8080"

Any process that respects these variables routes its traffic through the specified proxy. This is how security testers intercept application traffic through tools like Burp Suite during an engagement without touching application code, without modifying config files, without restarting services. Set the variable, run the process, read the traffic.

The lowercase versions (http_proxy) are respected by most command-line tools. Some applications only check the uppercase versions. In practice, set both.

SSL_CERT_FILE and SSL_CERT_DIR

export SSL_CERT_FILE=/path/to/ca-bundle.pem
export SSL_CERT_DIR=/path/to/ca-certificates

Processes that read these variables trust the certificates you point them at. In an authorized testing context, this is how you get an application to trust your proxy’s self-signed certificate so you can inspect encrypted HTTPS traffic without the tool throwing certificate errors and refusing to connect.

Combined with http_proxy, these two variables give you a complete traffic interception setup without touching a single config file inside the application.

LD_PRELOAD

export LD_PRELOAD=/tmp/custom.so

This is the most powerful variable on this list. When set, the dynamic linker loads your specified shared library before anything else before the C standard library, before every other dependency the binary has. Functions in your library override functions in any other library the process loads.

The implication: you can intercept system calls, hook functions, and fundamentally alter the behavior of a running process entirely from outside its source code. No recompilation. No patching. Just an environment variable.

In CTF challenges and authorized penetration tests, LD_PRELOAD shows up in privilege escalation paths, sandbox bypasses, and function hooking scenarios. It’s also widely used legitimately memory profilers, debugging tools, and performance analyzers all use this same mechanism.

Why defenders care: monitor for LD_PRELOAD pointing to paths in /tmp or world-writable directories. Legitimate tools don't load libraries from there. Something else does.

LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/tmp/mylib:$LD_LIBRARY_PATH

Defines the directories the dynamic linker searches for shared libraries. By prepending a custom directory, you can substitute your own version of any library a binary depends on the binary loads your library instead of the real one and never knows the difference.

This is the environment variable equivalent of DLL hijacking on Windows. Same concept, same impact, different operating system. The attack works because most binaries trust that the libraries they load are the ones they expect.

SUDO_ASKPASS

export SUDO_ASKPASS=/tmp/fake-prompt
sudo -A whoami

When sudo is called with the -A flag, instead of prompting for a password in the terminal, it executes whatever program is set in SUDO_ASKPASS and uses that program's output as the password. In authorized social engineering simulations, this technique demonstrates how applications and GUI wrappers around sudo can be leveraged to intercept credential input without the user realizing the prompt they're responding to isn't the real one.

LD_DEBUG

export LD_DEBUG=libs
./somebinary

A reconnaissance variable. Setting it makes the dynamic linker print detailed output about every library being loaded the full path, the search order, which directories were checked, which version was found. No special permissions required. No tools to install.

In authorized engagements, this is how you identify which libraries a binary depends on and whether any of them are loaded from locations that could be hijacked with LD_LIBRARY_PATH. It turns library loading from a black box into a visible, auditable process.

IFS

export IFS=$'\n'

IFS Internal Field Separator tells bash how to split strings into tokens. The default is space, tab, and newline. Changing it changes how every command and script in your session parses its inputs.

In exploit development and CTF challenges, subtle IFS manipulation breaks input validation in scripts that weren’t written with this in mind. A script that sanitizes space-separated input suddenly behaves differently when the separator changes. It’s a small variable with outsized consequences in poorly written shell code which, in production environments, is not rare.

GDBINIT

export GDBINIT=/tmp/custom-gdbinit

GDB the GNU Debugger reads initialization commands from the file specified here on startup. In authorized assessments, this demonstrates how developer tooling itself can become an execution vector when environment variables aren’t controlled. CI pipelines, developer workstations, build servers anything that invokes GDB in an environment where GDBINIT can be influenced is a potential target.

TMOUT

export TMOUT=1

Sets bash to automatically terminate after the specified number of seconds of inactivity. Setting it to 1 closes a shell almost immediately after it goes idle. In an authorized engagement context, this is how you ensure a session closes cleanly without leaving an open shell exposed on a system you’re no longer actively using.

For defenders, it’s also worth setting in /etc/profile on any server where unattended sessions are a risk. Idle shells with elevated privileges sitting open are an opportunity nobody needs to create.

XDG_CONFIG_HOME

export XDG_CONFIG_HOME=/tmp/custom-config

Applications that follow the XDG Base Directory Specification read their config from this path. Redirect it to a controlled directory and you supply a completely custom configuration to any XDG-compliant application without modifying a single file on the real filesystem. Clean, contained, reversible.

Thirteen variables. Every one of them is in the man pages, in the official documentation, available to anyone who reads carefully enough. The difference isn’t access to secret knowledge it’s whether you took the time to understand what was already there.

How to audit and manage your environment properly

You’ve now got 30+ variables in your head. The natural next question is: what’s actually set on my system right now, and how do I control it properly?

Most people interact with environment variables reactively they set something when a tool breaks, forget where they set it, and spend twenty minutes debugging the wrong file six months later. The fix is understanding the three distinct layers your environment is built from, and having a handful of commands you can reach for without thinking.

Viewing what’s currently set

Four commands, four slightly different outputs:

# Everything exported to the current environment
printenv

# A specific variable
printenv PATH

# All exported variables (similar to printenv)
env

# All shell variables including unexported ones - verbose
set | less

printenv and env show you what's exported — what child processes will inherit. set shows everything including shell-local variables that don't get passed down. For most auditing purposes, printenv is what you want. For thoroughness, set | less and scroll.

Setting variables know your layers

This is where most confusion lives. There are three distinct scopes and they don’t interact the way people assume:

# Current session only — gone when you close the terminal
export MY_VAR="value"

# Permanent for your user - survives reboots, applies to new sessions
echo 'export MY_VAR="value"' >> ~/.bashrc
source ~/.bashrc
# System-wide for all users - requires root
echo 'MY_VAR="value"' >> /etc/environment

The hierarchy goes: system (/etc/environment) → user shell config (~/.bashrc, ~/.bash_profile) → current session (export). Each layer can override the one above it. When a variable is behaving unexpectedly, you're almost always looking at a conflict between two of these layers something set in /etc/environment getting overridden in .bashrc, or a session export shadowing both.

I’ve personally spent an embarrassing amount of time debugging a “why is this variable always wrong” issue that turned out to be set correctly in .bashrc, overridden in .bash_profile, and then overridden again by a script that exported it fresh on every run. Check all three layers before assuming the system is broken.

Removing and locking variables

# Remove a variable from the current session
unset MY_VAR

# Lock a variable so it can't be modified in the current session
readonly SECURE_VAR="value"

unset is straightforward. readonly is underused once set, any attempt to modify or unset that variable in the current session returns an error. Useful for variables that should never change after initialization, like paths to critical binaries in a script.

Passing variables to a single command without exporting

MY_VAR="value" some-command

This sets the variable in the environment of some-command only it doesn't persist to your shell, doesn't affect other processes, disappears immediately after the command finishes. Useful for one-off overrides without polluting your session. A lot of developers don't know this syntax exists and reach for export when they don't actually need it.

Auditing for suspicious variables

On any machine you’re responsible for especially one you’ve inherited or that’s been flagged for investigation:

# Check startup files for variables that shouldn't be there
grep -r "LD_PRELOAD|HISTSIZE=0|SUDO_ASKPASS" <br>  /etc/profile.d/ ~/.bashrc ~/.bash_profile ~/.profile

If any of those turn up unexpectedly, don’t assume it’s a misconfiguration. Investigate. Their presence in startup files especially HISTSIZE=0 and LD_PRELOAD pointing to /tmp is a meaningful signal, not noise.

That’s the full management toolkit. View, set, scope, lock, audit. Five operations that cover everything you’ll need in day-to-day work and in the more interesting situations this knowledge puts you in reach of.

Security rules most hardening guides forget

Configuration guides cover firewalls. They cover SSH keys. They cover fail2ban and port knocking and a dozen other things that are genuinely important. Environment variables don't make the list often, which is exactly why this attack surface stays underappreciated.

A few rules that actually matter:

Never store secrets in plain environment variables. They show up in printenv. They show up in /proc/<pid>/environ readable by any process running as the same user. They show up in crash dumps, in CI logs when a build fails mid-run, and in container inspection output if your orchestration config is even slightly misconfigured. Use a secrets manager. Pass secrets through files with tight permissions, not shell exports.

Lock down critical variables in scripts:

readonly PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
readonly SECURE_VAR="value"

Harden your history file:

chmod 600 ~/.bash_history

The default permissions on bash_history are often more permissive than they should be. Other users on a shared system can read your command history. One command does the job.

Audit environment files on every box you manage:

grep -r "LD_PRELOAD|HISTSIZE=0|SUDO_ASKPASS" <br>  /etc/profile.d/ ~/.bashrc ~/.profile ~/.bash_profile

Run this on any system you’ve inherited, any container base image you didn’t build yourself, any machine that’s been flagged in an incident. Unexpected results here aren’t curiosities they’re findings.

Conclusion

Here’s a slightly uncomfortable opinion: most Linux security hardening guides are written for people who already know this stuff. They assume you understand the environment variable layer, so they skip it entirely and go straight to network rules and access controls. That assumption leaves a real gap one that shows up repeatedly in CTF writeups, in post-incident reports, and in the gap between developers who use Linux and engineers who understand it.

Environment variables are not a trivia topic. They’re the control plane for every process running on your system. The 30+ variables in this article aren’t exhaustive they’re the ones that matter most, the ones that explain the most behavior, and the ones that show up when things go wrong or when someone with bad intentions goes right.

The terminal is the most honest interface on any system. It hides nothing if you know what to ask. Start asking better questions.

As containers and srverless architectures continue to take over infrastructure, environment variables are increasingly the primary configuration layer injected at runtime, scoped per service, and almost never audited properly. That makes this knowledge more relevant every year, not less.

Go through this list on a test machine. Run each export. Watch what changes. The best way to internalize this is to break something deliberately in a safe environment and understand exactly why it broke.

And if you’ve got a cursed environment variable story a HISTSIZE=0 you found where it shouldn’t be, an LD_PRELOAD incident, a production outage that traced back to TZ being unset drop it in the comments. I genuinely want to hear it.

Helpful resources

Linux man page: ld.so full documentation on LD_PRELOAD, LD_LIBRARY_PATH, LD_DEBUG behavior
GTFOBins practical reference for how environment variables feature in privilege escalation paths
Arch Wiki: Environment Variables one of the clearest practical guides on scoping and management
XDG Base Directory Specification official spec for XDG_CONFIG_HOME and related variables
OWASP Secrets Management Cheat Sheet why plain environment variables are the wrong place for credentials

Software development is having a second chance. Nobody saw this coming.

Thu, 07 May 2026 14:01:55 +0000

Everyone spent two years arguing about whether AI would kill the craft. Turns out it might be the thing that saves it.

Here’s a thing nobody wants to admit out loud: while half the dev community was busy writing “AI is going to take our jobs” threads, the other half was quietly shipping more software than ever before. Solo. Faster. With fewer standups.

There’s something deeply ironic about that.

For the past two years, the loudest conversation in tech has been about what AI is taking away from software development. The junior roles. The craft. The thinking. And look some of that is real. Nobody’s going to pretend the job market looks the same as it did in 2022. But here’s the angle that keeps getting skipped in all those LinkedIn hot takes: software development as a practice the act of building systems, shipping products, solving real problems with code is quietly having a moment.

Not a crisis. A comeback.

GitHub recently had to redesign its entire infrastructure to handle 30x its previous scale not because of more developers, but because agentic workflows and AI-assisted development exploded so fast the old architecture couldn’t keep up. That’s not a dying field. That’s a field that just found a second gear.

This article is about that second gear.

TL;DR: AI didn’t kill software development. It stress-tested it, stripped out the tedious parts, and handed the keys back to engineers who know what to do with them. Here’s what that actually looks like.

The myth of the dying craft

Let’s get something straight. “AI is killing software development” is doing a lot of heavy lifting as a take, and most of the people repeating it are confusing two very different things: the job market and the craft.

The job market? Yeah, it’s shifting. That’s real and worth a separate conversation. But the craft the actual act of designing systems, writing code, shipping things that work is not dying. It’s redistributing. And there’s a massive difference.

Here’s the tell: if software development were actually collapsing, you wouldn’t see GitHub scrambling to redesign its infrastructure for 30x capacity because agentic development workflows accelerated faster than anyone predicted. You wouldn’t see repository creation, pull request activity, and API usage all trending sharply upward at the same time. Those aren’t the numbers of a dying discipline. Those are the numbers of a discipline that just removed a ceiling.

Think about what the day-to-day used to look like. I remember spending a full afternoon building a pagination component. Not because it was intellectually interesting. Not because it required deep thought. Because there was no faster way. Someone had to write it, so someone did. That was the job a mix of genuinely hard problems and an enormous amount of mechanical, repetitive work that just had to get done.

AI ate the second category. Almost entirely.

And this is the part people keep misreading as loss. Calculators didn’t kill mathematicians. They killed arithmetic drudgery, which freed mathematicians to do more actual mathematics. Same play. Different stack. The pagination component was never the craft it was the toll booth before the craft.

The craft is still there. It just finally got a fast lane.

The 50-year debt AI is finally paying off

Software development has been dragging a 50-year backpack of accumulated bad decisions. Bad abstractions that got copy-pasted into frameworks. Over-engineered patterns that became industry standards before anyone could stop them. Wheels reinvented so many times they stopped being round.

Every senior dev has a version of this story. Mine involves a codebase where three different teams had independently written three different date formatting utilities none of which talked to each other, all of which were “the right way” according to whoever wrote them. Nobody meant for it to happen. It just did. Because software moves fast, documentation is always someone else’s problem, and the cost of fixing old decisions is always higher than the cost of living with them.

For decades, that debt just… compounded. Quietly. In the walls.

Here’s what’s actually interesting about AI: it’s the first thing that moves fast enough to surface and bypass that debt at the same time. Auth systems that used to take a week to spec out and another two to implement correctly? Afternoon work now. CI/CD pipelines that felt like a rite of passage the kind where you hadn’t really earned your DevOps stripes until you’d genuinely cried over a YAML indent? Scaffolded in minutes, debugged in context.

It’s not just that things are faster. It’s that old patterns are getting stress-tested at a pace humans alone never could have managed. The bad abstractions are getting caught earlier. The reinvented wheels are getting spotted before they ship.

Think of it like inheriting a house with thirty years of questionable DIY plumbing except now you have a contractor who can see every pipe in the wall before touching a single one.

The debt isn’t gone. But for the first time, we have a tool that doesn’t just add to it. Sometimes it actually pays some of it back.

That’s not a small thing. That’s kind of a big deal.

One dev, infinite surface area

Here’s the shift that doesn’t get talked about enough and it’s the one that actually changes everything.

The real unlock from AI-assisted development isn’t that code gets written faster. It’s that one engineer can now own and reason about dramatically more surface area than before. Not a little more. A lot more. Backend, DevOps, frontend, QA, monitoring the full stack of concerns that used to require four different people with four different specializations can now live inside one person’s working context.

I watched a friend ship a SaaS product in six weeks. Solo. Full auth layer, billing integration, CI/CD pipeline, a halfway decent UI. In 2020, that same scope took his startup four months with a team of three. Same person. Same skills. Wildly different output. The only meaningful variable was the tooling.

Tools like Cursor, Aider, and Claude Code aren’t just autocomplete on steroids. They’re surface-area expanders. They let a single engineer hold more of the system in their head or at least in their context window without dropping pieces of it on the floor.

This is the second chance for an archetype that never quite fit the enterprise era: the solo builder. The person who could see the whole product but never had the hours to execute all of it alone. That person now has a legitimate shot.

It’s not that one player got dramatically better overnight. It’s that the map got smaller and the spawn points moved. A solo dev in 2025 starts the game with resources that a small team in 2019 would’ve spent months unlocking.

The surface area didn’t shrink. The headcount required to cover it did.

What “good engineering” means now

Something quiet is happening to the definition of a good engineer. It’s not being announced anywhere. There’s no RFC, no industry memo, no Stack Overflow post with 4,000 upvotes explaining it. It’s just… shifting. And if you’re not paying attention, you’ll miss the transition entirely.

For most of software’s history, “good engineer” meant someone who could write correct, efficient code across a broad range of problems. The person who knew the right data structure without Googling it. Who could debug a memory leak at midnight without losing their mind. Who had enough language fluency to move fast without making a mess.

That still matters. But it’s no longer the whole game.

A colleague of mine spent three hours chasing a bug that an AI tool introduced into a service. Genuinely frustrating, genuinely subtle. Then he spent thirty minutes giving the same tool the right context the actual constraints, the edge cases, the downstream dependencies and watched it fix the bug cleanly. The skill wasn’t in writing the fix. It was in knowing exactly what information to hand over and when.

That’s the shift. Good engineering now has a new layer on top: judgment, taste, and the ability to reason about systems at a level where you’re directing work rather than just executing it.

Think less “master chef who cooks every dish” and more “executive chef who knows exactly what to order, from where, and what to do when it arrives wrong.”

AI is a very fast, occasionally overconfident intern. It needs direction. It needs someone who can spot when it’s confidently wrong which, if you’ve used any of these tools seriously, you know happens more than the demos suggest.

The engineers who’ll thrive aren’t the ones who out-type AI. They’re the ones who out-think it.

The uncomfortable truth about the second chance

Second chances come with conditions. This one’s no different, and it would be dishonest to skip past that part.

Junior developers have a harder path right now. The entry-level roles that used to absorb new grads the ones that were repetitive enough to be learnable and scoped enough to be survivable a lot of those look different now. Some are gone. That’s worth saying clearly, without dressing it up as “the market is evolving” or some other phrase that means the same thing while feeling less uncomfortable.

But here’s where it’s worth zooming out.

Every major shift in this industry looked like the end from inside it. Cloud computing arrived and everyone asked why anyone would ever manage their own servers again. Docker showed up and the “but it works on my machine” era started dying. Each time, the engineers who leaned in early who treated the new thing as infrastructure to understand rather than a threat to outlast came out with more leverage, not less.

This moment is opt-in. That’s the uncomfortable part. The second chance doesn’t land in your inbox automatically. It goes to the people who decide to pick it up.

The craft is still here. The problems are still real. The systems still need someone to design them, reason about them, and take responsibility when they fall over at the worst possible moment.

AI doesn’t do that last part. Not yet. Probably not for a while.

That gap is the opportunity.

The craft didn’t die. It leveled up.

Here’s where I land on this, for whatever it’s worth from someone who’s been watching this industry long enough to have strong opinions about tab versus space debates that nobody asked for.

Software development isn’t being replaced. It’s getting a difficulty reset. The grind that used to live in the boilerplate, the scaffolding, the “I’ve written this exact thing four times across three jobs” work that part is going away. And honestly, good riddance. Nobody got into this field because they loved writing the fifteenth variation of a user authentication flow.

The second chance is real. But it belongs to engineers who are willing to see AI as infrastructure the same way we eventually stopped arguing about whether to use the cloud and just started building on it.

The next decade of software will be built by fewer people, moving faster, covering more ground. The quality ceiling on everything they ship will depend almost entirely on the judgment, taste, and systems thinking of the humans steering it. That’s not a demotion. That’s actually a more interesting job than the one that came before it.

So the question worth sitting with isn’t “is AI taking over software development?”

It’s: are you picking up the controller or not?

Because the game didn’t end. The respawn screen just looked a lot like a warning, and most people stopped reading before the countdown finished.

Drop your take in the comments second chance or last chance? I want to know where you land.

Helpful resources

Kubernetes 1.36 killed your webhooks. Here are 10 other things it quietly changed.

Thu, 07 May 2026 13:56:43 +0000

Haru dropped with a Hokusai painting and a calligraphy inscription. Buried underneath all that poetry is a release that rearranged how your cluster actually works.

There’s a tradition in Kubernetes releases where the changelog reads like a corporate memo dry, dense, and written for people who already know what they’re looking for. Kubernetes 1.36 broke that tradition in the most unexpected way. The release is named Haru a Japanese word that carries three meanings at once: spring, clear skies, far-off horizons. The logo is a reimagining of Hokusai’s Red Fuji, with the Kubernetes helm floating in the sky above the mountain. The calligraphy brushed across it translates to

“soar into clear skies; toward tomorrow’s sunrise.”

And I’m sitting here like okay, K8s. We’re doing this now.

The poetry is earned though, because underneath it, 1.36 is one of the more consequential releases in recent memory. Not because it introduced a dozen shiny new alpha features (it did that too), but because it finally graduated things that have been in progress for years, killed things that should’ve died ages ago, and gave platform engineers actual tools to stop duct-taping their clusters together.

TL;DR: Mutating Admission Policies are GA and webhooks are on notice. User Namespaces finally hit stable after four years in alpha. Ingress NGINX is officially retired not deprecated, retired. DRA grew up for real GPU scheduling. OCI volumes made ML model distribution less embarrassing. HPA scale-to-zero is now a thing. And the gitRepo volume type is gone, eight years after everyone was told it was going away.

Webhooks: cooked

If you’ve run admission webhooks in production, you already know the drill. Every API request hits your webhook server on the way in. That server lives outside the cluster, needs its own deployment, its own TLS, its own on-call rotation. And when it goes down and it will go down pod scheduling stops. Not degrades. Stops.

I once spent a week chasing a stalled CI/CD pipeline. Deployments failing with no clear pattern, logs noisy but useless. Root cause: an OPA Gatekeeper webhook silently dropping pod CREATE requests under load. A week. For a dropped request.

That entire class of problem disappears with MutatingAdmissionPolicies, which hit GA in 1.36. Instead of an external service, you write mutation logic as CEL expressions evaluated inline inside the API server. No external server. No TLS certs to rotate. No 3am pages because the webhook pod got evicted. Define it as a Kubernetes object, version-control it in Git, ship it through your normal GitOps flow.

The asterisk: if your mutation logic needs to call an external service, you still need a webhook. But that’s maybe 20% of real-world use cases. Label injection, sidecar prepending, field defaulting all of that is now native and in-process.

Webhooks aren’t gone. They’re just no longer the only option. And for most teams, that’s a massive operational relief.

Root inside a container was always fiction

Here’s something nobody likes to say out loud: when your container runs as root, it’s running as root on the host too. The container boundary exists, sure but if something escapes it, it lands on your node with full administrative power. That’s not a hypothetical. Container breakout vulnerabilities are real, documented, and have CVEs attached to them.

The fix has existed in Linux for years. User Namespaces let you map UID 0 inside the container to an unprivileged user on the host. Your process thinks it’s root. The kernel knows it isn’t. If it escapes, it lands with essentially nothing.

Kubernetes has been working toward this since v1.25 in August 2022. Four years of alpha, beta, validation on production workloads, and edge case hunting. In 1.36 it’s finally Stable. You enable it with one field:

spec:
  hostUsers: false

That’s it. That’s the fence with a lock.

Before this, running genuinely rootless containers in Kubernetes meant layering on gVisor, Kata Containers, or some combination of third-party tooling and crossed fingers. Now it’s native, stable, and production-ready with no extra dependencies.

One watch-out: Images built assuming real root privileges may behave unexpectedly under UID remapping. Test on non-critical workloads first before rolling it cluster-wide.

Ingress NGINX is dead. Not deprecated. Dead.

On March 24, 2026, Kubernetes SIG Network and the Security Response Committee officially retired Ingress NGINX. No more releases. No bug fixes. No security patches. If you’re running it today, you’re running unsupported software in production and the maintainers have left the building.

This isn’t a soft deprecation with a two-release runway. There’s no “we recommend migrating by v1.38.” It’s done. The flaws were too deep, the maintainer bandwidth wasn’t there, and the Security Response Committee signed off on pulling the plug.

The uncomfortable part is how many production clusters are still running it. Ingress NGINX became the default answer to “how do I route traffic in Kubernetes” for years. It was good enough, it was everywhere, and nobody had a strong reason to migrate until now.

The migration path is Gateway API v1.5. It gives you structured routing, cross-namespace references, and a proper separation between infrastructure concerns and developer concerns things Ingress never cleanly handled. The Ingress2Gateway project hit 1.0 in March 2026 specifically to help with this transition. The tooling exists. The excuses don’t.

If you’re on Ingress NGINX, this is the conversation to have with your team this sprint, not next quarter.

DRA grew a brain. GPU scheduling makes sense now.

If you’ve ever tried to schedule GPU workloads on Kubernetes before Dynamic Resource Allocation existed, you know what it felt like. Node selectors, custom labels, resource limits that didn’t reflect actual hardware topology, vendor-specific device plugins that all invented their own interfaces. It worked, technically. The way duct tape works, technically.

DRA has been Kubernetes’ answer to this for a few releases now a proper framework for scheduling specialized hardware like GPUs, accelerators, and custom silicon. 1.36 is the release where it stops feeling experimental.

A few things landed here that matter. Taints and tolerations for hardware devices: you can now take a specific GPU offline for maintenance without touching the rest of the cluster. Same mental model you already use for nodes, applied to individual devices. Resource Health Status is now surfaced through standard Kubernetes tooling if a GPU is unhealthy, it shows up like any unhealthy pod or node. No custom monitoring stack per vendor. No guessing. Just a status field that’s either green or it isn’t.

Per-pod DRA resource visibility is also locked to GA, meaning monitoring tools, billing systems, and operators can reliably query exactly what hardware each pod has been allocated. That matters a lot when your GPU cluster costs more per hour than most engineers’ daily rate.

The AI/ML infra angle here is obvious. Training jobs are expensive. Scheduling a 12-hour run onto a degraded GPU because your health checks lived in a vendor sidecar that missed a signal is the kind of thing that ends sprints. 1.36 starts fixing the foundation.

Shipping ML models in init containers was embarrassing

Getting a large model into a container has always been an exercise in picking the least bad option. Bake it into the image and watch your pull times become a running joke. Use an init container to download it at runtime and accept the complexity and failure modes that come with that. Fight ConfigMap size limits trying to get config artifacts in. Build a custom distribution pipeline and maintain that forever.

None of these are good. All of them are common.

OCI VolumeSource hits GA in 1.36 and it’s the answer that should’ve existed earlier. Reference any OCI image as a volume. Kubernetes pulls it and mounts the contents into your pod exactly like a regular volume. Your 40GB model lives in its own OCI artifact, versioned and distributed through the same registry infrastructure you already use. Your app container stays lean. Updates to the model don’t require rebuilding the app image.

volumes:
  - name: model-weights
    image:
      reference: registry.example.com/models/llama-weights:v3

For AI/ML workloads specifically this is a meaningful quality-of-life change. Model and code have different update cadences, different owners, and different size profiles. Treating them as separate artifacts that get composed at runtime is just the correct architecture. OCI VolumeSource makes that native instead of something you have to engineer around Kubernetes to achieve.

HPA scale-to-zero: serverless K8s without the serverless tax

Every team has them. Pods sitting at 0.3% CPU at midnight, burning compute budget, waiting for a queue message that won’t arrive until morning. You know you should scale them down. You also know that wiring up a custom scaler, a KEDA setup, or a managed FaaS layer just to handle idle workloads is its own operational surface area to maintain.

1.36 introduces alpha support for HPA scale-to-zero for Object and External metrics. minReplicas: 0 is now a real configuration, not a validation error.

spec:
  minReplicas: 0
  maxReplicas: 50
  metrics:
    - type: External
      external:
        metric:
          name: sqs_queue_length
        target:
          type: AverageValue
          value: 10

Queue hits zero, pods go to zero. Message arrives, HPA spins them back up. Combine with Karpenter and the node drains too. That’s native scale-to-zero serverless architecture without handing control to a managed FaaS platform or bolting on a separate autoscaling tool.

It’s alpha, so you need to enable the HPAScaleToZero feature gate explicitly. More importantly audit your existing HPAs. If you're not explicitly setting minReplicas: 1 somewhere, your workloads may now behave differently than you expect. That's the kind of silent change that shows up in a production incident, not a code review.

SSH-ing into nodes to check logs was always shameful

You know the sequence. Something’s wrong on a node. You find the bastion host. You SSH in. You navigate to the worker node. You run journalctl -u kubelet and scroll through walls of output trying to find the one line that explains why your pods aren't scheduling. Meanwhile the incident is live and your team is waiting.

Every engineer who’s run Kubernetes in production has done this at least once. Most have done it more times than they’d like to admit.

Node log queries hit GA in 1.36. With NodeLogQuery enabled on the kubelet and enableSystemLogQuery set in your kubelet config, you can query node-level logs kubelet logs, system service logs directly through kubectl. No SSH. No bastion. No explaining to your security team why you needed direct node access during an incident.

kubectl get --raw "/api/v1/nodes/<node-name>/proxy/logs/kubelet"

It’s not glamorous. It’s not the kind of feature that gets a conference talk. But the number of minutes lost per engineer per year to that SSH chain in production is genuinely non-trivial, and now it’s gone.

This was SIG Windows work through KEP-2258, which also means Windows nodes get full parity here something that’s historically lagged behind Linux in the observability tooling space.

The kubelet was handing out too much access. Fixed.

The kubelet exposes a gRPC API that monitoring agents, device plugins, and observability tools use to query what’s running on a node which pods are scheduled, what hardware resources they’ve been allocated, container states. Useful stuff. Stuff that a lot of tools legitimately need.

The problem was access granularity. Getting a monitoring tool the access it needed often meant granting broad kubelet permissions. In regulated environments PCI-DSS, FedRAMP, SOC 2 that’s not a risk you can quietly accept. It’s a finding waiting to happen.

Fine-grained kubelet API authorization hits GA in 1.36. Tools get exactly the permissions they need, scoped to the specific API surfaces they actually call. Nothing broader. The least-privilege model that should’ve been there from the start is now the stable, production-ready default.

External ServiceAccount token signing also graduates to GA in this release. If your compliance framework requires key management outside Kubernetes’ default signing setup or you’re running in an environment where the control plane’s signing keys need to live in an external KMS this gives you a native, stable path to that without third-party workarounds.

Neither of these features will show up in a demo. Nobody’s going to tweet about kubelet auth granularity. But for teams running Kubernetes under any kind of compliance requirement, these two graduating to stable quietly removes two items from the audit findings list that have been sitting there for a while.

Your PVCs now tell you when they were last used

Orphaned PersistentVolumeClaims are one of those slow, invisible cost drivers that nobody notices until someone pulls up the storage bill and starts asking uncomfortable questions. PVC gets created, workload gets deleted or redeployed, claim sits there bound and unused, and the underlying disk keeps billing. Multiply that across a busy cluster over a few months and it adds up faster than you’d think.

Before 1.36, identifying idle PVCs meant either building a custom controller, running a third-party cleanup tool, or doing periodic manual audits none of which are things anyone actually wants to own.

1.36 adds an unusedSince timestamp field to PersistentVolumeClaimStatus. The PVC protection controller now stamps it when the last pod referencing that claim is deleted or hits a terminal state. When a new pod mounts it again, the field clears back to nil.

status:
  phase: Bound
  unusedSince: "2026-03-10T14:22:00Z"

Two states. Either it has a timestamp idle since that moment or it’s nil, meaning it’s currently mounted or has never been used. Simple, native, queryable through standard kubectl and the API.

It’s alpha, so it’s not on by default yet. But the pattern it enables list all PVCs where unusedSince is older than 30 days, review, clean up is something teams have been building custom tooling to approximate for years. Now it's just a field.

The gitRepo volume: eight years after deprecation, finally gone

Deprecated in v1.11. June 2018. That’s not a typo.

For eight years the gitRepo volume type lived in the Kubernetes codebase like a haunted house nobody wanted to deal with. It let you populate a volume directly from a Git repository at pod startup which sounds convenient until you think about what that actually means. Arbitrary git clone operations running with pod-level permissions, no real sandboxing, a well-documented attack surface. It was deprecated almost immediately after people understood the implications.

And yet there it sat. Release after release. Deprecation notice in place, removal perpetually deferred.

1.36 finally pulls it out. If you’re somehow still using gitRepo volumes in 2026, this is the migration you should have done in 2019. Init containers with a proper git clone step, or a CI/CD pipeline that bakes artifacts into the image, are both better in every dimension.

Also worth flagging in the same breath: externalIPs in the Service spec is deprecated in 1.36, with full removal planned for v1.43. It's been a known attack vector since CVE-2020-8554. If it's in your configs, start the conversation now rather than doing it in a panic seven releases from now.

The broader signal here is worth noting. Kubernetes is enforcing its own deprecation timeline now, not just suggesting it. That’s good for the project’s long-term hygiene and a sign the maintainers are serious about not carrying dead weight forever.

Haru didn’t come to play

There’s a version of this release that gets written off as a maintenance drop. No dramatic new directions, no paradigm-shifting alpha features with a flashy demo. Just a lot of things graduating, a lot of debt getting cleared, and a handful of quiet improvements that compound over time.

That reading misses the point entirely.

1.36 is the release where Kubernetes starts filling its own gaps instead of expecting you to build scaffolding around them. Admission webhooks were a workaround that became an institution MAPs replace the institution. Container root isolation required third-party tooling for years User Namespaces makes it native. GPU scheduling was a patchwork of vendor plugins and custom controllers DRA gives it a real foundation. ML model distribution was genuinely embarrassing OCI volumes fix the architecture.

None of these are new ideas. All of them are ideas that finally graduated from “you can kind of do this if you squint” to “this is stable and production-ready.”

The Ingress NGINX retirement and the gitRepo removal are the most underrated signals in the release. They tell you the project is serious about what Kubernetes should and shouldn’t be and serious about not carrying CVE-adjacent code forever because migration is inconvenient.

DRA’s trajectory over the next two or three releases will define how Kubernetes handles the AI infrastructure wave. The bones being laid in 1.36 matter more than they look.

Haru means spring. New season, cleared skies, distant horizon. The release name was earned.

Which of these 10 changes hits closest to home for your cluster? Drop it in the comments.

DEV Community:

I replaced GitHub Copilot with 3 tools. My team noticed within a week.

Cursor, Claude Code, and Windsurf didn’t just change how I code they changed what my PRs look like.

The Copilot breakup nobody talks about

Why Copilot stopped being enough

Cursor the IDE that started arguing back

What I actually use it for:

Claude Code the terminal agent I didn’t know I needed

Windsurf the dark horse nobody warned me about

How Cursor, Claude Code, and Windsurf work together

Final thoughts: you don’t need 40 tools

Helpful resources and links

Cursor, Claude Code, Windsurf?! My AI coding stack after 40 dev experiments

For devs drowning in AI tool hype who just want to know what actually stuck

40 tools, 3 survivors

Table of contents

Why your AI tool stack actually matters now

Cursor the IDE that started arguing back

Claude Code the terminal agent I didn’t know I needed

Windsurf the dark horse nobody warned me about

How Cursor, Claude Code, and Windsurf work together

Final thoughts: you don’t need 40 tools

Helpful resources and links

You’re the reason your React app is slow

You didn’t hit a framework limit. You wrote the bottleneck yourself and it’s been quietly billing you in FPS ever since.

The re-render killers

Mistake 1: Inline functions in JSX

Mistake 2: Using useCallback as a good luck charm

Mistake 3: Using array index as key in lists

State architecture sins

Mistake 4: Putting state too high up the tree

Mistake 5: Context without memoization

Mistake 6: useEffect doing too much

Bundle crimes

Mistake 7: Not using React.lazy and Suspense

Mistake 8: Importing entire libraries when you need one function

The tools you’ve been ignoring

Mistake 9: Rendering a thousand items when you could render twelve

Mistake 10: Never opening the React DevTools Profiler

You’re not off the hook just because React 19 exists

Helpful resources

The only AI agents article you’ll ever need

From ReAct loops to production multi-agent systems everything in one place, nothing left out.

What an agent actually is

The core building blocks

Four design patterns that actually work

Reflection

Tool use

Planning

Multi-agent collaboration

Shipping to production

Evaluate before you optimize

Latency and cost are the same problem

Log everything, assume nothing

The job changed. Most people haven’t caught up yet.

Helpful resources

Go vs Rust: the only backend language debate that actually matters in 2026

You don’t need to pick one. You need to know which fight each one was built for.

Go: the default loadout every backend team starts with

Rust: the late-game unlock you didn’t know you needed

Real architecture patterns: how teams actually use both

Pattern 1: Go orchestrates, Rust crunches

Pattern 2: Cost optimization at scale

Pattern 3: Latency-sensitive systems where GC pauses are a product problem

The wall: when Go stops being enough

How to know if you’ve actually hit it

The team cost is real too

The quick reference

The actual decision

Conclusion: Go builds companies, Rust survives them

Helpful resources

Junior devs catch exceptions. Senior devs prevent them. Here’s the difference.

Four patterns that replace most of the try-catch in your codebase and the mental model that makes the right choice obvious.

Why junior devs default to try-catch

Pattern 1: validate first, catch never

Pattern 2: build a custom exception hierarchy

Pattern 3: handle everything in one place

Pattern 4: result objects for expected failures

The mental model that ties it all together

Conclusion

Mistake 2: Using `useCallback` as a good luck charm

Mistake 3: Using array index as `key` in lists

Mistake 6: `useEffect` doing too much

Mistake 7: Not using `React.lazy` and `Suspense`