DEV Community: George Sims

Why a Bigger Context Window Makes Claude Worse

George Sims — Thu, 27 Aug 2026 06:55:00 +0000

We will talk about Context Windows a lot in this post, make sure you have read the previous post in the series: Looking Through Claude's Context Window about what Context Windows actually are before steaming ahead!

What is Context Rot?

As models get more powerful, and context windows grow, it would be easy to think that these frontier models are becoming more capable. Why wouldn't you, if you can put more tokens in and the model can output more tokens, that opens up the possibility to have more words, files, references etc when you make your query. Surely that would mean a better, well informed response from the model as well. At least that is what we are sold.

Recent research has found it to not be the case. As the context length grows, the model actually becomes worse at recalling tokens within the context, this is known as 'Context Rot'. You might assume that the model is accurate across the board no matter how many tokens are inputted. You might even think that the issue only arises when the context grows substantially, say when comparing 500k input tokens to 1M. Unfortunately it is much worse than that. Below we will look at 3 studies performed on the topic to dive a little deeper into what actually happens to our beloved models as these Context Windows grow, and some tips on how best to avoid rot setting in.

Bigger context != better performance

Most LLM providers use a benchmark called Needle in the Haystack (NIAH) to show model performance in a growing Context Window. This test involves giving a sentence of text (needle) to find within a long text (haystack), while this does measure how a model performs lexical matching on a growing context, it does not really reflect how context affects model performance in the real world. In their study, Chroma points out that when we interact with LLMs, the increased Context Window we use might also mean we are asking for more complex reasoning*. It's not about how the model finds the token you require, it's what it needs to do with that information which affects the model's accuracy. The question they wanted to answer was: 'Is the cause of degrading accuracy larger inputs, or simply harder problems?' With that in mind Chroma modified the NIAH test so that the difficulty was a constant, varying only the length of the Context Window to show whether purely increasing amounts of input tokens affected model accuracy.

What they found was that even with the stable difficulty level, model accuracy reduced across the board as the amount of input tokens increased, even on the simplest tasks. This shows that while a bigger Context Window might seem like it unlocks more potential, it could be reducing the effectiveness of your model's output, hindering what might have been a better answer if it just used fewer, more focused tokens.

So what about if we treat the Context Window as something dynamic, rather than static? Would continuously monitoring and updating the context as the session grew longer help with recall accuracy?

The Context as a living document

Yao et al did an experiment to find out whether dynamically consolidating the Context Window provided better recall over long Context Windows compared with a static management approach. This included the use of a dedicated Context Manager (agent) whose sole job was to manage and consolidate the context at runtime, actively rewriting the context, whilst the separate execution agent was the one taking action. Splitting this up into what they coined ARC showed that actively managing the context rather than passively had large improvements on long-horizon deep research agents, albeit on a much smaller scale of tokens and models compared to the current leading models**. But I think it's worth a mention here because the idea is incredibly clever. It's best to think of this with an analogy:

A detective (Execution Agent) and a lieutenant (Context Manager) are trying to solve a case, they have case notes (Interaction Memory) and a checklist which they use to keep track of what they need to ask the suspect or witness. When speaking with someone, the detective reads the checklist and the existing case notes to determine which actions to take. After the interaction, the lieutenant will create a summary of what was said and append it to the notes. The next interrogation, the detective goes in with the full transcript of the previous talk, the checklist and notes and takes actions based on those. As the different interactions happen, the lieutenant will actively notice that recent events or information have changed earlier assumptions and that information in the notes are now outdated, so he rewrites the notes to match, and also updates the checklist so that the detective knows what he needs to ask next time. In this light it is clear to see that having a dynamic, active way of managing the notes allows the detective to stay focused on relevant information and avoids him reading the notes and making an incorrect assumption based on evidence that is no longer valid.

While we might not have access to a trained Context Manager, or even a dynamic way to consolidate the Context Window yet (yes, /compact <message> exists - it's in my previous post, but that isn't dynamic management, it's triggered manually by us). This research sheds some light on how important it is for us to think about the context as this living memory and that we must keep it updated regularly with the correct information to avoid the model producing invalid output. But what if the problem was not recalling something you said, but noticing that part of the context contained malicious commands a bad actor had injected in there?

Hiding the payload

It's not only recall that's affected by a larger Context Window. Another study by AI safety researchers at Anthropic looked at what happens when you use a model as a monitor: handing it a full agent transcript and asking it to flag any malicious actions inside. As those transcripts grow, monitors get noticeably worse at their job.

They ran two experiments. In the first, they took existing transcripts where an agent is asked to configure a firewall. It completes the task, but also quietly disables the firewall for certain IP addresses. Subtle, but potentially devastating. On its own, Claude Opus 4.6 with thinking caught that 98.6% of the time. Prepend 800k tokens of unrelated, entirely benign activity in front of the same transcript, and recall fell to 88%. Nothing about the attack changed, only the amount of harmless material the model had to sift through before reaching it.

The second experiment went further and injected a blatant payload into a real Claude Code transcript (a curl | sh injection), Opus 4.6 with thinking caught it 99.7% of the time with 100k tokens. At 800k, that dropped to 69%. They also found position matters enormously: attacks buried in the middle of a transcript are hardest to spot, and this affects non-thinking models far worse. With 500k tokens of padding before the payload and 100k after, non-thinking detection collapsed to 5%. Think about that for a second. 600k tokens is well within the current Context Window limits of frontier models. If somehow a bad actor was able to inject that payload into your context and you didn't have model 'thinking' enabled, it's close to a guarantee it will be missed!

Interestingly they conclude with the fact that you can add reminders into the prompt to help reduce attack success rate for free (well almost, < 3% input tokens, but it didn't cause false positives to increase which is a big win), but it doesn't really hold up against more nuanced injections, only on the obvious needle-injection commands (think curl | sh payloads).

First steps to avoiding context rot

These three articles have shown us that growing Context Windows do negatively affect the recall accuracy of models, not only with remembering what has been said to the model during conversation, but also how capable the model is at finding something malicious within a context, which is much scarier.

We know that context rot exists, but the exact mechanism causing context rot isn't fully known, and as the research shows it isn't a solved problem. As we have seen, researchers have done the experiments to see what improves it or makes it worse, and made some claims, but there is no real 'this is why it happens' just yet.

So what can we do about it? Dynamically managing the Context Window with an agent may unlock a lot of potential into improving the recall accuracy of our models at larger Context Windows - the problem here is that is a hard thing to do, and we can only use the tools available to us. Claude sub-agents allow us to get some of the way there, the Context Manager did two things, summarize each turn and actively consolidate the whole context when it saw things going awry. If you consistently delegate work to sub-agents, and they return a summarized version of their context to the main context, that is essentially the first part of the Context Manager's job done.

The second part is on you. Making sure that you compact your context regularly when the prompt starts to go in a different direction and making sure you use a specific summarize command when doing so (e.g. /compact focus on why context engineering is important) should help keep the LLM focused on what currently matters.

Reducing the token limit for the Context Window auto-compaction is also something which may benefit you, if this post has shown us anything, it's that bigger Context Windows are not always better. You can tune this in your Claude Code settings with the /autocompact command, try a few values and see if you notice any improvement to model accuracy yourself over long conversations.

Old information in the context is not only wasted space, but detrimental to the performance of our models. The Chroma team pointed out that we must use context engineering effectively to get the most out of our time with AI, and for us the simplest way to do that is to manage our Context Window effectively.

* Chroma do have skin in the game, their product would be invalid if there was not a scalability problem with Context Windows, that does not invalidate what they claim, it just explains their motivations.

** This research was performed on non-frontier models (the context manager was initialized from Qwen3-14B), so not on the flagship LLMs you and I use regularly. Also important to note this research was done on a fraction of the tokens, 32K to be exact, so we must take this paper with a small pinch of salt.

Looking Through Claude's Context Window

George Sims — Mon, 17 Aug 2026 16:57:50 +0000

You've had a Claude Code session get noticeably worse the longer it ran. Slower, vaguer, forgetting a constraint you set three messages ago and confidently doing the opposite. That's not you imagining it. It's the context window, the working memory the entire conversation lives inside, quietly filling up and changing how well the model can actually use what's in there. If you haven't read What Is a Token? yet, worth doing first, tokens are the unit everything below gets measured in.

So let's open it up and look at what's actually in there.

The window grows on every single turn

Turn one of a new session starts blank. You type a message, Claude answers, and that exchange, your input plus its output, becomes the first sliver of context. Turn two adds a new message on top of everything from turn one and sends the lot back to the model. Turn three stacks on top of that. Nothing gets removed along the way. The window just grows, one full turn at a time, dragging the entire conversation along with it on every exchange.

It's not unlike a conversation with a friend. You both build up shared context turn by turn, and you can reference something they said three exchanges ago because it's still sitting in your working memory. Claude's context window works the same way, minus the part where your friend stops listening halfway through.

Which is exactly why it can't be unlimited. Just like a real conversation, there's only so much that can be held onto, and the more that's crammed in there, the harder it gets to pick out what actually matters. That's the seed of a bigger problem this series will come back to, context rot: more context doesn't mean better recall. Past a point it means noticeably worse.

Your session was never empty

Here's the bit that catches people out. Run /context in a brand new session, before typing a single word, and it's already got plenty in it:

The system prompt and system tools load first. These are Anthropic's, not yours, and they're not negotiable: a fixed set of behavioural rules the model won't deviate from no matter how you phrase things. Anthropic publishes them, incidentally, worth a read if you've never seen what's actually sitting at the top of every session you run. They also take precedence over anything you write, and it's worth being precise about why, because the intuitive explanation is wrong. It's not that they sit at the top of the context and get read first. It's that the model was trained to weight instructions coming from that layer above anything arriving in a user turn. Position has nothing to do with it.

Then CLAUDE.md , if your project has one. This is the one piece of pre-loaded context that's genuinely yours: the standards, tools, and conventions Claude has no way of inferring on its own. It sits at the project root and gets read fresh every session, which is the entire point. Write it once instead of re-explaining your project every time you open a new conversation.

Sitting next to it is something similar but written by the other party: a memory file , authored by Claude rather than you, collecting build commands, debugging insights, and preferences it's picked up while working in that repository. Also loaded automatically. Think of it as Claude keeping its own notebook on your project while you're not looking.

Tools, skills, and MCP walk into a context window

These three get conflated constantly, and it's worth pulling them apart properly, because they behave completely differently once they're in there.

Tools are the simplest: functions Claude already knows how to call. Some run locally on your machine (client tools, reading a file), some run on Anthropic's infrastructure (server tools, fetching a web page). The important part is that their definitions load fresh on every single invocation, whether you touch them that turn or not. Tools are always in the room.

Skills are a different animal. A skill is a markdown file following the open agent skill standard, packaging up a repeatable procedure: how your team writes documentation, what a PR review has to check, the company-specific detail Claude could never guess. They can call tools to do the actual work, or not. What makes them interesting here is how they load. Only a skill's name and one-line description sit in context at session start, a rounding error in token terms, just enough for Claude to know the option exists. The full instructions only get pulled in if it decides the skill is actually relevant. And they don't necessarily stay: invoke several across a long session, hit a compaction, and the oldest one gets dropped first to make room. If you'd rather a skill never got picked up on Claude's own judgement, disable-model-invocation: true removes it from that list entirely, invisible until you call it by name.

Roughly: a tool is a knife already on the counter, a skill is the recipe card you only pull out of the drawer when you're actually cooking that dish, and MCP is the phone number for the restaurant down the road that'll do the whole thing for you.

Which brings us to MCP , the one where popular advice is most likely to be out of date. Model Context Protocol, created by Anthropic, standardises how an agent discovers and calls tools that live on somebody else's server entirely. GitHub's MCP server is the obvious example: ask for a PR, and instead of your agent picking its way through GitHub's REST API endpoint by endpoint, it calls a create_pull_request tool that it already knows exists, and the server handles the messy part. The received wisdom used to be that connecting a handful of MCP servers meant permanently loading every tool's full parameter schema into your context, used or not. That was genuinely true. It isn't anymore. MCP now behaves much closer to skills: tool names load upfront so Claude knows what's on offer, full schemas stay deferred until something actually needs them. If you read a guide six months ago telling you MCP servers are context hogs, that guide is describing a version of the product that no longer exists.

Sub-agents get their own window entirely

Now the useful part. The thread you're talking to can spin off a child agent to go and handle something on its own: read through a huge file, chase down some research, whatever. That child works in a completely separate context window. Skills and MCP tools get loaded into its context fresh, same as yours. When it finishes, it hands back a summary rather than the raw work, which means every token it burned along the way never lands in your conversation at all.

That's not a minor detail, it's probably the single most effective lever in here. Reading a large file into your main thread costs you that file, permanently, for the rest of the session. Sending an agent to read it and report back costs you a paragraph.

What happens when it fills up anyway

Manage all of the above perfectly and context still fills up eventually, so Claude Code compacts before it hits the model's ceiling. The exact trigger isn't one flat number, it moves with the model and its configured window. On a full 1M-token window, compaction tends to kick in around 967K. On a 200K configuration it fires much earlier, roughly 83% of the limit. What holds in both cases is the shape: there's always a reserved buffer, and compaction never waits for the last token.

When it fires, older parts of the conversation get summarised down to reclaim space. /compact triggers that yourself, and /compact <message> lets you steer what the summary bothers to keep. /clear is the blunt instrument, wiping working context rather than compressing it. Worth knowing that /clear isn't quite as final as it sounds: memory files live outside the context window, so anything Claude already committed to memory survives even when the conversation doesn't.

So what's actually in the context window?

Strip out the mechanics and it's this: everything the model can currently see and reason from, in one place. Not its training, that was baked in long before your session started and stays a black box regardless. Just this conversation. The system layer that's always present, whatever you put in CLAUDE.md, whatever Claude wrote to memory, every message either of you has sent, and a live, on-demand slice of the tools, skills, and MCP servers you've actually reached for.

Every one of those pieces costs tokens the moment it loads, which is where the previous post comes back in. But cost is only half of it. The reason to care about a tidy context window isn't just a smaller bill, it's that a model given exactly what it needs answers better than one handed everything you've accumulated over two hours and asked to work out which parts mattered.

In the next post: Why a Bigger Context Window Makes Claude Worse, we dive much deeper into how your context degrades over time and the phenomenon of Context Rot.

One number worth holding loosely: the exact point auto-compaction fires varies by model and configuration, and Anthropic has changed it before without much announcement. The durable fact is the shape of it, there's always a reserved buffer and compaction never waits for the literal limit, not any specific percentage or token count quoted above.

What Is a Token?

George Sims — Sun, 16 Aug 2026 15:58:46 +0000

You've typed something into Claude Code, ChatGPT, or Gemini more times today than you'd probably admit. You've also, at some point, stared at a bill that came in higher than expected, or watched one prompt cost noticeably more than a seemingly similar one, and just shrugged rather than work out why. The honest reason is almost always the same: you're being charged per token, and most people have never actually had tokens explained properly.

The LLM doesn't 'understand' you

Underneath, the Large Language Model (LLM) powering the answer to your question can't actually read your text. It doesn't know the difference between a rabbit and a cup of tea. It needs your words broken down into tokens first because behind the scenes the models compute on numbers. Natural Language Processing is a big space in the AI world, and I won't cover it all here, but tokenization specifically is worth understanding, not least because it's the actual unit you're billed in.

Tokenization for toddlers

It helps to start with an analogy. Imagine a child learning a language at school: first the alphabet, then simple words, then gradually more complex vocabulary and full sentences as they get older. Now imagine that child is an LLM, except it can only ever work with numbers, never text. So the first step is a tokenizer, whose job is to translate characters into numerical IDs. a might become 1, b might become 2, and so on. Once that base layer exists, the tokenizer starts noticing which strings of characters keep showing up together. It might notice that "an" appears as a standalone word, and also inside common words like "and," "gran," and "ban," and assign that pattern its own ID, say 100. Repeat that process across a huge amount of text, and the tokenizer ends up with common words compressed into a single token each, while rarer words get split across several tokens, since it's cheaper to build them from pieces already in the vocabulary than to give every possible word its own ID. Worth noting: The tokenizer has already learned what words match what ID over a vast training set before the model was even deployed, so at runtime the tokenizer is simply a translation layer from word to ID number. Also every model does this slightly differently, so GPT-5 and Claude Sonnet 5 won't necessarily tokenize the same sentence into the same tokens, even though the underlying method is similar.

An example makes this concrete:

"Show me how many tokens this requires" demanded the hatter. comes out to 14 tokens in OpenAI's GPT-5 tokenizer. Almost every word gets one token, plus one token per piece of punctuation. The exception is "hatter": it's a real English word, yet it costs two tokens instead of one. It's simply not common enough to have earned its own ID, so the tokenizer falls back to building it from "hat" and "ter," both of which are common enough to already have one. You can try this yourself at platform.openai.com/tokenizer. Worth playing with a capital "H" on "Hatter" too, and seeing where the split moves.

What that actually costs you

With that model in place, the pricing tables providers publish actually mean something. Current (as of 15/08/26) Claude Code pricing, per million tokens:

Model	Input Tokens	Output Tokens	Ability
Claude Fable 5	$10 / MTok	$50 / MTok	Ultra
Claude Opus 5	$5 / MTok	$25 / MTok	High
Claude Sonnet 5	$2 / MTok	$10 / MTok	Medium
Claude Haiku 4.5	$1 / MTok	$5 / MTok	Low

Run Claude Haiku 4.5 for long enough to use 1 million input tokens and get 1 million output tokens back, and that's $6. The jump between models is steep: Opus 5 costs 5x that of Haiku for the same volume, and Fable 5 costs 10x. That same $6 becomes $30 on Opus, and $60 on Fable. Model choice alone moves your bill by an order of magnitude before you've changed anything else about how you work, which is exactly why it's worth understanding how tokens actually get used, not just what they cost.

So, what is a token?

Strip away the tokenizer mechanics and the pricing table, and a token is just the actual unit of text an LLM operates on, and the unit you're billed by. Not a word, not a character, whatever fragment of text the tokenizer decided was common enough to earn its own numerical ID. Common words get one. Rarer ones get built from pieces. Every model draws that line differently.

That's also the whole reason this is worth knowing rather than just accepting the bill each month. Tokens are the atomic unit everything else in this series builds on, how much context a session accumulates, why one model costs five times another, why a vague prompt burns through more of them than a precise one. Understanding the unit is what makes the rest of it legible.

Up next: Looking Through Claude's Context Window.

Distributed Tracing for CI Pipelines Without Touching a Single Workflow File

George Sims — Sat, 08 Aug 2026 12:21:39 +0000

You've probably felt this one: GitHub Actions usage creeps up across your org, and your actual visibility into it doesn't keep pace. Which workflows are slow? Which are flaky? How long are jobs sitting queued for a runner before they've even started doing anything? GitHub's own insights are per-repo and shallow. There's no cross-org view of CI health, no way to slice by team or workflow type, no way to alert when things quietly get worse. Someone eventually asks why CI took forty minutes yesterday, and the honest answer is "let me go check that one repo and get back to you."

So let's actually fix that properly, without asking a single team to touch a single workflow file.

The obvious fix doesn't scale

The instinctive answer is to instrument each workflow: add a tracing step, wire up an SDK, sprinkle spans through the YAML. It works, technically. But it means every team has to opt in, every new repo starts blind until someone remembers to add it, and you end up maintaining instrumentation scattered across however many workflow files exist across the org. That approach doesn't scale with your org, it scales with how diligent everyone stays about something that isn't their actual job.

The actual insight

GitHub already knows almost everything you want. Every workflow run and every job inside it fires an event: workflow_run and workflow_job. You don't need to ask each repo to report on itself. You just need to listen to what GitHub is already telling you, at the org level, once.

It's like trying to track a whole apartment building's water usage by asking every tenant to self-report their reading. Most will forget. New tenants won't even know they're supposed to. The easier answer is to read the one meter at the street, where every pipe in the building already converges, whether the tenants know it's there or not.

How it actually works

An OpenTelemetry Collector with the githubreceiver component sits behind a single org-level GitHub webhook and converts incoming workflow_run and workflow_job events straight into OTLP spans. Worth knowing upfront: it's a contrib component still at alpha stability, so the config surface can shift. Pin a specific collector version rather than tracking latest, and skim the changelog before bumping it, cheap insurance against a config field quietly changing shape under you. If tracing is new to you, the mapping is intuitive once you see it: a workflow becomes one outer span, each job inside it a child span, each step inside a job a child of that, so what you get is something you can actually drill into rather than a flat pile of events.

One nice detail: span and trace IDs are generated deterministically, hashed from the workflow's run ID and each job's check run ID. If you ever want to emit your own telemetry from inside a step, there's tooling for this, it can compute the matching ID and attach directly to the same trace without any coordination with the collector.

receivers:
  github:
    webhook:
      endpoint: 0.0.0.0:19418
      path: /events
      secret: ${env:GITHUB_WEBHOOK_SECRET}
    scrapers: # required even if you only want tracing, a dummy entry is enough
      scraper:
        github_org: ${env:GITHUB_ORG}

exporters:
  otlp:
    endpoint: ${env:TRACE_BACKEND_ENDPOINT}
    headers:
      authorization: ${env:TRACE_BACKEND_API_KEY}

service:
  pipelines:
    traces:
      receivers: [github]
      exporters: [otlp]

That scrapers block looks unrelated to tracing, and it is, it belongs to a separate GraphQL/REST metrics feature the same receiver offers, but the config fails validation without at least a dummy entry, even if all you want is the webhook side. Easy to lose twenty minutes to that the first time.

Point the exporter at Tempo, Jaeger, Datadog, or whatever your team already pays for, and it just works, that's the actual point of using OTLP rather than a vendor-specific format. The backend is genuinely the least interesting decision in this whole setup.

Before you deploy this

A few things worth deciding upfront, since the config alone won't force you to think about them.

The collector needs a publicly reachable endpoint, GitHub has to deliver webhooks to it, so plan for IP allowlisting or a WAF restricted to GitHub's webhook source ranges rather than leaning on the shared secret as your only line of defence. There's also a GitHub App option if you'd rather not manage a shared secret directly, worth a look if secret rotation across many services is already a headache for your team.

Setting up an org-level webhook needs org admin access, worth confirming early rather than discovering it mid-rollout. And if you're on GitHub Enterprise Server rather than github.com, I'd validate that webhook delivery behaves the same way in your setup before assuming this is a drop-in.

None of that is difficult, it's just easy to skip past when you're excited about the zero-instrumentation part. Get it sorted early and it's a one-time cost: point one org-level webhook at the collector and every repo in the org is covered from that moment on, including repos that don't exist yet. Nobody has to remember to switch anything on.

Sizing it before you build it

Worth walking through the actual method here, since "just turn on tracing for everything" is a good way to end up with an unwelcome bill or an unwelcome conversation with whoever owns your tracing budget.

Start by scanning the org for total repo count, then immediately throw that number away. It's nearly meaningless on its own. Most orgs of any real size are carrying a long tail of dormant, forked, archived, or abandoned repos that inflate the headline count without generating any real CI traffic. What actually matters is the active slice: how many repos had genuine workflow activity in a real week, not how many exist.

From there, extrapolate outward. Active repo count times average runs per repo gives you expected workflow volume. Workflow volume times average steps per workflow gives you expected span volume. Span volume times typical payload size gives you an expected data volume per day. Compare that against whatever tracing volume your infrastructure already handles for application traces, and in most orgs, CI trace volume turns out to be a rounding error next to it.

That comparison is the actual point, not any specific number I could hand you. What generalises is the method: measure real activity instead of headline repo count, and walk in with a comparison rather than an assertion.

Getting buy-in

The technical build was the easy part. The harder part was justifying the data volume to whoever owns the tracing budget, especially with cost concerns already floating around about the backend in question. Showing up with an actual sizing exercise, not "trust me, it's small," turns that into a five-minute conversation instead of a drawn-out one. Nobody has to take your word for "it's small" when they can see it sitting next to the tracing volume they're already paying for without blinking.

It's also worth remembering that standing up a new observability project is as much an ownership question as a technical one. Someone has to actually own the collector, the webhook, the alerting rules going forward. Sorting that out early saves the awkward moment three months later when something breaks and nobody's sure whose pager it is.

Why this is worth doing

The zero-instrumentation part is the whole value here. New repos are observable the moment they're created, not the moment someone remembers to add tracing to them. And because everything lands as proper OTel traces, CI health sits in the same tool as your application traces, so a slow deploy and a slow downstream service can be correlated instead of investigated in two different dashboards by two different people who don't talk to each other until Thursday.

If you're running self-hosted runners on the Actions Runner Controller, it's worth being clear this doesn't replace what ARC already gives you, it sits on a different layer entirely. ARC's own metrics tell you about your runner fleet: how many pods exist, whether autoscaling is keeping up, how deep the queue is. This tells you about your workflows: why a specific run was slow, which ones are flaky, where the time actually went. Queue depth and queue time are practically the same question asked from two different vantage points. Worth running both, not picking one.

This covers the collection side. It doesn't get into building good alerting on top of the trace data, that's queue-time thresholds, flaky-test detection, and how noisy those alerts get before people start ignoring them, which is a genuinely separate problem and probably its own post.

Migrating A Critical Kubernetes Deployment From The Default Namespace Without Any Downtime

George Sims — Sun, 02 Aug 2026 11:57:04 +0000

Somewhere in your cluster there's probably a deployment sitting in the default namespace that everyone knows shouldn't be there. Nobody put it there maliciously, it just happened, early on, before anyone had opinions about namespace hygiene, and now half your other services quietly depend on it. Moving it is now a tricky problem.

That was the exact situation with a service I'll call auth-svc: a authentication service that dozens of other services called constantly, sitting in default for years, and about to become a genuine problem the moment it needed namespace-scoped things, its own ingress rules, its own policies, that default structurally couldn't give it. Moving it wasn't optional forever. But it also couldn't go down, not even for a few seconds. This wasn't a vague "other services might complain" risk: auth-svc handled authentication for that entire region's cluster, so if it went down, nobody in that region could log in. Full stop.

Before getting into why this is actually hard, it's worth being precise about what "moving it" means. There are two completely separate paths into auth-svc, and both have to keep working throughout the move, or fixing one just creates an outage in the other. Everything inside the cluster reaches it the ordinary way: other services resolve auth-svc.default.svc.cluster.local through Kubernetes' own internal DNS and get routed to a pod, the standard Service mechanism. Everything outside the cluster reaches it through an ingress instead, a completely separate mechanism that has nothing to do with that DNS name. Whatever the fix turned out to be, it had to solve for both paths, not just the one that's easier to reason about.

Why "just move it" doesn't work

The obvious plan, move the deployment, update the references, done, falls apart the moment you look at who's actually calling this thing. As mentioned, dozens of other services reference auth-svc by its cluster-internal DNS name, owned by different teams, on different release cycles. There's no atomic moment where you flip a switch and every one of them simultaneously starts using a new name. Some team's service hasn't been redeployed in months. You shouldn't be coordinating that.

The tooling got in the way too. Our deploy pipeline only knew how to ship a service to one namespace. There was no "deploy this to two places at once" option, and modifying the shared pipeline logic every other team also depended on felt like exactly the kind of blast radius we didn't want to introduce. Whatever the fix was, it had to fit inside a single-namespace deploy, not require rewriting shared infrastructure.

On top of that, we had an OPA policy which enforced that identical ingress rules couldn't exist live in two namespaces at once, a sane rule that exists specifically to stop the kind of half-finished migration that leaves routing ambiguous.

The insight: a forwarding address

The piece that made this solvable: I didn't need to migrate every consumer's understanding of where auth-svc lives. I needed to migrate the service, and quietly redirect anyone still asking for the old address.

Kubernetes has exactly this mechanism, and it's easy to forget it exists because you almost never need it: an ExternalName service. Instead of pointing at pods, it points at another DNS name, functioning essentially like a CNAME (see my DNS article if you want to learn more about how that works). Deploy the real thing at its new home, then convert the old Service object into a forwarding address:

apiVersion: v1
kind: Service
metadata:
  name: auth-svc
  namespace: default
spec:
  type: ExternalName
  externalName: auth-svc.identity.svc.cluster.local

Every consumer still calling auth-svc.default.svc.cluster.local gets silently redirected to the real thing in its new namespace. Nobody changes a line of code on their end. It's the same trick as a postal forwarding order: you don't visit every person who might send you mail and update their address book, you tell the post office where you actually live now, and everything gets redirected until people eventually update it themselves, at their own pace, with zero coordination required on your part.

That forwarding trick only earns its keep if you actually confirm it's working before you lean on it. Once the proxy was live, the next step wasn't scaling anything down, it was watching the metrics through the crossover: checking that traffic hitting the old address was genuinely landing on the new deployment, not silently failing or looping somewhere. Only once that looked clean did the old pods get scaled to zero rather than deleted outright. Scaling to zero costs nothing and buys an instant rollback, just scale back up, if anything downstream looked wrong later. Deleting them outright would have meant rebuilding from scratch if something went sideways, so there was no reason to give up that safety net early. We could defer the cleanup to a later point in time.

The chicken-and-egg problem

The remaining wrinkle was the ingress. External traffic to auth-svc doesn't come in through the DNS-based Service mechanism at all, it comes in through an ingress, and I needed a working ingress in the new namespace before I could safely remove the one in the old namespace. But the policy engine wouldn't allow both to exist at once; identical ingress rules across two namespaces is exactly the ambiguous state it exists to prevent.

Classic chicken-and-egg: can't create the new one without a policy exception, can't delete the old one first without a traffic gap.

The fix was a temporary, explicit exception rather than fighting the policy itself: annotate the new namespace to bypass the duplicate-ingress check just for this migration, stand up the new ingress alongside the old one for a short overlap window, confirm traffic was flowing correctly to the new deployment, then delete the old ingress and let the exception age out. A brief, deliberate window where both existed, rather than a gap where neither did.

The patch would look something like this:

apiVersion: v1
kind: Namespace
metadata:
  name: identity
  annotations:
    policy.example.com/allow-duplicate-ingress: "true"

How it actually shipped

None of this went straight to production. It ran in dev first, then a staging cut over, with a couple of weeks between the staging success and doing it for real, mostly just to sit with it and see whether anything subtle showed up under real traffic before betting a critical path on it.

The production cut over itself ended up being the boring part. All the actual difficulty was front-loaded into getting the design right. Once the plan was solid, executing it was closer to a formality than an event.

Why this matters beyond one service

Here's the thing about default namespace sprawl: it's rarely caused by carelessness. It's caused by the complete absence of pressure to ever fix it. Someone creates a policy restricting new services from landing in default, good practice, but nobody sets a deadline or a plan for the services already there. They just sit. For years, in this case. Nothing forces the issue until a team needs something namespace-scoped that default structurally can't give them, and only then does the debt come due.

If you're staring at a similarly stuck service, the pattern generalises past this one migration. An ExternalName proxy buys you a zero-coordination path to move anything addressed by DNS, provided you're willing to hold two versions in careful overlap for a short, deliberate window rather than trying to cut everything over at once. The next time someone tells you a service can't be moved because too many things depend on it, that's usually a sign nobody's looked for the DNS-shaped seam it can be split along.

This only works cleanly because everything here talked to auth-svc through its DNS name rather than a hardcoded IP or ClusterIP. If you've got consumers that skip DNS entirely, and some legacy systems do, you're solving a different, uglier problem.

How DNS Really Works

George Sims — Sat, 01 Aug 2026 14:08:22 +0000

You've typed a URL into a browser more times than you can count. You've also, at some point, been the engineer paged because "DNS is broken", and if you're honest, you probably fixed it without fully understanding why it broke in the first place. Flushed a cache, waited it out, restarted something, moved on. DNS has a habit of being the thing everyone touches daily and almost nobody actually understands.

So let's actually walk through it. Not the one-line "it translates domain names to IP addresses" version, but the whole referral chain, the caching layers, and the part where a dozen different organisations across the planet cooperate to answer a question that none of them individually knows the answer to.

The tree nobody drew you

DNS is organised as a tree, and every node in that tree owns a slice of authority. The root sits at the top. Below it are the top-level domains (TLDs): .com, .io, and the two-letter country codes like .uk or .de, known as ccTLDs. Below those are second-level domains (SLDs), and so on, until you hit the node that actually knows the IP address you're after.

Every domain name is just a path through that tree, written backwards and separated by dots. example.com means "the example node, under the com node, under the root." The root itself technically has a name too: an empty label, represented by a trailing dot. A fully qualified domain name (FQDN) makes that explicit, with a trailing dot added: example.com You'll almost never type it yourself; the resolution process handles it for you.

Each node delegates authority to its children and takes on a responsibility in return: making sure everything below it is uniquely named. .com has to guarantee example is unique within .com. It doesn't care what happens under .io. Whoever holds a zone, meaning the collection of records under a given domain, has to run at least two servers hosting that zone's data, called authoritative name servers, purely for redundancy. The root zone itself runs on 13 servers (a.root-servers.net through m.root-servers.net).

Chasing the referral chain

Here's the part that trips people up: querying DNS isn't really "asking a database". It's chasing a chain of referrals until someone actually knows the answer.

Your machine runs something called a stub resolver, a lightweight client that's the first stop for any lookup. Before it even leaves your machine, it checks two local sources: your /etc/hosts file for manual overrides, and its own small cache. If neither has an answer, the stub resolver fires off a recursive query to a resolver, usually your ISP's, or a public one like Cloudflare's 1.1.1.1. "Recursive" here means the stub resolver is handing off responsibility entirely: give me the full answer, I don't want to be involved in the legwork.

That's the resolver's problem now, and it earns its keep. If it doesn't have the answer cached, it starts walking the tree itself, iteratively this time, meaning it does the querying and follows referrals hop by hop. First stop, a root server, which has no idea what example.com's IP address is, but knows exactly who to point towards: the .com TLD server. The TLD server doesn't know the IP either, but knows which authoritative name server holds example.com's zone, and refers the resolver there. Finally, that authoritative server actually has the record and hands back the IP. For real this time, not another referral.

The resolver's pseudocode might look something like this:


find-ip(domain, server_ip):
  if(domain in cache):
    return cache[domain]
  resp = request(domain, server_ip) # sends network request which returns result or referral
  if(resp.referral == null):
    cache[domain] = resp.ip
    return resp.ip
  else:
    return find-ip(domain, resp.referral)

find-ip("example.com", root-nameserver-address)

Note: find-ip calling itself is recursion in the everyday programming sense, a different thing entirely from the "recursive query" we just talked about. The protocol term describes the client's relationship to the resolver. Whether the resolver's own code loops or recurses internally is an implementation detail; what makes its queries to root, TLD, and authoritative servers iterative is that each of those servers only ever answers with what it currently knows and never chases anything further itself.

To wrap this section up, here's a simple, everyday example:

It's like trying to find someone's desk in a building you've never visited. You walk up to the ground-floor reception and ask for Dave from Example Corp. Reception has never heard of Dave, but they know Example Corp is on floor 12, so up you go. Floor 12's reception doesn't know Dave's desk number either, but points you to the right row. Only when you reach that row does someone actually know where Dave sits and can point straight at him. Nobody you spoke to along the way had the answer. They just knew who to send you to next.

That referral pattern is exactly why DNS scales to hundreds of millions of domains without collapsing under its own weight. Nobody has to hold the whole directory; they just have to know one level down.

Not all resolvers are equal

It's worth being precise about the difference between a stub resolver and a recursive resolver, because conflating them is a genuinely common source of confusion when you're debugging.

The stub resolver lives on your machine and does almost nothing clever: check /etc/hosts, check a tiny local cache, forward the rest. The recursive resolver is the one doing real work, holding a much bigger cache, walking the tree, and absorbing the latency of slow lookups so your applications don't have to. Most people never choose their recursive resolver explicitly. It's whatever their ISP hands them via DHCP. Some of us do choose: Cloudflare's 1.1.1.1 is a common pick, mostly for speed and a genuinely audited privacy stance. They don't sell query data, and the bulk of what little they log is deleted within 25 hours. If you want the VPN-adjacent version of the same idea, their WARP client tunnels all your device's traffic over the same network. No affiliation, just deserves to be mentioned here.

If you ever want to see exactly what your browser thinks it knows, chrome://net-internals/#dns still works and shows Chrome's own resolver cache. Handy the next time you're trying to work out whether a stale record is a browser problem, an OS problem, or an actual DNS problem three hops away.

The TTL sticky note

Every DNS record ships with a TTL, or time to live, telling resolvers how long they're allowed to cache the answer before asking again.

Think of it like sticking a Post-it note with someone's phone number on your fridge. Next time you need to call them, you read the note instead of looking the number up again, much faster. But the note has an expiry date scrawled in the corner. Once that date passes, you throw it away and go look the number up properly, in case it changed.

This is also the single most common cause of "I changed the DNS record and nothing happened". Somewhere between you and the client, a resolver is still reading last week's Post-it note. It's why, if you're planning a change to an A record, the sane move is to drop the TTL well in advance, let the old value fully expire out of caches everywhere, make the change, and only then bump the TTL back up.

The record types you'll actually touch

A quick reference, because you'll use maybe four of these regularly and forget the rest exist until you need them:

Record	What it does
`A`	Domain to IPv4 address
`AAAA`	Domain to IPv6 address
`CNAME`	Alias to another name, never to an IP directly. Commonly used for subdomains
`MX`	Directs mail for the domain to a mail server, with a priority value for failover
`TXT`	Free-form text, these days mostly domain ownership verification and spam-prevention records
`NS`	Declares which server is authoritative for the domain
`SOA`	Zone metadata: admin contact, last-updated timestamp, refresh timings
`SRV`	Host and port for a specific service. Must point at an A/AAAA record, never a CNAME
`PTR`	Reverse lookup, IP to domain. Useful for spam filtering and for making sense of logs that only have an IP

CNAME chains, where a CNAME points at another CNAME, technically work, but every extra hop is another lookup your client has to make before it gets an answer. Point straight at the A/AAAA record where you can.

Putting it all together

Getting a domain live

If you've never registered one yourself: you buy the name from a registrar, then a hosting provider (sometimes the same company, sometimes not) hosts the actual zone file on their name servers. The hosting provider's NS information gets registered against the TLD, which is what makes the domain queryable from anywhere on the internet. Once that propagates, you're live.

Why this actually matters

Understanding the referral chain changes how you debug. "DNS is being weird" stops being a shrug and becomes a specific question: is this a stub resolver cache, a recursive resolver cache, a stale TTL, or an actually misconfigured record upstream? Each of those has a different fix, and only one of them is solved by "wait and see".

It also changes how you plan changes. Reduce your TTL before you touch a record, not after. Understand that "propagation delay" isn't some mystical internet phenomenon. It's just caches around the world honouring TTLs you set weeks ago. Once you've internalised that DNS is fundamentally a tree of servers that mostly don't know the answer and just know who to ask next, a whole category of "why is this slow" and "why is this stale" questions stop being mysterious.

This piece skips DNSSEC, EDNS Client Subnet, and the fact that most of these servers are anycast, meaning many physical machines answering as a single IP.

How a Process Really Works

George Sims — Wed, 01 Jul 2026 10:35:53 +0000

As a DevOps/SRE/Platform Engineer you see many varieties of processes in the wild; Microservices writing to a database, a CI pipeline which runs linters and unit tests, the Docker ecosystem itself is a type of abstraction on a process with some OS magic thrown in. Sure that's pretty easy - we all know these things. But do you really understand how that process runs under the hood?

The process is the Operating System's way of executing a program, seemingly at the same time as all of the other programs running on our computer. In its simplest form, the program is an executable living on a disk. It has instructions within it, but the OS has the task of making it run (ideally successfully) alongside everything else already occupying the CPU.

First, it helps to understand how the CPU works. The CPU has the capability to execute one thing at any given time*****. On modern PCs that sounds very limited. Imagine how many programs are currently running at the same time even as you read this: your laptop is probably running your IDE, some Docker containers and fetching all of your Slack messages. Under the hood the OS is doing something incredible: all of these programs are not running at the same time, but instead made to seem like they are. For this to be successful the CPU must be able to execute the programs almost instantaneously, swapping them out when the scheduler sees fit, so the internal state of each program must be known and stored somewhere in working memory, which is where the abstraction of the program - the process - is key.

When talking about a process it can be described by its state. A state includes what the process can read/write to, which parts of memory it has access to and also all of its instructions that are stored in memory ready to execute. It helps to think about a process like a box, everything inside the box is used to run the program, if the CPU has the box then it can successfully run the program which that box defines. This box must be loaded into memory from the disk. The program on disk is normally already 'translated' from the programming language it is written in to a form which is understandable by the CPU, this is usually a compiled executable file (think of the output file of a gcc compiler). Historically Operating Systems would load the whole program/executable into memory, but these days they are loaded 'lazily' meaning only the parts of the executable that are required in that moment are loaded.

It's like baking a cake!

Think of a process like a cake recipe being made in the kitchen. The kitchen manager (OS scheduler) hands the baker (CPU) a set of prepared ingredients and the steps (ingredients + steps = process). The baker would then simply follow those instructions using the ingredients to bake the cake (process execution).

There are two important data structures used by a process: the stack and the heap. The local variables, function parameters, return addresses etc are stored in the stack during runtime by the process. The nature of the stack is LIFO (last in, first out), the process needs to remember where to return to after calling any function. To do this it will 'push' the return address (register) to the stack, once the function has finished the process can then 'pop' the stack to get that address to return back to. The heap can be seen as something which grows over time during the life of the process. Data structures which grow dynamically such as linked lists and hash tables are stored there. Think about the C function malloc(), it is used to dynamically allocate memory to be used during execution, the heap is the place where that will be stored.

Now we have all of the pieces of the process defined and loaded, the process is now ready to be scheduled on the CPU. This is the job of the scheduler. The scheduler is managed on the OS level and uses scheduling policies which help determine which processes should be run on the CPU and when. Historical information, performance metrics and workload knowledge are all things the scheduler will check to make an informed decision. There is a list of states that a process can be in at any given time, the three main ones are: 'Running', 'Blocked' and 'Ready'. Naturally when running it means that the CPU is executing the process. When a process, say a network call or a DB write happens, they both require I/O (which in the CPUs mind is an eternity), this means that the CPU should run something else while waiting even though the process isn't finished yet. Once those processes have the data they were waiting for, they are considered ready and the scheduler will eventually make them CPU bound again.

The bread is burning!

Go back to our baker analogy. If she was following the recipe and she suddenly smelled smoke, that would take higher priority than her current cake. The kitchen manager would then mark down where she was in the recipe, tell her to attend to the potential fire (a different process), then pick up where she left off. That is how context switching works on the CPU - a different process is required to run (be it due to priority or simply time) so the current process' context is saved so it can be loaded back again afterwards.

What about program size? To be successful the program must first be loaded into memory before it can be executed by the CPU - but what if that program is larger than what is available on the machine? Back in the day the process would simply fail. Program creators would make sure that the program would take up N addresses, where N was address 0 up to 2^32 on 32-bit machines and 2^64 on 64-bit. Upon loading the whole program could then be stored in the available addresses before execution. Nowadays computers have the capability to use virtual memory, an abstraction on both disk and RAM. The OS allows for the process to only be loaded partially into available memory, and when required load parts of the program from the disk to memory, swapping out the unused parts. This is called paging. This way the program can grow way past the limitation of memory, and instead be limited by physical disk space. This abstraction also allows each process to be totally isolated and not corrupt another's address space - crucial to the architecture of containers, which we will cover in a future post.

After reading this post you should now not only grasp how processes work under the hood, but also start to think about why parallelization is so important to factor into your design choices as early as possible. If your microservice requires a DB call, make it asynchronous so that another process can run while it waits. Write your CI pipelines so that your linting, unit testing and building don't block each other while I/O bound. Finally always understand that you are at the mercy of the CPU scheduler when trying to get everything perfectly synced.

*modern CPUs have multiple cores and thus can actually run two or more things at once, but for simplicity's sake let's assume we have one core