DEV Community

Cover image for AWS IMDS hop limit: the one-line change I wouldn't sign off until I understood it
Sudhakar
Sudhakar

Posted on

AWS IMDS hop limit: the one-line change I wouldn't sign off until I understood it

#aws #imds #kubernetes #ec2

A cryptic one-line change request — "bump the IMDS hop limit from 1 to 2 for CastAI" — and the trail of questions it took to approve it with a straight face.

The change request, in full:

Increase IMDS hop limit 1 → 2 on the EKS nodes. Required for CastAI configuration. Requires node patching. Possible downtime. Approver: you.

That's it. No rationale, no risk assessment, just an integer that needs to go from 1 to 2 on production nodes, a third-party name you don't remember installing, and the word downtime sitting there uncommented. You could approve it in two clicks. But your name is on it, and you have a rule: don't sign off on a production change you can't explain to whoever you'd have to wake up if it went sideways.

(If you've not had the tidy version, you may have met the same problem from the other end: a pod that won't start, logging could not get EC2 instance identity metadata: context deadline exceeded, or half your workloads stuck Pending after an EKS upgrade. Same root cause, less polite packaging.)

So instead of approving it, you start pulling the thread. This is where it goes.

In a hurry? The answer, before the story.

  • The symptoms — a context deadline exceeded timeout, pods stuck Pending, or a CastAI change request — all point at one setting: the IMDS hop limit, which needs to go from 1 → 2 on the EKS nodes.
  • IMDS lives locally on each EC2 instance (169.254.169.254) — not in Kubernetes, not the public AWS API. That locality is the whole reason hops matter.
  • The hop limit is the TTL on IMDS's response; it caps how many network layers a request may cross (range 1–64).
  • A pod sits in its own Linux network namespace, so pod → IMDS costs 2 hops. At limit 1 the request dies at the host; limit 2 lets it through.
  • CastAI's spot-handler (a per-node DaemonSet) reads IMDS to catch spot-interruption notices — the concrete thing the limit is blocking.
  • Limit 1 blocks every pod equally; normal app pods just don't notice, because they use IRSA / EKS Pod Identity instead of IMDS.
  • Applying it means replacing the nodes (the setting lives in the launch template) via a rolling drain — hence the downtime caveat.

Everything below is the why — the trail one engineer followed to be able to defend that change at 3 a.m.

First, what is IMDS — and why does anything need it?

Every line of that request leans on one acronym — IMDS — so that's where the thread starts.

IMDS is the EC2 Instance Metadata Service — a small, read-only service any EC2 instance can query to learn facts about itself: its instance ID, type, Availability Zone, networking, and most importantly the temporary IAM credentials for whatever role is attached to it.

Why does that need to exist? Because the alternative is what gets people breached: long-lived AWS access keys baked into code or dropped on disk, rotated by hand until one leaks. IMDS removes that pattern. Code on the instance asks "what are my credentials right now?" and gets back short-lived, auto-rotating ones — no secrets in the repo, no manual rotation. That single idea, self-identity and keyless credentials on demand, is what the whole story is built around. The hop limit, containers, and the security argument later are all plumbing around it.

Where IMDS actually lives

Before going further, it's worth killing the most common wrong mental picture. IMDS is not a pod in your cluster, and it is not the public AWS API you reach over the internet.

It's a link-local endpoint — 169.254.169.254 (IPv6: fd00:ec2::254) — served locally, on each individual EC2 instance, by the underlying EC2/Nitro platform. "Link-local" is the operative word: the request never actually leaves the machine to reach a remote server. The instance turns to itself and asks "who am I, and what may I do?", and the platform answers from right there on the box.

That locality is the load-bearing fact. Because each instance serves its own IMDS, a request only succeeds if it arrives from that instance's own network — the caller effectively has to be standing on the box.

❓ Which raises the obvious question of distance: how far from the box may a caller sit before a request is refused?

So what is a "hop limit"?

The hop limit borrows straight from how IP networking already works. A hop is one pass through a router or network layer; every IP packet carries a TTL ("time to live") counter, each hop knocks it down by one, and when it hits zero the packet is dropped. It's the mechanism that stops lost packets from circling forever.

IMDS reuses exactly that. The "hop limit" is the TTL stamped on the metadata service's responses, and it decides one thing: how many network layers a metadata request may cross before the packet dies.

  • Default: 1 on a plain EC2 instance — deliberately, so the response can't leave the box it came from.
  • Maximum: 64.
  • Where it's set: a field called HttpPutResponseHopLimit on the instance's metadata options (and on the launch template that stamps out new instances).

So a limit of 1 means the answer isn't allowed to travel even one layer away from the instance. That sounds harmless — until you remember everything you run is in a container.

💡 Clarification — the thing that's easy to get wrong here
"IMDSv2 sets the hop limit to 1" is not right. The hop limit is an independent setting; IMDSv2 is a separate hardening, based on session tokens (more on that when we get to the security trade-off). They're usually configured together, which is why people treat them as one thing. And "the default is 1" has big exceptions: EKS managed node groups have defaulted the hop limit to 2 since 2020, and Amazon Linux 2023 AMIs default to 2 on purpose, for containers. So the real question isn't "why is 1 the default" — it's why are our nodes at 1? Someone hardened them, or they came up a non-standard path. That's the first hint the change has a backstory.

🤔 If containers are everywhere and a limit of 1 forbids leaving the box, how has anything been working?

Why a container is different

A process running directly on the host reaches the metadata service in a single hop: process → 169.254.169.254. But almost nothing runs directly on the host. It runs in a pod, and a pod doesn't share the host's network — it lives in its own isolated network sandbox, with its own virtual interface wired to the host through a virtual cable (a veth pair into a bridge). Crossing that cable is itself a hop.

Two hops (pod → host → IMDS)

Two hops, every time. So with the limit at 1, the packet's TTL hits zero the moment it reaches the host — one hop short. It doesn't fail loudly; it just times out. Set the limit to 2 and both jumps are allowed. That's the entire mechanism: a container sits one network layer too far from the metadata service, and the hop limit is the ruler measuring the gap.

Deep dive — the "extra layer" is a real kernel object
It's a Linux network namespace with a veth pair bridged to the host. That boundary is the first hop, full stop. Hold onto the phrase Linux network namespace — a second, unrelated thing also called a "namespace" shows up later, and the two get confused constantly.

🧵 That clears up the mechanism — but it leaves a loose end: if AWS runs this metadata service, why is **your* team the one deciding its hop limit?*

Who actually maintains it — AWS or you

Cleanly split:

Layer Owner What they handle
The service itself AWS (internal) Runs and maintains IMDS and the 169.254.169.254 endpoint. Nothing to patch, host, or babysit.
The configuration You (the engineer) Whether IMDSv2 is required, the hop limit, whether the endpoint is on at all, and any IAM/network restrictions.

Mechanically: attach an IAM role to an instance, and AWS mints temporary, auto-rotating credentials for it; the AWS SDKs inside your app fetch those from 169.254.169.254. No static keys, no manual rotation.

So there's nothing to keep alive at 3 a.m. — but the dials that decide how your instances talk to IMDS are yours, and the hop limit is one of them. Which means the request isn't asking AWS for anything — it's asking you to loosen how tightly your own nodes are locked down.

🧐 That makes the one clue you skipped worth chasing: **required for CastAI configuration* — and you've never installed anything called CastAI.*

The name in the request: what CastAI is, and why it forces the change

CastAI is a Kubernetes automation platform for cloud cost optimization. It watches your cluster continuously and, on its own, right-sizes workloads, bin-packs pods onto fewer nodes, and swaps pricey on-demand instances for cheaper (often Spot) ones — trimming the EC2 bill without anyone hand-tuning it.

And here's the part where it all clicks. To do that, one of CastAI's components runs as a pod on every node and has to read that node's own IMDS. That single dependency is the whole origin of the change — because reading IMDS from inside a pod is exactly what a hop limit of 1 blocks. Laid end to end, the request explains itself:

Causal chain (ends on the green

That's the why. But approving a change isn't about confirming the why — it's about confirming it's safe. Two questions now stand between you and the signature: what exactly does that per-node pod read, and is opening this door dangerous? Everything below is those two questions.

🤔 Starting with a reflexive worry: if CastAI has a pod on every node, is your traffic flowing through it?

It's a watcher, not a checkpoint

Good instinct, wrong picture — worth ruling out before it poisons everything downstream. A tempting model is that CastAI sits in the request path and everything routes through it. It doesn't. CastAI is not a reverse proxy, sidecar, or firewall in your traffic path. A user hits your app pod, the traffic goes straight to your app; CastAI never sees it.

What it actually is: a background observer and optimizer that pulls from two sources and correlates them.

  • The Kubernetes API server — how many pods are running, what CPU/memory they requested, whether any are stuck Pending because nodes are full.
  • The cloud provider — instance types, pricing, capacity, and (per node) spot-interruption signals.

From those it decides things like "this workload fits on a cheaper instance type" or "drain this spot node before AWS reclaims it." That work is spread across several pods — and only one of them is why the hop limit matters:

🔍 Under the hood — which CastAI component actually touches node IMDS
CastAI isn't one pod; it installs several into the castai-agent namespace, and only one is what the hop limit gates:

  • castai-agent — the read-only core (a Deployment); talks to the K8s API and streams telemetry to CastAI.
  • castai-cluster-controller — carries out scaling/node actions.
  • castai-evictor, castai-pod-mutator / pod-pinner — bin-packing and right-sizing.
  • castai-kvisor — the security agent (KSPM / runtime security; can run an eBPF DaemonSet). It's a DaemonSet too, so people point to it for the IMDS dependency — but it isn't the one.
  • castai-spot-handler — a DaemonSet on every node that reads IMDS (/latest/meta-data/spot/instance-action) to catch spot-interruption notices. This is the pod whose per-node IMDS access the hop limit gates. It only watches and reports; it doesn't drain nodes itself.

So CastAI is many pods, and the spot-handler is the one that needs IMDS.

❓ Which raises a smaller question you can't unsee now: people keep saying **agent* — is that a pod, or something lower-level running straight on the machine?*

"Agent" is just a role — the real actor is the DaemonSet

"Agent" is a role, not a privileged bare-metal binary. In Kubernetes it's still a container in a pod. What matters is which kind of workload wraps it, and two native Kubernetes shapes tell the story:

  • Deployment — the centralized-coordinator shape (the core castai-agent): a replica or two, somewhere in the cluster.
  • DaemonSet — Kubernetes guarantees exactly one copy on every worker node. Grow from 5 nodes to 50 and it drops a copy onto each newcomer automatically. It's the standard shape for per-node infrastructure: log shippers, security scanners, node-termination handlers — and CastAI's spot-handler.

That's why the problem is per-node: there's a spot-handler pod on each node, each one reaching for that node's IMDS from inside its own pod network. Fixing one node fixes one node — this is a fleet-wide job.

Deep dive — you've seen this shape in AWS's own tooling
aws-node-termination-handler is the identical pattern: a small DaemonSet pod on each host polling IMDS paths like /spot and /events, cordoning and draining the node when a notice arrives. CastAI's spot-handler is a specialized cousin.

🧐 Something still doesn't add up, though: CastAI already has a direct line to the Kubernetes API, which knows everything about pods and nodes — so why does it need to crawl down to each node's AWS metadata at all?

What Kubernetes can't see

Your instinct is half right, and the missing half is the point. Kubernetes knows the logical truth — this pod asked for 2 CPUs and 4 GB, that node is 80% full. It's blind to money and physical hardware.

  • Kubernetes says: "this pod needs 2 CPU / 4 GB."
  • The cloud says: "you're on an m5.large, on-demand, in us-east-1, at $X/hour — and there's a cheaper Spot option that fits."

A cost optimizer has to marry those two datasets, so it needs cloud-side facts Kubernetes can't supply. But the tidy version of this oversells IMDS's role:

💡 Nuance — it's not pricing that needs your node's IMDS
"CastAI reads instance types and prices from IMDS on every node" is wrong in the load-bearing detail. Instance catalogs and pricing come mostly from cloud provider APIs and CastAI's own pricing database (via the cluster-controller's IAM permissions) — not per-node IMDS. The thing that genuinely must be read from a node's own IMDS is live, node-local state, and the headline example is the spot-interruption notice the spot-handler watches for — real-time, specific to that box, available nowhere but its own 169.254.169.254. So the honest sentence is: the hop-limit change unblocks node-local IMDS reads (spot interruption chief among them), not CastAI's whole pricing engine.

So the spot-handler needs its node's IMDS.

🤔❓ Which finally detonates the question that should have been nagging since the container section: every normal app pod lives behind the exact same container network, so they're all two hops from IMDS too — why has a hop limit of 1 been "working" for them, and only breaks for CastAI?

The turn: a hop limit of 1 was never working for anyone

This is the part most explanations get subtly wrong. The comfortable assumption — that hop limit 1 is fine for normal pods — is false. It was never fine. A hop limit of 1 blocks every pod equally, CastAI's and yours. There's no per-app exemption.

The reason your app pods never complained is that they don't call IMDS at all.

  • Well-configured EKS apps get AWS credentials through IRSA (IAM Roles for Service Accounts) or the newer EKS Pod Identity. AWS injects a short-lived token straight into the container; the SDK finds it and talks to regional AWS endpoints — never touching 169.254.169.254.
  • Because they never knock on IMDS, they never hit the wall. They've been unaware it exists.

And the wall exists on purpose — it's a security control. If an attacker compromises a public-facing pod, a hop limit of 1 stops them from reaching 169.254.169.254 to steal the node's IAM credentials. It keeps a single compromised pod from becoming a compromised node.

💡 Clarification — "normal pods don't use IMDS" is only half true
Plenty of system pods do, and they break at hop limit 1 just like CastAI:

  • The EBS CSI driver reads instance identity from IMDS; at limit 1 in some setups, volumes fail to attach and pods hang in Pending (often the real cause of that post-upgrade outage).
  • Spot / node-termination handlers (AWS's and CastAI's) poll IMDS for interruption notices.
  • Some overlay CNIs need even more — Cilium can require a hop limit of 3.

This is exactly why EKS managed node groups default to 2. The honest framing isn't "CastAI is uniquely needy" — it's that CastAI's spot-handler joins the club of node agents that need IMDS, and your nodes happen to be locked at 1.

Compromised pod → BLOCKED (red) vs ALLOWED (green)

🧵 Two new threads pull at you: is IRSA-versus-IMDS a switch someone flips, and if a hacker can still use IRSA, why is that any less of a catastrophe?

Two keys: one opens the machine, one opens a drawer

Is IRSA a toggle? No. There's no "use IRSA instead of IMDS" checkbox. It's an architecture baked into the AWS SDK's credential provider chain — the SDK checks credential sources in a fixed order, and IRSA's injected token sits near the top. You wire it up by:

  1. Creating a Kubernetes ServiceAccount annotated with an IAM Role ARN.
  2. Attaching that ServiceAccount to the pod.

At launch, Kubernetes mounts a signed token into the container and points an env var at it. The SDK uses that first and never falls through to IMDS. (EKS Pod Identity reaches the same end by a newer route.)

So why isn't a stolen IRSA token as bad as a stolen IMDS one? Blast radius:

Node IMDS credentials IRSA / Pod Identity credentials
Whose identity The entire EC2 node's IAM role One pod's narrowly-scoped role
Typical power Broad — attach ENIs, pull images, edit routes, talk to the control plane Only what that app needs (e.g. one S3 bucket)
If stolen Attacker can manipulate the wider AWS account Contained to that app's sandbox

Steal an IMDS token and you have the keys to the machine; steal an IRSA token and you have the keys to one drawer. That asymmetry is why a hop limit of 1 is a sane default — and why raising it earns the scrutiny you're giving it.

🤔 Which lands on the question really holding up your signature: you're about to open the IMDS door for CastAI's pod — so what if CastAI's pod is the one that gets breached?

What if the breach comes through CastAI

The honest answer first: it's possible. No software is exploit-proof, and there's no 100% guarantee. The reason security teams still approve this isn't that a breach is impossible — it's that the blast radius is boxed in on three sides:

  1. IMDSv2 is session-based (this is the "v2" mentioned earlier). There's no static string to grab and run. An attacker must first PUT for a short-lived token, then pass it in a header on the GET. The token expires quickly and won't work outside that instance's network boundary — copying it to a laptop gets nothing.
  2. CastAI's IAM role is scoped tight — infrastructure/metadata actions (describe instances, node scaling), not your customer databases, app secrets, or private buckets. Even wearing CastAI's identity, an attacker inherits only CastAI's narrow powers.
  3. Namespace isolation + Network Policies — CastAI lives alone in the castai-agent namespace. With standard Kubernetes NetworkPolicies enforced, cross-namespace traffic can be blocked, so an intruder in the CastAI pod can't freely pivot into your app pods.

Stated plainly: raising the hop limit to 2 does widen network visibility for node agents. It's a calculated risk — defended by IMDSv2 session tokens, least-privilege IAM, and namespace/network isolation — taken in exchange for continuous, automated cost savings. That's a sentence you can defend at 3 a.m.

🧠 You've now said "namespace" several times, and a splinter of doubt lodges: isn't "namespace" also a Linux kernel thing from containers? Are these the same word doing two jobs?

Two things named "namespace" — and they're not the same

They share a word, they're deeply connected, and they operate in completely different universes.

Linux namespace — the real isolation technology, at the kernel level. It makes a process believe it's alone on the machine. Containers are Linux namespaces wrapped around a process:

  • Network (net) — its own interfaces, loopback, IP. This is the extra hop from earlier.
  • PID — the container thinks its main process is PID 1, though the host sees PID 48291.
  • Mount (mnt) — sees only its own filesystem.

No Linux namespaces, no containers.

Kubernetes namespace — an organizational label, at the orchestration level. A K8s namespace (default, castai-agent, …) is just a tag in the Kubernetes database (etcd). The Linux kernel has never heard of it. It exists to group resources, scope RBAC, and set quotas — a locked department suite inside an office building.

Where they meet is your exact problem:

The two

Kubernetes supplies the organizational wrapper (the K8s namespace); the Linux network namespace is what physically creates the hop that forces hopLimit = 2. Same word, two jobs, only one of them relevant to your change.

Deep dive — the escape hatch you're choosing not to use
A pod can set hostNetwork: true, dropping it into the host's network namespace — no pod net namespace, so it reaches IMDS in a single hop and ignores the limit. Some infra agents do this. CastAI keeps its components in the normal pod network (so it can map workloads cleanly), which is why the fix is a node-level hopLimit=2 rather than a hostNetwork special case.

The mechanism is fully understood now. What's left is operational — and one piece of it decides whether Sunday's window really needs the word downtime.

🤔 How hard is this pod hitting IMDS, anyway — is it a cron job?

Not a cron job — how often it really talks

A CronJob wakes on a timer, runs once, exits. CastAI can't work that way — cost, traffic, and scheduling pressure change by the second — so it runs as long-lived daemons, talking to different systems at different rhythms:

Signal Cadence Mechanism
Cluster state (pods/nodes) Real-time K8s API watchers — events pushed the instant a pod goes unschedulable
Node identity (ID / type / region) Once at boot, then cached A single IMDSv2 GET — the hardware doesn't change while the node lives
Spot-interruption notice Continuous light loop IMDS /spot/instance-action on the node
Cluster metrics Short interval (impl. detail) Metrics scrape

The spot loop is continuous because AWS gives little warning — up to ~2 minutes on AWS (≈30 seconds on GCP/Azure) — before reclaiming a Spot node; CastAI also runs an ML prediction model for a longer heads-up. (You'll see "every 15 seconds" quoted for metrics; treat it as approximate.)

The reassuring part: the node-identity read this change is really about happens once, at pod startup. This isn't a pod hammering the metadata service in a hot loop — it's a pod that needs one clean answer when it boots, and today, on your nodes, that answer never arrives.

❓ Last question, and it's the one that governs the maintenance window: to change the limit on a running cluster, do you have to replace the nodes — and is that where the downtime comes from?

Applying it: why the nodes have to be replaced

Yes. To land this on the existing cluster, the nodes have to be replaced (or at least relaunched).

The hop limit is baked into the EC2 launch template — the blueprint the node group / Auto Scaling Group uses to build nodes. Edit the blueprint from 1 to 2 and the already-running instances don't change; the new value only applies to newly launched nodes.

The exception worth knowing: on a single running instance you can flip it live, no reboot: aws ec2 modify-instance-metadata-options --http-put-response-hop-limit 2. But for a fleet you fix the launch template and roll the nodes, so the change is consistent and survives future scale-ups.

The choreography is a rolling replacement, not a big-bang reboot:

  1. Cordon — mark an old node SchedulingDisabled so no new pods land there.
  2. Drain — evict its pods; Kubernetes reschedules them onto other nodes.
  3. Terminate & replace — once it's empty, kill it; the ASG launches a fresh node from the updated blueprint, now at hop limit 2.

And that explains the word that stopped you on Friday. Why "possible downtime"?

  • Single-replica workloads — a one-pod app blinks out while it's killed on the old node and recreated on the new one. Mitigation: ≥2 replicas plus a PodDisruptionBudget.
  • Capacity churn — moving many pods at once can cause brief blips while replacement nodes pass health checks.

A well-architected service with multiple replicas and sane PDBs rides the roll with zero user-visible impact. The "possible downtime" line is there for the single-replica and tight-capacity cases — which you can now go check by name instead of guessing.

At which point you know what the change does, why it exists, which pod needs it, why it was blocked, what it risks, and how to do it without taking the site down. So you approve it — not because you were told to, but because you can finally explain it.

The whole picture, in one diagram

The whole picture (K8s API + CastAI cloud + node with agent/spot-handler → IMDS, 1 blocked / 2 allowed)

What common explanations get wrong

  • "The hop limit default is always 1." → Plain EC2, yes; but EKS managed node groups and Amazon Linux 2023 default to 2. Nodes at 1 are usually hardened/custom (or a spot/Karpenter path).
  • "IMDSv2 sets the hop limit." → The hop limit is an independent option; IMDSv2 is the session-token requirement. Configured together, but distinct.
  • "The kvisor DaemonSet needs IMDS." → The per-node IMDS consumer is the castai-spot-handler DaemonSet (monitoring-only). kvisor is the security agent.
  • "CastAI reads instance type & pricing from node IMDS." → Type/pricing come mostly from cloud APIs + CastAI's pricing DB. The essential node-local IMDS read is the spot-interruption notice.
  • "Normal pods don't hit the hop wall." → Only if they use IRSA. System pods (EBS CSI, termination handlers, some CNIs) do, and fail at limit 1 — the reason EKS defaults to 2.

Read further

Top comments (0)