DEV Community: Sudhakar

How Meilisearch works

Sudhakar — Tue, 21 Jul 2026 04:22:52 +0000

Here is the complete end-to-end journey of searching for "Evengers" (notice the typo!) in a movie database.

This flow ties together everything we've discussed—Rust application logic, LMDB offsets, Linux mmap, Page Faults, and CPU memory reads.

The End-to-End Execution Flow

1. API Ingestion & Typo Tolerance: User Space (Rust App). The HTTP request GET /indexes/movies/search?q=Evengers hits the Meilisearch Rust web server.

Tokenization: Meilisearch cleans and extracts the token "evengers".
FST Lookup: Meilisearch queries a memory structure called a Finite State Transducer (FST) to handle typos. It calculates a Levenshtein distance of 1 and maps the typo "evengers" $\rightarrow$ real indexed term "avengers".

2. Calculating the Virtual Address: Meilisearch + LMDB Logic. To find which movies contain "avengers", Meilisearch needs to read the Inverted Index entry for that word.

LMDB knows "avengers" sits at Offset `0x8400` inside the database file. Meilisearch performs pointer math:

$$\text{Virtual Address} = \text{Base Address } (0\text{x}10000000) + \text{Offset } (0\text{x}8400) = 0\text{x}10008400$$

Meilisearch asks the CPU: "Read memory at 0x10008400."

3. Kernel Memory Check & Page Fault: Linux Kernel + Hardware. The CPU intercepting 0x10008400 checks the Page Table in physical RAM:

Cold Search (Present Bit = 0): The CPU raises a Page Fault. The Linux OS pauses the thread, fetches the 4 KB page containing the "avengers" index node off the SSD, loads it into Physical RAM, and flips Present Bit = 1.
Warm Search (Present Bit = 1): The data is already in Physical RAM. The CPU reads it instantly in nanoseconds.

4. Bitset Intersection (Roaring Bitmaps): CPU Execution. At address 0x10008400, Meilisearch reads a Roaring Bitmap—a compressed set of document IDs containing the word "avengers" (e.g., [Doc ID 12, Doc ID 45, Doc ID 890]).

If you also searched for another word (e.g., "marvel"), Meilisearch would fetch both bitmaps and execute a CPU bitwise AND instruction to instantly find only the movies containing both terms.

5. Ranking & Document Retrieval: LMDB + Memory Read. Meilisearch runs its ranking rules (Typo score, Word matching, Proximity) on the matching IDs. Doc ID 45 (The Avengers, 2012) wins top spot.

Now it needs the actual movie details (Title, Plot, Poster URL):

Calculates: Virtual Address = Base Address + Offset_for_Doc_45
Reads the raw JSON bytes directly from memory (via RAM or a final Page Fault from SSD).

6. HTTP Response: Network Output. Meilisearch serializes the JSON payload into an HTTP 200 OK response and hands it off to the network socket back to the client—all completed in under 15 milliseconds.

The Big Picture Takeaway

Notice how Meilisearch never wrote a single line of code to handle reading from the SSD, managing buffer caches, or allocating disk blocks.

It simply performed pointer math (Base + Offset), asked for a Virtual Address, and let the Linux Kernel handle fetching bytes off the SSD whenever a Page Fault occurred.

AWS IMDS hop limit: the one-line change I wouldn't sign off until I understood it

Sudhakar — Sat, 18 Jul 2026 08:39:55 +0000

A cryptic one-line change request — "bump the IMDS hop limit from 1 to 2 for CastAI" — and the trail of questions it took to approve it with a straight face.

The change request, in full:

Increase IMDS hop limit 1 → 2 on the EKS nodes. Required for CastAI configuration. Requires node patching. Possible downtime. Approver: you.

That's it. No rationale, no risk assessment, just an integer that needs to go from 1 to 2 on production nodes, a third-party name you don't remember installing, and the word downtime sitting there uncommented. You could approve it in two clicks. But your name is on it, and you have a rule: don't sign off on a production change you can't explain to whoever you'd have to wake up if it went sideways.

(If you've not had the tidy version, you may have met the same problem from the other end: a pod that won't start, logging could not get EC2 instance identity metadata: context deadline exceeded, or half your workloads stuck Pending after an EKS upgrade. Same root cause, less polite packaging.)

So instead of approving it, you start pulling the thread. This is where it goes.

In a hurry? The answer, before the story.

The symptoms — a context deadline exceeded timeout, pods stuck Pending, or a CastAI change request — all point at one setting: the IMDS hop limit, which needs to go from 1 → 2 on the EKS nodes.

IMDS lives locally on each EC2 instance (169.254.169.254) — not in Kubernetes, not the public AWS API. That locality is the whole reason hops matter.

The hop limit is the TTL on IMDS's response; it caps how many network layers a request may cross (range 1–64).

A pod sits in its own Linux network namespace, so pod → IMDS costs 2 hops. At limit 1 the request dies at the host; limit 2 lets it through.

CastAI's spot-handler (a per-node DaemonSet) reads IMDS to catch spot-interruption notices — the concrete thing the limit is blocking.

Limit 1 blocks every pod equally; normal app pods just don't notice, because they use IRSA / EKS Pod Identity instead of IMDS.

Applying it means replacing the nodes (the setting lives in the launch template) via a rolling drain — hence the downtime caveat.

Everything below is the why — the trail one engineer followed to be able to defend that change at 3 a.m.

First, what is IMDS — and why does anything need it?

Every line of that request leans on one acronym — IMDS — so that's where the thread starts.

IMDS is the EC2 Instance Metadata Service — a small, read-only service any EC2 instance can query to learn facts about itself: its instance ID, type, Availability Zone, networking, and most importantly the temporary IAM credentials for whatever role is attached to it.

Why does that need to exist? Because the alternative is what gets people breached: long-lived AWS access keys baked into code or dropped on disk, rotated by hand until one leaks. IMDS removes that pattern. Code on the instance asks "what are my credentials right now?" and gets back short-lived, auto-rotating ones — no secrets in the repo, no manual rotation. That single idea, self-identity and keyless credentials on demand, is what the whole story is built around. The hop limit, containers, and the security argument later are all plumbing around it.

Where IMDS actually lives

Before going further, it's worth killing the most common wrong mental picture. IMDS is not a pod in your cluster, and it is not the public AWS API you reach over the internet.

It's a link-local endpoint — 169.254.169.254 (IPv6: fd00:ec2::254) — served locally, on each individual EC2 instance, by the underlying EC2/Nitro platform. "Link-local" is the operative word: the request never actually leaves the machine to reach a remote server. The instance turns to itself and asks "who am I, and what may I do?", and the platform answers from right there on the box.

That locality is the load-bearing fact. Because each instance serves its own IMDS, a request only succeeds if it arrives from that instance's own network — the caller effectively has to be standing on the box.

❓ Which raises the obvious question of distance: how far from the box may a caller sit before a request is refused?

So what is a "hop limit"?

The hop limit borrows straight from how IP networking already works. A hop is one pass through a router or network layer; every IP packet carries a TTL ("time to live") counter, each hop knocks it down by one, and when it hits zero the packet is dropped. It's the mechanism that stops lost packets from circling forever.

IMDS reuses exactly that. The "hop limit" is the TTL stamped on the metadata service's responses, and it decides one thing: how many network layers a metadata request may cross before the packet dies.

Default: 1 on a plain EC2 instance — deliberately, so the response can't leave the box it came from.
Maximum: 64.
Where it's set: a field called HttpPutResponseHopLimit on the instance's metadata options (and on the launch template that stamps out new instances).

So a limit of 1 means the answer isn't allowed to travel even one layer away from the instance. That sounds harmless — until you remember everything you run is in a container.

💡 Clarification — the thing that's easy to get wrong here
"IMDSv2 sets the hop limit to 1" is not right. The hop limit is an independent setting; IMDSv2 is a separate hardening, based on session tokens (more on that when we get to the security trade-off). They're usually configured together, which is why people treat them as one thing. And "the default is 1" has big exceptions: EKS managed node groups have defaulted the hop limit to 2 since 2020, and Amazon Linux 2023 AMIs default to 2 on purpose, for containers. So the real question isn't "why is 1 the default" — it's why are our nodes at 1? Someone hardened them, or they came up a non-standard path. That's the first hint the change has a backstory.

🤔 If containers are everywhere and a limit of 1 forbids leaving the box, how has anything been working?

Why a container is different

A process running directly on the host reaches the metadata service in a single hop: process → 169.254.169.254. But almost nothing runs directly on the host. It runs in a pod, and a pod doesn't share the host's network — it lives in its own isolated network sandbox, with its own virtual interface wired to the host through a virtual cable (a veth pair into a bridge). Crossing that cable is itself a hop.

Two hops, every time. So with the limit at 1, the packet's TTL hits zero the moment it reaches the host — one hop short. It doesn't fail loudly; it just times out. Set the limit to 2 and both jumps are allowed. That's the entire mechanism: a container sits one network layer too far from the metadata service, and the hop limit is the ruler measuring the gap.

Deep dive — the "extra layer" is a real kernel object
It's a Linux network namespace with a veth pair bridged to the host. That boundary is the first hop, full stop. Hold onto the phrase Linux network namespace — a second, unrelated thing also called a "namespace" shows up later, and the two get confused constantly.

🧵 That clears up the mechanism — but it leaves a loose end: if AWS runs this metadata service, why is **your* team the one deciding its hop limit?*

Who actually maintains it — AWS or you

Cleanly split:

Layer	Owner	What they handle
The service itself	AWS (internal)	Runs and maintains IMDS and the `169.254.169.254` endpoint. Nothing to patch, host, or babysit.
The configuration	You (the engineer)	Whether IMDSv2 is required, the hop limit, whether the endpoint is on at all, and any IAM/network restrictions.

Mechanically: attach an IAM role to an instance, and AWS mints temporary, auto-rotating credentials for it; the AWS SDKs inside your app fetch those from 169.254.169.254. No static keys, no manual rotation.

So there's nothing to keep alive at 3 a.m. — but the dials that decide how your instances talk to IMDS are yours, and the hop limit is one of them. Which means the request isn't asking AWS for anything — it's asking you to loosen how tightly your own nodes are locked down.

🧐 That makes the one clue you skipped worth chasing: **required for CastAI configuration* — and you've never installed anything called CastAI.*

The name in the request: what CastAI is, and why it forces the change

CastAI is a Kubernetes automation platform for cloud cost optimization. It watches your cluster continuously and, on its own, right-sizes workloads, bin-packs pods onto fewer nodes, and swaps pricey on-demand instances for cheaper (often Spot) ones — trimming the EC2 bill without anyone hand-tuning it.

And here's the part where it all clicks. To do that, one of CastAI's components runs as a pod on every node and has to read that node's own IMDS. That single dependency is the whole origin of the change — because reading IMDS from inside a pod is exactly what a hop limit of 1 blocks. Laid end to end, the request explains itself:

That's the why. But approving a change isn't about confirming the why — it's about confirming it's safe. Two questions now stand between you and the signature: what exactly does that per-node pod read, and is opening this door dangerous? Everything below is those two questions.

🤔 Starting with a reflexive worry: if CastAI has a pod on every node, is your traffic flowing through it?

It's a watcher, not a checkpoint

Good instinct, wrong picture — worth ruling out before it poisons everything downstream. A tempting model is that CastAI sits in the request path and everything routes through it. It doesn't. CastAI is not a reverse proxy, sidecar, or firewall in your traffic path. A user hits your app pod, the traffic goes straight to your app; CastAI never sees it.

What it actually is: a background observer and optimizer that pulls from two sources and correlates them.

The Kubernetes API server — how many pods are running, what CPU/memory they requested, whether any are stuck Pending because nodes are full.
The cloud provider — instance types, pricing, capacity, and (per node) spot-interruption signals.

From those it decides things like "this workload fits on a cheaper instance type" or "drain this spot node before AWS reclaims it." That work is spread across several pods — and only one of them is why the hop limit matters:

🔍 Under the hood — which CastAI component actually touches node IMDS
CastAI isn't one pod; it installs several into the castai-agent namespace, and only one is what the hop limit gates:

castai-agent — the read-only core (a Deployment); talks to the K8s API and streams telemetry to CastAI.

castai-cluster-controller — carries out scaling/node actions.

castai-evictor, castai-pod-mutator / pod-pinner — bin-packing and right-sizing.

castai-kvisor — the security agent (KSPM / runtime security; can run an eBPF DaemonSet). It's a DaemonSet too, so people point to it for the IMDS dependency — but it isn't the one.

castai-spot-handler — a DaemonSet on every node that reads IMDS (/latest/meta-data/spot/instance-action) to catch spot-interruption notices. This is the pod whose per-node IMDS access the hop limit gates. It only watches and reports; it doesn't drain nodes itself.

So CastAI is many pods, and the spot-handler is the one that needs IMDS.

❓ Which raises a smaller question you can't unsee now: people keep saying **agent* — is that a pod, or something lower-level running straight on the machine?*

"Agent" is just a role — the real actor is the DaemonSet

"Agent" is a role, not a privileged bare-metal binary. In Kubernetes it's still a container in a pod. What matters is which kind of workload wraps it, and two native Kubernetes shapes tell the story:

Deployment — the centralized-coordinator shape (the core castai-agent): a replica or two, somewhere in the cluster.
DaemonSet — Kubernetes guarantees exactly one copy on every worker node. Grow from 5 nodes to 50 and it drops a copy onto each newcomer automatically. It's the standard shape for per-node infrastructure: log shippers, security scanners, node-termination handlers — and CastAI's spot-handler.

That's why the problem is per-node: there's a spot-handler pod on each node, each one reaching for that node's IMDS from inside its own pod network. Fixing one node fixes one node — this is a fleet-wide job.

Deep dive — you've seen this shape in AWS's own tooling
aws-node-termination-handler is the identical pattern: a small DaemonSet pod on each host polling IMDS paths like /spot and /events, cordoning and draining the node when a notice arrives. CastAI's spot-handler is a specialized cousin.

🧐 Something still doesn't add up, though: CastAI already has a direct line to the Kubernetes API, which knows everything about pods and nodes — so why does it need to crawl down to each node's AWS metadata at all?

What Kubernetes can't see

Your instinct is half right, and the missing half is the point. Kubernetes knows the logical truth — this pod asked for 2 CPUs and 4 GB, that node is 80% full. It's blind to money and physical hardware.

Kubernetes says: "this pod needs 2 CPU / 4 GB."
The cloud says: "you're on an m5.large, on-demand, in us-east-1, at $X/hour — and there's a cheaper Spot option that fits."

A cost optimizer has to marry those two datasets, so it needs cloud-side facts Kubernetes can't supply. But the tidy version of this oversells IMDS's role:

💡 Nuance — it's not pricing that needs your node's IMDS
"CastAI reads instance types and prices from IMDS on every node" is wrong in the load-bearing detail. Instance catalogs and pricing come mostly from cloud provider APIs and CastAI's own pricing database (via the cluster-controller's IAM permissions) — not per-node IMDS. The thing that genuinely must be read from a node's own IMDS is live, node-local state, and the headline example is the spot-interruption notice the spot-handler watches for — real-time, specific to that box, available nowhere but its own 169.254.169.254. So the honest sentence is: the hop-limit change unblocks node-local IMDS reads (spot interruption chief among them), not CastAI's whole pricing engine.

So the spot-handler needs its node's IMDS.

🤔❓ Which finally detonates the question that should have been nagging since the container section: every normal app pod lives behind the exact same container network, so they're all two hops from IMDS too — why has a hop limit of 1 been "working" for them, and only breaks for CastAI?

The turn: a hop limit of 1 was never working for anyone

This is the part most explanations get subtly wrong. The comfortable assumption — that hop limit 1 is fine for normal pods — is false. It was never fine. A hop limit of 1 blocks every pod equally, CastAI's and yours. There's no per-app exemption.

The reason your app pods never complained is that they don't call IMDS at all.

Well-configured EKS apps get AWS credentials through IRSA (IAM Roles for Service Accounts) or the newer EKS Pod Identity. AWS injects a short-lived token straight into the container; the SDK finds it and talks to regional AWS endpoints — never touching 169.254.169.254.
Because they never knock on IMDS, they never hit the wall. They've been unaware it exists.

And the wall exists on purpose — it's a security control. If an attacker compromises a public-facing pod, a hop limit of 1 stops them from reaching 169.254.169.254 to steal the node's IAM credentials. It keeps a single compromised pod from becoming a compromised node.

💡 Clarification — "normal pods don't use IMDS" is only half true
Plenty of system pods do, and they break at hop limit 1 just like CastAI:

The EBS CSI driver reads instance identity from IMDS; at limit 1 in some setups, volumes fail to attach and pods hang in Pending (often the real cause of that post-upgrade outage).

Spot / node-termination handlers (AWS's and CastAI's) poll IMDS for interruption notices.

Some overlay CNIs need even more — Cilium can require a hop limit of 3.

This is exactly why EKS managed node groups default to 2. The honest framing isn't "CastAI is uniquely needy" — it's that CastAI's spot-handler joins the club of node agents that need IMDS, and your nodes happen to be locked at 1.

🧵 Two new threads pull at you: is IRSA-versus-IMDS a switch someone flips, and if a hacker can still use IRSA, why is that any less of a catastrophe?

Two keys: one opens the machine, one opens a drawer

Is IRSA a toggle? No. There's no "use IRSA instead of IMDS" checkbox. It's an architecture baked into the AWS SDK's credential provider chain — the SDK checks credential sources in a fixed order, and IRSA's injected token sits near the top. You wire it up by:

Creating a Kubernetes ServiceAccount annotated with an IAM Role ARN.
Attaching that ServiceAccount to the pod.

At launch, Kubernetes mounts a signed token into the container and points an env var at it. The SDK uses that first and never falls through to IMDS. (EKS Pod Identity reaches the same end by a newer route.)

So why isn't a stolen IRSA token as bad as a stolen IMDS one? Blast radius:

	Node IMDS credentials	IRSA / Pod Identity credentials
Whose identity	The entire EC2 node's IAM role	One pod's narrowly-scoped role
Typical power	Broad — attach ENIs, pull images, edit routes, talk to the control plane	Only what that app needs (e.g. one S3 bucket)
If stolen	Attacker can manipulate the wider AWS account	Contained to that app's sandbox

Steal an IMDS token and you have the keys to the machine; steal an IRSA token and you have the keys to one drawer. That asymmetry is why a hop limit of 1 is a sane default — and why raising it earns the scrutiny you're giving it.

🤔 Which lands on the question really holding up your signature: you're about to open the IMDS door for CastAI's pod — so what if CastAI's pod is the one that gets breached?

What if the breach comes through CastAI

The honest answer first: it's possible. No software is exploit-proof, and there's no 100% guarantee. The reason security teams still approve this isn't that a breach is impossible — it's that the blast radius is boxed in on three sides:

IMDSv2 is session-based (this is the "v2" mentioned earlier). There's no static string to grab and run. An attacker must first PUT for a short-lived token, then pass it in a header on the GET. The token expires quickly and won't work outside that instance's network boundary — copying it to a laptop gets nothing.
CastAI's IAM role is scoped tight — infrastructure/metadata actions (describe instances, node scaling), not your customer databases, app secrets, or private buckets. Even wearing CastAI's identity, an attacker inherits only CastAI's narrow powers.
Namespace isolation + Network Policies — CastAI lives alone in the castai-agent namespace. With standard Kubernetes NetworkPolicies enforced, cross-namespace traffic can be blocked, so an intruder in the CastAI pod can't freely pivot into your app pods.

Stated plainly: raising the hop limit to 2 does widen network visibility for node agents. It's a calculated risk — defended by IMDSv2 session tokens, least-privilege IAM, and namespace/network isolation — taken in exchange for continuous, automated cost savings. That's a sentence you can defend at 3 a.m.

🧠 You've now said "namespace" several times, and a splinter of doubt lodges: isn't "namespace" also a Linux kernel thing from containers? Are these the same word doing two jobs?

Two things named "namespace" — and they're not the same

They share a word, they're deeply connected, and they operate in completely different universes.

Linux namespace — the real isolation technology, at the kernel level. It makes a process believe it's alone on the machine. Containers are Linux namespaces wrapped around a process:

Network (net) — its own interfaces, loopback, IP. This is the extra hop from earlier.
PID — the container thinks its main process is PID 1, though the host sees PID 48291.
Mount (mnt) — sees only its own filesystem.

No Linux namespaces, no containers.

Kubernetes namespace — an organizational label, at the orchestration level. A K8s namespace (default, castai-agent, …) is just a tag in the Kubernetes database (etcd). The Linux kernel has never heard of it. It exists to group resources, scope RBAC, and set quotas — a locked department suite inside an office building.

Where they meet is your exact problem:

Kubernetes supplies the organizational wrapper (the K8s namespace); the Linux network namespace is what physically creates the hop that forces hopLimit = 2. Same word, two jobs, only one of them relevant to your change.

Deep dive — the escape hatch you're choosing not to use
A pod can set hostNetwork: true, dropping it into the host's network namespace — no pod net namespace, so it reaches IMDS in a single hop and ignores the limit. Some infra agents do this. CastAI keeps its components in the normal pod network (so it can map workloads cleanly), which is why the fix is a node-level hopLimit=2 rather than a hostNetwork special case.

The mechanism is fully understood now. What's left is operational — and one piece of it decides whether Sunday's window really needs the word downtime.

🤔 How hard is this pod hitting IMDS, anyway — is it a cron job?

Not a cron job — how often it really talks

A CronJob wakes on a timer, runs once, exits. CastAI can't work that way — cost, traffic, and scheduling pressure change by the second — so it runs as long-lived daemons, talking to different systems at different rhythms:

Signal	Cadence	Mechanism
Cluster state (pods/nodes)	Real-time	K8s API watchers — events pushed the instant a pod goes unschedulable
Node identity (ID / type / region)	Once at boot, then cached	A single IMDSv2 `GET` — the hardware doesn't change while the node lives
Spot-interruption notice	Continuous light loop	IMDS `/spot/instance-action` on the node
Cluster metrics	Short interval (impl. detail)	Metrics scrape

The spot loop is continuous because AWS gives little warning — up to ~2 minutes on AWS (≈30 seconds on GCP/Azure) — before reclaiming a Spot node; CastAI also runs an ML prediction model for a longer heads-up. (You'll see "every 15 seconds" quoted for metrics; treat it as approximate.)

The reassuring part: the node-identity read this change is really about happens once, at pod startup. This isn't a pod hammering the metadata service in a hot loop — it's a pod that needs one clean answer when it boots, and today, on your nodes, that answer never arrives.

❓ Last question, and it's the one that governs the maintenance window: to change the limit on a running cluster, do you have to replace the nodes — and is that where the downtime comes from?

Applying it: why the nodes have to be replaced

Yes. To land this on the existing cluster, the nodes have to be replaced (or at least relaunched).

The hop limit is baked into the EC2 launch template — the blueprint the node group / Auto Scaling Group uses to build nodes. Edit the blueprint from 1 to 2 and the already-running instances don't change; the new value only applies to newly launched nodes.

The exception worth knowing: on a single running instance you can flip it live, no reboot: aws ec2 modify-instance-metadata-options --http-put-response-hop-limit 2. But for a fleet you fix the launch template and roll the nodes, so the change is consistent and survives future scale-ups.

The choreography is a rolling replacement, not a big-bang reboot:

Cordon — mark an old node SchedulingDisabled so no new pods land there.
Drain — evict its pods; Kubernetes reschedules them onto other nodes.
Terminate & replace — once it's empty, kill it; the ASG launches a fresh node from the updated blueprint, now at hop limit 2.

And that explains the word that stopped you on Friday. Why "possible downtime"?

Single-replica workloads — a one-pod app blinks out while it's killed on the old node and recreated on the new one. Mitigation: ≥2 replicas plus a PodDisruptionBudget.
Capacity churn — moving many pods at once can cause brief blips while replacement nodes pass health checks.

A well-architected service with multiple replicas and sane PDBs rides the roll with zero user-visible impact. The "possible downtime" line is there for the single-replica and tight-capacity cases — which you can now go check by name instead of guessing.

At which point you know what the change does, why it exists, which pod needs it, why it was blocked, what it risks, and how to do it without taking the site down. So you approve it — not because you were told to, but because you can finally explain it.

The whole picture, in one diagram

What common explanations get wrong

"The hop limit default is always 1." → Plain EC2, yes; but EKS managed node groups and Amazon Linux 2023 default to 2. Nodes at 1 are usually hardened/custom (or a spot/Karpenter path).
"IMDSv2 sets the hop limit." → The hop limit is an independent option; IMDSv2 is the session-token requirement. Configured together, but distinct.
"The kvisor DaemonSet needs IMDS." → The per-node IMDS consumer is the castai-spot-handler DaemonSet (monitoring-only). kvisor is the security agent.
"CastAI reads instance type & pricing from node IMDS." → Type/pricing come mostly from cloud APIs + CastAI's pricing DB. The essential node-local IMDS read is the spot-interruption notice.
"Normal pods don't hit the hop wall." → Only if they use IRSA. System pods (EBS CSI, termination handlers, some CNIs) do, and fail at limit 1 — the reason EKS defaults to 2.

Read further

AWS — Use the Instance Metadata Service (hop limit / IMDSv2): https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html
AWS — Instance metadata access considerations (container / hop-limit-2 guidance): https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
AWS — Amazon EKS now supports IMDSv2 (why EKS node groups default to hop limit 2): https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-eks-supports-ec2-instance-metadata-service-v2
Amazon Linux 2023 — IMDSv2 (AL2023 defaults hop limit to 2): https://docs.aws.amazon.com/linux/al2023/ug/imdsv2.html
CastAI Docs — Read-only agent & Spot handler: https://docs.cast.ai/docs/about-the-read-only-agent · https://docs.cast.ai/docs/spot-handler
AWS — aws-node-termination-handler (the same DaemonSet-reads-IMDS pattern): https://github.com/aws/aws-node-termination-handler
Kubernetes — Namespaces: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
Linux — namespaces(7): https://man7.org/linux/man-pages/man7/namespaces.7.html
AWS EKS — IRSA & EKS Pod Identity: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

CastAI, AWS IMDS IRSA EC2, K8S

Sudhakar — Sat, 18 Jul 2026 07:17:54 +0000

From a Timeout Error to Patching EKS Nodes: An IMDS Rabbit Hole

A learning journey through AWS Instance Metadata, container networking, CastAI, Kubernetes vs. Linux namespaces, and the security reasoning behind a one-line node config change.

The 2AM version of why you're here

You didn't wake up curious about instance metadata. Something broke first. Maybe it looked like one of these:

A pod refuses to start and the logs read could not get EC2 instance identity metadata: ... context deadline exceeded.
After an EKS upgrade, half your pods are stuck Pending and the EBS CSI driver is quietly failing to attach volumes.
A change request lands on your desk: "Bump the IMDS hop limit from 1 to 2 on the EKS nodes for CastAI — requires node patching, possible downtime," and you're the one who has to approve or apply it.

All three trace back to the same tiny setting, and none of the one-line "just set the hop limit to 2" answers on the internet tell you why — or whether it's safe. That "why" turns out to run through container networking, IAM, two completely different meanings of the word namespace, and a real security trade-off. This article follows that thread from the error message all the way down.

First, what is IMDS — and why does anything need it?

IMDS is the EC2 Instance Metadata Service. It's a small, read-only service that every EC2 instance can query to learn facts about itself — its instance ID, type, Availability Zone, networking details, and (most importantly) the temporary IAM credentials for whatever role is attached to it.

Why does that need to exist? The alternative is ugly: baking long-lived AWS access keys into your application code or onto the server's disk, then rotating them by hand forever. Instead, code running on the instance just asks IMDS "what are my credentials right now?" and gets short-lived, auto-rotating ones back. That single idea — self-identity and keyless credentials, on demand — is what this whole article orbits. Hop limits, containers, and the security trade-offs are all just the plumbing wrapped around it.

And where does this "IMDS" thing actually live?

Before anything else, kill the most common wrong mental picture. IMDS is not a pod in your cluster, and it is not the public AWS API / control plane you reach over the internet.

It's a link-local endpoint — 169.254.169.254 (IPv6: fd00:ec2::254) — served locally on each individual EC2 instance by the underlying EC2/Nitro infrastructure. "Link-local" means the request never actually leaves the box to travel to some remote server; the instance asks itself "who am I, and what credentials do I have?" and the platform answers.

That single fact is the seed of the entire problem:

Because each instance serves its own IMDS locally, a request only reaches it from the instance's own network — in effect, the caller has to look like it's running directly on the box.
A container doesn't share the instance's network directly — it runs in its own isolated network sandbox with its own virtual network interface, wired to the host through a virtual cable (a veth pair into a bridge). So a packet leaving a container has to first cross that virtual boundary onto the host's network before it can go anywhere else. That crossing is the "extra layer." The moment your caller lives in a pod, "ask the instance about itself" stops being a local question and becomes a network hop — and that's where the hop limit starts dropping packets.

Keep that image — a local endpoint on the box, and a container sitting one layer away from it — and the rest of this article is really just unpacking its consequences.

TL;DR

Symptoms like context deadline exceeded, pods stuck Pending, or a CastAI change request all point at one setting: the IMDS hop limit, which typically needs to go from 1 → 2 on EKS nodes.
IMDS lives locally on each EC2 instance (169.254.169.254), not in Kubernetes and not the public AWS API. That locality is why network hops matter.
The hop limit is the TTL (time-to-live) on the metadata service's network response — it caps how many network layers a request can cross before packets are dropped. Default range is 1–64.
A pod lives in its own Linux network namespace, so a pod → IMDS request crosses two boundaries (pod network → host → IMDS). With a limit of 1, the request dies at the host. Limit 2 lets it through.
CastAI's spot-handler runs as a DaemonSet on each node and reads IMDS to catch spot-interruption notices. That's the concrete thing the limit is blocking.
Hop limit 1 isn't a CastAI-specific wall — it blocks every pod equally. Most app pods just don't notice because they use IRSA / EKS Pod Identity instead of IMDS.
Applying the change means replacing nodes (the setting lives in the launch template), done as a rolling drain — hence the downtime caveat.

Each section below starts with a question that actually came up, answers it, then hands you the next doubt it triggers. Boxes marked Deep dive and Clarification go beyond the surface.

1. What does "hop limit" actually mean?

Think of a hop as one pass through a router or network layer. Every IP packet carries a TTL (time-to-live) counter; each hop decrements it by one, and when it hits zero the packet is dropped. IMDS uses exactly this mechanism: the "hop limit" is the TTL stamped on the metadata service's responses.

Default: 1 on a plain EC2 instance. This deliberately keeps the response inside the instance — it can't be forwarded off the box.
Maximum: 64.
How you read/change it: the HttpPutResponseHopLimit field on the instance's metadata options (or the launch template).

💡 Clarification — a common oversimplification
"IMDSv2 sets the hop limit to 1" is not quite right. The hop limit is an independent instance setting, not something IMDSv2 turns on. Two are often conflated because they're configured together. And the "default is 1" rule has big exceptions: EKS managed node groups have defaulted the hop limit to 2 since 2020, and Amazon Linux 2023 AMIs default to 2 specifically to support containers. So if your EKS nodes are sitting at 1, that's usually a hardened or custom config (or a Karpenter/spot path that didn't inherit the default) — which is exactly why an explicit change request exists.

→ If containers are so common on EC2, why would a limit of 1 ever block them? What's special about a container?

2. Why containers need a hop limit of 2

A normal process running directly on the EC2 host reaches IMDS in one hop: process → 169.254.169.254. A containerized process does not, because it lives behind an extra network layer.

Container process
      │  (Hop 1)  leaves the pod's virtual network, crosses the bridge into the host
      ▼
EC2 Host OS network
      │  (Hop 2)  host forwards the request to the metadata service
      ▼
AWS IMDS (169.254.169.254)

With a limit of 1, the packet's TTL is exhausted the moment it reaches the host — before it can make the second jump to IMDS. The request silently times out. Bumping to 2 authorizes both jumps.

Deep dive — where the extra hop physically comes from
It's not metaphorical. The container runtime places each pod in its own Linux network namespace with a virtual ethernet pair (veth) bridged to the host. Crossing that veth boundary is the first hop. We'll come back to this when we compare Kubernetes namespaces with Linux namespaces (§10) — it's the same boundary that makes the whole hop-limit problem exist.

→ I get what IMDS is for now. But is it something AWS just runs for me, or something an engineer has to set up and maintain?

3. Who actually maintains IMDS — AWS or you?

We've settled what IMDS is for and where it runs. The question left from the journey is ownership: do you operate this thing, or does AWS? The mechanics make the answer clear.

When you attach an IAM role to an instance, AWS mints temporary, auto-rotating credentials for it. AWS SDKs inside your app know to fetch those credentials from the IMDS link-local address (169.254.169.254). No static keys to leak, no manual rotation.

Shared responsibility:

Layer	Owner	What they handle
The service itself	AWS (internal)	Hosts and maintains IMDS and the `169.254.169.254` endpoint. Nothing for you to patch or install.
The configuration	You (the engineer)	Decide whether IMDSv2 is required, set the hop limit, enable/disable the endpoint, restrict it with IAM or network policy.

So there's no "IMDS server" to babysit — but the knobs that decide how your instances talk to it are yours, and the hop limit is one of them.

→ Right — so what is CastAI, and why is a tool I might not have installed myself the reason someone wants to change my node settings?

4. What CastAI is — and why it's the reason for the change

We've been saying "CastAI" since the opening ticket without ever defining it. Time to fix that — because it's the whole reason the hop-limit change is on the table.

CastAI is a Kubernetes automation platform for cloud cost optimization. It continuously watches your cluster and automatically right-sizes workloads, bin-packs pods onto fewer nodes, and swaps pricey on-demand instances for cheaper (often Spot) ones — trimming your EC2 bill without anyone hand-tuning it.

To do that, one of CastAI's components runs as a pod on each node and has to read that node's own AWS metadata from IMDS. That single dependency is the entire origin of the ticket — because reading IMDS from inside a pod is exactly what a hop limit of 1 blocks (§2). Put the pieces together and the change request explains itself:

The ticket names CastAI
        │
        ▼
CastAI runs a per-node pod that must read the node's own IMDS
        │
        ▼
A pod → IMDS request costs 2 hops (§2) — but the nodes are capped at 1
        │
        ▼
⇒ raise the hop limit  1 → 2      ← the whole reason for the change request

Everything else in this article is really just the two follow-up questions that chain provokes: what does that per-node component actually read? and is it safe to open the door? We'll take them in turn.

First, a fair worry: does CastAI sit in my traffic path? A natural mental model is "CastAI is a pod on the node, so maybe all traffic routes through it first." It doesn't. CastAI is not a reverse proxy, sidecar, or firewall in your request path. If a user hits your application pod, that traffic goes straight to your app — CastAI never sees it.

Instead, CastAI is a background monitoring and optimization system. It gathers data from two places:

The Kubernetes API server — "How many pods are running? What CPU/memory did they request? Are any pods stuck Pending because nodes are full?"
The cloud provider — instance types, pricing, capacity, and (per node) spot-interruption signals.

It correlates the two to decide things like "this workload fits on a cheaper instance type" or "drain this spot node before AWS reclaims it." Concretely, that work is spread across a handful of pods — and only one of them is the reason the hop limit matters:

🔍 Under the hood — which CastAI component actually touches node IMDS
CastAI isn't a single pod; it installs several components into the castai-agent namespace, and only one of them is what the hop limit gates:

castai-agent — the read-only core agent (a Deployment); talks to the K8s API and streams telemetry to CastAI.

castai-cluster-controller — executes scaling/node actions.

castai-evictor, castai-pod-mutator/pod-pinner — bin-packing and right-sizing.

castai-kvisor — the security agent (KSPM / runtime security; can run an eBPF DaemonSet when enabled). (It's a DaemonSet too, so it's easy to mistake for the IMDS consumer — but it isn't the one.)

castai-spot-handler — a DaemonSet that runs on nodes and reads IMDS (/latest/meta-data/spot/instance-action) to detect interruption notices. This is the component whose per-node IMDS access the hop limit gates. It's monitoring-only; it reports events, it doesn't drain nodes itself.

→ So CastAI has multiple pods. Then when someone says "agent," is CastAI even a pod, or something lower-level? And where does it live in Kubernetes?

5. "Agent" vs. pod, and why DaemonSet is the key native feature

"Agent" is just a role — background software doing a monitoring job. It's not a special bare-metal binary; in Kubernetes it still runs as a container inside a pod. CastAI absolutely does need pods.

Two native Kubernetes workload types matter here:

Deployment — the centralized coordinator pattern (the core castai-agent). One (or a few) replicas somewhere in the cluster.
DaemonSet — Kubernetes guarantees exactly one copy on every worker node. Scale from 5 nodes to 50 and Kubernetes drops a copy onto each new node automatically. This is the standard pattern for per-node infrastructure agents (logging, security scanners, node termination handlers, and CastAI's spot-handler).

The DaemonSet pattern is why the hop-limit issue is per node: there's a spot-handler pod on each node, each one trying to reach that node's IMDS from inside a pod network namespace.

Deep dive — this same pattern in AWS's own tooling
AWS's own aws-node-termination-handler runs the identical shape: a small DaemonSet pod on each host polling IMDS paths like /spot and /events, then cordoning/draining the node when a notice appears. CastAI's spot-handler is a specialized cousin. If you understand one, you understand both.

→ Let's return to the hop limit with fresh eyes. If CastAI mostly talks to the Kubernetes API — and Kubernetes already knows about traffic and resource usage — why does it need node-level AWS metadata at all?

6. Why the Kubernetes API isn't enough

Your intuition is partly right: Kubernetes knows logical resource facts — this pod requested 2 CPUs and 4 GB, that node is 80% full. What Kubernetes is completely blind to is money and physical hardware:

Kubernetes says: "This pod needs 2 CPU / 4 GB."
The cloud says: "You're on an m5.large, on-demand, in us-east-1, costing \$X/hour — and there's a cheaper Spot option that fits."

A cost optimizer has to join those two datasets. So it needs cloud-side facts Kubernetes can't provide.

💡 Nuance — not all of that comes from node IMDS
The tidy story "CastAI reads instance type and pricing from IMDS on every node" overstates IMDS's role. Instance-type catalogs and pricing come largely from cloud provider APIs and CastAI's own pricing database (via the cluster-controller's IAM permissions), not from per-node IMDS calls. The thing that genuinely must be read from a node's own IMDS is live, node-local state — most importantly the spot-interruption notice the spot-handler watches for. That's real-time, node-specific, and only available at 169.254.169.254 on that box. So the honest phrasing is: the hop-limit change unblocks node-local IMDS reads (spot interruption being the key one), not "CastAI's entire pricing engine."

→ Fine — but here's what really bugs me: normal application pods live behind the exact same container network layer. They'd follow the identical two-hop path. So why does a hop limit of 1 seem to "work" for them but not for CastAI?

7. The question that unlocks everything: hop limit 1 doesn't "work" for normal pods either

This is the crux, and the usual explanation gets it subtly wrong.

A hop limit of 1 blocks every pod equally — CastAI's and yours. There's no per-application exception. The reason your app pods don't complain is that modern EKS apps don't call IMDS at all.

Instead of IMDS, well-configured app pods get AWS credentials through IRSA (IAM Roles for Service Accounts) or the newer EKS Pod Identity. AWS injects a short-lived token straight into the container; the SDK finds it and talks to regional AWS API endpoints — never touching 169.254.169.254.
Because they never ping IMDS, they never hit the 1-hop wall. They're blissfully unaware it exists.

💡 Clarification — "normal pods don't use IMDS" is only half true
Plenty of system pods absolutely do use node IMDS and will break at hop limit 1. Real-world examples people hit constantly:

The EBS CSI driver fetches instance identity from IMDS; at hop limit 1 in some EKS setups, volumes fail to attach and pods get stuck Pending.

Spot / node-termination handlers (AWS's and CastAI's) poll IMDS for interruption notices.

Certain CNI configurations need it (some overlay CNIs like Cilium even need a hop limit of 3: one for the node, one for the pod, one for the overlay).

This is precisely why EKS managed node groups ship with hop limit 2 by default — AWS knew containerized infra components need it. So the real framing isn't "CastAI is uniquely needy"; it's "CastAI's spot-handler joins the set of node agents that need IMDS, and your nodes happen to be locked at 1."

Why hop limit 1 exists at all: it's a deliberate security control. If an attacker compromises a public-facing pod, a limit of 1 stops them from reaching 169.254.169.254 to steal the node's IAM credentials (which are powerful — see §8). It contains the blast radius to that pod.

[ Compromised app pod ]
      ├─► tries IMDS  (hop limit 1) ──► BLOCKED (can't steal the node's admin-ish creds)
      └─► tries IRSA  ─────────────────► works, but only its own narrow permissions

→ That raises two things at once. Is IRSA-vs-IMDS a setting I flip per app? And if a hacker can still use IRSA, why is that any safer than IMDS?

8. IRSA vs. IMDS: not a toggle, and why the attacker gains little

Is it a config switch? No single "use IRSA instead of IMDS" toggle. It's an architecture baked into the AWS SDK credential provider chain. The SDK checks credential sources in a fixed priority order; IRSA-injected credentials sit near the top. To wire up IRSA you:

Create a Kubernetes ServiceAccount annotated with an IAM Role ARN.
Attach that ServiceAccount to the pod.

At launch, Kubernetes mounts a signed token into the container and sets an env var pointing at it. The SDK sees that token first, uses it, and never falls through to IMDS. (EKS Pod Identity achieves the same outcome with a different, newer mechanism.)

Why can't a hacker just use IRSA? They can — but it buys them almost nothing, thanks to least privilege:

	Node IMDS credentials	IRSA / Pod Identity credentials
Scope	The entire EC2 node's IAM role	One pod's narrowly-scoped role
Typical power	Broad: attach ENIs, pull images, modify routes, talk to the control plane	Only what that app needs (e.g., one S3 bucket)
Blast radius if stolen	Can manipulate broader AWS infra	Limited to that app's tiny sandbox

Stealing IMDS creds is a catastrophe; stealing one pod's IRSA creds is a contained incident. That asymmetry is the whole reason a hop limit of 1 is a sane default.

→ So we're deliberately opening that door for CastAI. What's my guarantee CastAI's own pod won't be the thing that gets popped? Is that even possible — and if not, why not?

9. What if the CastAI pod itself is compromised?

Honest answer: it's possible. No software is exploit-proof, and there's no 100% guarantee. Security engineers approve this change not because breach is impossible, but because the blast radius is heavily constrained by three overlapping controls:

IMDSv2 is session-based — no static string to copy-paste out. An attacker must do a PUT to get a short-lived token, then pass it in a header on the GET. The token expires quickly and won't work off the instance's network boundary. (This is the "v2" hardening; the hop limit and IMDSv2 together are what you're standardizing on.)
CastAI's IAM role is tightly scoped — infrastructure/metadata actions (describe instances, node scaling), not your customer databases, app secrets, or private buckets. Even assuming CastAI's identity, an attacker inherits only CastAI's narrow permissions.
Namespace isolation + Network Policies — CastAI lives in its own castai-agent namespace. With standard Kubernetes NetworkPolicies enforced, cross-namespace traffic can be blocked, so an attacker in the CastAI pod can't freely pivot into your app pods.

The trade-off, stated plainly: raising the hop limit to 2 does widen network visibility for node agents. It's a calculated risk, defended by IMDSv2 session tokens + least-privilege IAM + namespace/network isolation, in exchange for continuous automated cost optimization.

→ You keep saying "namespace." Is a Kubernetes namespace the same thing as a Linux namespace I've heard about in containers? Are they related?

10. Kubernetes namespace vs. Linux namespace — related, but different universes

They share a word and are deeply connected, but they operate at completely different layers.

Linux namespace = the real isolation technology (kernel level).
A Linux namespace is a kernel feature that makes a process believe it's alone on the machine. Containers are Linux namespaces wrapped around a process:

Network (net) — its own interfaces, loopback, and IP. ← this is the extra hop.
PID — the container's main process thinks it's PID 1, even though the host sees PID 48291.
Mount (mnt) — sees only its own filesystem.

Without Linux namespaces, containers wouldn't exist.

Kubernetes namespace = an organizational label (orchestration level).
A Kubernetes namespace (default, castai-agent, …) is just a metadata tag in the Kubernetes API database (etcd). The Linux kernel has no idea it exists. It's an admin construct for grouping resources, scoping RBAC, and setting quotas — like a locked department suite inside an office building.

How they meet in this exact problem:

[ EC2 Node ] ── single Linux kernel, owns the host network namespace
   │
   ├── K8s Namespace "castai-agent"   ← logical label in etcd only
   │      └── Pod: castai-spot-handler
   │             └── Linux net namespace (isolated IP)  ─┐
   │                                                     │ Hop 1 (cross the veth to host)
   └── Host network namespace (the EC2 OS) ◄─────────────┘
                                                         │ Hop 2 (host → IMDS)
                                     [ AWS IMDS 169.254.169.254 ]

Kubernetes provides the organizational wrapper (the K8s namespace); the Linux network namespace is what physically creates the hop that forces hopLimit=2. Two different meanings of "namespace," both true, meeting on the same node.

Deep dive — the hostNetwork escape hatch
A pod can set hostNetwork: true, which drops it into the host's network namespace — no separate pod net namespace, so it reaches IMDS in a single hop and doesn't care about the limit. Some infra agents use this. CastAI's model keeps components in the normal pod network (so it can map workloads cleanly), which is why the node-level fix is hopLimit=2 rather than hostNetwork.

→ Now I understand the plumbing. Operationally, how does CastAI run — is it a cron job that wakes up, checks IMDS and Kubernetes, and exits?

11. How often does CastAI actually poll? (It's not a cron job)

A Kubernetes CronJob wakes on a schedule, runs to completion, and exits. CastAI can't work that way — costs, traffic, and scheduling pressure change second to second. It runs as long-lived daemons.

Kubernetes state — real-time, event-driven. CastAI uses native informers/watchers: it holds a persistent streaming connection to the API server, which pushes events the instant a pod becomes unschedulable or usage shifts. No fixed polling timer for cluster state.
Node IMDS core facts — essentially once, then cached. Instance ID / type / region don't change while a node runs, so the agent reads them at startup and caches them; it only re-reads on pod restart or when a brand-new node joins.
Spot-interruption watch — continuous, lightweight loop. On spot nodes, the spot-handler keeps polling the IMDS interruption path, because AWS gives only a short notice (up to ~2 minutes on AWS; ~30 seconds on GCP/Azure) before reclaiming the instance. CastAI additionally runs ML-based interruption prediction for a longer heads-up.

Signal	Cadence	Mechanism
Cluster state (pods/nodes)	Real-time	K8s API watchers (push)
Node identity (ID/type/region)	Once at boot, cached	IMDSv2 GET
Spot interruption notice	Continuous short loop	IMDS `/spot/instance-action` on the node
Cluster metrics	Short interval (impl. detail)	Metrics scrape

(The often-quoted "every 15 seconds" metrics figure is an implementation detail that varies by setup — treat the exact number as approximate.)

→ Last question. To actually get hop limit 2 onto our existing cluster — do we have to restart or replace nodes? Is that why the change request mentions downtime?

12. Applying the change: why nodes get replaced (and why "potential downtime")

Yes — existing nodes must be replaced (or at least relaunched) to pick up the new limit.

The hop limit is baked into the EC2 launch template (the blueprint the EKS node group / Auto Scaling Group uses to build nodes). Editing the template from 1 to 2 does not retroactively change already-running instances. New settings only apply to newly launched nodes built from the updated template.

Note: on a single running instance you can change it live with aws ec2 modify-instance-metadata-options --http-put-response-hop-limit 2 (no reboot). But for a managed fleet you fix the launch template and roll the nodes, so replacements are consistent and the change survives future scale-ups.

How the rolling update works (no big-bang reboot):

Cordon — mark an old node SchedulingDisabled so no new pods land on it.
Drain — evict its pods; Kubernetes reschedules them onto other nodes.
Terminate & replace — once empty, delete it; the ASG launches a fresh node from the updated template, now with hop limit 2.

Why the change request still warns about downtime:

Single-replica workloads — a 1-pod app is briefly unavailable while it's killed on the old node and recreated on the new one. Mitigate with ≥2 replicas + a PodDisruptionBudget.
Capacity churn — moving many pods at once can cause short blips while new nodes pass health checks.

A well-architected app with multiple replicas and sane PDBs typically rides through the roll with zero user-visible impact. The "potential downtime" language is there for the single-replica and tight-capacity cases.

Putting it all together — one diagram

                          ┌─────────────────────────────────────────────┐
                          │           EKS Node (EC2 instance)            │
                          │  Linux kernel · host network namespace       │
                          │                                              │
   K8s API  ◄──watchers───┤  [castai-agent]  (Deployment, read-only)     │
   (push events)          │        │ telemetry → CastAI cloud            │
                          │        ▼                                     │
   CastAI cloud ◄─────────┤  optimization decisions                     │
   (pricing DB,           │                                              │
    cloud APIs)           │  [castai-spot-handler]  (DaemonSet)          │
                          │        │ reads node-local IMDS               │
                          │        │  Hop 1: pod netns → host  ──────┐   │
                          │        └─────────────────────────────────┘   │
                          │                              Hop 2: host → IMDS
                          └──────────────────────────────────┬───────────┘
                                                              ▼
                                              AWS IMDS (169.254.169.254)
                                   hop limit 1 = blocked · hop limit 2 = allowed

The mental model in five lines

Hop limit = TTL on IMDS responses; a pod-to-IMDS request costs 2 hops because a pod sits in its own Linux network namespace.
Limit 1 is a security default (contain node-credential theft); it blocks all pods, not just CastAI.
Most app pods skip IMDS via IRSA / Pod Identity, so they never notice — but node/infra agents (EBS CSI, spot handlers, CastAI's spot-handler) need it and break at 1.
Raising to 2 is defended by IMDSv2 tokens + scoped IAM + namespace/network isolation.
The setting lives in the launch template, so applying it means a rolling node replacement — hence the downtime caveat.

The common explanations, clarified

"Default hop limit is always 1." → Generic EC2 yes; but EKS managed node groups and Amazon Linux 2023 default to 2. Nodes stuck at 1 are usually a hardened/custom (or spot/Karpenter) config.
"IMDSv2 sets the hop limit." → Hop limit is an independent metadata option; IMDSv2 is the session-token requirement. Configured together, but distinct.
"The kvisor DaemonSet needs IMDS." → The per-node IMDS consumer is the castai-spot-handler DaemonSet (monitoring-only). kvisor is the security agent.
"CastAI reads instance type & pricing from node IMDS." → Type/pricing come mainly from cloud APIs + CastAI's pricing DB; the essential node-local IMDS read is the spot-interruption notice.
"Normal pods don't hit the hop wall." → They don't if they use IRSA. But system pods (EBS CSI, termination handlers, some CNIs) do and fail at limit 1 — the reason EKS defaults to 2.

The 'Disconnected' Dilemma: A Mac User's Journey to Connecting the Gemini CLI and Salesforce MCP server

Sudhakar — Mon, 27 Oct 2025 04:59:10 +0000

🧩 The "Disconnected" Dilemma: A Mac User’s Journey to Connecting the Gemini CLI and Salesforce

Connecting the Gemini CLI to Salesforce felt like unlocking a superpower — using natural language to query orgs, analyze code, and deploy metadata, all from the terminal.

But what I thought would be a 10-minute setup turned into a multi-step troubleshooting puzzle.

The main villain?

🔴 salesforce - Disconnected

This is the story of that journey, the problems I faced on Mac (Apple Silicon), and the “Aha!” moment that finally fixed it.

If you’re seeing that same red dot, this guide is for you.

🎯 The Goal & The Setup

The Goal: Configure the Gemini CLI to use the Salesforce MCP Server.

The Machine: A Mac with Apple Silicon (M1/M2/M3).

🧰 The Key Software

Gemini CLI – The AI agent.
Salesforce CLI (sf) – Installed and authorized with a default org.
npm / npx – Used to run the MCP server package.

🧪 The Troubleshooting Timeline

Phase 1: The “By the Book” Attempt (And the First Failure)

I started by following the developer documentation. It suggested using the Salesforce CLI directly to start the server.

My ~/.gemini/settings.json:

{
  "mcpServers": {
    "salesforce": {
      "command": "sf",
      "args": ["force", "mcp:server:start"]
    }
  }
}

The Result:
I launched Gemini and ran:

bash /mcp list

The Output:

✗ salesforce: ... - Disconnected

The command seemed logical but clearly wasn’t establishing the persistent connection Gemini needed.

Phase 2: Finding the Right Package (And a New Mystery)

After more research, I found that the standard way to run the server is by using the dedicated npm package @salesforce/mcp.

My ~/.gemini/settings.json (Attempt 2):

json { "mcpServers": { "salesforce": { "command": "npx", "args": [ "-y", "@salesforce/mcp@latest", "--orgs", "DEFAULT_TARGET_ORG", "--toolsets", "orgs,metadata,code-analysis" ] } } }

The Result:
After restarting Gemini and running /mcp list:

✗ salesforce: npx -y @salesforce/mcp ... - Disconnected

This was frustrating. The command was correct, the package official, and the org authorized. So why was it still disconnecting?

Phase 3: The “Aha!” Moment (Testing the Command Directly)

We realized we were assuming the command itself was failing.
But what if Gemini was the one failing to connect?

The Test:
Run the command directly in the terminal:

bash npx -y @salesforce/mcp@latest --orgs DEFAULT_TARGET_ORG --toolsets orgs,metadata,code-analysis

The Result:

Salesforce MCP Server v0.23.4 running on stdio

✅ The server worked perfectly.
The Salesforce CLI authorization and npx command were fine.
The issue wasn’t Salesforce — it was how Gemini was spawning the process.

Phase 4: The Mac Silicon Clue (Environment Mismatch)

Here’s where the Apple Silicon clue came in.

When Gemini (a Node.js app) spawns a child process (like npx), it doesn’t always inherit the full $PATH from your interactive shell (like .zshrc).
This happens frequently when using version managers like nvm or fnm.

Gemini was trying to run npx, but its isolated environment couldn’t find the command.
The process exited instantly, and Gemini reported Disconnected.

Phase 5: The Fix (Absolute Path to the Rescue)

If the $PATH is unreliable, don’t rely on it.
Tell Gemini exactly where npx lives.

Step 1: Find the Path

bash which npx

Example output:

/usr/local/bin/npx

Yours might differ, such as /Users/<your-name>/.nvm/versions/node/v20.x.x/bin/npx.

Step 2: Update settings.json

Final, working version:

json { "mcpServers": { "salesforce": { "command": "/usr/local/bin/npx", "args": [ "-y", "@salesforce/mcp", "--orgs", "DEFAULT_TARGET_ORG", "--toolsets", "orgs,metadata,code-analysis", "--tools", "deploy_metadata,retrieve_metadata" ] } } }

Phase 6: Success!

After saving the file and relaunching Gemini, I ran:

bash /mcp list

The Output:

✓ salesforce: /usr/local/bin/npx -y @salesforce/mcp ... (stdio) - Connected

It finally connected 🎉

⚙️ Now What? How to Use Your Connected Server

Getting Connected is just the first step.
Next, you need to use the available tools through natural language.

You don’t explicitly tell Gemini which tool to use — it infers that based on your prompt.

💬 How to Prompt Gemini

Use descriptive keywords that align with the tools (deploy_metadata, retrieve_metadata, code-analysis, etc.).

Examples:

Retrieve metadata:

“Please retrieve the metadata for the Account object from my default Salesforce org.”

Deploy metadata:

“Take the file src/classes/MyNewClass.cls and deploy it to Salesforce.”

Code analysis:

“Analyze the Apex class MyController.cls for any security vulnerabilities or performance issues.”

Each time, Gemini asks for confirmation and shows the exact command it plans to execute — proof that the MCP server is being used.

📚 Appendix: Installation & Useful Links

🧩 1. Running via npx (Standard Method)

bash npx -y @salesforce/mcp@latest --orgs DEFAULT_TARGET_ORG --toolsets orgs,metadata,data,users

What it does:
Downloads and runs the latest MCP package, then removes it.
Perfect for testing.

🧰 2. Installing Globally (Optional Troubleshooting Step)

If npx is unreliable, you can install globally:

bash npm install -g @salesforce/mcp

This ensures a stable global package version.
Not required for most users, but helpful if npx caching fails.

🔗 Related Documentation

💡 Key Lessons for Future Users

"Disconnected" = "Instant Exit" → The server process likely failed to start.
Test Commands Directly → Always run the command + args outside Gemini first.
Mac/Linux Users (especially with nvm/fnm): Use the absolute path to npx or other executables.
Windows Users: Use "cmd" with "/c" in settings.json, e.g.:

json "command": "cmd", "args": ["/c", "npx", ...]

🏁 Conclusion:
The “Disconnected” message wasn’t a bug — it was a clue.
Understanding how Gemini spawns its processes revealed the real issue: environment inheritance.
With the absolute path fix, the Gemini–Salesforce bridge now runs perfectly.

Salesforce MCP Server for Gemini CLI (Mac, Linux, Windows)

Sudhakar — Mon, 27 Oct 2025 04:52:37 +0000

⚙️ Configuring the Salesforce MCP Server for Gemini CLI

🧭 Hands-On Guide for macOS, Linux, and Windows

Connecting the Gemini CLI to the Salesforce MCP Server unlocks a powerful workflow — allowing you to query orgs, analyze code, and deploy metadata using natural language.

However, many users — especially on macOS with Apple Silicon — immediately encounter a frustrating

🔴 salesforce - Disconnected error.

This guide walks you through the correct, stable configuration to get you up and running.

💡 Root cause: usually an environment $PATH issue.

The fix: explicitly define the full command path Gemini should run.

📑 Table of Contents

Prerequisites
Step 1: Find Your settings.json File
Step 2: Get the Server Command (The Gotcha & Fix)
- macOS & Linux Users
- Windows Users
Step 3: Configure Your settings.json
- macOS / Linux Configuration
- Windows Configuration
Step 4: Verify the Connection
Step 5: Use the Connected Server
Appendix: Installation & Useful Links

🧰 Prerequisites

Before you begin, ensure you have the following installed and configured:

Gemini CLI – Google’s AI agent for your terminal
Salesforce CLI (sf) – The standard Salesforce command-line tool
Node.js / npm / npx – The Node.js runtime and package manager
Authorized Salesforce Org – You must have an org authorized and set as default in Salesforce CLI

Authorize an org:

sf org login web

Or set an existing org as default:

bash sf config set target-org <your-org-alias>

🏠 Step 1: Find Your `settings.json` File

Gemini CLI uses a configuration file named settings.json, located in your home directory.
This file tells Gemini how to start the Salesforce server.

Location:

~/.gemini/settings.json

If this file or directory doesn’t exist, create them manually.

🧩 Step 2: Get the Server Command (The Gotcha & Fix)

When Gemini starts an MCP server, it launches a child process.
This process might not inherit your shell’s full $PATH, especially on macOS/Linux when using Node version managers like nvm or fnm.

If your settings.json command is simply "npx", Gemini won’t find it — causing the Disconnected error.

✅ Fix: provide the absolute path to your npx executable.

For macOS & Linux Users

Run this command outside Gemini:

bash which npx

Example outputs:

/usr/local/bin/npx

/Users/your-name/.nvm/versions/node/v20.x.x/bin/npx

Use this exact path in your configuration.

For Windows Users

On Windows, npx must be invoked through the command interpreter.

Command: "cmd"
First argument: "/c"

🧾 Step 3: Configure Your `settings.json`

Open the file:

~/.gemini/settings.json

Then add the mcpServers block according to your OS.

💻 Final Configuration for macOS / Linux

json { "mcpServers": { "salesforce": { "command": "/usr/local/bin/npx", "args": [ "-y", "@salesforce/mcp@latest", "--orgs", "DEFAULT_TARGET_ORG", "--toolsets", "orgs,metadata,code-analysis", "--tools", "deploy_metadata,retrieve_metadata" ] } } }

💡 Replace "DEFAULT_TARGET_ORG" with your Salesforce org alias (e.g., "my-sandbox-alias").

🪟 Final Configuration for Windows

json { "mcpServers": { "salesforce": { "command": "cmd", "args": [ "/c", "npx", "-y", "@salesforce/mcp@latest", "--orgs", "DEFAULT_TARGET_ORG", "--toolsets", "orgs,metadata,code-analysis", "--tools", "deploy_metadata,retrieve_metadata" ] } } }

✅ Step 4: Verify the Connection

Save your configuration and restart Gemini CLI.
Then run:

bash /mcp list

Expected output:

✓ salesforce: /usr/local/bin/npx -y @salesforce/mcp ... (stdio) - Connected

🚀 Step 5: How to Use Your Connected Server

You don’t need to explicitly tell Gemini to “use the Salesforce server.”
Just write prompts describing what you want done — Gemini automatically routes them to the right MCP tools.

Example Prompts

Retrieve Metadata

“Please retrieve the metadata for the Account object from my default Salesforce org.”

Deploy Metadata

“Take the file src/classes/MyNewClass.cls and deploy it to Salesforce.”

Code Analysis

“Analyze the Apex class MyController.cls for any security vulnerabilities or performance issues.”

💬 When you run such prompts, Gemini will ask for confirmation before executing — confirming your MCP server is working correctly.

📚 Appendix: Installation & Useful Links

How to Install the Salesforce MCP Server

The setup uses npx to dynamically fetch and run the MCP package.

Standard method:

bash npx -y @salesforce/mcp ...

This command:

Downloads the latest @salesforce/mcp package
Runs it with your arguments (--orgs, --toolsets)
Removes it after execution

Optional (for troubleshooting):

bash npm install -g @salesforce/mcp

Installing globally may help if npx fails to find or cache the package properly.

🔗 Official Documentation

🧭 Salesforce DX MCP Server and Tools (Beta)
Official developer guide describing MCP server functionality and tools.
💻 Salesforce MCP Server GitHub Repository
Source code and community issue tracker.

🏁 You’re all set!
Your Gemini CLI is now successfully connected to the Salesforce MCP Server — ready to analyze, query, and deploy with AI assistance.

Syncing two Git Repos (By using below commands you can move contents of one git repo to another without losing history and tags)

Sudhakar — Sat, 10 Aug 2019 17:32:04 +0000

Cloning SourceRepo and Pushing that to TargetRepo

git clone --mirror SourceRepoURL

The above git clone command will create a directory in your local machine with the name of your repo.

Change to that directory and add a remote to the target repo.

cd SourceRepo
git remote add NEW-REMOTE TargetRepoURL
It creates a connection with TargetRepo.
Here'NEW-REMOTE' is just a name, you can give your own Name. Remember that name to sync these two repos.

git push NEW-REMOTE --mirror
It will push all folders and history of Source repo to Target Repo.

Syncing Those two Repos

After cloning source and pushing to target if you make any more changes to source simply use below two commands from the same directory to sync.

git fetch origin
To pull SourceRepo changes

git push NEW-REMOTE --all
To push(sync) SourceRepo changes to TargetRepo.

DEV Community: Sudhakar

How Meilisearch works

The End-to-End Execution Flow

The Big Picture Takeaway

AWS IMDS hop limit: the one-line change I wouldn't sign off until I understood it

First, what is IMDS — and why does anything need it?

Where IMDS actually lives

So what is a "hop limit"?

Why a container is different

Who actually maintains it — AWS or you

The name in the request: what CastAI is, and why it forces the change

It's a watcher, not a checkpoint

"Agent" is just a role — the real actor is the DaemonSet

What Kubernetes can't see

The turn: a hop limit of 1 was never working for anyone

Two keys: one opens the machine, one opens a drawer

What if the breach comes through CastAI

Two things named "namespace" — and they're not the same

Not a cron job — how often it really talks

Applying it: why the nodes have to be replaced

The whole picture, in one diagram

What common explanations get wrong

Read further

CastAI, AWS IMDS IRSA EC2, K8S

From a Timeout Error to Patching EKS Nodes: An IMDS Rabbit Hole

The 2AM version of why you're here

First, what is IMDS — and why does anything need it?

And where does this "IMDS" thing actually live?

TL;DR

1. What does "hop limit" actually mean?

2. Why containers need a hop limit of 2

3. Who actually maintains IMDS — AWS or you?

4. What CastAI is — and why it's the reason for the change

5. "Agent" vs. pod, and why DaemonSet is the key native feature

6. Why the Kubernetes API isn't enough

7. The question that unlocks everything: hop limit 1 doesn't "work" for normal pods either

8. IRSA vs. IMDS: not a toggle, and why the attacker gains little

9. What if the CastAI pod itself is compromised?

10. Kubernetes namespace vs. Linux namespace — related, but different universes

11. How often does CastAI actually poll? (It's not a cron job)

12. Applying the change: why nodes get replaced (and why "potential downtime")

Putting it all together — one diagram

The mental model in five lines

The common explanations, clarified

Further reading (for your own deep dive)

The 'Disconnected' Dilemma: A Mac User's Journey to Connecting the Gemini CLI and Salesforce MCP server

🧩 The "Disconnected" Dilemma: A Mac User’s Journey to Connecting the Gemini CLI and Salesforce

🎯 The Goal & The Setup

🧰 The Key Software

🧪 The Troubleshooting Timeline

Phase 1: The “By the Book” Attempt (And the First Failure)

Phase 2: Finding the Right Package (And a New Mystery)

Phase 3: The “Aha!” Moment (Testing the Command Directly)

Phase 4: The Mac Silicon Clue (Environment Mismatch)

Phase 5: The Fix (Absolute Path to the Rescue)

Phase 6: Success!

⚙️ Now What? How to Use Your Connected Server

💬 How to Prompt Gemini

📚 Appendix: Installation & Useful Links

🧩 1. Running via npx (Standard Method)

🧰 2. Installing Globally (Optional Troubleshooting Step)

🔗 Related Documentation

💡 Key Lessons for Future Users

Salesforce MCP Server for Gemini CLI (Mac, Linux, Windows)

⚙️ Configuring the Salesforce MCP Server for Gemini CLI

🧭 Hands-On Guide for macOS, Linux, and Windows

📑 Table of Contents

🧰 Prerequisites

🏠 Step 1: Find Your settings.json File

🧩 Step 2: Get the Server Command (The Gotcha & Fix)

For macOS & Linux Users

For Windows Users

🧾 Step 3: Configure Your settings.json

💻 Final Configuration for macOS / Linux

🪟 Final Configuration for Windows

✅ Step 4: Verify the Connection

🚀 Step 5: How to Use Your Connected Server

Example Prompts

📚 Appendix: Installation & Useful Links

How to Install the Salesforce MCP Server

🏠 Step 1: Find Your `settings.json` File

🧾 Step 3: Configure Your `settings.json`