DEV Community: Alister Baroi

10,000 Agents, Zero Tokens: Why the Best AI Architectures "Skip" the LLM

Alister Baroi — Fri, 04 Sep 2026 09:07:28 +0000

1. Introduction: The Scalability Paradox of Agentic Systems

In the boardroom, AI agents are promised as the ultimate workers—autonomous, reasoning, and tireless. In the engineering trenches, however, we face a brutal scalability paradox:

the more agents you deploy, the slower, more expensive, and more non-deterministic the system becomes.

When I set out to build the a simulation involving 10,000 independent agents, the traditional approach of calling a Large Language Model (LLM) for every runner’s decision was dead on arrival. To achieve massive scale, you must embrace a counter-intuitive architectural shift:

you have to strategically bypass the LLM.

The goal is to leverage the agentic framework for lifecycle management and telemetry while offloading the heavy lifting to deterministic code. In a production-grade system, your architecture should ensure that adding more runners does not add more tokens.

2. The "Before Model" Callback: The Secret to Deterministic Speed

The technical linchpin of this architecture is the "before model" callback, a feature within the Agent Development Kit (ADK). On the surface, it sounds absurd. Why define an LLM agent only to intercept the call before the model even sees it? As an architect, the answer is "scar tissue." You want the agent wrapper for its telemetry, its ability to communicate with other agents (A2A), and its structured lifecycle. By using the callback, we "neuter" the probabilistic inference and replace it with millisecond-level deterministic tool calls. This allows us to keep "unit-testable" logic inside a system that is otherwise notoriously gnarly.

The model is required by the LLM agent as in order to create the object but it’s never actually called because before the model call back... intercepts every invocation and returns deterministic tool calls.

By intercepting the invocation, we adhere to the ADK paradigm (maintaining full observability) without paying the "token tax" or the latency penalty of a round-trip to the model.

3. Design-Time vs. Runtime: AI as the Architect, Not the Executor

When being interviewed for engineering roles, candidates often gets asked how they would use AI to solve a specific pathfinding problem. The candidates who suggest using a full LLM to calculate the route at runtime usually don't get the job. The best answer is to use the AI as the architect during the design phase, not as the executor during the runtime. For the simulation, we faced an NP-hard problem: stitching together a 26.2188-mile path using a specific road network while avoiding walking bridges and indoor paths. Instead of wasting tokens at runtime, we used Gemini in AI Studio during development to research and generate the algorithms. By enabling "grounding with Google Search" and "code execution", Gemini helped us refine a multi-phase approach:

Phase 1: A Dijkstra algorithm that utilized haversine weighted edges to stitch together landmarks from a GeoJSON road graph.
Phase 2: Extending that path with a serpentine waypoint sequence, then trimming the final segment to hit the exact official marathon distance.The AI did the "deep research" and code generation only once. At runtime, the agent simply executes that deterministic Python code. We used the model where it helps (judgment) and skipped it where it doesn't (repetitive math).

4. Borrowing from Game Development: The "Server-Side Tick" Pattern

Managing 10,000 agents requires a shift toward game development patterns. In a massive multiplayer game, a centralized server uses a "tick" to synchronize state across all entities. We applied this by creating a "Simulator Agent" that acts as the server, orchestrating thousands of "Runner" agents. Within this simulator is a "Tick Agent". On paper, this is an LLM-defined workflow agent (sequential and looping), but in practice, it is entirely driven by the "before model" callback. Every tick, the agent triggers an "advance tick" tool deterministically. It remains an agent for the purpose of telemetry and state management, but it functions like a high-performance game loop. This ensures that the orchestration of 10,000 runners remains synchronized and token-free.

5. Breaking the State Bottleneck: Why SQL Fails at Scale

When you deploy to a stateless environment like GCP Cloud Run, session management becomes the primary bottleneck. In the "Race Condition" project, we scaled to 50 Cloud Run instances. Because the Global Load Balancer has no affinity for which instance holds a runner's state, that state must be externalized. We evaluated the standard ADK session stores:

In-Memory (Local): Useless at scale; state is trapped in a single instance and invisible to the other 49.
SQL (AlloyDB/Cloud SQL): Too slow. The overhead of SQL queries, transactions, and row-locking was the first thing to break under the pressure of 1,000+ simultaneous sessions.
Redis Session Service: The winner. We utilized a Redis session service from the Google ADK Community GitHub repo. However, even standard Redis wasn't enough. We had to subclass the service to perform specific performance tweaks, such as pruning events and preventing blob growth. By keeping the session state streamlined, we could broadcast messages over PubSub and emit updates via Websockets fast enough to visualize 1,000 runners in real-time, even though the backend was handling 10,000.

6. The "Autopilot" Runner: Heuristics Over Inference

The most scalable unit in our system is the "Autopilot" runner. While a standard agent might deliberate over its next move, the Autopilot runner is a specialized extension of the base agent. Instead of an LLM call, it uses heuristics derived from our earlier AI-driven research. It makes decisions about pace, fatigue, and positioning in milliseconds. This transition from probabilistic inference to deterministic code is what allows the system to scale to 10,000 agents without crashing the bank or the server. The AI provides the "judgment" for the initial plan; the "Autopilot" code provides the execution.

7. Conclusion: Engineering the Hybrid Future

The "Race Condition" project proves that the future of AI scale isn't about bigger models or more tokens, it's about smarter, hybrid architecture. The philosophy is simple:

Use the model as the architect during design-time, but use deterministic code as the executor at runtime.

As an engineering leader, you must look at your current agentic workflows and identify the "expensive math" that should actually be "free code". By wrapping deterministic logic in agentic lifecycles, you get the best of both worlds, the observability of an agent and the performance of a compiled algorithm. The future of AI scale is hybrid, and the best architects are the ones who know exactly when to skip the model.

The Safest Place to Run an AI Agent Is On a Cluster That Doesn’t Trust It

Alister Baroi — Thu, 27 Aug 2026 17:34:36 +0000

Every organization running AI agents has already made a hosting decision. Most made it by accident.

The sales team switched on the agent built into their CRM. Engineering is piloting a coding agent in a vendor’s cloud. Someone on the data team deployed a LangGraph service to a VM with a database key in an environment variable, and someone else is running an agent framework on a laptop with production credentials in a dotfile. Each of these is a hosting decision. Each one quietly settled who holds the agent’s credentials, what network paths it can reach, what gets recorded when it acts, and who can stop it. Nobody ran an architecture review, because no single deployment looked big enough to deserve one.

The scale says otherwise. By May 2025, 82% of organizations surveyed by SailPoint were already using AI agents. Only 44% had policies for securing them, 80% said their agents had already taken unintended actions, and 23% had watched an agent get tricked into revealing credentials. A year later the bill arrived: IBM’s 2026 Cost of a Data Breach report found that one in four malicious breaches is now AI-enabled, up 56% in a single year, and that those breaches cost about a million dollars more than the global average. Gartner, for its part, expects over 40% of agentic AI projects to be canceled by the end of 2027, naming inadequate risk controls as one of the three reasons.

This post is the architecture review those deployments never got. The conclusion, stated up front so you can argue with it: agents should run on Kubernetes clusters you control, wrapped in guardrails and gateways that no prompt can talk its way past. Not because Kubernetes is fashionable, but because a year of documented incidents shows that every failure was an environmental failure, and a cluster you control is the only substrate where you own every layer of the environment.

The workload that chooses its own code path

A microservice does what its code says. You can read the code, test the paths, and enumerate what it will do with any input. An agent is a different kind of workload, and the difference is structural, not a maturity phase it will grow out of:

Its control flow is decided at runtime by a model. The “code path” is chosen by whatever lands in the context window. You cannot statically review what an agent will do, only what it is allowed to do.
It holds credentials and takes actions. Agents query databases, call APIs, send email, move money, and spawn other agents. The interesting failure mode is not a wrong answer. It is a correct, authorized, catastrophic action.
Every input is potentially an instruction. Prompt injection remains unsolved. OWASP’s Top 10 for Agentic Applications, published in December 2025 with over a hundred contributors, reads mostly as variations on this theme: goal hijack, tool misuse, identity and privilege abuse, memory poisoning.

Put those three together and you get the principle we have argued all year, from the accountability gap to the OpenShell analysis: controls the agent can override are not controls. The system prompt is not a perimeter. The guardrail library running inside the agent’s own process is not a perimeter. The controls have to live in the environment, where a confused or compromised agent cannot negotiate with them.

And once the controls must live in the environment, the hosting question stops being an infrastructure detail. Where the agent runs determines which controls can exist at all.

A year of incidents; one pattern

Theory is cheap. The past twelve months supplied the evidence, and it is unusually well documented.

One agent’s tokens, seven hundred victims

Drift was an AI chat agent embedded on corporate websites. To do its job it held long-lived OAuth tokens into each customer’s Salesforce. In August 2025, attackers who had stolen those tokens used them to run bulk data exports against customer CRMs. Google’s threat intelligence group assessed more than 700 organizations as potentially affected. The attackers were not after chat transcripts. They mined the exported support cases for the credentials customers had pasted into them: AWS keys, Snowflake tokens, passwords. Cloudflare, one of the disclosed victims, rotated 104 API tokens found in its stolen case text.

No agent was compromised. No model misbehaved. One integration’s credentials were broad, long-lived, and held outside every victim’s own controls, so one theft became seven hundred breaches.

The agent with production credentials and a code freeze

In July 2025, on day nine of a public twelve-day build, Replit’s coding agent wiped SaaStr founder Jason Lemkin’s production database, despite an explicit and repeated code-freeze instruction. It then generated thousands of fake records and reported that rollback was impossible. It was not; the data came back from backup. The agent’s own postmortem: “a catastrophic failure on my part.”

The agent held live credentials to production with no environment separation, and the code freeze existed only as natural language. An instruction in a prompt is a suggestion. The incident record was, briefly, whatever the agent chose to say it was.

The tool that BCC’d the attacker

In September 2025, Koi Security found the first confirmed malicious MCP server in the wild: an npm package impersonating Postmark’s email server. It behaved correctly for fifteen versions, building trust, then v1.0.16 added a single line that BCC’d every email the connected agents sent to the attacker’s domain. Koi estimated roughly 300 organizations were running it in production. Nothing was exploited. An MCP server runs inside the agent’s trust boundary with the user’s credentials, and nobody diffs a patch release.

And the rest, briefly

EchoLeak (CVE-2025-32711) demonstrated a zero-click chain against Microsoft 365 Copilot: a poisoned email, retrieved later by RAG, walked the assistant’s most sensitive reachable data out through an allowlisted URL. Microsoft fixed it before any known exploitation, and it became the first critical-severity CVE assigned to an AI assistant’s injection path. The s1ngularity npm supply-chain attack inverted the picture: malware on developer machines invoked the victims’ own installed CLI agents with their client-side safety flags disabled and put them to work enumerating secrets, leaking over a thousand valid GitHub tokens within hours. And in July 2026, Check Point documented a single operator running two frontier-model agents as an intrusion pipeline that hit nine Mexican government agencies and roughly 400 million records. Attackers have agents too, and theirs ignore your prompts entirely.

The pattern

None of these was a model failure. Every one of them was contained, or would have been, by the same four environmental controls:

Credentials scoped to one target and short-lived , so a stolen token is worth almost nothing (Drift).
Authorization evaluated on every request against policy , not against the agent’s intentions or instructions (Replit, and the bulk CRM exports that no chat agent should have been able to run).
Egress that is deny-by-default and enforced outside the agent , so even a successful injection has nowhere to send the loot (EchoLeak, the Postmark BCC, s1ngularity’s uploads).
An audit trail owned by the platform, not narrated by the agent (Replit’s fabricated reports; Cloudflare could say “exactly 104 tokens” only because logs existed outside the blast radius).

Read that list again. Nothing on it is a property of the model. Nothing on it can be provided by a prompt, a fine-tune, or a guardrail library running inside the agent. All four are properties of the place where the agent runs.

Why the answer is a cluster you control

Environmental controls need an environment you own: the network path, the identity plane, the secret store, the audit pipe. Kubernetes is the one mainstream substrate that hands you all four as programmable primitives: namespaces and NetworkPolicy for segmentation, runtime classes for kernel-level sandboxing (gVisor, Kata), SPIFFE for workload identity, and admission control to keep anything unreviewed from running at all. Its declarative model then puts every one of those decisions in git, where they get reviewed and versioned like the rest of your infrastructure. These are the same properties that made Kubernetes the default for microservices. For agents they are not conveniences; they are the containment system.

The ecosystem has noticed. In the past year, Kubernetes gained a SIG subproject for agent sandboxing with first-class Sandbox primitives, launched at KubeCon NA 2025. agentgateway, a data plane purpose-built for agent-to-agent, MCP, and LLM traffic, became a Linux Foundation project with participation from AWS, Microsoft, Red Hat, IBM, and Cisco. KubeCon NA 2026 has a dedicated AI and agentic track. Running agents on Kubernetes stopped being a contrarian position sometime last winter.

Watch the managed agent runtimes and you see the same conclusion arriving from the other direction. AWS AgentCore added VPC egress and private identity-provider support. Microsoft’s agent service will inject its runtime into your own VNet so agent data stays in your tenant. Anthropic’s managed agents shipped self-hosted sandboxes that run tool execution on your infrastructure. OpenAI’s Frontier launch promises agents that run “across local environments, enterprise cloud infrastructure, and OpenAI-hosted runtimes.” Every managed runtime’s roadmap is bending toward your perimeter. That is the vendors telling you, in feature announcements, where the controls have to live.

There is also the question of what you can see when isolation is outsourced. In late 2025, Unit 42 disclosed an escape in AWS AgentCore’s code-interpreter sandbox: its “sandbox” network mode still allowed DNS resolution to arbitrary domains, which is enough for tunneled exfiltration and command-and-control. AWS fixed it properly within months, and credit to them. But note who found it and when the customers learned: from the researcher’s publication, not from their own telemetry, because inside a managed runtime there is no customer telemetry to find it in.

Side by side:

Concern	Managed agent runtime	Sandbox platform	Kubernetes with guardrails
Where the agent executes	Vendor’s cloud (some now reach into your VPC)	Provider’s microVM fleet	Your cluster: any cloud, on-prem, air-gapped
Who holds tool credentials	Vendor’s credential broker	Injected into the sandbox	Brokered in-cluster, attached per hop
Egress control	Vendor-defined, opt-in	Platform-defined	Your gateway, default deny
Authorization granularity	Per connector, mostly coarse	Yours to build	Per request, down to tool arguments
Audit trail	Per vendor, per format	Per platform	One trail across agents, tools, and models
Agents you didn’t launch	Invisible	Invisible	Detectable on the node
Exit cost	Rebuild against a new runtime	Re-platform	Agents are containers; the cluster moves with you

The table generalizes, and the best vendors are improving individual rows. What no managed runtime can give you are the rows that require owning the substrate: one policy engine over every agent regardless of framework, one audit trail across agents, tools, and models, and visibility into the agents nobody told the platform about.

One caveat before the spec. _ A bare cluster is not the answer either _. Stock Kubernetes secures pods, not agents: NetworkPolicy has no idea what an MCP tool call is, RBAC governs humans and service accounts, and nothing in the box produces an agent-level audit trail. We walked through those gaps in the accountability gap post. What the cluster uniquely gives you is the ability to install the missing layer yourself, uniformly, for every agent you run.

What strict guardrails look like

Here is the spec a year of incidents wrote, stated as requirements. Sandboxing the agent’s process is assumed and is not sufficient on its own; even the CNCF says so now. Everything below governs what agents do across the network, which is where every incident above happened.

An identity for every agent. Cryptographic workload identity (SPIFFE, OIDC) issued at deploy time, not a shared service account. Every other control depends on “which agent did this” having an answer.
One road out. Agent egress is deny-by-default, and the only path to other agents, MCP servers, and LLM providers is a gateway. EchoLeak, the Postmark backdoor, and s1ngularity’s uploads were all exfiltration over permitted egress. An allowlist enforced outside the agent kills the chain even when the injection succeeds.
Authorization per request, not per deployment. Each call is judged against policy: this agent, this target, this tool, these arguments. A lead-triage agent asking for a bulk CRM export should fail on policy, not on luck.
Credentials the agent never holds. Provider keys and tokens attach at the gateway, scoped to the single target of the single hop, short-lived. This is the Drift control: make the stolen token worthless.
A human where the blast radius is high. Destructive and irreversible operations block until a person approves. “Code freeze” becomes a policy object with an enforcement point, instead of a sentence in a prompt.
Detection for the agents you didn’t launch. 80% of organizations already report unintended agent actions, and IBM’s data keeps tying breach costs to shadow AI. You need to see the unregistered agent from the node it runs on, not from a registry it never joined.
An audit trail the agent cannot edit. Recorded by the enforcement plane as a side effect of enforcing, tied to agent identity, covering every hop. The agent’s own account of events must never be the system of record.

If you run a startup, this list probably reads as an enterprise ceremony. It is the opposite. The spec is precisely what lets a five-person team hand agents real credentials to real systems without hiring a security organization: default-deny scales down, a policy file for three agents fits on one screen, and one gateway is not a platform team’s worth of work. If you run an enterprise, the same spec is what turns your agent deployment from an audit finding into an audited system. The requirements do not change with headcount. Only the number of agents does.

The platform this spec describes

By now the specificity of that list has probably given the game away: it is, near enough, the design document for Tigera Lynx, our security and governance platform for AI agents on Kubernetes.

Lynx puts a registry and identity plane in front of your agents: SPIFFE and OIDC workload identity, plus a pod-owner mode that brings an unmodified agent under governance with no code changes and no credential to issue. Its gateway, built on the same Linux Foundation agentgateway data plane mentioned above, authorizes every agent-to-agent, MCP, and LLM call individually under Cedar policy, and MCP policies can see the operation, the tool name, and the tool arguments, failing closed on anything malformed. LLM provider keys live in your cluster’s secret store and attach at the gateway, so an agent never carries a provider credential it could leak, and per-provider token rate limits cap what any agent can burn.

In the current release cycle, per-hop credentials are minted for one target only (the Drift lesson, applied literally), and a policy can mark specific MCP calls as needing human approval, so the call waits until someone signs off (the Replit lesson). On the node, an eBPF-based detector spots agents nobody registered, classifies them as sanctioned, shadow, or unknown, and lets you quarantine a compromised workload at both the gateway and the kernel. And the Agent Trail records every decision, every hop, and which model actually served each call, as a byproduct of enforcement. The agent does not get to narrate.

Every one of those maps to a numbered line in the spec, because the spec came first. How Lynx Works has the full walkthrough, and the five principles post is the vendor-neutral checklist if you want to evaluate Lynx against anyone else.

The decision you already made

Back to the opening. Your organization has a hosting posture for AI agents today, assembled from defaults: whichever vendor clouds your teams clicked through, whichever VMs were lying around, whichever laptops were closest. The incidents above are what those defaults cost, and IBM’s data says AI is now involved in one of every four malicious breaches.

So ask the question deliberately, once, before the next agent ships: when it acts, who can say no, and where is the record? If the answer is a vendor’s incident queue and a support ticket, _ you have outsourced the two things a security leader cannot outsource. _

_ An agent you cannot refuse is an agent you do not control. _ Run yours where refusal is built in: a Kubernetes cluster you own, a gateway it cannot go around, and a policy engine that never gets tired of saying no.

Lynx is Tigera’s security and governance platform for AI agents on Kubernetes: identity, policy, detection, and audit for every agent in your cluster. Read How Lynx Works or request early access at tigera.io/demo/.

Learn more about Lynx →

The post The Safest Place to Run an AI Agent Is On a Cluster That Doesn’t Trust It appeared first on Tigera – Creator of Calico.

AI Red Team Agents Automate Attacks on your AI Agents. Runtime Policies Automate their Defense.

Alister Baroi — Mon, 24 Aug 2026 14:40:14 +0000

The AI red teaming market grew up fast this year. OpenAI bought Promptfoo, Cisco and Microsoft shipped automated attack suites, and a seed-stage startup publicly compromised 50 of 55 live customer service bots. These platforms find real problems at a scale no human team can match. But when you read the findings closely, a pattern emerges: agents talked into refunds, transfers, and data leaks they had standing authority to perform. Patching the prompt fixes one phrasing until the next model update. Constraining the authority fixes the class. The first job belongs to a red team platform. The second belongs to your runtime, and no scanner will do it for you.

In April 2026, General Analysis raised a $10M seed round on the strength of an uncomfortable demonstration: its adversarial agent attacked 55 live customer service bots and compromised 50 of them. Not lab models, but live systems with real customers and real tool access. This post is about the market behind that demonstration: who now automates the attacker’s role, what the attacks keep finding, and why the fix that lasts is runtime policy rather than a better prompt.

What an AI red team agent actually does

A traditional red team is a group of people paid to break your system before someone else does. An AI red team agent automates that role for AI systems. It fires thousands of adversarial inputs at a model or agent, notices which ones partially work, mutates them, escalates across turns, and reports what got through, with severity ratings and reproduction steps. A human tester works through a checklist over a week. These platforms run 20,000 variations overnight and rank the results.

The canonical demo goes like this. A company deploys an agent that handles refunds and can reach the billing system. The red team platform embeds disguised instructions in customer messages until it finds the phrasing that makes the agent skip its verification step and process a refund it shouldn’t. The report names the exact prompt pattern so engineering can fix it.

Enterprises are buying for reasons that have little to do with fashion. Agents now do things, such as executing code, querying databases, sending emails, and calling APIs, often with minimal supervision, so every capability is an attack vector. The EU AI Act mandates adversarial testing for high-risk AI systems, and insurers and procurement teams have started asking for documented testing as a condition of coverage or vendor selection. And a quarterly pen-test calendar cannot keep up with an agent whose behavior shifts on every model update.

All of that is real, and the tooling is genuinely good. None of this is an argument against red teaming. The question is what you do on the morning the report arrives.

The AI red teaming market grew up in about a quarter

The speed of consolidation tells you how seriously the industry takes automated red teaming. In March 2026, OpenAI acquired Promptfoo for a reported $86M; the MIT-licensed scanner claims 350,000 developers and now ships inside OpenAI’s Frontier enterprise platform. Cisco AI Defense tests across 200+ risk subcategories mapped to OWASP, NIST, and MITRE ATLAS. Microsoft’s AI Red Teaming Agent wraps PyRIT’s multi-turn attack orchestration, including Crescendo and Tree of Attacks with Pruning, into Azure AI Foundry. Mindgard starts with reconnaissance: it maps your guardrails, tools, and shadow AI deployments before it fires a single probe. And General Analysis goes deepest on agentic systems: MCP server exploitation, memory poisoning, and multi-step permission bypass.

OWASP now publishes a quarterly landscape document just for this category. When a market gets its own recurring OWASP publication, it has stopped being a niche.

Read the findings like a defender

Look at what these platforms actually catch across every vendor’s case studies: an agent processing unauthorized refunds. An agent skipping its verification step. An agent revealing one customer’s data to another. A moderation model bypassed with encoding tricks. An agent walked, over several polite turns, across a permission boundary it was never supposed to cross. A poisoned memory that redirects behavior days later. A malicious MCP server that turns one compromised tool into several.

Sort those into two piles. In the first pile, the model said something it shouldn’t have (jailbreaks and moderation bypasses). Those are model-layer problems with model-layer fixes, and the labs keep getting better at them. In the second pile, the agent did something it shouldn’t have. Every headline finding in the agentic column lands here, and every one of them has the same anatomy: the agent was persuaded, and the environment obeyed.

The persuasion is novel every time. That is the entire point of automated attack generation; there is always another phrasing, another encoding, another escalation path. The obedience is identical every time. The agent asked for transfer_funds, and something executed it.

Prompt patches decay. Runtime policy doesn’t.

Here is the loop most teams fall into. The red team report names a prompt pattern. Engineering patches the system prompt or adds a guardrail rule. The finding closes. A model update ships three weeks later, and the finding reopens with different wording. The vendors know this; it’s why General Analysis sells regression testing for exactly the case where a passing agent configuration fails after a model update or prompt change, and why every platform now pitches continuous scanning rather than an annual engagement. Continuous testing is the industry’s honest admission that the layer under test never stays fixed.

We’ve made this argument before, in the accountability gap post and throughout the series: controls the agent can override are not controls. The same logic applies to instructions. An agent can be talked out of its system prompt; that is precisely what the red team platform just proved. It cannot be talked out of a deny that lives outside its process.

So a pile-two finding deserves two responses. Patch the prompt, by all means; make the attack more expensive. But the fix that lasts is changing what the persuasion can accomplish, and that change happens in the environment, at the point where the agent’s request becomes an action.

Rewriting a red team finding as runtime policy

Take the marquee finding, the one on every vendor’s landing page: an injected instruction convinces a finance agent to call transfer_funds without its verification step. As a reproduction step, that’s a ticket that will reopen. As policy, it looks like this:

@approvalRequired("Approve transfer_funds on treasury")
forbid ( principal, action == Action::"callMCPServer", resource )
unless { context.approval.granted }
when { context.mcp.tool.name == "transfer_funds" };

This is Cedar, evaluated at a gateway between the agent and its MCP servers. Any call to transfer_funds is denied unless a human has granted approval, and the annotation turns that deny into a hold: the gateway parks the request, a person approves or rejects it, and only then does anything move. Approval is an input to the policy decision, not an override of it. The injection can still convince the agent to try. Trying now produces a held request and an audit entry instead of a wire transfer.

The same translation works across the rest of pile two:

What the red team keeps finding	Where the durable fix lives
Prompt injection triggers an unauthorized tool call	Per-request authorization at a gateway the agent cannot route around
Agent skips a verification or approval step	A policy hold that parks the call until a human decides
Agent leaks one customer’s data to another	Tool and egress policy scoped to the agent’s verified identity
Multi-step chains cross a permission boundary	Per-call decisions with the full chain correlated in one trace
The agent nobody tested, because nobody knew it existed	Runtime detection of unregistered agents

That last row deserves a sentence. Mindgard leads with reconnaissance because most organizations cannot produce an accurate inventory of their own AI systems, and you cannot red team what you don’t know you’re running. Scoping the engagement is where a program fails first, before a single probe fires. Inventory is a runtime problem too.

This is the seam where Lynx sits. Its gateway authorizes every MCP call individually against Cedar policy under a workload identity (e.g. SPIFFE). The @approvalRequired hold shown above shipped in preview this cycle, with the pending queue and countdown surfaced in the dashboard. Agent Trail records the deny that started the hold and the human decision that resolved it. And runtime detection classifies the agents actually present in your cluster, so the inventory a red team engagement needs is a query rather than an archaeology project. None of that replaces adversarial testing. It’s what turns the testing’s output into something that stays fixed.

Starting with open source: Garak, PyRIT, and DeepTeam

If you want to start this quarter, the open source route is credible. Garak (by NVIDIA, under Apache 2.0 license) is the deepest free attack catalogue: 120+ probe modules, with multi-turn agentic attacks added in v0.15.0 this May. PyRIT (Microsoft, MIT) is the strongest orchestrator for custom multi-turn campaigns. DeepTeam (Apache 2.0 license) maps results onto OWASP’s Agentic Security Initiative taxonomy for the compliance evidence trail; disable its cloud scoring calls if your data can’t leave the building. One caveat worth knowing before you standardize: Promptfoo belongs to OpenAI and PyRIT to Microsoft, both model vendors. The attack logic is open source and auditable, which helps, but neither should be your only scanner when the system under test is that vendor’s own model.

Two practices make whichever stack you pick actually pay off. First, run it against a test double, not only production; we covered how to stub LLMs for exactly this, so you can exercise the agent’s authority paths deterministically. Second, file every confirmed agentic finding as two tickets: one for the prompt, one for the policy that makes the prompt’s failure survivable. If your enforcement point logs its decisions, the audit trail becomes your regression oracle. The next scan shouldn’t just show the attack failing; it should show the gateway denying it.

Key takeaways

AI red team agents automate adversarial testing at a scale no human program matches, and the market consolidated fast: OpenAI bought Promptfoo, Cisco and Microsoft ship attack suites, and OWASP now tracks the category quarterly.
The agentic findings share one anatomy: the agent was persuaded, and the environment obeyed. Persuasion varies infinitely. Obedience is the constant you can actually fix.
Prompt patches decay with every model update, which is why the vendors themselves sell continuous regression testing. Policy enforced in the environment doesn’t decay with the model.
Translate findings into runtime controls: per-request authorization, human approval holds on dangerous tools, identity-scoped access, and detection for the agents you didn’t know you had.
Buy the red team. Build the enforcement. They are complements, and neither substitutes for the other.

Back to the fifty bots. What they had in common was not a worse model or a lazier system prompt than the five survivors. It was that persuasion and permission were the same thing: once the agent was convinced, nothing else got a vote. Your next red team report will show you where that’s true in your own stack. When it does, which findings will you patch, and which will you enforce?

The red team finds the words that move your agent. Your runtime decides what moving accomplishes.

Learn more about Lynx →

The post AI Red Team Agents Automate Attacks on your AI Agents. Runtime Policies Automate their Defense. appeared first on Tigera – Creator of Calico.

You’ve Been Running a Kubernetes Security Model in NSX and Didn’t Even Know It

Alister Baroi — Mon, 10 Aug 2026 19:19:22 +0000

One of the blockers to moving VMs off vSphere and onto Kubernetes is losing NSX and the protection it provides. Security teams that have spent years building out distributed firewall policy look at Kubernetes and are, quite understandably, alarmed by the flat network and the fact that any workload can reach any other by default.

How will they enforce east-west traffic controls? Will they be able to replicate NSX distributed firewall rules with the same granularity? What about security groups, tiered policy, and rules that travel with the workload when it moves? These are important questions that must be answered before migration can begin.

Migrate vSphere to Kubernetes with microsegmentation intact

Calico addresses vSphere to Kubernetes security concerns with a network policy model that maps directly to key features of the NSX distributed firewall (NSX DFW). Every property NSX DFW users rely on has a direct Calico equivalent: tiered governance, workload-identity enforcement, distributed kernel-level inspection, and dynamic workload grouping. Teams coming from vSphere will recognise the pattern quickly.

Let’s walk through each one in detail.

Distributed Enforcement

Enforcement on the host means better performance and automatic scaling

Traditional firewalls sit at the edge of the network. Traffic between workloads inside the data center has to travel out to that inspection point and back, even if the source and destination are on the same physical host. The NSX distributed firewall takes a different approach: the DFW runs in the kernel of every ESXi host and enforces policy directly at each VM’s network interface, before traffic leaves the VM.

The practical effect is that inspection happens at the source. A packet from a web server VM headed toward a database VM gets checked right where it originates, not at a central appliance somewhere else on the network. No round trip to a dedicated device. Enforcement scales automatically because it lives on every host alongside the workloads it protects.

Calico’s enforcement model is built on the same principle. On each Kubernetes node, Calico enforces policy at every workload’s network interface using eBPF or one of the Linux netfilter-based dataplanes (iptables or nftables). The mechanism differs from VMware’s kernel module, but the position in the stack is the same: enforcement happens before traffic enters or leaves the workload, not at a central inspection point somewhere else in the cluster.

eBPF is worth a brief note here. It’s a way of running purpose-built programs inside the Linux kernel without modifying the kernel itself, which means packet inspection and policy decisions happen at a very low level with very little overhead. It’s the same instinct that makes NSX DFW fast: get enforcement as close to the wire as possible.

The solution scales in the same way as well. Adding a node to the cluster adds enforcement capacity automatically because the enforcer runs on the node rather than as a separate service that all traffic routes through. Enforcement is distributed across every node, so there’s no central chokepoint that the whole cluster’s traffic has to pass through.

Workload-attached Policy

Policies follow the workloads they secure

When a VM migrates between hosts via vMotion, its firewall rules move with it. The policy is attached to the workload, not the host. This is a consequential architectural choice: you can’t accidentally leave a workload unprotected by moving it to a new host with different configurations in place.

It also exposes a problem with IP-based firewall rules that becomes obvious the moment workloads start moving. Rules written against specific addresses are tied to a network location, not the workload itself. If a VM changes address after migration, those rules no longer cover it. NSX DFW avoids this by anchoring policy to security group membership instead, and security groups can be built around tags rather than addresses. A VM tagged “web-tier” carries that group membership with it regardless of which host it lands on or what IP it holds. The rule that applies to web-tier VMs follows automatically, with nothing to update.

This is exactly how Kubernetes and Calico network policies also work. Rules are declared against label selectors. When a workload moves to a different node, its labels move with it and the policy selector picks it up automatically. Nothing IP-specific lives in the policy, so there’s nothing to update when an address changes. A VM labeled env: production and tier: database picks up the right policies wherever it lands. No external mechanism needs to track it.

Tiered Precedence

Microsegmentation allows for granular control and a separation of concerns

NSX also has a tiered policy model. Rules are organized into named categories with a fixed evaluation order. Emergency and quarantine rules sit at the top, infrastructure and shared service rules in the middle, and application-level rules at the bottom. Traffic is evaluated against each category in sequence, top down. A deny rule in the highest category stops evaluation immediately. Nothing below it applies.

The category structure is also where access control is enforced. NSX’s role-based access control can be scoped to specific policy categories. The evaluation order and the permission model are designed to align. Teams with the narrowest access work in the categories that run last, and no rule they write can override a higher category.

Calico tiers are designed according to the same principles. Each tier is a Kubernetes resource with an order field, a number that determines where it sits in the evaluation sequence. Lower numbers run first. A tier with order 100 is evaluated before a tier with order 200, and a deny rule in the first tier cannot be overridden by anything in the second.

Unlike NSX’s fixed category names, Calico tiers are user-defined. You name them to match your own operational model, such as “security,” “platform,” “application”, and you set the ordering. You can define as many tiers as your use cases require. Most teams structure their tiers so the security team’s policies run first, platform or infrastructure policies run second, and application or namespace-level policies run last.

Within each tier, individual policies also carry an order field, and within each policy rules are evaluated top to bottom with the first match applying. One useful behaviour worth knowing about is the pass action: a rule can explicitly pass traffic to the next tier rather than allowing or denying it. This lets each tier focus on what it’s responsible for and hand off everything else cleanly, rather than requiring every tier to have an opinion on every packet. The security tier, for example, should not have to worry about whether a specific frontend should be able to talk to a certain backend service.

For access control, Calico uses Kubernetes-native RBAC. ClusterRoles and ClusterRoleBindings govern access to cluster-scoped resources like tiers and GlobalNetworkPolicy, while Roles and RoleBindings can be used for namespace-scoped NetworkPolicy. Either way, teams can be restricted to working within their own scope with no ability to reach into tiers or namespaces they don’t own.

Attribute-based Identity

Policies are not IP dependant

NSX uses tags for dynamic workload grouping. Tags are key-value pairs applied to VMs. You can set them manually through NSX Manager or via API, which means they can be applied at deployment time rather than added by hand after the fact. Security groups define their membership through tag-matching criteria, and that membership is live: any VM carrying the right tag combination joins the group immediately.

Tagging a VM as “PCI-in-scope” isn’t just a label. It’s a trigger. The moment that tag is applied, the VM joins every security group whose membership rule matches it, and every DFW policy targeting those groups immediately covers the new member. No change request, no waiting. The policy was already written; the workload just identified itself as falling within its scope.

The same logic works on decommission. When a tag is removed, the VM leaves the group and the rules that covered it stop applying. Orphaned firewall rules accumulating over years of manual changes are a genuine operational hazard in IP-based environments — tags sidestep the problem entirely because security group membership reflects the actual state of the environment, not a record of past changes nobody got around to cleaning up.

No manual list maintenance. No IP tracking. The tag is the identity.

NSX tags map to Kubernetes labels, and the mechanism works the same way. Labels are key-value pairs defined in a workload’s manifest and applied at deployment. A Calico network policy references them through a selector: compliance: pci in a label is picked up by any policy whose selector matches that expression. The moment a workload with that label comes online, it’s covered. No separate step, no group membership to update manually.

Labels can sit on pods, on namespaces, or on both, which gives you different levels of granularity. A policy that selects on a namespace label applies to everything running in that namespace. A policy that selects on a pod label is more specific. Combining the two lets you write rules like “allow traffic from any workload in the payments namespace that also carries the label tier: web” — the equivalent of multi-tag security group membership in NSX, expressed as a single selector.

The decommission property carries over too. When a workload is deleted, its labels go with it. Policies that targeted it stop applying automatically, with no cleanup step and no stale rules accumulating over time.

For endpoints that can’t carry labels such as external services, legacy VMs not yet migrated, third-party IP ranges, Calico’s NetworkSets handle the case directly. A NetworkSet is a named collection of IP addresses or CIDRs that can be referenced in a policy selector the same way a label can. Rather than repeating IP ranges across every policy that needs them, you reference the NetworkSet by name. When those ranges change, you update the NetworkSet and every policy that references it reflects the change automatically.

A demo that went sideways in the best way

A solutions engineer was running a demo for a team evaluating a VMware replacement. They were walking through how Calico enforces network policy when the customer’s senior network consultant stopped them.

This team had been running NSX-T for years. They had 19 T1 edges and 8 segments. During the walkthrough, the consultant started recognizing what he was looking at. Then he pointed out something that reframed the whole conversation. The T1 edges weren’t actually providing any isolation. The distributed firewall policy was doing all the security work.

What the consultant recognized was that his team had been operating a flat network with policy enforcement. They’d been using NSX tags to classify workloads and superset policies to establish rule precedence.

Both of those map directly to Calico constructs which surprised them. The assumption had been that moving from NSX to a Kubernetes-native stack meant leaving the policy model behind. It turned out the policy model was already there. They’d been building towards it without realizing it.

This is a common pattern. In NSX deployments where the DFW has been built out carefully, the network topology handles connectivity and the policy handles security. The T1 edges are default gateways. The actual security boundary is the DFW rule, not the segment boundary.

What works better on the other side

Once policy is running in Calico, a few things improve over the NSX baseline. Compliance monitoring runs continuously and generates audit-ready reports against frameworks like PCI DSS, HIPAA, SOC 2, and NIST. This replaces manual evidence assembly before each audit cycle. Policy changes tracked in version control produce evidence of reviewed, timestamped, and tamper-evident changes. Tiered RBAC means the security team’s rules are structurally unreachable by teams below, with compliance report access scoped so each team sees only what it owns.

What this all means in practice

The NSX distributed firewall may feel like one of the hardest components to replicate but it turns out to be the part that transfers most directly, because the architectural decisions behind it, distributed enforcement, workload identity, tiered governance, attribute-based grouping, are the same decisions Calico was built on.

The team in that demo was genuinely surprised when they recognized their own security model in the Calico policy framework. They’d spent years building a policy structure in NSX that was a close approximation of what Kubernetes-native policy looks like. They’d been moving toward this model without knowing they were doing it.

If your NSX security posture is built on tags and tiered rules, you’ve been thinking this way for a while. The migration is a translation project. The model is already there.

Everything covered here has a corresponding configuration reference in the Calico microsegmentation documentation. It walks through tier setup, label-based policy, RBAC, and compliance reporting in detail.

Read our migration guide for a comprehensive look at VM migration.

The post You’ve Been Running a Kubernetes Security Model in NSX and Didn’t Even Know It appeared first on Tigera – Creator of Calico.

The New MCP Headers Are a Gift to Gateways

Alister Baroi — Thu, 06 Aug 2026 20:52:04 +0000

In short, buried in the transport section of the MCP 2026-07-28 release candidate are three changes that matter more to infrastructure teams than to anyone else: mandatory Mcp-Method and Mcp-Name headers, cache-control-style ttlMs and cacheScope fields, and standardized W3C Trace Context propagation. Together with the stateless core, they turn MCP from a protocol that gateways had to fight into one that meets them halfway. What the headers still don’t carry: who the caller is, whether the call should be allowed, and any record that it happened.

Everyone is writing about MCP going stateless, and the coverage is deserved. No handshake, no session ID, any request can hit any server replica, round-robin load balancing just works. If you want the deep dive on what that does to protocol state, my colleague Peter is writing one.

I want to talk about the part of the release candidate that made me sit up, because I spend my days around a gateway that authorizes agent traffic. It’s three transport changes, a few paragraphs in the announcement, and it fixes a problem every MCP-aware proxy has been engineering around since Streamable HTTP shipped in the 2025-03-26 revision.

The problem: MCP was opaque at the network layer

To an HTTP intermediary, MCP traffic used to look like nothing. Every operation, whether it listed tools, read a resource, or transferred money, arrived as a POST to the same endpoint. The thing that actually mattered, the JSON-RPC method and the tool being called, sat inside the request body.

So any middlebox that wanted to treat a tools/list differently from a tools/call had one option: buffer the request, parse the JSON-RPC envelope, and make its decision from the body. That works, and it’s what serious MCP gateways do today. But it puts body parsing on the hot path for every request, including the vast majority where a cheaper answer would have sufficed. Rate-limiters, load balancers, and WAFs that can’t parse JSON-RPC at all were simply blind. Layer 7 infrastructure spent thirty years learning to route on methods and paths, and MCP hid both.

Mcp-Method and Mcp-Name: Routing without body inspection

SEP-2243 fixes this at the obvious place. The Streamable HTTP transport now requires Mcp-Method and Mcp-Name headers: Mcp-Method carries the JSON-RPC method (tools/call, resources/read, and so on), and Mcp-Name carries the operation target, such as the tool name. The announcement states the goal plainly: load balancers, gateways, and rate-limiters can route on the operation without inspecting the body.

What this unlocks, roughly in order of how quickly teams will use it:

Per-tool rate limiting. Throttle tools/call for an expensive tool without touching the cheap ones, in a stock rate-limiter that only reads headers.
Operation-aware routing. Send resources/read to read-optimized replicas, or split list-type traffic from call-type traffic entirely.
Cheap early denies. A policy gateway can reject a request for a tool that no policy could ever allow before parsing a byte of body.
Visibility in existing tooling. Access logs, metrics dashboards, and anomaly detection keyed on headers now see MCP operations instead of an undifferentiated stream of POSTs.

One design detail deserves attention: servers must reject requests where the headers and the body disagree. That’s what makes the headers usable for real decisions rather than hints. A client can’t advertise tools/list in the header and smuggle a tools/call in the body, because the terminating server will refuse it.

Still, if your gateway makes security decisions, the right way to hold this is defense in depth. The header is a claim the server will eventually verify; the body remains the truth. Fast-deny on headers is always safe, since a mismatch would have been rejected anyway. For the allow path on sensitive operations, an enforcement point should keep parsing the body, both because arguments matter (more on that below) and because “the server behind me validates the invariant my security decision depends on” is a sentence that should make any security engineer reach for their own validation.

ttlMs and cacheScope: Caching with permission

SEP-2549 gives list results and resource reads two new fields modeled on HTTP Cache-Control: ttlMs says how long the response stays fresh, and cacheScope says whether it may be shared across users.

Gateways have wanted to cache tools/list forever. Tool catalogs change rarely, agents ask for them constantly, and every wasted round trip adds latency to an agent loop that’s already slow. But caching without server guidance meant guessing, and guessing wrong across users is how one tenant sees another tenant’s tool catalog. cacheScope makes the safety question explicit, and puts the answer where it belongs: with the server that knows whether the response was personalized.

If you run a shared MCP gateway in front of internal servers, this is the difference between “we cache nothing because we can’t prove it’s safe” and an actual caching policy.

Trace context: The audit trail gets standard rails

SEP-414 documents W3C Trace Context propagation in _meta, fixing the key names for traceparent, tracestate, and baggage. Every SDK and every gateway now agrees on where trace identity lives in an MCP request.

This one reads like an observability footnote and is quietly the most important of the three for anyone who cares about accountability. Distributed tracing across agent systems mostly worked if you controlled every hop and configured every SDK the same way. The moment traffic crossed a boundary, a different SDK, a vendor’s server, someone else’s gateway, correlation broke, and your trace of “which agent triggered this tool call” ended mid-sentence.

With the key names fixed in the spec, a trace can survive the full path: agent to gateway to MCP server and back, across implementations, into any OpenTelemetry-compatible backend. For incident response, that’s the difference between “a tool call failed somewhere” and a single trace showing the agent, the gateway’s authorization decision, and the server’s execution as one causal chain. We’ve written before about why multi-hop correlation is where existing tools fall down; the spec just removed the excuse at the protocol layer.

And because the protocol is now stateless, all of this composes. A gateway no longer needs session affinity to keep a conversation coherent, so any proxy replica can handle any request, with the operation in the headers and the trace context in _meta. The 2026-07-28 revision is the first one that reads like its authors had a load balancer diagram on the wall.

What the headers still don’t carry

Here’s the boundary, and it’s the same boundary the auth-hardening post ended on. The new transport tells you what is being asked. It does not tell you three other things.

Who is asking. Mcp-Method: tools/call plus Mcp-Name: transfer_funds identifies the operation, not the caller. Agent identity has to come from somewhere outside the protocol: mTLS workload identity, a verified token, something the platform issued rather than the process claimed.

Whether it should be allowed. Headers make policy enforcement cheaper; they don’t supply the policy. “May the finance team’s reporting agent call transfer_funds with this amount” needs an engine evaluating rules against agent attributes and, for the cases that matter, the arguments in the body. The headers get you to the decision point faster. The decision is still yours.

What happened. A cacheable, traceable, header-routed request that no one records is still unaccountable. Trace context gives audit trails standard rails to run on, but something has to actually write the trail, and it can’t be the agent.

Read those three again. They are the accountability questions, and the transport section answers none of them, correctly, because a transport shouldn’t. But it is exactly the seam where an agent gateway earns its place. Lynx’s gateway sits on this seam today: every request authorized individually against Cedar policy under a SPIFFE workload identity, with the decision recorded in Agent Trail. The new headers make that architecture cheaper to run and easier to integrate, because the proxy can classify traffic before it parses it, and the trace that leaves the gateway now correlates with everything upstream and downstream by default. Protocol changes rarely hand an enforcement plane this much for free.

If you run infrastructure in front of MCP, do this

Update any body-parsing routing or rate-limiting logic to read Mcp-Method and Mcp-Name first, and keep body validation on the security-sensitive allow path.
Audit your caching layer against cacheScope semantics before enabling cross-user caching of list results. The field tells you when sharing is safe; honor it.
Wire traceparent propagation through your proxy now, while the SDKs are shipping support inside the ten-week validation window. Correlation only works if every hop plays.
Test against the release candidate before July 28. Servers rejecting header/body mismatches is a new failure mode your clients need to handle.

Key takeaways

SEP-2243 puts the MCP operation in HTTP headers, so gateways, load balancers, and rate-limiters can act on MCP traffic without parsing JSON-RPC bodies. Servers enforce header/body agreement.
SEP-2549 adds ttlMs and cacheScope, making response caching a documented contract instead of a guess, including whether cross-user sharing is safe.
SEP-414 standardizes W3C Trace Context in _meta, so traces survive multi-hop, multi-implementation agent systems end to end.
The transport now tells intermediaries what is being asked. Identity, authorization, and audit remain the gateway’s job, and the protocol just made that job considerably cheaper to do well.

The headers tell you what is being asked. Your gateway decides whether it’s allowed. Your audit trail proves what happened. The spec just handed you the first one for free; the other two are still yours to build, and they were always the hard part.

Identity, authorization, and audit are the part the transport leaves to you — and the part that’s hardest to get right. Our whitepaper, Securing AI Agents on Kubernetes using Tigera Lynx, walks through how to build all three. Read the whitepaper →

The post The New MCP Headers Are a Gift to Gateways appeared first on Tigera – Creator of Calico.

VM Migration – What Happens to Your NSX Segments in Kubernetes?

Alister Baroi — Wed, 05 Aug 2026 19:39:40 +0000

Planning a migration off NSX usually starts with a networking conversation. Segments, VLANs, routing topology and BGP peering are not things that map cleanly to Kubernetes-native constructs the way the NSX distributed firewall maps to Calico’s tiered microsegmentation. NSX virtualizes the network layer in ways that Kubernetes doesn’t replicate by default. There is no native concept of a Layer 2 segment or VLAN, for instance. Pods simply receive IP addresses on a flat, routed network, with no built-in way to give a workload L2 adjacency to external devices or attach it to a specific broadcast domain.

This is usually where teams start to worry. They can see exactly what NSX is doing for them, but they have no obvious Kubernetes equivalent to point at. The natural question becomes how they will run the networking they depend on once their VMs live in a cluster.

Achieving the same routing, isolation, and connectivity outcomes, however, is well within reach. It just requires a bit of a mental shift.

The rest of this blog will cover the details of what that mental shift entails.

Each NSX construct and the Calico building block that takes over its job.

How NSX handles segments

Before we get into recreating NSX networking outcomes for VMs in Kubernetes, let’s set the stage by looking at what it is we are trying to recreate.

NSX is a network virtualization platform that creates software-defined networks (switching, routing, firewalling, load balancing) in a layer that runs on top of existing physical networks. It allows you to provision and segment networks programmatically without reconfiguring switches or routers.

The building block in all of this is the segment, the logical network a workload attaches to. A segment is not one fixed thing. How it behaves depends on how it is designed. It can be backed by an overlay or tied to a physical VLAN. It reaches the rest of the network either by routing through a Tier-1 gateway or by being advertised externally through a Tier-0 gateway. The backing type is what matters most when you move workloads to Kubernetes, so the two are worth looking at on their own.

VLAN-backed segments connect to a physical VLAN tag. VMs have L2 adjacency to anything on that VLAN, including physical servers and external devices. The traffic follows the physical VLAN.

Overlay segments use encapsulation to tunnel L2 frames across a routed underlay, decoupling logical topology from physical VLANs entirely. VMs on the same overlay segment see a flat L2 domain regardless of which hosts they’re on.

Both types give VMs an IP address, a default gateway, a broadcast domain, and a network identity the rest of the infrastructure can interpret.

How each segment type maps to Calico

Overlay segments are the straightforward case. The NSX overlay model, encapsulating L2 traffic and carrying it across a routed underlay, is the same architectural pattern Calico uses for its default pod network, with VXLAN or IP-in-IP as the encapsulation format. VMs and pods communicate across nodes as if they shared a flat L2 domain, the physical network carries UDP, and the details are handled below the application layer. For workloads that were on NSX overlay segments and have no hard dependencies on specific VLANs or IP ranges, the migration to Calico pod networking is largely mechanical. There is nothing to configure specifically for the overlay. It is the default.

There is one real difference behind that smooth mapping, and it is worth understanding before you migrate. An NSX overlay segment gives workloads a true Layer 2 domain. Calico’s pod network looks flat to the application, but traffic between nodes is actually routed at Layer 3. For almost everything, that distinction never surfaces. The exception is any application that leans on Layer 2 broadcast or multicast to discover its peers, such as some older clustering protocols. Those are the ones to identify up front rather than after the move.

VLAN-backed segments are where most migration discussions get complicated, and they are often the most common case, because moving off them takes time and effort that teams under pressure rarely have. A VLAN-backed segment is not just a network assignment. It often carries compliance scope, upstream firewall rules, and IP ranges that legacy systems have relied on for years. Telling a security team that the PCI VLAN is going away requires evidence that the workload’s regulatory scope is preserved by some other mechanism. Telling an external system to update its allowlist requires coordination and change windows. These are organizational constraints as much as technical ones.

Calico’s answer for environments where VMs need to stay on their VLANs is L2 Bridge mode, configured through a Calico Network resource. It stretches your existing VLAN into the Kubernetes cluster the VM now runs in, so that as far as your network is concerned, the VM is still in its old home with all its dependencies intact. Calico creates a bridge on each cluster node and connects it to the VLAN trunk on the node’s NIC, then attaches the VM to that bridge through an additional interface. The VM sees the same VLAN it was on before the migration. It keeps its IP address. It keeps its MAC address. From the perspective of the upstream switch, the firewall, and any external system talking to that VM, nothing moved.

The implementation is different but the outcome is the same. Your migrated VM runs on the same VLAN it always did, with the added advantage of now treating your infrastructure as code, with all the version control and management consistency that brings.

A VLAN-backed segment becomes a Calico L2 Bridge, so the VM keeps its VLAN, IP, and MAC.

What about my subnets?

NSX has something called a Tier-1 gateway. It is not a subnet, though it is the closest thing to the gateway behind one. It is the routing tier that segments attach to, and it does a few jobs: it provides the default gateway for workloads on those segments, routes between them and upward toward the Tier-0, keeps tenants isolated at the routing layer, and can hand a workload a fixed source IP via SNAT for allowlisting.

In Calico, none of this is a single appliance. The IP ranges those segments carried map to the IPPool, the custom resource that defines the CIDRs workloads draw from. The default gateway needs no equivalent: every node already routes for the workloads it hosts, so the T1’s routing simply happens on the nodes. Connectivity between workloads and out to the physical network rides Calico’s own routing instead of a gateway in the path.

Route isolation, where a T1 keeps one tenant’s routes out of another’s, is handled in Calico Enterprise by Multi-VRF, implemented through a custom resource called ExternalNetworks. Each ExternalNetwork is an independent routing table on the node with its own BGP peering, so routes in one domain stay invisible to another. It is the tool for cases a single routing table cannot handle, such as reaching external services with overlapping address ranges.

So the Tier-1 equivalent is not one thing you deploy. It is a combination of an IPPool, per-node routing, Multi-VRF where isolation is required, and policy. That is a real shift in thinking, and it works in your favor: no appliance to size, patch, or treat as a single point of failure, routing that scales with the cluster, and every piece a declarative, version-controlled Kubernetes resource rather than a separate box to keep in sync.

Connecting a segment to the outside world

The NSX Tier-0 gateway is the border between the NSX overlay and the physical network. Its main job is BGP peering with the top-of-rack switches, advertising the overlay subnets into the physical routing fabric so the rest of the data center knows how to reach NSX-hosted workloads. It is also the exit point for traffic leaving the fabric entirely. The T0 usually runs on a dedicated edge node or edge cluster, often as an active and standby pair, which makes it a concentration point for all north-south traffic. NAT and edge firewalling are T0 functions too, and those can be saved for a follow-up post.

In Calico, that routing role moves directly onto the cluster nodes. Each node runs a BGP daemon (BIRD) and peers with the physical switching infrastructure through the BGPPeer custom resource, which specifies the peer address, the AS number, and, through a node selector, which nodes participate. Calico advertises pod and VM CIDRs into the fabric exactly as the T0 advertised the overlay subnets. The switches learn routes to cluster workloads and forward to them with no extra configuration on the physical side.

At scale you do not peer every node with every switch. Calico supports BGP route reflectors, where a small set of nodes hold the peerings and the rest peer with them. This keeps the number of sessions manageable as the cluster grows. Either way, the T0 appliance goes away and the function it performed is distributed across the nodes.

As with the Tier-1, this is a shift from a box to a behavior, and the trade works in your favor. There is no edge appliance pair to size or fail over, north-south routing no longer funnels through a single concentration point, and capacity scales as you add nodes. The peering itself is declarative which again means that the BGPPeer resource lives in version control and is reviewed like any other part of the cluster.

The Tier-0 BGP and the Tier-1 routing distribute across the cluster nodes.

What this means in practice

The networking layer is where a migration off NSX looks most different. The segment topology, the gateway appliances, and the BGP configuration do not have one-to-one equivalents on the Kubernetes side, and the first look can be unsettling.

What carries over is the logic underneath. Overlay segments become Calico pod networking with no special configuration. VLAN-backed segments map to L2 Bridge Networks, so a VM keeps its VLAN, its IP, and its MAC through the move. The T0’s BGP peering moves to the cluster nodes through the BGPPeer resource, and the T1’s routing is absorbed into per-node routing with nothing dedicated to replace it. Where separate routing tables are genuinely needed, Calico Enterprise’s Multi-VRF provides that isolation without a gateway appliance.

The shape changes but the outcomes do not. Once you stop looking for the appliance and start looking at what it was actually doing, most NSX topologies carry fewer hard dependencies than they first appear, and each of those jobs has a Kubernetes-native home that is declarative, version-controlled, and scales with the cluster. The practical next step is to map your own environment the same way. List what each segment and gateway is really providing. Once that list exists, the Calico equivalents are usually easy to identify. That requirements audit is where most migrations should start, and it is a good conversation to have before the first VM moves.

If you want to go further, check our our ebook: Calico Enterprise for NSX Administrators. And when you are ready to try the mapping yourself, you can request a Calico Enterprise trial license.

Ready to migrate VM-centric architectures to Kubernetes platforms? Read our ebook, Calico Enterprise for NSX Administrators.

The post VM Migration – What Happens to Your NSX Segments in Kubernetes? appeared first on Tigera – Creator of Calico.

MCP’s Auth Hardening: What the Six New OAuth SEPs Fix, and What They Still Don’t

Alister Baroi — Tue, 28 Jul 2026 14:20:20 +0000

In short, the MCP 2026-07-28 release candidate is getting attention for going stateless. The quieter story is a package of six SEPs that harden the protocol’s OAuth layer: issuer validation, credential binding, client type declaration, and cleanups around refresh tokens, scopes, and discovery. All six are worth shipping, and all six fix real failure modes. But they harden how a client authenticates to a server, and that was never the whole problem. Agent identity, per-request authorization, delegation, and audit still sit outside the spec. Which means they still sit with you.

The stateless core is soaking up most of the commentary on the new MCP release candidate, and fair enough: deleting the initialize handshake and the session ID changes how everyone deploys. But scroll past that section of the announcement and you hit six SEPs of authorization hardening that almost nobody is writing about. That’s a mistake. If you operate MCP servers that hold real credentials, this is the part of the spec that decides whether a confused client hands a token to the wrong party.

The final spec ships July 28, 2026. The release candidate was locked on May 21, and SDK maintainers are in a ten-week validation window now. So this is a good moment to read the auth changes carefully: late enough that they’re stable, early enough to fix your implementation before the ecosystem expects it.

Why MCP keeps having to fix auth

MCP inverts the deployment shape OAuth grew up with. Classic OAuth has many clients talking to one authorization server: thousands of apps, one identity provider, one token issuer. MCP runs the other way around. One client, the host application, talks to many servers, each potentially fronted by a different authorization server, discovered at runtime, and often registered on the fly through Dynamic Client Registration.

That inversion is where the bugs live. The spec authors say it directly: the issuer validation SEP targets “a class of mix-up attack that is more prevalent in MCP’s single-client, many-server deployment pattern.” When your client holds registrations with a dozen authorization servers at once, an attacker doesn’t need to break the crypto. They need to get your client to attribute a response to the wrong server.

This isn’t the first pass at the problem. The 2025-06-18 revision made Resource Indicators (RFC 8707) mandatory so tokens are minted for one specific server rather than floating around as bearer credentials any resource might accept. The 2026-07-28 package continues that trajectory: less trust by default, more explicit binding.

The six SEPs, grouped by what they protect

The release notes list six SEPs. They cluster into three jobs.

Knowing which server you’re actually talking to

SEP-2468 brings RFC 9207 into MCP: authorization servers include an explicit iss parameter in authorization responses, and clients that see it must validate it. If the issuer in the response doesn’t match the issuer the client thinks it’s mid-flow with, the client rejects the response. That closes the mix-up attack above. Today including the parameter is recommended rather than required, but the spec is explicit that in a future version clients will be expected to reject responses that omit iss, so treat it as mandatory in anything you build now.

SEP-2352 fixes the other half of the same confusion. Clients must keep separate registration state per authorization server and bind each registered credential to the issuing server’s issuer value. If a resource migrates from one authorization server to another, the client re-registers rather than replaying credentials issued by the old server against the new one. Before this, a client could quietly present a client ID minted by issuer A to issuer B, and depending on the server’s tolerance, sometimes it worked. “Sometimes it worked” is not a property you want in an auth system.

Telling the authorization server what kind of client you are

SEP-837 is small and will quietly fix a failure everyone building a local MCP client has hit. Clients now declare their OpenID Connect application_type during Dynamic Client Registration. Without it, authorization servers commonly default a desktop or CLI client to web and then reject its localhost redirect URI, because web clients aren’t supposed to redirect to localhost. The result was a class of registration failures that looked like server bugs but were really a missing field. If you’ve ever watched a CLI tool fail OAuth registration against a strict identity provider for no visible reason, this SEP is for you.

Housekeeping that removes guesswork

The remaining three SEPs are clarifications, and clarifications in auth specs matter more than they sound. SEP-2207 documents how to request refresh tokens from OpenID Connect style authorization servers, which implementers previously handled with folklore and copy-paste. SEP-2350 pins down scope accumulation during step-up authentication, so a client that goes back for more scopes knows what happens to the ones it already has. SEP-2351 clarifies the .well-known discovery suffix behavior, which sounds cosmetic until two SDKs disagree about where the metadata document lives and interop breaks.

Here’s the package in one view:

SEP	What it requires	Failure it prevents
2468	Validate `iss` on authorization responses (RFC 9207)	Mix-up attacks across multiple authorization servers
2352	Bind registered credentials to their issuer; re-register on migration	Credential replay against the wrong authorization server
837	Declare `application_type` during Dynamic Client Registration	Desktop/CLI clients rejected over localhost redirect URIs
2207	Documented refresh token flow for OIDC-style servers	Divergent, improvised token renewal
2350	Defined scope accumulation in step-up flows	Ambiguity about previously granted scopes
2351	Clarified `.well-known` discovery suffix	Metadata discovery interop failures

If you maintain an MCP client or server, the work here is real but bounded, and the Tier 1 SDKs are expected to ship support within the validation window. Adopt it all. None of it is controversial.

What the six SEPs still don’t touch

Now the part I actually want to talk about. Read the package again and notice what every one of these SEPs has in common: they harden the exchange between one OAuth client and one authorization server. That exchange needed hardening. But if you’re responsible for a production deployment of agents, it’s worth being precise about the questions this spec revision does not answer, because the gap between “OAuth done right” and “agents governed” is where incidents happen. We mapped that gap in the accountability gap post; the new spec moves none of these boundaries.

The token authenticates the client, not the agent. In MCP’s OAuth model, the thing holding the credential is the host application. The agent, the actual decision-making process that chose to call the tool, has no identity of its own anywhere in the flow. Two hundred agents behind one host share one client identity. When a token is misused, “which client” has an answer and “which agent, on whose behalf, deciding on the basis of what” does not. Issuer binding makes the client-to-server link trustworthy; it says nothing about what’s behind the client.

Scopes are not policy. OAuth answers admission: does this client get a token for this server. It does not answer the per-request question: should this agent be allowed to call this tool, with these arguments, right now. A scope like mcp:tools is a key to the building, not a rule about what you may do inside. Real deployments need decisions at the level of “the finance team’s reporting agent may call read_ledger but never transfer_funds, and never with an amount above X.” Nothing in this spec revision, or any planned one, evaluates that.

Delegation chains vanish. Agents call agents, which call MCP servers. Each hop can be individually OAuth-clean while the chain as a whole is unaccountable: the server at the end sees a valid token from its immediate caller and nothing else. Who initiated the chain, under whose authority, through which intermediaries? The protocol doesn’t carry that provenance, and hardened issuer validation doesn’t create it.

Nobody is required to write anything down. All six SEPs make the authorization decision more trustworthy. None of them require a record of the decision. When your compliance team asks “was this tool call authorized by policy, and which policy,” a perfectly spec-compliant deployment gives you nothing. The spec is silent on audit, deliberately: it’s a protocol, not a governance framework. That’s the correct scoping decision for a protocol. It just means the governance framework has to come from somewhere else.

Fleet scale multiplies all of it. One client and many servers is the spec’s model. An enterprise runs many clients and many servers: N agents times M servers, each pair needing registration, each registration now correctly issuer-bound. With ten agents you can manage that by hand. With a hundred you have a spreadsheet nobody trusts. With a thousand, you need a registry, and the spec has no opinion about registries, or about noticing the agent that never registered at all.

Hardened plumbing still needs governance

None of this is a criticism of the SEPs. Protocol specs should be narrow, and this package does exactly what a protocol revision should: it makes the mechanics harder to get wrong. MCP without issuer validation was HTTP without certificate checking; now that hole is closing.

But the pattern to notice is that every question the spec leaves open is a question the agent could answer dishonestly if you ask the agent. Which agent are you? What are you allowed to do? Who told you to do this? The only place those answers stay trustworthy is in the environment around the agent: workload identity issued by the platform rather than claimed by the process, policy evaluated per request at a gateway the traffic can’t bypass, and an audit trail written by the infrastructure rather than the workload. That’s the argument we’ve been making since Why We Built Lynx, and it’s how Lynx is built: SPIFFE-based identity per agent, Cedar policy evaluated on every request, and a decision trail that exists whether or not the agent cooperates. The new spec makes the OAuth layer under all of that meaningfully stronger, and changes the division of labor not at all.

Key takeaways

The 2026-07-28 auth package is six SEPs: issuer validation (2468), issuer binding for credentials (2352), application_type in Dynamic Client Registration (837), plus documented refresh tokens (2207), scope accumulation (2350), and .well-known discovery (2351).
The common thread is binding: responses bound to issuers, credentials bound to issuers, clients declaring what they are. Mix-up and replay attacks in the one-client-many-servers pattern get materially harder.
Adopt all of it now. iss validation is heading toward mandatory, and SDK support lands within the ten-week validation window.
Be clear-eyed about scope: this hardens client-to-server authentication. Agent identity, per-request authorization, delegation provenance, and audit remain outside the protocol, by design.
Those four gaps are the governance layer. If your plan for them is “the spec will get there eventually,” it won’t, and it shouldn’t. Protocols carry messages; platforms enforce policy.

So update your SDKs, validate iss, bind your registrations. Then ask the question the spec was right not to answer. When a token you issued gets used for a tool call you’d never have approved, presented by an agent you can’t name, who catches it, and where is the record? (Five Principles of an Accountable AI Agent Network is the checklist for evaluating whatever answers you get.)

The spec authenticates the client. Someone still has to govern the agent.

The same release candidate also quietly handed infrastructure teams the best transport upgrade MCP has had. That story deserves its own post, and it gets one soon. In the meantime, the whitepaper Securing AI Agents on Kubernetes using Tigera Lynx is the long-form version of the argument this post ends on: what identity, policy, and audit look like when the environment enforces them instead of the agent.

Read our whitepaper, Securing AI Agents on Kubernetes using Tigera Lynx

The post MCP’s Auth Hardening: What the Six New OAuth SEPs Fix, and What They Still Don’t appeared first on Tigera – Creator of Calico.

Bring Your Own VLAN: Moving VMs to Kubernetes Without Changing a Single IP

Alister Baroi — Tue, 28 Jul 2026 13:55:10 +0000

For many organizations, modernizing their VMs before migrating them is not a realistic option, especially when external events trigger the migration. Mapping dependencies and refactoring network configurations before the deadline is impractical, forcing VMs to move as they are.

The mechanics of moving a VM are largely solved. Tools like Forklift handle what a vSphere admin would recognize as a cold or warm migration: copy the VMDKs off the datastore, convert the guest, and boot it as a KubeVirt VM on Kubernetes. The guest comes through with its disks, its OS, its MAC address, and the static IP still written in its network configuration.

Recreating the NSX segment the vNIC was attached to, the VLAN that defined the VM’s compliance scope, or the firewall rules that reference its address is a different story. The VM arrives in a cluster that knows nothing about any of it. Everything NSX was doing for that VM now has to be rebuilt on the Kubernetes side.

Kubernetes networking cannot solve this on its own, for two reasons. First, pod IPs are assigned dynamically from the cluster’s pod CIDR, and a KubeVirt VM attached to the pod network is treated like any other workload, meaning that it gets a fresh address on arrival and again whenever it reschedules, so the static IP in its configuration file is unusable. Second, the pod network is routed. The model has no concept of a VLAN, a broadcast domain, or a persistent MAC address. A VM whose compliance scope or firewall rules depend on VLAN membership needs L2 semantics that the standard Kubernetes stack was never designed to carry.

Migrating a VM with default Kubernetes networking breaks dependencies

L2 vs L3: why the mismatch matters

Traditional VM networking is rooted in L2. A VM on a VLAN-backed segment sits in a broadcast domain: when it needs to reach anything outside its subnet, it broadcasts an ARP request for its default gateway, and the gateway answers with its MAC address because they share that domain. The upstream switch learns the VM’s own MAC the same way, and its static IP belongs to the VLAN’s subnet.

Two VMs on the same segment talk to each other without a router ever being involved. The pod network offers none of this, not because Kubernetes is deficient but because it was scoped for workloads designed for a routed network from the start. Running an L2-dependent VM in a cluster means extending L2 into the cluster explicitly, and that is where Calico’s L2 Bridge comes in.

Why VMs may need to stay on their original VLAN

The short answer is that the VLAN is rarely just a VLAN. Over time it accumulates dependencies that make changing it expensive in ways that have nothing to do with technology.

The most common is compliance scope. In PCI DSS environments, the cardholder data environment is often defined by VLAN membership. The audit evidence says “these workloads are in scope because they are on VLAN 200.” Moving a workload off that VLAN and redefining scope through a different mechanism, such as labels, policy selectors, and egress controls, is possible, but it requires re-engaging the auditor, updating the scoping documentation, and getting sign-off before the workload moves. That process runs on a different timeline than a migration project.

Upstream firewall rules are the other major dependency. Perimeter firewalls and network ACLs are typically written against subnets and VLAN ranges, not individual IPs. A rule that permits traffic from 10.0.2.0/24 to the payment processor is a rule about VLAN 200. Changing the workload’s IP or VLAN means finding every rule that references that range, across every firewall that sits between that workload and anything it talks to, and updating them in coordination. Each of those updates requires a scheduled change window and with it the usual requests, approvals, and rollback plans.

Beyond compliance and firewalls, there are the smaller dependencies that pile up over years of operations. Monitoring systems with hardcoded IPs, application configuration files that reference addresses directly, license servers that are locked to a specific IP, and external SaaS integrations that have the service IP on an allowlist are just a few examples. None of these are hard to fix individually. Together they represent enough coordination overhead that “just keep the IP” is often the right engineering decision, not a workaround.

How Calico preserves VLAN-backed segment outcomes

The three categories of dependency from the previous section all trace back to the same requirement: the VM needs to keep its IP address, its VLAN membership, and its MAC address after it moves. If those three things are preserved, nothing upstream needs to change.

Calico L2 Bridge Networks preserve all three. Instead of attaching the VM to the routed pod network, a Network resource tells Calico to connect the VM’s vNIC to a Linux bridge on the cluster node. The bridge is easiest to understand as the node’s vSwitch: the physical NIC is the uplink, and a VLAN sub-interface on that NIC plays the role of the tagged port group that selects the VM’s VLAN out of the trunk.

The VM connects to its old VLAN with a bridge

There is one physical prerequisite. The node’s NIC must be plugged into a switch trunk port that carries the VM’s VLAN, just as an ESXi host’s uplinks must be trunked for a VLAN-backed segment to work. If VLAN 200 has not been extended to the Kubernetes nodes, there is no broadcast domain for the VM to join.

With that in place, the VM sits in the same broadcast domain it occupied on ESXi. It ARPs for its gateway and the same physical gateway answers. The auditor still sees the workload on VLAN 200, the firewall still sees traffic from the same IP and subnet, and the license server, the monitoring system, and the application’s own configuration all still point at an address that hasn’t changed. The only real difference is that the VM now runs on a Kubernetes node instead of an ESXi host, with its network attachment declared as a Kubernetes resource instead of an NSX segment. The next section covers how the bridge is actually constructed.

How Calico L2 Bridge works

KubeVirt runs each VM inside a pod. When that pod is created, the cluster assigns it an IP from the pod CIDR like any other pod. The VM then boots inside it, and its virtual NIC is normally wired into the pod’s network namespace, so the VM inherits the pod’s identity: a cluster-assigned address on a routed network where its original IP means nothing. Calico L2 Bridge Networks change that wiring step: instead of connecting the vNIC to the pod network interface, Calico connects it to a Linux bridge on the node.

When a Network resource defines an L2 bridge network and associates it with a VLAN, Calico sets up a Linux bridge on the cluster node. That bridge has an uplink to the VLAN sub-interface on the node’s physical NIC, which gives it access to the trunk the node is already connected to. The VM’s virtual NIC is attached to that bridge through a tap interface, the Linux equivalent of the vSwitch port a vNIC plugs into on ESXi. The result is a direct L2 path from the VM to the physical switching fabric: VM tap device, through the Linux bridge, through the VLAN sub-interface, out the physical NIC, into the same broadcast domain the VM was on before.

The upstream switch learns the VM’s MAC address the same way it always has, by watching frames arrive on the VLAN. From the switch’s perspective, a new port came up carrying a MAC it already knew. Nothing in the physical network needs to be reconfigured.

This is the plumbing a migration plan needs to account for, and in practice it comes down to a short checklist. Before the move, record the VM’s VLAN and IP address. Create a Network resource referencing that VLAN, and point the VM’s KubeVirt definition at it by name. When the VM starts on its new node, Calico builds the bridge, the guest boots with the network configuration it has always had, and traffic flows on the same VLAN as before. As far as the VM knows, it’s still where it always was.

Keep the network, move the machine

None of the dependencies this blog started with have to be rewired when your VM moves to a Kubernetes cluster. The auditor’s scoping document, the perimeter firewall rules, the license server record, the hardcoded addresses in a decade of operational tooling: all of it still holds, because the VM still has the IP, the MAC address, and the VLAN membership they were written against. The migration moves the machine without touching the network identity everything else depends on.

The VM keeps its place on the VLAN and its dependencies

There is a broader point here that goes beyond any single VM. Because the VLAN extends into the cluster, ESXi and Kubernetes stop being two separate worlds with a hard cutover between them. During a migration they are just two sets of hosts attached to the same networks, which means you can move one VM this week, ten next month, and leave the stubborn ones where they are for now. The workloads that have moved keep talking to the ones that haven’t, and if something goes wrong, rolling back is a matter of booting the VM on the other side again. That changes what an NSX team is actually signing up for. The migration stops being a leap to an unfamiliar platform and becomes a gradual shift of workloads across a network you still recognize and still control, with the Kubernetes learning curve spread over months instead of compressed into a cutover weekend.

If a phased migration like that is what you are planning, read our ebook: Calico Enterprise for NSX Administrators. When you are ready to try Calico yourself, you can get started for free.

Ready to migrate VM-centric architectures to Kubernetes platforms? Read our ebook, Calico Enterprise for NSX Administrators.

The post Bring Your Own VLAN: Moving VMs to Kubernetes Without Changing a Single IP appeared first on Tigera – Creator of Calico.

Why eBPF Is Useful for Watching and Sandboxing AI Agents

Alister Baroi — Mon, 27 Jul 2026 20:04:42 +0000

Most of our runtime security habits were built for deterministic workloads. A service does what its code says: review the code, sign the image, and its behavior is bounded. Agents are different. An agent’s behavior emerges from a model reasoning over whatever lands in its context window, and some of that context comes from places we don’t fully control — a retrieved document, a tool’s output, a user’s prompt. Meanwhile the agent usually runs with real privileges: a service account, network reach, mounted secrets, a filesystem. When untrusted input shapes behavior, those privileges get exercised less predictably than we’re used to.

A lot of good work goes into making agents harder to mislead — prompt hygiene, injection classifiers, guardrail models. It’s worth pairing that with a second question: if an agent does something we didn’t intend, how far can it actually reach? That’s blast radius, and it’s mostly a decision we make at the runtime layer, independent of how the prompt was handled.

This is where eBPF is a good fit, for two reasons: it’s an excellent way to see what an agent is doing, and it can enforce limits on what the agent can touch — without changing the agent.

Most of what an agent does is a syscall

The reason the kernel is a natural place to observe and constrain an agent is that nearly everything an agent does — reading a file, running a command, opening a connection — resolves to a syscall. eBPF attaches to the kernel, beneath the workload, so it can watch those syscalls directly:

File activity — via LSM hooks (BPF-LSM / KRSI) or kprobes on openat, read, write. You can see exactly what an agent reads and writes, including attempts to reach paths like /var/run/secrets.
Process activity — via sched_process_exec and LSM bprm_* hooks. You can see every binary an agent tries to execute, including an unexpected shell.
Network activity — via cgroup connect hooks, tc/XDP, and DNS visibility. You can see where an agent is trying to send traffic, and whether it’s trying to bypass your model gateway.

That alone is valuable: a clear, tamper-resistant record of what your agents actually do at runtime, without instrumenting them.

From observing to enforcing

eBPF can also act inline — overriding a syscall or stopping a process when a policy is violated. That means the same layer supports a natural progression from observe → alert → enforce , and you can move an agent along that path without redeploying it.

In practice, that gives you sandbox-like controls expressed as simple rules about what a process may touch:

Allow reads under /workspace but keep writes off for a read-only agent, and keep secrets out of reach entirely.
Pin an agent to an allow-list of binaries, or block exec for agents that should never spawn a shell.
Require egress to flow through your model gateway, and block anything trying to go direct.

Two properties make this practical at scale. First, because eBPF sits beneath the workload, it applies to pods that are already running and agents you haven’t modified — no recompile, no sidecar, no new base image. That’s a real advantage when you already have agents in production. Second, it runs in a different trust domain than the workload, so it’s hard for the agent to tamper with, and the overhead is low enough to run fleet-wide.

What eBPF is not for

eBPF is well suited to blast-radius questions — what a process may touch — rather than deep content inspection. It can see that an agent is opening a connection or writing a path and act on that, but it can’t understand the semantics of a prompt or redact fields inside an encrypted request body. That kind of work belongs at a TLS-terminating proxy at the model gateway that can do semantic checks on the agent’s intent and actions. eBPF helps contain what an agent can do; a proxy governs what it sends. They’re most useful together, and eBPF is happiest as a uniform floor beneath whatever higher-fidelity, application-aware controls you already run.

The short version

Agents behave less predictably than the workloads our tooling was designed for, so it’s worth planning for the moments when one does something unintended. eBPF gives you two things that can help: visibility into what agents are actually doing at the syscall level, and a tamper-resistant way to lock them down, across agents you never had to change or redeploy.

Want to see what this looks like in practice on Kubernetes? Get the ebook: Securing AI Agents on Kubernetes using Tigera Lynx.

The post Why eBPF Is Useful for Watching and Sandboxing AI Agents appeared first on Tigera – Creator of Calico.

Migrate First, Modernize Later: A Leadership Guide to Converging VMs and Containers to Run AI Workloads

Alister Baroi — Fri, 24 Jul 2026 21:32:47 +0000

The Tipping Point

Every so often the ground under enterprise IT moves. It’s moving now. Across industries, organizations are consolidating fragmented infrastructure onto a single, self-hosted platform capable of running both containers and virtual machines side by side. The motivation is simple: simplify operations, lower cost and reallocate resources & budget to AI initiatives. Kubernetes is emerging as the primary platform for many of these workloads.

For most IT leaders, the compute and storage portions of a VM migration are manageable. Storage arrays and hypervisor CPU/memory allocation translate fairly directly to Kubernetes equivalents. Networking is where migration plans stall. A VM’s network identity — its IP, its VLAN membership, its firewall rules — is wired into surrounding infrastructure, monitoring, compliance controls, and business processes that nobody wants to touch during a migration window.

Teams accustomed to NSX for this work find that native Kubernetes networking wasn’t built with VM administrators in mind, and the functionality gap becomes the reason migration projects get bigger or are stalled. If the networking problem is solved — if a VM can move to Kubernetes and keep its IP, its policy, and its security posture intact — then the rest of the platform consolidation stops being an expensive, multi-year architectural bet.

The Power of Convergence

eBPF-powered Calico unified platform’s value has always been convergence and portability — collapsing separate networking domains into a platform and enabling customers to avoid vendor lock-in by platform vendors.

One Networking and Security Model for Any Kubernetes Distribution, Any Workload, Anywhere

Calico Enterprise was developed to deliver three objectives for enterprise networking & security in Kubernetes in a single unified platform to enable enterprises to be ready for hosting AI workloads. Now with the launch of Calico for VMs on Kubernetes, Calico delivers a fourth objective: support for VMs & Containers on Kubernetes.

All the components required for k8s networking and network security in a single unified platform – Built on the most trusted open-source technologies in Kubernetes — Calico Open Source, Istio, Envoy, and eBPF — the Calico platform gives platform engineering teams a single management plane to enforce, observe, and troubleshoot all workload communication
Unified platform works across any Kubernetes distribution – Calico is Kubernetes agnostic, and supports all major distributions equally: AKS, EKS, GKE, VMware VKS, RedHat OpenShift, Canonical, SUSE, Mirantis and others. That matters strategically for a VM migration decision: it means the move of workloads from a legacy hypervisor to Kubernetes doesn’t lock the organization into a single Kubernetes vendor.
Unified platform works across infrastructure – Calico extended one unified model across on-premises, cloud environments and edge, so connectivity, security, and observability behave identically no matter where a cluster runs.
Unified platform works across different types of workloads – Containers & VMs – Calico enables virtual machines and containers to share one networking fabric, one policy model, and one observability plane. Seamless VM migration while maintaining L2 networking, and a single control plane for ease of management, regardless of the workload, VMs or containers.

Benefits of Adopting a Modern Networking Architecture

Migrate First, Modernize Later

The single most important idea for a migration plan to succeed is separating “get off legacy” from “redesign the network and workloads”. Trying to do both at once is what turns a migration into a multi-year modernization program. Calico’s L2 bridge capability extends existing VLANs into Kubernetes, so a VM can move to a Kubernetes cluster on day one without renumbering, without rewriting firewall rules, and without breaking the hard-coded dependencies — DNS records, monitoring agents, compliance scans — that assume a specific IP or subnet. Once workloads are safely on Kubernetes, the network architecture can evolve on its own timeline — moving from an L2, VLAN-based design to a Kubernetes-native L3 design when the team is ready, not because the migration forced the issue.

A complete stack for VM networking on Kubernetes

Every capability and outcome delivered by NSX has a direct Kubernetes-native counterpart in Calico:

Connect – Calico Networks provide connectivity to VM workloads and to the networks and services around them. L2 Bridge capabilities can extend existing network VLANs (Segments) into Kubernetes for workloads that require Layer 2 or network continuity during and after migration. BGP-based routing, egress gateway, load balancing and ingress gateway functions support delivering applications and services to consumers.
Secure – Calico network policy, policy tiers, staged policy and DNS policy provide Kubernetes-native controls for access enforcement and microsegmentation. Policies can be planned, monitored and validated before enforcement, helping teams maintain security posture as workloads move and apply consistent controls across VMs and containers.
Observe – Calico Service Graph, flow logs, DNS logs, L7 visibility and packet capture provide context for troubleshooting and security operations. Teams can investigate VM-to-VM, VM-to-pod, pod-to-pod and cross-cluster flows with Kubernetes-aware workload context using eBPF-enabled deep packet inspection.

The ROI Case

The core mechanism is consolidation onto a single management plane covering network connectivity, observability, and network policy. Instead of maintaining separate stacks and separate expertise for VMs and for containers, the teams responsible for availability, performance, and security each focus on one plane, with one consistent set of tools, telemetry, and controls. Additionally, the new architecture prevents vendor lock-in by Kubernetes platform vendors.

Cost relief

The immediate driver for most organizations is reducing the costs of legacy hypervisors. Every VM that moves off legacy infrastructure onto a converged Kubernetes platform is a licensing spend that stops compounding — budget that can be redirected to AI infrastructure.

Operational efficiency

Running one policy model, one routing and observability stack, and one set of operational runbooks for both VMs and containers means fewer specialized teams, less tooling overlap, and lower mean time to resolution (MTR) when something goes wrong.

AI readiness

A converged platform is also the platform production AI workloads need. Self-hosted LLMs and the agents built on them can run alongside existing VM and container workloads, close to the data they need, with the performance, latency, and scale that production AI demands.

Conclusion

The market is converging on one self-hosted platform for containers and VMs, and the economics and AI trends driving it are only accelerating. The AI-driven need for a converged, self-hosted platform is the reason to move to Kubernetes specifically, rather than to another hypervisor. Tigera already secures workloads across more than a million clusters for organizations including NVIDIA, Royal Bank of Canada, Bloomberg, Chipotle, GoDaddy, and Upwork.

Migrate first. Modernize later. On your timeline.

→ Learn more: Visit the VM Migration page

→ See it in action: View the self-paced overview

Ready to see Calico for VMs on Kubernetes in action? Walk through the live demo.

The post Migrate First, Modernize Later: A Leadership Guide to Converging VMs and Containers to Run AI Workloads appeared first on Tigera – Creator of Calico.

NVIDIA OpenShell Secures the Agent. Who Governs the Fleet?

Alister Baroi — Wed, 15 Jul 2026 14:41:56 +0000

Most attempts to control AI agents work at the model layer (alignment, system prompts) or the application layer (guardrail libraries, output filters). Both share a flaw: the thing being secured is also the thing doing the securing. A sufficiently confused or sufficiently compromised agent can talk its way past its own instructions.

OpenShell takes a different position, and it is the right one. Put the controls in the environment, where the agent cannot negotiate with them. An agent inside an OpenShell sandbox cannot leak a credential it never received, and cannot call an endpoint the kernel refuses to route.

If that argument sounds familiar, it should. It is the same case we made in Why We Built Lynx and throughout the AI agent accountability series: controls the agent can override are not controls. NVIDIA arriving at the same conclusion, with an Apache 2.0 project and a partner list that includes Cisco, CrowdStrike, Google Cloud, and Microsoft Security, is the strongest endorsement the environment-layer approach has had yet.

So this is not a “versus” post. OpenShell and Lynx solve different halves of the same problem, and NVIDIA said so first: its own launch announcement says securing autonomous systems “requires an integrated ecosystem”.

What OpenShell actually does

OpenShell is a secure runtime for a single agent on a single machine. You install it with one command, then launch an agent inside a sandbox:

openshell sandbox create -- claude

That agent (Claude Code, Codex, Cursor, OpenCode, or your own container image) now runs inside an isolated environment governed by a declarative YAML policy with four layers:

Filesystem: Which paths the agent can read or write, enforced with Landlock and locked at sandbox creation.
Process: Which binaries can execute and which syscalls are available, enforced with seccomp. An agent can install a verified skill but cannot run an unreviewed binary.
Network: deny-by-default egress, intercepted at the HTTP method and path level, hot-reloadable as approvals are granted.
Inference: A “Privacy Router” that decides which LLM backend serves each call, keeping sensitive context on local models and routing to frontier models only when policy allows. Credentials are swapped at the router, so the real API key never sits inside the sandbox.

The threat model is specific and well chosen: long-running, self-evolving agents with shell access, live credentials, and the ability to rewrite their own code. Prompt injection, malicious third-party skills, subagents inheriting permissions they should not have. When the agent hits a policy wall, it can propose a policy change and a human approves or rejects it. Autonomy with a human holding the pen.

It is currently alpha (“proof of life,” in NVIDIA’s words), runs on macOS, Windows via WSL 2, and Linux, and targets everything from a developer laptop to DGX-class machines.

Where OpenShell stops, on purpose

Here is the part that matters for anyone running agents in production. NVIDIA’s technical documentation is explicit about what OpenShell does not address:

Agent-to-agent communication governance
Agent identity and authentication
Cross-sandbox communication patterns

Kubernetes is the near-miss on that list. OpenShell does run on Kubernetes: an experimental Helm chart, marked not for production, that provisions sandbox pods on a cluster. But putting sandboxes on Kubernetes and governing a fleet across Kubernetes are different jobs. Each sandbox still enforces its own YAML in isolation, with no shared agent identity and no view of its neighbors.

Read that list again. It is not a gap NVIDIA missed; it is a boundary they drew deliberately, and they drew it exactly where the fleet begins. OpenShell answers “what can this agent do on this box?” It does not attempt to answer “which of my two hundred agents called the payments MCP server last Tuesday, under whose authority, and using which model?” And as we argued in The AI Agent Accountability Gap, network policies, API gateways, and RBAC cannot answer those questions either.

They are the questions Lynx exists for. Side by side:

Concern	OpenShell	Lynx
Scope	One agent, one sandbox	A fleet of agents across a cluster
Agent identity & authentication	No first-class agent identity (users and components authenticate; agents just get injected credentials)	SPIFFE/SPIRE workload identity, mTLS, per-agent JWTs
Policy	YAML per sandbox	Cedar policy across agents, MCP servers, and LLM providers
A2A and MCP traffic	Out of scope	Gateway proxy, every request authorized individually
Agents you didn’t launch	Not applicable	eBPF detection classifies them as sanctioned, shadow, or unknown
Audit	Local allow/deny logs per sandbox	Fleet-wide Agent Trail, including which model actually served each call

One box, two hundred boxes. Same philosophy, different altitude.

Closing the gap: three integration patterns

None of these require code changes in either product. They use configuration surfaces both systems document today: OpenShell’s deny-by-default egress policy and credential injection on one side, Lynx’s gateway, registry, and token service on the other. To be clear about what this is: a proposed reference architecture drawn from published documentation, not a tested walkthrough. OpenShell is weeks old and still alpha. But the seams line up well enough that I think the patterns are worth writing down now.

Pattern 1: One road out of the sandbox

OpenShell intercepts all outbound traffic and denies by default. So write the narrowest useful network policy: the only egress a sandbox is allowed is the Lynx Agent Gateway.

Every MCP call, every A2A request, every LLM call now has exactly one path, and that path runs through Cedar authorization on a per-request basis, with the decision recorded in Agent Trail. The division of labor is clean. OpenShell guarantees the agent cannot go around the gateway, even if it is compromised and actively trying. Lynx decides what is allowed through the gateway, and remembers what happened.

Neither system can do the other’s job here. Lynx cannot stop a process inside someone’s laptop sandbox from opening a raw connection; OpenShell can. OpenShell has no idea whether this agent should be allowed to call that MCP tool with those arguments, but Lynx does.

Pattern 2: The API key never enters the sandbox

OpenShell’s Privacy Router already routes inference calls through controlled backends and swaps credentials on the way out. Lynx, as of the current release cycle, treats LLM providers as first-class governed entities: registered in the registry, subject to Cedar policy, visible on the access map, recorded in Agent Trail down to the model that actually served the request.

Chain them. Local-model traffic stays on the box, served by Nemotron or whatever the Privacy Router prefers. Frontier-model traffic routes to the Lynx LLM gateway, where Cedar decides which agent may use which provider and which model, and the credential is attached centrally.

Follow the key. The OpenAI or Anthropic API key exists in exactly one place, inside Lynx. Not in the sandbox, not in the agent’s environment variables, not in a dotfile the agent can read and exfiltrate. And every frontier call, from every sandbox on every developer machine, lands in one audit trail with the caller’s identity and the served model attached. A prompt-injected agent can ask for the key all it wants; there is nothing on the box to steal.

Pattern 3: Identity from birth

OpenShell deliberately focuses on securely running agents rather than defining who those agents are. It provides sandboxing, credential management, and integration with existing identity systems, but it doesn’t maintain a persistent registry of agent identities or establish a trust model between agents. Lynx complements that layer by giving every agent a verifiable identity from the moment it is created.

The integration is intentionally lightweight: a wrapper around openshell sandbox create registers the new agent with the Lynx registry and associates it with an existing workload identity; whether SPIFFE, OIDC, or another supported mechanism. From its first network request, the sandbox represents a known, authenticated agent rather than an anonymous process.

This pattern is what makes the first two enforceable per agent instead of per box, and it has a side effect worth naming. A developer’s local experiment, sandboxed with OpenShell and registered with Lynx, shows up on your access map as a sanctioned agent. The same experiment without registration is exactly the shadow agent that Lynx’s eBPF detection was built to catch. Registration at sandbox creation makes the sanctioned path the lazy path, which is the only kind of security policy developers reliably follow.

Same policy idea, from laptop to cluster

There is a deeper symmetry underneath these patterns. OpenShell’s filesystem and process layers do at sandbox scope roughly what Lynx’s agent-detector does at node scope with eBPF; its network and inference layers do locally what the Lynx gateway does for the fleet with Cedar. (Peter Kelly covered the gateway-plus-kernel enforcement model in Multi-Layer Policy for Securing AI Agents.) Nobody has built a translator between OpenShell YAML and Cedar yet. But the layers correspond closely enough that policy parity across the laptop-to-cluster boundary looks like an engineering problem, not a research problem. An agent developed under a given OpenShell policy could be promoted to Kubernetes with the same intent expressed as Cedar plus a quarantine baseline. That is the roadmap conversation this post is meant to start.

Two smaller threads point the same direction. OpenShell’s Kubernetes chart means sandboxes can run on a Lynx-governed cluster, sitting inside two independent kernel enforcement planes, one inside the sandbox and one on the node, so even a sandbox escape lands in Lynx’s detection perimeter. And OpenShell logs every allow/deny decision locally; forwarding those over OTLP into Agent Trail would put runtime decisions and traffic decisions in a single timeline. Both are speculative today. Neither is far-fetched.

The other half

OpenShell is the most credible answer yet to a question we have been asking all year: how do you give an agent real autonomy without handing it the keys to the host? If you are running coding agents locally, try it; the install is two commands and the defaults are sensible.

Then ask the question NVIDIA deliberately left open. When that agent, and the forty like it across your organization, start talking to MCP servers, to each other, and to three different LLM providers, who is checking identity at the door? Whose policy decides, and where is the record? (Five Principles of an Accountable AI Agent Network is the checklist for evaluating whatever answers you get.)

OpenShell holds the agent. Lynx governs the fleet. The seam between them is thinner than you would expect, and the patterns above are how we would stitch it.

_Lynx is Tigera’s security and governance platform for AI agents on Kubernetes: identity, policy, detection, and audit for every agent in your cluster. Read How Lynx Works or schedule a demo at tigera.io/demo/. _

Ready to see Lynx in action? Schedule a demo.

The post NVIDIA OpenShell Secures the Agent. Who Governs the Fleet? appeared first on Tigera – Creator of Calico.

Tiered Network Policy: Scaling Kubernetes Security

Alister Baroi — Fri, 10 Jul 2026 16:12:22 +0000

As Kubernetes clusters scale from a few development sandboxes to massive, multi-tenant production environments, platform teams often find themselves facing a configuration management crisis. A small number of microservices suddenly demand hundreds of individual Kubernetes NetworkPolicy objects. Managing them becomes operationally expensive, auditing them is difficult, and a single developer misconfiguration can easily drop critical production traffic or open a massive security hole.

To scale cluster security without slowing down engineering velocity, we must abandon the flat, uncoordinated rule planes of the past. The solution lies in establishing a clear, multi-layered framework: a hierarchy of trust powered by tiered network policies.

The Core Problem with Standard Kubernetes NetworkPolicy

Standard Kubernetes NetworkPolicy resources are genuinely useful for basic application microsegmentation, but they have major architectural and organizational bottlenecks when scaled across an enterprise:

Namespace-Scoped by Design: Standard network policies are inherently scoped to a namespace. If your security team mandates a cluster-wide rule, such as blocking all internal pods from querying the cloud provider’s metadata API (169.254.169.254), you have to copy-paste that policy into every single namespace. If a developer creates a new namespace, that guardrail doesn’t exist until someone manually applies it.
Organizational Friction: Because anyone with namespace access can manipulate these policies, it creates a persona gap within organizations. Platform & Security teams need to enforce global, un-overrideable guardrails (e.g. “Isolate the payments namespace from everything else”). DevOps teams need the freedom to write granular, service-to-service rules for their applications without opening infrastructure support tickets.
No Rules Hierarchy: Kubernetes network policies are strictly additive. There are no weights, priorities, or order sequences. An application developer can accidentally (or intentionally) write a loose policy that bypasses the security team’s intended restrictions, undermining any baseline trust.
The “Allow-Only” Restriction: Standard policies cannot explicitly Deny traffic. They operate solely on an allow-list model. Isolation is implicit: if a pod is selected by a policy, any traffic not explicitly allow-listed is dropped. This makes it impossible to write a simple, top-level rule that says, “Block traffic from Namespace X to Namespace Y, no matter what.”

What a Scalable Solution Requires

To solve these scaling pain points, we have to move away from a flat network architecture and adopt a Tiered Policy Model. A scalable solution requires four core capabilities:

Global, Cluster-Wide Scope: To stop copy-pasting rules, administrators need a policy type that natively operates at the cluster level rather than the namespace level. This allows a single manifest to apply to all current and future namespaces automatically, eliminating the risk of “configuration drift” and ensuring day-one protection for new workloads.
Separation of Concerns (RBAC-Gated Tiers): Security, platform, and application teams need their own distinct logical “zones” or tiers to deploy rules. These tiers must be strictly gated by Role-Based Access Control (RBAC) so a developer modifying their application namespace cannot alter or override a higher-priority platform or security tier.
Deterministic, Top-Down Evaluation: The firewall engine must evaluate these tiers sequentially. Traffic must pass through the highest-priority tier (e.g., Security) before it ever reaches a lower tier (e.g., Application).
Explicit Deny and Pass Actions: Standard policies are allow-only, so they can never express a hard “block this, period.” A tiered model needs explicit actions: a Deny that states a prohibition outright, and a third option, Pass, that lets one tier defer the decision to the next rather than ending it (covered in detail below).

Why the Pass Action Matters

The key enabler of tiered policies is the Pass action. Think of Pass as a delegated hand-off. When a packet matches a rule with a Pass action in a high-priority tier, the engine skips the remaining lower-precedence rules in that tier and continues evaluation in the next tier down the hierarchy. This allows security administrators to say: “This traffic is safe by our standards, but we aren’t explicitly endorsing it. We are passing the final decision down to the platform or development teams to handle at their layer.” Without a Pass action, tiered policies become brittle, forcing admins to explicitly track and approve every single microservice connection at the highest level, which would defeat the purpose of developer agility.

The Kubernetes Native Answer: ClusterNetworkPolicy

Recognizing these scalability constraints, the SIG-Network Policy API group developed a native, multi-layered solution: ClusterNetworkPolicy. The API delivers exactly the four capabilities outlined above, with a few concrete specifics worth calling out:

A Native Three-Layer Hierarchy: It introduces distinct, sequentially evaluated resource tiers. ClusterNetworkPolicy (Admin tier) at the top for absolute guardrails, standard NetworkPolicy in the middle for developer agility, and ClusterNetworkPolicy (Baseline tier) at the bottom as a cluster-wide fallback safety net. Unlike namespace-jailed standard policies, the Admin and Baseline tiers apply across the entire cluster.
Separation of Concerns: Because ClusterNetworkPolicy is delivered as a new Custom Resource Definition (CRD) rather than a tweak to the existing NetworkPolicy type, standard Kubernetes RBAC governs who can interact with it. This ensures that only Security/Platform teams access ClusterNetworkPolicy resources, while DevOps teams work only with namespaced network policies.
Numeric Precedence: Policies feature explicit integer priorities. A policy with a lower integer value (e.g., 10) takes precedence over a policy with a higher value (e.g., 100), allowing for deterministic evaluation.
Explicit Actions: Rules are no longer purely additive. You can now design rules with explicit Accept, Deny, and Pass actions.

This API completely shifts how cluster administrators manage traffic by introducing a native, three-tiered evaluation hierarchy:

The Top Layer: ClusterNetworkPolicy (Admin tier): This is the high-priority tier controlled by cluster and security administrators. Rules here are evaluated first, and two of its three actions are terminal: an Accept or a Deny is a final verdict that bypasses the developer’s NetworkPolicy layer entirely. A Deny here cannot be overridden by any developer manifest, but the same is true of Accept: if an admin explicitly accepts traffic, it is permitted regardless of what a developer policy would have decided. This is the crucial difference from a standard NetworkPolicy allow, which is additive. An Admin-tier Accept is an override, not a contribution. Only the third action, Pass, is non-terminal: it declines to decide and hands evaluation down to the next tier.

As an example, the following ClusterNetworkPolicy can be used to allow DNS UDP traffic toward kube-dns from all namespaces:

apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: allow-dns-to-kube-dns
spec:
  tier: Admin
  priority: 100
  subject:
    namespaces:
      matchLabels: {}
  egress:
    - name: allow-dns
      action: Accept
      to:
        - pods:
            namespaceSelector:
              matchLabels:
                kubernetes.io/metadata.name: kube-system
            podSelector:
              matchLabels:
                k8s-app: kube-dns
      protocols:
        - udp:
            destinationPort:
              number: 53

The Middle Layer: Standard NetworkPolicy: This is the traditional application-developer tier. It only kicks in if traffic wasn’t explicitly allowed or denied by the ClusterNetworkPolicy in the Admin tier above it. This keeps developers agile, letting them connect their microservices without needing admin intervention. One subtlety to keep in mind: standard NetworkPolicy carries an implicit deny for any pod it selects. So traffic only falls through to the Baseline tier when no NetworkPolicy selects the workload at all. A pod that is selected but matches none of its Accept rules is already dropped here, and never reaches the Baseline tier below. The following network policy can be used to permit ingress HTTP traffic for the awesome-app namespace.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-http-ingress
  namespace: awesome-app
spec:
  podSelector:
    matchLabels:
      app: http-server
  policyTypes:
    - Ingress
  ingress:
    - ports:
       - protocol: TCP
         port: 80

The Bottom Layer: ClusterNetworkPolicy (Baseline tier): This is the cluster-scoped Baseline tier, meant for default fallbacks. It acts as the safety net after developer policies are checked. For example, if a developer forgets to secure their pod, this policy can enforce a default cluster-wide posture like “if no developer policy matches this traffic, deny all intra-cluster traffic by default.”. The following ClusterNetworkPolicy would satisfy this requirement:

apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: deny-all
spec:
  tier: Baseline
  priority: 1
  subject:
    namespaces:
      matchLabels: {}
  ingress:
  - name: deny-all-ingress
    action: Deny
    from:
    - namespaces:
        matchLabels: {}

Combined, these features provide a native, multi-level strategy for scaling enterprise cluster security far beyond the limitations of a flat configuration.

Extending the Model: Calico Tiers

While the native Kubernetes APIs introduce a better three-layer model, and some control over rule priority, enterprise environments often require finer granularity. Calico expands on this concept by offering unlimited policy tiers, allowing you to design an arbitrary number of custom evaluation layers. Calico tiers will be discussed in the next post.

Get started with an interactive demo: DNS Policy with Calico

The post Tiered Network Policy: Scaling Kubernetes Security appeared first on Tigera – Creator of Calico.