The Cloud Native Control Plane: Running AI Agents on Kubermetes

Deploying an AI agent script in a local container is easy. Running a fleet of autonomous, state-seeking, tool-executing agents in production without burning through your budget or creating a security nightmare is a distributed systems problem. The difference becomes apparent when an agent stops behaving like a conventional application.

A traditional microservice receives a request, executes a predictable sequence of operations, returns a response, and terminates. An autonomous agent might receive a task, reason about its objectives, invoke an external API, execute generated code, wait for another agent, retrieve historical state, retry a failed operation, and continue working long after the original HTTP request has disappeared.

Now imagine running a thousands of these agents simultaneously.

Some need GPU-backed inference. Others spend most of their time waiting for external tools. A few execute untrusted Python code. Some maintain conversations for hours, while others wake up in response to asynchronous events. One agent might trigger another, creating execution chains that are difficult to predict before they happen.

This is where the infrastructure conversation changes.

We are no longer discussing how to deploy an LLM-powered application. We are discussing how to operate a distributed system whose workloads are autonomous, resource-intensive, stateful, and capable of executing actions.

Kubernetes already has a vocabulary for many of these problems: scheduling, reconciliation, workload isolation, identity, resource quotas, event processing, and observability. The open-source ecosystem provides the remaining building blocks to connect agent frameworks, execution environments, and external tools into a common infrastructure platform.

The interesting engineering challenge is not putting an agent inside a Kubernetes Pod. It is building a platform where agents can be created, scheduled, authorized, executed, observed, suspended, and terminated without compromising the rest of the infrastructure.

The following architecture shows what that requires.

The Reality Check: Why Agentic AI Is a Distributed Systems Problem in Disguise

Consider a relatively ordinary recruitment automation workflow. A user submits a request to an HR agent to screen a collection of candidate resumes. The agent needs to:

1. Retrieve candidate information from a database.
2. Read and parse uploaded resumes.
3. Compare candidate experience against a job description.
4. Invoke an LLM for semantic evaluation.
5. Call an internal HR API to update candidate records.
6. Generate an evaluation report.
7. Notify the recruitment team.

A simple implementation might package the entire workflow into a Python application and deploy it as a container. Initially, everything works. Then the workload grows.

Recruiters submit hundreds of resumes simultaneously. LLM requests begin hitting rate limits. Some resumes contain malformed documents. A tool call times out after the agent has already completed three other operations. One agent retries a failed request repeatedly, while another loses its conversation state after a container restart.

Suddenly, the application is dealing with problems that have very little to do with prompt engineering.

The infrastructure is now responsible for managing autonomous execution.

Let's break down the operational problems.

1. Agent execution is not predictable.

A conventional application generally follows a known execution path. An agent can make decisions dynamically, invoke different tools, and change its execution plan based on intermediate results.

A workflow expected to finish in ten seconds might take several minutes because an external API is unavailable or the agent decides to investigate additional information. Traditional CPU and memory requests are not enough to describe every execution requirement.

2. Agent state outlives individual containers.

Agents frequently maintain conversation history, intermediate reasoning state, task checkpoints, tool results, and execution metadata. If a Pod disappears, the application should not necessarily lose the entire workflow. This introduces a familiar distributed systems requirement: separating application execution from durable state.

3. Tool execution introduces a security boundary.

An agent capable of executing generated Python or shell commands is fundamentally different from a service that only serves HTTP requests.

What happens when an agent executes:

cat /var/run/secrets/kubernetes.io/serviceaccount/token

Or attempts to access internal network services that were never intended to be available to it? Running generated code inside an ordinary application container does not automatically make that execution safe.

4. Workloads have different compute requirements.

An agent invoking an external REST API might need only a few hundred megabytes of memory and minimal CPU. An agent running local inference might require an expensive GPU-backed node.

Running both workloads inside the same resource configuration creates unnecessary infrastructure costs and scheduling inefficiencies.

5. Autonomous execution can create runaway workloads.

Consider an agent stuck in a reasoning loop:

Agent
  |
  v
Call LLM
  |
  v
Evaluate response
  |
  v
Tool invocation failed
  |
  v
Retry
  |
  v
Call LLM again
  |
  v
Repeat indefinitely

If every iteration consumes tokens, GPU time, and external API requests, one poorly controlled workflow can exhaust resources and affect unrelated workloads.

This is not an imaginary infrastructure concern. Any production system that permits autonomous execution needs explicit limits on retries, execution duration, token consumption, concurrency, and resource allocation.

The architectural shift

The important distinction is that an agent should not be treated as a long-running application container with unlimited authority.

It should be treated as a managed workload with:

• A declared execution specification.
• A controlled lifecycle.
• An isolated runtime.
• A defined identity.
• Explicit resource limits.
• Durable state.
• Observable execution history.
• A termination policy.

This is precisely where Kubernetes becomes interesting.

The Architectural Stack: Mapping Agent Needs to Cloud-Native Primitives

Kubernetes does not need to become an AI framework. LangGraph does not need to become a container scheduler. SPIFFE does not need to understand agent reasoning.

Each component should solve the problem it was designed to solve. The platform layer connects them.

Consider the following architecture:

                  Developer / API / User
                           |
                           v
                  Agent Gateway / API
                           |
                           v
              Agent Orchestration Layer
             LangGraph / AutoGen / CrewAI
                           |
                           v
                 Kubernetes API Server
                           |
                Agent Custom Resources
                           |
                           v
                  Agent Operator
                           |
          +----------------+----------------+
          |                |                |
          v                v                v
      GPU Runtime     Tool Runtime     Event Workers
      Inference       Wasm / gVisor     CPU Workloads
          |                |                |
          +----------------+----------------+
                           |
                           v
                Identity & Authorization
                  SPIFFE / SPIRE / OIDC
                           |
                           v
                 External Tool Services
             APIs / Databases / Cloud Services

                           |
                           v
                   Observability Layer
             OpenTelemetry / Prometheus
                 Grafana / Tempo

This architecture separates responsibilities instead of forcing everything into one application.

Agentic requirement	Cloud-native primitive	Responsibility
Agent lifecycle	Kubernetes CRDs and Operators	Declare and reconcile agent workloads
Workflow execution	LangGraph, AutoGen, CrewAI	Manage agent execution logic
Asynchronous communication	NATS, Kafka, Knative Eventing	Distribute tasks and events
Dynamic scaling	KEDA	Scale workers based on workload demand
Node provisioning	Karpenter	Provision suitable compute capacity
GPU scheduling	Kubernetes device plugins	Allocate GPU resources
Code isolation	gVisor, WebAssembly	Restrict untrusted execution
Workload identity	SPIFFE/SPIRE	Provide verifiable workload identities
Authorization	OIDC, OAuth 2.0, policy engines	Control access to external tools
Distributed tracing	OpenTelemetry	Correlate execution across components
Metrics and dashboards	Prometheus, Grafana	Track resource consumption and reliability

There is one important architectural decision here.

The agent framework owns the workflow. Kubernetes owns the infrastructure lifecycle.

This distinction prevents the orchestration framework from becoming responsible for problems that already have mature solutions in the cloud-native ecosystem.

Now let's examine each layer.

Layer 1: Compute & Sandboxing (Scheduling GPUs, CPUs, and Isolating Tool Executions with gVisor/Wasm)

The first question when deploying an agent fleet is deceptively simple: Where should each execution happen?

Not every agent needs the same infrastructure. Treating all agent workloads as identical Pods is one of the quickest ways to create an expensive and inefficient platform.

1.1 Separate inference workloads from tool execution

An agent workflow usually contains multiple types of operations.

Inference workloads

These execute model inference, either through an external LLM provider or through locally hosted models.

They might require:

• GPU-backed compute.
• High memory capacity.
• Model weights and caching.
• High-throughput networking.
• GPU-aware scheduling.

Tool execution workloads

These execute individual actions requested by an agent.

Examples include:

• Running Python scripts.
• Executing shell commands.
• Processing uploaded documents.
• Calling internal REST APIs.
• Performing data transformations.

Most of these operations require CPU and memory rather than GPUs.

Agent coordination workloads

These manage workflow state, task scheduling, event processing, and communication between agents. They are generally lightweight compared with inference workloads. A reasonable infrastructure design separates these workloads into different Kubernetes node pools.

Kubernetes Cluster
|
+-- GPU Node Pool
|    |
|    +-- Local LLM Inference
|    +-- Embedding Models
|    +-- GPU-Accelerated Agents
|
+-- CPU Node Pool
|    |
|    +-- Agent Workers
|    +-- Workflow Execution
|    +-- API Integrations
|
+-- Isolated Execution Pool
     |
     +-- Python Tool Runtime
     +-- Shell Execution Runtime
     +-- Untrusted Code Workers

Kubernetes node labels, taints, tolerations, affinity rules, and resource requests help direct workloads to appropriate infrastructure.

For example:

apiVersion: v1
kind: Pod
metadata:
  name: local-llm-worker
spec:
  nodeSelector:
    workload-type: gpu

  containers:
    - name: inference
      image: my-llm-runtime:v1

      resources:
        limits:
          nvidia.com/gpu: 1
          memory: 16Gi
          cpu: "4"

This is a simplified example of GPU scheduling. A real deployment also needs the NVIDIA device plugin or GPU Operator, appropriate drivers, runtime configuration, and capacity planning.

The important point is that GPU allocation becomes an infrastructure decision rather than something buried inside an agent framework.

1.2 Scheduling heterogeneous workloads

GPU scheduling introduces a different set of challenges.

Imagine a cluster with four GPU nodes. Each node has a fixed amount of accelerator memory, and several agent workloads are competing for capacity.

A newly created inference workload might remain Pending because no suitable GPU is available, even though the cluster has plenty of ordinary CPU capacity. This is where Karpenter becomes useful. Karpenter provisions Kubernetes nodes based on unschedulable Pod requirements and available infrastructure capacity.

For an agent platform, it can help provision different instance types based on workload characteristics:

• GPU instances for local inference.
• Memory-optimized instances for model-heavy workloads.
• General-purpose instances for workflow workers.
• Smaller instances for lightweight tool execution.

The scheduling path becomes:

Agent Execution Requested
          |
          v
Pod Created
          |
          v
Kubernetes Scheduler
          |
          v
Is Suitable Capacity Available?
          |
      +---+---+
      |       |
     Yes      No
      |       |
      v       v
   Schedule  Karpenter
      |       |
      |       v
      |   Provision Node
      |       |
      +-------+
          |
          v
    Agent Execution

Karpenter is not an agent-aware scheduler by itself. It responds to Kubernetes scheduling requirements. The platform must express those requirements correctly through resource requests, labels, affinity, and related scheduling constraints. That distinction matters when debugging capacity problems.

A workload that requests an unavailable GPU configuration will not magically become schedulable because Karpenter is installed.

1.3 The dangerous part: Executing generated code

Now consider an agent that receives this instruction:

"Analyze the uploaded CSV, calculate the average transaction value, and generate a chart."

The agent decides to execute Python. Where does that Python code run?

A naive implementation might execute it directly inside the agent's application container using Python's subprocess module. That is a bad security boundary.

Generated code is untrusted input, even when it originates from an agent running inside your own platform. A malicious document, prompt injection, or compromised tool response could influence what gets executed.

A container is a useful isolation mechanism, but an ordinary container shares the host kernel. Its security depends on namespaces, capabilities, seccomp, Linux security modules, runtime configuration, and the absence of exploitable kernel vulnerabilities.

For workloads that execute arbitrary code, stronger isolation deserves serious consideration. Two technologies are particularly interesting.

gVisor: A stronger application sandbox

gVisor provides an additional isolation boundary between an application and the host kernel.

Rather than allowing the application to interact directly with the host kernel through the usual system-call interface, gVisor implements much of that interface in its own application kernel, called the Sentry.

This reduces the host kernel surface exposed to potentially hostile application code.

A Kubernetes RuntimeClass can select the gVisor runtime:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

A tool execution Pod can then request that runtime:

apiVersion: v1
kind: Pod
metadata:
  name: python-tool-execution
spec:
  runtimeClassName: gvisor

  automountServiceAccountToken: false

  containers:
    - name: executor
      image: python:3.12-slim

      command:
        - python
        - /workspace/task.py

      resources:
        requests:
          cpu: "250m"
          memory: "256Mi"

        limits:
          cpu: "1"
          memory: "512Mi"

This is a simplified abstraction. Production implementations should use controlled, preferably immutable execution images, read-only filesystems where practical, restricted capabilities, non-root execution, network policies, and ephemeral storage.

Also, gVisor requires compatible node runtime configuration. Adding runtimeClassName to a manifest does not install or configure gVisor automatically.

WebAssembly: Lightweight isolated execution

WebAssembly (Wasm) offers another interesting approach.

Instead of creating a conventional container for every small tool operation, the platform can execute compatible workloads inside a Wasm runtime.

Projects such as Wasmtime and Spin provide building blocks for running WebAssembly workloads.

Wasm is particularly useful for:

• Lightweight execution tasks.
• Deterministic data transformations.
• Portable tool implementations.
• Short-lived functions.
• Controlled access to host capabilities.

A Wasm runtime can expose only the filesystem, network, or host functions explicitly granted to a module.

That makes capability-based execution easier to reason about.

However, Wasm is not a universal replacement for containers. Native dependencies, arbitrary Python packages, system-level operations, and GPU workloads may require different execution environments.

A practical platform can support both.

Workload	Suitable execution model
Lightweight data transformation	WebAssembly
Small deterministic tools	WebAssembly
Python execution with third-party packages	gVisor-backed container
Shell execution	Strongly isolated container or VM
Local LLM inference	GPU-backed container
Workflow coordination	Standard Kubernetes Pod

The engineering rule: Never let the agent's reasoning process and its untrusted code execution share an unrestricted security boundary.

The agent decides what needs to happen. The execution runtime decides what is permitted to happen.

Layer 2: Identity & Security (SPIFFE/SPIRE, Workload Identity, and Zero-Trust Tool Access)

Once an agent can invoke tools, identity becomes a platform concern. Consider a recruitment agent that needs to read candidate records and update interview status.

The simplest implementation stores an API key in an environment variable:

env:
  - name: HR_API_KEY
    value: "some-static-api-key"

It works, but it creates several problems.

What happens when the key leaks through application logs? How do you rotate it across hundreds of agents? How do you prevent a resume-screening agent from accessing payroll information? What happens when an agent's permissions need to be revoked immediately?

Static credentials do not provide a particularly good foundation for autonomous systems. The better approach is to give workloads identities and authorize individual actions.

2.1 SPIFFE: Giving every workload a verifiable identity

SPIFFE defines a standard for identifying software workloads independently of their physical location or infrastructure.

A SPIFFE identity is represented using a SPIFFE ID.

For example:

spiffe://agent-platform.example.com/ns/hr/sa/resume-agent

This identity tells us:

• The workload belongs to the agent-platform.example.com trust domain.
• It operates in the hr namespace.
• It uses the resume-agent service account.

The identity does not depend on the Pod IP address, node name, or container hostname. That is particularly useful in Kubernetes, where Pods are frequently created, destroyed, and rescheduled.

SPIRE, the reference implementation of SPIFFE, provides the infrastructure for issuing and validating workload identities.

SPIRE typically involves:

• A SPIRE Server responsible for managing trust and identity issuance.
• SPIRE Agents running on infrastructure nodes.
• Workload attestation to establish which workload is requesting an identity.
• Short-lived X.509 certificates or JWT-SVIDs.

The resulting identity can be used for secure communication between services.

For example:

Resume Agent
     |
     | SPIFFE Identity
     v
HR Tool Gateway
     |
     | Validate Identity
     v
Authorization Policy
     |
     +-- Read Candidate Records
     |
     +-- Update Interview Status
     |
     +-- Deny Payroll Access

The gateway does not simply trust a request because it originated from a Kubernetes Pod. It verifies the workload's identity and checks whether the requested operation is permitted.

2.2 OIDC and workload identity

Another important part of the architecture is OpenID Connect (OIDC). OIDC builds on OAuth 2.0 and provides a standardized way to represent authenticated identities using signed identity tokens.

In Kubernetes, projected service account tokens can be issued with specific audiences and limited lifetimes. An external service can validate these tokens against the cluster's configured issuer and signing keys.

Cloud workload identity systems can use this mechanism to exchange Kubernetes service account credentials for short-lived cloud credentials.

For example, an agent running in Kubernetes might need to read an object from Amazon S3.

Instead of storing permanent AWS access keys inside the container, the workload can use an appropriate AWS workload identity mechanism, such as EKS Pod Identity or IAM Roles for Service Accounts (IRSA).

The flow looks like this:

Agent Pod
   |
   v
Kubernetes Service Account
   |
   v
Projected OIDC Token
   |
   v
AWS Identity Provider
   |
   v
Temporary AWS Credentials
   |
   v
Authorized S3 Access

The agent receives temporary credentials associated with its assigned identity. When those credentials expire, they can be refreshed through the supported identity integration. This removes the need to distribute permanent cloud credentials across agent workloads.

2.3 The missing piece: Authorization at the tool level

Identity alone does not solve the problem.

Knowing that a workload is a legitimate resume-screening agent does not automatically mean it should be allowed to perform every operation in the HR system.

This is where authorization policies enter the architecture.

A tool gateway can enforce rules such as:

Agent identity	Tool	Permission
Resume Agent	Candidate API	Read
Resume Agent	Candidate API	Update interview status
Resume Agent	Payroll API	Deny
Scheduling Agent	Calendar API	Read and write
Reporting Agent	Analytics API	Read

The platform can implement these policies through an authorization service, an API gateway, or a policy engine such as Open Policy Agent (OPA).

A simplified Rego-style policy might look like:

package agent.authorization

default allow := false

allow if {
    input.identity == "resume-agent"
    input.tool == "candidate-api"
    input.action == "read"
}

allow if {
    input.identity == "resume-agent"
    input.tool == "candidate-api"
    input.action == "update_interview_status"
}

This example illustrates the decision logic, not a complete production policy. A real policy must also validate the trustworthiness of the identity assertion, resource scope, tenant boundaries, and relevant request attributes.

2.4 Treat every tool invocation as an authorization decision

One mistake I would avoid when designing an agent platform is giving an agent a collection of credentials and trusting the framework to decide how to use them. The framework should not be the security authority.

Every tool invocation should pass through an authorization boundary.

For example:

Agent Framework
      |
      v
Tool Invocation Request
      |
      v
Identity Verification
      |
      v
Authorization Policy
      |
      +---- Denied ----> Reject Request
      |
      v
Credential Exchange
      |
      v
External Tool API
      |
      v
Audit Event

There is another subtle issue worth addressing: an agent should not automatically inherit the permissions of the user who initially created its task.

A user might request a report, but the agent should only receive the narrowly scoped permissions needed to generate that report. Delegation should be explicit, bounded, and auditable.

Short-lived credentials, audience restrictions, narrowly scoped tokens, and centralized authorization help establish that boundary.

The security principle is straightforward: An agent should only be able to perform the actions explicitly granted to its workload identity, regardless of what its LLM-generated instructions request.

Layer 3: Orchestration & Scaling (CRDs, Event-Driven Agent Runtimes, and KEDA)

Now that compute and identity are handled, we reach the part that makes the architecture operationally interesting.

How do we manage the lifecycle of thousands of agent executions?

Creating Pods manually is not a platform strategy. Building a custom orchestration API inside a Python application is not particularly attractive either.

Kubernetes already provides an extensibility mechanism for representing application-specific resources. That mechanism is the Custom Resource Definition (CRD).

3.1 Representing agents as Kubernetes resources

A CRD allows us to introduce new resource types into the Kubernetes API.

Instead of managing agents as anonymous application processes, we can represent them as declarative resources. Imagine a custom resource named AgentRun. A developer could describe an agent execution like this:

apiVersion: agents.platform.io/v1alpha1
kind: AgentRun

metadata:
  name: resume-screening-001
  namespace: hr

spec:
  framework: langgraph

  workflow:
    name: resume-screening
    version: "1.0"

  execution:
    runtime: gvisor
    timeout: 300s
    maxSteps: 20

  resources:
    cpu: "1"
    memory: 1Gi

  identity:
    serviceAccountName: resume-agent

  state:
    backend: postgresql

  tools:
    - candidate-api
    - resume-parser

  retryPolicy:
    maxAttempts: 3

This is an illustrative API design, not a built-in Kubernetes resource or an existing standard shared by all agent frameworks. Its purpose is to demonstrate how an agent execution can become a declarative infrastructure object.

The platform can now reason about:

• Which framework should execute the workflow.
• Which runtime should isolate its tools.
• Which service account should represent the workload.
• How long the execution may run.
• Which tools it can invoke.
• How many retries are permitted.
• Where its state should be stored.

The resource becomes the desired state of the execution.

3.2 Why an Operator is needed

A CRD defines a resource. It does not automatically implement the resource;s behavious. That is the responsibility of a Kubernetes Operator. An Operator watches custom resources and reconciles the actual cluster state with the desired state declared in those resources.

For our AgentRun example, the Operator might:

1. Watch for newly created AgentRun objects.
2. Validate the execution specification.
3. Resolve the agent framework and workflow version.
4. Create the required execution resources.
5. Attach the appropriate service account.
6. Configure runtime isolation.
7. Track execution progress.
8. Update the resource status.
9. Clean up temporary resources after completion.

The reconciliation loop might look like this:

             AgentRun CRD
                  |
                  v
             Agent Operator
                  |
                  v
          Read Desired State
                  |
                  v
        Validate Execution Policy
                  |
                  v
          Create Runtime Pods
                  |
                  v
          Observe Pod Status
                  |
                  v
          Update AgentRun Status
                  |
                  v
             Reconcile Again

A simplified status section might look like:

status:
  phase: Running

  execution:
    currentStep: 4
    completedSteps: 3
    failedSteps: 0

  conditions:
    - type: Ready
      status: "True"

The Operator should be idempotent. If it receives the same reconciliation request repeatedly, it should not create duplicate executions or accidentally restart a completed workflow.

That is an important detail because Kubernetes controllers operate through repeated reconciliation, not one-time imperative commands.

For an actual implementation, I would also distinguish between an Agent resource, which describes a reusable agent definition, and an AgentRun resource, which represents one execution.

That separation makes lifecycle management significantly cleaner.

3.3 Mapping agent frameworks to Kubernetes

The platform should not force every agent framework into the same execution model.

Different frameworks have different strengths.

LangGraph

LangGraph is particularly interesting for workflows that require explicit state management, conditional transitions, checkpoints, and human intervention.

Its graph-based execution model maps naturally to long-running Kubernetes-managed workflows.

For example:

Resume Received
      |
      v
Resume Parsing
      |
      v
Candidate Evaluation
      |
      v
Policy Check
      |
      +---- Needs Review ----> Human Approval
      |
      v
Update Candidate Record
      |
      v
Generate Report

The workflow engine handles the execution graph. Kubernetes handles the compute and infrastructure lifecycle.

A production implementation should persist LangGraph checkpoints outside the worker Pod so that execution can resume after infrastructure failures.

AutoGen

AutoGen supports multi-agent collaboration and conversation-driven workflows.

A platform can run individual agent participants as separate workers or group them into a managed workflow execution, depending on the collaboration model.

For example, a research agent, validation agent, and reporting agent can communicate through controlled message channels.

However, separating every participant into a Pod is not automatically beneficial. It adds network overhead, scheduling activity, and operational complexity. The deployment boundary should reflect actual isolation and scaling requirements.

CrewAI

CrewAI provides a role-oriented approach to coordinating agents around tasks and responsibilities.

A crew can be packaged as a workflow deployment, while individual task execution can be delegated to isolated workers when the workload requires it.

The important design choice is to avoid coupling the infrastructure API too tightly to one framework.

A common Kubernetes platform should expose infrastructure-level concepts and allow framework-specific adapters to translate those concepts into execution behavior.

3.4 Event-driven orchestration: Agents should not spend their lives waiting

One of the biggest inefficiencies in naive agent deployments is keeping application workers alive while they wait for external operations. Consider an agent that calls a document-processing service. The service takes 45 seconds to process a large document.

Does the agent need to hold a worker thread, a container, and all its execution resources for the entire duration? Not necessarily. This is where event-driven architecture becomes useful. Projects such as NATS, Apache Kafka, and Knative Eventing provide mechanisms for asynchronous communication.

The workflow can be split into independent stages.

Agent Workflow
      |
      v
Submit Document Job
      |
      v
Publish Event
      |
      v
Message Broker
      |
      v
Document Processing Worker
      |
      v
Publish Completion Event
      |
      v
Agent Workflow Resumes

The agent does not need to hold an execution thread while the document processor works.

Instead, it persists its checkpoint, waits for the completion event, and resumes when the result becomes available. NATS is useful for lightweight messaging and request/reply patterns, while JetStream provides persistence and replay capabilities. Kafka is appropriate when the platform needs durable event streams, replayable histories, partitioned consumption, and high-throughput event processing.

Knative Eventing provides Kubernetes-native event routing abstractions through resources such as Brokers, Triggers, and Sources.

These technologies are complementary in some architectures, but they are not interchangeable in every situation. The platform should select a messaging backbone based on delivery guarantees, retention, throughput, ordering, and operational requirements rather than deploying every available broker.

A critical detail is event delivery semantics.

Most distributed event systems provide at-least-once delivery under common configurations. Therefore, a completion event may be delivered more than once.

The workflow must use idempotency keys, durable checkpoints, and deduplication to avoid executing the same side effect repeatedly. An agent updating a candidate record should not accidentally update it three times because the completion event was redelivered.

3.5 KEDA: Scaling based on actual agent demand

Traditional Kubernetes Horizontal Pod Autoscaling primarily reacts to metrics such as CPU and memory utilization. That is useful, but agent workloads introduce other signals that matter more. Imagine a tool execution queue with 500 pending tasks. The worker Pods might be using almost no CPU because they are waiting for messages. A CPU-based autoscaler sees no reason to scale.

Meanwhile, the queue continues growing. This is exactly the type of problem KEDA (Kubernetes Event-driven Autoscaling) addresses. KEDA allows workloads to scale based on external and event-driven metrics.

Depending on the configured scaler, it can respond to:

• NATS message backlog.
• Kafka consumer lag.
• Queue depth.
• HTTP request rates.
• Prometheus metrics.
• Other supported external signals.

A simplified ScaledObject might look like:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject

metadata:
  name: agent-tool-workers

spec:
  scaleTargetRef:
    name: agent-tool-workers

  pollingInterval: 10

  minReplicaCount: 0
  maxReplicaCount: 50

  triggers:
    - type: prometheus

      metadata:
        serverAddress: http://prometheus.monitoring:9090

        metricName: pending_agent_tasks

        threshold: "20"

        query: |
          sum(agent_task_queue_depth)

This example assumes the Prometheus metric is correctly exposed and the scaler is configured with appropriate authentication and permissions. The result is a worker pool that responds to actual pending work rather than waiting for CPU utilization to increase.

There is an important interaction between KEDA and Karpenter:

• KEDA decides how many workload replicas are needed.
• Kubernetes Scheduler determines whether those Pods can be placed.
• Karpenter can provision additional nodes when existing capacity is insufficient.

Together, they provide a useful scaling chain.

Agent Task Queue
       |
       v
Queue Backlog Increases
       |
       v
      KEDA
       |
       v
Increase Worker Replicas
       |
       v
Kubernetes Scheduler
       |
       v
Insufficient Node Capacity?
       |
       v
    Karpenter
       |
       v
Provision Additional Nodes
       |
       v
Execute Agent Tasks

3.6 Preventing the infinite agent loop

Scaling an agent platform is not just about adding more workers. Sometimes the correct response to an increase in workload is to stop accepting additional execution.

Imagine an agent that continuously retries a failed tool call. KEDA sees a growing queue and creates more workers. Those workers encounter the same failure and publish more retry events.

The infrastructure scales successfully while the application becomes progressively less healthy. This is a classic positive feedback loop. The platform needs execution budgets.

For every AgentRun, I would enforce:

Control	Example limit
Maximum execution duration	5 minutes
Maximum reasoning steps	20
Maximum tool retries	3
Maximum concurrent tool calls	5
Maximum token budget	50,000
Maximum memory	1 GiB
Maximum CPU	1 core
Maximum GPU allocation	1 device

These are illustrative values, not universal production defaults.

The platform should also implement:

• Exponential backoff with jitter.
• Circuit breakers for failing external services.
• Dead-letter queues for repeatedly failing tasks.
• Per-tenant concurrency limits.
• Token and cost budgets.
• Explicit cancellation and timeout handling.

Autonomy without execution limits is not a production feature. It is an uncontrolled resource allocation problem.

Layer 4: Observability (Tracing Agent Reasoning Loops and Tool Latency with OpenTelemetry)

Here is a scenario that will feel familiar to anyone who has operated distributed applications. An agent takes 90 seconds to complete a task. The user reports that the system is slow.

You inspect the application logs.

You find:

Agent started
LLM request completed
Tool invoked
Tool response received
LLM request completed
Workflow completed

Nothing appears obviously wrong. But where did those 90 seconds go?

Was the LLM provider slow? Did the tool API take 60 seconds? Did the agent execute unnecessary reasoning steps? Was it waiting for an event? Did the workflow retry a request?

Ordinary application logs do not provide enough context to answer these questions. Agentic workloads need distributed tracing that follows execution across the entire workflow.

4.1 OpenTelemetry as the common telemetry layer

OpenTelemetry provides vendor-neutral APIs, SDKs, instrumentation, and collection protocols for generating and exporting telemetry.

It supports:

• Distributed traces.
• Metrics.
• Logs.
• Context propagation.

For agent infrastructure, OpenTelemetry provides a way to correlate events across frameworks, worker processes, tool gateways, inference services, and external integrations.

Consider the recruitment workflow again.

A single execution might generate the following trace:

AgentRun: resume-screening-001
|
+-- Workflow Execution
|   |
|   +-- Resume Parsing
|   |
|   +-- LLM Inference
|   |    |
|   |    +-- Provider Request
|   |    +-- Token Usage
|   |
|   +-- Candidate API
|   |    |
|   |    +-- Identity Verification
|   |    +-- Authorization Check
|   |    +-- HTTP Request
|   |
|   +-- Human Approval
|   |
|   +-- Report Generation
|
+-- Workflow Completion

Every operation should carry trace context so that the platform can reconstruct the complete execution path. A trace identifier can connect operations across services, even when those operations execute in different Pods.

4.2 Instrumenting agent execution

A simplified Python example using the OpenTelemetry SDK:

from opentelemetry import trace

tracer = trace.get_tracer("agent-platform")

def execute_agent(task):

    with tracer.start_as_current_span(
        "agent.workflow"
    ) as workflow_span:

        workflow_span.set_attribute(
            "agent.name",
            "resume-screening"
        )

        workflow_span.set_attribute(
            "agent.run_id",
            task["run_id"]
        )

        with tracer.start_as_current_span(
            "agent.llm.inference"
        ) as llm_span:

            response = call_llm(task)

            llm_span.set_attribute(
                "llm.model",
                "configured-model"
            )

            llm_span.set_attribute(
                "llm.tokens.input",
                response.input_tokens
            )

            llm_span.set_attribute(
                "llm.tokens.output",
                response.output_tokens
            )

        with tracer.start_as_current_span(
            "agent.tool.candidate_api"
        ) as tool_span:

            result = call_candidate_api(response)

            tool_span.set_attribute(
                "tool.name",
                "candidate-api"
            )

        return result

This is a conceptual instrumentation example. A real implementation needs an initialized SDK, exporter, appropriate instrumentation, and framework-specific hooks.

There is also an important distinction between observing execution and collecting internal reasoning.

Do not indiscriminately capture private chain-of-thought, raw prompts, or complete model responses in telemetry.

Those records can contain personal information, secrets, customer data, and sensitive business context.

The platform generally needs operational information:

• Agent execution ID.
• Workflow name and version.
• Framework and model identifiers.
• Step number.
• Tool name.
• Tool latency.
• Retry count.
• Token usage.
• Resource consumption.
• Execution outcome.
• Error classification.

This provides useful debugging information without turning the observability backend into an unrestricted repository of sensitive agent conversations.

4.3 Correlating agent metrics with infrastructure metrics

Agent observability becomes much more useful when application-level signals are connected to Kubernetes-level signals.

For example, a slow agent execution might correlate with:

• GPU utilization reaching saturation.
• Pod scheduling delays.
• CPU throttling.
• Memory pressure.
• High message broker consumer lag.
• LLM provider latency.
• Tool gateway timeouts.

I would build dashboards around four categories.

Agent execution metrics

agent_runs_total
agent_runs_active
agent_runs_failed_total
agent_execution_duration_seconds
agent_steps_total
agent_retries_total
agent_cancellations_total

LLM metrics

llm_requests_total
llm_request_duration_seconds
llm_tokens_input_total
llm_tokens_output_total
llm_provider_errors_total

Tool execution metrics

agent_tool_calls_total
agent_tool_duration_seconds
agent_tool_failures_total
agent_tool_authorization_denied_total

Infrastructure metrics

agent_worker_cpu_usage
agent_worker_memory_usage
agent_gpu_utilization
agent_queue_depth
agent_pod_pending_seconds

Metric names are illustrative and should be aligned with the organization's instrumentation conventions.

One thing I would avoid is using agent run IDs, user IDs, or arbitrary prompt identifiers as Prometheus labels.

That creates extremely high cardinality and can overwhelm the metrics backend.

Use trace attributes for high-cardinality execution details. Keep Prometheus labels bounded to dimensions such as workflow type, model family, environment, and outcome.

4.4 Tracking non-deterministic failures

This is where agent observability becomes different from conventional microservice monitoring. A microservice failure might be reproducible by sending the same request with the same input. An agent failure is not necessarily reproducible.

The same task can produce different execution paths because of:

• Model sampling.
• Different tool responses.
• Context window changes.
• External API availability.
• Dynamic workflow decisions.
• Changes in retrieved information.

A failed agent run therefore needs more than a stack trace.

The platform should preserve a structured execution record containing:

{
  "run_id": "resume-screening-001",
  "workflow": "resume-screening",
  "workflow_version": "1.0",
  "status": "failed",
  "failed_step": "candidate-api-update",
  "retry_count": 3,
  "model": "configured-model",
  "tool": "candidate-api",
  "error_type": "authorization_denied",
  "execution_duration_ms": 48200
}

This record makes the failure operationally useful without exposing the entire conversation or sensitive prompt contents. For long-running workflows, trace context also needs careful propagation across asynchronous boundaries.

When an agent publishes a task to Kafka or NATS, the producer should inject trace context into message headers. The consumer should extract it and continue the trace or create a linked trace when the execution is no longer a direct synchronous child.

This allows an engineer to follow the workflow even when it crosses multiple services and asynchronous queues.

4.5 Building the observability pipeline

A practical architecture could look like this:

Agent Frameworks
       |
       v
OpenTelemetry SDK
       |
       v
OTel Collector
       |
       +------------------+
       |                  |
       v                  v
     Traces             Metrics
       |                  |
       v                  v
     Tempo            Prometheus
       |                  |
       +--------+---------+
                |
                v
             Grafana
                |
                v
       Agent Operations Dashboard

For logs, Fluent Bit can collect container logs and forward them to an appropriate backend such as Loki. OpenTelemetry Collector can also receive and process telemetry from different sources, including traces, metrics, and logs. The goal is not to collect everything.

It is to answer operational questions quickly:

• Which agent runs are failing most frequently?
• Which tools contribute the most execution latency?
• How much does each workflow cost in model tokens?
• Are GPU workloads waiting for capacity?
• Which agents are repeatedly exhausting their execution budgets?
• Are authorization denials increasing?
• How many executions are waiting for external events?
• Which framework or workflow version introduced a regression?

If an SRE cannot answer these questions from the platform's telemetry, the system is not operationally mature enough for a large agent fleet.

Top 3 Key Takeaways: Three Actionable Principles for Building the Platform

1. Treat every agent execution as a managed workload, not an unrestricted application process.

Define execution budgets, isolate generated code, separate GPU inference from CPU-based tool execution, and make agent lifecycle management declarative through Kubernetes resources and Operators.

2. Make identity and authorization independent of the agent framework.

Use SPIFFE/SPIRE, OIDC-based workload identity, short-lived credentials, and centralized authorization policies. Every tool invocation should be checked against an explicit permission boundary. Never trust an agent merely because it runs inside your cluster.

3. Build around asynchronous execution and end-to-end observability.

Use event-driven messaging, KEDA, and Karpenter to scale according to actual demand. Persist workflow state, propagate OpenTelemetry context across execution boundaries, and enforce limits on retries, tokens, concurrency, and runtime duration.

Strong Conclusion: Open Standards Will Define the Future of AI Infrastructure

There is a temptation to approach Agentic AI infrastructure by selecting one framework, one orchestration product, one execution runtime, and one proprietary cloud service, then connecting everything through application-specific integrations.

That approach can get a prototype running quickly.

It becomes considerably harder when the organization needs to change its model provider, introduce a new agent framework, move workloads between cloud environments, or replace its execution runtime without rewriting the entire platform.

This is why open-source infrastructure matters.

Kubernetes provides the common control plane for workload scheduling, reconciliation, resource allocation, and lifecycle management. OpenTelemetry provides a standard approach to telemetry collection and propagation. SPIFFE establishes workload identity. NATS and Kafka provide asynchronous communication primitives. KEDA and Karpenter address workload-driven scaling and infrastructure capacity. gVisor and WebAssembly provide alternative execution isolation models.

Agent frameworks such as LangGraph, AutoGen, and CrewAI can operate above these infrastructure components without owning the underlying security, scheduling, and observability responsibilities.

None of these technologies individually solves the Agentic AI infrastructure problem. Their value comes from the boundaries they establish and the contracts they expose to one another.

I would not build a production agent platform around the assumption that one framework will remain dominant or that one model provider will remain the best option. Both assumptions are likely to create unnecessary infrastructure coupling.

Instead, build a platform around replaceable components, explicit execution contracts, portable workload identities, durable state, and observable interfaces.

The next generation of cloud-native infrastructure will not simply host applications that happen to call LLM APIs. It will host autonomous workloads that make decisions, execute actions, collaborate with other workloads, and consume infrastructure dynamically.

Kubernetes already knows how to manage workloads that come and go. The challenge is teaching the platform how to manage workloads that decide what to do next.

The cloud-native agent is not a new kind of Pod. It is a new workload model built on top of infrastructure principles that distributed systems engineers have spent decades learning the hard way.
The Cloud Native