Fortune Ndlovu

Posted on Sep 3

“I’m not excited by what AI can do. I’m excited by what I can do.” AI and security takeaways from ContainerDays Hamburg

#ai #kubernetes #security #devops

I spent three days at ContainerDays Hamburg 2026 at Schuppen 52, speaking, listening, and taking too many notes on my phone. This year’s conference sat next to a dedicated AI Context track, which meant the hallway conversations kept sliding from Kubernetes controllers to agents that call tools on your behalf.

I was there to talk about building production-ready Kubernetes operators, using the Red Hat Developer Hub operator as the running example. Huge thanks to the rhdh-installs team and everyone who helped me prep that talk. The slides are public if you want the operator deep dive.

This post is not that talk, and it is not a session-by-session recap. It is the thread I could not stop pulling: if we already refuse to let an untrusted loop mutate production without validation, hashing, and observability, why would we let an agent do it?

I also had the chance to speak with Kelsey Hightower. He put the week in one sentence:

“I’m not excited by what AI can do. I’m excited by what I can do.”

That is the thesis. The doing part is getting cheaper. Understanding the system is not.

Takeaway 1: Understand the control loop

Kelsey’s Q&A kept coming back to the same ground: Kubernetes is strongest as a declarative system. You say what you want. A control loop compares that to what exists and tries to make them match. The imperative API is powerful, and it is also how we accidentally turn a cluster into a pile of scripts nobody can predict.

I felt that in my bones because that loop is my day job.

A Kubernetes operator is a program that watches a custom resource and keeps the cluster aligned with it. We build ours with Operator SDK, which sits on Kubebuilder and controller-runtime. Under the hood you get a manager, a cache, watches, and a Reconcile method. Reconcile is not magic. It is: load the spec, look at the cluster, apply the difference, report status, wait for the next event.

A chat demo is imperative. You type, the model answers, you hope. A production agent is a loop: plan, call a tool, observe, retry, maybe call another tool. If you cannot predict that loop, you do not have a system. You have a vibe.

The operator pattern we ship is one way to make a loop inspectable. Configuration is layered: defaults, then flavour, then the custom resource, then a deployment patch. Defaults give you a working Backstage on day one. Flavours fold in optional stacks such as an AI assistant. The custom resource is the user’s intent. The patch is the last surgical override. Other operators handle this differently; some never generate manifests at all. Layering is our pattern, not a law of nature. The point is that every layer is named, ordered, and reviewable.

We also keep large YAML out of the custom resource. App-config and dynamic-plugin files live in ConfigMaps and Secrets the CR points at. Extra environment variables and secret references belong in the CR. That is not taste. etcd is a bad place for documents you will iterate on. Reviewers need a diff they can read. Secrets need a different lifecycle from API objects. And we hash the watched external config so that when someone edits a ConfigMap, pods actually restart. If the loop cannot see the change, the loop cannot reconcile it.

That is the habit I want to steal for agents. Do not stuff unbounded context into the prompt the way people want to stuff unbounded YAML into a CR. Name the layers. Hash what you watch. Fail fast when a referenced secret is missing. If you cannot draw the loop on paper, Kubernetes will not save you, and neither will the model.

Takeaway 2: The attack surface is text, tools, and trust

Nico Meisenzahl’s session, Agentic AI Under Attack, showed how AI agents can be hijacked without exploiting application code. The attacks used text, tools, and trust. He walked through live demos of three problems that already have names in the OWASP lists.

Indirect prompt injection. This is when content you did not type, such as a ticket, a wiki page, a PDF, or a web page, tells the agent to change its goal. At the model layer that is LLM01:2025 Prompt Injection. Once the model can plan and act across several steps, it is also ASI01 Agent Goal Hijack.

Tool and MCP poisoning. MCP is the Model Context Protocol, a common way for agents to connect to tools. The agent does not need a compromised binary if the tool description is a lie. The OWASP MCP Top 10 covers this as MCP03 Tool Poisoning and MCP10 Context Injection and Over-Sharing. The agent list calls the runtime version ASI02 Tool Misuse and Exploitation. A poisoned description can steer the next call and leak the whole conversation.

RAG poisoning. RAG means retrieval augmented generation. The agent looks up “internal knowledge” and uses it to answer. If someone you do not fully trust can write into that knowledge store, the attacker can hide instructions there. That maps to LLM08:2025 Vector and Embedding Weaknesses and ASI06 Memory and Context Poisoning. Those instructions can keep working after the original prompt is gone.

These are the questions I wrote down during the talk.

What stops the model from calling a dangerous tool? Not a system prompt. You need least privilege, a list of allowed tools, and a human in the loop for actions you cannot undo. That is LLM06:2025 Excessive Agency in real systems. If a refund tool exists, someday the agent will find it.

What about code execution? Treat the sandbox as throwaway. Create it for the call. Destroy it when the call ends. ASI05 Unexpected Code Execution is what you get when a tool that looks like text becomes a shell.

And before you store anything in a vector database, check it. A prompt shield that runs after the model has already read the document is too late.

Takeaway 3: Gate the conversation, not just HTTP

Steven Thwaites’s session, Your Agents Have Trust Issues, was about why securing AI needs more than a web firewall. He called it a Layer 8 problem. In networking, people use “layer 8” to mean the human layer. Here it means you have to look at the meaning of the conversation, not only the HTTP request.

A WAF (web application firewall) is good at headers, TLS, and known payload shapes. It cannot see a prompt inside a support ticket that says “ignore the previous policy.” It also cannot see a tool schema that says “this helper only reads logs” while the code posts those logs somewhere else.

The defense he walked through is an AI gateway that inspects the conversation. It checks prompts before they reach the model. It redacts sensitive data on the way back out (that is DLP, data loss prevention). It also uses agent identity: which agent may call which tool, with which credentials, for which tenant.

You do not have to buy a particular product to use the pattern. Microsoft documents the same split as Prompt Shields. One check looks at what the user typed. Another looks at documents the user did not type. The important design choice is when that check runs. If untrusted text reaches the model, you are already negotiating with the attacker.

MCP makes this practical. Manfred Bjørlin and Awar Abdulkarim’s session, MCP Security: Keep Your AI Agents from Spilling the Tea, asked the question every platform team will hear this year. Can you just reuse the RBAC you already have for REST?

RBAC is role based access control: who is allowed to do what. You can reuse your identity provider. You should not pretend an agent is a browser tab.

The MCP authorization specification is clear. A protected MCP server is an OAuth 2.1 resource server. Clients learn how to get a token through OAuth 2.0 Protected Resource Metadata (RFC 9728). Tokens should be bound to a specific audience with resource indicators (RFC 8707). Public clients need PKCE, which stops someone else from swapping in an authorization code. A shared static token in an environment variable is not enough. If it leaks, every tool behind it leaks with it.

The talk also named a failure the specs do not stress enough: consent fatigue. If every tool call pops a dialog, people click Allow. If nothing ever pops a dialog, the agent is running with standing privilege. Ask for consent by class of action. Reading a public doc is not the same as deleting a namespace. Do not ask for a click on every JSON-RPC method.

And log it. MCP08 Lack of Audit and Telemetry is easy to skip. If you cannot reconstruct which tool description the agent saw, which token it used, and which arguments it sent, you will spend the incident arguing with a transcript that does not exist.

flowchart LR
  user[User_or_agent]
  shield[Prompt_shield]
  authz[AuthZ_and_agent_identity]
  model[LLM]
  allowlist[Tool_allowlist]
  sandbox[Ephemeral_sandbox]
  tools[MCP_tools]
  audit[Audit_log]
  user --> shield
  shield -->|blocked| audit
  shield -->|clean| authz
  authz --> model
  model --> allowlist
  allowlist --> sandbox
  sandbox --> tools
  tools --> audit
  model --> audit

The order is prompt shield, then identity, then the model, then an allowlist, then a sandbox that you destroy, then a log. If you skip a step, that is usually where the incident starts.

Takeaway 4: The supply chain now includes what agents pull

Mohamad Sanan’s session, Own Goal, was about software supply chain risk when AI agents install packages. When a developer pulls a dependency, that is a choice. When an agent does it, it can happen without anyone noticing.

He backed that with JFrog’s 2026 Software Supply Chain Security State of the Union, as presented in the room. Eight malicious packages leaking on the order of 83,000 secrets. A handful of packages driving millions of compromised downloads. A reported 451% rise in malicious npm packages. Those are vendor research numbers, not my measurements. Treat them as a signal, then check your own registries.

The standards already expected this. LLM03:2025 Supply Chain covers models, datasets, plugins, and third party components. ASI04 Agentic Supply Chain Vulnerabilities is the version where the agent is the one running install. MCP09 Shadow MCP Servers is the version where nobody’s inventory even knows the server exists.

The rest of the security track told the same story at different layers.

Lior Kaplan and Yuval Saggie looked at CVE fixes across thousands of open source projects. A CVE is a published software vulnerability. They showed the lag that actually hurts you. A fix lands in the upstream project, then waits on a package, then a base image, then your final container. The time you are exposed is all of those delays added together. Project Copacetic exists because waiting for a full rebuild is how weeks disappear.

Mario Fahlandt generated SBOMs across a large slice of the CNCF landscape. An SBOM is a software bill of materials, a list of what is inside an artifact. His plain point was that an SBOM you cannot search for licenses and affected packages is paperwork. Use a real format such as SPDX or CycloneDX, and show the problems. The EU Cyber Resilience Act is the calendar that makes “we will get to it” a weaker answer than it was last year.

Dominik Mathern’s session on untrusted containers closed this at the cluster gate. Pulling latest from the public internet is convenient. It is also how incidents start. Admission control is how you make “only signed, scanned images from this registry” a property of the cluster instead of a wiki page. Pair that with provenance. Sigstore is the usual starting point, so you are not trusting a tag.

Two nearby talks belong here even though they were not billed as security. Jeffrey Sica’s walk through CNCF maturity was a reminder that reviews, shared infrastructure, and stewardship are how open source stays safe at scale. Akshay Jain’s Own your Agentic AI asked the private cloud questions out loud. What if the data never leaves the building? What if burning tokens should not mean burning SaaS fees? Where data is allowed to live is a design constraint. If you do not plan for it, you leak.

Takeaway 5: Production agents are distributed systems

Donald Forbes’s session, From Prototype to Production, was about making agentic AI reliable after the demo. AI demos are easy. Running agents as a distributed system is not.

The failures he listed are ones we already know, just in a new place: dependencies that flake, outputs that are not deterministic, workflows that run for a long time, partial failures, hidden state, weak observability, and rollback that does not actually roll back. The patterns he offered are familiar too: durable execution, retries with a budget, stateful orchestration, a human in the loop for steps that can hurt, and traces you can replay.

That last one is the eval question I wrote down and then kept hearing answered. When you evaluate an agent, do not only score the final answer. Score the trace. Which tool did it pick? What state did it change? Did it retry? Did it recover? Did it ask a human? A correct paragraph produced by the wrong tool is a future incident.

Philipp Westphalen’s incident response talk is the operations version of the same idea. Gathering logs, lining up metrics, digging through code, checking past incidents, and writing the postmortem is work. Agents can help with the gathering. The interesting design choice is the guardrail on fixes the agent is allowed to make by itself. Enriching a ticket is one kind of agency. Restarting a deployment is another. Deleting a namespace is a third. Map those to approval. Do not hide them behind a single flag that says “the AI is in the loop.”

René Schramowski and Ruben Verma argued you do not need a new infrastructure silo to run this. Enterprise Kubernetes already knows how to isolate workloads, inject identity, and kill a pod. That matches the operator instinct. Reuse the platform. Do not invent a parallel one because the workload is now called an agent.

NIST already gave this a vocabulary. The AI Risk Management Framework is four verbs: Govern, Map, Measure, Manage. In plain language that is who is accountable, what you think the system is, how you test it, and what you do when it goes wrong. The Generative AI Profile (NIST AI 600-1) is the closer companion for large language models. ISO/IEC 42001 is the management system standard buyers will ask for. None of those replace a tool allowlist. They are the process around the process.

From Kelsey’s Q&A I took a note about tokens that belongs here. Treat tokens like a scarce production resource. Prefer to pay for a result once and cache it close to the work, instead of asking the model to rediscover the same facts on every turn. That is not a reason to stop thinking. It is a reason to notice what the system is actually doing, what it is good at, and what it will never be good at no matter how many times you retry.

Takeaway 6: Humans are still the missing layer, and Kelsey gave us homework

Hossein Rafieekhah’s session, The Missing Layer in the AI Stack, was about human decision making as AI gets faster. We scale models, agents, and GPUs, and we treat human attention as a constant. The risk is not only slow work. It is bad decisions at machine speed. If the workspace cannot tell that the person on call is exhausted, the agent will keep handing them choices they cannot undo at 6pm on a Friday.

That is ASI09 in real life: Human-Agent Trust Exploitation. We over trust fluent output. Attackers know that. Tired humans know even less.

Kelsey’s Q&A was the other half of that warning. What I took from the room, in my words rather than his: models are average by design. Great programmers were already building serious systems before this wave. If we get comfortable shipping “good enough,” we are choosing the average. Design, taste, architecture you can draw, mentoring, and teaching people how to use the product are still ours. If you are an engineer, building is not the whole job.

He also left an action item.

Make two lists. What is the computer good at? What are you good at? If the lists match, you might be in trouble.

Lucas Hornung and Christian Matthaei’s session, From “No Time For GitOps” To Enterprise Adoption, was about how Tchibo adopted Flux. GitOps was ignored for years because “YAML will be more consistent” does not sell. They stopped pitching Flux as a tool and started with why: developer pain, numbers you can check, and stories people remember. Their five lessons were:

Be visible.
Talk to people. Listen. Ask for advice.
Make the value measurable, then check the numbers.
Create memories. Make people feel involved because they are.
Communicate. Say the thing clearly.

The turning point was not a better Helm chart. It was the Head of Webshop making GitOps a mandate. They even hacked a coffee machine on the floor with “Don’t be Hans-Peter. Use Flux.” That is not a security control. It is how humans actually change a system.

If you only take one thing from Hamburg besides the checklist below, take Kelsey’s action item. Write the two lists. Look at them again when you add a tool.

What I am taking back to work

Here is the short list I am taking back, with a source for each item.

Inventory every agent, tool, and MCP server. Shadow MCP is MCP09. You cannot gate what you cannot name.
Run a prompt and document shield before the model sees the context. Pattern: Prompt Shields. Risk: LLM01.
Allow only the tools you mean to allow, give them the least privilege they need, and destroy sandboxes at the end of the call. LLM06, ASI02, ASI05.
Put OAuth 2.1 on MCP. Bind tokens to an audience. Do not share them across servers. MCP authorization, RFC 9728, RFC 8707.
Log prompts, tool descriptions, tool calls, and traces. Evaluate the traces, not only the final answer. MCP08, NIST Measure.
Gate images and packages the agent pulls at the cluster and the proxy, not in a wiki. Admission controllers, Sigstore, ASI04.
Produce SBOMs you can search for licenses and CVEs, then shorten the time to patch. SPDX, CycloneDX, Copacetic, CRA.
Require a human for actions you cannot undo. Map that to NIST Govern and ASI09.
Keep a written picture of the system. Defaults, flavours, custom resource, patches. Prompt shield, identity, allowlist, sandbox, audit. If you cannot draw it, you cannot secure it.
Do Kelsey’s homework. Two lists. Computer versus you. Update them when the catalog grows.

Close

I went to Hamburg to talk about operators. I left thinking about the same loop for agents.

A production operator does not “just apply YAML.” It loads intent, checks references, merges layers in a known order, hashes what it watches, writes status, and waits to be told the world changed. A production agent that skips those steps is an unattended loop against a spec nobody wrote.

Kelsey is right. I am not particularly excited by what the model can generate. I am excited by what we can still do: design the loop, refuse the dangerous tool, teach the next person how the system actually works, and keep a list of the work we are not willing to hand over.

If you were in the room, I would like to hear which takeaway you are taking back. If you were not, start with the two lists.

Sources and further reading

Primary references used above. Talk attributions are to sessions at ContainerDays Hamburg 2026.

DEV Community