DEV Community

Devam Parikh
Devam Parikh

Posted on

Designing ChatOps Sessions for Kubernetes Agents

#kubernetes #chatops #sre #ai

Running a Kubernetes agent from Slack or Teams sounds simple:

Mention the bot and ask it to inspect the cluster.

The hard part is not sending a message to an agent. The hard part is deciding what context that message belongs to.

In incident response, context boundaries matter.

If two unrelated incident threads share the same agent memory, the agent may mix assumptions. If a bot can be invoked from any channel or personal chat, sensitive operations can drift outside approved spaces.

This post covers a practical model for ChatOps sessions with Kubernetes agents.

The core rule

For incident workflows, the safest default is:

one chat thread = one agent context
Enter fullscreen mode Exit fullscreen mode

That means:

Incident thread A -> agent session A
Incident thread B -> agent session B
Personal chat      -> blocked or separate session
Enter fullscreen mode Exit fullscreen mode

This keeps debugging context, approvals, tool results, and follow-up questions scoped to one operational conversation.

Chat threads are not always obvious to bots

Humans see a thread as a clear conversation.

Bot frameworks may expose several identifiers:

  • team ID
  • channel ID
  • conversation ID
  • message ID
  • reply/root thread ID

Depending on the platform, multiple channel threads may share a conversation ID and differ only by reply metadata.

If your bot stores agent state only by conversation.id, two threads may accidentally share an agent session.

A better session key includes the thread root:

session_key = channel_id + thread_root_id
Enter fullscreen mode Exit fullscreen mode

The exact fields vary by provider, but the intent is stable:

bind the agent session to the incident thread, not only to the channel.

The bot may not see old thread history

Another subtle issue: if an agent is mentioned halfway through a long thread, it may not automatically receive the previous messages.

The bot might only receive:

@agent can you inspect this?
Enter fullscreen mode Exit fullscreen mode

It may not receive:

Earlier: deployment failed after rollout
Earlier: image pull errors were seen
Earlier: only one namespace is affected
Enter fullscreen mode Exit fullscreen mode

There are three ways to handle this:

  1. Ask users to include context when mentioning the agent.
  2. Use the chat provider API to fetch thread history.
  3. Summarize thread history before sending it to the agent.

For production usage, fetching thread context requires careful permissions and data handling. Do not add broad chat-read permissions casually.

Control where the agent can be used

There are two layers of access control:

  1. Where the app can be installed.
  2. Where the runtime will actually respond.

The first layer is app configuration. For example, you might remove personal chat or group chat support and allow only team/channel usage.

The second layer is runtime enforcement.

A Kubernetes operations bot should support allowlists such as:

ALLOWED_TEAM_IDS
ALLOWED_CHANNEL_IDS
ALLOWED_CONVERSATION_IDS
Enter fullscreen mode Exit fullscreen mode

The bot should reject normal messages and approval actions outside approved locations.

This matters because an approval click is operational authority. It should be constrained to the same trusted space as the request.

Approval state belongs to the session

If the agent asks to run a protected tool, the approval decision should be tied to:

  • user
  • request
  • tool
  • target cluster/namespace/resource
  • chat thread
  • timestamp

Avoid approval flows that can be replayed or detached from their context.

An approval should mean:

I approve this specific tool call for this specific target in this specific thread.
Enter fullscreen mode Exit fullscreen mode

Not:

I approve the agent to do something generally.
Enter fullscreen mode Exit fullscreen mode

Make the agent summarize risk before approval

A useful ChatOps flow looks like:

User:
  Check this workload and restart it if needed.

Agent:
  I found failing readiness probes.
  Restarting the deployment is a Yellow operation because it mutates live workload state.

Approval card:
  Tool: restart deployment
  Target: namespace/app
  Risk: Yellow
  Approve / Reject
Enter fullscreen mode Exit fullscreen mode

The agent explains. The platform enforces.

That separation keeps the UX friendly without relying on the model as the policy boundary.

Final thought

Chat is a great interface for Kubernetes incident response because it is where people already coordinate.

But a chat interface is not automatically safe.

Design the session model deliberately:

  • one incident thread maps to one agent context
  • approved channels are enforced at runtime
  • old thread history is handled intentionally
  • approval actions are bound to a specific request
  • sensitive Kubernetes tools require real gates

The agent should feel easy to use, but the boundaries should be boringly explicit.

Top comments (0)