DEV Community

Kavin Arvind Ragavan
Kavin Arvind Ragavan

Posted on

Creating a Custom Copilot Agent for Chaos Engineering Using LitmusChaos and MCP

#ai #kubernetes #chaos #mcp

Chaos engineering is often introduced as a command-line or dashboard workflow: install a platform, register a Kubernetes cluster, create an experiment, and run it. That workflow is useful, but it leaves a gap between an SRE's or platform engineer's intent and safe execution.

The engineer should be able to work with the agent through a sequence of small, reviewable prompts: check readiness, inspect the target, review the experiment, approve execution, and verify recovery. This keeps discovery and approval separate from the destructive action.

That is the idea behind this project. I created a custom GitHub Copilot agent whose job is to operate LitmusChaos safely. The agent uses a Go-based Model Context Protocol (MCP) server for LitmusChaos API operations and uses Kubernetes read-only checks to verify local targets and recovery.

This post walks through the agent design, skill workflow, MCP integration, safety model, local Litmus setup, troubleshooting, and lessons learned.

Scope note: This article is about the LitmusChaos MCP integration and the custom Copilot agent. The demo-nginx workload is only an optional local Kubernetes validation target; it is separate from the MCP server and the agent.

Target Audience

This project is intended for engineers who already work with Kubernetes workloads and need a safer, more conversational way to run resilience tests:

  • Site reliability engineers (SREs) validating service recovery and steady-state behavior.
  • Platform engineers providing controlled chaos capabilities to development teams.
  • Chaos engineers designing and running LitmusChaos experiments.
  • Performance and resilience engineers testing failure behavior under controlled conditions.
  • Developers who need a disposable local workflow before promoting an experiment to a shared environment.

Throughout this article, the custom agent supports the SRE, platform engineer, or chaos engineer responsible for reviewing and approving a chaos action. It helps make the workflow repeatable, but it does not replace engineering judgment or approval.

The Agent Is the Product

The implementation contains five pieces, but the custom agent is the control layer that ties them together:

  1. Docker Desktop Kubernetes as the local Kubernetes cluster.
  2. LitmusChaos ChaosCenter as the control plane.
  3. Litmus chaos infrastructure deployed into the local cluster.
  4. A Go MCP server that translates MCP tool calls into LitmusChaos GraphQL requests.
  5. A custom Copilot agent that applies preflight checks, safety rules, confirmation gates, execution, and recovery verification.

The agent is designed to work with an explicitly identified workload and environment. It is instructed to reject control-plane namespaces, production environments, and unknown targets unless the responsible engineer provides an approved safety boundary.

Why Create a Custom Agent?

An MCP server exposes capabilities. It does not automatically define a safe operating procedure.

Without an agent, an engineer might ask for a chaos run and jump directly to an execution tool. With an agent, the request becomes a controlled workflow:

natural-language request
  |
  v
find experiment and inspect target
  |
  v
check infrastructure and Kubernetes workload
  |
  v
summarize impact and ask for confirmation
  |
  v
run through MCP and poll status
  |
  v
verify Kubernetes recovery and report evidence
Enter fullscreen mode Exit fullscreen mode

The agent does not replace LitmusChaos. It provides an opinionated operating model around LitmusChaos.

Custom Agent Structure

The workspace agent is defined in:

.github/agents/litmuschaos.agent.md
Enter fullscreen mode Exit fullscreen mode

Its frontmatter restricts the agent to the Litmus MCP server plus read-only or observational terminal work:

---
description: "Use for LitmusChaos MCP operations: inspect experiments, validate local Kubernetes chaos targets, run or stop disposable chaos experiments, investigate queued or failed runs, and report recovery. Requires confirmation before destructive actions."
name: "LitmusChaos Operator"
tools: [litmuschaos/*, execute, read]
user-invocable: true
---
Enter fullscreen mode Exit fullscreen mode

The agent body defines the behavior that matters more than the persona:

  • Inspect before executing.
  • Verify the infrastructure is active and confirmed.
  • Verify the namespace and workload selector.
  • Ask for explicit confirmation before run or stop operations.
  • Never expose the access token.
  • Verify the target recovers after the experiment.

The Agent Skill

The repeatable run workflow lives in:

.github/skills/litmus-experiment-runner/SKILL.md
Enter fullscreen mode Exit fullscreen mode

The skill is deliberately procedural. It tells the agent to:

  1. Find the experiment by name when necessary.
  2. Load the full experiment definition.
  3. Check infrastructure state.
  4. Validate the target namespace and selector.
  5. Present a confirmation summary.
  6. Run the experiment through MCP.
  7. Poll the run until it reaches a terminal state.
  8. Verify Kubernetes recovery.

This separation is useful: the .agent.md file defines the agent's role and boundaries, while the skill defines a reusable operational workflow.

Tutorial: Create the Agent and Skill

This is the smallest useful implementation. The agent defines the operating policy; the skill defines the repeatable run procedure.

1. Register the MCP Server

Create .vscode/mcp.json in the workspace:

{
  "servers": {
    "litmuschaos": {
      "type": "stdio",
      "command": "${workspaceFolder}/litmus-mcp-server/bin/litmuschaos-mcp-server.exe",
      "envFile": "${workspaceFolder}/litmus-mcp-server/.env"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Keep the token in the ignored .env file. Do not embed it in the agent, skill, README, screenshots, or chat prompts.

2. Create the Custom Agent

Create .github/agents/litmuschaos.agent.md:

---
description: "Operate LitmusChaos through MCP with target validation and confirmation gates."
name: "LitmusChaos Operator"
tools: [litmuschaos/*, execute, read]
user-invocable: true
---

You are the LitmusChaos Operator.

- Inspect experiments and infrastructure before execution.
- Verify the target namespace and workload.
- Require explicit confirmation before run or stop operations.
- Never expose credentials.
- Verify Kubernetes recovery after a local experiment.
Enter fullscreen mode Exit fullscreen mode

The description is important because it is the agent's discovery surface in VS Code. The body establishes behavior and safety boundaries.

3. Add the Operational Skill

Create .github/skills/litmus-experiment-runner/SKILL.md:

---
name: litmus-experiment-runner
description: 'Run safe LitmusChaos experiments through MCP with preflight, confirmation, polling, and recovery checks.'
---

# Litmus Experiment Runner

1. Find the experiment by name or ID.
2. Inspect the experiment definition and infrastructure.
3. Verify the namespace and workload selector.
4. Summarize impact and ask for confirmation.
5. Run through MCP only after confirmation.
6. Poll the run until it reaches a terminal state.
7. Verify Kubernetes recovery.
Enter fullscreen mode Exit fullscreen mode

A skill is a good fit here because the sequence is reusable and procedural. Keep policy in the agent and task-specific procedure in the skill.

4. Select the Agent in Copilot Chat

Reload the VS Code window if needed, select LitmusChaos Operator in the agent picker, and use Agent mode.

Start with a read-only request:

Check whether Litmus is ready for a chaos test, then list my chaos experiments.
Enter fullscreen mode Exit fullscreen mode

For an execution request against the optional local validation target, the agent should ask for confirmation:

Run the local `pod-delete-1` experiment against `demo-nginx` in namespace `chaos-test`.
Inspect the experiment and infrastructure first, then ask for confirmation.
Enter fullscreen mode Exit fullscreen mode

After confirmation, the agent can call run_chaos_experiment, poll list_experiment_runs, and use Kubernetes read-only commands to verify the local test Deployment recovered. The MCP server and custom agent remain the subject of this article; demo-nginx is only a sample workload used to exercise them.

5. Test the MCP Server Directly

The MCP transport can be tested independently of the agent with a read-only JSON-RPC request:

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"list_chaos_experiments","arguments":{}}}
Enter fullscreen mode Exit fullscreen mode

This distinction helps debugging:

  • If MCP discovery fails, inspect .vscode/mcp.json and server startup logs.
  • If MCP works but the run fails, inspect the experiment, infrastructure, workflow, and Kubernetes events.
  • If the agent skips confirmation, tighten the agent instructions and skill procedure.

Architecture

The request path looks like this:

Copilot Chat
    |
    | MCP over stdio
    v
LitmusChaos MCP server (Go)
    |
    | HTTP GraphQL request
    v
ChaosCenter GraphQL server
    |
    | WebSocket/subscriber communication
    v
Litmus chaos infrastructure in Kubernetes
    |
    v
ChaosEngine and experiment runner
    |
    v
Target workload in the approved application namespace
Enter fullscreen mode Exit fullscreen mode

The MCP server is a local stdio process. VS Code starts it from .vscode/mcp.json, and the process reads credentials from a local .env file.

A simplified MCP configuration looks like this:

{
  "servers": {
    "litmuschaos": {
      "type": "stdio",
      "command": "${workspaceFolder}/litmus-mcp-server/bin/litmuschaos-mcp-server.exe",
      "envFile": "${workspaceFolder}/litmus-mcp-server/.env"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The MCP server exposes these 16 enabled tools:

Tool Purpose
list_chaos_experiments Discover experiments with optional filters and pagination.
get_chaos_experiment Inspect an experiment, including its manifest and infrastructure.
run_chaos_experiment Start an existing experiment immediately.
stop_chaos_experiment Stop an active experiment or specific run.
list_experiment_runs Review experiment run history and filter by status.
get_experiment_run_details Inspect a run, execution state, and optional logs.
list_chaos_infrastructures Find registered infrastructures and filter by status.
get_infrastructure_details Inspect infrastructure details and optionally its manifest.
register_chaos_infrastructure Create an infrastructure registration request.
list_environments List environments used to organize infrastructures.
create_environment Create a PROD or NON_PROD environment.
list_resilience_probes Discover configured resilience probes.
create_resilience_probe Create supported HTTP, command, Kubernetes, or Prometheus probes.
list_chaos_hubs Discover available ChaosHubs.
get_chaos_faults Browse faults exposed by a ChaosHub.
get_experiment_statistics Review experiment and resiliency-score statistics.

The server currently exposes 16 enabled tools. Experiment creation is intentionally disabled: create and configure experiments in ChaosCenter first, then use MCP to list, inspect, run, stop, and review them.

What the MCP Server Does

The Go server is an API adapter. It accepts MCP tool calls over standard input and output, adds the configured project ID and bearer token to the request, calls the ChaosCenter GraphQL endpoint, and returns structured results to Copilot. It does not inspect Kubernetes workloads, decide whether a target is safe, or approve a destructive action.

Those responsibilities belong to the custom agent and its skill. Before a run, the agent reads the experiment manifest returned by get_chaos_experiment, checks that the manifest target matches the requested workload, verifies the infrastructure, and asks for explicit confirmation. After starting a run, it treats the start response as an acknowledgement rather than proof of success, polls the run, and investigates the workflow when the run remains active beyond its expected duration.

This separation also makes failures easier to diagnose:

  • MCP connectivity failure: the local GraphQL endpoint or port-forward is unavailable, so the server cannot reach ChaosCenter.
  • API failure: ChaosCenter rejects the GraphQL request or returns an error.
  • Workflow failure: ChaosCenter accepts the request, but Kubernetes or Argo rejects or cannot execute the generated workflow.
  • Target verification failure: the experiment manifest does not match the requested workload, or the local Kubernetes check cannot verify it.

The agent should report which layer failed and should not retry a destructive action automatically.

Experiment Coverage

MCP is not a second experiment catalog. It controls experiments already created in the configured ChaosCenter project, so the available experiments depend on the ChaosHub faults and probes installed there. Depending on that project, an engineer may use MCP with pod or container disruption, CPU or memory stress, network or storage faults, node-level faults, and probe-backed experiments that check HTTP, Kubernetes, command, or Prometheus signals.

The reliable workflow is to discover the actual catalog first:

List my chaos experiments.
Show the available faults in ChaosHub <hub-id>.
Show details for experiment <experiment-id>, including its manifest and recent runs.
Enter fullscreen mode Exit fullscreen mode

These are capability categories, not a guarantee that every Litmus installation contains every fault. The ChaosHub and the experiment definition determine which faults and parameters are available.

Prerequisites

You need:

  • Windows with Docker Desktop
  • Docker Desktop Kubernetes enabled
  • kubectl
  • Helm 3 or later
  • Go 1.21 or later to build the MCP server
  • A LitmusChaos project and API token
  • VS Code with Copilot Chat and MCP support

A full ChaosCenter installation is resource-heavy for a small laptop. My host had 8 GB of RAM and an i3 processor, so WSL 2 was configured conservatively:

[wsl2]
memory=4GB
processors=2
swap=2GB
localhostForwarding=true
Enter fullscreen mode Exit fullscreen mode

The WSL configuration lives at:

%UserProfile%\.wslconfig
Enter fullscreen mode Exit fullscreen mode

After changing it, apply the settings with:

wsl --shutdown
Enter fullscreen mode Exit fullscreen mode

Install ChaosCenter

First, select the local Docker Desktop context:

kubectl config use-context docker-desktop
kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

The node should be Ready.

Add the Litmus Helm repository:

helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm repo update
Enter fullscreen mode Exit fullscreen mode

For a constrained laptop, use a single MongoDB replica set member rather than the default three-member configuration:

helm install chaos litmuschaos/litmus `
  --namespace litmus `
  --create-namespace `
  --set portal.frontend.service.type=NodePort `
  --set mongodb.architecture=replicaset `
  --set mongodb.replicaCount=1 `
  --set mongodb.persistence.size=1Gi `
  --wait --timeout 15m
Enter fullscreen mode Exit fullscreen mode

The important detail is architecture=replicaset with replicaCount=1. A standalone MongoDB setting looked attractive for a low-resource installation, but Litmus init containers expected the StatefulSet DNS name chaos-mongodb-0.chaos-mongodb-headless. A one-member replica set preserves that expected service-discovery behavior.

Verify the installation:

kubectl get pods -n litmus
kubectl get svc -n litmus
Enter fullscreen mode Exit fullscreen mode

Forward the ChaosCenter frontend:

kubectl port-forward -n litmus svc/chaos-litmus-frontend-service 9091:9091
Enter fullscreen mode Exit fullscreen mode

Open:

http://localhost:9091
Enter fullscreen mode Exit fullscreen mode

Sign in with the administrator credentials configured for your local installation, and change the initial password immediately. Do not publish administrator credentials, tokens, or screenshots containing them.

Register the Local Chaos Infrastructure

ChaosCenter requires an environment and an infrastructure before it can run experiments.

Create a non-production environment, for example:

Name: local-docker
Type: NON_PROD
Description: Local Docker Desktop Kubernetes test environment
Enter fullscreen mode Exit fullscreen mode

Use ChaosCenter's Enable Chaos workflow to generate the infrastructure manifest. Apply the downloaded YAML with:

kubectl config use-context docker-desktop
kubectl apply -f local-docker-chaos-litmus-chaos-enable.yml
Enter fullscreen mode Exit fullscreen mode

The manifest installs components including:

  • Chaos operator
  • Chaos exporter
  • Subscriber
  • Event tracker
  • Workflow controller
  • Litmus CRDs and RBAC

Check readiness:

kubectl get pods -n litmus
kubectl logs -n litmus deployment/subscriber --tail=80
Enter fullscreen mode Exit fullscreen mode

A useful confirmation in the subscriber log is:

AgentID ... has been confirmed
Server connection established, Listening....
Enter fullscreen mode Exit fullscreen mode

On a 2 CPU local node, the generated infrastructure requests may exceed available CPU. For local validation, reduce only the infrastructure deployment requests:

kubectl set resources deployment/chaos-operator-ce deployment/chaos-exporter deployment/subscriber deployment/event-tracker deployment/workflow-controller `
  -n litmus --requests=cpu=50m,memory=100Mi
Enter fullscreen mode Exit fullscreen mode

This is a local scheduling workaround, not a production sizing recommendation.

Connect the MCP Backend

From the server directory:

cd litmus-mcp-server
go test ./...
go build -o .\bin\litmuschaos-mcp-server.exe .
Enter fullscreen mode Exit fullscreen mode

The server uses these environment variables:

CHAOS_CENTER_ENDPOINT=http://localhost:9002
LITMUS_PROJECT_ID=your-project-uuid
LITMUS_ACCESS_TOKEN=your-token
DEFAULT_INFRA_ID=your-infrastructure-id
DEFAULT_ENVIRONMENT_ID=your-environment-id
Enter fullscreen mode Exit fullscreen mode

The project ID is the UUID in the ChaosCenter project URL, not the display name. For example, a URL like:

/account/<account-id>/project/<project-id>/dashboard
Enter fullscreen mode Exit fullscreen mode

contains the project ID after /project/.

Forward the GraphQL server used by the MCP implementation:

kubectl port-forward -n litmus svc/chaos-litmus-server-service 9002:9002
Enter fullscreen mode Exit fullscreen mode

The local MCP endpoint is then:

http://localhost:9002
Enter fullscreen mode Exit fullscreen mode

The frontend port 9091 is for the browser UI. The MCP server should use the GraphQL port 9002.

Set Up a Local Validation Target

Create an isolated test namespace and workload:

kubectl create namespace chaos-test --dry-run=client -o yaml | kubectl apply -f -
kubectl create deployment demo-nginx `
  --image=nginx:alpine `
  --replicas=1 `
  -n chaos-test `
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout status deployment/demo-nginx -n chaos-test
Enter fullscreen mode Exit fullscreen mode

The expected label is:

app=demo-nginx
Enter fullscreen mode Exit fullscreen mode

The experiment should target this Deployment, not a pod name. Kubernetes will replace the pod after the fault runs.

Create the Experiment

In ChaosCenter, create a Pod Delete experiment from a ChaosHub template.

Recommended local values:

App kind: deployment
App namespace: chaos-test
App label: app=demo-nginx
Total chaos duration: 15 seconds
Ramp time: 0
Chaos interval: 5 seconds
Pods affected: 100 percent
Default health check: false
Sequence: parallel
Enter fullscreen mode Exit fullscreen mode

For the first test, either use no probe or create a Kubernetes probe that checks:

Group: apps
Version: v1
Resource: deployments
Resource name: demo-nginx
Namespace: chaos-test
Operation: present
Enter fullscreen mode Exit fullscreen mode

The probe must be attached to the Pod Delete fault, not to a workflow helper step such as run-chaos. Litmus stores the relationship in an annotation similar to:

annotations:
  probeRef: '[{"name":"k8sprobe","mode":"SOT"}]'
Enter fullscreen mode Exit fullscreen mode

The probe name must exactly match the saved probe name. A stale or mismatched probe reference causes errors such as:

Probe in fault is not attached to a proper reference
Enter fullscreen mode Exit fullscreen mode

Select and Test the Custom Agent

In Copilot Chat, select the LitmusChaos Operator custom agent and use Agent mode. Start with a read-only request:

Check whether Litmus is ready for a chaos test, then list my chaos experiments.
Enter fullscreen mode Exit fullscreen mode

A successful response includes the experiment name and ID. The agent should also report whether the infrastructure is active and confirmed. The server returned a successful read-only response with zero experiments before the experiment was created, which was a useful first connectivity test.

Run Through the Custom Agent

The safe conversational workflow is:

Run the pod-delete experiment through LitmusChaos.
First inspect the experiment and infrastructure.
Verify that the target is only demo-nginx in the chaos-test namespace.
Ask for confirmation before running it.
Enter fullscreen mode Exit fullscreen mode

The agent should resolve the display name to an experiment ID and then call MCP. Before execution, it should verify:

  • Infrastructure is active
  • Infrastructure is confirmed
  • The target namespace is chaos-test
  • The target selector is app=demo-nginx
  • The experiment is not pointed at litmus, kube-system, or production

Monitor Kubernetes in another terminal:

kubectl get pods -n chaos-test -w
Enter fullscreen mode Exit fullscreen mode

For a successful Pod Delete experiment, the target pod disappears and the Deployment creates a replacement pod.

The important design decision is that the agent does not silently execute a destructive tool. It turns the request into a preflight report and confirmation gate first.

Small Prompts for the Agent

Use these prompts one at a time. Wait for the agent's result before sending the next prompt.

Prompt 1: Check Readiness 

Is Litmus ready for a local chaos test?
Enter fullscreen mode Exit fullscreen mode

Prompt 2: Check the Target 

Check `Deployment/demo-nginx` in namespace `chaos-test`.
Enter fullscreen mode Exit fullscreen mode

Prompt 3: List Experiments 

List my chaos experiments.
Enter fullscreen mode Exit fullscreen mode

Prompt 4: Inspect the Experiment 

Show details for `pod-delete-1`.
Enter fullscreen mode Exit fullscreen mode

Prompt 5: Validate the Target 

Verify that `pod-delete-1` targets only `demo-nginx` in `chaos-test`.
Do not run it.
Enter fullscreen mode Exit fullscreen mode

Prompt 6: Ask for a Preflight 

Prepare the preflight summary for `pod-delete-1`.
Enter fullscreen mode Exit fullscreen mode

Prompt 7: Request Execution 

Run `pod-delete-1`.
Ask for confirmation first.
Enter fullscreen mode Exit fullscreen mode

Prompt 8: Confirm Execution

Only send this after reviewing the preflight summary:

Yes, run it.
Enter fullscreen mode Exit fullscreen mode

Prompt 9: Check the Run 

What is the status of the latest run?
Enter fullscreen mode Exit fullscreen mode

Prompt 10: Verify Recovery 

Verify that `demo-nginx` recovered.
Enter fullscreen mode Exit fullscreen mode

Prompt 11: Investigate Failure 

Why did the latest run fail or remain queued?
Enter fullscreen mode Exit fullscreen mode

Prompt 12: Stop Safely 

Check whether the experiment is still running.
Enter fullscreen mode Exit fullscreen mode

If it is still active, review the target and then send:

Stop the active run. Ask for confirmation first.
Enter fullscreen mode Exit fullscreen mode

Prompt 13: Inspect Infrastructure 

List only active chaos infrastructures.
Show details and the installation manifest for infrastructure <infra-id>.
Enter fullscreen mode Exit fullscreen mode

Prompt 14: Manage Environments 

List NON_PROD environments.
Create a NON_PROD environment named local-test with description "Disposable Kubernetes test environment".
Ask for confirmation before creating it.
Enter fullscreen mode Exit fullscreen mode

Prompt 15: Work With Resilience Probes 

List Kubernetes resilience probes.
Create a Kubernetes probe that checks whether Deployment/demo-nginx exists in namespace chaos-test, with a 5 second timeout and 5 second interval.
Enter fullscreen mode Exit fullscreen mode

Prompt 16: Discover ChaosHubs and Faults 

List all ChaosHubs.
Show pod-related faults available in ChaosHub <hub-id>.
Enter fullscreen mode Exit fullscreen mode

Prompt 17: Review Statistics 

Show experiment statistics including resiliency score distribution.
Give me a read-only health summary of experiments, infrastructures, environments, and ChaosHubs.
Enter fullscreen mode Exit fullscreen mode

Prompt 18: Register Infrastructure 

Register a namespace-scoped Kubernetes infrastructure named local-test-2 in environment <environment-id>.
Show me the request details and ask for confirmation before creating it.
Enter fullscreen mode Exit fullscreen mode

Creating an environment or infrastructure registration request changes ChaosCenter state, so the agent should confirm those actions too. Creating an experiment definition remains a ChaosCenter UI workflow in this implementation.

This incremental style keeps each decision visible. The agent should never combine discovery, approval, execution, and recovery into one opaque action.

What Failed During the Build

The setup produced several useful lessons.

Docker Desktop Kubernetes was unstable at 2 GB

The default Docker Desktop resource limit was 2 GB. ChaosCenter starts several services and MongoDB, so the Kubernetes API repeatedly became unavailable with TLS handshake timeouts.

The practical fix was to configure WSL 2 with 4 GB and 2 CPUs, then use a reduced MongoDB configuration and smaller infrastructure requests.

Standalone MongoDB broke expected DNS

A standalone MongoDB deployment removed the StatefulSet hostname expected by Litmus init containers. The compatible low-resource choice was a one-member replica set.

Helm retained a pending operation

An interrupted Helm installation left the release in pending-install. The clean recovery was:

helm uninstall chaos -n litmus
kubectl delete namespace litmus
kubectl create namespace litmus
Enter fullscreen mode Exit fullscreen mode

Then install the chart again with the corrected values.

Generated Argo labels were invalid

The generated workflow used a label value containing an unresolved template:

subject: "{{workflow.parameters.appNamespace}}_pod-delete"
Enter fullscreen mode Exit fullscreen mode

Kubernetes rejected the literal braces because they are not valid label characters. A static value such as this is valid:

subject: chaos-test_pod-delete
Enter fullscreen mode Exit fullscreen mode

This is a general lesson: do not assume every workflow field expands templates. Labels are validated before Argo can necessarily substitute workflow parameters.

The first MCP run was queued and rejected

The MCP request itself succeeded and returned a success response, but the resulting workflow was rejected by Kubernetes because of the invalid label. That distinction matters:

MCP transport: working
ChaosCenter API: reachable
Infrastructure: confirmed
Workflow validation: failed
Enter fullscreen mode Exit fullscreen mode

A successful MCP request does not guarantee that the generated chaos workflow will execute successfully. Always inspect the run status and Kubernetes events.

Keep Local Validation Repeatable

The repository includes an optional helper for restarting the local validation environment:

powershell -ExecutionPolicy Bypass -File .\scripts\start-litmus-demo.ps1
Enter fullscreen mode Exit fullscreen mode

It:

  1. Starts Docker Desktop if necessary.
  2. Selects the docker-desktop Kubernetes context.
  3. Waits for the local node.
  4. Waits for Litmus deployments.
  5. Verifies or recreates the local validation workload.
  6. Starts the 9091 frontend port-forward.
  7. Starts the 9002 GraphQL port-forward.

Stop the port-forwards without deleting Kubernetes resources:

powershell -ExecutionPolicy Bypass -File .\scripts\stop-litmus-demo.ps1
Enter fullscreen mode Exit fullscreen mode

This makes normal Windows or Docker Desktop restarts cheap. A Docker Desktop Reset Kubernetes cluster is different: it deletes the local Kubernetes resources and requires reinstalling ChaosCenter and the infrastructure manifest.

References and Useful Links

For the complete workspace implementation, the relevant files are:

  • .github/agents/litmuschaos.agent.md
  • .github/skills/litmus-experiment-runner/SKILL.md
  • .vscode/mcp.json
  • litmus-mcp-server/MCP_CHAT_GUIDE.md

Attribution

This project and tutorial integrate and build on:

The MCP server integration, custom agent instructions, skill workflow, restart scripts, and tutorial content in this repository are custom project work. Please consult each upstream project's repository and license before redistributing code, documentation, or generated experiment assets. LitmusChaos, Kubernetes, Docker, GitHub Copilot, VS Code, and related product names are trademarks or project names of their respective owners.

The custom agent instructions, skill workflow, restart scripts, and supporting documentation in this tutorial were created for this example. Replace local IDs, endpoints, credentials, and infrastructure details with your own values before publishing or sharing the repository.

Security Notes

Do not publish:

  • .env
  • API tokens
  • Infrastructure access keys
  • Downloaded infrastructure manifests containing credentials
  • Private project or account identifiers unless intentionally redacted

The .env file is ignored by Git, but a Git ignore rule does not protect a token that has already appeared in chat logs, screenshots, terminal transcripts, or public commits. Rotate any token that was exposed during testing.

For a team or public blog, replace all real values with placeholders:

LITMUS_PROJECT_ID=<project-uuid>
LITMUS_ACCESS_TOKEN=<redacted-token>
DEFAULT_INFRA_ID=<infrastructure-uuid>
Enter fullscreen mode Exit fullscreen mode

Final Takeaways

The custom Copilot agent is the central design artifact. LitmusChaos remains the chaos platform, Kubernetes remains the execution environment, and MCP provides the API bridge. The agent supplies the operational discipline around those systems.

MCP does not replace Kubernetes or ChaosCenter. It adds a conversational control layer over existing APIs and workflows, while the custom agent turns that capability into a repeatable and safer SRE and resilience-engineering workflow.

The most reliable pattern is:

preflight -> inspect -> confirm -> execute -> poll -> verify recovery
Enter fullscreen mode Exit fullscreen mode

The preflight and confirmation steps are not ceremony. They are what prevent a natural-language request from becoming an accidental production outage.

For a small local machine, resource sizing is part of the experiment design. A constrained target and a restart helper make local validation repeatable without pretending that a laptop-sized cluster is production infrastructure.

The reusable pattern is broader than LitmusChaos:

custom agent policy
  +
MCP tools
  +
domain-specific skill
  +
explicit confirmation
  =
conversational operations with guardrails
Enter fullscreen mode Exit fullscreen mode

Top comments (0)