DEV Community: LitmusChaos

Making Chaos Conversational: A Beginner-Friendly Guide to the LitmusChaos MCP Server

Pritesh Kiri — Thu, 20 Nov 2025 19:04:06 +0000

Modern software systems are becoming more distributed, more complex, and more dependent on reliability than ever before. But reliability isn’t something you bolt on at the end, it’s something you test deliberately. That’s exactly where chaos engineering comes in: by intentionally injecting controlled failures, teams can understand how their systems behave under stress and make them more resilient.

LitmusChaos has been one of the most widely adopted open-source frameworks for cloud-native chaos engineering. But even with its intuitive UI (ChaosCenter), CRDs, and CLI, we know that YAMLs and APIs can still feel intimidating, especially for teams just starting out.

So, how do we make chaos engineering easier, faster, and more accessible?

This is where the LitmusChaos MCP Server comes in.

What Is MCP and Why Does It Matter?

MCP (Model Context Protocol) is a new standard that allows AI assistants (like Claude or other LLM-powered tools) to communicate with external systems through structured, well-defined tools.

Think of MCP as a bridge that lets your AI assistant:
• List chaos experiments
• Run experiments
• Stop them
• Check statuses
• Explore infrastructures
• Build probes

Using just simple natural language.

So instead of writing YAML or navigating multiple UI screens, you can simply ask:

“Run a pod-delete experiment in my staging environment.”

And the MCP Server will take care of it.

Why Did LitmusChaos Build an MCP Server?

Chaos engineering is powerful, but sometimes feels “too technical” or “too risky” for new practitioners. The LitmusChaos MCP Server solves this problem by:

Lowering the barrier to chaos. Anyone on your team can run chaos with natural language — no CRDs required.
Improving speed and accessibility. Fast discovery, instant experiment triggers, quick monitoring — all via an AI assistant.
Integrating chaos into existing workflows. Teams already using AI agents can now tie chaos steps directly into operational workflows.
Enforcing safe, scoped access. MCP tools only expose specific, controlled actions. Tokens and namespaces ensure safe execution.
Making chaos collaborative. Experiment reviews, run summaries, and probe outputs become conversational and easy to share.

Ultimately, this brings chaos engineering closer to everyday development and reliability workflows.

Check out our LitmusChaos MCP Server YouTube Playlist for a detailed guide.

Setting Up the LitmusChaos MCP Server

Below is a simple guide to getting the MCP Server running on your cloud desktop or local system.

Prerequisites

You will need:
• Go 1.21 or newer
• Access to an existing LitmusChaos ChaosCenter
• A valid project and API token
• Any MCP-enabled AI client (e.g., Claude Desktop)

Clone the repository

git clone https://github.com/litmuschaos/litmus-mcp-server.git
cd litmus-mcp-server

Build the binary

make build

Getting Your Credentials

Chaos Center Endpoint: URL of your LitmusChaos installation
Project ID: Found in Chaos Center project settings
Access Token: Generate from Chaos Center → Settings → Access Tokens

Connecting it with your MCP Client (Using Claude here)

{
  "mcpServers": {
    "litmuschaos": {
      "command": "/path/to/litmuschaos-mcp-server",
      "env": {
        "CHAOS_CENTER_ENDPOINT": "http://localhost:8080",
        "LITMUS_PROJECT_ID": "your-project-id",
        "LITMUS_ACCESS_TOKEN": "your-token"
      }
    }
  }
}

Available Tools

The server provides 17 comprehensive tools for chaos engineering operations:

Experiment Management

list_chaos_experiments - List all chaos experiments with filtering
get_chaos_experiment - Get detailed experiment information
run_chaos_experiment - Execute experiments immediately
stop_chaos_experiment - Stop running experiments

Execution Monitoring

list_experiment_runs - List experiment execution history
get_experiment_run_details - Get detailed run information with logs

Infrastructure Management

list_chaos_infrastructures - List all registered infrastructures
get_infrastructure_details - Get detailed infrastructure information
register_chaos_infrastructure - Register new Kubernetes infrastructures

Environment Organization

list_environments - List all environments
create_environment - Create new environments for the organization

Resilience Validation

list_resilience_probes - List all configured resilience probes
create_resilience_probe - Create HTTP, CMD, K8s, or Prometheus probes

Discovery & Analytics

list_chaos_hubs - List available ChaosHubs
get_chaos_faults - Browse available chaos faults
get_experiment_statistics - Get comprehensive platform statistics

Examples

These are real-world prompts you can use once your MCP server is connected:

List all experiments

Show me all chaos experiments available in my staging environment.

Trigger an experiment

Run the pod-delete-basic experiment and share the run ID.

Check experiment status

Show me the timeline and probe results of run <RUN_ID>.

Stop running chaos

Stop the currently running pod delete experiment.

Explore ChaosHub

List all Kubernetes pod-level faults from the ChaosHub.

Build a probe

Create an HTTP probe that checks /health returns 200 within 2 seconds.

Add a new environment

Create a new environment called chaos-lab.

Why use MCP?

As systems become more distributed and reliability becomes more critical, we need chaos engineering tools that integrate into everyday workflows.

The LitmusChaos MCP Server delivers:

Ease of use → chaos through natural language
Faster adoption → no need to learn CRDs or YAML
Better collaboration → experiments become team-friendly

The LitmusChaos MCP Server is more than just a technical integration, it’s a new way of practicing chaos engineering. Making chaos accessible through natural language removes barriers and encourages teams to adopt reliability as a mindset, not a one-time activity.

LitmusChaos October Highlights - Hacktoberfest, Meetups & More!

Pritesh Kiri — Wed, 05 Nov 2025 04:14:40 +0000

October has been one of the most exciting and eventful months for the LitmusChaos community. With Hacktoberfest 2025 in full swing, we went all in, from hosting community-driven events to supporting contributors, shipping releases, and celebrating amazing open-source contributions.

Hacktoberfest 2025 (The Month of Open Source)

Hacktoberfest has always been special for LitmusChaos, and this year, we made it even bigger. The community came together to contribute to LitmusChaos and Litmus Docs, improving the project’s security, user experience, and accessibility with Litmus 4.0.

Here’s what we achieved together in October:

GitHub Stars: 112 new stars
LitmusChaos Issues: 42 created → 27 closed (PRs merged), 15 in progress
Litmus Docs Issues: 34 created → 31 closed (PRs merged), 3 in progress
New Contributors
- 23 new contributors to LitmusChaos
- 18 new contributors to Litmus Docs

These numbers reflect the incredible passion our contributors showed throughout Hacktoberfest. A big shoutout to everyone who helped strengthen LitmusChaos through code, docs, and discussions!

Release Spotlight: LitmusChaos MCP Server

One of the biggest highlights of October was the release of the LitmusChaos MCP (Model Context Protocol) Server, a major step toward making Chaos Engineering more accessible and intelligent.

The Litmus MCP Server allows users to interact with chaos experiments using natural language, bridging the gap between developers and automation. Imagine simply asking, “Run a pod-delete experiment on the checkout service,” and seeing it happen seamlessly — that’s the power MCP brings to the Litmus ecosystem.

Adoption Continues to Grow

LitmusChaos continues to see strong adoption within the cloud-native ecosystem.

October Installation Trends:

Mid-month spike: around 5.2K installations (Oct 6–7)
End-of-month range: roughly 3.8K–4K installations
Average daily installations: around 3.5K–4K

It’s heartening to see teams worldwide continue to adopt ChaosCenter to build more resilient systems.

New Blog of the Month

We published a new blog this month:

Pritesh Kiri for LitmusChaos

Oct 28 '25

Making Chaos Engineering Accessible: Introducing the LitmusChaos MCP Server

#mcp #chaosengineering #litmus #litmuschaos

Comments

3 min read

The post dives into how the Model Context Protocol (MCP) is bringing natural language interfaces to chaos engineering, making it easier than ever to design and execute experiments using LitmusChaos.

Growing Our Content & Community Reach

We’ve been consistent in sharing knowledge, updates, and community stories across our platforms.

YouTube

13 new videos published:

Community & Contributor Calls
Getting Started with Litmus” — a 10-part video series
16 new subscribers joined our channel

Twitter

+12 new followers and some great engagement on Hacktoberfest updates

+51 new followers — welcome to all the new faces joining the chaos community!

Slack

42 new members joined our community Slack Community

October Offline Meetups

We hosted two exciting in-person meetups in October to celebrate Hacktoberfest and connect with our amazing contributors!

📍 LitmusChaos x Hacktoberfest Meetup — Bangalore (Oct 4)

We kicked off October with a bang at the Harness Bangalore office, where 130+ attendees joined us to celebrate open source and learn about chaos engineering.

📍 Hacktoberfest Hyderabad Meetup — in Collaboration with DigitalOcean (Oct 11)

Next up, we partnered with DigitalOcean for the Hyderabad Hacktoberfest meetup, proudly sponsored by Harness.

Community & Contributor Calls

Our monthly calls were buzzing with energy and participation!

Community Call

Topics we discussed:

Release of Litmus v3.22.0
Active Hacktoberfest activities and contributions
Introduction to Litmus MCP

Contributor Call

Highlights included:

Recap of community contributions during Hacktoberfest
Open PRs discussion and shoutouts to active contributors
Contributor best practices and guidelines

These calls continue to be a great space for collaboration, feedback, and celebrating progress together.

Wrapping Up

October was nothing short of amazing — full of learning, contribution, and connection.
From new contributors joining the fold to community-driven meetups and continuous adoption growth, LitmusChaos continues to thrive, powered by its community ❤️

As we move into November, we’re excited to keep the momentum going. More releases, new experiments, and continued community engagement await!

🔗 Join the conversation on Slack and follow us on LinkedIn, YouTube, and Twitter for the latest updates.

Making Chaos Engineering Accessible: Introducing the LitmusChaos MCP Server

Pritesh Kiri — Tue, 28 Oct 2025 03:56:16 +0000

Chaos Engineering has always been about one thing: building resilient systems. But let’s be honest, while the idea sounds great, getting started with chaos experiments has never been easy.

YAMLs, CLI commands, complex setups, and a fear of “what if I break something?” often stop developers and SREs before they even begin. Even with tools like LitmusChaos, which simplify a lot of this, there’s still a learning curve, especially for those new to resilience testing.

That’s exactly what the LitmusChaos MCP Server aims to change.

The Challenge with Chaos Engineering

Chaos Engineering helps you intentionally inject failures into your systems to test how they respond.

But here’s the catch for many teams: it feels too technical, too complex, or too risky.

You need to understand multiple CRDs and YAML definitions.
You have to set up and manage chaos infrastructures.
You have to know how to interpret experimental results.

And if you’re a developer or DevOps engineer already juggling deployments, observability, and CI/CD, adding “run chaos experiments” to that list can feel overwhelming.

Where LitmusChaos Fits In

LitmusChaos was built to solve exactly that. To make Chaos Engineering simpler, more organized, and more collaborative.
With ChaosCenter, you get a single pane of glass to:

Run and manage chaos experiments.
Track resilience scores.
Monitor chaos infrastructures (agents).
Visualize experiment runs and results.

It has helped teams integrate chaos into their CI/CD pipelines and production workflows.

But… even with all these features, there’s still one big barrier left: You still need to know how to use it.

Now imagine if you could talk to your system and say:

“Run a pod-delete experiment on the payment service in production.”

And it just does it.

That’s exactly what LitmusChaos MCP Server enables. It turns Chaos Engineering into a conversational experience.

No YAMLs. No CLI. No UI clicks.

Just natural language.

LitmusChaos MCP Server

The LitmusChaos MCP Server is a Go-based Model Context Protocol (MCP) server that connects your AI assistant (like Claude) directly to your ChaosCenter.
It provides a complete interface to:

Manage experiments
Control infrastructures
Organize environments
Run and monitor resilience probes

All through natural language interactions.

In simple words, the MCP Server acts as a bridge between your AI assistant and your Chaos Engineering setup, letting anyone perform chaos operations just by asking.

What You Can Do With It?

Here are some examples of what the MCP Server can handle through natural language:

1. Experiment Management

“List all chaos experiments.”
“Run the pod-delete experiment on the frontend pods.”
“Stop the network latency experiment.”
You can trigger, stop, or describe chaos experiments instantly.

2. Infrastructure Operations

“List all active chaos infrastructures.”
“Show me the status of the production infra.”
You get a complete overview of all connected Litmus agents.

3. Environment Organization

“Create a new environment called staging.”
“Show me all experiments in the PROD environment.”
Organize chaos operations by environments for better structure.

4. Resilience Probes

“Create an HTTP probe that checks the payment API every 5 seconds.”
“List all existing probes in my project.”
You can validate steady-state conditions or system health with simple requests.

5. Analytics & Monitoring

“Show me all failed experiments last week.”
“What’s the resiliency score of my production environment?”
It makes chaos data easy to access and interpret.

Getting Started with MCP Server

You can get up and running in just a few minutes.
Quick setup

# Clone and build
git clone https://github.com/litmuschaos/litmus-mcp-server.git
cd litmus-mcp-server
make build

Now, just add it to your Claude or any other MCP Host, and you’re ready to talk to your ChaosCenter through natural language.

Follow this guide to learn more about setting up the Litmus MCP server.

Why It Matters

The MCP Server is more than just a new integration; it’s a mindset shift for chaos engineering.

For Developers: No more YAML confusion. Learn chaos the easy way.
For SREs: Run, monitor, and analyze chaos experiments faster.
For DevOps Teams: Automate resilience testing with AI-driven workflows.
For Organizations: Empower every engineer, not just chaos experts, to build resilient systems.

It lowers the barrier of entry so that anyone, regardless of chaos experience, can perform meaningful resilience testing.

The Future of Accessible Chaos Engineering

The LitmusChaos MCP Server marks a big step forward, where Chaos Engineering meets AI-driven accessibility.

It makes resilience testing approachable, interactive, and intuitive.
The future of chaos isn’t about running YAML files; it’s about asking the right questions, getting answers, and improving system resilience effortlessly.

So go ahead, try it out, connect your assistant, and let’s make Chaos Engineering something everyone can do.

LitmusChaos Community Highlights - September 2025 Recap

Pritesh Kiri — Thu, 23 Oct 2025 11:00:52 +0000

September was an exciting month for the LitmusChaos community!

From insightful discussions in our calls to strong GitHub activity and growing engagement on social platforms, the community continued to thrive as we geared up for Litmus 4.0 and Hacktoberfest. Here’s a look at what happened in September.

GitHub updates

The community remained active on GitHub throughout September:

Stars gained: 35 ⭐
Issues: 9 total (6 open, 3 closed)
Pull Requests: 7 total (4 open, 3 merged)

The activity shows steady engagement from contributors and continued momentum in maintaining and improving LitmusChaos.

Litmus Installations

Here is the graph showing our installation chart for September:

Mid-month spike: Around 1.2K installations
End-of-month jump: Roughly 3.6K installations on Sep 30
Average daily installations: ~300–350

This surge indicates growing interest in LitmusChaos as more users start experimenting and running chaos experiments.

New Videos & Resources

We released 4 new videos in September to help the community stay informed and get started with LitmusChaos:

1) “New Look of Litmus” – a fresh tour of Litmus features

2) Community Call – discussions on releases and upcoming events

3) “Learn Litmus in 20 Minutes” – a quick-start tutorial

4) Contributors Call – recap and guidance for contributors

Community and Contributor Calls Recap

Community Call Highlights

Litmus v3.21.0 release updates
Upcoming events: Hacktoberfest & meetups
GitHub activities & contributions
Sneak peek into Litmus 4.0 and Litmus MCP

Watch the full video here

Contributors Call Highlights

Recap of community contributions so far
Review of open PRs and recent contributions
Best practices for contributing
Hacktoberfest introduction and contribution planning
How to start contributing to LitmusChaos and follow the contributor guidelines

Watch the full video here

Social & Community Growth

September also saw engagement across our social platforms:

LinkedIn: 20+ new followers
Slack: 15+ new members
Twitter: 5+ new followers

We shared the new videos and key discussions across these channels, encouraging more people to join the community and participate in LitmusChaos.

Join the Litmus Community

If you’re passionate about open source and want to help improve developer experience, we’d love to have you!
Here’s how you can get involved:

Star the LitmusChaos GitHub repo — it really helps!
Join our Slack community to chat, ask questions, or collaborate.
Subscribe to our YouTube channel for tutorials, community calls, and updates. Recently released the "Getting Started with LitmusChaos" Playlist.

What’s next?

October is shaping up to be another exciting month:

Hacktoberfest is around the corner. Get ready to contribute!
More updates from Litmus 4.0 and Litmus MCP
Continued growth in community engagement and learning resources

Stay tuned, keep experimenting, and let’s continue building chaos together! ⚡

LitmusChaos is Heading to KubeCon + CloudNativeCon Europe 2025 – Join the Chaos!

Sayan Mondal — Wed, 26 Mar 2025 05:39:15 +0000

March 26, 2025
Project post by Sayan Mondal, Community Manager, LitmusChaos

The cloud native community is gearing up for one of the most anticipated events of the year, KubeCon + CloudNativeCon Europe 2025, happening in the vibrant city of London from April 1 - 4, 2025. And guess what? LitmusChaos is back and thrilled to be part of the action once again!

No matter if you’ve been doing chaos engineering for a while or you’re just starting to explore the world of reliability, we’re excited to connect with you. We’ve got some fun and insightful talks lined up, hands-on demos at our booth, and lots of conversations waiting to happen. Here’s a sneak peek at what LitmusChaos has planned for KubeCon EU 2025

🔥 Chaos Talks You Won’t Want to Miss

We’re excited to be part of the conference schedule with two impactful sessions that dive deep into chaos engineering and reliability practices in cloud native ecosystems.

Lightning Talk: Project Lightning Talk: Chaos Unleashed: LitmusChaos and Its Journey Towards CNCF Graduation - Vedant Shrotria, Maintainer 🗓 Tuesday April 1, 2025 🕐 14:22 - 14:27 BST 📍 Platinum Suite | Level 3

Join us to explore the essence of LitmusChaos and its roadmap to becoming a CNCF Graduated project

Session Talk: Driving Chaos Engineering Forward: What’s New and Next With LitmusChaos - Sarthak Jain & Saranya Jena, Harness 🗓 Thursday, April 3, 2025 🕐 16:00 - 16:30 BST 📍 Level 3 | ICC Capital Suite 7-9

Explore the latest advancements in chaos engineering for cloud-native systems. This session will cover key updates from recent releases, including enhanced resilience testing, observability, and scalability features, while showcasing how they address real-world challenges faced by Developers and SREs.

📍 Visit Us at the LitmusChaos Booth

Want to get hands-on with chaos engineering? Come meet the maintainers, explore live demos, and learn how LitmusChaos can integrate into your SRE and platform engineering workflows.

Booth Details:

Kiosk Number: 10A
Location: Excel London | Level 1 | North Hall | Entrance N8-N9
Project Pavilion Hours (BST):

Wednesday, April 2: 15:30 – 19:45
Thursday, April 3: 14:00 – 17:00
Friday, April 4: 12:30 – 14:00

We’ll be showcasing new features, running live chaos experiments, and walking through use cases from real production environments. If you’ve been curious about how chaos fits into GitOps, platform engineering, or if you even need an additional layer in your reliability journey, this is the place to be.

🤝 Project Engagement Opportunities

There are multiple ways you can engage with us at KubeCon EU with the latest project engagement opportunities for the community as well as for projects.

☕ Meet us at the Project Pavilion Lounge

The Project Pavilion Lounge, situated at the heart of the Pavilion, offers projects and attendees a retreat for unwinding and networking in a laid-back environment. Come chat with us at booth 10A, where the chaos engineers will be hanging.

☕ Let’s Chat at Coffee Stations:

Coffee stations will be conveniently located on both sides of the Pavilion, offering a refreshing array of hot beverages to energize and invigorate attendees throughout the event.

📍 Ad-Hoc Meeting Tables:

Adjacent to the Project Pavilion, designated tables will be available for projects interested in hosting impromptu on-site meetings. These tables operate on a first-come, first-served basis, providing projects and event attendees the opportunity to convene and collaborate throughout the duration of the week.

Want to plan a quick sync or deep-dive chat with us during the event? Drop a note at sayan.mondal@harness.io and we’ll make sure we catch up at KubeCon EU 2025!

🚀 Come Along and Learn More About Chaos Engineering

Here’s your opportunity to meet the LitmusChaos team, grab some cool LitmusChaos stickers, and ask questions in person. Whether you’re a Chaos Engineering expert, user, enthusiast, or just getting started, the LitmusChaos community is here to help you get the most out of your reliability journey.

At our booth, you’ll get guidance on:

Getting Started with Chaos Engineering
Getting Started with LitmusChaos
Learning more about Chaos Engineering adoption
Questions on LitmusChaos (blockers, doubts, next steps, choosing the right experiments)
Latest updates and enhancements in LitmusChaos
LitmusChaos Roadmap
Participating in the LitmusChaos community
Becoming a contributor or a future maintainer

🌐 Stay Connected with LitmusChaos

Whether you're attending KubeCon EU 2025 or following the updates from afar, there are plenty of ways to stay connected and be part of the LitmusChaos community.

We’re an open, welcoming space for chaos engineering enthusiasts, platform engineers, reliability folks, contributors, or anyone simply curious to learn more about building resilient systems. And we’d love to have you join us!

Here’s how you can get involved:

💻 Visit the LitmusChaos website to learn about the project, use cases, architecture, and everything chaos engineering.
💬 Join the conversation on the #litmus channel in the Kubernetes Slack, ask questions, share ideas, or just hang out with the community.
🛠 Contribute to the project! Check out our Contributing Guide and start with an issue, suggestion, or PR — every contribution counts.
📺 Subscribe to the LitmusChaos YouTube Channel for community meetings, demos, tutorials, and project updates.
🐦 Follow @LitmusChaos on Twitter/X or @litmuschaos on LinkedIn for the latest news, release updates, and community shoutouts.
✍️ Explore the LitmusChaos blog for stories, tutorials, and deep dives. Want to share your own experience with Litmus? Publish a blog on DEV.to and tag it with #litmuschaos, we’d love to read it!

See you in London!

Best Practices for Creating Custom Chaos Experiments

Smriti S — Tue, 25 Feb 2025 07:26:09 +0000

Chaos Engineering is all about breaking things with a purpose- to build resilient systems. Many chaos engineering tools offer predefined faults, but where do you start if you want to build custom chaos experiments to suit your infrastructure? And how do you design chaos experiments that are both effective and safe? Let’s break it down.

1. Define Clear Objectives

Answer this question: What are you testing?
Chaos experiments should have a specific goal. Before creating a custom experiment, ask:

What failure scenario are we simulating? (CPU spikes, network latency, disk failures, etc.)
What is the expected behavior of the system? (Auto-recovery, failover, degraded performance)
What metrics define success or failure? (Response time, error rate, downtime) For example: "I want to test how my application’s microservices handle sudden node failures and verify if traffic automatically shifts to healthy nodes."

2. Choose the Right Disruption

Answer this question: For your application/system, what faults are relevant? What resources should be disrupted?
All faults have different consequences (or blast radius). It is recommended to start with low-risk faults (such as network latency, CPU/memory spike, pod restarts) before moving towards destructive tests (such as node failures, killing database instances, packet loss in critical services).
Start executing chaos experiments in a staging environment, analyze results, and gradually increase the chaos impact to different environments (QA, pre-production, production).

3. Ensure RBAC Is in Place

Chaos experiments are disruptive if not controlled. Use RBAC policies to prevent unauthorized users from running high-risk experiments. This way, only authorized users/engineers will have the permissions to execute chaos experiments.
The manifest below describes the rules for a user (“chaos-engineer”) who can perform the operations (create, get, and list) on specified resources.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: chaos-engineer
rules:
  - apiGroups: ["litmuschaos.io"]
    resources: ["chaosengines", "chaosexperiments"]
    verbs: ["create", "get", "list"]

4. Monitor System Behavior

When a chaos experiment is defined, it is important to audit what goes on when the experiment is in execution. Set up monitoring/observability tools to monitor CPU, memory, and response time, as well as detect unusual errors. This way, you can answer some of the questions such as:

Did auto-scaling trigger correctly?
Did users experience downtime?
Were ale rts sent to engineers?

5. Recovery Plan in-place

Sometimes, chaos experiments may not go as expected, and may disrupt to a higher extent than expected. To address such cases, rollback and recovery should be set up.
For example, Kubernetes self-healing, restarting pods after executing the experiment, and so on.
The manifest below uses Kyverno to auto-restart pods after chaos so that the affected pods don’t persist and affect other resources.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restart-failed-pods
spec:
  validationFailureAction: Enforce
  rules:
    - name: restart-on-failure
      match:
        resources:
          kinds: ["Pod"]
      mutate:
        patchStrategicMerge:
          spec:
            restartPolicy: Always

Conclusion

Define clear objectives → Know what failure you’re testing.
Start small, then scale → Begin with low-risk chaos.
Use RBAC and monitoring → Keep experiments safe
Monitor system behavior → Audit what goes on when the experiment runs.
Always have a recovery plan → Have rollback/recovery plans in case something doesn’t go as expected.

Want to create custom experiments but don’t know where to start? Join the Litmus Slack channel, check out the official documentation and blogs.

The Hidden Risks of Relying Solely on Testing: Why Chaos Engineering is Essential

Smriti S — Thu, 06 Feb 2025 04:46:35 +0000

This blog explores the challenges organizations face when relying solely on testing to validate deployments, highlights how chaos engineering can bridge these gaps, and showcases various adopters of chaos engineering practices along with their use cases.

Organizations are rolling out changes almost everyday with the help of CI/CD. You can perform automated, manual, unit and integration tests to validate features before deployment. But what if your system encounters unexpected failures?

What if there are network disruptions, application crashes or resource exhaustions that would impact the application resulting in downtime or cascading failures?

These unforeseen failures can’t be addressed using testing because traditional testing methodologies aren’t designed to simulate such complex, real-world conditions.
This is where chaos engineering comes into play.

You can address the gaps mentioned earlier by incorporating chaos engineering experimentation in your application to build resilient and reliable systems.

Described below are some organizations (by sector) that leverage chaos engineering to enhance their cloud-native applications and operations:

Technology & E-Commerce: The Challenges and Solutions

1. Flipkart: Handling High-Traffic Events Without Downtime
Challenge: During peak shopping events, Flipkart experienced traffic spikes that led to performance degradation and downtime. Traditional load testing failed to capture the full complexity of real-world failures like network congestion and database latency.
Impact: Shoppers faced checkout failures, slow page loads, and abandoned carts, affecting revenue and customer satisfaction.
Solution: Flipkart integrated chaos engineering to simulate failures in production-like environments, testing how microservices and databases handled stress conditions. This approach helped them optimize auto-scaling strategies and build robust failover mechanisms, ensuring seamless shopping experiences.

2. Delivery Hero: Ensuring Reliability in Food Delivery Services
Challenge: Delivery Hero operates a high-demand food delivery service, where real-time order processing and delivery tracking are critical. Network failures and API downtime led to order failures and frustrated customers.
Impact: Customers faced delays or lost orders, affecting restaurant partnerships and overall brand trust.
Solution: Using chaos engineering, Delivery Hero injected failures into their APIs, databases, and network connections to identify weak points. By proactively fixing these, they improved system redundancy, reduced downtime, and ensured smooth operations even during peak hours.

3. Talend: Improving Data Pipeline Resiliency
Challenge: Talend, a data integration company, processes massive datasets across multiple cloud environments. Issues like database failures or unexpected API rate limits disrupted data workflows, affecting analytics and reporting.
Impact: Businesses relying on Talend’s data pipelines faced inconsistencies in analytics, leading to misinformed decisions.
Solution: Talend used chaos engineering to introduce controlled disruptions in their ETL (Extract, Transform, Load) processes. This approach helped them refine retry logic, optimize failover configurations, and ensure consistent data processing even under failure conditions.

4. Kitopi: Enhancing Reliability in Cloud Kitchens
Challenge: Kitopi, a cloud kitchen platform, depends on efficient order routing and delivery logistics. Unforeseen infrastructure failures, like database slowdowns or application crashes, led to delayed or missed orders.
Impact: Customers experienced long wait times, and restaurant partners faced reduced efficiency.
Solution: Kitopi adopted chaos engineering to simulate database latencies and system crashes. By analyzing system responses, they improved recovery mechanisms, reduced downtime, and ensured reliable operations for food preparation and delivery.

5. Lenskart: Ensuring E-Commerce Stability During Scaling
Challenge: As Lenskart scaled its e-commerce operations, performance bottlenecks emerged due to unpredictable user behavior and traffic surges.
Impact: Checkout failures and slow load times led to cart abandonment and lost sales.
Solution: Lenskart employed chaos engineering to test their microservices against network failures and sudden traffic bursts. These experiments helped them fine-tune their cloud infrastructure for better scalability and higher uptime.

6. iFood: Strengthening Food Delivery Operations
Challenge: iFood operates a large-scale food delivery service that depends on real-time coordination between customers, restaurants, and delivery partners. System failures disrupted order processing and driver dispatching.
Impact: Delayed or canceled deliveries hurt customer satisfaction and restaurant partnerships.
Solution: iFood used chaos engineering to test service degradations, ensuring their systems could handle API timeouts and database failures. By enhancing their error-handling strategies, they minimized order disruptions and maintained seamless delivery services.

7. Wingie Enuygun Company: Improving Online Travel & Finance Platform Resilience
Challenge: Wingie Enuygun, an online travel and finance platform, faced outages due to third-party API failures and unpredictable network issues.
Impact: Users encountered booking failures and delayed transactions, leading to loss of trust.
Solution: By integrating chaos engineering, they simulated API slowdowns and network failures, allowing them to improve fallback strategies and system stability. As a result, they delivered a more reliable user experience.

Conclusion

Organizations that do not use chaos engineering risk falling into a cycle of operational inefficiencies and poor customer experiences. By injecting controlled failures, chaos engineering helps address these challenges, thereby building robust and scalable systems.

Fintech Leader's journey to resilience with Litmus Chaos

Smriti S — Thu, 19 Dec 2024 05:40:06 +0000

In this blog, you will understand the importance of chaos engineering in Fintech and see how Wingie Enuygun has leveraged Litmus Chaos, an open-source CNCF-hosted platform to build and enhance their application’s resilience.

Why is Chaos Engineering Essential in Fintech?

In the fintech industry, resilience is crucial. Millions of users rely on financial platforms for transactions, payments, and investments. In such cases, system downtime can lead to significant financial loss, regulatory scrutiny, and loss of customer trust. The complexity of fintech applications often involve microservices, third-party integrations, and real-time data processing, making them vulnerable to unexpected failures.

What is Chaos Engineering?

Chaos Engineering addresses these challenges by intentionally injecting failure into a system before it is in production, to test its ability to withstand and recover from unplanned failures. This proactive approach allows companies to:

Mitigate Financial Risks: Detect issues before they escalate into costly outages or downtime.
Ensure Compliance: Maintain uninterrupted service to comply with strict financial regulations and SLAs.
Build Customer Trust: Provide a seamless, uninterrupted user experience, which is vital for customer confidence in financial transactions.

By testing for failures before they happen, organizations can achieve operational resilience, reduce downtime, and gain the confidence to innovate faster.

One such organization that uses LitmusChaos is Wingie Enuygun.

Wingie Enuygun Group, a leader in travel and fintech, employs chaos engineering practices to enhance the resilience of their applications. Using LitmusChaos, they conduct controlled failure experiments during quality assurance (QA) cycles in pre-production environments. This proactive approach allows them to identify and address potential system weaknesses before deployment, ensuring a resilient application and reliable user experience.

By simulating real-world failure scenarios, such as server crashes or network outages, Wingie analyzes how their application responds and recovers. This process helps in uncovering vulnerabilities that might not be evident through traditional testing methods.

Why Wingie Enuygun Uses Litmus Chaos?

Travelers expect instant access to booking platforms, and even minor disruptions can have significant consequences. Recognizing this, Wingie Enuygun incorporates Litmus Chaos Engineering to:

Identify Bottlenecks: By pinpointing performance and scalability issues, Litmus helps optimize its infrastructure to handle peak loads.
Detect Issues Early: Controlled failures reveal potential issues before they impact users, enabling the company to address problems preemptively.
Foresee Potential Errors: Proactive testing allows teams to predict and mitigate issues, reducing the likelihood of costly downtime.

This approach gives the organization a strategic advantage, allowing teams to take preventive measures that keep their systems resilient, and highly available.

How Wingie Enuygun Uses Litmus Chaos?

To maximize the benefits of chaos engineering, Wingie Enuygun has integrated Litmus into its Quality Assurance (QA) cycles. This process ensures that every update, change, or new feature undergoes chaos testing for resilience before going into the production environment. Here’s how it works:

Controlled Chaos in Pre-Production: Before releasing changes into production, controlled disruptions are introduced to stress-test the systems. These disruptions simulate real-world failures, such as server crashes or network interruptions.
Automated Resilience Testing: By automating chaos experiments, the company validates its system’s ability to recover from critical failures like network latency, resource depletion, and service interruptions.
Bug Detection and Verification: Chaos experiments force failures that might not otherwise surface while testing or in production. This approach helps identify and resolve bugs before they impact users.

By embedding chaos engineering within its QA process, Wingie Enuygun strengthens its infrastructure and builds confidence in its system’s ability to withstand adverse conditions.

Conclusion

From QA cycle integration to proactive resilience testing and custom chaos experiments, Wingie Enuygun’s use of chaos engineering has driven continuous improvement and system innovation.

By leveraging LitmusChaos, the company enhances its ability to detect issues early, optimize performance, and ensure uninterrupted service for its users by improving the resilience of the application.

LitmusChaos is joining Kubecon + CloudNativeCon North America 2024!

Sayan Mondal — Tue, 05 Nov 2024 12:10:29 +0000

Hello LitmusChaos Community! 👋

Over the past few years, LitmusChaos has evolved tremendously, becoming a leading open-source tool for Chaos Engineering within the Cloud Native ecosystem. It's been inspiring to watch our community grow, from engineers just getting started with chaos testing to large teams running complex fault scenarios across cloud and on-prem environments.

On behalf of the maintainers, we’re deeply grateful for the energy and passion each of you brings to the project. Your contributions, ideas, and feedback have fueled LitmusChaos' growth and helped tackle real-world resilience challenges in a collaborative, open-source environment. Whether it’s through insightful discussions in our Slack channels, pull requests on GitHub, or your experiences shared at events, it’s clear we’re building something special—together!

Our maintainers and contributors will be on-site, eager to meet with attendees, discuss how to get started with LitmusChaos, answer questions, and share opportunities to contribute to the project. We invite you to stop by, say hello, and learn more about how chaos engineering is transforming modern cloud-native applications.

Where to Find Us: Project Pavilion Kiosk #16A

Location: Level 1 | Halls A-C + 1-5 | Project Pavilion (Hall 1)

Kiosk: #16A

Shift: Half-shift AM schedule

Project Pavilion Hours:

Wednesday, November 13: 10:45 AM – 3:15 PM
Thursday, November 14: 10:30 AM – 1:45 PM
Friday, November 15: 10:30 AM – 12:30 PM

Our team will be at the Project Pavilion ready to talk chaos engineering, share resources, and hand out some awesome LitmusChaos stickers, candies and more. No matter your experience level—whether you’re an expert or just beginning to explore chaos engineering—this is the perfect opportunity to get personalized guidance and learn best practices for using LitmusChaos.

Project Engagement Opportunities at KubeCon NA

There are plenty of ways to connect with us beyond our kiosk! Here’s where you can find us around the venue:

Project Pavilion Lounge: Located in the heart of the Pavilion, the Lounge is a welcoming spot to relax, network, and have insightful discussions with other community members in a casual setting.
Coffee Stations: Fuel up and chat with us at the coffee stations on both sides of the Pavilion. Grab a coffee, and let’s discuss your ideas for chaos engineering!
Ad-Hoc Meeting Tables: Need more time to dive deeper into LitmusChaos? Ad-hoc meeting tables are available next to the Project Pavilion on a first-come, first-served basis, offering a chance for impromptu discussions.

If you’d like to set up a specific time to meet with us, please feel free to email matthew.schillerstrom@harness.io or sayan.mondal@harness.io, we’d love to connect at KubeCon NA 2024!

What You Can Learn at the LitmusChaos Booth

Our booth offers a hands-on opportunity to dive into chaos engineering, ask questions, and explore the latest developments in LitmusChaos. Here are some highlights:

Getting Started with Chaos Engineering
Introduction to LitmusChaos: First Steps & Best Practices
Adopting Chaos Engineering at Scale
Solving Challenges & Troubleshooting with LitmusChaos
Latest Updates and Enhancements in LitmusChaos
Roadmap for LitmusChaos
How to Get Involved as a Contributor or Future Maintainer

Whether you’re facing blockers, looking for experiment guidance, or curious about the project roadmap, our team is here to help you make the most of chaos engineering with LitmusChaos.

Connect with LitmusChaos Year-Round

Can’t make it to KubeCon NA 2024? You can still stay connected with the LitmusChaos community:

Website: Visit the LitmusChaos Website for the latest resources.
Slack: Join the conversation on #litmus in the Kubernetes Slack to learn, ask questions, and contribute.
Contributing Guide: Get started as a contributor by checking out our Contributing Guide.
YouTube: Subscribe to the LitmusChaos YouTube Channel for the latest demos and tutorials.
Twitter: Stay updated by following @LitmusChaos on Twitter.
LinkedIn: Connect with @litmuschaos on LinkedIn for latest updates.
DEV.to: Share your insights on DEV.to with #litmuschaos, or check out community-written blogs to learn from other contributors.

Introduction to k6 Load Chaos in LitmusChaos

Namkyu Park — Mon, 27 May 2024 03:06:13 +0000

In today's complex digital landscape, application resilience is crucial. Chaos engineering, using tools like LitmusChaos, intentionally introduces faults to uncover systemic issues missed by traditional tests. From now on, we can use k6, a load-testing tool with LitmusChaos. This post explores chaos engineering with LitmusChaos and demonstrates a k6 load chaos experiment.

What are chaos engineering and LitmusChaos
Introduction to k6
Injecting k6 load chaos with LitmusChaos (demo)

What are chaos engineering and LitmusChaos

What if our systems suddenly experience an outage? It's difficult to pinpoint the problem these days, especially since our systems are on Kubernetes, meaning they are microservices. While unit tests and integration tests can detect our application's weaknesses, they cannot detect weaknesses in our overall platform.

The above diagram shows the impact of resilience. Using chaos engineering can be a great way to achieve more than 90% resilience. Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production [1]. Chaos engineering involves intentionally injecting faults into a system to test its resilience. LitmusChaos makes this process easier to implement by simplifying Chaos Engineering.

LitmusChaos (CNCF incubating project) is a Cloud-Native Chaos Engineering Framework with cross-cloud support. You can use Litmus to inject controlled chaos and run chaos experiments in staging and production environments, allowing SREs to identify bugs and vulnerabilities. If you want to know more about Litmuschaos, Check out the documentation!

Introduction to k6

k6 is an open source load testing tool designed for developers to allow teams to create tests-as-code, integrate performance tests as part of the software development lifecycle, and help users test, analyze, and fix application performance issues. k6 supports various types of load testing (ex. spike, smoke, stress).

Now LitmusChaos supports k6 load testing as a chaos fault so that we can simulate load generation to the target application as a part of chaos testing on Kubernetes.

To know more about it, check out this documentation. You can also find our integration in the k6 documentation 🚀

Injecting k6 load chaos with LitmusChaos (demo)

Let us run the k6-loadgen chaos experiment. For simplicity, we will be injecting chaos into an OpenTelemetry Demo.

Prerequisites

Docker, minikube (If you use your k8s cluster, skip this)
Check out the otel-demo's Prerequisites
If you haven't installed LitmusChaos yet, check out this documentation.

Install opentelemetry demo

After installing Minikube and LitmusChaos, We now install the opentelemetry demo. All you have to do is enter the code below.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install my-otel-demo open-telemetry/opentelemetry-demo

If you want to customized a deployment setting, check out this documentation.

We cannot directly access the otel-demo service, so we are using minikube's command to get an external URL

minikube service my-otel-demo-frontendproxy --url

the result is like one below

We now access the frontend service

You can access Grafana using given URL

http://<<given_url>>/grafana

We will use Home > Dashboards > Demo Dashboard today.

Setup Probes

You can easily create a probe following this documentation. Enter a value like below.

// URL: http://my-otel-demo-frontendproxy.default.svc.cluster.local:8080
// METHOD: GET
// CRITERIA: ==
// Response Code: 200

Writing a k6 script

You don't have to install a k6 engine! We just have to write a k6 script. We will use the below code and save it as script.js

import http from 'k6/http';
import { sleep } from 'k6';
export const options = {
  vus: 1000,
  duration: '30s',
};
export default function () {
  http.get('http://my-otel-demo-frontendproxy.default.svc.cluster.local:8080');
  sleep(1);
}

If you'd like to write a more professional script, you can check out this documentation.

and let's make a secret

kubectl create secret generic k6-script \
    --from-file=<<script-path>>/script.js -n <<chaos_infrastructure_namespace>>

Now that all the preliminary work is done, let's create a chaos experiment.

Let's make a chaos experiment

Click Chaos Experiments > + New Experiment to create a new chaos experiment.

After clicking the Blank Canvas and Add buttons, we can choose chaos fault in ChaosHub. We use k6-loadgen

In the Tune Fault tab, we enter the secret name and key we created before.

Lastly, We add a probe that was created before.

Okay, all done! Let's execute a chaos experiment 🚀

A few minutes later, Our chaos experiment succeeded. Congratulations 🎉

We can see the load testing result on the Grafana dashboard. Go to Dashboards > Demo Dashboard > Requests Rate for frontend by span name > frontend-proxy. We can see the result below.

Summary

k6-loadgen fault simulates load generation on the target hosts for a specific chaos duration. The effects of chaos engineering can be maximized by designing experiments with k6-loadgen like other chaos faults in LitmusChaos.

If you are interested in LitmusChaos, Join the community! You can join the LitmusChaos community on GitHub and Slack.

Thank you for reading 🙏

Namkyu Park
Maintainer of LitmusChaos
LinkedIn | GitHub

Reference

[1] PRINCIPLES OF CHAOS ENGINEERING

Strategies for Writing More Effective Tests in Golang

Namkyu Park — Mon, 08 May 2023 06:32:42 +0000

Unit tests are functions that test specific pieces of code from a program or package. The primary objective of unit tests is to check the correctness of an application, leading to better software that is more robust, has fewer bugs, and is more stable. There are many ways to test software other than unit tests. Let's look at the test pyramid.

image source

As the test pyramid metaphor suggests, unit tests are the foundation of all tests. Unit tests are the easiest to write code and the most powerful. Unit tests are especially helpful for microservices. In microservices, each system must work independently. Therefore, unit tests that allow testing against one service while mocking another are very important.

This LFX quarter I got to get my hands on LitmusChaos, a CNCF incubating opensource project that dives deep on making cloud-native chaos-engineering accessible to multiple developer personas.

In this post, I'll discuss about the testing methodologies and strategies I took to improve the code coverage as well as modify the structure (for a Go based framework) of the litmus project to make it more robust and agile for testing. Before starting, We highly recommend watching this video.

Naming Convention
Testing Structure
What to test?
Test Interface
Mocking
Table-Driven Test
Solitary Tests vs Sociable Tests
Defer vs t.Cleanup()
Dealing with Before & After tests
Do not assert an Error message
Avoid Flaky Test
Fuzz testing
Code Coverage
Test Coverage Report(UI)
CI Integration
Conclusion

Naming Convention

According to the Golang testing package, we follow these naming conventions.

func helloWorld() {} // target function
func TestHelloWorld(t *testing.T) {} // test function

type FooStruct struct {}
func (f *FooStruct) Bar() {} // target method
func TestFooStruct_Bar(t *testing.T){} // test function

Testing Structure

A good structure for all unit tests follows these,

Set up the test data
Call your method under the test
Assert that the expected results are returned

These three steps are replaced with “given”, “when”, “then” in BDD. You can make unit test codes easier if you adopt this pattern.

// example of given-when-then pattern
func TestChaosHubService_DeleteChaosHub(t *testing.T) {
  t.Run("success", func(t *testing.T) {
     // given
     findResult := bson.D{
        {"project_id", "1"},
        {"hub_name", "hub1"},
        {"hub_id", "1"},
     }

     // when
     _, err := mockService.DeleteChaosHub(context.Background(), "1", "1")

     // then
     assert.NoError(t, err)
  })
}

What to test?

Tests that are too close to the production code are not recommended. As soon as you fix your production code, You need to change the test code too(the test code will be broken)! You rather test for observable behavior. Here’s what Martin Fowler’s Blog suggests.

Think about
if I enter values x and y, will the result be z?
instead of
If I enter x and y, will the method call class A first, then call class B and then return the result of class A plus the result of class B?

We can accomplish this by subtests in the Golang testing package. We don’t have to write separate functions. Instead, use t.Run() so that we can verify the result by various inputs in one function.

// example of subtests
func TestChaosHubService_UpdateChaosHub(t *testing.T) {
  t.Run("cannot find same project_id hub", func(t *testing.T) {
     // given codes
     // when codes
     // then codes
  })

  t.Run("success : updated hub type is remote", func(t *testing.T) {
     // given codes
     // when codes
     // then codes
  })

  t.Run("success : updated hub type is not remote", func(t *testing.T) {
     // given codes
     // when codes
     // then codes
  })

  t.Run("success : updated hub type is not remote, not changed data", func(t *testing.T) {
     // given codes
     // when codes
     // then codes
  })
}

Test Interface

As previously mentioned, We need to test functions' desirable results, not all lines of production code. With subtests, Interface can help what you focus on.

The interface is like a contract that expresses desired behavior. For example, the Service interface has an AddChaosHub function.

type Service interface {
  AddChaosHub(chaosHub CreateChaosHubRequest) (*model.ChaosHub, error)
}

We have not implemented the interface yet. But We can write test code. This method is the method for adding a ChaosHub. If the request parameter is valid, Method success creates a chaoshub object and return object. If not, return the error. According to these instructions, We can write test code like below.

// example of unit test of AddChaosHub function
func TestChaosHubService_AddChaosHub(t *testing.T) {
  // given
  newHub := model.CreateChaosHubRequest{
     ProjectID: "4",
     HubName:   "Litmus ChaosHub",
  }

  t.Run("already existed hub name", func(t *testing.T) {
     // given
     findResult := []interface{}{
        bson.D{{"project_id", "3"}, {"hub_name", "Litmus ChaosHub"}},
     }
     cursor, _ := mongo.NewCursorFromDocuments(findResult, nil, nil)
     mongoOperator.On(
        "List", mock.Anything, mongodb.ChaosHubCollection, mock.Anything,
     ).Return(cursor, nil).Once()

     // when
     _, err := mockService.AddChaosHub(context.Background(), newHub)

     // then
     assert.Error(t, err)
  })

  t.Run("success", func(t *testing.T) {
     // given
     findResult := []interface{}{
        bson.D{{"project_id", "1"}, {"hub_name", "hub1"}},
     }
     cursor, _ := mongo.NewCursorFromDocuments(findResult, nil, nil)
     mongoOperator.On(
        "List", mock.Anything, mongodb.ChaosHubCollection, mock.Anything,
     ).Return(cursor, nil).Once()
     mongoOperator.On(
        "Create", mock.Anything, mongodb.ChaosHubCollection, mock.Anything,
     ).Return(nil).Once()

     // when
     t.Cleanup(func() { clearCloneRepository(newHub.ProjectID, newHub.HubName) })
     target, err := mockService.AddChaosHub(context.Background(), newHub)

     // then
     assert.NoError(t, err)
     assert.Equal(t, newHub.HubName, target.HubName)
  })
}

Mocking

LitmusChaos project adopted layered architecture.

By applying this architecture, we can see the effect of low coupling and high cohesion. Changes to GraphQL logic only require modifications to the resolver layer, and changes to business logic only require modifications to the service layer. Changes to MongoDB logic only require changes to the operator layer. Since we will be testing on all layers, there is no need to test the sub-layers of each layer, so we mock the sub-layers. In LitmusChaos, we used a library called testify for mocking.

Here’s an example. In graphql-server, ChaosHubService needs a MongoOperator to interact with MongoDB. But in unit tests, We don’t have to use real databases, We mocked MongoOperator.

// example of MockOperator (mongoDB)
type MongoOperator struct {
  mock.Mock
}
// we don't have to write real logic. Mock object's method will
// be replaced at test function.
func (m MongoOperator) Get(ctx context.Context, collectionType int, query bson.D) (*mongo.SingleResult, error) {
  args := m.Called(ctx, collectionType, query)
  return args.Get(0).(*mongo.SingleResult), args.Error(1)
}

func (m MongoOperator) Update(ctx context.Context, collectionType int, query, update bson.D, opts ...*options.UpdateOptions) (*mongo.UpdateResult, error) {
  args := m.Called(ctx, collectionType, query, update, opts)
  return args.Get(0).(*mongo.UpdateResult), args.Error(1)
}


// chaoshub_test package
// Mock object is injected instead of real object.
var (
  mongoOperator = new(mocks.MongoOperator)
  mockOperator  = dbSchemaChaosHub.NewChaosHubOperator(mongoOperator)
  mockService   = chaoshub.NewService(mockOperator)
)

func TestChaosHubService_DeleteChaosHub(t *testing.T) {
  t.Run("cannot find same project_id hub", func(t *testing.T) {
     // given
     // setup expectation by using On() function.
     mongoOperator.On(
        "Get", mock.Anything, mongodb.ChaosHubCollection, mock.Anything,
     ).Return(&mongo.SingleResult{}, errors.New("")).Once()

     // when
     _, err := mockService.DeleteChaosHub(context.Background(), "1", "1")

     // then
     assert.Error(t, err)
  })

  t.Run("success", func(t *testing.T) {
     // given
     findResult := bson.D{
        {"project_id", "1"}, {"hub_name", "hub1"}, {"hub_id", "1"},
     }
     singleResult := mongo.NewSingleResultFromDocument(findResult, nil, nil)
     mongoOperator.On(
        "Get", mock.Anything, mongodb.ChaosHubCollection, mock.Anything,
     ).Return(singleResult, nil).Once()
     mongoOperator.On(
        "Update", mock.Anything, mongodb.ChaosHubCollection, mock.Anything, mock.Anything, mock.Anything,
     ).Return(&mongo.UpdateResult{MatchedCount: 1}, nil).Once()

     // when
     _, err := mockService.DeleteChaosHub(context.Background(), "1", "1")

     // then
     assert.NoError(t, err)
  })
}

Table-Driven Test

You can check the basics of table-driven tests here. By adopting a table-driven test approach, We can reduce the amount of repetitive code compared to repeating the same code for each test and make it straightforward to add more test cases. More details on the Golang dev blog.

// example of table-driven Test
func TestChaosHubService_UpdateChaosHub(t *testing.T) {
  // given
  utils.Config.RemoteHubMaxSize = "1000000000"
  testCases := []struct {
     name    string
     hub     model.UpdateChaosHubRequest
     got     bson.D
     isError bool
  }{
     {
        name: "cannot find same project_id hub",
        hub: model.UpdateChaosHubRequest{
           ProjectID: "1",
           HubName:   "updated name",
        },
        isError: true,
     },
     {
        name: "success : updated hub type is remote",
        hub: model.UpdateChaosHubRequest{
           ProjectID: "1",
           HubName:   "updated name",
           RepoURL:   "https://github.com/litmuschaos/chaos-charts/archive/refs/heads/master.zip",
        },
        got:     bson.D{{"project_id", "1"}, {"hub_name", "hub1"}, {"hub_type", "REMOTE"}},
        isError: false,
     },
     {
        name: "success : updated hub type is not remote",
        hub: model.UpdateChaosHubRequest{
           ProjectID:  "1",
           HubName:    "updated name",
           RepoURL:    "https://github.com/litmuschaos/chaos-charts",
           RepoBranch: "master",
           IsPrivate:  false,
        },
        got:     bson.D{{"project_id", "1"}, {"hub_name", "hub1"}},
        isError: false,
     },
     {
        name: "success : updated hub type is not remote, not changed data",
        hub: model.UpdateChaosHubRequest{
           ProjectID:  "1",
           HubName:    "updated name",
           RepoURL:    "https://github.com/litmuschaos/chaos-charts",
           RepoBranch: "master",
           IsPrivate:  false,
        },
        got:     bson.D{{"project_id", "1"}, {"hub_name", "updated name"}, {"repo_url", "https://github.com/litmuschaos/chaos-charts"}, {"repo_branch", "master"}, {"is_private", false}},
        isError: false,
     },
  }

  for _, tc := range testCases {
     t.Run(tc.name, func(t *testing.T) {
        // given
        if tc.isError {
           // given
           mongoOperator.On("Get", mock.Anything, mongodb.ChaosHubCollection, mock.Anything).Return(&mongo.SingleResult{}, errors.New("")).Once()

           // when
           _, err := mockService.UpdateChaosHub(context.Background(), tc.hub)

           // then
           assert.Error(t, err)
        } else {
           singleResult := mongo.NewSingleResultFromDocument(tc.got, nil, nil)
           mongoOperator.On("Get", mock.Anything, mongodb.ChaosHubCollection, mock.Anything).Return(singleResult, nil).Once()
           mongoOperator.On("Update", mock.Anything, mongodb.ChaosHubCollection, mock.Anything, mock.Anything, mock.Anything).Return(&mongo.UpdateResult{MatchedCount: 1}, nil).Once()

           // when
           t.Cleanup(func() { clearCloneRepository(tc.hub.ProjectID, tc.hub.HubName) })
           target, err := mockService.UpdateChaosHub(context.Background(), tc.hub)

           // then
           assert.NoError(t, err)
           assert.Equal(t, tc.hub.HubName, target.HubName)
        }
     })
  }
}

Solitary Tests vs Sociable Tests

There are two terms in the unit test world, Sociable Tests and Solitary Tests. See the illustration below for an explanation of the two terms.

image source

Previously, We talked about Mocking. With Mocking, We can make all unit test codes Solitary Tests. However, if you want your code to behave based on the results of actual actions in the lower layers, you should use Sociable Tests.

For example, In graphql-server’s ChaosHub package, ChaosHubService uses chaosHubOps.GitClone() in the AddChaosHub method. The AddChaosHub method performs the git clone through a real ChaosHub url, which means that if you mock the git clone part, you need additional logic to determine if the url is valid. Also, the GetExperiment method performs file I/O operations based on the cloned repository. For these cases, Sociable Tests, which do not mock chaosHubOps.GitClone(), is more appropriate.

Defer vs t.Cleanup()

If you need to clean up the resources used by your test, use t.Cleanup(). Unlike the defer function, this function also works fine in the event of a panic. You can check the details in this link.

// Example of t.Cleanup() for cleanup resources
func TestChaosHubService_AddChaosHub(t *testing.T) {
  t.Run("success", func(t *testing.T) {
     // given codes ...
     // when : called t.Cleanup() functions before when codes
     t.Cleanup(func(){clearCloneRepository(newHub.ProjectID, newHub.HubName)})
     target, err := mockService.AddChaosHub(context.Background(), newHub)

     // then codes
  })
}

Dealing with Before & After tests

Sometimes, We need to add additional tasks before or after tests. The Golang testing package gave us a solution. TestMain function can be declared per package. You can add additional processes like below.

func TestMain(m *testing.M) {
  // pre-process
  os.Exit(m.Run())
  // post-process
}

You can use the init() function. But init() function cannot be used in After logic. So I recommend using the TestMain function rather than the init function.

Do not assert an Error message

The Error message is only for human consumption. That means, It can easily change. So, Rather than using an Error message, you can just check if the error is not nil. More details are in the following conversations.

Avoid Flaky Test

Test functions needed to be deterministic. Do not make Flaky Tests. Here are common causes of flakiness include:

Poorly written tests.
Async wait
Test order dependency
Concurrency

Fuzz testing

The unit test has limitations in that unit tests’ input must be added by the developer. Fuzz testing can test many edge cases like coding interviews so that we can prevent SQL injection, buffer overflow, and more. Fuzz testing involves injecting random data with your original test cases. You can make the Fuzz test function like below.

func Reverse(s string) string {
    // function to reverse a string
}

func FuzzReverse(f *testing.F) {
    // table-driven fuzzing
    testcases := []string{"word1", "word2", "word3"}
    for _, tc := range testcases {
        f.Add(tc) // Use f.Add to provide a seed corpus
    }
    f.Fuzz(
        func(t *testing.T, a string) { // Value of a will be auto generated and passed
            // Assert that the length of the reversed string is the same as the original
        },
    )
}

And you can run unit tests with Fuzz like below.

go test -fuzz=Fuzz

Code Coverage

We use the Golang command tool to check test coverage. It's simple but really powerful. You don't need any extra tools if you already installed Go.

Golang command tool automatically installs, builds, and tests Go programs using nothing more than the source code as the build specification.

# Check Specific package's code coverage
# in the package root
go test -cover ./...

# Check the entire backend code coverage
# in the backend module root (in my case, graphql-server)
go test --coverpkg ./... -coverprofile cover.out ./... ; \
 echo -n "total: " ; \
 go tool cover -func=cover.out | tail -1 | awk '{print $NF}'

Test Coverage Report(UI)

The Golang command tool also gave us to check coverage by HTML UI. Official guide is here. Once you execute the entire backend unit tests, You can find cover.out file. If you executed the below code, now you can see a beautiful UI that can check coverage. the green color of codes is covered by your unit tests code and the red one is not.

# Check the entire backend code coverage
# in the project root
go test --coverpkg ./... -coverprofile cover.out ./... ; \
 echo -n "total: " ; \
 go tool cover -func=cover.out | tail -1 | awk '{print $NF}'
# cover.out to cover.html
go tool cover -html=cover.out -o cover.html

CI Integration

If you adopt CI / CD pipeline to your current project, you can integrate unit testing jobs to CI pipeline. CI jobs in the GitHub actions will execute unit tests so developers do not have to run them locally. Here's a sample.

name: build-pipeline
on:
  pull_request: # this example CI job runs when you raise PR
    branches:
      - master

env:
  DOCKER_BUILDKIT: 1

jobs:
  changes:
    runs-on: ubuntu-latest

  backend-unit-tests:
    runs-on: ubuntu-latest
    needs:
      - changes
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
      - uses: actions/setup-go@v2
        with:
          go-version: "1.16"
      - name: Backend unit tests
        shell: bash
        run: |
          # cd to the backend directory
          # run your test here!
          go test -cover ./...

  docker-build-backend-server:
    runs-on: ubuntu-latest
    needs:
      - changes
      - backend-unit-tests
    steps:
      - name: Build backend server docker image
        shell: bash
        run: |
          # run docker build job after all the tests are passed

As you seen, you can run unit tests before build docker image. If unit tests failed, no further jobs will be executed.

Conclusion

Through this post, We discussed several tips of how to write efficient unit test codes. If you have any questions or suggestions, please comment below or send me an email(lak9348@gmail.com)

As I mentioned before, I worked on writing unit tests in LitmusChaos. LitmusChaos can help you find weaknesses in your project by injecting chaos.

If you are interested in LitmusChaos, Join community! You can join the LitmusChaos community on GitHub and Slack.

Thank you for reading 🙏

Namkyu Park
Contributor of LitmusChaos
LinkedIn | GitHub

Introduction to HTTP Chaos in LitmusChaos

Akash Shrivastava — Tue, 13 Sep 2022 15:58:52 +0000

This article is a getting-started guide for HTTP Chaos in LitmusChaos. We will be talking about

Introduction to LitmusChaos
How does HTTP Chaos work — Architecture
Types of HTTP Chaos Experiments
HTTP Chaos Demo

What is LitmusChaos

LitmusChaos is a toolset to do cloud-native chaos engineering. It provides tools to orchestrate chaos on Kubernetes to help SREs find weaknesses in their deployments. SREs use Litmus to run chaos experiments initially in the staging environment and eventually in production to find bugs and vulnerabilities. Fixing the weaknesses leads to increased resilience of the system.

Litmus takes a cloud-native approach to creating, managing and monitoring chaos. Chaos is orchestrated using the following Kubernetes Custom Resource Definitions (CRDs):

ChaosEngine: A resource to link a Kubernetes application or Kubernetes node to a ChaosExperiment. ChaosEngine is watched by Litmus’ Chaos-Operator which then invokes Chaos-Experiments
ChaosExperiment: A resource to group the configuration parameters of a chaos experiment. ChaosExperiment CRs are created by the operator when experiments are invoked by ChaosEngine.
ChaosResult: A resource to hold the results of a chaos experiment. The Chaos-exporter reads the results and exports the metrics into a configured Prometheus server.

For more information, you can visit litmuschaos.io or github.com/litmuschaos/litmus

Architecture

The experiments internally use two things to inject HTTP chaos and redirect traffic properly. First, it runs a proxy server that acts as a middleman and modifies the request/response per the experiment type. Second, it creates a routing rule in the network routing table using the IPtables library to redirect all incoming traffic on the targeted service port to the proxy port.

This diagram shows a request without HTTP chaos injected. The request to access Service A comes at port 80 and is forwarded to Service A to be processed

Now, after we inject HTTP chaos, the request to access Service A comes to port 80 but is forwarded to port 8000, on which the proxy server listens for requests. This is done by adding a routing rule in the routing table using IPtables. After the proxy server has modified the request, if required, it will forward the request to Service A to be processed. Now the response will follow the same path, the proxy server will modify the response if required and then send it back to the client to complete the request loop.

The proxy server is running inside the service pod and the service pod routing table is updated by running commands inside the service pod using the nsenter tool.

To facilitate the creation of a proxy server and adding rules to the routing table, a helper pod is run which uses nsenter to enter inside the target pod to run commands to achieve this.

Experiments

Currently, there are 5 different types of HTTP experiments available. They are

Let’s know more about them

HTTP Latency

HTTP latency adds latency to the HTTP requests by adding a sleep timer before sending the request forward from the proxy server. It can be used to simulate delayed responses from the APIs. To tune the latency value, use the LATENCY experiment variable and provide the value in milliseconds.

HTTP Reset Peer

HTTP Reset Peer simulates TCP connection reset error by closing the connection after a specified timeout. It can be used to simulate connection failures. To tune the timeout value, use the RESET_TIMEOUT experiment variable and provide the value in milliseconds

HTTP Status Code

HTTP Status Code can modify the status code of the response from the service as well as change the body of the response for the defined status code with a predefined template. It can be used to simulate API failures. To specify the status code using the STATUS_CODE experiment variable to tune the value. Supported values are available in the docs. You can also provide a comma-separated list of values and the experiment will select a random value from the list to use. If no value is provided then any random value from the supported values will be chosen.

You can use the MODIFY_RESPONSE_BODY variable to tune whether the response is changed with a predefined template according to the status code or not.

HTTP Modify Header

HTTP Modify Header can modify, add or remove headers from a request or response based on provided values. To specify whether you want to modify the request or response, use the HEADER_MODE variable. You can set it to request or response.

The HEADERS_MAP needs a JSON-type input. Suppose you want to add a header litmus with a value 2.12.0 then you should provide it like this {“litmus”: “2.12.0”} similarly for multiple values as well. To remove a header, you can overwrite its value to an empty string, currently removing the header key is not possible.

HTTP Modify Body

HTTP Modify Body can modify the request/response body completely. This can be used to modify API responses. You can use the RESPONSE_BODY variable to provide the overwrite value, this can be an HTML, plain text or JSON object.

Important Tuneable

These are the tuneable specific to all the HTTP Chaos experiments

Toxicity

TOXICITY can be used to provide the probability of requests being affected. Suppose you want only 50% of the requests to be affected, by setting the value TOXICITY to 50, the probability of a request getting affected is 50%. This doesn’t mean every alternate request will be affected, but each request has a 50–50 chance of being affected. In large requests count, this comes out to be around 50% of requests being affected.

Target Service Port

TARGET_SERVICE_PORT is the port of the service you want to target. This should be the port where the application runs at the pod level, not at the service level. This means if the application pod is running the service at port 8080 and we create a service exposing that at port 80, then the target service port should be 8080 and not 80, which is the port at pod-level.

Proxy Port

PROXY_PORT is the port at which the proxy server will be running. You are not required to change the default value (which is 20000) if this port is being used explicitly by any of your other services. If the experiment fails due to a port bind issue for the proxy server, you can change this value to an empty port to make it work.

Network Interface

NETWORK_INTERFACE is the interface name for the network which your service is using. The default value is eth0. If the experiment injection is failing due to a network interface error, you can use this to change it to the correct value.

Demo

Let us run the HTTP Status Code experiment. For simplicity, we will be injecting chaos into an Nginx service.

The service is running on port 80, we will be targeting this.

If we access the service, we are getting a 200 OK response with the default Nginx webpage. I will be using Postman to verify the status code.

Now, that we have the application set up on which we will be injecting chaos, let’s start creating a chaos scenario with the HTTP Status Code experiment.

Login to Chaos Centre and get to the Chaos Scenario section. Click on the Schedule a Chaos Scenario button. Select your agent and then select the chaos hub (HTTP experiments are available from ChaosHub version 2.11.0). Add a name for your scenario and move ahead. Now we are at selecting the experiments page.

We will be selecting the generic/pod-http-status-code experiment from the list of experiments. Moving ahead, we will tune the experiment variables.

Click on the pencil icon next to the experiment name to edit the experiment. Now we have to select the app to inject chaos into. The NGINX application we are using is running in the default namespace, it is of the deployment kind and has the label app=nginx. We will skip adding probes to keep it simple. The next section is to tune the experiment variables. Change the STATUS_CODE to 500 and the TARGET_SERVICE_PORT to the port of the service, in this case, it is port 80. The MODIFY_RESPONSE_BODY is a boolean to specify whether the response body should be changed to a pre-defined HTTP template according to the status code. Now we are done with tuning the required variables in this experiment, let’s move ahead and run this.

Now LitmusChaos will set up the experiment and then run it, once it starts injecting chaos we will be seeing the status code changing for the service. The output will be something similar to this.

That’s it, we have injected HTTP chaos into our application. The experiment passed because we haven’t specified any criteria to verify, we can do this using probes.

Summary

Through this article, we could understand how the HTTP chaos experiment works internally and then talk about the current types of HTTP chaos experiments available. Then we injected the HTTP Status code experiment on a sample NGINX service and saw the experiment in live action. In further tutorial blogs, I will be talking about running the other HTTP experiments as well.

You can join the LitmusChaos community on GitHub and Slack. The community is very active and tries to solve queries quickly.

I hope you enjoyed this journey and found the blog interesting. You can leave your queries or suggestions (appreciation as well) in the comments below.

Show your ❤️ with a ⭐ on our Github. To learn more about Litmus, check out the Litmus documentation. Thank you! 🙏

Thank you for reading

Akash Shrivastava

Software Engineer at Harness

Linkedin | Github | Instagram | Twitter