Automated Load Testing in CI/CD Pipelines: Lessons from a Custom Bash Script

Introduction: Avoiding Web App Performance Potholes with Automated Load Testing

Let's face it, performance hiccups can make your web application as sticky as Joffrey Baratheon’s character on Game of Thrones—nobody likes it.
At Etsy, their engineers understood this well and decided to get proactive. Instead of relying on off-the-shelf tools for load testing, they decided to craft their own custom Bash scripts, integrating them seamlessly into their CI/CD pipelines. It’s like creating your own suit of armor, perfectly fitted to your application's needs.

Let’s dive in and build something impactful together!

In this article, we’ll explore:

• Crafting a custom Bash script for load testing web applications.
• Automating the script with Jenkins, turning it into a key component of a CI/CD pipeline.
• Enhancing the pipeline’s functionality by integrating SonarQube for code quality checks and brainstorming other ways to future-proof your workflow (funny how this just came to head now but we'll explore this area too).

Whether you’re new to DevOps or looking for creative ways to integrate performance testing into your workflows, this article offers insights, practical examples, and room for collaboration. Let’s dive in and build something impactful together.

---

The Load Testing Script

Optimizing your web application’s performance often entails making compromises between flexibility, functionality, and resource consumption. Off-the-shelf load testing tools provide many features, but they often come with a significant overhead in terms of setup, customization, and resource usage.

Why a custom script?
In many cases, the flexibility of a lightweight, purpose built solution outweighs the feature heavy options. For instance:

• Customizability: You can tailor the script to specific requirements like logging, test duration, or unique headers.
• Efficiency: A Bash script can be quickly executed on any Unix-based system without requiring additional software or libraries.
• Integration: It can seamlessly integrate with existing CI/CD pipelines, enabling automated performance testing.

The Script’s Capabilities
The script is designed to handle critical aspects of load testing while remaining user-friendly and adaptable to various use cases. Here's what it offers:

1. Input Handling
The script accepts three essential parameters:

Target URL: The web application endpoint to be tested.
Concurrency: The number of simultaneous requests to be sent.
Test Duration: The time (in seconds) for which the test will run.

validate_inputs() {
    local url=$1
    local concurrent=$2
    local duration=$3

    if [[ ! $url =~ ^https?:// ]]; then
        echo "Error: Invalid URL format. Must start with http:// or https://"
        exit 1
    fi

    if ! [[ "$concurrent" =~ ^[0-9]+$ ]] || [ "$concurrent" -lt 1 ]; then
        echo "Error: Concurrent requests must be a positive number"
        exit 1
    fi

    if ! [[ "$duration" =~ ^[0-9]+$ ]] || [ "$duration" -lt 1 ]; then
        echo "Error: Duration must be a positive number"
        exit 1
    fi
}

Why is this important?
Validating inputs ensures the script doesn’t crash or produce inaccurate results due to invalid parameters. It’s a simple yet essential step for robustness.

2. Metrics Collection
The script uses Apache Benchmark (ab) to perform the load test, collecting the following metrics:

Total Requests: Number of requests sent during the test.
Failed Requests: Number of requests that failed due to timeouts or server errors.
Response Times: Includes both mean and maximum response times.
Success Rate: Percentage of successful requests out of total requests.
The results are logged in JSON format for easy integration with monitoring tools:

cat > "$log_file" << EOF
{
    "timestamp": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")",
    "url": "$url",
    "configuration": {
        "concurrent_requests": $concurrent,
        "duration": $duration
    },
    "results": {
        "total_requests": $total_requests,
        "failed_requests": $failed_requests,
        "success_rate": $(echo "scale=2; ($total_requests-$failed_requests)/$total_requests*100" | bc),
        "mean_response_time": $mean_time,
        "max_response_time": $max_time
    }
}
EOF

output example;

{
    "timestamp": "2024-12-14T12:00:00Z",
    "url": "https://example.com",
    "configuration": {
        "concurrent_requests": 10,
        "duration": 60
    },
    "results": {
        "total_requests": 600,
        "failed_requests": 5,
        "success_rate": 99.17,
        "mean_response_time": 200,
        "max_response_time": 1000
    }
}

3. Logging
The script generates a unique log file for each test run, storing results in a dedicated directory:

setup_logging() {
    local timestamp=$(date +%Y%m%d_%H%M%S)
    local results_dir="load_test_results"
    mkdir -p "$results_dir"
    echo "${results_dir}/load_test_${timestamp}.json"
}

So the reason why this is important is to organized logging which makes it easy to compare results across multiple test runs, enabling trend analysis.

Extracting meaningful data from ab’s raw output was tricky due to its verbose nature. I used grep and awk to extract key metrics like total requests and response times.

local total_requests=$(grep "Complete requests:" "${log_file%.json}_raw.txt" | awk '{print $3}')
local mean_time=$(grep "Mean:" "${log_file%.json}_raw.txt" | awk '{print $4}')

Here's how you can run the script
./main.sh https://example.com 10 60


---

Jenkins Pipeline: Automating Load Testing

Jenkins is a powerhouse for automating DevOps workflows. With its flexibility repetitive tasks like load testing are automated and integrated seamlessly into the software development lifecycle. For this project, Jenkins not only automates the execution of our custom load testing script but also handles result archiving and workspace cleanup effortlessly.

Key Features of the Pipeline
Here’s a quick overview of how the pipeline is designed to function:

• Code Checkout:
  The pipeline pulls the latest version of the Bash script from a GitHub repository, ensuring you always use the most up-to-date code.

• Environment Preparation:
  It sets up the necessary directories and makes the script executable. This stage ensures that all prerequisites are in place before the script runs.

• Load Test Execution:
  The custom Bash script is executed with input parameters like the target URL, concurrency, and test duration, all defined in the pipeline’s environment variables.

• Result Archival:
  After the load test completes, the pipeline archives all results for easy access and further analysis.

• Post Actions:
  The pipeline includes cleanup and error-handling steps to maintain a tidy workspace and handle failures gracefully(yepp gracefully).

The complete Jenkinsfile, along with detailed comments explaining each stage, is available on my GitHub repository. Check it out here

---

The Open Question

Funny how, as I was writing this, I started wondering—could we make this pipeline even smarter? That’s when SonarQube came to mind. But can it even work with Bash scripts? I’d love to hear your thoughts!

Anticipating your responses!