DEV Community

Cover image for CI Load Test for http-echo using Kind and k6
🚀 Vu Dao 🚀
🚀 Vu Dao 🚀

Posted on

CI Load Test for http-echo using Kind and k6

#simflexcloud #k8s #kind #k6

Assignment

Build a CI Load Test for http-echo applications using Kind (Kubernetes in Docker) and the k6 load testing tool.

Solution Overview

This project implements a comprehensive load testing solution using:

  • Kind: Creates a multi-node Kubernetes cluster for realistic testing environments
  • NGINX Ingress Controller: Routes HTTP requests to different services based on hostnames
  • k6: Performs load testing with detailed metrics and performance analysis

Architecture 

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   k6 Load Test  │───▶│  NGINX Ingress   │───▶│  http-echo apps │
│                 │    │   Controller     │    │                 │
│ - foo.localhost │    │                  │    │ - foo service   │
│ - bar.localhost │    │  (Port 80/443)   │    │ - bar service   │
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                │
                                ▼
                       ┌──────────────────┐
                       │   Kind Cluster   │
                       │                  │
                       │ - Control plane  │
                       │ - Worker node    │
                       └──────────────────┘
Enter fullscreen mode Exit fullscreen mode

Project Structure 

kind-http-echo-test/
├── README.md                   # This documentation
├── kind-config.yaml           # Kind cluster configuration
├── k8s/                       # Kubernetes manifests
│   ├── http-echo-foo.yaml     # Foo service deployment
│   └── http-echo-bar.yaml     # Bar service deployment
└── loadtest/                  # Load testing scripts
    └── load-test.js           # k6 load test script
Enter fullscreen mode Exit fullscreen mode

Prerequisites

Ensure you have the following tools installed in your CI environment:

CI Workflow

This project is designed for automated CI/CD pipelines that trigger on each pull request to the default branch. The workflow consists of four main stages executed sequentially:

Stage 1: Environment Setup

Purpose: Prepare the CI runner with all necessary tools and dependencies, supporting both AMD64 and ARM64 agents.

Key Activities:

  • Checkout the source code from the repository
  • Install k6 load testing tool via package manager
  • Download and install Kind for local Kubernetes clusters
  • Set up kubectl for cluster management
  • Configure necessary permissions and environment variables

Stage 2: Cluster Provisioning

Purpose: Create and configure a complete Kubernetes environment for testing.

Key Activities:

  • Create a multi-node Kind cluster using the project configuration
  • Install NGINX Ingress Controller for request routing
  • Wait for all control plane components to become ready
  • Deploy the http-echo applications (foo and bar services)
  • Verify all pods are running and services are accessible
  • Configure ingress rules for hostname-based routing

Stage 3: Load Testing

Purpose: Execute comprehensive load testing against the deployed services.

Key Activities:

  • Verify endpoint availability with health checks
  • Execute k6 load test script with predefined scenarios
  • Monitor real-time performance metrics during testing
  • Generate detailed test results in JSON format
  • Track custom metrics for both foo and bar services
  • Capture response times, error rates, and throughput data

Test Configuration:

  • Ramp-up: 10s to reach target load
  • Sustained load: 30s at peak virtual users
  • Ramp-down: 10s to zero load
  • Performance thresholds: P95 < 500ms, Error rate < 10%

Stage 4: Results Analysis

Purpose: Validate performance requirements and clean up resources.

Key Activities:

  • Parse JSON test results for key performance indicators
  • Validate against predefined performance thresholds
  • Generate human-readable test summary
  • Fail the pipeline if performance criteria are not met
  • Upload test artifacts for later analysis
  • Clean up the Kind cluster to free resources
  • Provide detailed feedback on test outcomes

Success Criteria:

  • All requests complete successfully (< 10% error rate)
  • Response times meet SLA requirements (P95 < 500ms)
  • Both services perform within acceptable ranges
  • No infrastructure errors during testing

🚀 Load Testing Configuration

The load-test.js script provides comprehensive load testing capabilities with the following configuration:

📊 Test Execution Stages

The load test follows a three-phase approach for realistic performance assessment:

Phase Duration Virtual Users Purpose
🔼 Ramp-up 10 seconds 0 → 10 users Gradual load increase to target capacity
🎯 Sustained Load 30 seconds 20 users Peak performance evaluation
🔽 Ramp-down 10 seconds 20 → 0 users Graceful load reduction

⚡ Performance Thresholds

Critical performance criteria that must be met for test success:

  • Response Time: 📈 95% of requests must complete under 500ms
  • Error Rate: ❌ Must remain below 10%
  • Service Availability: ✅ Both foo and bar services must be responsive

📈 Custom Metrics & Monitoring

Advanced monitoring capabilities for detailed performance analysis:

🎯 Host-Specific Metrics

  • Per-host Response Times: Individual performance tracking for foo and bar services
  • Load Distribution: Ensures balanced traffic across both deployments
  • Service Health: Real-time availability monitoring

🔍 Real-Time Tracking

  • Error Rate Monitoring: Continuous failure rate assessment
  • Request Distribution: Balanced load verification across services
  • Performance Degradation: Early detection of performance issues

📋 Test Results & Output

Comprehensive reporting for performance analysis:

🖥️ Console Output

  • Real-time metrics display during test execution
  • Live performance indicators and alerts
  • Immediate feedback on test progress

📄 Detailed Reports

  • JSON Report: Machine-readable results saved as loadtest-results.json
  • Per-Host Breakdown: Individual service performance analysis
  • Threshold Validation: Pass/fail status for each performance criterion

📊 Key Metrics Captured

  • Request rate (requests/second)
  • Response time percentiles (P90, P95, P99)
  • Error distribution by service
  • Host-specific performance characteristics

📊 Monitoring and Metrics

k6 Metrics Collection

Comprehensive performance data captured during load testing:

  • Request Rate: Requests per second across all endpoints
  • Response Time: Average, minimum, maximum, P90, and P95 percentiles
  • Error Rate: Percentage of failed requests with detailed error categorization
  • Host-specific Metrics: Individual performance analysis per service (foo/bar)

📋 Sample Test Results

Example output from a successful load testing execution:

{
  "Total Requests": 450,
  "Requests/sec": "9.00",
  "Request Duration": {
    "avg": "25.34ms",
    "min": "10.12ms",
    "max": "89.45ms",
    "p90": "45.67ms",
    "p95": "52.34ms"
  },
  "Failed Requests": "0.00%",
  "Foo Host Duration": {
    "avg": "24.56ms",
    "p90": "44.23ms",
    "p95": "51.12ms"
  },
  "Bar Host Duration": {
    "avg": "26.12ms",
    "p90": "47.11ms",
    "p95": "53.56ms"
  }
}
Enter fullscreen mode Exit fullscreen mode

📝 Project Completion Notes

💡 Development Experience

  • GitHub Copilot Integration:

    • Successfully leveraged GitHub Copilot (GPT-4.1) to generate code with the following prompts:
    • Create a Kind configuration for a Kubernetes cluster with 2 nodes, adding taints to the control-plane node to ensure only the ingress-nginx controller is deployed on it
    • Add a step for deploying an Ingress controller to handle incoming HTTP requests in the CI workflow
    • Create 2 http-echo deployments using the hashicorp/http-echo:1.0 image, one serving a "bar" response and another serving a "foo" response. Configure cluster/ingress routing to send traffic for the "bar" hostname to the bar deployment and "foo" hostname to the foo deployment on the local cluster (routing both http://foo.localhost and http://bar.localhost), where ingress rules are based on the request's host header
    • Generate randomized traffic loads for both bar and foo hosts and capture load testing results using k6
    • Add a CI workflow step at the beginning to modify /etc/hosts, aliasing foo.localhost and bar.localhost to 127.0.0.1
    • Create a CI workflow that triggers on each pull request to the master branch
    • Use Github-copilot (Claude Sonnet 4) to write this document

  • Kind Learning Curve: This was my first experience with Kind (Kubernetes in Docker). I spent considerable time understanding how Kind works, particularly the configuration required to expose services to localhost for external access and testing.

  • Time Investment: The complete project took approximately 5 hours to complete, including research, implementation, testing, and documentation.

Source: https://github.com/vumdao/kind-http-echo-test

Ref: Goodnotes Take Home Assignment - DevOps Engineering

Note: The solution did not meet the team's expectations.

Top comments (0)