Using mitmproxy with k6

Mitmproxy is one of the best-known proxies used in everyday work. What sets it apart from Fiddler and Charlesproxy is that it is open source.

The tool provides us with three main modes of operation. Each of them has a different task, and their use will depend on what we want to achieve.

1. mitmproxy - an interactive, SSL/TLS-enabled proxy with a console interface for HTTP/1, HTTP/2 and WebSockets. Its main task is to track passing traffic.
2. mitmdump - a command line companion to mitmproxy. It provides functionality similar to tcpdump, allowing you to view, log and programmatically transform HTTP traffic. In other words, mitmdump is used to record, filter and aggregate network traffic.
3. mitmweb - a web interface for mitmproxy covering the point of functionality described in point 1 and 2.

Why do we need mitmproxy in k6?

One of the fundamental problems in the context of writing performance tests is capturing traffic from both mobile and web applications.

Mitmproxy's target task is to record network traffic into a suitable format, which can then be quickly transferred to a k6 scenario and then parameterized.

Installation

Installation, depending on the operating system, is done in several ways. The simplest is to use pip3 packet manager.

pip3 install mitmproxy

mitmproxy in action

The requests and responses intercepted by mitmproxy are called HTTP flows. While they can be easily converted, this is not the HAR format. Let's check step by step how.

Capture requests and responses

A simple startup of mitmproxy in console mode is done by the command:

mitmproxy

After running the above command, an interactive console will open. For the time being, no traffic is visible in it, as nothing has been sent through the proxy yet.

After configuring, for example, JMeter (in HTTP Request Defaults set as Server IP the address 127.0.0.1 and the port as 8080 [default]) or a browser to use the proxy, records will begin to appear in the console.

Flows
21:07:08 HTTPS POST …[ackbugs.com](http://ackbugs.com) /get/new_content        307    [no content]       342ms
21:07:09 HTTPS POST …[ackbugs.com](http://ackbugs.com) /get/new_content        307    [no content]       209ms
21:07:09 HTTPS POST …[ackbugs.com](http://ackbugs.com) /get/new_content        307    [no content]       241ms
21:07:10 HTTPS POST …[ackbugs.com](http://ackbugs.com) /get/new_content        307    [no content]       270ms
21:07:10 HTTPS POST …[ackbugs.com](http://ackbugs.com) /get/new_content        307    [no content]       240ms  

These are the HTTP Flows that we mentioned earlier.

Save HTTP Flows.

After generating the appropriate traffic in the console, we press the w button to go to save HTTP Flows, and then type the path to the file to be created.

: save.file @shown C:\Users\BackBugs\Desktop\flows

This will save a file named flows on the desktop. Unfortunately, it doesn't accomplish much in our case.

To get a file in .HAR format, which we can then convert to k6 script, we can't use the saved HTTP flows. This is because converting them from file level to HAR format is impossible. However, it is possible to save real-time traffic captured by mitmproxy to HAR format. This is done by mitmdump.

Converting HTTP Flows to HAR.

Files in HAR format are standard in all web browsers. In the mitmproxy repository, we can find script with which it is possible to kownersion HTTP Flows to HAR archives. It is done through the following command.

mitmdump -s ./examples/contrib/har_dump.py --set hardump=./dump.har

The console will show summaries of the requests.

[11:40:41.279] Loading script /har-dump.py
[11:40:41.284] HTTP(S) proxy listening at *:8080.
[11:40:45.051][127.0.0.1:12464] client connect
[11:40:45.142][127.0.0.1:12464] server connect k6.io:443 (18.66.233.97:443)
127.0.0.1:12464: GET https://k6.io/
              << 200 OK 514k
127.0.0.1:12464: GET https://k6.io/
              << 200 OK 514k
127.0.0.1:12464: GET https://k6.io/
              << 200 OK 514k
127.0.0.1:12464: GET https://k6.io/
              << 200 OK 514k
127.0.0.1:12464: GET https://k6.io/
              << 200 OK 514k
[11:40:47.201][127.0.0.1:12464] client disconnect
[11:40:47.202][127.0.0.1:12464] server disconnect k6.io:443 (18.66.233.97:443)
[11:40:48.702] HAR dump finished (wrote 2672545 bytes to file)

HAR conversion to k6 script.

The next step is already the target conversion of the HAR format file to k6 script. For this we use script provided in Grafana's repository.

har-to-k6 dump.har -o k6-scenario.js

After installing and using the above command, a test scenario will be generated. Unfortunately, har-to-k6 is not an ideal tool. This is not due to its primitiveness, but to its inability to cover, for example, dynamic paths in some of the applications. Therefore, such elements as parameterization or checks we have to add ourselves.

An example of how the script looks after conversion:

/*
 * Creator: mitmproxy har_dump 0.1
 * mitmproxy version mitmproxy 9.0.1
*/

import { sleep } from 'k6'
import http from 'k6/http'


export const options = {}

export default function main() {
  let response

  response = http.get('https://k6.io/', {
    headers: {
      Connection: 'keep-alive',
      Host: 'k6.io',
      'User-Agent': 'Apache-HttpClient/4.5.13 (Java/19.0.1)',
    },
  })

  response = http.get('https://k6.io/', {
    headers: {
      Connection: 'keep-alive',
      Host: 'k6.io',
      'User-Agent': 'Apache-HttpClient/4.5.13 (Java/19.0.1)',
    },
  })

  response = http.get('https://k6.io/', {
    headers: {
      Connection: 'keep-alive',
      Host: 'k6.io',
      'User-Agent': 'Apache-HttpClient/4.5.13 (Java/19.0.1)',
    },
  })

  response = http.get('https://k6.io/', {
    headers: {
      Connection: 'keep-alive',
      Host: 'k6.io',
      'User-Agent': 'Apache-HttpClient/4.5.13 (Java/19.0.1)',
    },
  })

  response = http.get('https://k6.io/', {
    headers: {
      Connection: 'keep-alive',
      Host: 'k6.io',
      'User-Agent': 'Apache-HttpClient/4.5.13 (Java/19.0.1)',
    },
  })

  // Automatically added sleep
  sleep(1)
}