DEV Community: Bartek Wilczek

Stubbing HTTP communication in E2E test with Hoverfly

Bartek Wilczek — Wed, 06 Nov 2024 14:01:07 +0000

Stubbing out communication with external services is a common practice in automated testing. It brings various advantages, for example:

reduced flakiness
faster execution
ability to test edge cases (network errors)

It is pretty easy to stub requests to remote services in unit tests, where the test runner process is executing the code under test. There's a variety of tools that hook into the HTTP client libraries and alter their behavior in runtime, making them an important tool in the tester's tool belt. Ruby, Java, Python, Node, and other have their VCR, sinon, puffing-billy, Betamax, Talkback. None of these tools however has been designed to provide a technology agnostic solution for complete distributed app stack.

For E2E testing communication stubbing is not simple because the test runner lives in a different process then the application under test. And this makes changing the behavior of the running app in runtime impossible. Or at least very, very hacky and standing in the way of E2E testing paradigm.

Additionally the application under test can be multi-process itself: consider clustered HTTP server and a separate service for processing of background jobs. In order to provide consistent behavior of the whole stack all processes should be experiencing the same responses for the same requests. Tools listed in previous paragraph won't help in this case.

Hoverfly enters the stage

The solution to this problem is making all the processes in the stack route their HTTP communication through a proxy server and let this proxy perform any manipulation on the responses, as required by the test scenario.

Hoverfly is (almost) ideal tool for this purpose. It hooks itself into the HTTP requests processing seamlessly, without needing to alter the applications under test.

Settings things up

All applications involved in E2E test scenarios have to be configured to use an HTTP proxy server for external request they make. Setting this up can vary, depending on the libraries used.

Starting Hoverfly

But first things first. Let's start a local instance of Hoverfly, that the other services will use:

docker run --name hoverfly -d -p 8888:8888 -p 8500:8500 spectolabs/hoverfly:latest

It is important to know where services under test run. If they run in docker containers Hoverfly container should be created in the same network, and referred by name hoverfly. If the services run on the host (locally) the correct address for Hoverfly service would be localhost or 127.0.0.1.

As one can see two ports are being exposed: 8500 for the actual proxy and 8888 for the admin interface and the REST API.

Configuring proxy with env variables

In ideal scenario the desired configuration should be achieved without needing to change a single line of application's code, using only environment variables. There's an (informal) standard for proxy configuration, that involves three variables:

http_proxy - holds proxy URL to be used for HTTP requests
https_proxy - holds proxy URL to be used for HTTPS requests
no_proxy - holds a coma separated list of URLs that should not go through a proxy. Typically localhost or other services from the same app stack.

As stated before our instance of Hoverfly runs at http://hoverfly:8500. Then a typical set of variables for E2E test stack would be:

export http_proxy=http://hoverfly:8500
export https_proxy=http://hoverfly:8500
export no_proxy=localhost,127.0.0.1,hoverfly,db,web,search,other,services

Most likely these would be declared in GitHub Action workflow, or docker-compose.yml, or kubernetes manifests, or wherever the stack is started.

Configuring proxy per library

Some HTTP client libraries do not respect these env vars. If this is the case the configuration will have to be provided in the code. For example node-fetch requires some tweaking:

const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');

const agent = new HttpsProxyAgent(process.env.HTTPS_PROXY);

fetch('https://example.com', { agent })
  .etc()

Fortunately, the changes are minimal and (assuming that apps under test are properly designed) should happen in just one place in the code.

Configuring SSL certificate

In order to support requests to HTTPS sites Hoverfly comes with a bundled SSL certificate. However, as this is a testing tool, this certificate is self-signed, and needs to be marked as "trusted" before the applications using the proxy will attempt to connect to secure sites.

Similarly as with proxy URL this goal can be achieved in a way transparent to the apps' code, by trusting the certificate on a global (OS) level. On debian systems it can be done with:

cp cert.pem /usr/local/share/ca-certificates/hoverfly.crt && update-ca-certificates

Not all apps however respect this approach. If this is the case with your HTTP client please refer to its documentation to find a proper solution. It shouldn't be too hard though, as often the issue can be resolved with an environment variable. For example:

Python's requests package: REQUESTS_CA_BUNDLE=cert.pem
Node's extra CA setting: NODE_EXTRA_CA_CERTS=cert.pem
Curl's extra CA setting: CURL_CA_BUNDLE=cert.pem (it should respect system settings though)

Example workflow

Once the applications are configured to use Hoverfly proxy the actual testing workflow can start. Before the tests can be executed the responses have to be prepared. In Hoverfly terms a collection of request/response pairs is called a Simulation.

There are two ways of creating simulations.

Recording a simulation

Hoverfly can work in several modes. For this article we care only about capture and simulate modes. First for recording real traffic, second for serving the pre-recorded responses, not allowing any connections to real systems.

The are several ways to set the mode to the required value. First is using the admin panel at http://localhost:8888.

Another way is using CURL and REST API:

curl -X PUT -H "Content-Type: application/json" -d '{"mode": "capture"}' http://localhost:8888/api/v2/hoverfly/mode

The last way is calling the REST API from the code. The code examples in this article use TypeScript and @bwilczek/hoverfly-client package.

import { Client } from "@bwilczek/hoverfly-client"

const client = new Client("http://localhost:8888")
await client.setMode({mode: 'capture'})

After the mode has been set Hoverfly is ready to record the traffic going through it. It's time to let the apps under test perform their requests and let Hoverfly record them. Typically app user (or acceptance tester, so most like you, dear reader) will just one the app UI and click through the functionality in question.

When the scenario is completed it's time to save the recorded traffic to a file, so that it could be reused in the future. This can be achieved it two ways.

Using CURL and REST API:

curl http://localhost:8888/api/v2/simulation > simulation.json

Or from TypeScript code:

const sim = await client.getSimulation()
saveSimulationToFile(sim, 'simulation.json')

The saved simulation.json file can now be edited in any text editor to remove any redundant data, simplify response body, or provide responses for specific edge case scenarios.

Crafting a simulation in the code

Another approach is crafting the simulation programmatically, in the code. Here's an example:

import { 
  buildSimulation,
  ResponseData,
  RequestMatcher,
  saveSimulationToFile
} from "@bwilczek/hoverfly-client"

const response: ResponseData = {
  status: 402,
  body: '{"result": "error", "message": "Insufficient balance"}',
  encodedBody: false,
  templated: false
}

const request: RequestMatcher = {
  path: [{ matcher: 'exact', value: '/api/invoices' }],
  destination: [{ matcher: 'exact', value: 'payment.provider' }],
}

const pair = { request: request, response: response }

const sim = buildSimulation([pair])
saveSimulationToFile(sim, 'simulation.json')

Serving responses from a simulation

Once the traffic is recorded in a simulation file it's time to change Hoverfly mode to simulate and start using the pre-recorded responses instead of real services.

Changing mode is easy:

# curl:
curl -X PUT -H "Content-Type: application/json" -d '{"mode": "capture"}' http://localhost:8888/api/v2/hoverfly/mode

# TypeScript:
await client.setMode({mode: 'simulate'})

Uploading the simulation JSON isn't hard as well:

# curl:
curl -X PUT -H "Content-Type: application/json" -d @simulation.json http://localhost:8888/api/v2/simulation

# TypeScript:
await client.uploadSimulation(buildSimulationFromFile('simulation.json'))

Obviously simulations crafted by hand can be uploaded without having to be saved to a JSON file first.

const sim = buildSimulation([pair])
await client.uploadSimulation(sim)

Middleware and Journal

Hoverfly comes with two more handy tools that can provide more flexibility to the test framework. First one is middleware,
a mechanism that can modify the responses dynamically, make requests to real systems, trigger callback webhooks or perform any logic that could be implemented inside a simple HTTP application. This topic is so broad and project specific that it goes far beyond the scope of this article. It's being mentioned here so that you know that if there's something more sophisticated that your stubbed communications needs to do, middleware can help.

The other concept is Journal, which is basically a registry of performed HTTP requests that went through Hoverfly. It's essential for making assertions like "Expect that payment provider has been queried for pending transactions". More on this in the example below.

Example workflow with hoverfly-client

Let's demonstrate the features of Hoverfly in an example jest test:

import { describe, expect, test } from '@jest/globals'
import {
  buildSimulation,
  ResponseData,
  RequestMatcher,
  saveSimulationToFile,
  Client
} from "@bwilczek/hoverfly-client"

const client = new Client("http://hoverfly:8888")

describe("Fetch invoice", () => {
  beforeEach(async () => {
    await client.purgeSimulation()
    await client.setMode({mode: 'simulate'})
    await client.purgeJournal()
    // upload some default requests/responses that should be always active, for example authenthication
    await client.uploadSimulation(buildSimulationFromFile('default_traffic.json'))
  })

  test('sufficient balance', async () => {
    // append simulation for this scenario to the one already present in Hoverfly
    await client.appendSimulation(buildSimulationFromFile('payment_sufficient_balance.json'))

    // do some actions in the UI, that will make the app under test perform a request to payment provider
    await browser.submitPaymentButton.click()

    // assert that the backend really performed a request to the payment provider
    const paymentsJournal = await client.searchJournal({request: {destination: [{matcher: "exact", value: "payment.provider"}]}})
    expect(paymentsJournal.journal.length).toBe(1)
  })
})

Conclusions

Stubbing HTTP communication in a distributed, multi-process system is not only possible, but it's also not that hard. HTTP proxy is the right tool for this purpose, and Hoverfly provides just the right features:

Flexible request matchers
Middleware for custom logic
Support for SSL
Journal for tracking of the processed requests
JSON REST API for easy configuration
Simulating latency and outages

As demonstrated in this article, introduction of Hoverfly to any app stack is easy. With minimal or no changes to the code it opens the apps for testing of use cases not achievable with requests to real external services.

Caveats

As great as Hoverfly is it comes with a few caveats:

destination, no_proxy and direct requests

Switching Hoverfly on and off dynamically for certain URLs is hard. no_proxy variable provides a static list, while destination filtering relies on a whitelist regular expression. Implementation of a test suite that runs some scenarios against real payment provider and some against a simulated one
is tricky and requires some fancy logic in a stateful middleware. It's doable though.

not every lib respects http_proxy

As described above: some HTTP libraries won't respect the standard ENV variables and require changes to client initialization.

not every lib respects default location of SSL certificates

As described above: some HTTP libraries won't trust all system's certificates and require changes to client initialization.

default certificate does not work well with time traveling

Scenarios that involve time traveling might not work as expected, as the SSL certificate shipped with Hoverfly has more "present" validity period. A custom certificate, valid +/- 30 years from now could be generated and used instead.

one, huge simulation, cannot upload/delete single pairs

REST API operates (upload/download) on a whole simulation - not individual request/response pairs. Adding or subtracting responses requires some extra coding, fortunately it can be abstracted away in a client library, like the one used in the examples in this article.

JSON format of simulations

Pretty often request or response payload is also in JSON format, what makes manual changes to the simulation files hard. JSON does not support line splitting and the JSON content is at easiest case escaped and stored in a very long line. In a harder case it can be gzip or brotli compressed and then serialized in base64. Of course editing such payload is still possible, but a bit harder then if a different format was used.

Cover image created with ChatGPT

Practical approach to software complexity

Bartek Wilczek — Tue, 29 Oct 2024 08:46:29 +0000

As software engineers we intuitively know what is software complexity and what impact it makes on the long term maintainability. (In case you didn’t know: the smaller the complexity the easier the code is to maintain).

In long living projects complexity tends to grow over time, and that’s normal. It can be reduced (by certain amount) by refactoring, tooling upgrade or other kinds of redesign. Our (engineers') problem is that such initiatives do not bring immediate business value to the stakeholders. Therefore it could be hard to convince them that they should invest team’s effort into such projects.

In this post I’d like to provide stakeholder-friendly definition of complexity and also explain why keeping it under control is good not only for engineers’ happiness, but also for business’s sustainability.

What is complexity?

Let’s use the image below to explain what complexity is.

Complexity is a metric describing how difficult to change the software is.

It consists of two components:

essential complexity - originates from the business rules. The very core of the system. Cannot be reduced without sacrificing the functionality.
accidental complexity - originates from the implementation. Brings no business value.

The smaller accidental complexity overhead the better.

Now let’s go back to the picture above.

Think of the clean wiring as the essential complexity with very little accidental complexity added. The system is not simple, because the domain is not simple. It looks easy to comprehend and change though, as the non-domain overhead is minimal.

Now think of the messy wiring as the essential complexity with a lot of accidental complexity on top of it. It does the same functionality as the first one, but is much harder to comprehend and change.

Whenever engineers say that certain initiative will reduce the complexity they mean removing the obsolete wires and simplifying the layout of the essential ones.

Where does the accidental complexity come from?

Accidental complexity can be added to the software in two ways:

unintentionally, when the engineering team does not have the experience with the technical solutions required to complete certain task. This happens despite the best efforts and good will displayed by the team. It can be mitigated by involving people with more experience early in the solution design. Reaching out to architects should be a good starting point.
intentionally, when certain shortcuts are being made in order to achieve a short term goal. Can be mitigated by being more flexible with deadlines. When in doubt refer to the messy wiring picture and ask yourself whether adding a random, loose wire that can make one project go live a bit earlier is beneficial for other projects in the long run. Spoiler alert: it is not :)

How to measure the complexity?

This usually depends on the programming language used in the project. Different languages and different tools use different formulas, but as long we we select one of them and stick to it we can track how the complexity changes over time.

Taking actual measurements is pretty easy and usually limited to execution of a single command. In ruby universe that would be flog.

Pseudocode for the calculation of an impact made on complexity by a given commit ABC could look like this:

# checkout commit prior to commit in question
git checkout ABC^

# compute complexity for that commit
complexity_before = `parse output from flog`

# checkout the commit in question
git checkout ABC

# compute complexity for it
complexity_after = `parse output from flog`

# calculate the impact
delta = complexity_after - complexity_before
print "Commit ABC has changed overall complexity by #{delta}"

To assess the impact made by the whole project simply run this formula for every commit associated with the project.

Why we should measure it?

Because you cannot manage what you cannot measure. Prior to giving green light to a potential refactoring project (with no direct business value associated) stakeholders would need a metric that could be used to assess the project impact. They will not be willing to invest resources into vague promises that this project will make the code better.

However after they have been familiarized with the notion that high complexity is bad for long-term maintainability they could be more eager on investing into a project that will e.g. reduce the complexity by 3%.

After the project is approved and completed it is worth to assess the long term impact it made on other, more common metrics:

Team velocity/capacity - it should increase as changing less complex code is easier, giving more time to implement new features.
Infrastructure cost - it should decrease as less complex code requires less resources to run on production and CI environments.
Engineering team satisfaction - it should increase for two reasons:
- engineers love making code better and letting them work on a refactoring project makes them happy
- engineers hate working with hacky, overcomplicated code. Making the code more elegant heals that pain

Summary

Whenever technical projects (refactoring, tooling upgrades) face resistance from the stakeholders the metaphor of reducing complexity could convince them to reassess the priorities. It's like reducing the fixed costs, or other liabilities: frees up the resources to be allocated on actual development. It's measurable, traceable, and easy to introduce to the development process.