DEV Community: mikotian

Making a server believe you're from somewhere else

mikotian — Sun, 22 Jun 2025 18:00:00 +0000

Recently I had to test a scenario where the laravel application detects which region a request is coming from and based on that chooses the correct payment gateway (domestic/international) to effect the payment. This was tricky for the following reasons:

We were testing on servers which were in a VPN enclosed environment. And the VPN gateway always shows a United States IP.
All gateways available to us were US based only. Using a different VPN wasn't possible as its against company policy (and also unsafe).

Now normally, when such systems are built, they are usually built on the basis of geolocation (invasive, think of your browser asking you to share your location) or on the basis of IP Ranges (non-invasive, you can find these on many websites that tell you that you are from such and such country). We were going with IP Ranges, so spoofing geolocation using chrome, playwright etc was out of the question.

The Solution:

We're using an Apache Server as our web server. One of the features of the apache web server is the modules you can load onto it that helps you do things before requests hit the hosted applications. One such module is the mod_headers, which can be used to add/modify basically any header one may need.

So we added the module, enabled it and wrote in a rule that would replace the RemoteIPHeader with whatever IP we fedinto each request in the X-Forwarded-For header.

$ sudo a2enmod mod_headers

$ sudo a2enmod headers

Restart apache2

sudo systemctl restart apache2

Add the following to virtualhost config for application site

RemoteIPHeader X-Forwarded-For

That is the configuration needed at the server side.

At the client side, we need to ensure that the X-Forwarded-For header is added/modified for each request. Since this is a web application and not an API, it needs some extra helping to do this.

Use a chrome extension to modify each request to add the required X-Forwarded-For
Use a proxy that does the above.

We used a proxy. More to the point, we used Burp Suite Community Edition to do the same.

Burp has a built in browser that will use the configured proxy. Browse the application with this browser and voila! You have fooled your application that you are from Berlin when in truth you sit in Baramati!

The process in a nutshell:

Running a simple chaos toolkit experiment

mikotian — Sun, 11 Jul 2021 20:02:46 +0000

Install Chaos Toolkit & required drivers as mentioned here:

Setting up ChaosToolkit Execution Environment

Make sure the virtual environment is activated.

Create the following file:

{
  "version": "1.0.0",
  "title": "random website testing",
  "description": "Just checking the chaostoolkit base stuff",
  "steady-state-hypothesis": {
    "title": "Website is OK",
    "probes": [
      {
        "type": "probe",
        "name": "website-must-be-up",
        "tolerance": 200,
        "provider": {
          "type": "http",
          "timeout": [
            3,
            5
          ],
          "url": "https://httpbin.org/forms/post",
          "method": "GET"
        }
      }
    ]
  },
  "method": [
    {
      "type": "probe",
      "name": "website-must-return-202",
      "tolerance": 202,
      "provider": {
        "type": "http",
        "url": "https://httpbin.org/status/202"
      }
    },
    {
      "type": "probe",
      "name": "response should be json",
      "tolerance": {
        "type": "jsonpath",
        "path": "$.slideshow.author",
        "expect": [
          "failed"
        ],
        "target": "body"
      },
      "provider": {
        "type": "http",
        "url": "https://httpbin.org/json"
      }
    }
  ],
  "rollbacks": []
}

Running the experiment:

chaos run SimpleExperiment.json

ScreenGrab:

The same can be run in a Jenkins Pipeline:

The jenkins pipeline script is as follows:

pipeline {
    agent any

    stages {
        stage('Deploy') {
            steps {
                // Get some code from a GitHub repository
                git 'https://github.com/mikotian/resilientchaos.git'

            }

            post {
                success {
                    echo 'Hello World'
                }
            }
        }
        stage('Run Chaos Script') {
            steps {

                sh ". /home/mithun/.venvs/chaostk/bin/activate && chaos run SimpleExperiment.json"
            }

            post {
                success {
                    echo 'Success'
                }
            }
        }
    }
}

The execution looks like this:

Setting up ChaosToolkit Execution Environment

mikotian — Sun, 11 Jul 2021 20:01:32 +0000

Base Setup

Install Python 3.5 on your execution machine.

Windows : Download the latest binary installer from the Python website.

Linux: https://www.python.org/download/other/

Create a virtual environment. Virtual environments help us encapsulate our changes away from the larger system.

python3 -m venv ~/.venvs/resilient

source ~/.venvs/resilient/bin/activate

Install chaostoolkit

pip install chaostoolkit

Verify version

chaos --version

This will install the base version of chaostoolkit with its inbuilt probes, tolerances & actions.

Install AWS Drivers

pip install -U chaostoolkit-aws

Passing Credentials to ChaosToolkit.

For AWS actions/probes to work as expected we need to place a credentials file like below in the following location. ~/.aws

The credentials can be added in the experiment itself, like so

{
  "secrets": {
    "aws": {
      "aws_access_key_id": "your key",
      "aws_secret_access_key": "access key",
      "aws_session_token": "token"
    }
  }
}

Thats it, you're all set to run AWS Chaos/resiliency experiments!

Install Toxiproxy Drivers

Install Toxiproxy Server from here: (https://github.com/Shopify/toxiproxy/releases)

Install the toxiproxy CLI (https://github.com/Shopify/toxiproxy/releases)

These are required in order to make the network changes work.

Install the chaos toolkit drivers:

pip install -U chaostoolkit-toxiproxy

Add the following configuration to your experiment.

"configuration": {
    "toxiproxy_host" : "10.124.23.183",
    "some_environment_variable": {
      "type": "environment",
      "key": "ENVIRONMENT_VARIABLE"
    }
  }

Start the toxiproxy server.

Now you are ready to run the experiment.

ChaosToolkit Basics

mikotian — Sun, 11 Jul 2021 19:58:49 +0000

The major aim of Resiliency Experiments is always the same -

To observe a steady state
Do something that attempts to disrupt that steady state
Observe if we still have a steady state

These three steps are always constant in any resiliency experiment.

What changes are the ways we measure or cause these steps to happen.

For instance,

To see if a service is up we can have multiple methods viz. Health Checks, Monitoring Checks or Request Checks.
To simulate a service downtime, we can shutdown the machine or the specific process (each has its own case)

In order to correctly perform the experiment when automated and to maintain the sameness for the tests, we needed to implement a tool that would provide a homogeneous way of doing this.

We evaluated Gremlin and Chaos Toolkit for these experiments.

This document describes the working of Chaos Toolkit, keeping in mind what is listed above.
Chaos Toolkit is a declarative framework for Chaos Engineering experiments which can be extended for any system by using drivers. It has open source drivers for most major systems.

We will take a look at the declarative framework and its basics:

Each Experiment will have the following sections:

Description Section:

This is just meta data for the test. Version, Title & Description mention specifics of the experiment. The tags mention what the system being tested is composed of or what the test tags are. There can be multiple tags but only singular version, title & description.

Steady State Hypothesis Section:

This is the heart of the experiment. This section is executed when the experiment is started and after the "method" section is executed. The first time execution is to ensure that steady state hypothesis is met before we do any action. If it is not, then the experiment is failed. The Steady State

The way to verify a state or hypothesis is by using a probe. ChaosToolkit provides a set of probes that can be used to check on services or URLs. Each probe has a provider which mentions the type of the probe and arguments, if applicable. Each probe is different and may have multiple arguments or none at all.

In the example below, there are two probes.

The first one is a python function that returns a boolean. This is verified in the "tolerance" parameter.

The second one is a built in http probe that checks a url and verifies if the response code is 200

Writing a good steady state hypothesis is essential to get your resiliency tests be repeatable and accurate. Too simple and you risk missing issues. Too complex and you risk false negatives.

"Action" Section:

This section is where we perform our "disruptions"

Depending on the drivers, we can have many types of disruptions, for e.g:

The AWS driver can shutdown instances, random machines in availability zones etc.

Kubernetes drivers have the ability to kill random pods.

In the example below, we can see that the action is terminating a pod based on a random name from a given pattern. After the action is performed, we can pause execution for a provided value in seconds to let the system stabilize and then proceed to probe for our expected values. In this case we are probing by reference of a probe that has already been defined earlier in the experiment.

We can see the other probes defined as well. Just as there can be multiple probes, there can be multiple actions within the "methods" section.

It might be better to keep the action itself brief and limited so that we can objectively identify what that action does to our system. If there are multiple actions, determining what caused the failure might be difficult.

This is not always true though as many issues will come up because of multiple actions performed simultaneously. In short, YMMV.

Rollback Section:

This section is to undo any permanent change to the system we may have caused during the course of the experiment. For example, we could start an instance that we shut down in the experiment.

Overall, an experiment descriptor file looks like below:

Simple ChaosToolkit Experiment

{
    "version": "1.0.0",
    "title": "Stop an instance",
    "description": "it should stop",
    "tags": [
        "stop"
    ],
    "steady-state-hypothesis": {
        "title": "Application responds",
        "probes": [
            {
                "name": "count-instances",
                "type": "probe",
                "tolerance": 1,
                "provider": {
                    "func": "count_instances",
                    "type": "python",
                    "arguments": {
                        "filters": []
                    },
                    "module": "chaosaws.ec2.probes"
                }
            }
        ]
    },
    "method": [
        {
            "type": "action",
            "name": "stop-an-ec2-instance",
            "provider": {
                "type": "python",
                "module": "chaosaws.ec2.actions",
                "func": "stop_instance",
                "arguments": {
                    "instance_id": "i-0e1f0c1d97589b5e9"
                }
            },
            "pauses": {
                "after": 30
            }
        },
        {
            "name": "count-instances",
            "type": "probe",
            "tolerance": 0,
            "provider": {
                "func": "count_instances",
                "type": "python",
                "arguments": {
                    "filters": []
                },
                "module": "chaosaws.ec2.probes"
            }
        },
        {
            "type": "probe",
            "name": "healthcheck-service-must-still-respond",
            "provider": {
                "type": "http",
                "url": "http://localhost:8080/healthcheck"
            }
        }
    ]
}

Resiliency

mikotian — Sun, 11 Jul 2021 19:56:43 +0000

In a complex setup where hundreds of microservices talk to and depend on each other to function properly, things can always go wrong. Especially when multiple teams are involved in making changes. Sometimes multiple teams are involved in making changes on the same microservice! This is a sure shot recipe for issues that will crop up at stages starting from integration to weird production issues that may have no functional basis at all. Some issues are purely related to improper error handling, dependency management or infrastructure glitches causing cascading issues. It is this problem that Resiliency Testing tries to solve. While these need not be automated all the time, but having these automated (for a mature product) is most likely to be beneficial.

Core Aim of Resiliency Tests:

Identify Resiliency Issues with the Application Under Test
Help the team identify markers of issues that may otherwise take multiple hours to decode in a production environment
Make a more observable system by introducing errors that will expose the need for more logging and tracing.

The two main areas for inducing uncertainty in a system are:

Infrastructure: Randomly shutting down instances and other infrastructure parts

Application: Introduce failures during runtime at a component level (e.g. endpoint/request level)

You then enable uncertainty randomly or intentionally via experiments:

Randomly (Chaos)
- More suitable for ‘disposable’ infrastructure (e.g. ec2 instances)
- Tests redundant infrastructure for impact on end-users
- Used when impact is well-understood
Experiments (Resilience)
- Accurately measure impact
- Control over experimental parameters
- Suitable for complex failures (e.g. latency) when impact is not well understood

Finally, you can categorize failure modes as follows:

Resource: CPU, memory, IO, disk
Network: Blackhole, latency, packet loss, DNS
State: Shutdown, time, process killer

Many of these modes can be applied or simulated at the infrastructure or application level:

After running an experiment, there are typically two potential outcomes. Either the system is found to be resilient to the introduced failure, or a problem is identified that may needed to be fixed. Both of these are good outcomes. In the first case, the confidence in the system and its behaviour is enhanced. In the other case, a problem has been found before it causes an outage.

By proactively testing and validating our system’s failure modes we can reduce operational burden, increase resiliency, and will eventually lead to reduced palpitations at the time of production issues.

To kill a MockingBird...

mikotian — Mon, 09 Mar 2020 14:02:09 +0000

In the previous posts of the series I recounted why the need for MockingBird arose and what we did to implement it.

To recap, it was implemented using Flask and Python and was started on the machine using the following command:

python mockingbird.py

This meant that Flask was running in its default mode - development, on its own server and was running a single process - ideally with a single thread but I have read conflicting documents that mention that flask is multi-threaded by default currently (i.e Threaded=true is the default way now).

If a single thread is around, then flask will serve only one request at a time. This combined with the 0.5s sleep we applied would mean that we were able to serve only 2 requests per second. Which is not ideal.

My target was 1200 requests per minute at the bare minimum.

My tests initially went well, we were able to handle the load thrown at the MockingBird pretty well (probably because of the aforementioned threading) for short test durations and low load. But I noticed something weird in the logs of my tests when I applied the full load.

Timeouts, lots of them.

The timeout in the originating MS was set at 60s. And Mockingbird was unable to send responses to a majority of the requests (>60%) in that time.

And that was how I killed the MockingBird.

Looking at the logs I started cursing at myself for not thinking the solution through. Redoing the whole thing would mean loss of time. And time is something we cannot afford at my current organization.

So off I went to read the flask documentation(great resource btw) which mentioned that the inbuilt webserver was not ideal for production loads (which I had thrown at it). It might have worked if I didn't have the blocking sleep call but that was needed. So what else could work?

I decided to take a look at some recommended servers. One I liked in particular was Gunicorn, with its support for different types of workers.

A worker is to put it simply, a copy of your code that is run in parallel to other workers. Read the design doc for more info.

pip install gunicorn

is how to install gunicorn on your system

I also decided to use the gevent worker for gunicorn to handle multiple requests. This decision was made since we needed a AsyncIO worker to optimize for our forced IO wait times. For more information I suggest reading the documentation linked above.

pip install gevent

is how to install gevent.

After doing these two things, it is a matter of just issuing the proper command (no code changes needed to existing files!) to start the application.

All that is needed is one extra file in the root directory of your project:

from flaskapi import api

if __name__ == "__main__":
    api.run()

Name it whatever you want. I named it wsgi.py

The command:

gunicorn --worker-class gevent -w 4 flaskapi:api --bind 0.0.0.0:5000 --daemon

0.0.0.0 is important if you want intranet access to your API/server
-w 4 means 4 workers : ideal value should be number of processors + 1
--daemon means that processes will be spawned in the background

This can be seen to start 5 processes on the server, one master and 4 workers each serving multiple requests per second. This worked sorta fine for the load we were throwing at it, but we had to eventually upgrade our puny server (CPU was hitting 100% at peak load) to land consistent results, even under prolonged peak load.

And thus the MockingBird lives to sing another day.

So this brings to an end the MockingBird series of posts. One hopes it has been informative!

To make a MockingBird

mikotian — Sat, 08 Feb 2020 11:11:22 +0000

Prologue https://dev.to/mikotian/to-birth-a-mockingbird-ep8

I began researching mocking tools and found one that seemed quick enough for my needs.

Mockoon[https://mockoon.com/] was a nice little gui based tool that I had heard of before and without giving it much thought, I started implementing my little mockingbird.

The good part about Mockoon:

*Its fairly simple to start with. Anybody who has used Postman will find this to be intuitive and easy to understand.
*You can have multiple apis running in parallel on your chosen ports
the ability to define multiple responses based on what is the incoming payload
*https support out of the box, if you need it
*Very very good templating support (e.g. generating unique ids for fields)
*Latency Simulation (this was the number 1 requirement I had)
*latency simulation on a per endpoint basis as well (this is cumulative over base path latency)

The dealbreaker(s):

*It is a localhost only application (so no intranet exposure, at least I wasn't able to get that to work)
*No commandline or gui-less invocation, so would not work on no-desktop environments without major tinkering (one way could be x11 forwarding, perhaps)

Some Screenshots from my experiment

I could simulate every API endpoint I wanted to mock and it worked flawlessly for me, except for the intranet part. Might be interesting for Devs who want to mock complex API behavior.

So I had to think of something else.

Having worked on Python for a while, and had always wanted to write an API using it. I had researched a bit on flask back in the day but had never quite got around to using it. This was my chance.

And it turned out to be quite easy.

I used the standard python installation on an AWS installation to run the following code:

It is pretty boilerplate stuff except for a few things:

*In order to introduce some latency in the responses, I used time.sleep() before sending the response
*Used the inbuilt support for json to manipulate json objects.
*Used uuid to generate a couple of different types of guids (learnt about the library too along the way!)

The instance I used to host this was a AWS t2.micro instance with 1 vCPU and 1 GB of RAM.

It was exposed on the standard 5000 port that flask uses and I decided not to use https since that would have increased setup time on the many instances of the producing microservice.

In order to expose it to the internal network, I used 0.0.0.0 as the address which means that the program will listen to all binded IPs for the machine.

There is good support for debugging in flask as well and that requires us to set the debug flag while starting the service/application.

The command I used to start the application was a fairly straightforward one:

python mockingbird.py

The application worked wonderfully for our purposes. It was giving the correct responses with the desired latency and it looked like it was absorbing all the load as well. A couple of load tests went well with parameters showing to be on expected lines.

But as it is in the world of Software Development (and especially in the world of Software Testing), the first version is never quite the finished project.

In the next post, I'll recount/explain how i killed the mockingbird (and what I did to fix it)

To need a MockingBird

mikotian — Sat, 01 Feb 2020 12:09:35 +0000

Weird title, but it'll get clear by the end of the post, I promise!

Mocking is something that is associated with developers more than it is with testers, especially if one is dealing with Unit Tests.

That is not to say that testers don't use it. Mocking is helpful when testers have API specifications but don't have the actual API itself and they have to develop API Automation (because Agile).

In my current project, we are migrating over a legacy ticketing platform (standard .NET stuff & a lot of confused applications) to a new platform comprised of a SaaS based Incident Management system that is supported by multiple "new" microservices with every new age technology stuffed into the architecture somehow (I could fill up a couple of blog posts with snark aimed at the overall thinking in doing something like this, but I digress. Maybe later).

So, the overall system when tested for performance (by yours truly) was doing extremely poor in terms of the lofty expectations set by people who were good at setting expectations :)

So a few "war" rooms with much yelling and panicky reactions, the performance issue was narrowed down to some faulty API calls that were setting databases on fire in the third party ticketing system (imagine 30-40 second response times on good calls and 62 seconds about 70% of the times, where the expected value was sub-second). Perfectly legal, good API calls, mind you. This in turn dragged down the performance of our "victim" microservice to abysmal levels.

After much wringing of hands, late night builds & some more panicked conference calls, we ran performance tests that absolved the Third Party SaaS Incident Management System of any wrongdoing. This caused a lot of frayed tempers to be soothed and the hand-wringing moved down to just twitching of fingers. What it did though was give us pause to think.

What was the actual throughput of our "victim" service?

Now that we knew that the service's performance was tied to how the SaaS API performed, will we ever know its true performance given how flaky the API was turning out to be?

We decided to setup a performance lab, abstract out the rest of the stuff and just keep the microservice, its common infra and have a mock do the work of the SaaS API.

And that is how we gave birth to our MockingBird - thats what I call the mock API that I built.

The next post will be about how exactly that was accomplished.