Alaz: Open-source, Self-Hosted, eBPF-based K8S Monitoring 🐝 🚀

Kürşat Aktaş — Wed, 06 Sep 2023 17:43:26 +0000

Hello Everyone ✋

I'm excited to introduce a new open-source observability platform and would love to hear your feedback.

We are aware that there are lots of open-source/commercial tools out there. However, we believe that monitoring the clusters and extracting actionable insights requires deep know-how about the tools/domain. We mainly focused on this problem.

🐝 Alaz is an eBPF agent installed on your K8S cluster as DaemonSet. Thanks to eBPF - Alaz collects traces directly from Linux kernels. This means there's no need for sidecars, instrumentations, or service restarts.

🎯 The UI not only visualizes data but also provides actionable insights. Using the Service Map, you can:

View latencies and RPS between services.

Detect zombie services and underperforming SQL queries.

Monitor golden signals, such as 5xx status codes.

In addition, Alaz can capture system resources like CPU, Memory, Disk, and Network through the Prometheus Node Exporter, which is embedded in the agent.

Setting up is straightforward: just install Alaz as a DaemonSet, and the platform will handle the rest.

Finally, the combination of Alaz and Ddosify Performance Testing makes it possible to do load testing and simultaneously monitor the system to find bottlenecks instantly.

🚀 For those interested, check out Alaz on GitHub.
Your feedback would be greatly appreciated!

Load Testing a Fintech API with CSV Test Data Import

Kürşat Aktaş — Mon, 13 Feb 2023 18:10:19 +0000

Introduction

We have organized this write-up into two parts to demonstrate two different features of Ddosify. In the first part, we will perform a load test on a GET endpoint that accepts base and target currency and returns their exchange rate of them. The rand() utility method is used to send different currencies on each request. In the second part, we will test a POST endpoint that performs exchange operations. We will use a CSV file that contains test data stored on our Test App's Database. Then we import this CSV file into Ddosify to replay the same transactions stored on DB, but in high concurrency. In both parts, we will gain insights into the reliability of our exchange API across high traffic.

The Environment

As in the previous blog post we will again use the Ddosify test API as a backend service.
We will use Ddosify Open Source Load Engine version v0.13.2. If you didn't install it yet, you can follow the readme to find the proper installation method for your operating system.
The configuration files used in this blog are available in this repository.

Load testing the exchange rate info API

Almost all fintech APIs have an endpoint that provides the exchange rate between two currencies. Our test backend API has a similar one, GET exchange_rate/<base_currency>/<target_currency>/. In this use case, we would like to learn the performance of this endpoint by providing random <base_currency> and <target_currency> on high IPS (iteration per second). Let's take a look at the below configuration first, then we will talk about the details of each section.

fetch_exchange_rates.json

{
  "iteration_count": 1000,
  "duration": 10,
  "load_type": "incremental",
  "env": {
    "currencies": [
      "AED",
      "ARS",
      "AUD",
      "BGN",
      "BHD",
      "BRL",
      "CAD",
      "CHF",
      "CNY",
      "DKK",
      "DZD",
      "EUR",
      "FKP",
      "INR",
      "JEP",
      "JPY",
      "KES",
      "KWD",
      "KZT",
      "MXN",
      "NZD",
      "RUB",
      "SEK",
      "SGD",
      "TRY",
      "USD"
    ],
    "baseUrl": "https://testserver.ddosify.com/exchange_rate"
  },
  "steps": [
    {
      "id": 1,
      "name": "Random Currency Fetch",
      "url": "{{baseUrl}}/{{rand(currencies)}}/{{rand(currencies)}}/"
    }
  ]
}

First of all, we have a list of currencies in our environment variables. If you wonder, we have fetched the supported currencies from the https://testserver.ddosify.com/currencies/ endpoint. The important part of this configuration is the URL field of the step. We used the rand() utility method on both <base_currency> and <target_currency> fields to inject a random currency. Before starting the test, we will use the --debug flag to inspect request headers, request body, response headers, and response body. Ddosify sends only 1 request in debug mode.

$ ddosify -config fetch_exchange_rates.json -debug

<.. truncated ..>
STEP (1) Random Currency Fetch
-------------------------------------
- Environment Variables
    currencies:    [AED ARS AUD BGN BHD BRL CAD CHF CNY DKK DZD EUR FKP INR JEP JPY KES KWD KZT MXN NZD RUB SEK SGD TRY USD]
    baseUrl:       https://testserver.ddosify.com/exchange_rate

- Test Data

- Request
    Target:     https://testserver.ddosify.com/exchange_rate/CNY/RUB/
    Method:     GET
    Headers:
    Body:

- Response
    StatusCode:    200
    Headers:
        Strict-Transport-Security:     max-age=31536000
        Content-Length:                17
        Connection:                    keep-alive
        Server:                        nginx/1.23.3
        Date:                          Wed, 08 Feb 2023 10:16:16 GMT
        Allow:                         GET, HEAD, OPTIONS
        Referrer-Policy:               same-origin
        Vary:                          Accept
        X-Frame-Options:               DENY
        Cross-Origin-Opener-Policy:    same-origin
        Content-Type:                  application/json
        X-Content-Type-Options:        nosniff
    Body:
        {
            "rate": 10.44834
        }

You may notice that we use the -debug flag on CLI instead of putting the debug:true key in the configuration file as we did in the previous article. Although these two ways are valid ways to run Ddosify in debug mode, the CLI flag has priority over the debug key. In this use case, we choose to enable debugging on CLI for ease of use.

Looks like everything is as expected. We have successfully injected random currencies on the request URL and the response contains the exchange rate (10.44834) along with the 200 HTTP OK status code. We can remove the debug flag and start the test.

$ ddosify -config fetch_exchange_rates.json

⚙️  Initializing...
🔥 Engine fired.

🛑 CTRL+C to gracefully stop.
✔️  Successful Run: 35     100%       ❌ Failed Run: 0        0%       ⏱️  Avg. Duration: 0.08713s
✔️  Successful Run: 101    100%       ❌ Failed Run: 0        0%       ⏱️  Avg. Duration: 0.12753s
✔️  Successful Run: 207    100%       ❌ Failed Run: 0        0%       ⏱️  Avg. Duration: 0.13816s
✔️  Successful Run: 326    100%       ❌ Failed Run: 0        0%       ⏱️  Avg. Duration: 0.13880s
✔️  Successful Run: 455    100%       ❌ Failed Run: 0        0%       ⏱️  Avg. Duration: 0.13609s
✔️  Successful Run: 560    100%       ❌ Failed Run: 0        0%       ⏱️  Avg. Duration: 0.29128s
✔️  Successful Run: 821    100%       ❌ Failed Run: 0        0%       ⏱️  Avg. Duration: 0.43275s
✔️  Successful Run: 998     99%       ❌ Failed Run: 2        1%       ⏱️  Avg. Duration: 0.44565s


RESULT
-------------------------------------
Success Count:    998   (99%)
Failed Count:     2     (1%)

Durations (Avg):
  DNS                  :0.0851s
  Connection           :0.0120s
  Request Write        :0.0000s
  Server Processing    :0.2938s
  Response Read        :0.0000s
  Total                :0.4457s

Status Code (Message) :Count
  200 (OK)    :998

Error Distribution (Count:Reason):
  2     :connection timeout

If we follow the real-time logs, we can easily analyze the Avg. Duration increases over time. The reason is we set the load_type as incremental in the configuration file. That means the iteration count increases every second, so our API receives more and more requests per following second. As a final note, 2 requests have been timed out at the last second of the test, so our test API couldn't totally handle 1000 requests in 10 seconds on incremental traffic.

Incremental Load Type

Importing CSV data to replay network traffic

In our test API there is a POST /exchange endpoint that expects the amount, base currency, and target currency as request payload and it exchanges the given amount from the base currency to the target currency. Let's assume that our imaginary customers have used this endpoint many times and our servers started to slow down. The development team did some performance improvements and we would like to replay the last 1000 transactions to test the performance of the new system. This is a great example to demonstrate how to supply test data to Ddosify Engine.

We exported the last 1K transactions from the database and save them to a CSV file called test_data.csv.

test_data.csv

e746d910-bece-4c37-8e6b-325f89acc4d2,969.2,DZD,KWD
531dae65-2cf4-4194-867e-71c0ea7b6381,485.04,ARS,KZT
c0ec5795-2c37-4c08-a384-82f3444d445e,750.09,FKP,JEP
ba53bbdc-715d-4b72-99a2-867d4167861d,374.03,RUB,JPY
a087955c-3dbc-4164-b5a1-9def635107c7,971.02,SGD,SEK
<.. truncated ..>

The CSV file consists of 4 columns. These are api_key for authenticating the user for the transaction, amount, base currency, and target currency. Now we can use the CSV test data import feature of Ddosify Engine to resend these transactions.

transaction_replay.json

{
  "iteration_count": 1000,
  "duration": 60,
  "load_type": "waved",
  "env": {
    "baseUrl": "https://testserver.ddosify.com/exchange/"
  },
  "data": {
    "transactions": {
      "path": "./test_data.csv",
      "vars": {
        "0": {
          "tag": "api_key"
        },
        "1": {
          "tag": "amount",
          "type": "float"
        },
        "2": {
          "tag": "base_currency"
        },
        "3": {
          "tag": "target_currency"
        }
      },
      "delimiter": ",",
      "allow_quota": false,
      "order": "sequential",
      "skip_first_line": false,
      "skip_empty_line": true
    }
  },
  "steps": [
    {
      "id": 1,
      "name": "Exchange",
      "url": "{{baseUrl}}",
      "method": "POST",
      "headers": {
        "X-API-KEY": "{{data.transactions.api_key}}",
        "Content-Type": "application/json"
      },
      "payload_file": "./exchange_body.json"
    }
  ]
}

transaction_replay.json is our main configuration file. In the data section we provide our CSV file in transactions name scope, so we can use it in our steps as {{data.transactions.tag}} format. There are lots of options we configured in this section, here are the details of them:

We used path to point the location of our CSV file.
vars field is for matching the column index in the CSV to a variable name to use it in step configuration. For example, we named the first column in the CSV as api_key then we use it in the headers section of the Step, like X-API-KEY": "{{data.transactions.api_key}}. Note that we also assigned the type of the second column as float since we want that Ddosify should treat amount values as a float instead of a string.
delimiter is the delimiter character of the CSV. While the default value of it is "," we type it to show you can use a custom delimiter.
allow_quota enables the situation if a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. Default is false. We put it to show this option. In our case, we don't expect quoted data in our CSV.
We set the the order as sequential since we want to fetch the lines in the same order that is located in the CSV. The default is random, in that case, Ddosify fetches lines randomly from the CSV.
We set skip_first_line as false because the test_data.csv has no headers in the first line.
Although we don't have any empty lines on our CSV file, we configured skip_empty_line to show that there is an option like this.

exchange_body.json

{
  "amount": "{{data.transactions.amount}}",
  "base": "{{data.transactions.base_currency}}",
  "target": "{{data.transactions.target_currency}}"
}

exchange_body.json is our POST payload, we filled the fields with the test data fetched from the CSV file. Let's debug our scenario and see what will happen.

$ ddosify -config transaction_replay.json -debug

<.. truncated ..>

STEP (1) Exchange
-------------------------------------
- Environment Variables
    baseUrl:    https://testserver.ddosify.com/exchange/

- Test Data
    data.transactions.base_currency:      DZD
    data.transactions.target_currency:    KWD
    data.transactions.api_key:            e746d910-bece-4c37-8e6b-325f89acc4d2
    data.transactions.amount:             969.2

- Request
    Target:     https://testserver.ddosify.com/exchange/
    Method:     POST
    Headers:
        Content-Type:    application/json
        X-Api-Key:       e746d910-bece-4c37-8e6b-325f89acc4d2
    Body:
        {
            "amount": 969.2,
            "base": "DZD",
            "target": "KWD"
        }

- Response
    StatusCode:    200
    Headers:
        Allow:                         POST, OPTIONS
        <.. truncated ..>
    Body:
        {
            "amount": 2.1690696000000003,
            "currency": "KWD",
            "success": true
        }

The Test Data section in the Debug result shows that we have successfully fetched the first row of our CSV file. Also, everything looks correct on the request payload and X-Api-Key header value. Since we receive the 200 HTTP OK status code we can start the actual test. Remember our configuration on transaction_replay.json, we set iteration_count to 1000 since we have 1000 lines of data in our CSV. Instead of incremental, we will use waved load type to simulate more realistic network traffic.

If the iteration_count is greater than the data count on the CSV file, then Ddosify Engine loops over the file .

Waved load type

$ ddosify -config transaction_replay.json

<.. truncated ..>

RESULT
-------------------------------------
Success Count:    1000  (100%)
Failed Count:     0     (0%)

Durations (Avg):
  DNS                  :0.0037s
  Connection           :0.0015s
  Request Write        :0.0000s
  Server Processing    :0.0831s
  Response Read        :0.0000s
  Total                :0.0886s

Status Code (Message) :Count
  200 (OK)           :1000

The result shows that our backend API can handle 1000 exchange transactions in 60 seconds in the waved load type. We can test the system under different conditions by changing duration, load_type, and iteration_count parameters.

Conclusion

At first, we showed how to use rand() utility method of Ddosify Engine to use environment variables in advance. Then we demonstrated the usage of Test Data import via CSV file along with a traffic replay scenario.

Ddosify Engine has more capabilities on each release, follow it on GitHub to stay updated.

You can find the files we used in this article at Ddosify Blog Examples repository. If you need assistance, you can join our Discord channel.

Testing the Performance of User Authentication Flow

Kürşat Aktaş — Sat, 14 Jan 2023 12:56:06 +0000

Testing the performance of the backend API on high traffic is a must-have step before deploying it to production. We have to simulate the traffic as realistically as possible. This includes testing the endpoints that require authentication. In this article, we explained how to simulate hundreds of users that first get their JWT token from the Login endpoint and hits the private endpoints. We have used Ddosify Open Source Load Engine, so we didn't need to write any line of code.

Happy reading
Testing the Performance of User Authentication Flow

DEV Community: Kürşat Aktaş