DEV Community: GoChronicles

Bench-marking RESTful APIs | Part - II: On the Cloud

Nikhil Akki — Sun, 26 Sep 2021 08:07:43 +0000

Have you ever load tested your APIs on the cloud? In this post we explore how to load test and benchmark different RESTful framework's performance

Recap 🤠

In our previous post, we did some benchmarks of frameworks from different languages. Our test hardware/server was my Raspberry Pi 3 Model B from 2016, it was a good experiment, but this time around we needed something more realistic.

TL;DR - We used Kubernetes 😉 on Cloud (Google Cloud Platform) -> jump to benchmarks section.

Let's get back to our story if are following the storyline from the first post 😄

Dave went back to his peers to show benchmark results that he had got in the initial tests which he ran on his Raspberry Pi 3, while some of his peers liked the idea and appreciated the outcome, some pointed out that they would need the test on real production-grade hardware to believe the results! Dave went back to his garage to tweak his test bench.

Intro

Our setup is quite straightforward, each RESTful service gets the same amount of CPU and memory (enforced via k8s config). The entire setup takes few minutes to initialize, thanks to tools like gcloud CLI, Terraform & Pulumi. You can get it up and running with the environment without much hassle. And if you want to run the benchmark without fancy infra (i.e. without private VPC etc.). We recommend that you use the CLI wrapper as it is built over gcloud SDK and for the adventurous type, we have a slightly more elaborate setup with Terraform (GCP) & Pulumi (GCP & Digital Ocean).

Environment Review

Kubernetes is a planet scale tool that can orchestrate containerized applications and more.

Checkout our series on Kubernetes it is swell! 😃

Since we didn't want to scale the application as load increases, we have put some limits. The config ensures that the deployments stay put and do not auto-scale in the K8s cluster. The whole point of this exercise is to simulate a prod environment (but without auto-scaling). We then load test and measure performance.

Quite the step up from the test on Raspberry Pi 3, isn't it? 😉

It took us a while to figure out the right configuration for the cluster so that you could replicate the tests on your own with the optimal amount of resources. The K8s environment can be setup on GCP free tier (at the time of writing this article)

Source code link for this entire project is given the references section! 😎

Let's review our K8️⃣s Config file

Deployment config looks like this - 👀

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rest-net-http-golang
spec:
  selector:
    matchLabels:
      app: rest-net-http-golang
  template:
    metadata:
      labels:
        app: rest-net-http-golang
    spec:
      containers:
        - name: rest-net-http-golang
          image: ghcr.io/gochronicles/benchmark-rest-frameworks/rest-net-http-golang:latest
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              memory: "1Gi"
              cpu: "500m"
          ports:
            - containerPort: 3000

Notice that we have allocated memory - 1Gi and CPU 500m or half vCPU. This constraint is given to all frameworks, this will ensure that the amount of compute given to each deployment is consistent.

apiVersion: v1
kind: Service
metadata:
  name: rest-net-http-golang
spec:
  type: LoadBalancer # provide public ip to the service
  selector:
    app: rest-net-http-golang
  ports:
    - port: 80
      targetPort: 3000

Service config exposes the RESTful apps in the cluster network via Public IP so that our bench-marking client can connect and run load simulations.

Our attack tool for load testing🍾

This time around, we decided to play around with different benchmark & load testing tools. Finally, we chose Hey.

Hey is a drop-in replacement for ab tool (Apache benchmark).

hey -c 800 -n 35000 <http://ip-addr-url/>

This command will send 800 concurrent & total 35k requests to the RESTful API services on the K8s cluster.

Honorable mentions -

Locust! 🛩️- This was the ideal tool for this test for couple of important reasons. We could deploy this python based web app like load test tool on K8s cluster and run benchmarks from with in the cluster network (no need for public IP) It comes with a nice UI dashboard to visualize the results. The test results was same across frameworks, it looked like we couldn't schedule enough number of workers to really push the throttle on the RESTful APIs. We had a limit on the number of processors we could deploy on our GCP instance (free tier has 8CPU limit for the entire project) If you want to tinker with locust, here's the k8s config we created.
Apache Benchmark - Good old tool we could still probably use, but the results were better and faster with hey and it shares the almost similar CLI options. CPU monitoring tool (htop) revealed that ab tool didn't take advantage of all the CPU cores, where as hey tool fired up on all CPU cores with same parameters out of the box. Benchmarks 📈 The order of slowest to fastest framework is as expected in the benchmark results. Go frameworks are at a minimum 10x faster than Node & Python-based frameworks. However, the interesting bit is FastAPI (Python framework) isn't too far off from NestJS (which is about ~12% faster).

FastAPI (Python)

NestJS (Node)

ExpressJS (Node)

Gin (Golang)

Net-http (Go std libray)

Fiber (Golang)

Close thoughts 🤔

Results are as we anticipated - Go-based frameworks are at least 10x faster than Node & Python-based frameworks. One thing surprised us and possible areas for more research -

In our local testing, Gin has always performed faster than Net/HTTP (Golang). However, in this test, it has scored lower. The source code for this service and the Kubernetes config can be found here and here respectively.

Let us know in the comments if you found a better way to do these tests.

Your feedback 💁 and support🤝 means a lot, do share some love🥰 by sharing our posts on social media and subscribe to our newsletter! Until next time! 🎂😎

References

This article was originally published on GoChronicles.com by the same author and has been repost with permission.

Bench-marking RESTful APIs

Nikhil Akki — Sat, 25 Sep 2021 04:43:19 +0000

Are your RESTful services ready to handle a surge in traffic? One way to find out....

Do you know if your RESTful services will take the heat when your service/website gains multi-fold momentum in traffic?

Dave works for an eCommerce start-up, one day, he was given a task to find out if the micro-services which he has been writing is scalable and optimized. The marketing team wants to run a big flash sale for a couple of days and expects a massive influx of online traffic to the website and the mobile app.

Dave's task is to figure out how services like order, checkout and other product-related services would perform in such situations.
In this article, we will load-test three REST API frameworks written in NodeJS, Python & Golang and, the benchmark will be using the Apache Benchmark tool.

The beauty of micro-services is that different teams could write services backed by business logic in whichever programming language & Eco-system conducive to their use case.

Our Rigg -

Raspberry PI 3 Model B

1.2Ghz Quad-core Cortex A53
whopping 1GB RAM
16GB Class 10 SD Card.

Well, you could run this benchmark on any compatible system. Benchmarks make sense as long as you test them on the same hardware environment.

Environment -

Server machine [mighty rpi3] -

OS - Raspberry PI OS on armv7l
Kernel - 5.10.52-v7+
Application runtimes -

Client machine [my laptop ;-)]-

OS - Fedora 34
Kernel - 5.13.5-200.fc34.x86_64
Ryzen 5500U (6 Cores & 8 GB DDR4 3200 MHz RAM)
Apache Benchmark tool (AB tool)

And the test -

ab -k -c 350 -n 5000 http://<rpi ip>:3000/

The above command sends 5k requests in total with a concurrency of 350 requests. One could tweak these numbers as per requirements.

The entire testing session ran for a few minutes and, we have the results in the below screenshots. We will start with FastAPI, followed by ExpressJS and then Fiber.

FastAPI (Sync) - Python

FastAPI in synchronous mode clocks in at ~178 requests per second.

FastAPI (Async) - Python

FastAPI in asynchronous mode clocks in at ~228 requests per second. One important thing to point out, that we are running FastAPI's ASGI server (uvicorn) with default settings (1 worker). If we tweak the worker count to the number of available CPUs, there should be a significant jump in the performance.

Express JS (Sync) - NodeJS

Express JS in the synchronous mode does 447 requests per second which are over ~2x jump from the FastAPI.

Express JS (Async) - NodeJS

A similar trend is apparent in asynchronous mode, ~509 requests per second. Again, we must note that FastAPI and Express JS apps are running on one worker by default. In both frameworks, we see a single-core performance.

Now comes our star performer - Fiber (Golang)

Fiber - 5k requests

Fiber takes less than a second to process 5k requests! Since Fiber framework is implemented in Golang, concurrency is handled out of the box which means there is no need to specify async await syntax in your code.

CPU utilization by Fiber (Golang)

Now, you may be wondering Golang is using all CPU cores, but FastAPI and ExpressJS are utilizing only one CPU core per instance. Let's see how they perform when running on all CPU cores.

FastAPI Async with 4 workers on 4 Cores

ExpressJS Async with 4 workers on 4 Cores

ExpressJS is 1.5x faster than FastAPI. However, Fiber is still ~7.5 and ~11 times faster than ExpressJS and FastAPI.

What do Sync and Async mean?

Dave is also a champion chess player and, he has been challenged to play chess with not one but ten different players all at once! He has essentially two approaches

Approach 1- Dave decides to play one player at a time* and, if the average time per game is 10minutes (assuming other players are making quick decisions like our champ!), even then it takes 10 mins X 10 players = at least 100 minutes (i.e. over 1.6 hours)

Approach 2 - Dave plays all players at once! How is that possible? Every time an opponent takes time to think about their next move, Dave moves on to the next player and the next. It continues till the last move of the last remaining game.

Now swap the crazy madness of a guy playing chess with multiple people with a RESTful service. The opponents are external DB or service/APIs with which the RESTful service will communicate. Yes, that's how Sync & Async works and, since we are dealing with computers and not humans, they always perform actions as instructed!

Our thoughts -

Well, it is going to be a long one and yet may not be complete. Let's get to it, shall we?

Fiber (Golang) is way ahead of the game in terms of speed and raw performance. Express (NodeJS) is ~1.5x faster than FastAPI (Python) and, these frameworks are ~7.5 and ~11.35 times slower than Fiber (as per multi-core async performance).

However, this doesn't mean that you no longer use Python or NodeJS based frameworks. Factors like existing software stack, dev skills & experience in the team will play a significant role in making such decisions. Remember, good Go devs are hard to find compared to Python & JavaScript devs (at the time of writing this article) simply because these languages and their ecosystem have been here for longer.

A few salient features FastAPI brings like Automatic data model documentation, JSON validations, serialization and more. One could achieve the same results in the other two frameworks by writing more code as they are minimalistic by design.

We have used FastAPI extensively. Absolute performance becomes relevant when you are doing things at scale. These benchmarks are solely in terms of raw speed/throughput and do not consider development speed, DB I/O operations, JSON serialization and de-serialization etc.

A general recommendation is that you choose FastAPI when working with AI/ML-based services or, your team knows Python well. And same goes for NodeJS based framework decisions. There is no silver bullet here!

If you are starting a new project and you are willing to give GoLang a try. Fiber is a great place to start and, there are many frameworks to choose from

We hope this gave you some perspective. Feel free to share comments and feedback. Let us know if we should do another post comparing full-feature frameworks from NodeJS & Golang eco-system such as NestJS & Go-buffalo.

Source code -

GitHub - gochronicles/benchmark-rest-frameworks: Benchmarking RESTful APIs written in Python, NodeJS and Golang

If you want to follow further development on this project, make sure you give the repo a star! :-)

This article was originally posted on GoChronicles.com by the same author and has been repost with permission.

For latest updates & posts, please subscribe to GoChronicles.