HttpArena - Benchmark Web Frameworks

#opensource #performance #showdev #webdev

HttpArena is a recent project which goal is to build an open source platform where web frameworks are benchmarked for throughput performance, CPU usage, memory consumption and latency. Sounds familiar? Yes.. there are a few of them so what does HttpArena brings different?

Much broader test coverage including Http/1.1, Http/2, Http/3, gRPC and Websocket tests.
Community driven, we are not a company or sponsored by companies that compete in the benchmarks.
Tries to benchmark workloads closer to you see in real world applications including benchmarks with reverse proxies, caching services like redis and distributed systems.
Cares all about competitive fairness, entries are subdivided by Production, Tuned, Infrastructure and Engine, these have clear specific rules to avoid comparing apples to oranges and potatoes.

So, what does HttpArena wants to benchmark?

Both micro benchmarks and workload types, see the full test suite here.

Target audience

We target both framework developers and users, while micro benchmarks target development metrics, every test specially workload like ones can be useful for users when picking a technology or framework.

Hardware

Currently we run all the benchmarks in a single box, a 64 core AMD Threadripper with 256GB RAM. This has pros and cons, some pros are the fact that we don't get networking bottlenecked as we would if using a multiple box option like having 2 or more servers. The cons are sharing the cpu between server and load generators, while this is not optimal we take a lot of measures to minimize this such as running the servers and load generators in separate containers with pinned Cores for each, see Hardware and Topology.

Who can join?

Everyone is welcome! Not just to add a framework but also to improve the existing implementations and give us valuable feedback and ideas on existing and new tests.