Hyperman into Fetch

#performance #programming #software #softwaredevelopment

This is the story of two pieces of software I published recently - Hyperman, an event-loop PSGI server, and Fetch, a Future based HTTP client - and of why building the first one enticed me to build the second.

This story really starts a few months ago when I published another module Hypersonic. Hypersonic is fast, really fast but it's also flawed in its just in time nature when it comes to compiling. I have deliberately left Hypersonic out of the numbers below - it is not the point here, and lining that "crazy" JIT up against ahead-of-time hard code mostly invites the wrong conclusions.

What Hypersonic taught me was where the speed actually comes from, and Hyperman and Fetch are an attempt to keep that speed while moving the compilation to a saner time. Instead of compiling just in time, the hot path is written once, in C, and compiled ahead of time into XS - so there is no warmup and the code running under load is exactly the code you, I and cpantesters has tested.

All numbers below were measured on a 10-core macOS laptop (4 performance + 6 efficiency cores) over loopback, with a trivial PSGI app returning a 256-byte body. They are relative, your mileage will vary.

Hyperman

Hyperman is a preforking, event-loop PSGI server. A supervisor process forks a pool of workers, each worker runs a single-threaded event loop written in C that accepts connections, parses HTTP, calls your PSGI app, and writes the response - all without blocking, so one worker can juggle thousands of connections at once.

Pluggable readiness backends behind one vtable. kqueue on macOS and the BSDs, epoll or io_uring on Linux, and a portable poll fallback everywhere else (Except windows). Selection is automatic; you can force one with HYPERMAN_BACKEND.

A real async story. Handlers can return a Hyperman::Future (or any on_ready-compatible object) instead of a response, and the worker parks the connection and services others until the future resolves. psgi.streaming delayed responses, a streaming writer, psgix.io, and cancellation on client disconnect all work. The Future implementation is entirely XS over an array slot object, with continuations trampolined through a fire queue so long then-chains run iteratively with bounded stack depth.

Production behaviour. Worker respawn with crash loop backoff, SIGHUP zero-downtime recycle, graceful TERM/INT drain, idle and slow request timeouts, a pipelining fairness cap, a request-size ceiling, opt-in SO_REUSEPORT per-worker listeners on Linux, an access-log callback, and per-worker stats.

HTTP/2 and TLS. A non-blocking OpenSSL layer with SNI and ALPN, and an HTTP/2 path (nghttp2 for framing and HPACK) negotiated via ALPN over TLS. Both degrade gracefully to stubs when the libraries are absent - HTTP/1.1 keeps working with no OpenSSL at all.

It runs any Plack app: plackup -s Hyperman.

How fast is it?

Same PSGI app, same load (wrk -t4 -c100), access logging off (PLACK_ENV=deployment), four workers where the server supports them, best of three runs:

Server	req/s
HTTP::Server::PSGI (1 process)	17,231
Gazelle (4 workers)	34,490
Starman (4 workers)	76,403
Hyperman (4 workers)	185,932

Roughly 2.4x Starman and an order of magnitude over the single-process core server, on this box. (Gazelle is tuned for Linux with Server::Starter and SO_REUSEPORT; it likely does better in its native habitat than under plackup on macOS. The point of the table is the harness is identical for everyone.)

The problem: how do you even measure that?

Here is the wall I hit. A server that can answer ~186,000 requests a second is only interesting if you can prove it, and to prove it you need a client that can ask 186,000 times a second. That turns out to be surprisingly hard to do from Perl.

The standard Perl HTTP clients are all blocking: they send one request, wait for the reply, then send the next. One request in flight at a time, per process. So the number you measure is not the server's capacity - it is the client's round trip latency. Point a blocking client at Hyperman and you learn how fast the client is, and conclude, wrongly, that your server does ~30k/s.

You can work around it by forking dozens of client processes and summing their throughput, but that is fiddly to coordinate, noisy to measure, and awkward to put in a test suite. You can reach for wrk or ab - excellent C tools - but now your all Perl stack has a C dependency in its own test harness, and you cannot easily express "hammer this server with 500 concurrent requests and assert the p99 latency" as a Perl test.

What I actually wanted was a Perl HTTP client that could keep hundreds of requests in flight on one event loop - fast enough to saturate Hyperman from inside Perl. There wasn't one other than what exists inside of Hypersonic. So Fetch happened.

Part 2: Fetch

Fetch is an HTTP user agent whose socket, TLS, HTTP/2 framing and HTTP/1.1 parsing live in vendored C, and whose asynchronous results are Fetch::Future objects. It works out of the box with no event loop to set up - it ships its own - and it cooperates with an existing loop when you have one.

use Fetch;

# no framework, no setup
my $res = Fetch->new->get('https://example.com')->get;
print $res->content;

# hundreds of requests, one loop, all concurrent
my @f = map { $ua->get($_) } @urls;
Fetch::Future->needs_all(@f)->get;

The same C core as the server. Fetch reuses Hyperman's readiness backends and its Future primitive. A request is a non-blocking connect + send + recv + parse driven by a C readiness callback - no Perl runs per event. HTTP/1.1 and HTTP/2 (ALPN over TLS), streaming bodies, redirect following, per-request timeouts, cookies (RFC 6265), keep-alive connection pooling, WebSockets (RFC 6455), and JSON decode are all there, with the hot paths in C.

It rides any loop. Fetch's C core arms readiness through one tiny method an adapter provides, so Fetch runs on IO::Async, AnyEvent, or - the natural pairing - a Hyperman::Loop. A Hyperman worker can make concurrent outbound HTTP/2 calls on the same loop it serves inbound connections with.

How fast is it?

All clients pointed at one Hyperman server, keep-alive on, measured with the core Benchmark module (cmpthese), one request per iteration:

Client	req/s	vs LWP
LWP::UserAgent	9,934	1.0x
HTTP::Tiny	25,862	2.6x
Furl	41,379	4.2x
Fetch (sequential)	100,000	10x

Sequentially, Fetch is already ~10x LWP and ~2.4x Furl (the fastest of the blocking clients), because the request/response hot path never leaves C.

But sequential is not the point. The point is this:

Client	req/s
Fetch, 200 requests in flight	164,342

One process, one core, 200 requests concurrently on a single event loop - and now we are within striking distance of the 186k the server can serve, and far past anything a blocking client can do. (Benchmark's one-call-per-iteration model can't express in-flight concurrency, so that last figure is a straight wall-clock measurement over the batch.)

Okay we are not quite wrk just yet, but I have a few ideas on how to get that with an option for a more optimised response without any blessing of objects... BUT we are close enough with a full featured user agent and currently far ahead of other perl equivalents.