Experiments with h3 clients + Envoy

moderation — Tue, 13 Apr 2021 03:07:13 +0000

I've been experimenting with HTTP/3 (h3) support in Envoy Proxy. I now have both upstream and downstream working

Inspired by the paper referred to in

Theophilus A. Benson

@dottheophilus

Interested to understand why (or rather when) QUIC (HTTP3) outperforms TCP (HTTP2)? Check out my new WWW paper w/ @triplewy1 on demystifying QUIC: cs.brown.edu/~tab/papers/QU…

Hint: Performance is a function of implementation and operational choices not necessarily the protocol.

06:41 AM - 16 Feb 2021

I set out to expand the number of h3 clients for testing

Thanks to @howardjohn at Google for helping with the initial h3 config. Thanks to @triplewy1 for helping me sort out the correct parameters to pass to proxygen.

Big thanks to the Envoy team who have helped with configs, testing, ideas etc. In particular @alyssawilk, @danzh2010 + @mattklein123

Envoy

I'm building Envoy from source (main branch) on Linux with a limited set of extensions

Clients

I've ended up building and testing 7 h3 clients:

#.	Client	Language	Compilation
1.	curl/curl	C	cloudflare/quiche + BoringSSL
2.	hyperium/h3	Rust	musl static
3.	proxygen/hq	C++
4.	mozilla/neqo	Rust	NSS
5.	istio/quic-go	Go
6.	cloudflare/quiche	Rust	musl static
7.	quinn-rs/quinn	Rust	musl static

Testing

hyperfine

I used the excellent hyperfine for testing. Please note that benchmarking is hard and this is in no way a proper benchmark. This is more for fun, learning how to build and use new h3 clients and working out how to configure h3 / QUIC for Envoy. Please take all results with a huge grain of salt

tl;dr - neqo generally slightly quicker followed closely by proxygen, quic-go + h3 (not always in that order). Then quinn, curl and cloudflare/quiche. I'm surprised by cloudflare/quiche being so slow however I believe it has not been optimized at this point

h3spec

I've also tested using the excellent h3spec. We found one crashing bug using this test suite which has subsequently been fixed

45 examples, 13 failures. This suite has been great for catching crashes but it should be noted the goal is not to attain 100% as there are a number of performance trade-offs to consider

Config

Downstream h3 with local direct responses + h2 upstream

The first config shows how to set up a TCP + UDP listener on the same port, testing JSON structed logging, an Envoy direct response on /local, alt-svc headers on h2 responses

I use CUE for all of these configs and these are exported YAML. The process is started on the Fish shell with:

<path to>/envoy --concurrency 1 --log-level debug --config-path (cue export downstream_httpbin_org.cue | psub)

Note that for h3 to worked today you'll need to set concurrency to 1

Downstream h1 with h3 upstream

This is a simpler config with a stock h1 listener but talks h3 to the upstream service

Future

Would be fun to test this with things like dynamic forward proxies
The testing above is done on an Envoy proxy with a the runtime value envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing: true set and Linux Capabilities of sudo setcap cap_bpf+ep <path to>/envoy on a kernel >= 5.8.x. However as per the following issue it is not sure what effect this has
Docs: clarify QUIC BPF operation #15845
moderation posted on Apr 06, 2021
Title: Docs: clarify QUIC BPF operation

Description: Prompted by the Twitter chat at https://twitter.com/mattklein123/status/1378172039870091265 I looked into the runtime flag that allows QUIC routing in the kernal via BPF - https://github.com/envoyproxy/envoy/blob/main/source/common/runtime/runtime_features.cc#L82
layered_runtime: layers: - name: static-layer static_layer: envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing: true
Installing BPF rules like this requires one of:
1. Envoy is running as root
2. For kernels >= 5.8, Envoy is running with sudo setcap cap_bpf+ep <envoy binary>
3. For kernels < 5.8, Envoy is running with sudo cap_net_admin,cap_sys_admin+ep <envoy binary>
From initial testing Envoy doesn't display any different output when launched in different modes. QUIC / h3 listeners work whether Envoy was launched with the elevated permissions or not.

It would be good to clarify in the docs what steps need to be taken to enable QUIC BPF kernel routing and what platforms work and don't work. It looks like this is Linux only at the moment. It might be worthwhile logging whether the BPF rule has been installed successfully - https://github.com/envoyproxy/envoy/blob/main/source/common/quic/active_quic_listener.cc#L234-L298

Relevant Links: Handy reference on determine what Linux Capabilities your system supports - https://linux-audit.com/linux-capabilities-101/

/cc @ggreenway @alyssawilk @danzh2010 @mattklein123
View on GitHub

Updates

quinn removed support for their h3 test client in https://github.com/quinn-rs/quinn/pull/1155

Hello dev.to

moderation — Mon, 06 May 2019 04:48:26 +0000

Setting up my dev.to

DEV Community: moderation