Experiments with h3 clients + Envoy

I've been experimenting with HTTP/3 (h3) support in Envoy Proxy. I now have both upstream and downstream working

Inspired by the paper referred to in

Theophilus A. Benson

@dottheophilus

Interested to understand why (or rather when) QUIC (HTTP3) outperforms TCP (HTTP2)? Check out my new WWW paper w/ @triplewy1 on demystifying QUIC: cs.brown.edu/~tab/papers/QU…

Hint: Performance is a function of implementation and operational choices not necessarily the protocol.

06:41 AM - 16 Feb 2021

I set out to expand the number of h3 clients for testing

Thanks to @howardjohn at Google for helping with the initial h3 config. Thanks to @triplewy1 for helping me sort out the correct parameters to pass to proxygen.

Big thanks to the Envoy team who have helped with configs, testing, ideas etc. In particular @alyssawilk, @danzh2010 + @mattklein123

Envoy

I'm building Envoy from source (main branch) on Linux with a limited set of extensions

Clients

I've ended up building and testing 7 h3 clients:

#.	Client	Language	Compilation
1.	curl/curl	C	cloudflare/quiche + BoringSSL
2.	hyperium/h3	Rust	musl static
3.	proxygen/hq	C++
4.	mozilla/neqo	Rust	NSS
5.	istio/quic-go	Go
6.	cloudflare/quiche	Rust	musl static
7.	quinn-rs/quinn	Rust	musl static

Testing

hyperfine

I used the excellent hyperfine for testing. Please note that benchmarking is hard and this is in no way a proper benchmark. This is more for fun, learning how to build and use new h3 clients and working out how to configure h3 / QUIC for Envoy. Please take all results with a huge grain of salt

tl;dr - neqo generally slightly quicker followed closely by proxygen, quic-go + h3 (not always in that order). Then quinn, curl and cloudflare/quiche. I'm surprised by cloudflare/quiche being so slow however I believe it has not been optimized at this point

h3spec

I've also tested using the excellent h3spec. We found one crashing bug using this test suite which has subsequently been fixed

45 examples, 13 failures. This suite has been great for catching crashes but it should be noted the goal is not to attain 100% as there are a number of performance trade-offs to consider

Config

Downstream h3 with local direct responses + h2 upstream

The first config shows how to set up a TCP + UDP listener on the same port, testing JSON structed logging, an Envoy direct response on /local, alt-svc headers on h2 responses

I use CUE for all of these configs and these are exported YAML. The process is started on the Fish shell with:

<path to>/envoy --concurrency 1 --log-level debug --config-path (cue export downstream_httpbin_org.cue | psub)

Note that for h3 to worked today you'll need to set concurrency to 1

Downstream h1 with h3 upstream

This is a simpler config with a stock h1 listener but talks h3 to the upstream service

Future

• Would be fun to test this with things like dynamic forward proxies
• The testing above is done on an Envoy proxy with a the runtime value envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing: true set and Linux Capabilities of sudo setcap cap_bpf+ep <path to>/envoy on a kernel >= 5.8.x. However as per the following issue it is not sure what effect this has

  Docs: clarify QUIC BPF operation #15845

  

  moderation posted on Apr 06, 2021

  Title: Docs: clarify QUIC BPF operation

  Description: Prompted by the Twitter chat at https://twitter.com/mattklein123/status/1378172039870091265 I looked into the runtime flag that allows QUIC routing in the kernal via BPF - https://github.com/envoyproxy/envoy/blob/main/source/common/runtime/runtime_features.cc#L82

  layered_runtime:                                                             
    layers:                                                                    
      - name: static-layer                                                     
        static_layer:                                                          
          envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing: true

  Installing BPF rules like this requires one of:

  1. Envoy is running as root
  2. For kernels >= 5.8, Envoy is running with sudo setcap cap_bpf+ep <envoy binary>
  3. For kernels < 5.8, Envoy is running with sudo cap_net_admin,cap_sys_admin+ep <envoy binary>

  From initial testing Envoy doesn't display any different output when launched in different modes. QUIC / h3 listeners work whether Envoy was launched with the elevated permissions or not.

  It would be good to clarify in the docs what steps need to be taken to enable QUIC BPF kernel routing and what platforms work and don't work. It looks like this is Linux only at the moment. It might be worthwhile logging whether the BPF rule has been installed successfully - https://github.com/envoyproxy/envoy/blob/main/source/common/quic/active_quic_listener.cc#L234-L298

  Relevant Links: Handy reference on determine what Linux Capabilities your system supports - https://linux-audit.com/linux-capabilities-101/

  /cc @ggreenway @alyssawilk @danzh2010 @mattklein123

  View on GitHub

Updates

• quinn removed support for their h3 test client in https://github.com/quinn-rs/quinn/pull/1155

Comments

[Alan Sill]

Interested in helping me write up the current state of this topic for a technical article in IEEE Cloud Continuum? Let me know if so and I'll send you some questions.