DEV Community

moderation
moderation

Posted on • Edited on

1

Experiments with h3 clients + Envoy

I've been experimenting with HTTP/3 (h3) support in Envoy Proxy. I now have both upstream and downstream working

Inspired by the paper referred to in

I set out to expand the number of h3 clients for testing

Thanks to @howardjohn at Google for helping with the initial h3 config. Thanks to @triplewy1 for helping me sort out the correct parameters to pass to proxygen.

Big thanks to the Envoy team who have helped with configs, testing, ideas etc. In particular @alyssawilk, @danzh2010 + @mattklein123

Envoy

I'm building Envoy from source (main branch) on Linux with a limited set of extensions

Clients

I've ended up building and testing 7 h3 clients:

#. Client Language Compilation
1. curl/curl C cloudflare/quiche + BoringSSL
2. hyperium/h3 Rust musl static
3. proxygen/hq C++
4. mozilla/neqo Rust NSS
5. istio/quic-go Go
6. cloudflare/quiche Rust musl static
7. quinn-rs/quinn Rust musl static

Testing

hyperfine

I used the excellent hyperfine for testing. Please note that benchmarking is hard and this is in no way a proper benchmark. This is more for fun, learning how to build and use new h3 clients and working out how to configure h3 / QUIC for Envoy. Please take all results with a huge grain of salt

tl;dr - neqo generally slightly quicker followed closely by proxygen, quic-go + h3 (not always in that order). Then quinn, curl and cloudflare/quiche. I'm surprised by cloudflare/quiche being so slow however I believe it has not been optimized at this point

hyperfine --ignore-failure --warmup 5 --runs 100 \
'~/rust/curl/src/curl --silent --alt-svc ~/.altsvc.cache https://x1.local:4433/get' \
'~/rust/h3/target/x86_64-unknown-linux-musl/release/examples/client https://x1.local:4433/get' \
'~/proxygen/proxygen/_build/proxygen/httpserver/hq -logdir=/tmp -mode=client -early_data=true -connect_udp=true -draft-version=29 -port=4433 -host=x1.local -path=/get' \
'~/istio/pkg/test/echo/cmd/client/client --http3 https://x1.local:4433/get' \
'~/rust/neqo/target/release/neqo-client --output-read-data --resume --alpn h3-29 https://x1.local:4433/get' \
'~/rust/quiche/tools/apps/target/x86_64-unknown-linux-musl/release/quiche-client https://x1.local:4433/get' \
'~/rust/quinn/target/x86_64-unknown-linux-musl/release/examples/h3_client https://x1.local:4433/get'
Benchmark #1: ~/rust/curl/src/curl --silent --alt-svc ~/.altsvc.cache https://x1.local:4433/get
Time (mean ± σ): 146.1 ms ± 55.3 ms [User: 23.9 ms, System: 17.1 ms]
Range (min … max): 105.0 ms … 525.4 ms 100 runs
Benchmark #2: ~/rust/h3/target/x86_64-unknown-linux-musl/release/examples/client https://x1.local:4433/get
Time (mean ± σ): 129.2 ms ± 52.0 ms [User: 9.4 ms, System: 7.0 ms]
Range (min … max): 100.4 ms … 402.4 ms 100 runs
Benchmark #3: ~/proxygen/proxygen/_build/proxygen/httpserver/hq -logdir=/tmp -mode=client -early_data=true -connect_udp=true -draft-version=29 -port=4433 -host=x1.local -path=/get
Time (mean ± σ): 142.0 ms ± 55.8 ms [User: 16.9 ms, System: 9.0 ms]
Range (min … max): 104.4 ms … 507.2 ms 100 runs
Benchmark #4: ~/istio/pkg/test/echo/cmd/client/client --http3 https://x1.local:4433/get
Time (mean ± σ): 133.2 ms ± 70.2 ms [User: 10.1 ms, System: 9.3 ms]
Range (min … max): 101.6 ms … 625.9 ms 100 runs
Benchmark #5: ~/rust/neqo/target/release/neqo-client --output-read-data --resume --alpn h3-29 https://x1.local:4433/get
Time (mean ± σ): 125.7 ms ± 38.0 ms [User: 9.2 ms, System: 4.6 ms]
Range (min … max): 99.5 ms … 376.6 ms 100 runs
Benchmark #6: ~/rust/quiche/tools/apps/target/x86_64-unknown-linux-musl/release/quiche-client https://x1.local:4433/get
Time (mean ± σ): 218.5 ms ± 91.1 ms [User: 9.1 ms, System: 4.0 ms]
Range (min … max): 121.5 ms … 738.7 ms 100 runs
Benchmark #7: ~/rust/quinn/target/x86_64-unknown-linux-musl/release/examples/h3_client https://x1.local:4433/get
Time (mean ± σ): 151.0 ms ± 78.2 ms [User: 9.4 ms, System: 7.7 ms]
Range (min … max): 100.7 ms … 558.9 ms 100 runs
Summary
'~/rust/neqo/target/release/neqo-client --output-read-data --resume --alpn h3-29 https://x1.local:4433/get' ran
1.03 ± 0.52 times faster than '~/rust/h3/target/x86_64-unknown-linux-musl/release/examples/client https://x1.local:4433/get'
1.06 ± 0.64 times faster than '~/istio/pkg/test/echo/cmd/client/client --http3 https://x1.local:4433/get'
1.13 ± 0.56 times faster than '~/proxygen/proxygen/_build/proxygen/httpserver/hq -logdir=/tmp -mode=client -early_data=true -connect_udp=true -draft-version=29 -port=4433 -host=x1.local -path=/get'
1.16 ± 0.56 times faster than '~/rust/curl/src/curl --silent --alt-svc ~/.altsvc.cache https://x1.local:4433/get'
1.20 ± 0.72 times faster than '~/rust/quinn/target/x86_64-unknown-linux-musl/release/examples/h3_client https://x1.local:4433/get'
1.74 ± 0.90 times faster than '~/rust/quiche/tools/apps/target/x86_64-unknown-linux-musl/release/quiche-client https://x1.local:4433/get'

h3spec

I've also tested using the excellent h3spec. We found one crashing bug using this test suite which has subsequently been fixed

QUIC servers
MUST send FLOW_CONTROL_ERROR if a STREAM frame with a large offset is received [Transport 4.1]
MUST send TRANSPORT_PARAMETER_ERROR if initial_source_connection_id is missing [Transport 7.3]
MUST send TRANSPORT_PARAMETER_ERROR if original_destination_connection_id is received [Transport 18.2]
MUST send TRANSPORT_PARAMETER_ERROR if preferred_address, is received [Transport 18.2]
MUST send TRANSPORT_PARAMETER_ERROR if retry_source_connection_id is received [Transport 18.2]
MUST send TRANSPORT_PARAMETER_ERROR if stateless_reset_token is received [Transport 18.2]
MUST send TRANSPORT_PARAMETER_ERROR if max_udp_payload_size < 1200 [Transport 7.4 and 18.2]
MUST send TRANSPORT_PARAMETER_ERROR if ack_delay_exponen > 20 [Transport 7.4 and 18.2]
MUST send TRANSPORT_PARAMETER_ERROR if max_ack_delay >= 2^14 [Transport 7.4 and 18.2]
MUST send FRAME_ENCODING_ERROR if a frame of unknown type is received [Transport 12.4]
MUST send PROTOCOL_VIOLATION on no frames [Transport 12.4] FAILED [1]
MUST send PROTOCOL_VIOLATION if reserved bits in Handshake are non-zero [Transport 17.2] FAILED [2]
MUST send PROTOCOL_VIOLATION if PATH_CHALLENGE in Handshake is received [Transport 17.2.4]
MUST send PROTOCOL_VIOLATION if reserved bits in Short are non-zero [Transport 17.2] FAILED [3]
MUST send STREAM_STATE_ERROR if RESET_STREAM is received for a send-only stream [Transport 19.4]
MUST send STREAM_STATE_ERROR if STOP_SENDING is received for a non-existing stream [Transport 19.5]
MUST send PROTOCOL_VIOLATION if NEW_TOKEN is received [Transport 19.7]
MUST send STREAM_STATE_ERROR if MAX_STREAM_DATA is received for a non-existing stream [Transport 19.10]
MUST send STREAM_STATE_ERROR if MAX_STREAM_DATA is received for a receive-only stream [Transport 19.10]
MUST send FRAME_ENCODING_ERROR if invalid MAX_STREAMS is received [Transport 19.11]
MUST send STREAM_LIMIT_ERROR or FRAME_ENCODING_ERROR if invalid STREAMS_BLOCKED is received [Transport 19.14]
MUST send FRAME_ENCODING_ERROR if NEW_CONNECTION_ID with invalid Retire_Prior_To is received [Transport 19.15]
MUST send FRAME_ENCODING_ERROR if NEW_CONNECTION_ID with 0-byte CID is received [Transport 19.15] FAILED [4]
MUST send PROTOCOL_VIOLATION if HANDSHAKE_DONE is received [Transport 19.20]
MUST send unexpected_message TLS alert if KeyUpdate in Handshake is received [TLS 6]
MUST send unexpected_message TLS alert if KeyUpdate in 1-RTT is received [TLS 6] FAILED [5]
MUST send no_application_protocol TLS alert if no application protocols are supported [TLS 8.1]
MUST send missing_extension TLS alert if the quic_transport_parameters extension does not included [TLS 8.2] FAILED [6]
MUST send unexpected_message TLS alert if EndOfEarlyData is received [TLS 8.3]
MUST send PROTOCOL_VIOLATION if CRYPTO in 0-RTT is received [TLS 8.3] FAILED [7]
HTTP/3 servers
MUST send H3_FRAME_UNEXPECTED if DATA is received before HEADERS [HTTP/3 4.1]
MUST send H3_MESSAGE_ERROR if a pseudo-header is duplicated [HTTP/3 4.1.1] FAILED [8]
MUST send H3_MESSAGE_ERROR if mandatory pseudo-header fields are absent [HTTP/3 4.1.3] FAILED [9]
MUST send H3_MESSAGE_ERROR if prohibited pseudo-header fields are present[HTTP/3 4.1.3] FAILED [10]
MUST send H3_MESSAGE_ERROR if pseudo-header fields exist after fields [HTTP/3 4.1.3] FAILED [11]
MUST send H3_MISSING_SETTINGS if the first control frame is not SETTINGS [HTTP/3 6.2.1]
MUST send H3_FRAME_UNEXPECTED if a DATA frame is received on a control stream [HTTP/3 7.2.1]
MUST send H3_FRAME_UNEXPECTED if a HEADERS frame is received on a control stream [HTTP/3 7.2.2]
MUST send H3_FRAME_UNEXPECTED if a second SETTINGS frame is received [HTTP/3 7.2.4]
MUST send H3_SETTINGS_ERROR if HTTP/2 settings are included [HTTP/3 7.2.4.1]
MUST send H3_FRAME_UNEXPECTED if CANCEL_PUSH is received in a request stream [HTTP/3 7.2.5] FAILED [12]
MUST send QPACK_DECOMPRESSION_FAILED if an invalid static table index exits in a field line representation [QPACK 3.1]
MUST send QPACK_ENCODER_STREAM_ERROR if a new dynamic table capacity value exceeds the limit [QPACK 4.1.3]
MUST send H3_CLOSED_CRITICAL_STREAM if a control stream is closed [QPACK 4.2] FAILED [13]
MUST send QPACK_DECODER_STREAM_ERROR if Insert Count Increment is 0 [QPACK 4.4.3]

45 examples, 13 failures. This suite has been great for catching crashes but it should be noted the goal is not to attain 100% as there are a number of performance trade-offs to consider

Config

Downstream h3 with local direct responses + h2 upstream

The first config shows how to set up a TCP + UDP listener on the same port, testing JSON structed logging, an Envoy direct response on /local, alt-svc headers on h2 responses

I use CUE for all of these configs and these are exported YAML. The process is started on the Fish shell with:

<path to>/envoy --concurrency 1 --log-level debug --config-path (cue export downstream_httpbin_org.cue | psub)

Note that for h3 to worked today you'll need to set concurrency to 1

admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /tmp/admin_access.log
address:
socket_address:
address: ::0
ipv4_compat: true
port_value: 9901
protocol: TCP
layered_runtime:
layers:
- name: static-layer
static_layer:
envoy.http.headermap.lazy_map_min_size: 3
envoy.reloadable_features.new_tcp_connection_pool: true
envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing: true
envoy.reloadable_features.remove_legacy_json: true
static_resources:
clusters:
- name: service_httpbin
connect_timeout: 2s
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_httpbin
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: httpbin.org
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
'@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
alpn_protocols: h2
sni: httpbin.org
type: LOGICAL_DNS
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
'@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options: {}
listeners:
- name: listener_udp
address:
socket_address:
address: ::0
ipv4_compat: true
port_value: 4433
protocol: UDP
filter_chains:
- filter_chain_match:
transport_protocol: quic
transport_socket:
name: envoy.transport_sockets.quic
typed_config:
'@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
downstream_tls_context:
common_tls_context:
alpn_protocols: h3
tls_certificates:
certificate_chain:
filename: cert.pem
private_key:
filename: cert-key.pem
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
access_log:
- name: envoy.access_loggers.file
typed_config:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
log_format:
json_format:
bytes_received: '%BYTES_RECEIVED%'
bytes_sent: '%BYTES_SENT%'
duration: '%DURATION%'
http_response: '%RESPONSE_CODE%'
protocol: '%PROTOCOL%'
request_authority: '%REQ(:AUTHORITY)%'
request_method: '%REQ(:METHOD)%'
request_path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
response_flag: '%RESPONSE_FLAGS%'
start_time: '%START_TIME%'
upstream_host: '%UPSTREAM_HOST%'
upstream_service_time: '%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%'
user_agent: '%REQ(USER-AGENT)%'
x_forwarded_for: '%REQ(X-FORWARDED-FOR)%'
x_request_id: '%REQ(X-REQUEST-ID)%'
path: http3_downstream.log
codec_type: HTTP3
http_filters:
- name: envoy.filters.http.router
route_config:
name: local_route
max_direct_response_body_size_bytes: 428
virtual_hosts:
- name: local_service
domains:
- '*'
routes:
- match:
prefix: /local
direct_response:
body:
inline_string: |-
<!DOCTYPE html>
<head><meta charset=utf-8><title>envoy_http3_downstream</title><link rel=icon href="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogIDxyZWN0IHg9IjAiIHk9IjAiIHdpZHRoPSI1MDAiIGhlaWdodD0iNTAwIiBzdHlsZT0iZmlsbDojZmYwMGZmO3N0cm9rZTojZmYwMGZmOyIgLz4KPC9zdmc+IA==" type=image/svg+xml sizes=any><style> body { font-size: 70px; }</style></head><body>direct_response: You found it!</body>
status: 200
response_headers_to_add:
- header:
key: content-type
value: text/html;charset=utf-8
- match:
prefix: /
route:
cluster: service_httpbin
host_rewrite_literal: httpbin.org
stat_prefix: ingress_h3
reuse_port: true
udp_listener_config:
downstream_socket_config:
prefer_gro: true
quic_options: {}
- name: listener_tcp
address:
socket_address:
address: ::0
ipv4_compat: true
port_value: 4433
protocol: TCP
filter_chains:
- transport_socket:
name: envoy.transport_sockets.tls
typed_config:
'@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
certificate_chain:
filename: cert.pem
private_key:
filename: cert-key.pem
alpn_protocols: h2
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
access_log:
- name: envoy.access_loggers.file
typed_config:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: http3_downstream.log
http_filters:
- name: envoy.filters.http.router
route_config:
name: local_route
max_direct_response_body_size_bytes: 428
virtual_hosts:
- domains:
- '*'
name: local_service
response_headers_to_add:
- header:
key: alt-svc
value: h3=":4433"; ma=86400, h3-29=":4433"; ma=86400
routes:
- match:
prefix: /local
direct_response:
body:
inline_string: |-
<!DOCTYPE html>
<head><meta charset=utf-8><title>envoy_http3_downstream</title><link rel=icon href="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogIDxyZWN0IHg9IjAiIHk9IjAiIHdpZHRoPSI1MDAiIGhlaWdodD0iNTAwIiBzdHlsZT0iZmlsbDojZmYwMGZmO3N0cm9rZTojZmYwMGZmOyIgLz4KPC9zdmc+IA==" type=image/svg+xml sizes=any><style> body { font-size: 70px; }</style></head><body>direct_response: You found it!</body>
status: 200
response_headers_to_add:
- header:
key: content-type
value: text/html;charset=utf-8
- match:
prefix: /
route:
cluster: service_httpbin
host_rewrite_literal: httpbin.org
stat_prefix: ingress_h2

Downstream h1 with h3 upstream

This is a simpler config with a stock h1 listener but talks h3 to the upstream service

admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /tmp/admin_access.log
address:
socket_address:
address: ::0
ipv4_compat: true
port_value: 9902
protocol: TCP
layered_runtime:
layers:
- name: static-layer
static_layer:
envoy.http.headermap.lazy_map_min_size: 3
envoy.reloadable_features.new_tcp_connection_pool: true
envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing: true
envoy.reloadable_features.remove_legacy_json: true
static_resources:
clusters:
- name: service_quic_tech
connect_timeout: 2s
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_quic_tech
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: quic.tech
port_value: 8443
transport_socket:
name: envoy.transport_sockets.quic
typed_config:
'@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicUpstreamTransport
upstream_tls_context:
allow_renegotiation: true
common_tls_context:
alpn_protocols: h3-29
sni: quic.tech
type: LOGICAL_DNS
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
'@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
common_http_protocol_options:
idle_timeout: 1s
explicit_http_config:
http3_protocol_options: {}
upstream_http_protocol_options:
auto_sni: true
listeners:
- name: listener_0
address:
socket_address:
address: ::0
ipv4_compat: true
port_value: 10000
protocol: TCP
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains:
- '*'
routes:
- match:
prefix: /
route:
host_rewrite_literal: quic.tech
cluster: service_quic_tech
http_filters:
- name: envoy.filters.http.router

Future

  • Would be fun to test this with things like dynamic forward proxies
  • The testing above is done on an Envoy proxy with a the runtime value envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing: true set and Linux Capabilities of sudo setcap cap_bpf+ep <path to>/envoy on a kernel >= 5.8.x. However as per the following issue it is not sure what effect this has

    Docs: clarify QUIC BPF operation #15845

    Title: Docs: clarify QUIC BPF operation

    Description: Prompted by the Twitter chat at https://twitter.com/mattklein123/status/1378172039870091265 I looked into the runtime flag that allows QUIC routing in the kernal via BPF - https://github.com/envoyproxy/envoy/blob/main/source/common/runtime/runtime_features.cc#L82

    layered_runtime:                                                             
      layers:                                                                    
        - name: static-layer                                                     
          static_layer:                                                          
            envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing: true
    Enter fullscreen mode Exit fullscreen mode

    Installing BPF rules like this requires one of:

    1. Envoy is running as root
    2. For kernels >= 5.8, Envoy is running with sudo setcap cap_bpf+ep <envoy binary>
    3. For kernels < 5.8, Envoy is running with sudo cap_net_admin,cap_sys_admin+ep <envoy binary>

    From initial testing Envoy doesn't display any different output when launched in different modes. QUIC / h3 listeners work whether Envoy was launched with the elevated permissions or not.

    It would be good to clarify in the docs what steps need to be taken to enable QUIC BPF kernel routing and what platforms work and don't work. It looks like this is Linux only at the moment. It might be worthwhile logging whether the BPF rule has been installed successfully - https://github.com/envoyproxy/envoy/blob/main/source/common/quic/active_quic_listener.cc#L234-L298

    Relevant Links: Handy reference on determine what Linux Capabilities your system supports - https://linux-audit.com/linux-capabilities-101/

    /cc @ggreenway @alyssawilk @danzh2010 @mattklein123

Updates

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

Top comments (1)

Collapse
 
alansill profile image
Alan Sill

Interested in helping me write up the current state of this topic for a technical article in IEEE Cloud Continuum? Let me know if so and I'll send you some questions.

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more