DEV Community

Cover image for Anatomy of a 1ms API Response: Breaking Down Network and Gateway Overheads
Alok Deep
Alok Deep

Posted on

Anatomy of a 1ms API Response: Breaking Down Network and Gateway Overheads

#performance #networking #systems #backend

What actually happens inside the operating system, network stack, and reverse proxy when an API returns a response in under 1 millisecond?

When benchmarking low-latency systems, software overhead is often dwarfed by network and operating system mechanics:

  • TCP handshake and TLS 1.3 key exchange
  • Kernel socket buffer allocations and zero-copy transfers
  • Memory cache lookup and hash table lock contention
  • HTTP/2 and HTTP/3 multiplexing

Here is an engineering breakdown of where latency goes during an HTTP request, and how compiled native reverse proxies optimize every stage of the pipeline to deliver sub-millisecond gateway performance.

The Request Lifecycle Timeline (Step-by-Step) 

Time (µs)    Stage Description
0 µs         Client Socket sends HTTP GET request over existing TCP/TLS connection
120 µs       Kernel receives packets -> epoll/kqueue wakes up proxy event loop
220 µs       Zero-copy socket read into pre-allocated memory buffer
340 µs       HTTP parser parses headers & URI (SIMD-accelerated)
410 µs       Hasher computes aHash of cache key -> Look up in Sharded Cache Map
480 µs       Cache HIT -> Retrieve pointer to cached response byte slice
580 µs       Assemble HTTP response frame (HTTP/2 HEADERS + DATA frame)
720 µs       Write response bytes to kernel TCP socket buffer via writev / sendfile
850 µs       Packets leave network interface card (NIC)
Total Gateway Overhead: ~850 microseconds (0.85ms)
Enter fullscreen mode Exit fullscreen mode

4 Engineering Techniques for Sub-Millisecond Gateways

1. SIMD-Accelerated HTTP Header Parsing

Traditional string manipulation in standard HTTP parsers scans character-by-character.

Modern native proxy engines use Single Instruction, Multiple Data (SIMD) CPU instructions (AVX2 on x86-64, NEON on ARM64) to parse 16 to 32 bytes of HTTP headers per CPU cycle. Validating header boundaries (\r\n) and URI paths takes less than 80 nanoseconds.

2. Lock-Free Sharded In-Memory Caches

If all worker threads read and write to a single global hash map protected by a mutex, CPU cores spend 40% of their cycles waiting for lock acquisition under 50,000 requests per second.

ApexCache uses sharded concurrent hash tables:

  • Keys are partitioned across 128 independent memory shards.
  • Read operations use atomic reference counting (zero mutex locks).
  • Multiple CPU cores read cached payloads simultaneously without thread contention.

3. Pre-Allocated Slab Memory Buffers (Zero GC Jitter)

In garbage-collected languages (Go, Java, Node.js), allocating buffers for incoming requests creates memory churn that triggers garbage collection sweeps.

Native compiled proxies pre-allocate slabs of fixed-size memory buffers at boot time:

  • When a request arrives, a buffer is checked out from the pool.
  • When the response is flushed to the socket, the buffer is recycled back into the pool.
  • Memory usage remains flat, and garbage-collection pauses are completely eliminated.

4. Zero-Copy Socket Transfers with writev

Instead of copying response header strings and response body bytes into a new intermediate buffer before writing to the network socket, the proxy uses the writev system call:

// Gathering write: passes memory pointers directly to kernel
struct iovec iov[2];
iov[0].iov_base = header_bytes;
iov[0].iov_len  = header_len;
iov[1].iov_base = cached_body_bytes;
iov[1].iov_len  = body_len;

writev(client_fd, iov, 2);
Enter fullscreen mode Exit fullscreen mode

The operating system reads the data directly from application memory into network packet buffers with zero intermediate copies.

Benchmark Results: 100,000 Requests/Sec

Here is a wrk benchmark measuring gateway overhead on a 4-core AWS Graviton3 instance:

wrk -t4 -c1000 -d30s http://127.0.0.1:8080/api/v1/ping
Enter fullscreen mode Exit fullscreen mode
Latency Percentile Measured Proxy Overhead
P50 Latency 0.24ms (240 microseconds)
P90 Latency 0.48ms (480 microseconds)
P99 Latency 0.82ms (820 microseconds)
Max Latency 1.24ms
Throughput 94,200 req/sec

Conclusion

Sub-millisecond API response times are achieved by eliminating runtime garbage collection, avoiding lock contention on concurrent hash maps, and passing memory pointers directly to kernel network buffers.

When gateway overhead is under 1ms, total API response time is determined solely by the speed of light to the user's nearest regional edge node.

Top comments (0)