A byte is usually beneath the resolution of capacity planning. In a cache holding more than 250 billion entries, it is a line item: one extra byte per entry consumes over 250 GB across the fleet.
That arithmetic changes how software should be designed. A convenient Vec<T> is no longer just an API choice. A repeated domain name is no longer harmless duplication. The padding inside an enum, the size class chosen by an allocator, and the distance between two heap objects all become infrastructure decisions.
Cloudflare’s Big Pineapple platform—the Rust system behind 1.1.1.1, Gateway DNS, DNS Firewall, and other DNS services—recently applied that way of thinking to its cache. Five related changes cut the benchmarked footprint of an entry from 953 bytes to 420 bytes. Fleet-wide working-set memory fell by roughly 100 TB. The same layout also increased insert throughput by 43% and reduced lookup latency by 19%.
The interesting lesson is not simply “use less memory.” It is how to make a mature hot-path data structure smaller without turning every read into decompression work. The winning design kept the fields that need interpretation structured, encoded the bulk record data compactly, and arranged the common path so that it could often copy bytes directly into the response.
The cache is a second representation of DNS
A recursive resolver receives a question such as “what is the A record for example.com?” If the answer is already cached and its time-to-live has not expired, the resolver can answer without repeating the upstream resolution process.
The key identifies the question. It includes the queried name and record type, plus context such as authentication state and service-specific tags. The value contains metadata and the DNS resource records divided into answer, authority, and additional sections.
That division follows the DNS message model defined by RFC 1035: a header and question are followed by three lists of resource records. A first implementation naturally mirrors the protocol with Rust structures:
struct CacheEntry {
created_at: Instant,
ttl: Ttl,
hits: u32,
answers: Vec<Record>,
authority: Vec<Record>,
additional: Vec<Record>,
// more metadata...
}
That is pleasant code. Each section has its own collection, each record is parsed into a typed enum, and callers can index or inspect fields directly. But this in-memory form is neither the compact DNS wire format nor a database page designed for density. It is a graph of headers, pointers, capacities, enums, and heap allocations.
The distinction matters: the representation that is easiest to build is not automatically the representation that should live for millions of lookups.
Big Pineapple fills from empty after startup. Once an instance reaches its configured entry count, it evicts older or less useful records to admit new ones. Cache size varies by location and traffic. EDNS Client Subnet can multiply entries because authoritative servers may return different answers for different client networks. Memory efficiency therefore affects both the number of answers a resolver can retain and the volume of upstream work it must repeat.
Measure the whole path, not just size_of
Rust’s size_of::<T>() is useful, but it cannot describe an object graph on its own. A 24-byte vector header may point to a much larger allocation. That allocation may reserve unused capacity. An allocator may round a 40-byte request into a 48-byte size class. Several individually small fields may force alignment padding into a containing struct.
Cloudflare built a benchmark around generated entries shaped like production traffic: roughly 56% A, 25% AAAA, and 19% variable-length records represented by TXT, with one to four records per entry. A custom allocator wrapper tracked the number and size of allocations. The benchmark also measured insert throughput and lookup latency so a lower byte count could not hide extra CPU work.
Synthetic measurements were only the first filter. Traffic mix, occupancy, allocator state, and non-cache process data all affect resident memory. Each release was therefore rolled out to production and checked against steady-state working-set measurements.
That two-level method is worth copying:
- Use a controlled benchmark to attribute changes to a representation decision.
- Use production memory and latency to verify that the decision survives real distributions and allocator behavior.
First: stop paying for growth after construction
On a 64-bit system, a Vec<T> conceptually carries a pointer, a length, and a capacity. Capacity is what makes appending efficient: the vector can grow into already reserved space before allocating again.
A cached DNS answer does not grow. It is assembled once, inserted, read many times, and eventually evicted. Retaining growth machinery throughout that lifetime pays for a capability the value will never use.
Converting a completed Vec<T> into Box<[T]> removes the capacity word from the long-lived header. Converting a String into Box<str> does the same for text. The Rust standard-library documentation explicitly recommends a boxed slice when excess capacity should be discarded and the long-lived structure should be smaller.
Each cache entry had eight vector or string fields. Removing one 8-byte capacity word from each saved 64 bytes in the entry header, before accounting for abandoned spare slots in the backing allocations. Multiplied across the cache, this first change was already worth more than 15 TB.
This is a general ownership rule: use growable containers during assembly, then freeze them at the boundary into the smallest type that expresses the stored value’s real behavior.
Second: represent boundaries with offsets, not allocations
Answer, authority, and additional records are logically separate, but they do not need three independent heap objects. They can occupy one contiguous sequence with two offsets marking where the later sections begin.
Because DNS sections contain small record counts, each boundary fits in a u16. Two boxed slices would each require a pointer and a length—16 bytes apiece on a 64-bit target. Replacing those two extra slices with two 2-byte offsets saves 28 bytes per entry and removes two allocation relationships.
The same reasoning applies to flags. Several booleans can be packed into one bit field. The direct savings may look tiny, yet layout changes are not perfectly additive. Rust must respect field alignment, and a structure’s total size is rounded to its alignment. Removing a two-byte field or collecting booleans can eliminate neighboring padding as well. The Rust Reference explains why the size of a value includes this alignment padding.
Contiguity also improves locality. A lookup that touches adjacent bytes can make better use of each CPU cache line than one that follows pointers across unrelated heap regions. At this scale, memory footprint and latency are often the same problem viewed from different levels of the hierarchy.
Third: infer the owner on the common path
Every DNS resource record has an owner name. In an ordinary answer, that owner is often exactly the name in the cache key. A query for example.com A might return two A records, both owned by example.com.
Storing the full owner inside both records duplicates information already held by the key. Big Pineapple changed the record to store an optional owner:
struct Record {
owner: Option<Box<Name>>,
class: Class,
ttl: Ttl,
kind: Rtype,
data: RecordData,
}
None means “use the queried name from the cache key.” A different name is stored only when the response requires one, as in a CNAME chain where later records belong to the canonical target.
This deliberately makes an individual record less self-contained. That is a good trade because every cache lookup already has the key. The complete context exists at the point where the record is read, so copying it into every stored child object buys convenience rather than capability.
DNS already makes a similar trade on the network. RFC 1035 name compression allows later names to refer to an earlier suffix with a two-octet pointer. Big Pineapple did not keep those compression pointers in its parsed cache representation—the lookup cost would be awkward—but it borrowed the deeper idea: repeated context can be referenced or inferred instead of expanded everywhere.
Fourth: stop sizing common records for the rare giant
A typed enum looks ideal for DNS record data:
enum RecordData {
A(Ipv4Addr),
Aaaa(Ipv6Addr),
Txt(Txt),
Naptr(Naptr),
Svcb(Svcb),
// many more variants
}
An enum value must have enough space for any of its variants, plus the discriminant and padding. In this case, NAPTR was the largest variant at 136 bytes, making the full enum 144 bytes. Yet A needs four bytes and AAAA needs sixteen, and together they represented more than 80% of the benchmark’s traffic.
The common records were therefore carrying well over a hundred bytes of unused inline space so the same slot could occasionally hold a rare large record.
The first repair boxed large variants. Small, common variants remained inline; large variants became pointers to right-sized heap allocations. This reduced the enum to 24 bytes and saved about 120 bytes for each A or AAAA record.
Boxing is a useful lever, but it introduces a second bill. Each large value needs an allocation. Allocators such as jemalloc group requests into size classes, so a requested size may be rounded upward. Pointer chasing also scatters related data and can cause extra CPU cache misses.
The intermediate enum proved the distribution-aware idea—optimize the container for common variants instead of the maximum—but it was not the final layout.
Fifth: keep structure where it helps, wire bytes where they win
One extreme would cache an entire ready-to-send DNS packet. That is compact, but responses are not identical for every client. The transaction ID changes. DNSSEC records are included only when the client sets the DNSSEC OK flag. Name compression depends on the final message layout. A full packet would need duplicate variants or parsing and filtering on every hit.
The opposite extreme is the fully parsed object graph. It makes manipulation easy but spends memory and serialization work on fields that are usually copied back to the wire.
The final design sits between them. Metadata remains structured. Record payloads are encoded into one Box<[u8]>; each record is stored as a two-byte length followed by its wire-format bytes.
Most records—including A, AAAA, TXT, and DNSSEC types—can then be copied directly into the outgoing response. Records containing domain names, such as CNAME, NS, MX, and SOA, are parsed when needed so the resolver can apply name compression in the new message.
Random indexing becomes less convenient because the length-prefixed buffer must be scanned. Big Pineapple sometimes rotates address records for round-robin behavior, but entries contain few records, so sequential iteration is cheap. In exchange, all payloads occupy one allocation and arrive together in cache lines.
Insertion uses a reusable scratch buffer. Repeated writes grow it to a practical high-water mark, so building the next entry usually needs no temporary reallocation. Once serialization is complete, the exact byte count is copied into a boxed slice. That creates one right-sized long-lived allocation and avoids hoping that shrinking a vector will return its unused tail to the allocator. This last change alone raised insert throughput by 13% in the benchmark.
Smaller became faster because less work moved through the machine
The completed set of changes reduced the benchmarked net footprint per entry by 56%, from 953 to 420 bytes. Allocations per entry fell from about 1.1 KB to 461 bytes, a 58% reduction.
Performance moved in the same direction:
| Metric | Before | After | Change |
|---|---|---|---|
| Cache insertion | 625,000 entries/s | 893,000 entries/s | +43% |
| Cache lookup latency | 828 ns | 670 ns | -19% |
| Per-entry footprint | 953 bytes | 420 bytes | -56% |
| Per-entry allocations | 1.1 KB | 461 bytes | -58% |
This is not mysterious once the hierarchy is visible. Fewer allocations mean less allocator bookkeeping. Smaller values move fewer bytes during insertion. Contiguous records need fewer pointer dereferences and use CPU cache lines more fully. Direct copying avoids reconstructing common records field by field.
During the production rollout, instance memory fell in steps as new releases arrived and caches refilled. At the 99th percentile, steady-state resident memory dropped from 9.3 GB to 5.3 GB; at the 90th percentile it fell from 6.5 GB to 3.8 GB. Across the fleet, the working-set reduction settled near 100 TB.
The freed memory is not merely an accounting win. Cloudflare plans to spend it on more cache entries. A larger cache raises hit rate, which reduces upstream queries and can improve response time and resilience. Efficient representation creates a compounding return: the same hardware retains more answers and performs less external work.
A practical playbook for high-scale data structures
The techniques here are specific to DNS, but the design process applies to search indexes, telemetry buffers, object stores, compilers, databases, and any service with millions of long-lived objects.
Model the real distribution
Do not optimize every enum variant equally. Measure which cases dominate count, bytes, and access frequency. A rare large value should not dictate the inline size of billions of common small values.
Separate construction from storage
Builders benefit from mutation and spare capacity. Stored objects benefit from exact sizing and immutability. Make that phase transition explicit with frozen strings, boxed slices, arenas, pages, or another compact representation.
Charge pointers the full price
A pointer costs more than its width. It implies an allocation, allocator metadata, possible size-class slack, another object lifetime, and a potential cache miss. Replacing two pointers with a four-byte offset can save far more than twelve bytes in practice.
Treat duplicated context as a schema question
If every child repeats a value available from its parent or key, ask whether the child truly needs to stand alone. Optional overrides are often enough. The tradeoff must be explicit: inference saves space but increases coupling to the surrounding context.
Choose an internal wire format deliberately
Parsed structures and serialized bytes are endpoints, not commandments. A hybrid can keep searchable metadata typed while storing opaque payloads compactly. The right boundary is the one that minimizes total work across insertion, residency, and lookup.
Validate improvements at several layers
Track type sizes, heap allocations, throughput, tail latency, resident memory, and downstream behavior such as cache hit rate. An optimization that wins one microbenchmark can lose after allocator rounding or production traffic shifts.
The real optimization was removing capabilities
Every major saving came from taking something away:
- growable capacity after the value became immutable;
- independent allocations for sections that can share storage;
- owner names already present in the key;
- inline room for rare enum variants;
- parsed fields that the hot path only serializes again.
That is the durable lesson. Compact systems are not created by clever packing alone. They emerge when the representation offers exactly the capabilities its lifetime requires—and no more.
At ordinary scale, these choices look like implementation details. At 250 billion entries, they become 100 terabytes, 268,000 additional inserts per second, and 158 fewer nanoseconds on a cache hit. Scale does not invent new costs. It reveals the ones that were already hiding in every object.
Top comments (0)