DEV Community: Galadd

Cutting Beacon State Storage by ~97%: Validator Sectors with Structural Sharing

Galadd — Fri, 02 Oct 2026 10:54:18 +0000

A design prototype. I’d like your feedback before I build the reconstruction measurements.

TL;DR

Storing full beacon states at regular intervals costs roughly 3 TB. Diffing against a base reference (eth-state-diff) brings that down to about 400 GB. But ~85% of a beacon state is the validators list, which barely changes. By treating validators separately as sectors of structurally shared chunks, plus layered checkpoints of diffs, the estimate drops to ~70 GB, a reduction of roughly 97% from the naive approach.

All numbers are maximum approximations. This post is about the design prototype, not benchmarks.

Background

I’ve been looking into designs that reduce redundant on-disk data and speed up state reconstruction and historical state lookups. That work led to eth-state-diff, which computes the difference between two beacon states and can be reapplied to reconstruct one from the other.

It has a clear limitation: it always needs a base reference, and the farther the base is from the target, the larger the diff. A pure diff scheme is not a complete solution. It needs periodic full-state anchors (I’m thinking every 256 epochs).

Full anchors are still wasteful, because most of each one is almost identical to the last.

The observation

About 85% of the beacon state is the validators list, and that list is append-mostly:

New validators are appended, and the appends are clustered.
In-place mutations are extremely rare—as few as 0.015% of validators.

If the list were strictly append-only, a checkpoint would only need to store a length. It isn’t, but it’s close enough that storing a full copy every time is almost pure redundancy.

So the design question is: how do you exploit near-100% redundancy?

Validators need their own representation.

The design

Validators are stored in two cooperating layers: sectors (structural sharing) and checkpoints (diffs).

Layer 1: Sectors (structural sharing)

A sector spans 8 checkpoints of 256 epochs each → 2,048 epochs.

At every sector boundary the full validators list is materialised, but not as a monolithic blob. It is split into chunks of 64 validators (64 × 121 bytes ≈ 7.7 KB each). Each chunk is stored once and addressed by name (think content-addressed). A sector’s validator list is simply a list of chunk references:

The reference list is small (~250 KB), so a plain list is sufficient. A persistent tree would be over-engineering here.
When validators change, only the affected chunks are written. Unchanged chunks are shared with earlier sectors.

Cost at a maximum of 150 changes per sector:

150 changes × 121 B × 64 (whole chunk copied) ≈ 1.2 MB

Appends are clustered, so they create few chunks (max ~25 KB). Across the full history the changed chunks total roughly 300 MB and the chunk-reference lists roughly 57 MB.

The tradeoff is intentional: a change to a single 8-byte field still copies its whole chunk, so this is not space-optimal in the small. But it never requires a deep copy of the validators list, and at this scale the overhead is tiny.

Layer 2: Checkpoints (layered diffs)

Inside a sector, checkpoints record diffs of the validators list:

Every 256 epochs a “major” checkpoint stores a diff against the sector base.
Every 32 epochs a “minor” checkpoint stores a diff against the nearest preceding major checkpoint.

Concrete layout (example at epoch 8768)

Sector 0 (epoch 0): full validators (chunks)
  ├─ minor 32:   diff(32, sector 0)
  ├─ minor 64:   diff(64, sector 0)
  ├─ … 
  └─ major 256:  diff(256, sector 0)
       ├─ minor 288: diff(288, major 256)
       ├─ … 
       └─ major 512: diff(512, sector 0)
            …

Sector 8192 (epoch 8192): full validators (new ChunksRef list)
  ├─ minor 8224: diff(8224, sector 8192)
  ├─ …
  ├─ major 8448: diff(8448, sector 8192)
  ├─ …
  ├─ major 8704: diff(8704, sector 8192)
  └─ minor 8768: diff(8768, major 8704)

Reconstruction at epoch E

Locate the sector base: floor(E / 2048) × 2048.
Locate the nearest major checkpoint ≤ E: floor(E / 256) × 256.
Locate the exact minor checkpoint (or the target epoch itself).
Start from the sector’s full validators (resolved via its ChunksRef list).
Apply the major-checkpoint diff.
Apply the minor-checkpoint diff.

Example for epoch 8768:

validators@8768 = sector@8192 + diff@8704 + diff@8768

Estimated storage

Component	Size
Naive: full state every 32 epochs	~3 TB
Diffs every 32 epochs, base every 256	~400 GB
This design
Sector bases (full state without validators)	~60 GB
Changed validator chunks	~300 MB
Chunk references	~57 MB
Validator patches / minor diffs	~550 MB
Diffs (remaining state + major checkpoints)	~10 GB
Total	~70 GB

≈ 97.7 % reduction versus the naive approach and ≈ 82 % reduction versus plain eth-state-diff.

What I still need to measure

Storage is only half the story. I will look into concrete reconstruction-cost numbers:

Latency vs distance from the sector base
Bytes read per reconstruction
Number of diffs applied and chunks fetched

Feedback wanted

Is a 64-validator chunk the right granularity? Smaller chunks copy less on mutation but grow the reference list.
Has anyone seen similar layered structural-sharing + diff approaches in other clients’ state stores?

All figures above are max approximates. The focus is the design prototype.

Optimizing Persistent Storage with State Deltas

Galadd — Thu, 22 Jan 2026 11:20:28 +0000

Optimizing persistent storage through deltas might sound like a general solution—but this article is intentionally specific. This approach worked because I had full control over the data model and could make assumptions about how the data evolves over time.

That won’t always be the case. You might not be able to reshape your data this way, or your workload might look completely different. Still, optimization often comes down to understanding your data deeply and being willing to look at it from unconventional angles. Even if you don’t apply this exact approach, I hope it offers a useful perspective.

The idea itself is simple—almost offensively so. There’s nothing particularly novel or academically complex here. But sometimes, stepping away from heavy abstractions and focusing on the shape of your data can yield outsized wins.

So I’ll get straight to the point.

The Problem

Every 3.5 hours, a full system state was written to disk. Even after compression, each snapshot was roughly 140 MB.

This works, but it doesn’t scale. A new node syncing the network would need to download and store every historical snapshot. Over time, this balloons into terabytes of required disk space.

To put concrete numbers on it:

Before: ~10 TB of required storage
After: ~2 TB

That’s roughly an 80% reduction in disk usage.

Why Not Just Snapshot Less Often?

Although a new state exists every 12 seconds, not all states are equal. Each state contains specific information that allows reconstruction of prior states and verification of the chain’s history.

As snapshots get farther apart, reconstructing intermediate states becomes increasingly expensive. The farther a target state is from the nearest snapshot, the more work is required to replay or reconstruct it. This directly impacts sync time and verification latency.

Snapshots therefore can’t be spaced arbitrarily far apart without shifting cost elsewhere. There’s a practical lower bound on snapshot frequency if you want reasonable reconstruction times.

The Core Idea: State Deltas

The key observation was simple: consecutive full states written only 3.5 hours apart are extremely similar.

Instead of storing the entire state every time, I switched to storing:

One base state
Followed by deltas, which encode only the differences between states

When a full state is needed, the base state is loaded and the deltas are applied sequentially.

This slightly changes complexity—from pure O(x) reads to O(x + delta_application)—but in practice, the tradeoff is well worth it. Applying a delta took roughly ~200 ms, which was acceptable for this workload.

The result is dramatically less data written to disk and significantly less data read during sync.

Avoiding Disk Reads: Caching the Base State

Initially, delta computation was slower than expected. The reason was obvious in hindsight: computing a delta requires access to both the base state and the new state.

My first implementation read the base state from disk every time. That was a mistake.

If something is accessed repeatedly and fits comfortably in memory, it should be cached.

Each base state is reused for roughly 7 subsequent deltas, making caching an easy win. I introduced a simple in-memory cache using an Arc<RwLock<Option<_>>>:

base_state_cache: Arc::new(RwLock::new(None)),

The logic is straightforward:

If the base state is cached, reuse it
Otherwise, load it once from disk and cache it
Subsequent accesses should not hit disk again

fn load_base_state(&self, root: H256) -> Result<Option<Arc<BeaconState<P>>>> {
    if let Some(cached) = self.load_cached_base_state(root) {
        return Ok(Some(cached));
    }

    let Some(state) = self.state_by_block_root(root)? else {
        return Ok(None);
    };

    self.set_cached_base_state(root, &state);
    Ok(Some(state))
}

With this change, both read and write performance improved significantly.

Overall, the new approach reduced time spent on state storage by roughly 70%, while also drastically reducing disk usage.

A Closer Look: Optimizing Balances

One field deserved special treatment: balances.

Balances are stored as a Merkle tree containing over 2,000,000 u64 values, and roughly 1,000,000 of them change between states. At first glance, this suggests that storing half the values is unavoidable—but that assumption turns out to be wrong.

The key lesson here is simple: know your data.

After analyzing balance changes, four patterns emerged:

Many values don’t change
Many values change to zero
Many values change, but only by a small amount
Very few values change by a large amount

Once this distribution is clear, storing every changed value as a full u64 becomes wasteful.

Encoding Changes with 2 Bits

Each balance change can be classified into one of four categories, meaning it can be represented using 2 bits.

I used a compact tag vector where each entry encodes the change type:

00 → no change
10 → set to zero
11 → apply a small diff
01 → set to a full target value

Four balance entries fit into a single byte:

pub struct BitTagVec {
    data: Vec<u8>, // 4 entries per byte
    len: usize,
}

impl BitTagVec {
    pub fn new(len: usize) -> Self {
        let bytes = len.div_ceil(4);
        Self {
            data: vec![0; bytes],
            len,
        }
    }

    #[inline]
    pub fn set(&mut self, idx: usize, tag: u8) {
        let byte = idx / 4;
        let shift = (idx % 4) * 2;

        self.data[byte] |= (tag & 0b11) << shift;
    }

    #[inline]
    pub fn get(&self, idx: usize) -> u8 {
        let byte = idx / 4;
        let shift = (idx % 4) * 2;
        (self.data[byte] >> shift) & 0b11
    }
}

For ~2,000,000 balances, this tag vector alone takes roughly 500 KB.

The associated data is stored separately:

pub struct BalanceDiffs {
    tags: BitTagVec,
    small_diffs: Vec<i32>,
    target_values: Vec<Gwei>,
    new_balances: Vec<Gwei>,
}

small_diffs store small balance changes
target_values store large or special-case updates
new_balances represent newly added balances Applying the delta is a linear pass over the tag vector, applying the corresponding operation. The logic is predictable, cache-friendly, and fast.

Conclusion

This work started as a practical optimization and became a reminder of something fundamental: effective storage optimization often has less to do with clever algorithms and more to do with understanding your data.

By switching from full-state storage to delta-based storage—and by tailoring the delta format to how the data actually changes—I was able to significantly reduce both disk usage and runtime costs.

If you have ideas on how this approach could be improved, or alternative ways you would model these deltas, I’d love to hear your thoughts. The full PR is available here:

https://github.com/grandinetech/grandine/pull/523/files

Writing this was also an attempt to write more and engage more openly. Constructive critique—technical or otherwise—is very welcome.