DEV Community

Cover image for The I/O Scheduler That Stops Background Work From Killing Latency
Swaraj Puppalwar
Swaraj Puppalwar

Posted on

The I/O Scheduler That Stops Background Work From Killing Latency

#liorandb #database #nosql #indian

A database performs many kinds of disk activity:

WAL fsync
Primary data flushes
Document-store writes
Index flushes
Compaction
Text compaction
Checkpoint sync
Metadata writes
Cleanup
Enter fullscreen mode Exit fullscreen mode

If all of them hit the disk without coordination, foreground latency becomes chaotic.

LioranDB contains a global I/O scheduler to manage this contention.

I/O classes

Operations are assigned to classes such as:

WalFsync
Metadata
PrimaryFlush
DocStore
IndexFlush
IndexMaintenance
Compaction
TextCompaction
CheckpointSync
Enter fullscreen mode Exit fullscreen mode

Each class has a relative priority.

WAL fsync receives the highest weight because committed transactions may be waiting for it.

Background compaction receives less priority because it can usually pause briefly without breaking correctness.

Token-based budgeting

The scheduler maintains a byte budget per second.

Operations consume tokens as they perform I/O.

The budget supports short bursts but prevents background workers from continuously saturating the device.

The default budget can be configured through environment variables.

Foreground awareness

The scheduler records recent foreground activity.

When user-facing writes are active, optional background operations can yield.

Examples include:

  • Secondary compaction
  • Text compaction
  • Cleanup

This reduces the chance that a giant merge causes sudden WAL latency spikes.

Sync coordination

Physical syncs are especially expensive.

LioranDB tracks sync ownership for:

WAL
Checkpoint
Document store
Metadata
Compaction
Enter fullscreen mode Exit fullscreen mode

It also records whether WAL fsync was blocked by another subsystem.

That gives engineers evidence instead of vibes when investigating latency.

Circuit-breaker behaviour

The code also models background-I/O health states such as:

Healthy
Elevated
Critical
Recovery
Enter fullscreen mode Exit fullscreen mode

When WAL latency becomes dangerous, background work can use smaller chunks or pause more aggressively.

Databases are not CPU programs that occasionally touch disk.

They are I/O orchestration systems wearing an API as a hat.

Once several durability and maintenance pipelines exist, a scheduler becomes essential to keep them from wrestling in the storage aisle.

Built by Swaraj Puppalwar under Lioran Group.

More:

Top comments (0)