DEV Community

Oleksandr Kuryzhev
Oleksandr Kuryzhev

Posted on Originally published at kuryzhev.cloud

PostgreSQL 18 Asynchronous I/O: Settings Worth Testing

Originally published on kuryzhev.cloud


A reporting replica gets upgraded to the new major version. Someone reads a headline about "async I/O making scans faster," and a ticket appears: "turn on whatever gives us the speedup." Nobody can say which knob that is, or whether the workload even does the kind of reads that benefit. That is the normal state of PostgreSQL 18 asynchronous I/O adoption: a real feature with vague expectations.

The PostgreSQL 18 release notes introduce an asynchronous I/O subsystem. PostgreSQL 18 is the current stable line (18.6 at the time of writing). The feature matters now because it is active after the upgrade with its default settings, so you are already running it whether you tuned it or not. The useful question is which few settings deserve a controlled test on your data.

Short answer: keep the default io_method = worker unless two conditions hold:

  • You are on Linux with a kernel and PostgreSQL build that support io_uring.
  • Your queries are read-heavy with cold or large working sets.

If both hold, A/B test worker against io_uring, then io_workers, then effective_io_concurrency, one change at a time. If your hot data fits in shared_buffers and the OS cache, expect little or nothing.

What PostgreSQL 18 asynchronous I/O actually does

Before 18, a backend asked the kernel for a block and waited. Read-ahead and posix_fadvise hints helped in places, but reads were still synchronous from the backend's point of view. With the new subsystem, a backend can submit several read requests and continue working while they are in flight.

Two things decide how that happens. The first is the io_method setting, which picks the mechanism:

  • worker (the default) hands reads to dedicated I/O worker processes.
  • io_uring uses the Linux kernel's ring interface directly from the backend.
  • sync keeps the old blocking behavior.

The second is the set of code paths that use it. As documented for 18, the main beneficiaries are sequential scans, bitmap heap scans and vacuum. Writes and WAL are not part of this change, so write-heavy OLTP gains little. Verify the exact scope in the release notes.

Related defaults also moved: effective_io_concurrency and maintenance_io_concurrency were raised in 18 (check with SHOW on your build). In practice, an upgraded server behaves differently before you touch anything. That is why a before/after comparison on identical queries matters.

How people use it wrong

The most common error is copying a tuning snippet from a blog post and restarting production with several changed values. Then nobody knows which value helped, hurt, or did nothing. One plausible failure mode is a latency regression caused by oversized worker counts competing with query backends for CPU.

The second error is benchmarking on a warm cache. If the table is already in shared_buffers or the page cache, there is almost no read I/O to overlap, and every method looks identical. The reverse also happens: someone drops caches once, sees a gain, and treats it as steady-state behavior for a workload that is mostly cached.

Third, people assume io_uring is simply "the fast one." It depends on Linux, a PostgreSQL build with liburing support (--with-liburing), and a kernel that allows it. Watch out for container runtimes: default seccomp profiles and some hardened hosts restrict the io_uring syscalls. A setting that works on a VM can therefore fail inside a container. Check your runtime's documentation rather than assuming.

Finally, people expect gains on index-driven point lookups, write-heavy ingestion or WAL-bound workloads. The subsystem in 18 does not target those paths, so no setting will fix them.

The correct approach

Treat it as an experiment with one variable at a time. First, record your baseline: the current values and a representative set of slow read queries. Run the first block below on the target version to capture the starting point.

-- Baseline: current AIO-related settings (values shown are whatever your build uses)
SELECT name, setting, unit, context
FROM pg_settings
WHERE name IN ('io_method','io_workers','io_max_concurrency',
               'io_combine_limit','effective_io_concurrency',
               'maintenance_io_concurrency','track_io_timing');

-- Confirm I/O worker processes exist when io_method = worker
SELECT pid, backend_type
FROM pg_stat_activity
WHERE backend_type = 'io worker';

Next, enable timing so the numbers mean something, and write a candidate config. io_method and io_max_concurrency need a restart, while io_workers can be changed with a reload (confirm with the context column above). Put changes in a drop-in file so they are easy to revert.

# /etc/postgresql/18/main/conf.d/aio-test.conf  (path varies by distro)
track_io_timing = on          # adds I/O time to EXPLAIN and pg_stat_io
io_method = io_uring          # candidate; baseline run used the default 'worker'
# io_workers = 6              # only relevant for io_method = worker; test separately
# effective_io_concurrency = 32   # change in a later, separate round

Run the same queries with EXPLAIN (ANALYZE, BUFFERS, SETTINGS) against a cold cache on a non-production host. For a cold start, restart PostgreSQL and clear the OS page cache (sync; echo 3 > /proc/sys/vm/drop_caches as root, test machine only). Repeat each case several times and compare medians, not the best run.

To verify the change took effect, SHOW io_method; must return the new value after restart, and the server log must show a clean start. If io_uring cannot be used, the server refuses to start rather than silently falling back. This happens either because the build lacks liburing, which makes the value invalid, or because the kernel or seccomp blocks the syscalls. You find out immediately, which is another reason to test this on a non-production host first.

Advanced patterns

Once the method is chosen, tune concurrency against your storage. Low-latency local NVMe often reaches its useful limit with less concurrency than network storage, but measure rather than assume. On network-attached volumes such as cloud block storage, latency dominates, and more outstanding requests usually pay off up to the volume's IOPS or throughput cap. That cap is the real ceiling, so check it before concluding that PostgreSQL is the bottleneck.

For worker mode, size io_workers against CPU headroom. Each worker is a process, and on a busy host the workers compete with query backends. Increase the count in small steps and watch whether scan time improves or the CPU run queue grows instead.

Maintenance work is a separate test. Vacuum reads are governed by maintenance_io_concurrency, so vacuuming a bloated table on a staging copy is a clean way to measure it without touching query latency. Pair the test with whatever bloat monitoring you already run; more PostgreSQL operations material is on kuryzhev.cloud.

For live observation, sample the in-flight view while a test query runs. pg_aios requires superuser or membership in pg_read_all_stats.

-- Run repeatedly (e.g. with \watch 1) during a scan; empty output means nothing is in flight
SELECT * FROM pg_aios;

-- Read volume and time per backend type, since stats reset
SELECT backend_type, object, context, reads, read_bytes, read_time
FROM pg_stat_io
WHERE reads > 0
ORDER BY read_time DESC NULLS LAST;

If you export pg_stat_io to Prometheus and graph it in Grafana, record the delta of read_time and reads per test window. Average time per read is a better signal than raw totals. Column sets differ between major versions (18 added byte columns such as read_bytes), so verify exporter queries against the PostgreSQL 18 documentation.

Performance notes

Expect uneven results. Large sequential scans over data not in cache are the likeliest winners. Queries that are CPU-bound after the read, such as heavy aggregation or expression evaluation, stay limited by the CPU; faster I/O only exposes that. Highly selective index lookups mostly will not move.

Gotcha: changing io_combine_limit alters the size of individual read requests, which interacts with storage block sizes and OS read-ahead. Leave it at the default unless a test shows a specific reason. Never edit developer-only options such as debug_io_direct on a real system.

The costs are modest but real:

  • io_uring adds a kernel and build dependency and may complicate container hardening.
  • Extra workers consume CPU and memory.
  • Higher concurrency can let one noisy analytical query take a larger share of shared storage bandwidth, which hurts neighbors on a shared volume.

If your replica serves mixed traffic, test with the real mix running.

When not to bother: small databases that fit in RAM, write-dominated systems, or any server where the upgrade is already stable and the queries meet their SLOs. Defaults in 18 are chosen to be safe, and "no change" is a valid outcome of the experiment.

A simple decision rule to keep:

  1. If cold-cache EXPLAIN shows significant I/O time, change one setting.
  2. Rerun the identical query set.
  3. Keep the change only if the median improves without raising CPU or p99 latency elsewhere.
  4. Write the result next to the config file so the next engineer does not repeat the test.

Related

Top comments (0)