DEV Community

Parsa Gheiratian
Parsa Gheiratian

Posted on

Detecting Ceph LARGE_OMAP_OBJECTS Before Deep Scrub Finds It

#database #devops #infrastructure #sre

LARGE_OMAP_OBJECTS has an annoying property: it tells you about a problem that started days ago. An RGW bucket index shard, a CephFS metadata object, or an application object built on librados grows past the threshold, and nothing in ceph health changes until a deep scrub happens to walk that PG. With the default deep-scrub interval of seven days, the warning can trail the actual growth by most of a week.

I wanted to know exactly what Ceph knows about OMAP size before deep scrub, what it only learns during deep scrub, and which of the commonly suggested "check it earlier" techniques actually work and at what cost. This article is the result of reading the scrub code and then reproducing the condition on a Tentacle lab cluster.

The behavior I wanted to verify

  • Is the health warning purely a product of deep scrub, or does a normal scrub contribute?
  • What exactly is compared against the two thresholds: key count, value bytes, or both?
  • Does the OSD keep a live per-object OMAP size anywhere that I can query?
  • What do ceph pg dump, ceph df detail, ceph osd df, the OSD admin socket, and the Prometheus module expose, and which of those numbers are live versus scrub-time snapshots?
  • What does rados listomapkeys | wc -l actually cost on the OSD?
  • Is there any supported way to ask "top N objects by OMAP keys" without reading every object?
  • What is different for RGW bucket indexes?

What actually triggers LARGE_OMAP_OBJECTS

Two OSD options define "large". On the lab cluster both are at their defaults:

$ ceph config get osd osd_deep_scrub_large_omap_object_key_threshold
200000
$ ceph config get osd osd_deep_scrub_large_omap_object_value_sum_threshold
1073741824
Enter fullscreen mode Exit fullscreen mode

ceph config show-with-defaults osd.0 reports both with source default, so nobody has tuned them here. The key threshold is also read by the MDS: the open-file table code caps the number of entries it puts in one object at exactly this value and spreads the rest across up to 1024 objects. That is a useful hint about how the Ceph developers themselves avoid the warning: shard at the application layer.

The detection path, traced through the Tentacle source, is entirely inside deep scrub:

deep scrub of PG
    │
    ▼
for each object in the scrub chunk (5..15 objects per chunk by default)
    │
    ├── stat + xattrs                       (shallow and deep)
    │
    └── OMAP walk, 1024 keys per step   (deep only, osd_deep_scrub_keys)
          │  header → crc
          │  every key and value → crc, key count, value bytes
          ▼
        keys  > osd_deep_scrub_large_omap_object_key_threshold
        OR
        bytes > osd_deep_scrub_large_omap_object_value_sum_threshold
          │
          ▼
        object flagged in the scrub map
          │
          ▼
after each chunk: ScrubBackend::omap_checks()
    ├── sums keys/bytes into per-scrub counters
    ├── increments large_omap_objects
    └── clog WARN "Large omap object found. Object: ... PG: ...
                   Key count: ... Size (bytes): ..."
          │
          ▼
scrub_finish(), only if the scrub was deep:
    stats.sum.num_large_omap_objects = counter
    stats.sum.num_omap_keys          = sum
    stats.sum.num_omap_bytes         = sum
          │
          ▼
OSD publishes PG stats → mgr/mon PGMap
          │
          ▼
mon health check: if pg_sum.num_large_omap_objects > 0
    → LARGE_OMAP_OBJECTS (HEALTH_WARN), "N large objects found in pool 'X'"
Enter fullscreen mode Exit fullscreen mode

Details worth knowing because they affect how you interpret the numbers:

  • A shallow scrub never touches OMAP. The object scan calls stat and getattrs and only descends into the OMAP walk when the scrub is deep. A shallow scrub also leaves num_omap_keys, num_omap_bytes, and num_large_omap_objects untouched in the PG stats, so it neither raises nor clears the warning.
  • "Size (bytes)" is the sum of value lengths only. Key bytes are not counted toward the value-sum threshold. An object with 200000 keys and empty values trips the key threshold with a reported size of 0.
  • The comparison is strictly greater-than. Exactly 200000 keys is not "large".
  • One warning per scrub chunk. omap_checks() stops at the first flagged object in each chunk, so if two large objects sit in the same chunk of 5 to 15 objects, only one is logged and counted. The health message count can be lower than the truth.
  • PG OMAP stats are a snapshot. Nothing decrements or increments them on the write path. They only change at the next deep scrub of that PG.

Lab environment

  • Ceph 20.2.3 Tentacle (daemons), 20.2.4 client tools, deployed with cephadm.
  • 3 hosts, 3 mons, 2 mgrs, 6 BlueStore OSDs, all replicated pools with size 3.
  • No RGW deployed. radosgw-admin from the same release was used only to verify command availability.
  • Health was HEALTH_OK before the test and again after cleanup.
  • Test pool test-omap: 1 PG, pg_autoscale_mode off, application rados. Single test object test-object.
  • All scrub and threshold settings at defaults. No configuration was changed for the reproduction.

Reproducing the condition

The default key threshold is 200000, which is small enough to hit directly. I wrote 250000 keys with 32-byte values using librados from Python, batching 5000 keys per write op:

import rados
N = 250000
B = 5000
c = rados.Rados(conffile="/etc/ceph/ceph.conf")
c.connect()
io = c.open_ioctx("test-omap")
for start in range(0, N, B):
    with rados.WriteOpCtx() as op:
        keys = tuple("key-%08d" % i for i in range(start, min(start + B, N)))
        vals = tuple(b"v" * 32 for _ in keys)
        io.set_omap(op, keys, vals)
        io.operate_write_op(op, "test-object")
io.close()
c.shutdown()
Enter fullscreen mode Exit fullscreen mode

The whole write took 0.7 seconds. The object has no data payload at all:

$ rados -p test-omap stat test-object
test-omap/test-object mtime 0.000000, size 0
Enter fullscreen mode Exit fullscreen mode

rados stat says size 0. OMAP is invisible to it.

Before any scrub

Twenty seconds after the write, the PG stats already know that the object has OMAP, but not how much:

$ ceph pg 7.0 query | jq '.info.stats.stat_sum
    | {num_objects, num_objects_omap,
       num_omap_bytes, num_omap_keys, num_large_omap_objects}'
{
  "num_objects": 1,
  "num_objects_omap": 1,
  "num_omap_bytes": 0,
  "num_omap_keys": 0,
  "num_large_omap_objects": 0
}
Enter fullscreen mode Exit fullscreen mode

num_objects_omap is maintained on the write path, so it is live. The other three are zero because no deep scrub has run.

The pool-level view, on the other hand, is live and already large:

$ ceph df detail | grep -E 'POOL|test-omap'
POOL       ID  PGS   STORED   (DATA)   (OMAP)  OBJECTS     USED   (DATA)   (OMAP)  %USED  MAX AVAIL ...
test-omap   7    1   16 MiB      0 B   16 MiB        1   48 MiB      0 B   48 MiB   0.02     95 GiB ...
Enter fullscreen mode Exit fullscreen mode

Health:

$ ceph health detail
HEALTH_OK
Enter fullscreen mode Exit fullscreen mode

Shallow scrub 

$ ceph pg scrub 7.0
instructing pg 7.0 on osd.5 to scrub
Enter fullscreen mode Exit fullscreen mode

It completed in about two seconds. last_scrub_stamp advanced, last_deep_scrub_stamp did not, all OMAP counters stayed at zero, and health stayed HEALTH_OK. This matches the source: no OMAP walk in a shallow scrub.

Deep scrub 

$ ceph pg deep-scrub 7.0
instructing pg 7.0 on osd.5 to deep-scrub
Enter fullscreen mode Exit fullscreen mode

Also about two seconds for one object with 250000 small keys. Now the stats are populated:

$ ceph pg 7.0 query | jq '.info.stats.stat_sum
    | {num_objects, num_objects_omap,
       num_omap_bytes, num_omap_keys, num_large_omap_objects}'
{
  "num_objects": 1,
  "num_objects_omap": 1,
  "num_omap_bytes": 8000000,
  "num_omap_keys": 250000,
  "num_large_omap_objects": 1
}
Enter fullscreen mode Exit fullscreen mode

8000000 bytes is exactly 250000 keys times 32-byte values, confirming that keys are not counted toward the byte sum. The cluster log has the object name, and the monitor raised the health check five seconds after the OSD logged it:

$ ceph log last 300 warn cluster | grep -i 'large omap'
2026-09-05T14:16:07.644528+0000 osd.5 (osd.5) 55 : cluster [WRN] Large omap object found. Object: 7:5756f1fd:::test-object:head PG: 7.bf8f6aea (7.0) Key count: 250000 Size (bytes): 8000000
2026-09-05T14:16:12.723891+0000 mon.ceph01 (mon.0) 39043 : cluster [WRN] Health check failed: 1 large omap objects (LARGE_OMAP_OBJECTS)
Enter fullscreen mode Exit fullscreen mode 
$ ceph health detail
HEALTH_WARN 1 large omap objects
[WRN] LARGE_OMAP_OBJECTS: 1 large omap objects
    1 large objects found in pool 'test-omap'
    Search the cluster log for 'Large omap object found' for more details.
Enter fullscreen mode Exit fullscreen mode

ceph pg ls-by-pool now shows the numbers too, with the asterisk footnote that Ceph itself prints:

$ ceph pg ls-by-pool test-omap
PG   OBJECTS  ...  BYTES  OMAP_BYTES*  OMAP_KEYS*  ...  SCRUB_STAMP                      DEEP_SCRUB_STAMP
7.0        1  ...      0      8000000      250000  ...  2026-09-05T14:16:07.651563+0000  2026-09-05T14:16:07.651563+0000

* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization.
Enter fullscreen mode Exit fullscreen mode

The snapshot goes stale immediately

I then added 50000 more keys and waited fifteen seconds:

$ rados -p test-omap listomapkeys test-object | wc -l
300000
$ ceph pg ls-by-pool test-omap | awk 'NR==2 {print "OMAP_KEYS*=" $8}'
OMAP_KEYS*=250000
$ ceph df detail --format json \
    | jq '.pools[] | select(.name=="test-omap") | .stats.stored_omap'
20410766
Enter fullscreen mode Exit fullscreen mode

The PG stat is frozen at the last deep scrub. The pool-level OMAP figure moved. This is the core asymmetry of the whole problem.

Clearing also needs a deep scrub

After rados -p test-omap rm test-object, the warning stayed for as long as I waited. Another ceph pg deep-scrub 7.0 reset all counters to zero and the health check cleared within a few seconds. Deleting a large OMAP object does not clear the warning by itself.

What Ceph knows before deep scrub

This is the inventory I ended up with, split by when the information is produced.

                live, no scrub needed
                ─────────────────────
pool   ├── STORED/USED (OMAP) bytes     ceph df detail       RocksDB estimate
       ├── num_objects_omap             ceph pg dump pools   write-path count
       └── store_stats.omap_allocated   ceph pg dump pools   same as df detail

PG     └── num_objects_omap             ceph pg dump pgs / pg query

OSD    ├── OMAP column                  ceph osd df          estimate, all pools
       ├── omap_allocated per pool      ceph tell osd.N dump_pool_statfs <id>
       └── bluestore omap_* counters    ceph tell osd.N perf dump   aggregates

object └── has-omap flag only           (drives num_objects_omap)

                only produced by deep scrub
                ───────────────────────────
PG     ├── num_omap_keys                ceph pg dump / pg ls   snapshot
       ├── num_omap_bytes                                      snapshot
       └── num_large_omap_objects                              health check

object └── key count and value bytes    cluster log line only

                not stored anywhere
                ───────────────────
object ── per-object OMAP key count or byte size, outside a full key walk
Enter fullscreen mode Exit fullscreen mode

Some specifics on the live sources, because they are the only early signals Ceph gives you for free:

Pool-level OMAP bytes. With per-pool OMAP enabled in BlueStore, which has been the default for newly created OSDs for several releases, each OSD stores every pool's OMAP under a pool-prefixed RocksDB key range. ceph df detail asks each OSD for estimate_prefix_size on that range, which is a RocksDB GetApproximateSizes call over SST files plus memtables. It is an estimate of on-disk footprint, not a key count, and it includes tombstoned data until compaction runs. In the lab, deleting the test pool left the OSD-level OMAP column elevated for a while afterwards for exactly that reason. It is cheap, it is continuous, and it is the only OMAP size that Ceph updates without a scrub.

$ ceph df detail --format json | jq -r '.pools[]
    | [.name, .stats.stored_omap, .stats.omap_bytes_used] | @tsv'
.mgr      1215    3647
images    6823    20471
volumes   20659   61979
vms       0       0
backups   0       0
Enter fullscreen mode Exit fullscreen mode

Which PGs and pools have OMAP objects at all. num_objects_omap is incremented and decremented by the OSD as objects gain or lose OMAP. On this cluster it immediately reveals the RBD metadata objects in the image pools. The num_omap_keys column next to them is whatever the last deep scrub saw, which for these empty RBD pools is legitimately zero:

$ ceph pg dump pgs --format json | jq -r '.pg_stats[]
    | select(.stat_sum.num_objects_omap > 0)
    | [.pgid, .stat_sum.num_objects_omap, .stat_sum.num_omap_keys,
       .last_deep_scrub_stamp] | @tsv'
3.1c    1    0    2026-09-05T09:12:57.106912+0000
2.1c    1    0    2026-09-04T14:08:45.216107+0000
3.3     1    0    2026-09-04T14:08:54.384848+0000
2.3     1    0    2026-09-04T14:08:45.216107+0000
Enter fullscreen mode Exit fullscreen mode

Per-OSD, per-pool. ceph tell osd.N dump_pool_statfs <poolid> returns the same omap_allocated estimate for one OSD's share of one pool. In the lab the primary for the test PG reported 12594436 bytes for 250000 keys, before any scrub.

What is not there. I checked the obvious places for a per-object figure:

  • ceph tell osd.N bluestore onode metadata <ghobject> prints the onode: nid, size, shards, blobs, xattrs. No OMAP key count, no OMAP byte count. BlueStore does not store one.
  • ceph tell osd.N calc_objectstore_db_histogram walks the entire RocksDB and reports key and value size histograms per prefix. It is an aggregate, and it is a full DB scan.
  • ceph tell osd.N perf dump has bluestore.omap_setkeys_records, omap_setkeys_bytes, omap_get_keys_lat, omap_next_lat and friends. All per-OSD aggregates. Useful for spotting an OSD receiving a flood of OMAP writes, useless for naming the object.
  • RADOS has no "count my OMAP keys" operation. The read side of the OMAP op set is get-keys, get-vals, get-vals-by-keys, get-header, and compare. Nothing returns a count; every read that could tell you the size does so by iterating.
  • ceph-objectstore-tool can list OMAP for an object, but only against a stopped OSD. It is a forensic tool, not a monitoring one.

So the answer to "does Ceph provide a top-N-by-OMAP API" is no. Not in the OSD, not in the mgr, not in the mon. The only per-object numbers ever produced are the ones deep scrub writes into the cluster log.

Method 1 — Targeted OMAP inspection

If you already suspect a specific object, counting its keys is straightforward:

rados -p <pool> listomapkeys <object> | wc -l
Enter fullscreen mode Exit fullscreen mode

On the 250000-key test object this returned in 0.32 seconds. But look at what the primary OSD did during that call. I snapshotted its perf counters before and after:

osd.op_r                 985  →  1230     (+245 read ops)
osd.op_r_out_bytes  32153954  →  36155179 (+4.0 MB returned)
bluestore.omap_next_lat.avgcount  1408 → 1653  (+245 iterator batches)
Enter fullscreen mode Exit fullscreen mode

245 ops for 250000 keys is the batching: the rados tool asks for 1024 keys per request, and the OSD caps a single OMAP read at osd_max_omap_entries_per_request, also 1024 by default. Each of those requests opens a RocksDB iterator positioned after the last key returned and walks forward. The OSD reads every key. It also reads the values from the same SST blocks because that is how RocksDB stores them, even though only keys go on the wire.

wc -l therefore changes nothing about server-side cost. It only stops 250000 lines from hitting your terminal. The distinction matters:

small output on the client    ≠    low OSD / RocksDB read cost
Enter fullscreen mode Exit fullscreen mode

The cost is linear in key count and, for value-heavy objects, in bytes. A bucket index shard with 200000 entries of a few hundred bytes each is tens of megabytes of RocksDB reads on one OSD, on the primary, in the client I/O path with no scrub throttling. That is acceptable for a handful of candidate objects. It is not acceptable as a loop over a pool.

Other rados subcommands for reference:

  • rados listomapvals <object> returns keys and values. Same iteration cost, plus the values on the wire. The same test object produced 50 MB of output. Do not use it for counting.
  • rados getomapheader <object> returns only the header blob. It says nothing about key count or total size.
  • rados stat <object> reports data size and mtime. OMAP does not appear, as shown above with size 0.
  • ceph tell osd.N getomap <pool> <object> exists on the admin socket and dumps the entire map. Strictly worse than listomapkeys for this purpose.

Method 2 — Narrowing down suspicious PGs/objects

There is no supported API that ranks objects by OMAP size, so "narrowing down" means combining the live aggregates with the stale scrub snapshot and with knowledge of the application. The funnel that actually works:

1. pools with OMAP         ceph df detail                    live bytes per pool
        │
        ▼
2. growth trend            ceph df detail --format json      sample every few
                           .stats.stored_omap per pool       minutes, alert on
                                                             rate or level
        │
        ▼
3. PGs holding OMAP        ceph pg dump pgs --format json    num_objects_omap>0
   objects                 plus num_omap_keys and            last deep-scrub
                           last_deep_scrub_stamp             snapshot as a prior
        │
        ▼
4. candidate objects       application knowledge             RGW index shards,
                           (never rados ls + listomapkeys    your own key layout
                            across the pool)                 MDS/RBD metadata
        │
        ▼
5. targeted count          rados listomapkeys <obj> | wc -l  a few objects only
        │
        ▼
6. targeted deep scrub     ceph pg deep-scrub <pgid>         authoritative;
                           ceph osd pool deep-scrub <pool>   updates pg stats and
                                                             the health check
Enter fullscreen mode Exit fullscreen mode

Step 3 deserves a note. The last deep scrub gave you num_omap_keys per PG. A PG that already had 150000 keys across three OMAP objects a few days ago, in a pool whose live OMAP bytes have since doubled, is a much better candidate than a PG with zero. That prior is stale, but it is free, and it is per PG rather than per pool.

Step 6 is the part that people underrate. A deep scrub of a single PG is a bounded, throttled operation that runs through the OSD's scrub machinery with chunking and osd_scrub_sleep, and it produces the authoritative answer in the same place the periodic scrub would: num_large_omap_objects in the PG stats, the object name in the cluster log, and the health check. On the lab PG it took two seconds. On a production PG it is a full read of that PG's data as well as its OMAP, so pick PGs, not pools, unless the pool is an index-only pool that is small in bytes.

Options for making that scrub happen sooner without doing it by hand:

# one PG, now
ceph pg deep-scrub <pgid>

# every PG of a pool
ceph osd pool deep-scrub <pool>

# schedule rather than force, on the primary OSD
ceph tell osd.<id> schedule-deep-scrub <pgid>

# per-pool deep-scrub interval in seconds (overrides osd_deep_scrub_interval)
ceph osd pool set <pool> deep_scrub_interval <seconds>
Enter fullscreen mode Exit fullscreen mode

The pool-level deep_scrub_interval is the closest thing Ceph has to "watch this pool harder". Index pools are small in bytes and expensive in OMAP, which is exactly the profile where a shorter deep-scrub interval costs little.

rados -p <pool> ls is not part of the funnel. It lists names with no sizes, and a pool of any size makes the follow-up per-object listing the very scan that the section on full scans below argues against.

Method 3 — RGW bucket-index monitoring

RGW was not deployed in this lab, so this section is verified against the Tentacle source and the radosgw-admin binary from the same release, not against live buckets.

A bucket index shard is one RADOS object named .dir.<bucket_instance_id>.<shard> in the zone's .rgw.buckets.index pool, and every object in the bucket is at least one OMAP key in one of those shards. That mapping is what makes RGW the one workload where an early check exists that does not read OMAP at all:

radosgw-admin bucket limit check
radosgw-admin bucket limit check --warnings-only
radosgw-admin bucket limit check --uid=<user>
Enter fullscreen mode Exit fullscreen mode

Per bucket it prints num_objects, num_shards, objects_per_shard, and a fill_status of OK, WARN <pct>%, or OVER <pct>%. The numbers come from the per-shard index headers, which RGW maintains on every index update, so the check reads one small header per shard rather than walking the keys. The percentage is objects_per_shard against rgw_safe_max_objects_per_shard (default 102400), and WARN starts at rgw_shard_warning_threshold (default 90, so 92160 objects per shard).

Compare that to the OSD threshold of 200000 keys per object. Dynamic resharding, on by default via rgw_dynamic_resharding, triggers at rgw_max_objs_per_shard (default 100000) and is evaluated by the reshard thread every rgw_reshard_thread_interval seconds (default 600). When it is working, shards are split at roughly half the OSD's key threshold, and LARGE_OMAP_OBJECTS should never fire for a bucket index. When it does fire for an index object, one of these is usually true:

  • Dynamic resharding is disabled, or the bucket exceeded rgw_max_dynamic_shards (default 1999).
  • The deployment is multisite on a release before Reef, where dynamic resharding is not supported.
  • Resharding is queued but not completing. Check radosgw-admin reshard list and radosgw-admin reshard status --bucket=<name>.
  • The index holds more entries than the object count suggests, for example versioned buckets with many versions per name or a backlog of incomplete multipart uploads. Treat objects_per_shard as a proxy with margin, not as a key count.

To go from the cluster log line back to a bucket, take the instance id out of the .dir.<bucket_instance_id>.<shard> object name and match it against the id field in radosgw-admin bucket stats. To go the other way and count a specific shard's keys, the generic tool still applies:

rados -p <zone>.rgw.buckets.index listomapkeys .dir.<bucket_instance_id>.<shard> | wc -l
Enter fullscreen mode Exit fullscreen mode

with the same cost caveats as Method 1. Manual resharding is radosgw-admin bucket reshard --bucket=<name> --num-shards=<n>.

The practical RGW rule: run bucket limit check --warnings-only on a schedule and alert on any output, verify that reshard list drains, and let the OSD-level warning be the backstop rather than the detector.

What Prometheus can and cannot tell us

I enabled the mgr prometheus module on the lab for the duration of the test and pulled the endpoint while the large object existed and the health check was active. Grepping the 113 metric families for omap matched my pool's name in a ceph_pool_metadata label and exactly one real series:

ceph_health_detail{name="LARGE_OMAP_OBJECTS",severity="HEALTH_WARN"} 1.0
Enter fullscreen mode Exit fullscreen mode

That is the only one. It went from 0 to 1 after the deep scrub, and the underlying health check cleared after the object was removed and the PG deep-scrubbed again. It is a correct and useful alert, and it is exactly as late as the health check because it is the health check.

What the module does not export, verified in its source:

  • No per-pool OMAP bytes. The pool DF series are ceph_pool_stored, ceph_pool_stored_raw, ceph_pool_bytes_used, ceph_pool_objects and similar. stored_omap and omap_bytes_used exist in ceph df detail --format json but are not in the list the module publishes.
  • No per-PG num_omap_keys or num_omap_bytes.
  • No num_objects_omap.
  • No per-object anything.

Daemon perf counters are not served by the mgr module either. mgr/prometheus/exclude_perf_counters defaults to true and the expected source is ceph-exporter. Those counters include the BlueStore omap_* series shown earlier. Even when exported, they are per-OSD totals of calls and records. A spike in OMAP set records on one OSD tells you that something is writing OMAP through that OSD. It does not name a pool, a PG, or an object.

So the honest monitoring picture is:

metric exists?            per pool  per PG  per object  live?
------------------------  --------  ------  ----------  -----------------
OMAP bytes (df detail)    yes*      no      no          yes
OMAP key count            no        no      no          deep scrub only†
large omap object count   no        no      no          deep scrub only†
LARGE_OMAP_OBJECTS        cluster   -       -           after deep scrub

* in ceph df detail JSON, not exported by the Prometheus module
† and not exported by the Prometheus module at all
Enter fullscreen mode Exit fullscreen mode

The one PromQL expression I can honestly recommend on a stock Tentacle cluster:

ceph_health_detail{name="LARGE_OMAP_OBJECTS"} == 1
Enter fullscreen mode Exit fullscreen mode

If you want the live pool-level OMAP figure in Prometheus, you have to put it there yourself, for example a small textfile-collector script that runs ceph df detail --format json and emits stored_omap per pool. That is cheap because the OSDs already compute the number for every stats report. It is not something Ceph exports for you.

Why scanning every object is a bad monitoring strategy

The tempting cron job looks like this:

for obj in $(rados -p "$pool" ls); do
    n=$(rados -p "$pool" listomapkeys "$obj" | wc -l)
    [ "$n" -gt 150000 ] && echo "$obj $n"
done
Enter fullscreen mode Exit fullscreen mode

Measured against what the OSD actually does, this is a deep scrub of the pool's OMAP, minus everything that makes deep scrub safe:

  • It reads every key of every object, so the cost is the total OMAP key count of the pool, not the number of objects. An index pool with a few thousand shards of 50000 entries each is hundreds of millions of RocksDB iterations per run.
  • It runs in the client op path on the primary OSDs, competing with real I/O, with no chunking, no osd_scrub_sleep, no scrub reservations, no load or time-window gating.
  • It runs from one client, so the OSD-side reads are serialized behind whatever concurrency the script has and the client becomes a bottleneck long before the cluster does.
  • The result is stale the moment it finishes, exactly like the scrub snapshot, but without the cluster-log entry and health check that a real deep scrub gives you for free.
  • On erasure-coded pools OMAP is not supported at all, and on replicated pools you are still reading only the primary copy, so it checks less than scrub does.

Deep scrub already implements the full-walk detector, with backpressure, and writes the answer into the PG stats and cluster log. If you believe a whole pool needs checking now, the correct full-walk tool is ceph osd pool deep-scrub <pool>, not a shell loop.

A practical early-warning strategy

Everything above collapses into a workflow that leans on the live aggregates for detection, the application for candidates, and deep scrub for confirmation:

                 OMAP-heavy workload
                         │
                         ▼
        ┌── inventory: which pools carry OMAP ──┐
        │   ceph df detail  (STORED/USED OMAP)  │
        │   ceph pg dump pools  num_objects_omap│
        └───────────────────┬───────────────────┘
                            ▼
              trend pool-level OMAP bytes
              (stored_omap per pool, sampled)
                            │
             ┌──────────────┴──────────────┐
             │                             │
            RGW                     generic RADOS
             │                             │
   bucket limit check            application-level counters
   --warnings-only               (keys per object, per shard)
   reshard list drains?          shard keys across objects
             │                             │
             └──────────────┬──────────────┘
                            ▼
             candidate objects / PGs identified
                            │
                            ▼
          rados listomapkeys <obj> | wc -l   (a few objects)
                            │
                            ▼
                  ceph pg deep-scrub <pgid>
                            │
                            ▼
      pg stats num_large_omap_objects, cluster log entry,
      LARGE_OMAP_OBJECTS health check (or its absence)
                            │
                            ▼
      backstop: alert on ceph_health_detail LARGE_OMAP_OBJECTS
      and, for index-style pools, a shorter pool deep_scrub_interval
Enter fullscreen mode Exit fullscreen mode

Concretely:

  1. Know your OMAP pools. Run the ceph df detail and num_objects_omap queries once and write down which pools are OMAP-bearing. On this RGW-less lab the answer is the RBD pools, whose rbd_directory and related metadata objects carry OMAP. On yours it is probably .rgw.buckets.index, the CephFS metadata pool, and whatever your librados applications use.
  2. Trend the live number. Sample stored_omap per pool every few minutes. Alert on growth rate for pools that should be flat and on absolute level for index pools. This is the only continuous signal Ceph gives you and it costs nothing extra.
  3. Instrument the application. If you write OMAP through librados, you already know the key count per object because you wrote the keys. Count them where they are produced and cap them: the MDS uses the OSD threshold itself as the per-object limit and fans out. Do the same.
  4. RGW: check shards, not OMAP. bucket limit check --warnings-only on a schedule, plus confirmation that dynamic resharding is enabled and reshard list is draining.
  5. Inspect, then confirm. For the few candidates that the trend or the application points to, count keys with listomapkeys | wc -l, and deep-scrub the owning PG so the authoritative stats, the cluster log, and the health check all agree.
  6. Shorten the interval where it is cheap. Set deep_scrub_interval on index-style pools that are small in bytes and heavy in OMAP. You are paying for OMAP walks either way; paying more often on a small pool is the cheapest generic way to move the warning earlier.
  7. Keep the backstop. Alert on ceph_health_detail{name="LARGE_OMAP_OBJECTS"} == 1. It is late, but it is authoritative.

Commands cheat sheet

Purpose Command Live or snapshot Cost
Thresholds in force ceph config get osd osd_deep_scrub_large_omap_object_key_threshold / ..._value_sum_threshold config none
Per-pool OMAP bytes ceph df detail, or the stored_omap field of ceph df detail --format json live estimate none
Per-OSD OMAP bytes ceph osd df (OMAP column) live estimate none
One OSD, one pool ceph tell osd.N dump_pool_statfs <poolid> live estimate none
PGs holding OMAP objects ceph pg dump pgs --format json, filter stat_sum.num_objects_omap > 0 live count none
PG OMAP keys/bytes ceph pg ls-by-pool <pool> or ceph pg <pgid> query last deep scrub none
Health check ceph health detail last deep scrub none
Which object ceph log last 300 warn cluster, search for Large omap last deep scrub none
Count keys of one object rados -p <pool> listomapkeys <obj> piped to wc -l live full key walk on primary
Deep scrub one PG ceph pg deep-scrub <pgid> produces snapshot full PG read, throttled
Deep scrub a pool ceph osd pool deep-scrub <pool> produces snapshot full pool read, throttled
Pool-specific interval ceph osd pool set <pool> deep_scrub_interval <sec> scheduling ongoing
RGW shard fill radosgw-admin bucket limit check --warnings-only live from index headers one header read per shard, no key walk
RGW reshard backlog radosgw-admin reshard list live none
Prometheus backstop ceph_health_detail{name="LARGE_OMAP_OBJECTS"} == 1 last deep scrub none

Final result

Back to the scenario: an OMAP object is growing, deep scrub has not run, and ceph health is clean.

What Ceph already has. Per pool, a live estimate of OMAP bytes on disk, visible in ceph df detail and in the pool stats. Per PG and per pool, a live count of objects that have OMAP. Per OSD, aggregate OMAP byte estimates and OMAP operation counters. All of these update without any scrub.

What Ceph only learns during deep scrub. Per-PG OMAP key count and value bytes, the per-PG count of objects over the threshold, and the name, key count, and value size of the offending object. Shallow scrub contributes nothing. These are snapshots that do not move until the next deep scrub of that PG, in either direction: growth after the scrub is invisible, and deleting the object does not clear the warning.

What can be inspected proactively. The live aggregates, at zero cost. The last deep-scrub snapshot per PG, at zero cost, as a prior. Individual objects with listomapkeys, at the cost of the OSD iterating every key of that object.

How expensive. Aggregates are free. A single targeted listomapkeys is linear in the object's key count and runs unthrottled on the primary; fine for a handful, wrong for a loop. A targeted PG deep scrub is a full throttled read of one PG and yields the authoritative answer.

Is there a scalable cluster-wide solution. Not a generic one. Ceph 20.2 does not expose an inexpensive per-object OMAP inventory outside of scrub, and there is no top-N API. Cluster-wide early detection therefore has to come from pool-level trending, application awareness, RGW-specific tooling, and more frequent or targeted deep scrubs of the pools that matter.

What is different for RGW. The index-shard-to-object mapping is fixed and known, bucket stats track entries per bucket without reading OMAP, and bucket limit check plus dynamic resharding keep shards well under the OSD threshold when they work. RGW is the one workload with a cheap, early, per-shard check.

What monitoring to implement. Trend stored_omap per pool from ceph df detail; alert on bucket limit check warnings and a non-draining reshard queue; instrument key counts in your own librados writers; shorten deep_scrub_interval on small OMAP-heavy pools; and keep ceph_health_detail{name="LARGE_OMAP_OBJECTS"} as the authoritative, late backstop.

Top comments (0)