YugabyteDB yb_stats: table and tablet detail

This is a blogpost about using the yb_stats with the new options of --print-table-detail and --print-tablet-detail.

extra-data

Both the --print-table-detail and --print-tablet-detail do fetch table or tablet specific details. Fetching these requires extra calls to the tablet servers and masters. Therefore, these options only work when the --extra-data switch is set, which fetches this additional required data. Therefore normal runs without this parameter will be unaffected by this extra work.

print-table-detail

What does the --print-table-detail show?

➜ yb_stats --print-table-detail --extra-data --uuid 000033e8000030008000000000004100
Table UUID: 000033e8000030008000000000004100, version: 0, type: PGSQL_TABLE_TYPE, state: Running, keyspace: yugabyte, object_type: User tables, name: ysql_test_table
On disk size: Total: 639.60M WAL Files: 579.91M SST Files: 59.69M SST Files Uncompressed: 569.71M
Replication info:
Columns:
0    id                               int32 NOT NULL PARTITION KEY
1    f1                               string NULLABLE NOT A PARTITION KEY
Tablets:
9e75c351ddeb4b8aa4b16afd8bd086cd hash_split: [0xAAAA, 0xFFFF], Split depth: 0, State: Running, Hidden: false, Message: Tablet reported with an active leader, Raft: FOLLOWER: yb-2.local FOLLOWER: yb-1.local LEADER: yb-3.local
85ac22ace45542baa1543d172b8b7657 hash_split: [0x5555, 0xAAA9], Split depth: 0, State: Running, Hidden: false, Message: Tablet reported with an active leader, Raft: FOLLOWER: yb-2.local LEADER: yb-1.local FOLLOWER: yb-3.local
39f2c50a35d043f58bc0ade9e9399dba hash_split: [0x0000, 0x5554], Split depth: 0, State: Running, Hidden: false, Message: Tablet reported with an active leader, Raft: LEADER: yb-2.local FOLLOWER: yb-1.local FOLLOWER: yb-3.local
Tasks:

It shows:

• the table internal version (0).
• the table internal type (PGSQL_TABLE_TYPE).
• the state and keyspace (database) and object name.
• the object type (this will say Index tables for an index).
• the total on disk size, the total WAL size, the total uncompressed and compressed SST file size (for all tablets).
• the columns and their internal types, as well as what is part of the partition key.
• the tablet UUID, the way the tablets are split.
• the tablet state and whether the tablet is running.
• the tablet current last known good state.
• the tasks operating at the table level (such as index backfill).

print-tablet-details

If you want to look deeper into the table, you have to look at the fundamental logical storage units, which are the tablets. The --print-tablet-detail options shows the details for a single tablet.

➜ yb_stats --print-tablet-detail --extra-data --uuid 9e75c351ddeb4b8aa4b16afd8bd086cd
192.168.66.80:9000
 General info:
  Keyspace:       yugabyte
  Object name:    ysql_test_table
  On disk sizes:  Total: 212.42M Consensus Metadata: 1.4K WAL Files: 192.52M SST Files: 19.91M SST Files Uncompressed: 189.95M
  State:          RUNNING
 Consensus:
  State:          Consensus queue metrics:Only Majority Done Ops: 0, In Progress Ops: 328, Cache: LogCacheStats(num_ops=0, bytes=0, disk_reads=0)
  Queue overview:
  Watermark:
  Messages:
 LogAnchor:
  Latest log entry op id: 1.328
  Min retryable request op id: 1.325
  Last committed op id: 1.328
  Max persistent intents op id: 1.327
  Earliest needed log index: 325
 Transactions:
  - { safe_time_for_participant: { physical: 1678909347240643 } remove_queue_size: 0 }
 Rocksdb:
  IntentDB:
  RegularDB:
   total_size: 2088888, uncompressed_size: 19988404, name_id: 14, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
   total_size: 6270955, uncompressed_size: 59742289, name_id: 13, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
   total_size: 6215593, uncompressed_size: 59741912, name_id: 12, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
   total_size: 6296946, uncompressed_size: 59708999, name_id: 10, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
192.168.66.82:9000
 General info:
  Keyspace:       yugabyte
  Object name:    ysql_test_table
  On disk sizes:  Total: 212.96M Consensus Metadata: 1.5K WAL Files: 193.06M SST Files: 19.91M SST Files Uncompressed: 189.95M
  State:          RUNNING
 Consensus:
  State:          Consensus queue metrics:Only Majority Done Ops: 0, In Progress Ops: 0, Cache: LogCacheStats(num_ops=0, bytes=0, disk_reads=0)
  Queue overview: Consensus queue metrics:Only Majority Done Ops: 0, In Progress Ops: 0, Cache: LogCacheStats(num_ops=0, bytes=0, disk_reads=0)
  Watermark:
  - { peer: 61c349193a9847c784d8c781b37574bd is_new: 0 last_received: 1.328 next_index: 329 last_known_committed_idx: 328 is_last_exchange_successful: 1 needs_remote_bootstrap: 0 member_type: VOTER num_sst_files: 4 last_applied: 1.328 }
  - { peer: 4097103c4efe49019e7fd0aace42d046 is_new: 0 last_received: 1.328 next_index: 329 last_known_committed_idx: 328 is_last_exchange_successful: 1 needs_remote_bootstrap: 0 member_type: VOTER num_sst_files: 4 last_applied: 1.328 }
  - { peer: c10e5cfd1f3243238ee654667d49a391 is_new: 0 last_received: 1.328 next_index: 329 last_known_committed_idx: 328 is_last_exchange_successful: 1 needs_remote_bootstrap: 0 member_type: VOTER num_sst_files: 4 last_applied: 1.328 }
  Messages:
  - Entry: 0, Opid: 0.0, mesg. type: REPLICATE UNKNOWN_OP, size: 6, status: term: 0 index: 0
 LogAnchor:
  Latest log entry op id: 1.328
  Min retryable request op id: 1.325
  Last committed op id: 1.328
  Max persistent intents op id: 1.327
  Earliest needed log index: 325
 Transactions:
  - { safe_time_for_participant: { physical: 1678909347317049 } remove_queue_size: 0 }
 Rocksdb:
  IntentDB:
  RegularDB:
   total_size: 2088888, uncompressed_size: 19988404, name_id: 14, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
   total_size: 6270955, uncompressed_size: 59742289, name_id: 13, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
   total_size: 6215593, uncompressed_size: 59741912, name_id: 12, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
   total_size: 6296946, uncompressed_size: 59708999, name_id: 10, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
192.168.66.81:9000
 General info:
  Keyspace:       yugabyte
  Object name:    ysql_test_table
  On disk sizes:  Total: 212.42M Consensus Metadata: 1.5K WAL Files: 192.52M SST Files: 19.91M SST Files Uncompressed: 189.95M
  State:          RUNNING
 Consensus:
  State:          Consensus queue metrics:Only Majority Done Ops: 0, In Progress Ops: 328, Cache: LogCacheStats(num_ops=0, bytes=0, disk_reads=0)
  Queue overview:
  Watermark:
  Messages:
 LogAnchor:
  Latest log entry op id: 1.328
  Min retryable request op id: 1.325
  Last committed op id: 1.328
  Max persistent intents op id: 1.327
  Earliest needed log index: 325
 Transactions:
  - { safe_time_for_participant: { physical: 1678909347315157 } remove_queue_size: 0 }
 Rocksdb:
  IntentDB:
  RegularDB:
   total_size: 2088888, uncompressed_size: 19988404, name_id: 14, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
   total_size: 6270955, uncompressed_size: 59742289, name_id: 13, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
   total_size: 6215593, uncompressed_size: 59741912, name_id: 12, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd
   total_size: 6296946, uncompressed_size: 59708999, name_id: 10, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004100/tablet-9e75c351ddeb4b8aa4b16afd8bd086cd

The information categories for tablet followers and master are identical, however a leader is showing more data and slightly different data because of it being leader.

It shows:

• General info (keyspace name, object name, on disk sizes per tablet, state).
• Consensus state: for the followers, the state row is filled out, for the leader, the queue overview as well as the 'watermark'-s are shown, as well as messages about consensus.
• LogAnchor: the current log state.
• Transactions: the current transactional state is shown for all replicas. The leader will show pending transactions if these are active.
• Rocksdb: for both the IntentDB and RegularDB rocksdb instances the datafiles are shown, with their size, uncompressed size and id/number.

yb_stats Snapshots

If you are looking into an issue or want to store the cluster state for later investigation, then creating a yb_stats snapshot can have the --extra-data switch set. This will take longer, because it will obtain the state of all the tables and their tablets, but it allows you to investigate the state of each of them by specifying the snapshot number after the --print-table-detail or --print-tablet-detail switch.

Conclusion

Both the --print-table-detail and --print-tablet-detail switch require the additional switch --extra-data to obtain the data required, to prevent regular use of yb_stats from fetching individual table and tablet data.

By using the --extra-data switch, table and tablet detail can be investigated, that would otherwise require going in the web UI of the master (for the table data), and going into the web UI of the tablet server serving the tablet to investigate the tablet details.