DEV Community

AWS Community Builders profile image Aki
Aki for AWS Community Builders

Posted on

Building a CLI Tool for Incident Investigation on AWS Iceberg Tables (v2 Update)

#aws #iceberg #dataengineering

Original Japanese article: AWS上のIcebergテーブルの障害調査・運用診断ツールを作ってみた

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

A while back, I built a CLI tool called iceberg-navigator for inspecting the snapshot history of Apache Iceberg tables on AWS.

Building a CLI Tool to Visualize AWS Iceberg Table Snapshot History

The first version shipped with four commands: list, show, compare, and graph. After using it for a while, though, I kept running into the same situations: "an ETL job just failed, which snapshot caused it, and what actually happened?" or "when did the schema change, exactly?"

So this time, I added a set of features aimed specifically at incident investigation and operational diagnostics, and I'm calling it v2.

https://github.com/dataPenginPenguin/iceberg_navigator

To be upfront about it: most of what's under the hood here is still PyIceberg's API, just formatted for the terminal. But I think there's still some value in how it's packaged for actual investigation workflows.

What's New in v2

Four new commands were added on top of the original four:

Command Description
diagnose Show a timeline of snapshots and automatically flag anomalous record-count changes
schema-history Walk Iceberg's metadata.json history and list only the versions where the schema changed
schema-diff Show a side-by-side before/after diff for a given schema version
find-snapshot Find the snapshot that was current at a given point in time

The existing commands also got some upgrades:

  • list: added a Change (%) column showing the record-count change rate from the parent snapshot, plus a --threshold flag that flags rows whose change rate meets or exceeds the threshold
  • graph: color-coded nodes (APPEND = blue, DELETE = red, REPLACE = green), a chronological layout, and a --limit option

Installation was also cleaned up so the tool is available as a proper iceberg-navigator command via pip install -e ..

Installation 

git clone https://github.com/dataPenginPenguin/iceberg_navigator.git
cd iceberg_navigator
pip install -e .
Enter fullscreen mode Exit fullscreen mode

Make sure your AWS CLI credentials are configured beforehand.

Commands

List Snapshots with Change Tracking 

iceberg-navigator list --table <dbname>.<tablename>
iceberg-navigator list --table <dbname>.<tablename> --threshold 50
Enter fullscreen mode Exit fullscreen mode

Lists all snapshots for a table. When --threshold is set, any row whose record-count change from the parent snapshot meets or exceeds that percentage gets flagged with ⚠.

|    | Snapshot ID         | Timestamp            | Operation | Parent Snapshot ID   | Total Size (MB) | Record Count | Change (%) |
|----|---------------------|----------------------|-----------|----------------------|-----------------|--------------|------------|
|    | 1533347322559466931 | 2025-05-22T02:10:24Z | APPEND    | null                 |           13.48 |      729,732 |          - |
| ⚠  | 1485371543345582290 | 2025-05-22T02:10:54Z | DELETE    | 1533347322559466931  |            0.00 |            0 |    -100.0% |
|    | 6369576239134108166 | 2025-05-22T02:41:51Z | APPEND    | 3920289554540444894  |           13.48 |      729,732 |          - |
| ⚠  | 6216935665394419954 | 2025-05-22T02:41:54Z | APPEND    | 6369576239134108166  |           26.96 |    1,459,464 |    +100.0% |
| ⚠  | 8477810866715341247 | 2026-01-02T14:34:44Z | APPEND    | 5262386007961017819  |          187.45 |   10,445,650 |  +1331.4%  |
Enter fullscreen mode Exit fullscreen mode

You can raise the threshold to cut down the noise:

# only show full deletes (-100%) and changes of +100% or more
iceberg-navigator list --table <dbname>.<tablename> --threshold 100
Enter fullscreen mode Exit fullscreen mode

Show Snapshot Details 

iceberg-navigator show <snapshot_id> --table <dbname>.<tablename>
Enter fullscreen mode Exit fullscreen mode

Shows the schema, operation type, and summary statistics for a given snapshot.

Table: yellow_tripdata

Snapshot ID: 8362052512120533476
Timestamp: 2026-01-02T09:44:22Z
Operation: APPEND
Parent Snapshot ID: 9025884583559565847
Manifest List: s3://<your-bucket>/warehouse/.../snap-....avro

Schema:
  1: vendorid: optional int
  2: tpep_pickup_datetime: optional timestamp
  ...

Summary:
  added-records: 729,732
  total-records: 1,459,464
  total-data-files: 2
Enter fullscreen mode Exit fullscreen mode

Compare Snapshots 

iceberg-navigator compare <snapshot_id> --table <dbname>.<tablename>
Enter fullscreen mode Exit fullscreen mode

Compares a snapshot with its parent, showing the change in record count and file size.

----------------------------------------
Parent Snapshot
----------------------------------------
ID:         9025884583559565847
File Size:  13.48 MB
Records:    729,732

----------------------------------------
Current Snapshot
----------------------------------------
ID:         8362052512120533476
File Size:  26.95 MB
Records:    1,459,464

========================================
Summary
========================================
Added Records:   729,732
Deleted Records: 0
Enter fullscreen mode Exit fullscreen mode

Diagnose: An Anomaly-Detection Timeline 

iceberg-navigator diagnose --table <dbname>.<tablename> --threshold 50
Enter fullscreen mode Exit fullscreen mode

Shows every snapshot in chronological order and flags any record-count change that meets or exceeds the threshold with ⚠. At the end, it summarizes the flagged anomalies and suggests the compare command to dig into each one.

Anomaly threshold: ±50% record change

⚠ 2026-01-02T09:44:22Z  APPEND  records:    1,459,464  change:  +100.0%  [8362052512120533476]
⚠ 2026-01-02T14:34:44Z  APPEND  records:   10,445,650  change: +1331.4%  [8477810866715341247]
⚠ 2026-01-02T14:46:34Z  APPEND  records:   20,161,568  change:   +93.0%  [493829446962284285]
⚠ 2026-01-02T15:16:59Z  DELETE  records:            0  change:  -100.0%  [8442142038781057784]

============================================================
⚠  4 anomaly(ies) detected (threshold: ±50%)
============================================================
  2026-01-02T14:34:44Z  APPEND  +1331.4%  [8477810866715341247]
  → Run: iceberg-navigator compare 8477810866715341247 --table <dbname>.<tablename>
Enter fullscreen mode Exit fullscreen mode

The diagnose → compare flow lets you quickly narrow down when and where something happened.

To be precise about what this actually does: it's not diagnosing an incident in any deep sense, it's flagging record-count outliers as candidates for investigation. Think of it as a first pass that narrows down where to look next, not an automated root-cause finder.

Schema History 

iceberg-navigator schema-history --table <dbname>.<tablename>
Enter fullscreen mode Exit fullscreen mode

Walks Iceberg's metadata.json history and lists only the versions where the schema actually changed.

5 schema change(s) detected:
============================================================
Schema ID   : 0 -> 1
Timestamp   : 2026-06-12T09:38:24Z
  Added columns:
    + new_col: int
  -> Run: iceberg-navigator schema-diff 1 --table <dbname>.<tablename>
============================================================
Schema ID   : 2 -> 3
Timestamp   : 2026-06-12T10:00:51Z
  Added columns:
    + status: string
  -> Run: iceberg-navigator schema-diff 3 --table <dbname>.<tablename>
============================================================
Schema ID   : 3 -> 4
Timestamp   : 2026-06-12T10:01:56Z
  Removed columns:
    - test_col: string
  -> Run: iceberg-navigator schema-diff 4 --table <dbname>.<tablename>

... (truncated, 5 total)
Enter fullscreen mode Exit fullscreen mode

Schema Diff 

iceberg-navigator schema-diff <schema_id> --table <dbname>.<tablename>
Enter fullscreen mode Exit fullscreen mode

Pass a schema_id from schema-history to see the full before/after schema side by side, including columns that didn't change, so it's immediately clear what actually moved.

Schema diff: schema_id 3 -> 4

  Column         Before                After                 Change
  -----------------------------------------------------------------
  fl_date        date                  date
  dep_delay      int                   int
  arr_delay      int                   int
  air_time       int                   int
  distance       int                   int
  dep_time       double                double
  arr_time       double                double
  new_col        int                   int
  test_col       string                -                     - REMOVED
  status         string                string
Enter fullscreen mode Exit fullscreen mode

Find Snapshot by Timestamp 

iceberg-navigator find-snapshot --table <dbname>.<tablename> --at "2026-01-02T14:00:00+00:00"
Enter fullscreen mode Exit fullscreen mode

Finds and displays the snapshot that was current as of a given timestamp. Useful when you need to check what state the table was in around the time an incident started.

Snapshot at 2026-01-02T14:00:00+00:00:
Table: yellow_tripdata
Snapshot ID: 8362052512120533476
Timestamp: 2026-01-02T09:44:22Z
Operation: APPEND
Parent Snapshot ID: 9025884583559565847

Summary:
  total-records: 1,459,464
  total-data-files: 2
  total-files-size: 28,258,822
Enter fullscreen mode Exit fullscreen mode

Lineage Graph 

iceberg-navigator graph --table <dbname>.<tablename> --limit 10 --output graph_recent.png
Enter fullscreen mode Exit fullscreen mode

Layout and readability got a big upgrade from v1. It now supports a chronological layout (newest at the top), color-coded operations (blue/red/green for APPEND/DELETE/REPLACE), and a --limit option to cap how many snapshots are shown.

For a large table (316 snapshots), rendering everything at once makes the labels too cramped to read:

--limit to show only the most recent N snapshots is recommended for tables like this.

Use Cases

Here's how these new features play out in some actual investigation and operations scenarios.

1. An ETL job just failed 

# scan the timeline for anomalies first
iceberg-navigator diagnose --table db.table --threshold 50

# then dig into the suspicious snapshot
iceberg-navigator compare <snapshot_id> --table db.table
Enter fullscreen mode Exit fullscreen mode

diagnose narrows down which snapshot had a sudden spike or drop in record count, and compare shows exactly what changed against its parent.

2. A query started failing after a schema change 

# find out when the schema changed
iceberg-navigator schema-history --table db.table

# see exactly what changed, before/after
iceberg-navigator schema-diff <schema_id> --table db.table
Enter fullscreen mode Exit fullscreen mode

schema-history pinpoints when the schema changed, and schema-diff shows what changed.

3. You need to check the data as of a specific point in time 

# find the snapshot current at the time of the incident
iceberg-navigator find-snapshot --table db.table --at "2026-01-02T14:00:00+00:00"
Enter fullscreen mode Exit fullscreen mode

Combined with Iceberg's time-travel capability, you can take the resulting snapshot ID straight into an Athena query:

SELECT * FROM db.table FOR SYSTEM_VERSION AS OF 8362052512120533476;
Enter fullscreen mode Exit fullscreen mode

4. You want a picture of the table's operational history 

# get the big picture as a graph
iceberg-navigator graph --table db.table --limit 20

# check recent changes on the timeline
iceberg-navigator diagnose --table db.table --threshold 30
Enter fullscreen mode Exit fullscreen mode

Useful both for visually understanding what operations a table has been through, and for running diagnose periodically as a lightweight anomaly check.

Conclusion

With this update, iceberg-navigator moved from "a tool for browsing snapshots" to "a tool for incident investigation and operational diagnostics."

Under the hood, most of it is still PyIceberg's API calls formatted for the terminal, and I won't pretend otherwise. But I do think there's real value in the diagnose → compare and schema-history → schema-diff flows, which are designed to point to "what to look at next" rather than just dumping data, and in mapping the commands onto actual investigation scenarios rather than just listing features.

The source is here. Feedback and suggestions are always welcome via issues or comments:

https://github.com/dataPenginPenguin/iceberg_navigator

I hope this is useful to anyone running Apache Iceberg tables on AWS Glue Data Catalog.

Top comments (0)