A job that exits cleanly tells you one thing: the job finished. It does not tell you that every record you meant to write is actually in the destination.
Those are two different questions, and mixing them up is how you end up discovering missing rows days later, from a user report instead of from your own checks.
Here is a simple, manual way to answer the second question after an import, sync, migration, or scheduled batch job.
1. Pin down the scope first
Before comparing anything, write down exactly what you are checking:
- the run ID (or the time window of the run)
- the destination (table, bucket, index, API)
- any filters the job applied
Then use the same scope on both sides of the comparison. A surprising number of "missing data" investigations turn out to be two queries with slightly different time windows or filters.
2. Build the list of what should be there
Use the input of that run, or a snapshot saved from it, to list the IDs that were expected to arrive.
Two things matter here:
- Use the input from that run, not today's version of the source. Sources change; your expected list should not.
- Record intentional exclusions separately. If the job is supposed to skip some records (filtered, invalid, deleted), list them on their own so they do not get mixed up with real gaps.
If you cannot rebuild this list, stop here. Without an expected list, you cannot honestly say the data is complete — only that nothing obviously broke.
3. Check what was actually saved — and compare IDs, not counts
Get the IDs from the destination itself, not from the job's success log. The log tells you what the job tried to do.
Then compare IDs, not just totals. Counts can match while the data is wrong:
- 1,000 expected, 1,000 found — but 3 are missing and 3 are duplicates.
- Same total, different rows.
So check three lists separately:
- missing — expected, but not in the destination
- unexpected — in the destination, but not expected
- duplicates — the same ID written more than once
One more thing: if the destination has a documented write or visibility delay (replication lag, eventual consistency, indexing), wait for it before you call anything missing.
4. Before you rerun anything
When IDs are missing, the tempting fix is to rerun the whole job. Before you do, check whether repeating the operation could create duplicates or repeat side effects — a second charge, a second email, a second webhook.
A safer order:
- Trace one missing ID through the input, the processing logs, and the destination. Find where it dropped.
- Fix or confirm the cause.
- Try a small, controlled rerun on a few IDs before a full one.
When is the check "done"?
Call the review complete when:
- every expected ID is accounted for,
- unexpected and duplicate records are explained,
- and the evidence is written down.
If gaps remain, mark it unresolved and write down the next check and who will do it. Note that this checks that records are present — it does not prove every field is correct.
If you want this as a printable sheet
I turned the steps above into a two-page PDF checklist with fields for the scope, expected and confirmed IDs, missing / unexpected / duplicate records, evidence, and the next action. It is a manual review aid, not a scanner or a repair script.
It is $5 here: https://alkalenia.gumroad.com/l/vgnmup
But everything you need to do the check is in this post — the sheet just gives you a place to write it down.
How do you verify that a "successful" batch job actually delivered everything? I would like to hear what works for you.
This post was drafted with AI assistance and reviewed by a human.
Top comments (0)