DEV Community

Zijian Xiao
Zijian Xiao

Posted on

Exact-row duplication is a review signal, not proof

AI assistance disclosure: AI helped prepare this text; the author reviewed it. #ABotWroteThis

A transaction table may contain two genuine payments with the same date, description and amount. An exact-row detector can find candidates, but deciding to remove them is a separate step.

Keep source-row references, mark candidate groups, and show the rows before changing the output. Make removal explicit. Keep a list of excluded rows and compare before/after counts. Do not imply that a clean file proves the source was correct.

Date order, decimal separators and debit/credit signs also need explicit choices. A detector run after a wrong sign mapping can produce a tidy but incorrect result. Preserve the original and inspect a small sample first.

Here is a Python candidate index over synthetic, already-normalized rows. It groups matches without deleting either payment:

from collections import defaultdict

rows = [
    {"source_row": 2, "date": "2026-11-01", "description": "Fee", "amount": "-5.00"},
    {"source_row": 3, "date": "2026-11-01", "description": "Fee", "amount": "-5.00"},
]
groups = defaultdict(list)
for row in rows:
    key = (row["date"], row["description"], row["amount"])
    groups[key].append(row["source_row"])
candidates = [refs for refs in groups.values() if len(refs) > 1]
assert candidates == [[2, 3]]
assert len(rows) == 2
Enter fullscreen mode Exit fullscreen mode

Show candidate source rows to the reviewer. Store removal decisions separately, then derive the output and an exclusion list from those decisions. Test both genuine repeated payments and accidental duplicate imports; matching keys alone cannot tell them apart.

In my implementation, CSVQBO retains source-row references and offers optional removal of exact repeated date/description/amount rows after review. It does not guarantee that a generated bank file will be accepted by an importer.

Disclosure: I built this.

Top comments (0)