A few weeks ago I was testing my own receipt parsing API with a batch of a few hundred receipts pulled from a client's expense folder. Midway through, my upload script crashed on a network timeout. I re-ran it.
Nothing errored. Nothing warned me. I just ended up with roughly 40 receipts counted twice.
That's the failure mode I want to talk about, because it's boring, it's common, and it's the kind of bug that doesn't show up in a demo — it shows up three weeks later when someone's expense total is off by a few hundred dollars and nobody can figure out why.
The failure mode, concretely
If you're building anything that ingests receipts in bulk — an expense tool, a bookkeeping integration, a fintech product pulling from email or cloud storage — there are basically three ways duplicates sneak in:
Retry after a partial failure. A batch upload times out or errors halfway through. You retry the whole batch because you don't know exactly which files made it through. Now some receipts are processed twice.
Overlapping date-range pulls. You're syncing from a source (Gmail, Dropbox, a POS export) on a schedule, and your date windows overlap by a day or two "to be safe." Every receipt in that overlap gets ingested twice.
Multiple people, one folder. Two team members both upload the same photographed receipt because neither knew the other already had it.
None of these are edge cases. They're the normal, expected shape of bulk ingestion once more than one process or person touches the pipeline. And the failure is silent — the API doesn't error, the data doesn't look obviously wrong, it just quietly inflates totals.
How bulk upload works today, and where it stops short
I went and looked at how the major receipt/invoice parsing APIs currently handle bulk submissions, because I wanted to know if this was a solved problem I was about to reinvent badly.
Most of them fall into one of two patterns:
Bundle-and-submit. You package multiple files into a single request — a zip, an array of URLs — and the API processes them as one call. This is fast and simple, but it's really designed for "these pages belong to the same document," not "these are N separate receipts I want deduplicated against each other or against what I've already sent." Veryfi's own documentation is upfront about this: when you zip multiple receipt images together, the system treats them as multiple pages of a single transaction, and explicitly does not check for duplicate line items across them. That's not a criticism — it's an honest scope statement. But it means the bundling mechanism and the deduplication problem are two different things, and only one of them is solved.
True async batch processing. Push a batch into a queue, get results back per file via webhook or polling as they complete. Mindee's API works this way, and it's a solid pattern for throughput. But nothing in that flow checks whether file #47 in this batch is the same receipt as file #12 from last week's batch. The queue processes what you send it. If you send it the same receipt twice, you get two processed receipts back.
In both patterns, deduplication is left as an exercise for the caller. Which means every team building on top of these APIs is quietly re-solving the same problem, usually after they've already shipped and someone found the bug in production.
Where dedup actually needs to live
The reason this is worth building into the API layer rather than leaving it to the client is timing. By the time a duplicate receipt has been through OCR and handed back to your application as two structured JSON objects, you've already paid for two extractions and you're now trying to reconcile two records that might have slightly different OCR noise (a misread cent, a slightly different line-item order) even though they're the same physical receipt. Deduplication after the fact means writing fuzzy-matching logic against imperfect extracted data, which is a worse problem than the one you started with.
The alternative is to catch it at ingestion, before extraction runs twice on the same document. That means the batch endpoint itself needs to know: has this file, or a file materially identical to it, already been processed for this account? If yes, don't burn a second extraction — return the existing record, or flag it, depending on how you want to handle it.
That's the model I built into the bulk endpoint on my own receipt parser: batch upload and duplicate detection aren't two separate concerns bolted together, they're the same code path. When a batch comes in, each file gets checked against what's already been processed before it goes through extraction, not after. A retried batch after a timeout doesn't cost you double, and an overlapping sync window doesn't quietly inflate a total three weeks later.
It's a small, unglamorous feature. But "quietly double-counted expenses" is a small, unglamorous bug — and it's the kind that erodes trust in a financial tool faster than almost anything else, because it doesn't look like a bug. It looks like the numbers are just wrong.
If you're building something that ingests receipts or invoices in bulk, it's worth asking early: what happens when the same file shows up twice? If the honest answer is "I'm not sure," that's worth fixing before it's a support ticket.
--
I'm building ilovesreceipt.com, a receipt parsing API for developers, with batch upload and deduplication as core parts of the same flow. Happy to talk shop about extraction pipelines or dedup strategies in the comments.
Top comments (0)