Originally published on the Neblex Blog.
Every integration can move a record from A to B. The difference shows up the morning after something broke overnight.
A payout file with half a million rows failed at row 480,000 because the ERP rate-limited the connection at two in the morning. Now someone has to decide: rerun the whole job and hope the first 480,000 rows are safe to write twice, or hand-pick the tail and load it by hand.
Both options cost a day. The first one also risks doubled journal entries, which is a far worse problem than a late one.
What resuming from the failed step requires
Resuming from the failed step sounds like a feature toggle. It's closer to an architecture decision, and it has three parts.
- Checkpoints inside the run. Bulk steps process records in chunks and record where each chunk ended. A restart reads the last checkpoint and continues from there, so the 480,000 rows that already landed stay untouched.
- An event-sourced run history. Every step records what it received and what it produced. A replay doesn't re-fetch the source and hope it looks the same; it continues from the recorded inputs, so a replay the next morning sees the data from the night before. Those inputs are only kept for a retention window, so replay is a same-day or next-day tool, not an archive.
- Idempotent writes where the target supports them. Some APIs, such as Stripe's, accept an idempotency key: send the same key on a retry and the target returns the original result instead of acting twice. A retried step that reuses its key never double charges or double books. Where a target has no such key, be honest about it: don't promise exactly-once delivery you can't deliver, and match on an external ID before writing instead.
Without all three, "retry" means one of two things: start over and risk duplicates, or hand the tail to a person.
Worked example: Stripe payouts into NetSuite
Stripe payouts into NetSuite are a good test because the volumes are real and the target is strict.
-
Trigger. Stripe sends a
payout.paidevent to a webhook, so the run starts with the payout already in hand. - Fetch. The run pages through the balance transactions behind that payout.
- Transform. Each chunk becomes journal lines with the fee, the net, and the currency conversion split out.
- Write. NetSuite receives the journal lines in chunks, and each chunk is recorded as it lands. If the connection is throttled, the step retries with backoff. If it fails for good, the run stops with the chunk recorded.
- Replay. Someone opens the run, sees chunk 96 of 100 failed, fixes the cause, and replays. The step continues from chunk 96; chunks 1 to 95 are not sent again.
The finance close still happens on time, and nobody reconciles the same payout twice.
What changes for the team that gets paged
The practical change isn't speed. A failed integration becomes a bounded task with a known size, instead of an investigation. The run history says which step failed and why, and the fix is to replay from that step.
That's also why the run history matters more than any dashboard: it's the audit trail, not a report about it.
Questions that come up
Does a replay re-read the source system? It shouldn't. Replaying from recorded inputs reproduces what the run saw at the time. If you want fresh data, start a new run.
What if the target doesn't support idempotency keys? Checkpoints still keep already-written chunks from being resent. For the chunk that failed mid-write, match on an external ID before writing so a partial write isn't duplicated.
Should a person approve the replay? Put the approval in the flow before the ERP write, not on the replay button. Then limit who can replay, so a replay can't skip a control that the original run passed.
How we built it
At Neblex we built our integration runtime around these three parts: chunked bulk steps with checkpoints, a recorded run history that replay reads from, and per-step idempotency keys for targets that accept them. The original post links to how each part works in the product: Replay from the failed step, not from record one.
Top comments (0)