DEV Community: Gruv AI

Two people got paid twice because my webhook handler took six seconds

Gruv AI — Tue, 08 Sep 2026 10:31:08 +0000

A contractor emailed to say she had been paid twice. She was not complaining, which somehow made it worse.

That was nine days after the run. The batch held 3,412 items, every one reconciled, and my dashboard said 3,412 sent. It also said 3,414 payments had actually left, and nobody had ever thought to compare those two numbers because there had never been a reason to.

Two items. €1,284 between them. The interesting part is not the bug. It is that every individual piece of the system behaved exactly the way its documentation said it would.

The provider did nothing wrong

It posts a webhook when a payout item becomes payable, and waits five seconds for my service to answer. If no answer arrives, it assumes the message was lost and posts again. That is a sensible design and it is written plainly in their docs.

My handler's p99 that afternoon was 6.2 seconds.

So the provider waited its five seconds, heard nothing, and redelivered. The second delivery landed 41 milliseconds after the first by my own log timestamps, while the original was still running. Both of them went all the way through, and both of them sent a payment.

If you are integrating anything that pays people in batches, the three things worth reading before you write a line are idempotency, item-level errors and webhook behaviour. Between them they decide whether a retry is safe, and I had assumed rather than checked.

The shape of the mistake

My handler read the payout item, saw its status was still pending, sent the payment, then wrote the status back as sent.

Read that as one request and it is obviously fine. Read it as two requests arriving 41 milliseconds apart and the problem is immediate: both of them read, both of them saw pending, because neither had written anything yet. The gap between reading a value and acting on it was nearly six seconds wide, and I had left it completely unguarded.

This is check-then-act. I could have named it in an interview. Nobody caught it in review either, because in the single-request reading it looks correct, and the single-request reading is the one your eye does.

The reason it survived testing is duller still. A duplicate that arrives while you are still processing the first one is genuinely hard to produce by hand, so nobody ever had. I now replay exactly that case with a webhook simulator set to deliver twice before anything of mine goes near production.

Why deduplicating on the event ID did not save me

I did have deduplication. That is the part that stung.

There was a table of processed event IDs, and the handler wrote to it once it had finished. It changed nothing, for a reason that took an embarrassing while to see: the deduplication check was itself a read followed by an act. Same race, one table over. Both requests looked, both found nothing, both carried on.

There is a second reason it was the wrong key anyway. An event ID identifies a delivery. What I actually needed to guarantee was one payment per payout item, and those are not the same promise. Key on the delivery and you are protected against the provider sending the same envelope twice. Key on the thing that matters to the business and you are protected against everything that means pay this person, however it arrives.

The fix was a rule, not more code

The database is the only part of the system that can settle an argument between two requests arriving at the same instant. So I let it.

Now there is a small table whose only job is to record that a payout item has been claimed, and the item's identifier is its primary key. Before anything sends a payment, it tries to insert a claim. Exactly one of two simultaneous attempts can succeed, because the database will not accept the same key twice. The winner sends the payment. The loser is told the row already exists, does nothing, and returns.

No lock table. No coordination service. No clever application logic. The uniqueness rule is the entire mechanism and everything around it is bookkeeping.

The second half was making the handler fast enough that the provider stops retrying at all. It now acknowledges the message immediately and does the actual work on a queue, where the provider is not holding a stopwatch. My p99 went from 6.2 seconds to about 40 milliseconds and the retries stopped.

What it costs

Claiming an item before sending means a crash between the claim and the send leaves that item claimed and unpaid. It will not retry. The claim already exists, so every later delivery correctly declines to act, which is precisely what I asked the system to do. So I run a sweeper that releases claims older than fifteen minutes with nothing to show for them. That sweeper is now something I maintain forever, and it has a failure mode of its own, because one that is too eager reintroduces the original bug from the other direction. It is a far smaller problem than paying people twice. It is not nothing, though, and anyone telling you the idempotent version is strictly better has never operated one.

Acknowledging immediately has the same shape. I traded the provider's retry logic, which was free and well tested, for my own queue's retry logic, which is now my problem.

I also cannot fully explain why only two of the 3,412 duplicated. The slow window touched perhaps forty items and the rest presumably lost the race in the harmless direction. I never reproduced the exact interleaving outside a load test, and I have stopped trying.

The part that was never an engineering problem

I found this because a contractor was honest. That is not a control.

The check that would have caught it, comparing items marked sent against payments actually made, is not difficult. It existed nowhere in my stack because those two numbers lived in different systems owned by different teams, and no single person had ever been uncomfortable enough to go and put them side by side. If you need to argue for the time to fix something like this, someone measured a month of that kind of gap and found more than 40 per cent of the handling time sitting inside the exceptions rather than the ordinary work. That number is a better argument than any of mine.

There is also a version of this with no concurrency in it at all, where the file was already wrong before any of my code ran and
a spreadsheet had eaten the leading zeros. Different failure, same lesson about trusting an upstream you did not write.

Go and compare the number of items you marked sent last month against the number of payments your provider actually made. Those two figures should be identical, and until you have looked, you do not know that they are.

Figures and identifiers in this post are illustrative.

How did you find yours? Mine told me herself, nine days late.

Payout Engineering at Gruv

Your payout CSV was already broken before validation ran

Gruv AI — Tue, 08 Sep 2026 06:03:31 +0000

A partner sent their monthly contractor file on the first, the way they had for two years. 1,847 rows. Our importer validated every one of them and reported no errors. We funded the batch and submitted it.

63 payments came back as R13, invalid routing number.

The numbers were correct when their finance lead typed them, and correct in the system they were exported from. Every one of the 63 had the same thing wrong with it: a missing leading zero.

What a spreadsheet does to bank data

Open a CSV in Excel or Sheets, save it, and the file you get back is not the file you opened. Type inference runs on every cell, and finance data is exactly what it gets wrong.

021000021 is a valid routing number. Nine digits, leading zero, checksum passes. A spreadsheet sees a number, drops the zero, and writes back 21000021. Eight digits, still parses as an integer, still sitting in the column looking like data.

A 17-digit account number becomes 1.23457E+16 and loses its last digits permanently.
There is no repair for information that is gone.

An amount typed as 1.234,56 in a German locale and reopened under a US one becomes either 1.23456 or 1234.56 depending on the path, and no checksum will tell you which. That is the one that frightens us. A wrong routing number fails loudly at the bank. A wrong amount succeeds.

None of it is visible in the spreadsheet's own display.

Keep the string

Our importer's first mistake was letting a parser be helpful. Read the file as bytes, strip the byte order mark if it is there, and keep every value a string. No integer coercion, no date parsing. In pandas, dtype=str and keep_default_na=False, or a payee who wrote NA as a country code arrives as NaN.

The second mistake was validating the parsed value instead of the raw one. By the time int("21000021") has succeeded, the evidence is gone. Length is evidence. Leading characters are evidence.

Checks that catch it

Routing numbers carry their own checksum, so a mangled one can be caught without asking a bank.

def aba_valid(rtn: str) -> bool:
    if len(rtn) != 9 or not rtn.isdigit():
        return False
    w = (3, 7, 1, 3, 7, 1, 3, 7, 1)
    return sum(int(d) * k for d, k in zip(rtn, w)) % 10 == 0

The line that catches the stripped zero is not the arithmetic. It is the length test on the first line, and it is the one people leave out because it feels redundant next to a checksum.

Amounts have no checksum to lean on. Require one unambiguous format and reject the rest.

AMOUNT = re.compile(r"^\d{1,12}(\.\d{2})?$")   # no separators, cents explicit

Refusing 1.234,56 and 1,234.56 alike is not pedantry. A human reads both. A program cannot, without knowing which country produced the file, and you do not know that.

Why not just pad the zeros back on?

You can see the zero is missing, padding produces a valid checksum, and the run is due.

We stopped doing that. The spreadsheet operated on the whole file, not one column. If the routing numbers were mutated, the amounts met the same locale handling. You can repair the damage you have a checksum for. You cannot detect the damage you do not, and repairing the visible half produces a file that looks clean and is not.

So the importer rejects the file, not the row. It returns the failing line numbers, the column, the value received, and what was wrong with it. A rejection with 63 line numbers gets fixed in an afternoon. A padded batch gets found at month end, if at all.

We are less sure about this than about the rest. With thousands of small sellers uploading their own files, a hard reject may just mean nobody ever finishes an upload, and a per-row quarantine might serve them better. We have only run this on partner files, where a named person can produce a new export.

Stop the bad file being made

The files that arrive intact never touch a spreadsheet: a direct pull, or an upload that validates on the spot and shows the sender their own errors while they still have the original in front of them. The person who can fix a wrong account number is the person who typed it, and they stop being reachable the moment the file leaves their
hands. When you design how a file actually gets into the system, that upload moment is where the time goes.

Where a spreadsheet is unavoidable, ask for text-formatted columns in the template; a column formatted as text before the paste survives a save. Better still, take the data over a connection: the tradeoff of
file and connector intake is setup work against a class of silent corruption that never happens.

An earlier post on this blog has the other half of this. A designer in Lisbon lost forty-one days of cash flow to an invoice rejected for formatting nobody could see.
Same failure, opposite end of the wire.

Disclosure: we build payout infrastructure at Gruv. The routing number above is a valid ABA number; every other figure is synthetic.

Payout Engineering at Gruv

Forty emails, eighteen actual problems

Gruv AI — Mon, 07 Sep 2026 12:06:09 +0000

This is a composite, drawn from conversations with several people who run seller operations at marketplaces. The details have been changed and it is not one person's account, which is why there is no name on it.

The marketplace sells handmade ceramics. Around 430 active sellers across fourteen countries, most of them one person and a kiln. Payouts run on the fifth of every month.

On the sixth, the seller-operations lead opens an inbox with somewhere between thirty and fifty emails in it, and every one of them is a version of the same question.

Where is my money.

Why "let me check" takes three days

They cannot answer any of those emails from the inbox they are reading.

The payout run happened inside a system they have read access to, sometimes, if the finance lead is at their desk. Per-seller results live in a CSV that lands in a shared drive, sorted by internal payee ID rather than shop name, which means answering one seller's question involves finding that seller's ID first. There are 430 of them.

So the reply is "let me check and get back to you," written thirty times before lunch.Then a spreadsheet of names. Then a Slack message to an engineer who is mid-sprint and who will, kindly, run a query that afternoon or tomorrow. That query costs the engineer half a day and the seller-operations lead most of a week, and it produces information that was already sitting in a table. By the time an answer comes back, the seller has sent a second email, and some of them have posted in the seller forum, where the tone is worse and other sellers are now also wondering. The whole cycle then repeats on the sixth of the following month, with the same people, about a different set of invoices.

None of that work produced anything. The information existed the whole time.

What was actually wrong

Here is the breakdown from one of those months, which is the part that changed how they thought about it.

Twelve sellers had genuinely been held before the run, because their tax details were incomplete. All twelve had been sent an automated email about it eleven days earlier. None of the twelve had opened it, which is a fact about automated emails rather than about those sellers.

Six had bank details that bounced. They did not know, because nobody had told them, and the failure had been logged in a system none of them could see.

The other twenty-two had been paid. On time, correctly, on the fifth. They were writing in because the money had not appeared in their account yet and they had no way of knowing whether that was normal, and a payout that takes two working days to settle looks identical to a payout that never happened if you are watching a bank app.

Eighteen real problems. Twenty-two emails that were not problems at all.

What changed

Two things, and they are separate.

The first was giving sellers somewhere sellers can look without asking. Not a status page for the platform, a view of their own payout: what was sent, when, what state it is in, and whether anything is waiting on them. The twenty-something who had already been paid stopped writing in almost immediately. Not because they became more patient, but because the question they had was answerable and now it was answered.

The second was making the run itself report properly, so that status on each row of the run is visible to the person answering the email rather than to the person who ran the batch. "Let me check" became a sentence in the same reply instead of a promise about tomorrow. That mattered more for the eighteen genuine problems than for the twenty non-problems, because the twelve held sellers needed to be told something specific and actionable, and the six with bad account details needed to be told immediately rather than in four days.

The seller-ops lead described the change less as a time saving and more as no longer starting the week owing thirty people an answer they did not have.

What it does not fix

It does not reduce the number of failed payouts.

Bad account details still fail. Incomplete tax profiles still hold a payment before it moves. Nothing about better visibility makes a wrong IBAN correct, and any tool that claims otherwise is describing a different problem. What changes is who finds out, and when, and whether they are told something they can act on. Six people learning on the sixth that their bank details bounced is a much better month than six people learning it on the tenth, from someone who had to go and ask.

We are less certain how this holds up on a marketplace where selling is occasional. Some of these sellers ship every week and will happily check a portal. Someone who lists twice a year is not going to log in to look at anything, and for them the automated email that nobody opens is still the whole system. We do not have a good answer for that yet.

An earlier post on this blog is the same month from the other side of the wire, written by someone waiting for a payment that had, in fact, already been sent. Reading the two together is uncomfortable in a useful way.

Disclosure: we build payout infrastructure at Gruv, so we have a view on this. The story is a composite and the figures are illustrative.

If you run payouts for a marketplace: what fraction of your payout-day inbox is a real failure, and what fraction is someone who cannot see what happened? We would genuinely like to know whether eighteen out of forty is typical.

The Spreadsheet I Built to Compare Wise, Payoneer, and SWIFT: A Lahore Analyst's Notes on Getting Paid

Gruv AI — Mon, 29 Jun 2026 09:11:54 +0000

I am a data analytics consultant in Lahore. I build dashboards and forecasting models, mostly for SaaS companies in the United States and the UK, and occasionally for a regional bank that pays me in PKR and reminds me, every quarter, why I prefer dollar invoices. I have been doing this for five years. For most of those five years I was vaguely losing money on the way payments arrived in my account, and only in the last eighteen months did I sit down and quantify how much.

The trigger was a $7,200 invoice from a Boston client that, by the time it landed in my HBL account, had become approximately $6,830 of usable funds after the foreign exchange margin, the correspondent bank fee, and a small charge I never fully understood that my bank labelled "remittance handling." That is a 5.1 percent total cost on what should have been a clean transfer. I had been telling myself for years that the cost of receiving money was a fixed unavoidable tax. It is not. It is a choice, and I had been making the wrong one.

What I did, eventually, was build a spreadsheet. The spreadsheet had three columns, one each for Wise, Payoneer, and direct SWIFT into my Pakistani bank. It modelled the round trip from invoice to PKR-in-hand, on three representative invoice sizes: $1,500, $5,000, and $12,000. It included not just the visible fee but the exchange rate margin, which is where most of the real cost hides. I want to share what I found, because I suspect a meaningful number of Pakistani freelancers are paying the same hidden tax I was, and have not run the numbers.

For the $5,000 representative invoice, here is what the spreadsheet showed.

Direct SWIFT into my HBL account, with the client's bank using Citi as the correspondent: total cost, including the $20 outgoing wire from the client's side, a $25 correspondent fee, and an exchange margin of roughly 2.3 percent applied by HBL when converting to PKR, came to approximately $160. That is 3.2 percent of the gross. The funds were available in PKR in three working days. The receipt was a Proceeds Realization Certificate, which is what State Bank wants for documenting foreign currency inflows.

Payoneer, with the client paying into my Payoneer USD balance and me transferring to my HBL account: receiving fee from the US client was nominally free for ACH, but Payoneer's conversion margin to PKR ran around 2 percent, and there is a small withdrawal fee. Total cost was approximately $115, or 2.3 percent. Time to PKR was about two days. Payoneer issues its own documentation that State Bank has generally accepted, though some banks ask for additional paperwork.

Wise, with the client paying USD into my Wise account and me converting to PKR and withdrawing: Wise's conversion fee on USD to PKR is published and was around 0.55 percent at the time I tested, with a small fixed component. Total cost was approximately $42, or 0.84 percent. Time to PKR was under twenty-four hours for most transfers. The complication is documentation. Wise's transaction record is not always accepted by Pakistani banks in the same form as a SWIFT PRC, and the regulatory treatment of Wise inflows into Pakistan has changed twice in the past three years.

What helped me think through the documentation side, which is the part most fee comparisons skip, was [a guide to receiving USD payments into a Pakistani bank account. It covers the State Bank requirements, the Proceeds Realization Certificate, and what counts as adequate documentation when the inflow comes through a fintech rather than a direct SWIFT wire. I had been ignoring this question and hoping it would not catch up with me. It would have, eventually.

For the regulatory comparison between the fintechs themselves, a comparison of Wise and Payoneer for Pakistani freelancers was the cleanest writeup I found. It does not just compare fees. It walks through the operational reality of each platform under current Pakistani regulations, including which one is more reliable for larger invoices and which one is less likely to flag your account for review. The piece changed how I structure the question for myself. The right answer, for me, turned out to be different at different invoice sizes.

What I do now, after eighteen months of running the spreadsheet, is route invoices by size and client preference. Invoices under $3,000 go through Wise, where the proportional fee saving is largest. Invoices between $3,000 and $10,000 go through Payoneer, which gives me the best balance of fee and documentation. Invoices above $10,000, and any from clients whose accounts payable team prefers traditional wires, go through direct SWIFT, where the fee is highest in percentage terms but the documentation is unambiguous and the PRC arrives without me having to ask twice.

The total I have saved over the last twelve months, by my own measurement, is approximately $2,400. That is roughly a month of expenses for my household. It is also, for context, more than I spend in a year on the tooling I use to do the actual analytics work. The lesson, which feels obvious in retrospect, is that the cost of payment infrastructure is not a small detail. It is a meaningful line item in a freelance P&L, and it deserves the same analytical attention I give to a client's customer acquisition cost.

If you are reading this from somewhere with similar friction, the spreadsheet is the work. Three columns, three invoice sizes, every fee made visible including the exchange margin. An afternoon of work. A meaningful change in net income. The arithmetic, as in most things, rewards the person who actually does it.

A Rejected Invoice in Lisbon: What I Learned the Expensive Way

Gruv AI — Thu, 21 May 2026 12:47:23 +0000

There is a particular kind of silence that follows a rejected invoice. The client's accounts payable team did not call to argue. They did not even email back the same day. They simply paused the payment, copied their tax advisor, and asked me to "kindly correct the formatting." That polite sentence cost me forty-one days of cash flow.

I am an American creative director living in Lisbon. I run a studio of one. My clients are in Berlin, Stockholm, New York, and occasionally Tokyo. My work is precise. I obsess over kerning. I argue, gently, about whether a logo should sit two pixels to the left. And yet for nearly three years I treated my own invoicing the way a bored teenager treats a high school book report. Templates copied from a friend. VAT lines adapted from memory. A reverse charge clause that I had once read on a forum and never properly understood.

The German invoice that broke my streak was for a six-figure brand identity project. The client was a mid-sized B2B SaaS company. Their accounts team rejected the invoice because my reverse charge note was non-compliant for their jurisdiction, my VAT ID validation was missing the country prefix, and the line items did not clearly separate the design fee from the licence transfer. None of this was complicated. All of it was learnable. I had simply never sat down to learn it.

What I would tell my younger self, the one who set up the studio in 2021 with a spreadsheet template and a hopeful smile, is the following.

The rules for invoicing across borders are boring, and they are also non-negotiable. EU B2B work between VAT-registered businesses in different member states almost always uses the reverse charge mechanism. That means I do not charge German VAT on a German client invoice, but I have to say, on the invoice itself, that the reverse charge applies, and I have to validate their VAT number through VIES before I send it. The validation is not optional. If they later turn out to have a deregistered or invalid VAT ID, the tax authority can come back to me. I finally got the mechanics straight from a clean walkthrough of the reverse charge for cross-border B2B work, and even though it is framed for a UK Ltd, the structure translated cleanly to my situation.

Currency is the second thing I underestimated. I used to invoice in euros for European clients and in dollars for US clients, then accept whatever exchange rate landed in my account. After tracking it for six months, I realised I was bleeding roughly 2.4% per cross-currency invoice. Some of that is unavoidable. A lot of it was me being lazy. I now invoice in the client's preferred currency, but I receive in the currency of the country where I will spend the money, and I refuse to use the default conversion offered by the most convenient platform. The platform is not optimising for my margins. I am.

The third thing, and this one took the longest to admit, is that as a US citizen abroad I have a stack of compliance obligations that no amount of European tax knowledge will substitute for. The Foreign Earned Income Exclusion, foreign tax credits, the interaction between Portugal's tax regime and US filing obligations. I had been reading scraps for years. What pulled it together for me was an honest look at how the FEIE actually interacts with foreign tax credits and the rest of the stack. It does not pretend the FEIE is a magical shield. It is not. It is one tool in a stack, and using it wrong is worse than not using it at all.

The studio runs differently now. Every invoice goes through a checklist before it leaves my outbox. VAT ID validated and screenshotted. Reverse charge clause present and worded correctly for the destination country. Currency, payment terms, and bank details consistent with my contract. Identification of the supply, with the date of supply, separated cleanly from any licence transfers. It takes me eight minutes longer per invoice than it used to. It has not been rejected since.

The wine in Lisbon is still excellent. The light is still ridiculous. But I no longer have a quiet panic in my chest every time a large invoice goes out. That trade, eight minutes for forty-one days of waiting, is the best deal I have made in years.