Building a Payment Processor: My First Attempt Was a Disaster

I wanted to figure out how people build payment systems without losing everyone's money. It turns out, my first attempt was a great way to lose a lot of it.

I started with what felt like a simple Go service. One endpoint, one database table, and a third-party provider to handle the actual charging.

The plan was straightforward:

1. Decode the request.
2. Call the provider to charge the user.
3. Save the result to my database.
4. Respond with a 201.

It looked like this:

func (s Server) createCharge(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    var req createChargeRequest
    if err := decodeJSON(r, &req); err != nil {
        writeError(w, http.StatusBadRequest, err.Error())
        return
    }

    // 1. Charge the user
    if err := s.paymentProvider.Charge(ctx, provider.CreateCharge{
        Amount:    req.Amount,
        Currency:  req.Currency,
        Reference: req.Reference,
    }); err != nil {
        writeError(w, http.StatusBadRequest, err.Error())
        return
    }

    // 2. Record it in the DB
    c, err := s.store.CreatePayment(ctx, repository.CreatePaymentParams{
        Amount:    req.Amount,
        Currency:  req.Currency,
        Reference: req.Reference,
        Status:    "completed",
    })
    if err != nil {
        writeError(w, http.StatusBadRequest, err.Error())
        return
    }

    writeJSON(w, http.StatusCreated, ApiResponse{Data: c})
}

On paper, it works. I ran a "happy path" test and it passed. But as soon as I started thinking about network hiccups or a user clicking "Pay" twice, I realized I had built a liability.

The Double Charge Problem

If a user hits the Pay button twice, my API receives two identical requests. My code doesn't check if the reference already exists; it just calls the provider again.

1. Request A → Provider charges → DB write.
2. Request B → Provider charges again → DB write.

The user is out $100 twice, and I only have one "completed" record in my DB because the second write fails on a unique constraint. I took their money, but I have zero record of the second charge.

The Ghost Charge

This one is even more frustrating. What if the provider is slow? Say they take 5 seconds to respond, but the client (or a proxy) times out after 1 second and retries.

My server is still processing that first request. The provider eventually finishes the charge successfully. But when my server tries to save the result, the context has already been canceled by the timeout. The DB write fails.

The client retries, and the provider charges them again. Now I have two charges at the provider, but zero successful records in my database. The money is gone, and my system has no idea why.

Why this is hard

I realised that the core issue is that I’m trying to keep two independent systems i.e my database and the payment provider in sync. They aren't in a single transaction. If one fails after the other succeeds, I'm stuck in an inconsistent state.

In the next post, I’m going to try to fix this by introducing idempotency keys and moving the actual charging logic into a background worker.

The code for this version is here: github.com/oreoluwa-bs/dinero/tree/naive-approach.