A Successful Payment That Never Became a Booking
A customer completed their payment.
Razorpay showed Payment Successful.
The webhook returned HTTP 200.
The payment was captured.
And the booking sat there spinning on:
Still confirming...
Nothing had errored.
No exception.
No failed request.
No alert.
By every individual metric, the system was healthy.
Yet the one thing that mattered hadn't happened.
The customer had paid.
The creator had no booking.
This is the story of that bug, the investigation behind it, and the architecture we built so it can never happen again.
The lesson underneath all of it is deceptively simple:
Accepting money is easy. Guaranteeing that every successful payment becomes a confirmed booking is the actual problem.
What CreatorOS Is, and Why This Matters
CreatorOS is a booking and payments platform for independent creators, coaches, tutors, consultants, and freelancers.
A typical flow looks like this:
Client
↓
Select Slot
↓
Pay via UPI
↓
Booking Confirmed
Money moves through Razorpay.
Bookings live in our database.
When payments are part of the product, "mostly works" is not good enough.
A payment that vanishes into a stuck booking isn't just a bug.
It's:
- A customer who paid and received nothing
- A creator who appears unreliable
- A loss of trust
And trust is the entire business.
The goal was never:
Process payments.
The real goal was:
Every captured payment must eventually become a confirmed booking.
The Naive Architecture (What Most Tutorials Teach)
Most payment tutorials teach something like this:
flowchart TD
A[Razorpay]
B[Webhook]
C[Update Booking]
A --> B
B --> C
The webhook receives an event and immediately updates the booking.
Simple.
Clean.
Dangerous.
This architecture quietly carries every failure mode that matters:
- Webhook retries
- Duplicate deliveries
- Partial failures
- Race conditions
- No recovery mechanism
The moment business logic lives inside a webhook handler, correctness becomes dependent on delivery success.
That is a fragile system.
Moving To Event Sourcing
The first major architectural decision was:
Webhooks should record events, not perform business logic.
Instead of updating bookings directly, we introduced an event ledger.
flowchart TD
A[Razorpay Webhook]
B[payment_events]
C[Processor]
D[Bookings]
A --> B
B --> C
C --> D
Three tables became the foundation of the system:
| Table | Purpose |
|---|---|
| payment_orders | Provider truth |
| payment_events | Immutable event ledger |
| bookings | Business truth |
This design gives us:
Durability
Events are safely stored before processing.
Auditability
Every state transition can be traced.
Replayability
Reprocessing means replaying events.
Idempotency
Duplicate deliveries become harmless.
The webhook now has one responsibility:
Verify Signature
↓
Store Event
↓
Return 200
Nothing else.
Two State Machines, Kept Separate
One subtle but critical decision:
Payment state and booking state are not the same thing.
Treating them as identical creates hidden bugs.
Payment State Machine
This reflects provider truth.
stateDiagram-v2
[*] --> Created
Created --> Authorized
Authorized --> Captured
Authorized --> Failed
Created --> Failed
Booking State Machine
This reflects business truth.
stateDiagram-v2
[*] --> Pending
Pending --> PaymentPending
PaymentPending --> Confirmed
PaymentPending --> Cancelled
A payment being captured is merely an input into a booking becoming confirmed.
They are related.
They are not identical.
Keeping them separate makes reconciliation possible.
The Incident
Then it happened.
A customer paid.
Razorpay showed success.
The webhook arrived.
The event existed in the database.
And the booking remained:
payment_pending
The first query told the story:
SELECT event_type, processed
FROM payment_events;
Output:
payment.captured false
order.paid false
The events existed.
They simply had not been processed.
Everything looked healthy:
- Razorpay worked
- Webhook worked
- Database worked
Yet bookings did not.
The Investigation
We traced a single booking through the system.
Using its correlation ID we followed the trail across:
- payment_events
- payment_orders
- bookings
Every row looked correct.
Except for one field:
processed = false
Repeated everywhere.
So we manually invoked the processor:
curl \
-H "Authorization: Bearer xxx" \
https://creator-os.vercel.app/api/cron/process-events
Response:
{
"processed": 15
}
Immediately:
- Events became processed
- Orders became captured
- Bookings became confirmed
The processor wasn't broken.
It worked perfectly.
The moment it ran.
Root Cause: The Scheduler That Didn't Exist
The investigation ultimately revealed something surprisingly simple.
Webhook
↓
Event Stored
↓
Processor Exists
↓
Never Runs
Nobody was invoking:
processPendingEvents()
The processor was healthy.
The webhook was healthy.
The database was healthy.
The scheduler was missing.
This is the hidden cost of event-driven architecture.
Decoupling ingestion from processing is the right design.
But it creates a new dependency:
Something must reliably trigger processing.
Without that trigger, events accumulate forever.
Building Recovery Mechanisms
Adding a cron job wasn't enough.
The goal was:
Even if a webhook fails, the system must eventually recover.
Scheduled Processing
A scheduler drives several independent jobs.
*/5 * * * *
process-events
Processes payment events.
reconcile
Queries provider truth and recovers missed webhooks.
integrity
Validates system invariants.
Idempotency Everywhere
Retries should never create incorrect state.
The processor claims work using:
SELECT *
FROM payment_events
WHERE processed = false
ORDER BY created_at
FOR UPDATE SKIP LOCKED;
This guarantees:
- Multiple workers are safe
- Retries are safe
- Duplicate deliveries are safe
Whether one processor runs or one hundred, results remain correct.
Proving It With CI
A payment system you cannot test is a payment system you cannot trust.
We built CI that spins up a real Postgres instance.
flowchart TD
A[GitHub Actions]
B[Postgres Container]
C[Test Suite]
D[117 Passing Tests]
A --> B
B --> C
C --> D
The tests verify:
- Duplicate webhook deliveries
- Event replay
- Concurrent processors
- Reconciliation recovery
- Booking confirmation invariants
Not just the happy path.
The guarantees.
Final Production Architecture
The architecture follows a simple principle:
- Webhooks record
- Events persist
- Processors transform
- Reconciliation guarantees
- Schedulers trigger
- Idempotency protects
What It Taught Us
A Successful Payment Is Not The Same Thing As A Successful Booking
They are different facts.
The gap between them is where customers get hurt.
Webhooks Should Record Events, Not Perform Business Logic
Business logic belongs in processors.
Webhooks should be fast, durable, and boring.
Every Critical Workflow Needs A Recovery Path
If your system only works when every webhook arrives on time:
It doesn't actually work.
It just hasn't failed yet.
Schedulers Are Production Infrastructure
A queue that nobody drains is just a place where work waits forever.
The trigger is as important as the processor itself.
Reliability Is A Feature
Customers never notice reliability when it exists.
They immediately notice when it doesn't.
Closing Thoughts
The hardest part of payments isn't collecting money.
Stripe, Razorpay, PayPal, and countless providers already solved that problem.
The hard part is guaranteeing that successful payments eventually become business outcomes.
That means:
- Durable events
- Idempotent processing
- Reconciliation
- Recovery paths
- Operational visibility
The day we stopped thinking about payments as API calls and started thinking about them as distributed systems was the day the architecture became reliable.
And that reliability is ultimately what customers pay for.
Top comments (0)