While building the webhook delivery system, my API could only handle around 350 requests per second. The architecture looked fine, Redis wasn't the bottleneck and the workers were running normally. After profiling the system, I found the real bottleneck wasn't where I expected, it was the way I was writing events to PostgreSQL. A single architectural change increased throughput to around 1300 req/sec while reducing latency by more than 70%.
The original design
At the time, the platform already used an asynchronous architecture. Incoming events were accepted by the API, pushed to Redis buffer, and processed by an Ingestion worker in the background.
The ingestion worker continuously consumed events from Redis and persisted them into PostgreSQL before passing them to the rest of the delivery pipeline.
The workflow looked like this:
Why it became slow?
For each of the events, the Ingestion worker was inserting an event to PostgreSQL which is fine for a small number of operations. Even though PostgreSQL is fast, when the number of DB operations becomes thousands then this tiny database write operation becomes expensive.
At first, this doesn't seem like a problem. PostgreSQL is highly optimized and can process thousands of queries per second. However, the issue isn't the INSERT statement itself, it's everything that happens around it.
For every event, PostgreSQL has to:
Parse and execute the SQL statement.
Allocate a new transaction.
Write changes to the Write-Ahead Log (WAL).
Commit the transaction.
Send a response back to the application.
Now imagine processing 1,000 events.
Instead of writing data once, the database repeats this entire process 1,000 separate times.
Event 1 -> INSERT -> COMMIT
Event 2 -> INSERT -> COMMIT
Event 3 -> INSERT -> COMMIT
...
Event 1000 -> INSERT -> COMMIT
Although each operation is relatively inexpensive, the overhead accumulates quickly under sustained traffic. The database spends a significant amount of time managing thousands of small transactions instead of efficiently writing large batches of data.
As the event rate increased, the Ingestion worker couldn't persist events as quickly as they arrived. The Redis buffer continued to grow, throughput plateaued at around 342 requests per second, and overall system latency increased.
The architecture itself wasn't the problem. The bottleneck was the granularity of the database writes.
At that point, the next optimization became clear: reduce the number of database transactions instead of trying to make each transaction faster.
The Idea: Batch Everything
Why was I forcing PostgreSQL to process thousands of small transactions when it could process a few large ones instead?
Instead of writing every event individually, I decided to accumulate multiple events in memory and persist them together.
Redis
↓
Pop 100 Events
↓
INSERT
(Event 1)
(Event 2)
(Event 3)
...
(Event 100)
↓
Single COMMIT
With this approach, the database performs one transaction instead of one hundred.
The amount of data being written remains exactly the same, but the transaction overhead is dramatically reduced. Rather than repeatedly opening and committing transactions, PostgreSQL spends more of its time doing what it's optimized for: writing data efficiently.
To implement this, the Ingestion worker continuously collects events from Redis and flushes them in batches. A batch is written when either:
A configured batch size is reached (100 events), or
A short timeout expires, ensuring that events are not delayed during periods of low traffic.
This approach provides the best of both worlds. During traffic spikes, batches fill quickly, maximizing throughput. During quieter periods, the timeout ensures that events are still persisted promptly without waiting indefinitely for a full batch.
The architecture itself didn't change, the API still pushes events into Redis, and the Ingestion worker still persists them to PostgreSQL. The only difference was how the worker wrote the data.
"Sometimes, the biggest performance improvements don't come from introducing new technologies. They come from using existing ones more efficiently."
The New Architecture
The overall architecture remained exactly the same. The only change was inside the Ingestion worker.
Instead of processing one event at a time, the worker now consumes multiple events from Redis and persists them to PostgreSQL using a single bulk insert.
This small implementation change significantly reduced database transaction overhead while keeping the rest of the system unchanged.
The updated ingestion pipeline now looks like this:
One Challenge with Batch Processing
Batching significantly improved throughput, but like most optimizations, it introduced a new trade-off.
PostgreSQL executes a batch insert as a single transaction. If any row in the batch violates a database constraint for example, a NOT NULL constraint, a UNIQUE constraint, or a foreign key constraint, the entire transaction is rolled back.
That means if a batch contains 100 events and just one of them is invalid, none of the events are persisted.
Batch of 100 Events
│
▼
INSERT INTO events (...)
│
▼
Event #47 violates a database constraint
│
▼
Entire transaction is rolled back
This is the trade-off with batching: you gain much higher throughput, but you lose fault isolation within a batch.
There are several ways a production system can handle this:
Validate events before they reach the database.
Retry failed batches by splitting them into smaller batches to isolate the problematic event.
Move invalid events to a dead-letter queue for later inspection.
Use features such as
ON CONFLICTwhere duplicate records are expected.
In HookRelay, this wasn't a major concern because every event had already passed API-level validation before entering the ingestion pipeline. Even so, it's an important consideration whenever batch processing is introduced into a production system.
Conclusion
This optimization taught me an important lesson: performance problems aren't always solved by adding new technologies or redesigning an entire system. Sometimes, the biggest improvements come from identifying a single bottleneck and removing it.
In my case, the architecture was already solid. The problem wasn't Redis, the API, or PostgreSQL itself, it was the cost of performing thousands of tiny database transactions. By simply batching writes, throughput increased from 342 requests/sec to 1,259 requests/sec, while latency dropped significantly.
This is just one of several optimizations that went into building HookRelay. In future articles, I'll share other improvements, such as Redis authentication caching, per-tenant queues, circuit breakers, and idempotency, that made the system more scalable and reliable.
If you're building systems that process high volumes of events, I hope this article helps you avoid the same bottleneck and encourages you to measure first, then optimize where it actually matters.
Top comments (0)