Tansa is a Rails application that transforms Bitcoin data into structured, verifiable information.
Its processing pipeline uses PostgreSQL and Sidekiq:
Bitcoin Core → Layer1 → Cluster → ActorProfile → ActorBehavior → ActorLabels
When one module started falling behind, Rails and Sidekiq looked like the obvious suspects. The processing ran through background jobs, with several large writes going through Active Record.
We could have immediately added more workers.
That would probably have made the problem worse.
What the Measurements Revealed
We waited for the query responsible for inserting data into cluster_inputs, then observed its behavior inside PostgreSQL.
One execution took 114.39 seconds.
During our sampling, almost all the observed wait time was:
IO/DataFileRead
The query accessed approximately:
- 55 MB from storage
- 1.6 GB from PostgreSQL’s shared cache
The Rails job was running. Sidekiq was doing exactly what we had asked it to do. Ruby CPU execution was not the main issue.
The pipeline’s throughput was being limited by PostgreSQL reads and the characteristics of the underlying storage.
Why More Workers Would Have Been Dangerous
An additional worker would not have made the disk faster.
It would have sent more concurrent queries to an already constrained database, increased contention, and made the problem harder to diagnose.
We focused on the actual constraint instead:
- measuring the exact SQL query;
- inspecting PostgreSQL wait events;
- increasing
shared_buffersfrom 4 GB to 8 GB; - improving the insertion and flushing path;
- controlling the number of concurrent jobs.
During an initial tuning phase, one representative insertion dropped from 252 to 156 seconds. After further improvements, another measured execution completed in 114 seconds.
These are production observations, not controlled laboratory benchmarks. But they gave us the evidence needed to move in the right direction.
The Real Lesson
Saying that a Rails application is slow is not a diagnosis.
Rails was orchestrating the pipeline correctly. Sidekiq was executing the jobs correctly. But PostgreSQL and the storage layer determined the actual throughput.
Before adding workers, replacing a framework, or rewriting a service, ask a much simpler question:
What is the system actually waiting for?
In our case, the answer was not Ruby.
It was the disk.
I document these real engineering decisions and incidents in Building Tansa with AI, a book about the architecture of an application built with artificial intelligence.
Have you ever discovered that a performance problem blamed on your framework was actually caused by the database or storage layer?
Top comments (0)