Five signals, five short tests, and one rule for deciding when the system really needs architectural change.
A customer reaches checkout in an online store. The site calculates delivery, requests an inventory reservation in 1C, authorizes payment through YooKassa, and records each step of the process. The journey can look healthy for a long time: pages open, orders go through, and releases continue.
Yet growth can already be constrained before a visible outage. A slow tail of requests starts to affect one journey. Background work does not catch up after a peak. More instances barely move throughput. An external dependency sets the pace. Or a shared fix must travel through several release paths.
Those are signals to investigate, not proof that a system needs to be rebuilt.
At Qtim, we start with a specific user journey and a short experiment. The point is to identify the constraint that is actually limiting growth before a costly architecture programme begins.
The average is calm while p99 breaks the journey
Average response time smooths out the requests that take much longer than the rest. p99 marks the value below which 99% of measurements fall, leaving the slowest one percent beyond it. For a journey that combines several calls, that tail can become a normal user experience even when the average still looks reassuring.
Google’s SRE guidance groups latency with traffic, errors, and saturation. Read those signals together. A rising p99 can indicate an approaching limit while the average remains stable. It also matters to separate successful and failed requests: a fast error may improve an average while ending the user journey.
There is no universal p99 threshold. The boundary depends on the journey and its service objective.
Short test: step-load one defined user journey. Record p95 and p99 alongside errors, traffic, and saturation. Once the tail diverges from the average, profile the code and database queries, then inspect cache and connection limits.
A queue is still growing after the peak is over
Queue depth tells you how much work is waiting. It does not tell you whether the system will recover in time for the business process. Compare backlog, the age of the oldest message, and arrival and processing rates for equivalent task types and load windows.
The more useful question is whether the queue returns to its baseline after a comparable peak. If the oldest message keeps aging and processing falls behind arrivals, capacity and demand are out of balance. A nightly catalogue import and an order-processing queue need different tolerances, so message count alone is not a decision rule.
In the HOTZ Global project, BullMQ runs catalogue imports, 1C synchronisation, image processing, and product-feed rebuilds outside user requests. That illustrates how background work can be isolated; it does not claim that the project had a backlog incident.
Short test: compare recovery after equivalent peaks. Then test separate workers, concurrency limits, priorities, and backpressure against the acceptable business delay.
Another instance barely moves throughput
Adding application instances helps only while work can move between them and the shared dependencies still have headroom. A stateless component can handle the next request on any instance. Session affinity, local state, a database lock, a connection pool, or an external API can all remain the common limit.
That is why scale-out should be tested as a curve, not accepted as a promise. Microsoft’s scale-out guidance makes the same point: additional instances do not guarantee proportional capacity.
Short test: use the same request profile at several fixed instance counts. Compare total and per-instance throughput, database connections, lock waits, and read/write behaviour. A flat curve tells you where to look next; it does not name the cause on its own.
One dependency sets the pace of the whole checkout
An external dependency belongs on the critical path when the next business step waits for its response. In the HOTZ Global checkout, delivery is calculated through CDEK, inventory is reserved in 1C, and payment is authorized through YooKassa. The system keeps a journal of reservation and payment states.
This route shows how one user action spans several systems. It does not mean that any of those services was slow or failed in that project. Distributed tracing is what separates the path into operations and shows the time spent at each boundary.
Test time-outs, retries, and unfinished operations together. A retry can be safe only when it does not create a second business effect. Idempotency supports that safety. A circuit breaker limits repeated calls to an unstable dependency; it does not remove the cause of the instability. A queue can move the continuation of an operation out of the user request.
Short test: inject controlled latency or failure in a test environment. Verify that the user-facing request completes predictably, no second order or payment appears, and the unfinished operation is visible to the team.
One fix now has four release paths
The fifth signal does not appear on an infrastructure dashboard. It appears in change history: one product fix touches several modules or repositories, repeats the same checks, and waits for several releases.
For the B2B catalogues of Teplomash, Sonniger, Volcano, and WDB, Qtim used an architecture reused across four independent environments. Each has its own frontend, backend, database, admin area, and release; the code is split across eight repositories. That separation enables independent releases and isolates one brand’s failure from the others. The trade-off is clear when a common change must be applied and verified in every environment.
This is not an argument to merge every system. Independent boundaries can be the right choice when brands release at different rhythms or need separate failure isolation. The useful test is whether the component boundary matches a real responsibility and a real rhythm of change.
Short test: trace one recent cross-cutting change from decision to production. Count touched modules and repositories, test cycles, release gates, and waiting time.
Rewrite only after the constraint survives a short test
The diagnostic matrix below links each symptom to a measurement and an experiment. It does not prescribe an architecture.
The first solution may be local: a database index, separate queue workers, a concurrency limit, a cache, or isolation of one integration. A larger architectural change becomes justified when the measured constraint sits in the system boundary itself and a local fix preserves the same ceiling.
Change the system boundaries only when a measured constraint sits in those boundaries and a local fix preserves the same ceiling.
If you have one critical user journey, its current metrics, and a near-term growth scenario, our product support and development team can help turn them into a diagnostic and experiment plan.
Top comments (0)