At 4:30 PM every Friday, thousands of field workers on remote construction sites log off and submit their weekly hours, safety checklists, and equipment reports at the exact same time.
A few years ago, our reporting system crashed under this predictable end-of-week surge. When thousands of employees press save simultaneously, a traditional database attempts to process every incoming write request at once while also serving complex read requests to project managers pulling live site updates. The server memory fills up, database locks pile up, connections time out, and workers are left staring at error messages.
To fix this, we redesigned the platform around Command Query Responsibility Segregation, or CQRS. This is a software design pattern where you separate the code that updates information from the code that reads it. Think of it like a high-volume bakery. Instead of having one person take an order, walk to the back to bake the bread, and hand it to the customer, you split the operations entirely. One counter team takes order forms and drops them into a tray, while a specialized baking team pulls forms from the tray and fills them at a steady pace.
In our system, when a field worker submits a weekly log, the mobile app no longer writes directly to the primary report database. Instead, it sends a quick message into Azure Service Bus, a cloud messaging service that acts like a digital holding area, storing incoming requests safely until server resources are free to handle them. Independent background workers pull those messages from the queue, run validation logic, and write them into the database. Project managers read their reports from a pre-built data cache optimized purely for fast viewing.
This redesign eliminated our end-of-week downtime. The queue absorbed the sudden spike, spreading five minutes of heavy traffic over a manageable fifteen-minute background processing window.
However, every system decision involves a compromise. Our trade-off was accepting eventual consistency, which is a state where data on the screen takes a short period of time to reflect recent changes made in the system.
We underestimated the human impact of this delay. Project managers refreshed their screens immediately after a field team reported finishing, saw empty dashboards, and assumed the software failed. Site leads began resubmitting their hours four or five times in frustration, which created duplicate records and lengthened queue processing times even further.
We resolved this by adding unique tracking IDs to block duplicate submissions and updating the web interface to show a processing status indicator rather than a blank screen. The technical takeaway was clear. Splitting read and write paths solves hard database scaling limits, but it forces you to design carefully for the human element when real-time updates are no longer guaranteed.
When your team shifts to asynchronous background processing, what strategies do you use to keep end users from feeling lost during processing delays?
#architecture #azure #microservices #dotnet
Top comments (0)