In a local data center, business workloads naturally fluctuate. However, a cloud management platform enforced strict hardware utilization thresholds — if a server's CPU, memory, or disk usage fell too low for too long, the team would be held accountable for "wasted resources." To avoid this, the customer deployed a dummy background process that artificially consumed system resources. That process eventually starved the database nodes, leading to a severe performance meltdown. Here's the full investigation.
The Symptom
The customer reported that over 50% of critical business tasks were failing to complete on time. On the coordinator node, dozens of business SQLs were running for hundreds of seconds, and the longest INSERT SELECT had reached 10,000 seconds. Some data nodes were stuck in the flush commit phase. Normally, these queries complete in 10–60 seconds.
Investigation
1. Identify the Slow Nodes
We logged into the data nodes with the longest-running queries and checked their disks with iostat. Several nodes showed sustained 100% disk utilization with no idle periods, while other nodes fluctuated between 50–80% busy and occasional idle times.
2. Historical Resource Analysis
The sar logs told a clear story. Before February 11th, disk I/O peaked at 10–50% and had around 30% idle time. Over the following days, both the intensity and duration of the disk load increased steadily. By February 16th, the disks were running at full saturation continuously. Additionally, the node's entire 32 GB of SWAP space was exhausted, even though it had over 300 GB of physical RAM.
3. Pinpoint the Culprit Process
iotop confirmed that the gbased database process was generating the disk I/O. However, top revealed the real offender. The gbased process was using only a few hundred MB of swap, but a mysterious non‑database process XXX had consumed 30 GB of swap and 180 GB of physical memory.
Root Cause
After discussing with the customer, we learned that process XXX was a dummy workload generator deployed on every database node. Its sole purpose was to consume CPU, memory, and disk resources so that the servers would never appear idle. The servers were monitored by a cloud management platform that would flag low utilization as a hardware planning failure, for which the team would be reprimanded.
By hoarding physical memory, the dummy process forced the operating system to swap out the database's memory pages. Once swap was exhausted, disk I/O became the bottleneck, and the database performance collapsed.
Resolution
Stopping the dummy background process immediately resolved the issue. Disk I/O returned to normal levels, and all business queries dropped back to their expected 10–60 second execution times.
Lessons Learned
This is a classic case of a misguided metric causing real damage. In a pure cloud environment, resources can be scaled up and down on demand. On physical hardware, workloads have peaks and troughs. Rigidly enforcing a utilization floor — and using artificial means to meet it — will eventually destroy system stability. A gbase database cluster needs its resources to be genuinely available, not just "look busy" on a dashboard. GBASE's MPP engine is designed to use hardware efficiently when real work is being done; artificially inflating the numbers only breaks that efficiency.
Top comments (0)