When “Active” Doesn’t Mean Healthy: Debugging PostgreSQL ENOSPC Across a Multi-VM Linux Stack

Dwaragesh D — Sat, 05 Sep 2026 22:08:10 +0000

PostgreSQL was active.

Nginx was active.

PHP-FPM was active.

The application still returned:

HTTP 503

The actual root cause was a completely full filesystem backing a PostgreSQL tablespace.

I built this incident deliberately in my Ubuntu/KVM support lab to test a common troubleshooting mistake:

Treating process state as proof of service health.

The incident became a useful example of why application health, dependency health, and process health need to be investigated separately.

Lab architecture

I used three Ubuntu VMs connected through a private libvirt network:

Client
   |
   v
vm-web-01
192.168.100.10
Nginx + PHP-FPM
   |
   | TCP/5432
   v
vm-db-01
192.168.100.20
PostgreSQL 14
   |
   | node_exporter
   v
vm-monitor-01
192.168.100.30
Prometheus + Alertmanager

The database VM also had a dedicated disk mounted at:

/mnt/inc012-db

That filesystem was used for the PostgreSQL tablespace, which meant I could create storage pressure without filling the VM's root filesystem.

Establishing the baseline

Before introducing the failure, I checked the application, database, filesystem, and monitoring state.

The PostgreSQL tablespace filesystem was:

8.03% used

The application returned:

HTTP 200

Database writes succeeded.

Prometheus showed:

DBDiskNearlyFull state=inactive health=ok

This gave me a known-good state to compare against during the incident.

Monitoring detected the problem first

I then consumed storage on the dedicated database filesystem in a controlled way.

When usage reached:

88.49%

Prometheus changed the storage alert to:

DBDiskNearlyFull firing

But the application was still returning:

HTTP 200

This was one of the most important observations in the test.

The monitoring system had identified a capacity problem before the customer-facing service failed.

That is exactly what useful monitoring should do: create an intervention window before an operational condition becomes an outage.

Then the filesystem reached 100%

I continued the controlled storage consumption until the dedicated filesystem reached:

100%

The database VM's root filesystem remained healthy.

PostgreSQL also continued reporting:

active

So at a quick glance, the database server could appear healthy.

It wasn't.

Testing the operation that actually mattered

A connectivity test alone wasn't enough.

PostgreSQL could still accept connections, but when I performed a write that required additional filesystem blocks, it failed with:

ERROR: could not extend file
No space left on device
HINT: Check free disk space.

The PostgreSQL logs showed the same ENOSPC condition.

That exposed the distinction I was looking for:

PostgreSQL process health
        !=
PostgreSQL write health
        !=
Application health

A running database process does not necessarily mean the database can complete the operations the application depends on.

The user-facing failure appeared at the web tier

The actual failure originated in database storage.

But the visible symptom appeared somewhere else.

The web endpoint changed from:

HTTP 200

to:

HTTP 503
{"status":"degraded","database":"write_failed"}

At the same time:

Nginx       active
PHP-FPM     active
PostgreSQL  active

All three processes were alive while the application was degraded.

If my investigation had stopped at:

systemctl is-active nginx
systemctl is-active postgresql

I could easily have concluded that the web and database layers were fine.

They weren't.

My investigation path

I worked down the dependency chain instead:

HTTP 503
   ↓
Check Nginx and PHP-FPM
   ↓
Processes active
   ↓
Check PostgreSQL reachability
   ↓
Database reachable
   ↓
Test an actual database write
   ↓
Write fails with ENOSPC
   ↓
Inspect PostgreSQL error
   ↓
Check tablespace filesystem
   ↓
Filesystem 100% full
   ↓
Correlate with Prometheus
   ↓
Storage alert already firing

The key shift was moving from:

Is the process running?

to:

Can this dependency perform the operation the application requires?

That narrowed the failure to storage rather than networking, web-server availability, process crashes, or general VM capacity.

Recovery

Because the disk pressure had been generated intentionally for the lab, I removed the temporary filler data from the isolated database filesystem.

I did not restart PostgreSQL.

After the storage was released, filesystem usage returned to:

8.27%

PostgreSQL writes succeeded again.

The application returned:

HTTP 200

Prometheus returned the storage alert to inactive, and Alertmanager had no remaining active DBDiskNearlyFull alert.

Before, failure and recovery

Check	Baseline	Failure	Recovery
DB filesystem	8.03%	100%	8.27%
PostgreSQL process	Active	Active	Active
PostgreSQL writes	Successful	ENOSPC	Successful
Application	HTTP 200	HTTP 503	HTTP 200
Prometheus alert	Inactive	Firing	Inactive
Alertmanager	Clear	Active	Clear

The most interesting line is still:

PostgreSQL process
Active → Active → Active

The process never stopped.

The service still failed from the application's perspective.

What I took away from the incident

1. Process status is only one health signal

systemctl is-active tells me whether systemd considers a process active.

It does not prove that the service can perform useful work.

2. Check the dependency the application actually uses

For a PostgreSQL-backed application, TCP connectivity or pg_isready is useful, but it may not be enough.

The application needed database writes.

So write capability was the meaningful test during this incident.

3. Monitoring should create time to respond

The alert fired at 88.49% while the application was still healthy.

That gap between warning and outage is operationally valuable.

Capacity alerts should give engineers enough time to investigate and remediate before the filesystem reaches exhaustion.

4. Validate recovery through the entire stack

Freeing disk space was not enough for me to call the incident resolved.

I checked:

Filesystem capacity
→ PostgreSQL write
→ application HTTP response
→ Prometheus state
→ Alertmanager state

Only after all of those recovered did I consider the incident validated.

5. Restarting isn't automatically the fix

PostgreSQL did not need a restart.

The process wasn't broken.

Its storage dependency was.

Restarting it would have treated the symptom instead of the root cause.

Commands I would use during a similar investigation

Check filesystem block capacity:

df -hT /mnt/inc012-db

Check inode capacity:

df -i /mnt/inc012-db

Find large consumers:

sudo du -xhd1 /mnt/inc012-db | sort -h

Check PostgreSQL process state:

systemctl is-active postgresql

Check PostgreSQL connectivity:

pg_isready

But after those checks, I would still test the actual database operation required by the application.

Final takeaway

The entire incident can be summarized as:

Prometheus warned first.
PostgreSQL stayed active.
Database writes failed.
The application returned HTTP 503.
The real root cause was exhausted database storage.

The biggest lesson for me was simple:

“The service is running” and “the service is healthy” are not the same statement.

This experiment was performed entirely in an isolated Ubuntu/KVM lab using controlled fault injection and manual troubleshooting.

Full lab and incident report

The full INC012 incident report and the rest of my 16-incident Ubuntu/KVM support lab are available here:

View the Ubuntu/KVM support lab on GitHub

Read the full INC012 incident report

DEV Community: Dwaragesh D