PostgreSQL Monitoring Is Broken: Too Many Metrics, Not Enough Signals

A practical guide to actionable PostgreSQL alerts, latency percentiles, locks, PgBouncer, replication lag, WAL retention, and transaction ID wraparound.

The incident every on-call engineer recognizes

It's Friday evening at a streaming platform. A highly anticipated episode has just dropped. Twenty minutes later, customers begin contacting support: search takes ten seconds, the Watch button spins indefinitely, and some viewers cannot find the new episode in the catalog.

The on-call engineer opens Grafana.

Everything is green.

• PostgreSQL CPU utilization is around 33%.
• The disk is only half full.
• Health checks are passing.
• The replica is connected.
• There are roughly 300 dashboard panels, and none has raised an alert.

The graphs are not lying. They are simply answering the wrong question. The infrastructure looks healthy, but customers are waiting.

Our argument is that database monitoring is frequently designed around what is easy to measure rather than what an engineer can act on during an incident. The problem extends beyond PostgreSQL. In a February 2026 NeuBird survey of 1,039 SRE, DevOps, and operations professionals, 78% reported having experienced an incident that triggered no alert, and 57% said fewer than 30% of their alerts were actionable. This is a vendor-sponsored survey, not a neutral industry census, but it illustrates a familiar problem: too many notifications, too little useful signal.

So what should trigger a page? And what should remain on a dashboard until an engineer needs to investigate?

1. Three failures that can happen while CPU is idle

Scenario A: A migration creates a lock queue

At 8:55 p.m., someone deploys an ALTER TABLE migration to add a column to the catalog table. The operation needs a strong table lock. But a long-running analytical report is still reading that table.

The migration waits for the report. New queries, including ordinary SELECT statements from the search service, can queue behind the waiting migration. They are not performing expensive computation. They are waiting for locks.

Customer symptom: Search and playback-related operations become slow.

What the CPU dashboard shows: Plenty of free CPU.

Scenario B: PgBouncer has exhausted its server connections

The application connects through PgBouncer. All 50 server connections are occupied by slow queries or transactions that have been left open. New client requests wait in PgBouncer before reaching PostgreSQL.

Customer symptom: Requests stall while acquiring a database connection.

What PostgreSQL may show: Modest CPU usage and no dramatic increase in active work.

Scenario C: A read replica is stale

The catalog reads from a replica, but replay has fallen behind. The new episode exists on the primary and has not yet appeared on the read replica.

Customer symptom: Users see outdated content even though requests complete successfully.

What ordinary health checks show: The database responds, and nothing looks obviously broken.

All three cases are diagnosable. The harder question is whether you should page on database internals, user-visible symptoms, or both.

2. Two competing alerting philosophies

Approach A: Page on database state

This approach assumes that PostgreSQL often provides warning signs before users notice a problem. Examples include:

1. Connection counts approaching a limit.
2. Replication lag increasing.
3. Old transactions remaining open.
4. WAL files accumulating because a replication slot is inactive.
5. Free disk capacity falling.

The advantage is early warning. The weakness is poor threshold design.

An alert such as "connections exceed 80%" may indicate routine operation for one database and an imminent incident for another. Reusing generic thresholds across every cluster produces noise, and people eventually learn to ignore it.

The stronger version of this approach uses database-specific baselines and rates of change. For example: "This replication slot is retaining 40 GB of WAL, increasing by 5 GB per hour." That describes a concrete failure mode and gives the operator useful context.

Approach B: Page on user-visible symptoms

This approach follows the reasoning in Google's Site Reliability Engineering book: monitoring should distinguish what is broken from why it is broken.

Google's four golden signals are latency, traffic, errors, and saturation. For a PostgreSQL-backed service, the most important paging signals might be:

• p99 latency of critical user operations.
• Error rate of those operations.
• Time spent waiting for a database connection.
• Failed or stale reads that directly affect users.

Locks, checkpoints, disk statistics, and internal session counts are then diagnostic signals. Engineers open them to explain a symptom instead of receiving a page for every fluctuation.

This approach reduces noise, but it has limits. Instrumenting every service takes work, and some dangerous database conditions grow silently for days or weeks. A symptom-only policy could discover an exhausted resource only when PostgreSQL begins rejecting writes.

Our position: Page on user impact, with a small and deliberate set of predictive alerts for irreversible or rapidly escalating database risks.

3. Why averages and even p95 can hide an incident

Imagine 1,000 requests to the Watch operation during one minute:

• 980 complete in 20 ms.
• 20 wait on a lock and complete in 3 seconds.

The average is:

(980 × 20 ms + 20 × 3000 ms) / 1000 = 79.6 ms

That appears healthy against a 200 ms average-latency threshold.

The p95 is also 20 ms. But p99 is approximately 3 seconds. Two percent of viewers are experiencing a three-second delay.

This is why latency distributions matter. But they must be interpreted correctly:

1. Measure each critical operation independently. Fast search requests can conceal slow playback operations in an aggregate metric.
2. Display request counts beside percentiles. Percentiles calculated from five requests per hour are not particularly informative.
3. Separate connection acquisition from query execution. A slow request may never have reached PostgreSQL.
4. Use stable histograms. Application-side Prometheus histograms are a common way to aggregate and analyze request latency.

Do not expect pg_stat_statements to provide p95 or p99 per query. It tracks counts and timing statistics such as mean, minimum, maximum, and variability, but not those latency percentiles. Instrument the application or relevant proxy layer for request-level distributions.

4. Lock contention: Find the blocker, not just the blocked

PostgreSQL uses locks constantly. An alert saying "locks exist" is therefore not useful.

You need answers to three questions:

1. Which sessions are waiting?
2. How long have they waited?
3. Which sessions are blocking them?

You can investigate using pg_stat_activity and pg_blocking_pids():

-- Find sessions waiting on locks or left idle in a transaction.
SELECT
    pid,
    now() - xact_start AS xact_age,
    now() - state_change AS in_state,
    state,
    wait_event_type,
    pg_blocking_pids(pid) AS blocked_by,
    left(query, 60) AS query
FROM pg_stat_activity
WHERE wait_event_type = 'Lock'
   OR state LIKE 'idle in transaction%'
ORDER BY xact_start;

Pay special attention to idle in transaction. A query may have finished, but the surrounding transaction remains open while the application waits for an external service. That session can retain locks and interfere with cleanup of old row versions.

Two controls are worth understanding:

• idle_in_transaction_session_timeout closes sessions that remain idle within an open transaction too long.
• transaction_timeout, available starting with PostgreSQL 17, limits total transaction duration. Apply carefully, preferably to suitable application roles, because it can also interrupt legitimate long-running work.

Make schema migrations fail quickly instead of blocking traffic

For a migration requiring a strong lock, a short lock_timeout can prevent the migration from sitting in a queue indefinitely:

BEGIN;
SET LOCAL lock_timeout = '2s';
ALTER TABLE titles ADD COLUMN age_rating text;
COMMIT;

If the lock cannot be acquired within two seconds, the operation fails and the transaction must be retried deliberately. PostgreSQL does not automatically retry the migration. The point is to reduce the risk of a waiting schema change obstructing normal traffic. For more context, see Xata's explanation of migrations and exclusive locks.

With state-based monitoring, an unusually old transaction might produce an early warning, though normal reports can also be long-running. With symptom-based monitoring, an increase in playback p99 would page the on-call engineer, who could then inspect the lock queue.

log_lock_waits is another diagnostic option. As of the PostgreSQL 19 beta series, PostgreSQL 19 also proposes expanded lock observability through pg_stat_lock; version-specific behavior should be confirmed against the PostgreSQL 19 release notes.

5. PgBouncer: When the queue is outside PostgreSQL

Database connection counts are frequently compared with max_connections. That is a useful capacity metric, but it does not capture the entire request path. PostgreSQL also reserves connection slots for privileged access, so ordinary clients may fail before total occupancy reaches 100%.

If PgBouncer sits in front of the database, inspect its own queue:

SHOW POOLS;

Two fields are particularly important in the PgBouncer documentation:

• cl_waiting: Clients waiting for an available server connection.
• maxwait: How long the oldest waiting client has waited, measured in seconds.

A pool that is 80% occupied but has no waiting clients may be healthy. A growing maxwait can indicate real user-facing delay even if the database is only moderately busy.

Do not automatically increase default_pool_size or max_connections. If slow queries and abandoned transactions already occupy connections, allowing more work to compete for CPU, RAM, and disk can make the situation worse.

A more useful alert combines connection acquisition latency, waiting-client count, and queue age with the affected application operation.

6. Replication lag: The replica can respond while returning stale data

pg_stat_replication exposes replay_lag, but this is not a universal freshness guarantee.

When the replica catches up and there are no new commits, lag measurements may eventually become NULL. If a replica disconnects entirely, its row can disappear from the primary's replication statistics. An alert that only checks whether replay_lag exceeds a threshold can miss both cases.

For critical read paths, consider measuring freshness at the application level. One approach is to update a heartbeat timestamp on the primary every few seconds and read that same row from the replica. Comparing the stored timestamp with the current time estimates how stale the replicated data is, provided the clocks are synchronized.

This tells you something closer to what a customer experiences: Can a read from the replica observe recent writes?

7. WAL retention: A failure that stays invisible until it is urgent

A replication slot tracks the WAL position still needed by its consumer. When the consumer stops advancing, the slot may keep WAL files from being removed.

An abandoned logical subscriber or disconnected standby can therefore produce sustained WAL growth. If no effective retention limit is configured, WAL can continue consuming storage until disk exhaustion threatens availability.

Inspect retention from the primary:

-- Inspect WAL retained by replication slots on the primary.
SELECT
    slot_name,
    active,
    wal_status,
    pg_size_pretty(
        pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
    ) AS retained,
    safe_wal_size
FROM pg_replication_slots;

safe_wal_size indicates how much more WAL can be written before a slot risks losing necessary files when max_slot_wal_keep_size is configured. It may be NULL depending on slot state and configuration.

PostgreSQL 18 introduced idle_replication_slot_timeout. It is disabled by default and can invalidate qualifying inactive slots after the configured duration, with invalidation applied during a checkpoint. That protects against some abandoned-slot scenarios, but it also requires operational planning because affected consumers may need their slots recreated.

Why this belongs in alerting: Users may see normal latency right up until a disk or WAL retention problem reaches a dangerous limit.

8. Transaction ID wraparound: The risk p99 cannot predict

PostgreSQL uses 32-bit transaction IDs and relies on vacuuming and freezing old transaction identifiers to avoid wraparound problems.

If that maintenance falls too far behind, PostgreSQL first emits warnings and eventually refuses commands requiring new transaction IDs to protect data integrity. The practical outcome can be a database that stops accepting writes.

Monitor the age of the oldest unfrozen transaction ID in each database:

SELECT
    datname,
    age(datfrozenxid) AS xid_age
FROM pg_database
ORDER BY xid_age DESC;

By default, autovacuum_freeze_max_age is 200 million transactions. PostgreSQL's protective limits are much farther away, but waiting for critical warnings is not an acceptable monitoring strategy. Consult the PostgreSQL documentation on routine vacuuming for the relevant thresholds and safeguards.

This is not a theoretical failure mode:

• In February 2019, Mailchimp's Mandrill service experienced a major outage after transaction ID exhaustion affected a database shard. The disruption lasted roughly 40 hours.
• Sentry documented a transaction ID wraparound incident in 2015 that disrupted its service for a substantial part of the US working day.

In Mandrill's case, engineers had identified the growing risk months earlier, but planned monitoring work remained in the backlog. The alert arrived after the incident had already been resolved.

An early XID-age warning, for example around one billion with suitable follow-up escalation, can be more useful than waiting for user-facing errors. The precise threshold must be chosen for the database's transaction rate, vacuum behavior, and operational response time.

9. Disk I/O and checkpoints: Correlate them with latency

Checkpoints write dirty data to disk and can contribute to I/O pressure, but a checkpoint should not automatically trigger a page.

Starting in PostgreSQL 16, pg_stat_io provides useful I/O statistics. Relevant timing fields require track_io_timing, which is disabled by default for most database I/O, and WAL timings use track_wal_io_timing where applicable. I/O timing can include operations served by the operating system's page cache, so use host-level tools such as iostat or node_exporter to understand physical device behavior.

Frequent checkpoints caused by WAL pressure can create log warnings. PostgreSQL's WAL configuration documentation cautions against overreacting to a single warning; repeated warnings combined with rising write latency deserve investigation.

Do not blindly increase max_wal_size. More WAL capacity can reduce some checkpoint pressure but also consumes disk space and can increase recovery time after a failure.

Better diagnostic question: Are I/O latency and checkpoint activity correlated with slower writes or failed user operations?

10. Comparison: State-first vs. symptom-first alerting

Question	Approach A: Database state	Approach B: User symptoms
What triggers a page?	Connection thresholds, replication, WAL, locks, capacity	p99 latency, error rates, connection wait time
When do you find out?	Potentially before users notice	Once service quality degrades
Main failure mode	Excessive noise and alert fatigue	Gradual risks can stay invisible too long
Where is it strongest?	WAL retention, XID age, replication-slot health	Lock queues, pool waits, slow operations
What is required?	PostgreSQL exporters and meaningful per-cluster thresholds	Application instrumentation, histograms, agreed service objectives

Neither strategy is sufficient on its own.

11. The alerting policy we would actually use

Our preferred model is symptom-first paging with a small set of high-consequence predictive exceptions.

Page when users are affected

1. p99 latency of a critical operation exceeds its objective for a meaningful interval and volume.
2. The error rate of a critical operation increases.
3. Connection acquisition time or PgBouncer queue age affects requests.
4. Replica freshness falls behind an application's accepted limit.

Page before a silent resource failure

1. WAL retention is increasing toward available disk or configured limits.
2. A replication slot has stopped advancing and creates a credible retention risk.
3. Transaction ID age is approaching an escalation threshold.
4. Critical disk headroom is projected to run out before a human can safely respond.

Keep on dashboards unless correlated with risk or impact

• Ordinary lock counts.
• Raw CPU utilization.
• Connection occupancy without a waiting queue.
• Routine checkpoints.
• Normal variations in disk I/O.

Make every page actionable

Compare these notifications.

Poor alert:

PostgreSQL active connections: 80%.

Useful alert:

Watch operation p99 is 4.2 seconds, compared with a normal 300 ms. Thirty-seven sessions are waiting on titles. The queued ALTER TABLE (PID 81234) is blocked by an analytical report (PID 80112).

The second tells an engineer what customers are experiencing, how many sessions are affected, and where to investigate first.

For each proposed page, ask a simple question: If this reaches someone at 3 a.m., is there a clear action to take?

If the answer is only "look at it and close it," it probably belongs on a dashboard instead.

There is one important exception to the instrumentation ideal. If you operate an old monolith and cannot safely add service-level metrics, state-based monitoring is still better than no monitoring. Just tune its thresholds to the actual database, not to a generic configuration copied from the internet.

Conclusion: Monitor outcomes, investigate causes

A healthy CPU graph does not prove that PostgreSQL is serving users well.

A full dashboard does not prove that your monitoring is effective.

The useful distinction is between signals that should wake someone up and metrics that help that person investigate.

Start with customer-facing latency, errors, and connection waits. Add carefully chosen predictive alerts for replication, WAL, XID wraparound, and genuine capacity exhaustion. Keep the rest available for diagnosis.

Fewer pages. Better context. Faster decisions.

Over to you

What PostgreSQL alert last prevented a real outage in your environment? And which alert has been firing for months without anyone taking action?

If you prefer state-based paging, what incident convinced you it was better than a symptom-first approach?