DEV Community: Uptime Architect

Oracle Wait Events, Decoded: The Half-Dozen

Uptime Architect — Mon, 22 Jun 2026 16:09:49 +0000

A wait event is not a mystery. It's just a label for time a session spent not on CPU — blocked, waiting for a single block to come back from disk, for a commit to flush, for another session to let go of a lock. That's all it is. Oracle gives each kind of waiting a name, sums the time, and ranks it. The skill isn't memorizing the catalog — Oracle ships hundreds of events — it's knowing the half-dozen that actually show up and what each one is telling you to go fix.

So ignore the catalog. On a real database the time piles up behind a handful of events: single-block reads, multi-block scans, direct path reads, the commit wait, and a small cluster of concurrency waits. Learn those, learn to rank them by DB time instead of by raw count, and learn which ones to throw away entirely — and you can read almost any performance problem from the wait interface alone. This targets Oracle 19c, with notes on 23ai/26ai, and there's a free lab so you can induce each event and read its signature yourself.

The short version. A wait event is time a session spent off-CPU, waiting on a resource. DB time is CPU plus all non-idle waits; rank events by their share of it, not by how many times they fired. Six events carry most real problems: db file sequential read (index/rowid single-block I/O), db file scattered read (multi-block scans), direct path read (scans/sorts bypassing the cache), log file sync (commit), and the concurrency pair buffer busy waits / enq: TX - row lock contention. Idle waits like SQL*Net message from client top the raw counters and mean nothing — the database is waiting for you.

The wait interface, and why DB time is the ruler

Every session is, at any instant, in one of two states: running on a CPU, or waiting on a named event. Oracle records both. DB time is the sum across all foreground sessions of CPU time plus non-idle wait time — the total time the database spent doing user work. Because it sums sessions in parallel, it can exceed wall-clock: a 5-minute interval with 4 active sessions can show 20 minutes of DB time. Divide DB time by elapsed time and you get Average Active Sessions (AAS), the single most useful load number there is. AAS near your core count means CPU saturation; AAS dominated by one wait class points straight at the resource behind it.

The wait interface exposes this at four zoom levels:

View	Scope	Use it for
`v$session`	One row per session, right now	What's a session waiting on this second (`event`, `wait_class`, `state`, `seconds_in_wait`, `blocking_session`)
`v$session_event`	Cumulative per session since it started	One session's wait history — `total_waits`, `time_waited`, `average_wait`
`v$system_event`	Cumulative instance-wide since startup	The whole instance, joined to `v$event_name` for `wait_class`
`v$active_session_history` (ASH)	Active sessions sampled once per second	Attributing waits to a `sql_id`, object, or moment in time

The crucial discipline lives in how you rank. Oracle's Top 10 Foreground Events section of an AWR report (older reports: "Top 5 Timed Events") ranks events — plus DB CPU — by total time consumed and by % of DB time, with idle events already stripped out. So the top rows are your bottleneck. ASH gives you the same view live: because it samples active sessions every second, count(*) over a window approximates seconds of DB time, and grouping by wait_class or sql_id yields the breakdown.

And this is where most wait tuning goes wrong before it starts: the idle vs non-idle distinction. Oracle classifies every event into a wait class — User I/O, Commit, Concurrency, Configuration, Application, System I/O, Network, Cluster, and the rest — plus one class you must ignore: Idle. SQL*Net message from client, rdbms ipc message, the pmon/smon timers — these top the raw v$session_event totals on every database, because the server spends most of its life waiting for the next request. They are the database waiting for you, not a bottleneck. Oracle states it plainly: idle events "should be ignored when tuning, because they do not indicate the nature of the performance bottleneck." Rank by time, exclude idle, and the noise falls away.

The half-dozen that carry real problems

You don't need the catalog. You need these six. Each row tells you what the session is actually blocked on, the cause you should suspect first, and the first move that isn't a guess.

Event	What the session is waiting on	Usual cause	First move
`db file sequential read` (User I/O)	A single block (P3=1) to come back from disk — index branch/leaf or a table block by rowid	Index-driven access; normal on OLTP. A problem only when reads are too many (bad plan/stats/clustering) or too slow (storage)	Check `average_wait` and the histogram. Numerous-but-fast → tune the SQL/plan. Few-but-slow → look at storage
`db file scattered read` (User I/O)	A multi-block read into scattered cache buffers — a full scan or index fast full scan routed through the cache	A full-scan plan that probably should be an indexed access; missing index, stale stats	Decode the segment, check the plan: should this be a full scan? If not, fix the access path
`direct path read` / `… temp` (User I/O)	A read straight into the PGA, bypassing the cache — large serial/parallel scans, or sort/hash data spilled to TEMP	Adaptive serial direct read on large scans (often normal); or under-sized PGA spilling work areas to TEMP	Datafile P1 → segment scan (tune access path/DOP). Tempfile P1 → a spill (size PGA, cut the sorted/hashed rows)
`log file sync` (Commit)	LGWR to flush this session's redo and post it back after a `COMMIT`	Committing too often (row-by-row commit in a loop) far more than slow redo disk	Compare avg LFS to avg `log file parallel write`. Close → storage. LFS ≫ LFPW → batch commits / CPU
`buffer busy waits` (Concurrency)	A block another session has pinned in the cache — intra-cache contention, not disk	A hot block: concurrent inserts to the same block/segment header, a right-growing index leaf	Map the object via `ROW_WAIT_OBJ#` and P3 (block class); spread the hot block, don't blame I/O
`enq: TX - row lock contention` (Application)	A row lock held by another transaction that hasn't committed	Application design: many sessions updating the same row, long transactions, user think-time inside a lock	Walk the blocking tree (`BLOCKING_SESSION`, `v$lock`); fix the transaction, not the database

Two of those rows hide a common trap, so read them again: buffer busy waits is in the Concurrency class and read by other session (its close cousin — you want a block another session is mid-read on) is in User I/O, while enq: TX - row lock contention is in Application. The class is a hint, not a label to obsess over. What matters is the move.

The I/O events: count vs latency

The User I/O events are where beginners burn the most time, usually by blaming storage. Resist it. db file sequential read is the single-block read — the "sequential" in the name refers to walking blocks in access-path order (down an index, then to the table by rowid), not a sequential scan. On a healthy OLTP system it is the top non-idle event and that is fine. It only becomes a problem two ways, and the wait histogram tells you which: too many reads (a SQL/plan/stats/clustering problem — fix the SQL, gather stats, repair the index, raise the clustering factor) or reads that are too slow (a storage problem — look at per-read latency; as a rule of thumb a sustained average above ~10ms on spinning disk warrants a look, flash should be low single-digit ms, and you always compare against your own baseline). Numerous-but-fast and few-but-slow are opposite fixes. The classic error is dropping indexes or forcing full scans to "cure" it — that just converts it into db file scattered read and more total I/O.

db file scattered read is the multi-block read — the fingerprint of a full scan (table or index fast full scan) that Oracle chose to route through the buffer cache. Its presence on an OLTP system that should be doing small indexed lookups is a red flag for a missing index or a bad plan — a SQL problem, not a hardware one. But here's the modern wrinkle that confuses people: a large serial full scan usually does not produce scattered reads at all. Since 11g, Oracle can decide at runtime — based on internal size thresholds rather than a documented switch — to read large segments via direct path read straight into the PGA, bypassing the cache. So the absence of scattered reads on a big scan is normal, not a bug — and growing the buffer cache won't route that scan back through it. For direct path read temp, the cause is almost never slow TEMP storage; it's a work area too small for the sort or hash, spilling to disk. The cure is PGA sizing and fewer sorted/hashed rows, not faster disks.

The commit event: it's a commit count problem

log file sync is the wait a session sits in after COMMIT, while LGWR flushes its redo to the online log and posts it back. The instinct — "high log file sync means slow redo disk, buy SSD" — is the single most common wait-event misdiagnosis, and it's usually wrong. The tell is the comparison: log file sync (Commit class, a foreground wait) versus log file parallel write (System I/O class, LGWR's background write). LFPW is just the pure I/O slice; LFS is LFPW plus LGWR scheduling, queueing, and the post-back round trip. If average LFS ≈ average LFPW, the redo I/O genuinely is the bottleneck — then faster, dedicated redo storage helps. But if LFS ≫ LFPW, the disk is fine; the time is going to commit frequency (row-by-row COMMIT in a loop, each one forcing a synchronous write + post) or to LGWR being starved of CPU. The fix for the common case is free: batch the commits. Move COMMIT out of the loop, use array DML, commit per logical unit of work — routinely a 10–100x reduction with zero hardware change. Always compare the two averages before you touch storage.

The concurrency cluster: serialize on the same thing

The Concurrency family is a set of "something else has it, wait your turn" events. buffer busy waits means a session can't pin a block because another session already has it pinned mid-modification — intra-cache contention, not disk. The classic shape is a hot block: concurrent inserts to the same block or segment header, or a monotonically increasing primary key hammering the right-hand leaf of its index. Its User-I/O cousin read by other session is when you want a block another session is currently reading in — also a hot-block symptom, not a SAN problem. And latch: cache buffers chains is almost always a SQL problem wearing a latch mask: a statement doing far too many logical reads against a hot block. You don't add latches and you rarely fix it by growing the cache — you fix the SQL.

enq: TX - row lock contention is different in kind: it's an Application-class wait, pure application design. A session wants to modify a row another transaction has locked and not yet committed. No amount of database tuning fixes it — you walk the blocking tree (BLOCKING_SESSION, FINAL_BLOCKING_SESSION, v$lock with TYPE='TX'), find the row via ROW_WAIT_OBJ#/ROW_WAIT_ROW#, and fix the transaction that's holding too long. The whole cluster shares one rule: pivot from the event to the object, SQL, or blocker — never tune the latch or the wait itself in isolation.

Triage: from high DB time to the move

The path is always the same. Start at DB time, read the ranked events, find the dominant wait class, and let the class pick the move.

Wait-event triage: rank by DB time, read the top event's wait class, and let the class route you to the fix. Idle waits are excluded before you start.

What teams get wrong

Tuning idle waits. SQL*Net message from client tops the raw counters on virtually every transactional database, because the server is waiting for the client. It is not a bottleneck. Oracle says ignore idle events — so ignore them. If that wait is huge, look at the application, the network, or a slow client, not the database.
Chasing events instead of DB time. A million waits that sum to 2% of DB time are noise; one event at 60% is your headline. Rank by time consumed, never by wait count — and make sure TIMED_STATISTICS is on (it's the default) so events are ordered by time, not occurrences.
"db file sequential read is bad." It's the normal top event on healthy OLTP — index access doing its job. Don't reflexively "fix" it; check the histogram first. Too-many is a SQL/plan problem; too-slow is storage. They're opposite fixes, and dropping indexes to escape it usually makes total I/O worse.
Treating log file sync as a storage problem. It's usually a commit-count problem. Compare average LFS to average log file parallel write: only when they're close is redo I/O the culprit. When LFS ≫ LFPW, you're committing too often or starving LGWR of CPU — batching commits fixes the too-frequent case for free (CPU starvation needs CPU, not batching).

Want to see these for real? The wait-events/ lab in github.com/pyaroslav/oracle-labs induces four of these on Oracle Database Free and lets you read each signature directly in v$session_event — index access for db file sequential read, a buffered full scan for db file scattered read, a large scan that bypasses the cache for direct path read, and a row-by-row commit loop for log file sync (then move the COMMIT outside the loop and watch it collapse). Each drill flushes the cache, runs the workload, and re-queries the counter — the jump is the proof.

Where this fits

Wait events are the vocabulary; the AWR report is the sentence they form. Once you can name the half-dozen and rank them by DB time, the natural next step is reading them in context — alongside Load Profile, the SQL lists, and the segments that own the I/O. That's the cornerstone: How to Read an AWR Report Without Drowning, where DB time runs the whole report and the top event routes you to the fix. And if the top event turns out to be a gc cluster wait, you've crossed into the other half of the stack — the interconnect and the RAC vs Data Guard decision tree, where Cluster-class waits live. Same method, different resource: name the wait, rank it by time, follow it to the object.

Frequently asked questions

What is a wait event in Oracle?

A wait event is a named label for time a session spent not running on a CPU — time it was blocked waiting on a resource, such as a single block to return from disk, a commit to flush, or another session to release a lock. Oracle records the time per event so you can see where a session, or the whole instance, spent time off-CPU. DB time is the sum of CPU time plus all non-idle wait time across foreground sessions.

What are the most common Oracle wait events?

On most databases the time concentrates in a handful: db file sequential read (single-block index and rowid reads), db file scattered read (multi-block full scans through the cache), direct path read and direct path read temp (large scans and sort/hash spills bypassing the cache), log file sync (the commit wait), and the concurrency events buffer busy waits and enq: TX - row lock contention. Idle events like SQL*Net message from client top the raw counters but are not bottlenecks.

What is the difference between db file sequential read and db file scattered read?

db file sequential read is a single-block read (P3 = 1), characteristic of index access and table lookups by rowid — despite the misleading name, it is not a sequential scan. db file scattered read is a multi-block read (P3 greater than 1) used by full table scans and index fast full scans that Oracle routes through the buffer cache. Both are in the User I/O wait class; the block count is what tells them apart.

Why is SQL*Net message from client my top wait event?

Because it is an idle event. It is the database waiting for the next request from the client, so it tops the raw cumulative totals on virtually every database simply because the server spends most of its life waiting for work. It does not indicate a database bottleneck and should be ignored when tuning. If it is genuinely large and users are slow, the problem is in the application, the network, or a slow client — not the database.

Does high log file sync mean I need faster storage?

Not usually. Compare the average log file sync (a foreground commit wait) to the average log file parallel write (LGWR's background redo write). If they are close, redo I/O is the bottleneck and faster, dedicated redo storage helps. If log file sync is much larger than log file parallel write, the disk is fine — the time is going to committing too often (row-by-row COMMIT in a loop) or to LGWR being starved of CPU. The fix for the common case is batching commits, which costs nothing.

How do I find which SQL or object is causing a wait event?

Use ASH. v$active_session_history (and DBA_HIST_ACTIVE_SESS_HISTORY for history) samples active sessions once per second, so filtering by event and grouping by sql_id, current_obj#, or the P1/P2/P3 wait parameters attributes the wait to specific SQL and objects. For a live single block read, v$session exposes ROW_WAIT_OBJ# to map the object and P1/P2/P3 for the file, block, and block count. For row locks, walk the blocking tree via BLOCKING_SESSION and v$lock.

What is the difference between direct path read and db file scattered read?

Both are multi-block reads in the User I/O class, but they target different memory. db file scattered read brings blocks into the SGA buffer cache; direct path read reads straight into the session's private PGA, bypassing the cache entirely. Since 11g, Oracle decides at runtime to use direct path read for large serial scans, and typically for parallel query, so a big full scan often shows direct path read and nothing in the cache — which is normal, not a bug. direct path read temp specifically reads back sort or hash work areas that spilled to TEMP.

Should I rank wait events by number of waits or by time?

Always by time consumed, never by count. A million waits that sum to a tiny fraction of DB time are noise; one event holding a large percentage of DB time is your real problem. The AWR Top Timed Events section ranks by total time and percentage of DB time with idle events already excluded, so the top rows are your bottleneck. Make sure TIMED_STATISTICS is enabled (it is by default) so events are ordered by time rather than by occurrence count.

Originally published at uptimearchitect.com.

Data Guard Switchover vs Failover: Which Role Transition, and When

Uptime Architect — Mon, 22 Jun 2026 16:06:33 +0000

The two words get used interchangeably in incident bridges, and that confusion costs people data. A
switchover and a failover both end with your standby running as the primary — but they are not
the same operation, they don't carry the same risk, and they leave your old primary in very different
states. Pick the wrong one under pressure and you either lose data you didn't have to, or you stall a
healthy database for no reason.

Here's the distinction that matters, what actually happens to each database, and how to automate the
one you can't afford to do by hand.

The short version. A switchover is a planned, graceful role reversal: primary and standby
swap roles with zero data loss, and it's fully reversible — it's for maintenance, rolling
upgrades, and DR tests. A failover is what you do when the primary is gone: a standby is
promoted, possibly with some data loss (your protection mode decides how much), and the old
primary drops out of the configuration until you reinstate it (Flashback Database) or rebuild it.
Switchover is a choice; failover is a response. Fast-Start Failover (FSFO) automates the
response via an Observer.

The one-sentence test

Is the primary still healthy and reachable? If yes and you want to move off it deliberately, that's
a switchover. If no — it's crashed, the site is gone, it's unreachable — that's a failover. A
switchover negotiates a clean hand-off with a primary that's still talking; a failover promotes the
standby precisely because the primary isn't.

The role-transition decision. A switchover requires a living primary to hand off cleanly; a failover promotes the standby because the primary is gone — manually, or automatically via the Fast-Start Failover Observer.

What actually happens in a switchover

A switchover is a coordinated role reversal. The primary stops accepting new transactions, ships its
final redo, and becomes a standby; the chosen standby applies that last redo and becomes the
primary. Because the old primary participates in the hand-off, there is no data loss, and nothing
is thrown away — your old primary is now a perfectly good standby, already in the configuration,
already protecting the new primary. You can switch back whenever you like.

That's why switchover is the workhorse of planned availability: rolling patching, hardware
maintenance, OS upgrades, and — most importantly — DR rehearsals. If you've never run a switchover,
you don't actually know your standby works.

What actually happens in a failover

A failover is a promotion under duress. The primary is gone, so there's no graceful hand-off — the
standby is told to become the primary now, applying whatever redo it has already received. Two
consequences follow that catch people out:

You may lose data. How much depends entirely on your protection mode and whether the standby was synchronized at the moment of failure (next section). In the default Maximum Performance mode, "a few seconds of redo" is typical; in a synchronous mode, it can be zero.
The old primary is out. After a failover, the former primary is disabled and can no longer participate in the Data Guard configuration. When it comes back, its timeline has diverged from the new primary, so you can't just plug it back in. If you enabled Flashback Database, you can reinstate it (flash it back and turn it into a standby of the new primary) in minutes. Without Flashback, you're rebuilding it from a backup or a fresh copy — hours, not minutes.

This is the single biggest reason to run Data Guard with Flashback Database on both databases: it
turns "rebuild the old primary" into one REINSTATE command.

Fast-Start Failover: automating the response

A failover is the operation you least want to perform by hand at 3am, so Data Guard can do it for you.
Fast-Start Failover (FSFO) uses the Broker plus a separate process called the Observer to detect
that the primary is gone and promote the standby automatically, with no DBA intervention — and then
automatically reinstate the old primary when it returns (if Flashback is enabled).

The non-negotiable details:

The Observer should run on a separate, independent host — ideally a third location, and never on the primary itself: if the observer lives on the primary, the thing that's supposed to notice the primary died dies with it. (Oracle recommends a third site; a host in the standby's data center is an accepted fallback when one isn't available.)
FSFO promotes only when the failure is real and the standby is recoverable — it respects a configurable threshold so a brief blip doesn't trigger a needless failover.
Run it through the Broker (DGMGRL); FSFO is not a manual-SQL feature.

Protection mode decides your failover data loss

Switchover is always zero-loss. Failover loss is set long before the incident, by your protection
mode:

Protection mode	Redo transport	Failover data loss	Trade-off
Maximum Performance (default)	ASYNC	possibly seconds of redo	no commit latency on the primary
Maximum Availability	SYNC	zero if synchronized at failure	small commit latency; degrades to ASYNC if the standby is unreachable
Maximum Protection	SYNC	zero, guaranteed	the primary shuts down rather than commit without a standby ack

The mode is a business decision — what is a transaction worth? — not a technical default to accept
blindly. Most estates run Maximum Availability with Fast-Start Failover for the zero-loss-without-the-
hard-stall sweet spot. (FSFO is supported in both Maximum Availability and, with a configured lag
limit, Maximum Performance.)

How to run each (the Broker way)

The Broker turns both transitions into one verb each, with built-in validation:

-- Planned: swap roles, no data loss, fully reversible
DGMGRL> SWITCHOVER TO 'standby_db';

-- Unplanned: promote the standby because the primary is gone
DGMGRL> FAILOVER TO 'standby_db';

-- After a failover, bring the old primary back as a standby (needs Flashback Database)
DGMGRL> REINSTATE DATABASE 'old_primary';

-- Turn on automatic failover (after setting the target, protection mode, and starting the observer)
DGMGRL> ENABLE FAST_START FAILOVER;

You can do role transitions with raw SQL — ALTER DATABASE SWITCHOVER TO <db> on the primary, and
ALTER DATABASE FAILOVER TO <db> on the standby (12c+ syntax; the older ACTIVATE PHYSICAL STANDBY DATABASE is a legacy, last-resort path) — but the Broker validates prerequisites, orders the steps, and
handles the observer and reinstate for you. For anything beyond a learning exercise, use the Broker.

Want to practice this? The Data Guard switchover/failover forensics
lab gives you five Broker
situations to read — decide switchover vs failover, quantify the data loss, and handle the old primary
— with a grade.sh self-check. No standby required; it's transcripts and bash.

What teams get wrong

Confusing the two under pressure — calling a failover when the primary is fine (and needlessly losing data), or attempting a switchover against a primary that's already dead (it can't hand off). The one-sentence test prevents both.
No Flashback Database, so every failover means rebuilding the old primary from scratch instead of a one-command reinstate.
The observer co-located on the primary — it dies with the very failure it exists to detect.
Never testing a switchover. An untested standby is a hope, not a DR plan. Switchover is the test; run it on a schedule.
Accepting the default protection mode without deciding what a lost transaction actually costs.

Role transitions are one piece of the bigger picture — see where Data Guard sits against RAC and
backups in The Oracle HA Decision Tree. To drill the
decision itself — switchover vs failover, the data loss, the reinstate — work through the five Broker
situations in the no-Docker Data Guard switchover/failover forensics
lab. And to stand up a
real physical standby and run an actual switchover end-to-end (your own Enterprise Edition binaries),
the opt-in Data Guard module walks
through it.

Frequently asked questions

What is the difference between switchover and failover in Oracle Data Guard?

A switchover is a planned, graceful role reversal between a healthy primary and a standby, with no data loss, and it is fully reversible. A failover promotes a standby to primary because the original primary is gone or unreachable; it may involve data loss depending on the protection mode, and the old primary must be reinstated or rebuilt afterward.

Does a Data Guard switchover lose data?

No. A switchover is a coordinated hand-off in which the primary ships its final redo before giving up the primary role, so there is no data loss. The old primary becomes a standby and remains in the configuration.

How much data does a failover lose?

It depends on the protection mode and whether the standby was synchronized at the moment of failure. In the default Maximum Performance (asynchronous) mode, typically a few seconds of redo can be lost. In Maximum Availability or Maximum Protection (synchronous) modes, a failover can be zero data loss when the standby was synchronized.

What happens to the old primary after a failover?

It is disabled and can no longer participate in the configuration because its timeline has diverged from the new primary. If Flashback Database was enabled, you can reinstate it as a standby of the new primary with a single REINSTATE command. Without Flashback, you must rebuild it from a backup or a fresh copy.

What is Fast-Start Failover and when does it trigger?

Fast-Start Failover (FSFO) uses the Data Guard Broker and a separate Observer process to automatically fail over to the standby when the primary is lost and conditions are met, with no DBA intervention, then automatically reinstate the old primary when it returns if Flashback is enabled. It respects a configurable threshold so a brief outage does not cause a needless failover.

Where should the Fast-Start Failover observer run?

On a separate host from both databases — ideally a third, independent location, and never on the primary itself, since it would fail along with the primary and could not initiate the failover it exists to perform.

Do I need Flashback Database for Data Guard?

It is not strictly required, but it is strongly recommended. Flashback Database lets you reinstate the old primary as a standby after a failover with one command instead of rebuilding it, and it is what makes automatic reinstatement under Fast-Start Failover possible.

Can I reverse a failover?

Not directly. After a failover the standby is the new primary and the old primary is out of the configuration. You bring the old primary back by reinstating it (with Flashback Database) or rebuilding it, after which you can switch back if you want the original roles.

Originally published at uptimearchitect.com.

RAC Node Eviction: A Troubleshooting Checklist That Starts With "Why"

Uptime Architect — Mon, 22 Jun 2026 16:01:01 +0000

A node disappears from your cluster at 3am. crsctl stat res -t shows it down, the surviving node
logged a reconfiguration, and someone is already asking whether you lost data. You didn't — and that's
the entire point of eviction. The harder question is the one you actually have to answer: why, and
will it happen again tonight.

This is a checklist for that second question. Not "restart Grid Infrastructure and hope" — a way to read
the cluster's own logs and land on the real cause: the interconnect, the voting disks, or a node that
starved itself to death.

The short version. A RAC node is evicted when it can no longer prove it is healthy to the rest
of the cluster — it stopped answering the network heartbeat over the interconnect (misscount,
default 30s), it lost access to a majority of the voting disks (disktimeout, default 200s), or
its own local guardian processes found ocssd hung. Eviction is not the bug; it is the cluster
protecting your data from split-brain. The fix is always upstream — find which heartbeat failed, and
why.

Why eviction exists: split-brain is worse than downtime

Picture the interconnect between two nodes going dark. Node 1 can't see Node 2; Node 2 can't see Node 1.
Each concludes it is the lone survivor. Both keep opening the shared database and writing to the same
datafiles on shared storage — with no coordination of locks or buffer state between them. That is
split-brain, and it doesn't cause downtime; it causes corruption, the kind you discover weeks later in
a block that two nodes overwrote independently.

Eviction is Clusterware's refusal to let that happen. When membership becomes uncertain, it forcibly
removes nodes until exactly one consistent cluster remains. You trade one node's availability for the
integrity of the database. That is always the right trade — which is why the goal of troubleshooting is
never "stop the evictions," it's "remove the condition that made membership uncertain."

The three heartbeats — this is the whole mental model

Cluster Synchronization Services (CSS), via the ocssd daemon on each node, keeps three heartbeats
alive. Understand these and most evictions diagnose themselves.

Heartbeat	What it proves	Over	Timeout	A miss triggers
Network	"other nodes can still reach me"	private interconnect	`misscount` (≈30s)	suspected split-brain → the losing side is evicted
Disk	"I can still see the cluster's source of truth"	voting disks	`disktimeout` (≈200s)	the node evicts itself
Local	"my own `ocssd` is alive and responsive"	`cssdagent` + `cssdmonitor` on the node	short, internal	reboot — or a rebootless restart of the stack

The network heartbeat is sent every second across the interconnect. Miss it for misscount seconds
and CSS assumes the node is gone or partitioned. The disk heartbeat is written to the voting files
every second; lose access to a majority of them for disktimeout seconds and the node removes itself,
because a node that can't see the voting majority cannot safely claim membership. The local
heartbeat is the subtle one: cssdagent and cssdmonitor watch ocssd on the same machine, so a
node frozen by CPU starvation or an OS hang — where ocssd is alive but can't get scheduled — gets put
down by its own guardians.

The eviction decision, per node. Any one heartbeat failing for its timeout is enough. Network-heartbeat loss escalates to a split-brain vote resolved by the voting disks.

Split-brain resolution: who actually survives

When the cluster splits, CSS doesn't flip a coin. The sub-cluster that can see a majority of the voting
disks wins. Between otherwise-equal partitions, the larger sub-cluster survives; on a true tie (e.g., a
two-node cluster split clean down the middle), the node with the lowest node number survives and the
other is evicted. The losing nodes are fenced.

This is exactly why voting disks come in odd numbers (1, 3, 5) and should sit across independent
failure groups: a node must reach more than half of them to stay in the cluster. Three voting files on
three separate storage paths means a node can lose one path and still vote.

The usual causes, ranked by how often they're the culprit

The private interconnect — the number-one cause, by a wide margin. Dropped or corrupted packets, a NIC flapping, a flaky switch port, or — the classic intermittent gremlin — an MTU / jumbo-frame mismatch where 9000-byte frames work until something fragments. A saturated interconnect (sharing a NIC with backup or application traffic) starves the heartbeat the same way a dead link does.
Voting disk / storage I/O. A lost SAN path, multipath flapping, or storage latency that exceeds disktimeout makes a node unable to write its disk heartbeat. If the ASM disk group holding the voting files goes offline, every node that loses the majority self-evicts.
Node hang / resource starvation. A node pinned at 100% CPU, or thrashing in swap, can't schedule ocssd — so it misses heartbeats it is technically "up" to send. This looks like a network problem in the logs but is really a performance problem. (Diagnose the starvation itself the way you would any slow database — see How to Read an AWR Report Without Drowning — alongside OS-level data.)
Time synchronization drift. Large clock skew between nodes destabilizes membership. Grid Infrastructure runs the Cluster Time Synchronization Service (ctssd) in observer mode when NTP/chrony is configured, active mode when it isn't; a broken time setup undermines both.
Hardware faults and known bugs. A failing NIC/HBA, bad memory, or a Clusterware bug fixed in a later Release Update. Always check the eviction signature against current GI patches.

Diagnosis: read the logs in this order

Work top-down. The first log tells you when and that; the rest tell you why. (Paths are
Oracle-Base/version-dependent — 12c+ uses the ADR trace layout; 11.2 uses $GRID_HOME/log/<host>/....)

GI alert log — start here. Find the eviction timestamp and the reconfiguration message. This anchors every other log to a moment in time.
ocssd trace (ocssd.trc / ocssd.log) — the heartbeat story. Search for phrases like "missed checkin", "Polling", and eviction/kill messages around the alert-log timestamp.
cssdagent / cssdmonitor logs — read these if the node rebooted; they record the local guardian's decision to put the node down.
OS messages (/var/log/messages, journalctl) — the reboot time, hardware/driver errors, and any OOM-killer activity. A reboot with no Clusterware reason in the GI logs points at the OS or hardware.
Cluster Health Monitor (CHM / oclumon) and OSWatcher — the single most useful evidence: per-second CPU, memory, and network counters from the seconds before the eviction. If you don't have these running, the smoking gun is already gone by morning.

What you see in the logs	Likely cause	Confirm with
"missed checkin" / "Polling" on the interconnect	network loss or saturation	OSWatcher `netstat`/`ifconfig`, switch logs, MTU test
disk-heartbeat / voting-file I/O timeout	storage path loss or latency	multipath status, ASM disk state, SAN latency
node rebooted, `ocssd` "hung", killed by `cssdagent`	CPU/memory starvation or OS hang	CHM/OSWatcher CPU & memory at eviction time
clock-skew / `ctssd` warnings	broken time sync	`chronyc`/`ntpq`, `ctssd` trace
reboot, nothing in GI logs	hardware fault or OS panic	`/var/log/messages`, IPMI/ILOM, vendor diagnostics

Want to practice this? The RAC node-eviction forensics
lab gives you five realistic
scenarios — interconnect, voting disk, starvation, time drift, and one that isn't an eviction at
all — as raw logs to diagnose. No cluster required; it's text and a grade.sh self-check.

Rebootless restart: why the node sometimes doesn't reboot

On 11.2.0.2 and later, Grid Infrastructure tries a rebootless restart: instead of bouncing the whole
OS, it attempts to gracefully stop and restart just the GI stack when the failure is inside the stack and
I/O can be safely halted. When it can't safely stop I/O — a kernel-level hang, for instance — it falls
back to a full node reboot to guarantee fencing. So "the node restarted Clusterware but didn't reboot"
and "the node power-cycled" are two outcomes of the same protective logic, and the logs distinguish them.

Fix and prevent, by cause

Cause	Fix it now	Stop it recurring
Interconnect	repair the link/switch; verify MTU is consistent end-to-end	redundant private NICs (HAIP or bonding), a dedicated interconnect, validated jumbo frames, no app/backup traffic sharing it
Voting / storage	restore the storage path; check ASM disk-group state	odd number of voting files across independent failure groups; monitor I/O latency; healthy multipathing
Starvation	relieve the CPU/memory pressure	headroom on every node; don't co-locate greedy workloads; deploy CHM; diagnose the load like any perf issue
Time sync	fix chrony/NTP on all nodes	keep time sync healthy cluster-wide, or let `ctssd` run active consistently
Bug / hardware	apply the relevant GI Release Update; replace the faulty part	stay current on GI RUs; monitor hardware health proactively

What teams get wrong

Treating the eviction as the failure. The node did its job. The bug is whatever made membership uncertain — chase that.
Raising misscount to "fix" it. This usually masks the real problem and widens the window where split-brain is possible. Oracle generally advises against changing it; reach for it last, if ever.
No CHM or OSWatcher running. Without per-second history, the CPU/network spike that caused the eviction is unrecoverable by the time you log in. Deploy them before the next incident — they are the difference between a root cause and a guess.
A single interconnect NIC with no redundancy — one cable or port becomes a cluster-wide outage.
Voting disks on the same fragile storage path as everything else — the tiebreaker shouldn't share a failure domain with the data.

RAC is the layer that keeps a node failure from becoming an outage — see where it fits among the other
HA options in The Oracle HA Decision Tree. And to
drill the skill this whole post is about — reading the logs and naming the cause — work through the five
scenarios in the no-Docker RAC node-eviction forensics
lab. For a real cluster to break
and recover (you'll want a 32 GB+ host and your own Enterprise Edition binaries), Oracle's official
RAC-on-Docker and Vagrant projects are the supported, license-compliant path.

Frequently asked questions

What is node eviction in Oracle RAC?

Node eviction is when Oracle Clusterware forcibly removes a node from the cluster because that node can no longer prove it is healthy and reachable. It is a protective action that prevents split-brain — two nodes independently writing to the same shared database and corrupting it.

What is the most common cause of RAC node eviction?

Private interconnect problems are the most common cause: dropped or corrupted packets, a flapping NIC, a bad switch port, an MTU or jumbo-frame mismatch, or a saturated interconnect that shares bandwidth with other traffic. Storage and resource starvation are the next most common.

What is the difference between misscount and disktimeout?

Misscount is the network-heartbeat timeout (default about 30 seconds): how long a node can miss interconnect heartbeats before CSS treats it as gone. Disktimeout is the voting-disk timeout (default about 200 seconds): how long a node can fail to access a majority of voting disks before it evicts itself.

Does a node eviction always reboot the server?

Not always. On 11.2.0.2 and later, Grid Infrastructure attempts a rebootless restart — gracefully stopping and restarting only the GI stack when the failure is within the stack and I/O can be safely halted. It falls back to a full reboot when it cannot safely stop I/O, such as a kernel-level hang.

How do I find out why a node was evicted?

Read the logs in order: the Grid Infrastructure alert log for the eviction time, then the ocssd trace for the heartbeat misses, then the cssdagent and cssdmonitor logs if the node rebooted, then the OS messages, and finally Cluster Health Monitor and OSWatcher data for the CPU, memory, and network state in the seconds before the eviction.

How do voting disks prevent split-brain?

A node must be able to see a majority of the voting disks to remain a cluster member. When the cluster splits, the sub-cluster that sees the voting majority survives and the other is evicted. That is why voting disks are deployed in odd numbers across independent failure groups.

Can high CPU or memory pressure cause a node eviction?

Yes. A node pinned at 100% CPU or thrashing in swap may be unable to schedule the ocssd process, so it misses heartbeats even though the machine is technically up. This often appears as a network-heartbeat miss in the logs but is really a performance problem, and the local guardian processes may reboot the node.

Should I increase misscount to stop frequent evictions?

Generally no. Raising misscount usually masks the underlying problem and widens the window in which split-brain could occur. Oracle advises against changing it in most cases. Fix the root cause — interconnect, storage, time sync, or resource starvation — instead.

Originally published at uptimearchitect.com.

How to Read an AWR Report Without Drowning

Uptime Architect — Mon, 22 Jun 2026 15:49:18 +0000

An AWR report is a wall of numbers — dozens of sections, hundreds of rows, a thousand ways to get lost
chasing a metric that doesn't matter. Most people scroll to "Buffer Hit %", see 99%, and conclude the
database is healthy. That instinct is exactly backwards, and it's why so much "tuning" optimizes the
wrong thing.

There's a faster, more reliable way to read one — and it fits in your head. It rests on a single idea
(DB Time), follows a fixed five-section path, and tells you within about ninety seconds whether
you're CPU-bound, waiting on something, or chasing a ghost. This is that method. It targets Oracle
19c, the enterprise workhorse, with notes on 23ai/26ai, and it comes with a free lab so you can
generate a real report and read it alongside the guide.

The short version. One number runs the whole report: DB Time — all the time the database spent
in user calls. Divide it by elapsed time to get Average Active Sessions and compare that to your
CPU count. Then read Top Timed Events: DB CPU on top means do less work (inefficient SQL); a
wait on top means follow that wait. Let the top event pick which SQL ordered by… list to read.
Ignore the hit ratios. Reach for ASH when it's a short spike, not a steady state.

First, a licensing reality

AWR (and ASH, and ADDM, and the DBA_HIST_* views) are part of the Diagnostics Pack, a separately
licensed option on Enterprise Edition. Querying them on a database you're not licensed for is a
compliance problem, not a free lunch. The governing switch is CONTROL_MANAGEMENT_PACK_ACCESS:

SELECT value FROM v$parameter WHERE name = 'control_management_pack_access';
-- DIAGNOSTIC+TUNING  -> AWR/ASH/ADDM available (Diagnostics + Tuning Pack licensed)
-- NONE               -> not available; use Statspack (free) instead

If you're not licensed, Statspack is the free, built-in alternative — same ideas, fewer features.
For the lab below we use Oracle Database Free, where the packs are enabled for development use (not a
license waiver for production).

DB Time: the one metric everything hangs on

DB Time is the total time the database spent in user calls — CPU plus all non-idle waits — across all
sessions. It is the master metric because tuning has exactly one goal: reduce DB Time. Every other
number in the report is just a clue about where DB Time went.

The header gives you the two numbers that frame everything:

              Snap Id      Snap Time        Sessions Curs/Sess
            --------- ------------------- -------- ---------
Begin Snap:         1   13-Jun-26 10:00      42       3.1
  End Snap:         2   13-Jun-26 10:10      45       3.2
   Elapsed:               10.00 (mins)
   DB Time:               21.40 (mins)

Divide them: Average Active Sessions (AAS) = DB Time / Elapsed. Here that's 21.4 / 10 ≈ 2.1. That
single number tells you how much work the database was really doing on average. Compare it to your CPU
count:

AAS vs CPU count	What it means
AAS ≪ CPUs	Database is mostly idle; if users complain, the problem is probably outside the DB
AAS ≈ CPUs	Running hot but not necessarily wrong — check whether it's CPU or waits
AAS ≫ CPUs	Sessions are queuing — for CPU or for a wait; this is where real contention lives

Everything that follows is about explaining that DB Time — and DB Time only splits two ways, into CPU
and waits:

DB Time decomposes into DB CPU plus non-idle wait time, and waits group into classes. Every section of the report is just a finer breakdown of this one quantity.

The five sections that matter (in order)

You do not read an AWR report top to bottom. You read five sections, in this order, and you can
stop as soon as the story is clear.

1. Load Profile — the shape of the workload

A quick orientation: is this an OLTP system doing lots of small transactions, or a few heavy queries?
Scan for red flags — a hard-parse rate that isn't near zero, enormous redo, or logical reads that dwarf
physical reads (a sign of inefficient SQL doing far too many buffer gets).

Load Profile              Per Second    Per Transaction
~~~~~~~~~~~~~~~          -----------    ---------------
        DB Time(s):            2.1               0.8
         DB CPU(s):            2.1               0.8
  Logical read (blocks):   182,117             4,454
 Physical read (blocks):       210                 5
            Executes:         1,140                28
           Hard parses:           1               0.0

Logical reads massively exceeding physical reads, with DB CPU ≈ DB Time, is the classic fingerprint of
CPU burned on inefficient SQL — lots of buffer gets, not lots of disk.

2. Top Timed Events — where the time actually went

This is the heart of the report. It ranks what consumed DB Time. The first question it answers: CPU or
waits?

Top 10 Foreground Events by Total Wait Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Event                         Waits   Time (s)   Avg Wait   % DB time   Wait Class
----------------------------- -----   --------   --------   ---------   ----------
DB CPU                                    11.9                   99.1
enq: CR - block range reuse      12        0.1     6.35ms        0.6    Other

DB CPU at the top (like the 99.1% above) → you're CPU-bound. The fix is to do less work: inefficient SQL, excessive logical reads, hard parsing. Adding CPU only buys time.
A wait event at the top → follow it. db file sequential read (single-block I/O, usually index lookups), db file scattered read (multi-block, full scans), log file sync (commit/redo), enq: TX - row lock contention (locking), gc events (RAC interconnect). Each points somewhere specific.

The number that matters is % DB time, not raw waits. A million waits that sum to 2% of DB time are
noise; one event at 70% is your headline.

3. The dominant resource → the right Top SQL list

AWR gives you several "SQL ordered by …" lists, and reading the wrong one wastes time. Let section 2
choose the list:

If Top Timed Events says…	Read "SQL ordered by…"
DB CPU dominates	CPU Time (and Gets — CPU usually tracks logical reads)
`db file sequential/scattered read`	Reads (physical I/O)
Lots of executions, high parse	Executions / Parse Calls
`log file sync`	look at commit frequency, not a single SQL

In a CPU-bound report, the top of "SQL ordered by CPU Time" is your suspect. In the lab, that's
unmistakable:

SQL ordered by CPU Time
  CPU Time (s)   Elapsed (s)   Executions   SQL Text
  -----------   -----------   ----------   ---------------------------------------
        23.6          23.8            2     SELECT /*+ awr_demo */ SUM(SQRT(LEVEL)
                                            + LN(LEVEL + 1)) FROM DUAL CONNECT BY...

One statement, ~all the CPU. That's the whole investigation: you now have a SQL_ID to tune.

4. Instance Efficiency — the ratios to distrust

This section is a trap. Buffer Hit %, Library Hit %, and friends are not health scores. A 99.99%
Buffer Hit Ratio often means a query is doing millions of logical reads of cached blocks — burning CPU
while looking "efficient." Tuning to make a ratio go up is how you end up optimizing the wrong query.

A composite scenario (illustrative). A team is paged for a slow system. The AWR shows Buffer Hit %
at 99.9%, so they rule out I/O and start adding memory. The real story was two lines down: DB CPU at
95%, driven by one report query doing a nested loop over millions of cached rows. The ratio said
"healthy"; DB Time said "one query is eating the box." Trust DB Time.

Glance at Instance Efficiency for anomalies (e.g. a low parse ratio, a soft-parse problem) — never as
a pass/fail grade.

5. The supporting cast (only if you still need it)

If the first four didn't close the case, drop into the detail: Time Model Statistics (where DB Time
splits — SQL execute vs parse vs PL/SQL), Segments by Logical/Physical Reads (which table/index is
hot), and SQL ordered by Reads for I/O-bound systems. Most reports are solved before you get here.

A second worked example: when it's I/O, not CPU

The whole point of the method is that the top event routes you. Flip the workload from CPU to I/O
and the report tells a different story — you read a different SQL list and land in a different section.
The lab's drill-io flushes the cache and full-scans a table larger than the cache; here's what changes.

Load Profile — physical reads jump from a rounding error to the headline:

Load Profile              Per Second
  Logical read (blocks):    60,977
 Physical read (blocks):    47,886      <- now enormous

Top Timed Events — an I/O wait appears alongside the CPU:

Event                   Waits   Time (s)   % DB time   Wait Class
----------------------- -----   --------   ---------   ----------
DB CPU                            11.0        96.2
direct path read        4,299     0.2         2.1       User I/O

Now the routing changes. There's an I/O event in play, so you read SQL ordered by Reads (not CPU)
— and the fastest path to the culprit, Segments by Physical Reads, which names the object doing
the I/O:

Segments by Physical Reads
  Owner     Object Name    Physical Reads   % Total
  LABUSER   BIGTAB              545,472       99.7

One table, 99.7% of the physical reads. You found the hot segment in a single line — now the real
question is whether that full scan should exist at all (a missing index? an unselective predicate? a
report running far too often?).

An honest note about fast storage. On the lab's local NVMe, those ~48k reads/sec complete in
microseconds, so direct path read is only 2% of DB time and DB CPU still tops the report — the
scan burns more CPU than it waits on I/O. On production storage with real latency, the same physical-read
volume becomes the headline:

[illustrative — the same workload on latency-bound storage]
Event                       Waits     Time (s)   % DB time
db file scattered read    420,118       980        71.4
DB CPU                                   210        15.3

The diagnosis path is identical — Load Profile shows the reads, Top Events names the wait, Segments
names the object — but now I/O is unmistakably the bottleneck. Trust the read volume and the segment;
the wait time scales with your storage.

The wait events you'll actually meet

When a wait — not DB CPU — tops the report, the name tells you where to look. You don't need to
memorize Oracle's hundreds of events; you need the dozen that show up in real reports and what each is
really saying:

Wait event	Class	What it usually means	First thing to check
`db file sequential read`	User I/O	Single-block reads — usually index lookups	The SQL's access path; is the index selective?
`db file scattered read`	User I/O	Multi-block reads — full scans	Should that scan be an index range scan?
`direct path read`	User I/O	Large scans bypassing the buffer cache	Big-table scans, parallel query, sort/hash spills
`log file sync`	Commit	Sessions waiting for commit redo to flush	Commit frequency (committing per row?), LGWR/redo storage
`log file parallel write`	System I/O	LGWR writing redo to disk	Redo log device latency
`buffer busy waits`	Concurrency	Two sessions want the same block	Hot block / right-hand index growth; consider partitioning
`read by other session`	User I/O	Waiting for a block another session is reading in	Often the flip side of a hot full scan
`enq: TX - row lock contention`	Application	Row-level locking — app waiting on a lock	The blocking session/SQL; transaction design
`latch: shared pool` / `library cache`	Concurrency	Parsing/sharing pressure	Hard parsing, lack of bind variables
`gc cr / current block`	Cluster	RAC interconnect — fetching a block from another node	Interconnect health; block contention across nodes

That last one is your bridge to the other half of HA: gc events live on
Real Application Clusters. The rest follow the same
rule — the event names the resource; the SQL and segment sections name the culprit.

AWR vs ASH vs ADDM — pick the right tool

Tool	Granularity	Use it when
AWR	Aggregated over a snapshot interval (default 1h)	"The database was slow between 2 and 3pm" — the steady-state picture
ASH	Per-second active-session samples	"It froze for 90 seconds at 2:14" — short spikes AWR averages away; drill into one session/SQL
ADDM	Automated analysis of an AWR interval	A fast first opinion and a starting hypothesis

Rule of thumb: AWR for the interval, ASH for the moment. A 10-minute stall inside a 1-hour AWR
window is diluted to ~17% of the report; ASH shows it at full intensity.

Drilling into a moment with ASH

AWR aggregates an interval; ASH samples active sessions every second, so it shows what was running
at a specific moment — the right tool for a spike. You generate an ASH report the same way you'd
generate AWR (ashrpt.sql, OEM, or DBMS_WORKLOAD_REPOSITORY.ASH_REPORT_TEXT), but bounded by a time
range instead of snapshots. The lab's drill-ash runs a short burst and reports on it:

Top User Events            Avg Active Sessions   % Activity
-------------------------   -------------------   ----------
CPU + Wait for CPU                  1.0              98.5

Top SQL Statements                          % Activity   SQL Id
SELECT /*+ ash_demo */ SUM(SQRT(LEVEL)...      97.8      <sql_id>

ASH answers what AWR can't: which session, which SQL, which wait — at 2:14:32, not "sometime in
the last hour." For a hands-on look you can also query the view directly:

SELECT sql_id, event, COUNT(*) AS samples
FROM   v$active_session_history
WHERE  sample_time > SYSTIMESTAMP - INTERVAL '5' MINUTE
GROUP  BY sql_id, event
ORDER  BY samples DESC FETCH FIRST 10 ROWS ONLY;

Each row of ASH is roughly one second of one active session, so those samples counts are
effectively seconds of DB time broken down by SQL and wait — a histogram of where the database's
attention actually went. (A NULL event means the session was on CPU.)

The 90-second triage

Reading an AWR report: let DB Time and the top event route you. Stop as soon as the story is clear.

What teams get wrong

Reading ratios instead of DB Time. Buffer Hit % is not a grade. Start at Top Timed Events.
Too-wide a window. A 6-hour AWR averages your 10-minute incident into invisibility. Pick the tightest snapshot pair that brackets the problem.
Spanning a restart. An AWR report across an instance bounce is meaningless — stats reset.
Tuning the top SQL by the wrong metric. If you're I/O-bound, the biggest-CPU SQL may be irrelevant.
Confusing waits count with impact. Sort by % DB time, always.
Using AWR for a spike. Reach for ASH when the problem is short-lived.

Try it yourself: generate a real AWR report

The fastest way to internalize this is to make one. The awr/ lab in
github.com/pyaroslav/oracle-labs spins up Oracle Database
Free with Docker, runs a known CPU-bound workload between two snapshots, and hands you a real AWR report
to read — no Oracle account, no license:

./run.sh up         # start Oracle Database Free
./run.sh all        # setup, then all three drills below
# or run them individually:
./run.sh drill      # CPU-bound  -> awr-report.txt  (DB CPU ~99%, the workload tops SQL by CPU)
./run.sh drill-io   # I/O        -> io-report.txt   (huge physical reads; BIGTAB tops Segments by Reads)
./run.sh drill-ash  # ASH        -> ash-report.txt  (Top User Events / Top SQL for the recent window)

Every excerpt in this post — the CPU-bound report, the I/O signature, the ASH output — was generated by
these drills on Oracle Database Free 26ai. Change a workload, regenerate, and watch the report change
with it; that feedback loop is what turns "reading AWR" from intimidating into routine.

What about 23ai and 26ai?

The method is release-stable: DB Time, Top Timed Events, and the SQL lists work the same on 19c, 23ai,
and the current 26ai. Newer releases sharpen the tooling around AWR — Real-Time SQL Monitoring and a
richer Active Session History make drilling into a single statement easier — but the reading order above
doesn't change. (On the Free image, AWR/ASH are available for learning, as the lab shows.)

Frequently asked questions

What is DB Time in an AWR report?

DB Time is the total time the database spent working in user calls — CPU time plus all non-idle wait time — summed across all sessions during the snapshot interval. It is the master metric of Oracle performance: the goal of tuning is to reduce DB Time. Dividing DB Time by elapsed time gives Average Active Sessions, a measure of how busy the database really was.

Which section of an AWR report should I read first?

After checking DB Time and Average Active Sessions in the header, read the Top Timed Events (Top 10 Foreground Events) section. It ranks what consumed DB Time and immediately tells you whether the database is CPU-bound (DB CPU at the top) or waiting on something specific. Do not start with Buffer Hit % or other ratios.

Is a high Buffer Cache Hit Ratio good?

Not necessarily. A very high Buffer Hit Ratio often means a query is performing millions of logical reads against cached blocks, burning CPU while appearing efficient. Hit ratios are not health scores. Diagnose performance from DB Time and wait events, not from cache ratios.

When should I use ASH instead of AWR?

Use AWR for the aggregate picture over a snapshot interval (for example, "the database was slow from 2 to 3pm"). Use ASH (Active Session History) for short-lived spikes that AWR averages away — for example a 90-second stall — and to drill into a specific session, SQL, or wait at a precise moment.

Do AWR and ASH require a license?

Yes. AWR, ASH, ADDM, and the DBA_HIST_* views are part of the Diagnostics Pack, a separately licensed option on Oracle Enterprise Edition. The CONTROL_MANAGEMENT_PACK_ACCESS parameter governs access. If you are not licensed, use Statspack, the free built-in alternative.

How long should the AWR snapshot window be?

Pick the tightest snapshot pair that brackets the problem — typically a single default (hourly) interval or a custom snapshot pair around the incident. Wide windows (several hours) average out short spikes and hide the issue. Never run a report across an instance restart, because statistics reset.

What is Average Active Sessions (AAS) in an AWR report?

Average Active Sessions is DB Time divided by elapsed time over the snapshot interval. It measures how many sessions, on average, were actively working in the database at once. Compared against the server CPU count it tells you the load: AAS well below the CPU count means the database is mostly idle; AAS near the CPU count means it is running hot; AAS well above the CPU count means sessions are queuing for CPU or a wait, which is where real contention shows up.

What does the db file sequential read wait event mean?

db file sequential read is the wait for a single-block read from disk into the buffer cache, most often an index block or a table block reached by rowid during an index lookup. It is normal in small amounts. When it dominates an AWR report, look at the SQL doing those reads and whether the index is selective — sometimes the fix is a better index, and sometimes it is avoiding an index in favor of a full scan. It is distinct from db file scattered read, which is the multi-block read used by full scans.

The one-paragraph version

Compute Average Active Sessions (DB Time ÷ Elapsed) and compare it to your CPU count. Read Top Timed
Events to learn whether you're CPU-bound or wait-bound. Let that pick which SQL ordered by … list
to read, and pull the SQL_ID of the dominant statement. Ignore the hit ratios. Use ASH when the
problem is a short spike rather than a steady state. Everything else in the report is a supporting
detail you'll usually never need.

Originally published at uptimearchitect.com.

The Oracle HA Decision Tree: RAC vs Data Guard vs Both

Uptime Architect — Sun, 07 Jun 2026 23:02:33 +0000

"We have RAC, so we're covered for DR." It's one of the most expensive sentences in Oracle operations,
and I've watched variations of it play out more than once. Real Application Clusters (RAC) and Data
Guard both live under the "high availability" umbrella, so it's easy to assume they're interchangeable
— or that having one means you don't need the other. They are not interchangeable. They solve
different failures, and the cost of confusing them is usually discovered at the worst possible time.

This is the long version of how I think about the choice. We'll start where every good HA design
starts — not with a feature, but with the failure you're trying to survive — then work through what RAC
and Data Guard each actually do, what they cost (in licensing and in complexity), how to reason about
RTO and RPO, and finally a decision tree you can apply to a real system. Everything here targets
Oracle 19c, the enterprise workhorse, with notes on where the newer releases — 23ai and the
current 26ai — change the picture. It's written from general industry practice and lab work — your
environment will differ, so test before you trust.

The short version. RAC keeps you running through a node failure — but it's one copy of your
data on shared storage, so it is not disaster recovery. Data Guard keeps you running through
site loss and corruption by maintaining an independent standby you fail over to. Neither saves
you from a bad DELETE — only backups and Flashback do. Set RTO and RPO with the business, then
buy the cheapest combination that meets them.

Start with the failure, not the feature

Before you evaluate any technology, write down the failure modes you actually need to survive. There
are four that matter for an Oracle database, and they are genuinely different problems:

Instance or node failure — a database instance crashes, or the server it runs on dies.
Site or region loss — a data center, availability zone, or whole region becomes unavailable.
Data corruption — physical block corruption (bad storage, lost writes) or logical corruption.
Human error — an accidental DROP TABLE, a bad deploy, a DELETE without a WHERE clause.

No single feature covers all four. That is the entire reason this article exists. Here is the map we'll
spend the rest of the post justifying:

Failure mode	RAC	Data Guard	Backups + Flashback
Instance / node failure	Yes	Partial (failover)	No
Site / region loss	No	Yes	Partial (slow, if offsite)
Block corruption	No	Yes	Yes
Human / logical error	No	No	Yes

Notice that the bottom row — human error — is covered by neither RAC nor Data Guard. Hold that
thought; it's the mistake I see most often.

What RAC actually solves

RAC runs multiple database instances on multiple servers (nodes) against one shared copy of the
database. The instances coordinate through Oracle Grid Infrastructure (Clusterware) and a private
interconnect, using Cache Fusion to ship blocks between node memories. Clients connect through the SCAN
listener and node VIPs, so a failed node's sessions are redirected to survivors.

What that buys you:

Instance and node resilience. If a node dies, the surviving instances keep serving the same database. There's no "restore" and no "fail over to a copy" — the data was already open on the other nodes.
Online scale-out for reads and writes. Add a node, add capacity, without re-architecting.
Rolling maintenance. Patch or relocate one node at a time while the service stays up.
Brownout masking. With application services and Application Continuity / TAF, in-flight work can be replayed or transparently redirected during a node loss.

You check on it with Clusterware and srvctl:

# Cluster resource overview
crsctl status resource -t

# Is the database up, and on which instances?
srvctl status database -d ORCLCDB

# Service placement (services are how you steer connections across nodes)
srvctl status service -d ORCLCDB

Now the part that the "RAC is our DR" crowd misses: every RAC instance points at the same storage.
There is exactly one copy of your data. A storage array failure, a site outage, or a corrupt block is
seen identically by all nodes. RAC gives you redundancy of compute, not redundancy of data.

A composite scenario (illustrative). Picture a shop running a healthy 3-node RAC cluster. Uptime
dashboards are green for two years; leadership is told the database is "fully redundant." Then a SAN
controller pushes bad firmware and the shared LUNs go offline. All three nodes go down at once,
because all three were reading the same storage. The cluster did exactly what it was designed to do —
it just was never designed for that failure. That's not a RAC flaw; it's a design gap.

Licensing and complexity (the honest cost)

RAC is a separately licensed option on top of Oracle Database Enterprise Edition, priced per
processor (or in the cloud, baked into certain shapes/editions). On top of license cost you're taking
on real operational weight: Clusterware, a redundant private interconnect, shared storage (typically
ASM), and the skills to run all of it. That complexity is itself a source of outages if the team isn't
staffed for it — a RAC node eviction, where
Clusterware fences a node it can't verify is healthy, is the canonical 3am example.

RAC One Node is the pragmatic middle ground: a single active instance that Clusterware can fail
over (or you can online-relocate) to another node, with online rolling patching — most of the
availability benefit, far less of the multi-instance complexity, and you can scale up to full RAC later.

# RAC One Node: relocate the running instance to another node, online
srvctl relocate database -d ORCLCDB -node racnode2

What Data Guard actually solves

Data Guard maintains one or more standby databases — independent, physically separate copies of
your primary — kept in sync by shipping redo and applying it. A physical standby applies redo
block-for-block (Redo Apply); a logical standby reconstructs SQL (SQL Apply). For HA/DR, physical
standby is the default and the one I'll focus on. The Data Guard Broker (dgmgrl) is how you should
manage it — it removes most of the manual ALTER DATABASE foot-guns.

What it buys you:

Site and region survival. The standby is a different database on different storage, usually in a different location. Lose the primary site and you fail over to the standby.
Corruption protection. Because the standby is an independent copy with its own writes, it doesn't inherit the primary's physical block corruption. With Active Data Guard, Automatic Block Media Recovery can transparently repair a corrupt block on either side from the other.
A real failover/switchover target. Planned role transitions (switchover) for maintenance, and unplanned ones (failover) for disasters.
Read offload and more (with Active Data Guard): an open read-only standby for reporting, offloaded backups, and snapshot standbys you can open read-write for testing and then flip back.

You watch role and lag with SQL and the broker:

-- Where am I, and what mode am I in?
SELECT database_role, open_mode, protection_mode, switchover_status
FROM   v$database;

-- How far behind is apply? (the number that matters during an incident)
SELECT name, value, time_computed
FROM   v$dataguard_stats
WHERE  name IN ('transport lag','apply lag');

dgmgrl sys@ORCLCDB
DGMGRL> SHOW CONFIGURATION;
DGMGRL> SHOW DATABASE 'ORCLCDB_STBY';

-- Planned role swap (maintenance): primary and standby trade places
DGMGRL> SWITCHOVER TO 'ORCLCDB_STBY';

-- Unplanned (disaster): promote the standby
DGMGRL> FAILOVER TO 'ORCLCDB_STBY';

Protection modes set your RPO

Data Guard's protection mode is the dial that trades data-loss risk against primary performance:

Protection mode	Redo transport	Data loss (RPO)	Effect on primary
Maximum Performance (default)	ASYNC	Possible — seconds of redo	None
Maximum Availability	SYNC	Zero while in sync; falls back to ASYNC if the standby is unreachable	Small commit latency
Maximum Protection	SYNC	Zero, guaranteed	Primary stalls if no standby can acknowledge

Most enterprises run Maximum Availability with SYNC transport to a nearby standby — zero data loss
in normal operation, without the "halt production if the standby is down" behavior of Maximum
Protection.

Going further: Fast-Start Failover and Far Sync

Fast-Start Failover (FSFO) adds automatic failover. A lightweight Observer process (run it on a third, independent host) watches both databases and promotes the standby automatically if the primary disappears — turning a 2am page into an event you read about in the morning.

  DGMGRL> ENABLE FAST_START FAILOVER;
  DGMGRL> START OBSERVER;

Far Sync solves the distance problem. SYNC gives you zero data loss but adds latency proportional to distance, so a DR site 2,000 km away can't be SYNC without hurting production. A Far Sync instance — a tiny control-file-and-redo-only instance placed near the primary — receives redo SYNC (zero loss, low latency) and forwards it ASYNC to the distant standby. You get RPO ≈ 0 and geographic distance.

Far Sync gives you zero data loss over distance: synchronous redo to a nearby Far Sync instance, then asynchronous onward to a far-off standby. A Fast-Start Failover Observer in a third location promotes the standby automatically.

Licensing note

Plain Data Guard (a physical standby in mount mode, doing Redo Apply) is included with Enterprise
Edition — there's no excuse not to have one. Active Data Guard — the open read-only standby,
Automatic Block Media Recovery, Far Sync, and friends — is a separately licensed option. Decide
deliberately which capabilities you're actually licensed for.

A composite scenario (illustrative). A team has a standby and a green broker status, so DR is
"done." Nobody has ever run a switchover. During a real failover they discover apply has been lagging
for weeks behind a quietly-stuck archive gap, the network team never opened the ports for client
redirection, and the runbook references a host that was decommissioned. The technology worked; the
operational readiness didn't. A standby you've never failed over to is a hope, not a plan.

The combined topology: RAC + Data Guard

When you genuinely need both local zero-downtime and cross-site survival, you run RAC at each site
with Data Guard between them. This is the heart of Oracle's Maximum Availability Architecture (MAA):
local node failures are absorbed by RAC with no failover at all, while a site loss triggers a Data
Guard role transition.

It's the gold standard, and it's also the most expensive and most complex thing on the menu — you're
paying for (and operating) RAC and Active Data Guard, in two locations. The honest question is
whether your RTO/RPO targets and the business cost of downtime justify it. MAA frames this as tiers, so
you can match spend to requirement:

MAA tier	Adds	Protects against
Bronze	Single instance + RMAN backups + Flashback	Corruption, human error (slow recovery)
Silver	+ RAC or RAC One Node	Instance/node failure (near-zero RTO locally)
Gold	+ Active Data Guard	Site loss, corruption; read offload
Platinum	+ GoldenGate, Application Continuity, Edition-Based Redefinition	Zero-downtime maintenance, app-transparent failover

A useful way to read this table: you don't start at Gold. You start at Bronze and climb only as far
as your RTO/RPO and budget require.

What MAA Gold actually looks like

It helps to picture the topology. RAC handles failures inside each site; Data Guard handles losing a
site; and the Observer — deliberately in a third location — is what makes failover automatic without
becoming a casualty of the outage it's supposed to detect.

MAA Gold: RAC at each site for local node resilience, Active Data Guard between sites for DR + corruption protection + read offload, and an FSFO Observer in a third location for automatic failover.

Read it as two independent failure domains: lose a node and RAC absorbs it with no role change at all;
lose a site and Data Guard promotes the standby. The reporting team can run on the open Active Data
Guard standby, and backups can be offloaded there too — so the DR copy earns its keep every day, not
just during a disaster.

Don't forget the two failure modes nobody licensed for

Look back at that first table. RAC and Data Guard together still leave two rows uncovered well, and one
of them is the most common cause of "lost data" incidents.

Block corruption is partly handled by Data Guard (independent copy, Automatic Block Media Recovery)
but your baseline defenses are configuration and backups: enable DB_BLOCK_CHECKING and
DB_LOST_WRITE_PROTECT, run periodic RMAN VALIDATE/BACKUP VALIDATE, and keep recoverable RMAN
backups.

Human and logical error is the trap. A DELETE with no WHERE clause is a perfectly valid
transaction — so Data Guard faithfully ships it to the standby and applies it in milliseconds. Your
"redundancy" just replicated the mistake to every copy. The defenses here are a different toolset
entirely:

-- Flashback Database: rewind the whole database to just before the mistake
-- (requires flashback logging / a guaranteed restore point)
SELECT flashback_on FROM v$database;
FLASHBACK DATABASE TO RESTORE POINT before_bad_deploy;

-- Or recover a single object after an accidental drop
FLASHBACK TABLE app.orders TO BEFORE DROP;

Guaranteed restore points before risky changes, Flashback Database/Table/Query, and RMAN
point-in-time recovery are what save you here — not replication. If you take one thing from this
article beyond "RAC ≠ DR," take this: replication is not a backup.

Make the decision with RTO and RPO first

Every choice above maps cleanly onto two numbers you should set with the business, not in IT:

RTO (Recovery Time Objective): how long can you be down? RAC handles node failure in ~seconds with no failover. Data Guard with FSFO recovers a site loss in seconds-to-minutes. Backups mean hours.
RPO (Recovery Point Objective): how much data can you lose? RAC: zero (same data). Data Guard: zero with SYNC/Far Sync, seconds with ASYNC. Backups: back to your last backup plus available redo.

Get those two numbers agreed and most of the architecture chooses itself. Here's the tree I walk:

A practical RAC vs Data Guard vs Both decision tree. Backups + Flashback are mandatory in every branch.

A side-by-side, for the architecture review

Dimension	RAC	Data Guard	RAC + DG	Backups only
Node/instance failure	Yes (instant)	Partial (failover)	Yes (instant)	No
Site/region loss	No	Yes	Yes	Partial (slow)
Block corruption	No	Yes (ADG repair)	Yes	Yes (restore)
Human/logical error	No	No	No	Yes (Flashback/PITR)
Typical RTO	seconds	seconds–minutes	seconds	hours
Typical RPO	0	0 (SYNC) / seconds (ASYNC)	0	last backup
Read offload	Yes (all nodes)	Yes (Active DG)	Yes	No
Rolling patching	Yes	Yes (standby-first)	Yes	No
Scale-out writes	Yes	No	Yes	No
Cost beyond EE	RAC option ($$)	included; ADG extra	both ($$$)	none
Operational complexity	high	medium	highest	low

Where GoldenGate fits

GoldenGate is the other tool people reach for, and it's worth knowing why it's not usually the answer
to this particular question. It does logical replication — capturing changes and applying them
elsewhere — which makes it brilliant for things Data Guard can't do: heterogeneous targets, cross-version
and near-zero-downtime migrations and upgrades, active-active multi-master, and replicating a subset
of the data. But it's a separately licensed option, it's operationally heavier, and for plain "keep an
identical standby for DR," physical Data Guard is simpler and tighter. Use GoldenGate when you need its
logical flexibility (it's a Platinum-tier component for a reason) — not as a default DR mechanism.

A worked switchover (planned, zero data loss)

Choosing the architecture is half the job; the other half is being able to operate it under pressure.
A switchover is a planned, lossless role reversal — the primary becomes a standby and a standby
becomes the primary. You'll do this for site maintenance, hardware refreshes, and — critically — as the
rehearsal that proves your DR actually works. Always drive it through the Broker.

Step 1 — Validate before you touch anything. Modern Broker gives you a pre-flight check that catches
gaps, missing standby redo logs, and flashback problems before you commit:

DGMGRL> SHOW CONFIGURATION;          -- expect: Status SUCCESS
DGMGRL> VALIDATE DATABASE 'ORCLCDB_STBY';

A healthy result looks roughly like this (trimmed):

  Database Role:       Physical standby database
  Primary Database:    ORCLCDB
  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)
  Flashback Database Status:
    ORCLCDB       : On
    ORCLCDB_STBY  : On
  Transport-Related Information:
    Transport lag:   +00 00:00:00
  Apply-Related Information:
    Apply lag:       +00 00:00:00

If "Ready for Switchover" isn't Yes, stop and fix that first — usually an archive gap, missing
standby redo logs, or apply lag.

Step 2 — Switch over. One command; the Broker orchestrates both databases:

DGMGRL> SWITCHOVER TO 'ORCLCDB_STBY';

Step 3 — Verify the new roles and that redo is flowing the other way:

-- On the NEW primary (formerly the standby)
SELECT database_role, open_mode, switchover_status FROM v$database;
-- DATABASE_ROLE should now be PRIMARY, OPEN_MODE READ WRITE

-- Confirm the configuration is healthy again
-- DGMGRL> SHOW CONFIGURATION;   -> Status SUCCESS

Step 4 — Redirect the application. This is the step people forget. Clients need to land on the new
primary — via a role-based service that only starts in the PRIMARY role, or via a connect string that
lists both hosts. Test it, don't assume it.

Failover (unplanned) and reinstate

A failover is what you run when the primary is gone and not coming back soon. It's faster and more
decisive than a switchover, and with asynchronous transport it may cost you a small amount of redo (your
RPO):

DGMGRL> FAILOVER TO 'ORCLCDB_STBY';

With Fast-Start Failover enabled, you don't type that at all — the Observer detects the outage and
promotes the standby automatically, typically in seconds. Either way, when the old primary comes back to
life, you don't rebuild it from scratch: if it had Flashback Database enabled, the Broker can rewind and
re-enrol it as the new standby in one step:

DGMGRL> REINSTATE DATABASE 'ORCLCDB';

That Flashback-Database prerequisite is exactly why "enable Flashback on both databases" belongs in your
standard build — without it, a failover turns a returning primary into a full rebuild.

Monitoring: what to watch, and when to page

A standby silently falling behind is the classic way DR rots. You need two numbers alarmed at all times —
transport lag (redo not yet received) and apply lag (redo received but not yet applied) — plus
the health of the apply process and, if you use it, the FSFO state.

-- The two numbers that define your real-world RPO/RTO right now
SELECT name, value AS lag, time_computed
FROM   v$dataguard_stats
WHERE  name IN ('transport lag','apply lag');

-- Is the apply process actually running? (run on the standby)
SELECT process, status, sequence#
FROM   gv$managed_standby
WHERE  process LIKE 'MRP%';

-- Fast-Start Failover health (run on the primary)
SELECT fs_failover_status, fs_failover_current_target, fs_failover_observer_present
FROM   v$database;

Sensible starting thresholds — tune them to your RPO/RTO, not these defaults:

Signal	Warning	Critical	Why it matters
Transport lag	> 60s	> your RPO	Redo isn't reaching the standby — data-loss exposure
Apply lag	> 5 min	> your RTO	Standby is "behind"; failover would replay slowly
MRP process	not running	absent after retry	Apply has stopped — lag will grow unbounded
FSFO status	not SYNCHRONIZED / not within lag limit	observer absent	Automatic failover is not currently possible
Archive gap	any persistent gap	growing	A missing sequence blocks all further apply

Two operational notes: run the Observer on a third, independent host (not on either database server —
otherwise the thing that watches for failure can die with the failure), and if you run Oracle Enterprise
Manager, its Data Guard metrics wrap all of the above in alerting so you're not hand-rolling every check.

One subtlety worth calling out: when apply lag grows but transport is healthy and there's no archive
gap, the standby itself is usually the bottleneck — redo is arriving but the apply can't keep up because
the standby is I/O- or CPU-bound. That's not a Data Guard problem, it's a performance problem, and you
diagnose it the same way you'd diagnose any slow database: pull an AWR report on the standby and read it.
If that's unfamiliar territory, start with How to Read an AWR Report Without
Drowning.

Troubleshooting the usual suspects

When Data Guard misbehaves, it's almost always one of a handful of patterns. The Broker surfaces these as
ORA-16xxx messages — always read the Broker's StatusReport for the specific code and its recommended
action rather than guessing:

DGMGRL> SHOW CONFIGURATION;                 -- look for WARNING/ERROR
DGMGRL> SHOW DATABASE 'ORCLCDB_STBY' StatusReport;

Symptom	Likely cause	Where to look	Typical fix
Apply lag climbing, sequence stuck	Archive gap — a missing redo sequence	`v$archive_gap`, `gv$archived_log`	Broker/FAL usually auto-resolves; if not, ship the missing logs and re-register
Standby block corruption after a bulk load	NOLOGGING operation on the primary	alert log, `v$database.force_logging`	`ALTER DATABASE FORCE LOGGING`; restore affected datafile from primary
Transport lag grows under load	Network throughput < redo rate	`v$dataguard_stats`, redo generation rate	Tune TCP/socket buffers, enable redo transport compression, or use Far Sync
Real-time apply won't start	Standby redo logs missing/undersized	`v$standby_log`	Add standby redo logs (one more group than online, same size)
Apply stopped after a failover test	Flashback off, can't reinstate	`v$database.flashback_on`	Enable Flashback Database; reinstate via the Broker

The meta-lesson: most "Data Guard is broken" tickets are really forcing logging wasn't set, standby
redo logs were never created, or the network can't keep up with peak redo. Get those three right at
build time and you'll prevent the majority of incidents.

Test it for real: a DR game-day

A standby you have never failed over to is a hope, not a plan — so put it on a schedule. A practical
cadence is a switchover every quarter (it's lossless and reversible) and a full failover drill at
least annually. To exercise the application against standby data without disturbing replication, use
a snapshot standby: it opens read-write for testing, then discards its changes and catches back up.

-- Open the standby read-write for application testing
DGMGRL> CONVERT DATABASE 'ORCLCDB_STBY' TO SNAPSHOT STANDBY;
-- ... run your app test suite against it ...
-- Roll it back and resume keeping pace with the primary
DGMGRL> CONVERT DATABASE 'ORCLCDB_STBY' TO PHYSICAL STANDBY;

A repeatable game-day runbook:

Announce the window and the rollback plan.
Pre-check with VALIDATE DATABASE (Ready for Switchover = Yes).
Execute the switchover (or failover, for the annual drill).
Verify the application actually reconnects through your role-based service — this is the test, not the database role itself.
Measure the real RTO and RPO and compare them to target. Numbers, not vibes.
Switch back and confirm the configuration returns to SUCCESS.
Report: measured RTO/RPO, every gap you hit, and the owner/date for each fix.

That report is also the artifact that turns "I think we're covered" into something leadership can
actually rely on — and it's how you find the decommissioned-host-in-the-runbook problem in a drill
instead of during a real outage.

Patching and upgrading without downtime

Here's the payoff most teams undersell: the biggest day-to-day return on HA isn't surviving disasters
— it's making planned maintenance nearly invisible. The same building blocks let you patch and
upgrade with little or no downtime, and that benefit cashes in every single patch cycle.

Rolling patches with RAC. Most quarterly Release Updates are RAC-rolling: you patch one node at a time while the others keep serving the database. Connections drain off the node you're working on (via services with a drain timeout, or Application Continuity) and return when it rejoins. No outage, just a brief capacity dip.
Standby-first patching. For patches that aren't RAC-rolling, Data Guard gives you another route: apply the patch to the standby first, verify it there, switch over to the patched standby, then patch the old primary. The application sees one short switchover instead of a maintenance window. (Oracle marks which patches are "Standby-First Installable.")
Major upgrades with DBMS_ROLLING. A full release upgrade (say 19c → 23ai) normally means real downtime. DBMS_ROLLING converts your physical standby into a transient logical standby, upgrades it while the primary keeps running, and then switches over — so the application's downtime collapses to a single switchover rather than the whole upgrade window:

-- sketch of a DBMS_ROLLING upgrade, driven from the primary
EXEC DBMS_ROLLING.INIT_PLAN(future_primary => 'ORCLCDB_STBY');
EXEC DBMS_ROLLING.BUILD_PLAN;
EXEC DBMS_ROLLING.START_PLAN;     -- standby becomes a transient logical standby
-- ... upgrade the transient logical standby to the new release ...
EXEC DBMS_ROLLING.SWITCHOVER;     -- the application flips to the upgraded database
EXEC DBMS_ROLLING.FINISH_PLAN;

The thread tying all three together: planned downtime is a choice, not a law of physics. If your
SLA can't spare a maintenance window, the HA you built for disasters quietly pays for itself every time
you patch.

Try it yourself: a runnable lab

Reading about recovery is one thing; doing it is what builds the reflex. I put together a small lab
you can run on a laptop with nothing but Docker — no Oracle account required — so you can feel the most
important lessons here first-hand. It uses the community Oracle Database Free image and runs
every command inside the container, so you don't even need a local Oracle client.

A quick honesty note about scope, because it maps exactly to this article:

RAC isn't something you can meaningfully run on a single laptop. It needs shared storage, a private interconnect, and clusterware across nodes — a real cluster, not a container trick. So the lab doesn't pretend to.
Data Guard is an Enterprise Edition feature, and the zero-login Free image doesn't include it. So the no-setup lab focuses on the failure modes you can reproduce — and which this post argues are the most commonly mishandled: human error, media loss, and corruption. A separate, opt-in Enterprise Edition module covers a real primary/standby switchover and failover for when you want to rehearse those too.

Getting started is three commands:

./run.sh up        # pulls the image and creates the database (first run takes a few minutes)
./run.sh setup     # enables archivelog and creates a small demo schema
./run.sh all       # runs all three drills end to end

The three drills, and the lesson each one drives home:

Human-error recovery. The lab deletes every row (committed) and then drops the table — two perfectly valid statements a standby would have replicated in milliseconds — and recovers both locally with Flashback Query and Flashback Table. This is the "replication is not a backup" point you can now prove to yourself (and to a skeptical colleague) in thirty seconds.
RMAN backup & restore. Take a backup, take a datafile offline and delete it from disk to simulate media failure, then restore and recover just that file while the rest of the database stays open. That's the restore-drill muscle this post keeps insisting you build.
Block-corruption detection & recovery. Write garbage into a single on-disk block, detect it with RMAN VALIDATE CHECK LOGICAL, and repair it with block media recovery — no full restore needed.

The full lab — the docker compose file, the run.sh driver, every drill script, and the optional
Enterprise Edition Data Guard module — is the ha/ lab in
github.com/pyaroslav/oracle-labs. Clone it, run it, break
things on purpose. (No spare RAM on your laptop? The repo includes a guide to run the whole thing
free on an OCI Always Free cloud VM.) Discovering that your runbook references a decommissioned host
is a great thing to learn in a lab on a Tuesday afternoon — and a terrible thing to learn at 2am.

What about 23ai and 26ai?

If you're on or moving to a newer release — 23ai, or the current 26ai — the good news is that
none of the decision framework above changes: the failure modes are the same, RAC still protects
compute, Data Guard still protects data, and backups + Flashback still own corruption and human error.
The "ai"-era releases continue the same Maximum Availability Architecture lineage and add incremental
improvements across the stack (redo transport/apply efficiency, manageability, and — notably in 23ai —
new in-database capabilities like AI Vector Search that change what you run, not how you protect
it). What does shift between releases is the small print: default parameter values, which features are
enabled, and option licensing. So when you implement on 23ai or 26ai, confirm the exact behavior and
licensing against that release's documentation rather than assuming 19c defaults carry over — and, if
you want a free place to check, the Oracle Database Free image (currently 26ai) and OCI Always
Free Autonomous Database both let you verify on a real instance at no cost.

What teams get wrong (the short list)

Treating RAC as DR. It isn't. One copy of data, one storage, one site.
An untested standby. If you haven't done a real switchover, you don't have DR — you have a theory. Schedule game-days.
Assuming replication protects against mistakes. A bad DELETE reaches the standby before you can cancel it. Flashback and backups are your safety net, every time.
Buying Gold when Bronze/Silver was the requirement. Match the MAA tier to a stated RTO/RPO, not to fear. Complexity you can't operate is a liability, not insurance.
Ignoring the licensing line. RAC and Active Data Guard are paid options. Design within what you're actually licensed for, or get the budget approved on purpose.

Frequently asked questions

Is Oracle RAC a disaster recovery solution?

No. RAC protects against instance and node failure by running multiple instances against one shared copy of the database. Because there is only one copy of the data on shared storage, a site outage, storage failure, or block corruption affects all RAC nodes at once. Disaster recovery requires an independent copy, which is what Data Guard provides.

Do I still need Data Guard if I already have RAC?

Yes, if you need to survive losing a site or region, or to protect against data corruption. RAC and Data Guard solve different failures: RAC handles local node failure, while Data Guard maintains a separate standby database for site loss and corruption protection. Many mission-critical systems run both.

Does Data Guard protect against accidental data deletion?

No. An accidental DELETE or DROP is a valid transaction, so Data Guard faithfully ships and applies it to the standby within seconds. Protection against human and logical errors comes from Flashback Database, Flashback Table, guaranteed restore points, and RMAN point-in-time recovery — not from replication.

What is the difference between a switchover and a failover?

A switchover is a planned, lossless role reversal between the primary and standby, used for maintenance and DR testing. A failover is an unplanned promotion of the standby when the primary is lost; with asynchronous transport it may incur a small amount of data loss. Fast-Start Failover can perform failovers automatically.

Is Data Guard included with Oracle Enterprise Edition?

Basic Data Guard — a physical standby in mount mode doing Redo Apply — is included with Enterprise Edition. Active Data Guard, which adds a read-only open standby, Automatic Block Media Recovery, and Far Sync, is a separately licensed option. RAC is also a separately licensed option.

What RPO can Data Guard achieve?

Zero data loss is achievable using synchronous redo transport in Maximum Availability or Maximum Protection mode, optionally with a Far Sync instance to preserve zero RPO over long distances. Asynchronous transport (Maximum Performance) typically loses only seconds of redo but adds no commit latency on the primary.

What is the difference between RAC and RAC One Node?

Full RAC runs multiple active instances across nodes for both high availability and scale-out. RAC One Node runs a single active instance that Oracle Clusterware can fail over or online-relocate to another node, with rolling patching. RAC One Node offers most of the availability benefit with less complexity, and can be scaled up to full RAC later.

What is Oracle Maximum Availability Architecture (MAA)?

MAA is Oracle's set of best-practice reference architectures for high availability and disaster recovery, organized into tiers: Bronze (a single instance with RMAN backups and Flashback), Silver (adds RAC or RAC One Node for local failure), Gold (adds Active Data Guard for site loss and corruption), and Platinum (adds GoldenGate, Application Continuity, and Edition-Based Redefinition for zero-downtime maintenance). You choose the lowest tier that meets your RTO and RPO targets.

What is an Oracle Data Guard Far Sync instance?

A Far Sync instance is a lightweight Data Guard member — just a control file and redo, no datafiles — placed close to the primary. The primary ships redo to it synchronously (zero data loss, low latency), and Far Sync forwards that redo asynchronously to a distant standby. This achieves zero-data-loss protection (RPO near zero) across long geographic distances without the commit latency that synchronous transport directly to a far-away standby would impose.

The one-paragraph version

Set RTO and RPO with the business. Use RAC (or RAC One Node) to survive instance and node failure
at a site with no downtime. Use Data Guard to survive site loss and corruption, with Fast-Start
Failover for automatic recovery and Far Sync if you need zero data loss over distance. Use both —
MAA Gold — only when your targets genuinely demand it. And in every design, no exceptions, keep RMAN
backups and Flashback Database, because that's the only thing that saves you from the failure RAC and
Data Guard can't: the human one.

Originally published at uptimearchitect.com.