<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Uptime Architect</title>
    <description>The latest articles on DEV Community by Uptime Architect (@uptimearchitect).</description>
    <link>https://dev.to/uptimearchitect</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3973084%2Feafab265-85f4-404d-a6a8-1322e9be60e1.png</url>
      <title>DEV Community: Uptime Architect</title>
      <link>https://dev.to/uptimearchitect</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/uptimearchitect"/>
    <language>en</language>
    <item>
      <title>ORA-04068 After a Deploy: The Retry Works. The Package State Is Gone.</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Wed, 18 Nov 2026 20:46:13 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/ora-04068-after-a-deploy-the-retry-works-the-package-state-is-gone-29ah</link>
      <guid>https://dev.to/uptimearchitect/ora-04068-after-a-deploy-the-retry-works-the-package-state-is-gone-29ah</guid>
      <description>&lt;p&gt;The release went fine. Every script ran, every package compiled, the smoke test passed. A minute later the application log fills with this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-04068: existing state of packages has been discarded
ORA-04061: existing state of package body "APP.PRICING" has been invalidated
ORA-04065: not executed, altered or dropped package body "APP.PRICING"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then it stops. A few dozen errors, one per connection, and nothing after. Someone opens an incident, someone else suggests rolling back, and by the time anyone looks, the errors are gone.&lt;/p&gt;

&lt;p&gt;The package isn't broken. It holds state, the deploy threw that state away, and Oracle told each session about it the first time it came back. The error is the visible part. The quieter part is that the retry works with the state reset, and not only for the package you deployed.&lt;/p&gt;

&lt;p&gt;This post covers what the error means, which packages and which changes raise it (two of them look harmless), what a connection pool sees, and how to deploy PL/SQL without it. It ends with a lab that proves each step on Oracle Database Free 26ai.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the error means
&lt;/h2&gt;

&lt;p&gt;Each session that uses a package gets its own copy of the package's variables: its &lt;strong&gt;package state&lt;/strong&gt;. A counter, a cached config row, a flag set at login, a lookup table loaded once. Oracle's &lt;a href="https://docs.oracle.com/en/database/oracle/oracle-database/26/lnpls/package-state.html" rel="noopener noreferrer"&gt;PL/SQL Language Reference&lt;/a&gt; calls a package &lt;strong&gt;stateful&lt;/strong&gt; if it declares at least one variable, constant or cursor, and &lt;strong&gt;stateless&lt;/strong&gt; otherwise. That state lives for the whole session.&lt;/p&gt;

&lt;p&gt;Except when the package body is recompiled. A deploy (&lt;code&gt;CREATE OR REPLACE PACKAGE BODY&lt;/code&gt;), an explicit &lt;code&gt;ALTER PACKAGE ... COMPILE&lt;/code&gt;, or an automatic recompile after something the package depends on changed. The state in every session was built by the old code, so it can't be kept. The next time a session calls the package, Oracle discards that state and raises ORA-04068. The two errors under it say which package, and why.&lt;/p&gt;

&lt;p&gt;In the lab, a session called a package with one variable, a call counter, twice: 1, 2. The deploy replaced the package body. The session's next call failed with exactly the three errors above. The call after that worked, ran the new code, and returned &lt;strong&gt;1&lt;/strong&gt;. The counter started again from its initial value.&lt;/p&gt;

&lt;p&gt;Two things the error does &lt;strong&gt;not&lt;/strong&gt; do, both checked in the lab:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It doesn't roll back the session's transaction.&lt;/strong&gt; The session had an uncommitted insert before the deploy. It was still there after the error.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It doesn't come back.&lt;/strong&gt; It's raised once per session per recompile. The next call re-creates the package state from scratch and runs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  It's the state, not the change
&lt;/h2&gt;

&lt;p&gt;Whether a deploy raises ORA-04068 doesn't depend on how big the change is. It depends on whether the package holds state when it's recompiled. In the lab, a session used each package, one change was deployed, and the session called the package again:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Package&lt;/th&gt;
&lt;th&gt;Change deployed&lt;/th&gt;
&lt;th&gt;Next call&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;no package variables&lt;/td&gt;
&lt;td&gt;new body&lt;/td&gt;
&lt;td&gt;no error, new code at once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;no package variables&lt;/td&gt;
&lt;td&gt;new spec (a function added) and body&lt;/td&gt;
&lt;td&gt;no error, new code at once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;only a literal constant (&lt;code&gt;c_rate CONSTANT NUMBER := 5&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;new body&lt;/td&gt;
&lt;td&gt;no error, new code at once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a constant set at load (&lt;code&gt;c_loaded CONSTANT DATE := SYSDATE&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;new body&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;ORA-04068&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;one variable&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ALTER PACKAGE ... COMPILE BODY&lt;/code&gt;, no code change&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;ORA-04068&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a &lt;code&gt;%ROWTYPE&lt;/code&gt; variable on &lt;code&gt;ORDERS&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ALTER TABLE orders ADD&lt;/code&gt; a column&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;ORA-04068&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a variable, but the code names its columns&lt;/td&gt;
&lt;td&gt;the same &lt;code&gt;ADD&lt;/code&gt; column&lt;/td&gt;
&lt;td&gt;no error&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three of these surprise people.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A constant isn't always a constant.&lt;/strong&gt; Oracle treats a package as stateless if its state can't change for the life of a session, which is the case when every item is a compile-time constant. A literal is one. &lt;code&gt;SYSDATE&lt;/code&gt; isn't: its value is set when the session first uses the package. So &lt;code&gt;c_loaded CONSTANT DATE := SYSDATE&lt;/code&gt; makes the package stateful, and the deploy that changed nothing else raised ORA-04068.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A recompile with no change is enough.&lt;/strong&gt; &lt;code&gt;ALTER PACKAGE pricing COMPILE BODY&lt;/code&gt;, with the exact same code, raised it. So does any script that recompiles invalid objects after a release, for every stateful package it touches.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DDL on a table can do it, with no PL/SQL deploy at all.&lt;/strong&gt; A package body with a variable declared as &lt;code&gt;orders%ROWTYPE&lt;/code&gt; depends on the table's column list. &lt;code&gt;ALTER TABLE orders ADD&lt;/code&gt; a column invalidated it, and the next call raised ORA-04068. A package that used the same table but named its columns stayed &lt;code&gt;VALID&lt;/code&gt; and didn't raise it. So the &lt;a href="https://uptimearchitect.com/blog/oracle-ora-00054-resource-busy-ddl/" rel="noopener noreferrer"&gt;ADD COLUMN that waited politely for its lock&lt;/a&gt; can still throw errors at the application once it runs.&lt;/p&gt;

&lt;p&gt;The other direction also held. A package with no variables switched to the new code silently, even when its spec changed. In a hand run, a standalone function that called that package was invalidated by the spec change, recompiled itself on its next call, and also raised nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  It resets packages the release didn't touch
&lt;/h2&gt;

&lt;p&gt;The documentation has one more sentence that's easy to read past: all of a session's package states can be lost if any of its instantiated packages is invalidated and revalidated.&lt;/p&gt;

&lt;p&gt;The lab tested it. A session used two packages, &lt;code&gt;PRICING&lt;/code&gt; and &lt;code&gt;AUDIT_CTX&lt;/code&gt;, each with its own counter. The release recompiled only &lt;code&gt;AUDIT_CTX&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PRICING   (not redeployed): 1, 2, 3        then, after AUDIT_CTX's error: 1
AUDIT_CTX (recompiled):     1, ORA-04068, 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;PRICING&lt;/code&gt; kept counting until the session called &lt;code&gt;AUDIT_CTX&lt;/code&gt; and got ORA-04068. After that, &lt;code&gt;PRICING&lt;/code&gt; was back to 1 too. Its code hadn't changed and nobody had deployed it, but its state was gone. (In a hand run, recompiling a package &lt;strong&gt;without&lt;/strong&gt; variables didn't do this: the session's other packages kept their state.)&lt;/p&gt;

&lt;p&gt;So "we only changed one package" doesn't limit the damage. Any session that hits ORA-04068 for any package starts again with every package's state at its initial value. If one of them caches a user's permissions, a tenant ID or a feature flag, the retry that "fixed" the error runs with the defaults.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpdklFrGzEQhP_KoJe0cE7zUEoxJcHB91BiamKn0NArRT6tY9WyZLS6uIfj_549-eyGvh3LzXyzs9qrOhhSQ6ilC7t6pWPCw7jywPhnpca0daFFiIhUh83WOvqyiB-uNba6XusnwiKYtlK_MBhc434vCmuQVpbBxGyDR8Mkg6OsFxXQ3sAEYtiEVXAGnHSim0odOvJ95_ZSKR-G8AHPOlq9cMRFNpEwwbsWziaK2qEOXtQ-caVeML2T1N_ob0KtnUNsPHd0eNqhWzQ7iCfFGKLkfotriYfQZ5ykRN1EDrHodz6RZLeEZQwbzB_n49FD2ZHLt-AhprPR4Orj1afPWfsu-JqwpXiq5X3PLnNxM9HOKMUWuxDXXGDRJIwmE4TlubteeMHn6nNnkKojSaCiXy3hT8PpeIP-zwvuaT-EMy9nX8X68fes_D4f3U7K3HHu42jRsY_WTnNiKZvyTvnKl5J2epeT33apS2OTpBosNJORIIaW1ufRMJs5-0ynnRlrom0-R3fy8zm6xyBDG4_Y_zi1pGD5xlMI5t9A8HmiCqgNxY22Rh7xXonRJj9nSaIbl9Th8AprU_bM" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpdklFrGzEQhP_KoJe0cE7zUEoxJcHB91BiamKn0NArRT6tY9WyZLS6uIfj_549-eyGvh3LzXyzs9qrOhhSQ6ilC7t6pWPCw7jywPhnpca0daFFiIhUh83WOvqyiB-uNba6XusnwiKYtlK_MBhc434vCmuQVpbBxGyDR8Mkg6OsFxXQ3sAEYtiEVXAGnHSim0odOvJ95_ZSKR-G8AHPOlq9cMRFNpEwwbsWziaK2qEOXtQ-caVeML2T1N_ob0KtnUNsPHd0eNqhWzQ7iCfFGKLkfotriYfQZ5ykRN1EDrHodz6RZLeEZQwbzB_n49FD2ZHLt-AhprPR4Orj1afPWfsu-JqwpXiq5X3PLnNxM9HOKMUWuxDXXGDRJIwmE4TlubteeMHn6nNnkKojSaCiXy3hT8PpeIP-zwvuaT-EMy9nX8X68fes_D4f3U7K3HHu42jRsY_WTnNiKZvyTvnKl5J2epeT33apS2OTpBosNJORIIaW1ufRMJs5-0ynnRlrom0-R3fy8zm6xyBDG4_Y_zi1pGD5xlMI5t9A8HmiCqgNxY22Rh7xXonRJj9nSaIbl9Th8AprU_bM" alt="A recompile of a stateful package doesn't fail the deploy. It fails the next call in every" width="842" height="1022"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A recompile of a stateful package doesn't fail the deploy. It fails the next call in every session that had used the package, once, and that session's retry starts with every package's state at its initial value. Packages without state, SERIALLY_REUSABLE packages and edition-based redefinition avoid the error.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What a connection pool sees
&lt;/h2&gt;

&lt;p&gt;A connection pool turns one recompile into a burst. Each pooled connection is a session, so each one that had used the package raises ORA-04068 once.&lt;/p&gt;

&lt;p&gt;The lab opened 8 sessions; 6 of them called the package, 2 didn't. After one deploy, every session called it twice:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;first round after the deploy:  6 of 8 calls failed with ORA-04068
second round:                  0 of 8
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The 2 sessions that had never used the package got the new code with no error. The 6 that had used it failed once each, even though the call they made after the deploy only read a version string and never touched the variable. What's discarded is the session's copy of the package, not one variable.&lt;/p&gt;

&lt;p&gt;That's the shape to recognize in a log: a burst right after a deploy, about one error per connection, then nothing. A pool of 200 connections gives you up to 200 errors and 200 resets of the package state. Unless your code retries, each of those errors is a failed user request.&lt;/p&gt;

&lt;h2&gt;
  
  
  Find the stateful packages before you deploy
&lt;/h2&gt;

&lt;p&gt;There's no column in the data dictionary that says "this package is stateful", but PL/Scope can tell you. Compile with identifier collection on, then list the declarations at package level:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;SESSION&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;plscope_settings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'IDENTIFIERS:ALL'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;PACKAGE&lt;/span&gt; &lt;span class="n"&gt;pricing&lt;/span&gt; &lt;span class="no"&gt;COMPILE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;          &lt;span class="c1"&gt;-- for each package you want to check&lt;/span&gt;

&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;object_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;object_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;type&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;user_identifiers&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;object_type&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'PACKAGE'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'PACKAGE BODY'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;usage&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'DECLARATION'&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;type&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'VARIABLE'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'CONSTANT'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'CURSOR'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;usage_context_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;               &lt;span class="c1"&gt;-- declared at package level, not inside a subprogram&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;object_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;object_type&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In a hand run on 26ai it listed the counter variable, a package-level cursor, and both constants. It didn't list the packages without package-level declarations. Check the &lt;code&gt;CONSTANT&lt;/code&gt; rows by eye: a literal (&lt;code&gt;:= 5&lt;/code&gt;) is stateless, anything else (&lt;code&gt;:= SYSDATE&lt;/code&gt;, a function call) isn't. Note that &lt;code&gt;ALTER PACKAGE ... COMPILE&lt;/code&gt; is itself a recompile, so run this on a test database, not under a live application.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deploy without it
&lt;/h2&gt;

&lt;p&gt;There are four ways to deal with it. They're not equal.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Keep state out of packages.&lt;/strong&gt; A package without variables has nothing to discard. In the lab it took the new code with no error every time, spec change included. If a value must survive across calls, keep it somewhere a recompile doesn't touch: a table, or an application context (&lt;code&gt;SYS_CONTEXT&lt;/code&gt;), which belongs to the session rather than to a package. In a hand run, a tenant ID set in an application context was still there after ORA-04068 had reset the package counter in the same session. This is the real fix, and it means changing the code.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Retry once in the application.&lt;/strong&gt; The second call works, so catching ORA-04068 and retrying the call makes the burst invisible to users. It's cheap, and many teams do it. But the retry runs with every package's state reset, including packages the deploy didn't touch. Only retry if the code re-creates any state it needs, for example by re-running its login initialization.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. &lt;code&gt;PRAGMA SERIALLY_REUSABLE&lt;/code&gt;.&lt;/strong&gt; A &lt;a href="https://docs.oracle.com/en/database/oracle/oracle-database/26/lnpls/SERIALLY_REUSABLE-pragma.html" rel="noopener noreferrer"&gt;serially reusable package&lt;/a&gt; keeps its state only for the duration of one server call. In the lab, the deploy raised no error and the very next call ran the new code. The cost is in the name: the counter was 1 on every call, before and after the deploy, because nothing survives between calls. And a serially reusable package can't be called from SQL at all: &lt;code&gt;SELECT pricing.version FROM dual&lt;/code&gt; failed with &lt;code&gt;ORA-06534: Cannot access Serially Reusable package&lt;/code&gt;. It suits packages that only use their variables within one call. It isn't a switch you can turn on for a package that relies on its state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Edition-based redefinition (EBR).&lt;/strong&gt; &lt;a href="https://docs.oracle.com/en/database/oracle/oracle-database/26/adfns/editions.html" rel="noopener noreferrer"&gt;Editions&lt;/a&gt; let two versions of the same package exist at once. Each session sees the version in its own edition. The release is compiled in a new edition while running sessions keep using the old one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- once, as a DBA: enable editions for the schema owner (this can't be undone)&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;USER&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="no"&gt;ENABLE&lt;/span&gt; &lt;span class="no"&gt;EDITIONS&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- per release&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="no"&gt;EDITION&lt;/span&gt; &lt;span class="n"&gt;rel2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;GRANT&lt;/span&gt; &lt;span class="no"&gt;USE&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="no"&gt;EDITION&lt;/span&gt; &lt;span class="n"&gt;rel2&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- the deploy, in a session switched to the new edition&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;SESSION&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="no"&gt;EDITION&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rel2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;REPLACE&lt;/span&gt; &lt;span class="no"&gt;PACKAGE&lt;/span&gt; &lt;span class="no"&gt;BODY&lt;/span&gt; &lt;span class="n"&gt;pricing&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt; &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- when you're ready for new connections to get the release&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="no"&gt;EDITION&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rel2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab, a session in the old edition (&lt;code&gt;ORA$BASE&lt;/code&gt;) called the package before and after the new body was compiled in &lt;code&gt;REL2&lt;/code&gt;: 1, 2, 3. No error, no reset, still the old code. A new session also got the old code until &lt;code&gt;ALTER DATABASE DEFAULT EDITION = REL2&lt;/code&gt;; after that, new sessions got the new code. A connection pool moves to the release as its connections are replaced, and none of them sees ORA-04068 on the way.&lt;/p&gt;

&lt;p&gt;EBR is in every edition of the database, Free included. The catch is preparation: &lt;code&gt;ENABLE EDITIONS&lt;/code&gt; is one-way, and tables aren't editioned, so changes to table structure need editioning views and crossedition triggers to fully avoid downtime. For PL/SQL-only releases, it does exactly what this error needs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it
&lt;/h2&gt;

&lt;p&gt;The lab runs Oracle Database Free in a container. Each "user" session is a SQL*Plus session that calls the package, waits for the deploy, then calls it again:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;drill    case                                                 result
error    package body replaced under a live session           ORA-04068 / 04061 / 04065 once; the retry works,
                                                              counter back to 1; uncommitted insert kept
collat   only AUDIT_CTX recompiled, session also uses PRICING PRICING reset to 1 too, after AUDIT_CTX's error
which    no variables (new body; new spec)                    no error, new code
         a literal constant                                   no error, new code
         CONSTANT DATE := SYSDATE                             ORA-04068
         ALTER PACKAGE ... COMPILE BODY, no code change       ORA-04068
         %ROWTYPE variable, then ALTER TABLE ... ADD          ORA-04068 (named columns: no error)
pool     8 sessions, 6 had used it, one deploy                6 errors, then 0
serial   PRAGMA SERIALLY_REUSABLE                             no error, new code at once; counter always 1;
                                                              ORA-06534 from SQL
edition  new body compiled in edition REL2                    live session: no error, 1, 2, 3 on the old code;
                                                              new sessions: new code after the default switch
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't take my word for it — run it.&lt;/strong&gt; The &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/package-state" rel="noopener noreferrer"&gt;package-state lab&lt;/a&gt; &lt;strong&gt;asserts&lt;/strong&gt; every row: the three-error stack and the counter starting again at 1 after the retry, with the uncommitted insert still there; the untouched package reset by another package's error; which packages and changes raise it and which don't; exactly 6 errors from the 6 pooled sessions that had used the package and none on the second round; the serially reusable package's missing error, missing state and &lt;code&gt;ORA-06534&lt;/code&gt;; and the live session in the old edition counting on with no error. If any of that doesn't reproduce, the run fails. Proven on every CI push.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What teams get wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rolling back a good release.&lt;/strong&gt; The burst looks like a broken deploy. It's one error per connection, and it's over by the time anyone looks. Rolling back is another recompile, and another burst.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Assuming the retry is clean.&lt;/strong&gt; It runs, but with every package's state reset, including packages the release didn't touch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Thinking &lt;code&gt;CONSTANT&lt;/code&gt; means stateless.&lt;/strong&gt; Only compile-time constants do. &lt;code&gt;:= SYSDATE&lt;/code&gt; makes the package stateful.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recompiling "just to be safe".&lt;/strong&gt; A no-change &lt;code&gt;COMPILE BODY&lt;/code&gt; raises it for every session that had used the package.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Forgetting that table DDL recompiles PL/SQL.&lt;/strong&gt; An &lt;code&gt;ADD&lt;/code&gt; column under a &lt;code&gt;%ROWTYPE&lt;/code&gt; package variable raises it, with no PL/SQL in the release.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Adding &lt;code&gt;SERIALLY_REUSABLE&lt;/code&gt; to make the error go away.&lt;/strong&gt; It does, along with the state, and SQL can no longer call the package.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What does ORA-04068 existing state of packages has been discarded mean?
&lt;/h3&gt;

&lt;p&gt;A session had used a PL/SQL package that holds state (package-level variables, cursors, or constants that aren't compile-time constants), and the package body was recompiled since: by a deploy, an ALTER PACKAGE ... COMPILE, or an automatic recompile after a dependency changed. Oracle can't keep state built by the old code, so on the session's next call it discards the state and raises ORA-04068, with ORA-04061 and ORA-04065 naming the package. The next call re-creates the state from scratch and works.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does ORA-04068 go away when I run the call again?
&lt;/h3&gt;

&lt;p&gt;Because the error is raised once per session per recompile. The failing call discards the session's package state; the next call re-instantiates the package with its initial values and runs the new code. In the companion lab the retry worked every time, but a counter the session had built up was back to 1. Any value the package had cached for the session is gone.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I prevent ORA-04068 after deploying a PL/SQL package?
&lt;/h3&gt;

&lt;p&gt;Keep state out of packages you redeploy (a package with no variables, or only literal constants, switched to new code with no error in the companion lab), retry the call once in the application if your code can re-create the state it needs, use PRAGMA SERIALLY_REUSABLE for packages that only need state within one call, or deploy with edition-based redefinition so running sessions keep the old version. In the lab, a session in the old edition saw no error and kept its state while the new code was compiled in a new edition.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can ORA-04068 happen for a package I didn't change?
&lt;/h3&gt;

&lt;p&gt;Yes, in two ways. A package is recompiled automatically when something it depends on changes: in the companion lab, ALTER TABLE ADD COLUMN invalidated a package with a %ROWTYPE variable on that table, and the next call raised ORA-04068. And once a session gets ORA-04068 for any package, it loses the state of all its packages: in the lab, a package the release never touched was reset to its initial value after another package's error.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does PRAGMA SERIALLY_REUSABLE fix ORA-04068?
&lt;/h3&gt;

&lt;p&gt;It avoids it: in the companion lab, a serially reusable package raised no error after its body was replaced, and the next call ran the new code. But its state lasts only for one server call, so a counter was 1 on every call, and it can't be called from SQL: a SELECT that called it failed with ORA-06534. Use it for packages that don't rely on state between calls.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does a package constant cause ORA-04068?
&lt;/h3&gt;

&lt;p&gt;Only if it isn't a compile-time constant. Oracle treats a package whose items are all compile-time constants as stateless, and a literal (CONSTANT NUMBER := 5) is one: in the companion lab that package switched to new code with no error. A constant set from SYSDATE or a function call gets its value at run time, so the package is stateful, and in the lab it raised ORA-04068 after a deploy.&lt;/p&gt;

&lt;p&gt;ORA-04068 after a deploy isn't a failed release. It's every session that had used a stateful package finding out that its state is gone, once, and starting again with all of its package state at the initial values. Know which of your packages hold state, keep it out of the ones you redeploy, and when you can't, compile the release in a new edition so the live sessions never notice. Then prove it with the &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/package-state" rel="noopener noreferrer"&gt;package-state lab&lt;/a&gt;, where one deploy fails 6 of 8 pooled connections once and resets a package nobody deployed.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://uptimearchitect.com/blog/oracle-ora-04068-package-state-discarded/" rel="noopener noreferrer"&gt;uptimearchitect.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>oracle</category>
      <category>database</category>
      <category>devops</category>
      <category>sql</category>
    </item>
    <item>
      <title>ORA-00054 Resource Busy: Find the Session, and Know Which DDL Will Wait</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Tue, 17 Nov 2026 14:44:56 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/ora-00054-resource-busy-find-the-session-and-know-which-ddl-will-wait-5eee</link>
      <guid>https://dev.to/uptimearchitect/ora-00054-resource-busy-find-the-session-and-know-which-ddl-will-wait-5eee</guid>
      <description>&lt;p&gt;The release script gets to its third statement and stops:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You run it again. Same error. The table isn't busy in any way you can see: nothing is running against it. The application is quiet, and the database is fine.&lt;/p&gt;

&lt;p&gt;Somewhere there's a session with an uncommitted change to that table. It might be one row, changed an hour ago, by an application that forgot to commit, a SQL Developer window someone left open, or a pooled connection with an open transaction. That session isn't doing anything. It's just holding its lock, and that's enough.&lt;/p&gt;

&lt;p&gt;This post covers what the error means, how to find that session when the message doesn't name it, and three things that surprise people: some DDL doesn't fail at all, it waits; &lt;code&gt;DDL_LOCK_TIMEOUT&lt;/code&gt; makes every other writer wait too; and on some databases the waiting kind fails again. It ends with a lab that proves each step on Oracle Database Free 26ai.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the error means
&lt;/h2&gt;

&lt;p&gt;DML takes a table-level lock (a &lt;code&gt;TM&lt;/code&gt; lock) for as long as its transaction is open. It's a shared mode: any number of sessions can update different rows of the same table at once. It's released at commit or rollback, not when the statement finishes.&lt;/p&gt;

&lt;p&gt;Most DDL on a table needs that table to itself, so it needs every transaction on the table to end first. By default it doesn't wait: &lt;a href="https://docs.oracle.com/en/database/oracle/oracle-database/26/refrn/DDL_LOCK_TIMEOUT.html" rel="noopener noreferrer"&gt;&lt;code&gt;DDL_LOCK_TIMEOUT&lt;/code&gt;&lt;/a&gt; is 0, which means "no wait", and the statement fails at once with ORA-00054. One uncommitted row is enough. So is a &lt;code&gt;SELECT ... FOR UPDATE&lt;/code&gt; that was never committed.&lt;/p&gt;

&lt;p&gt;On 26ai the message says more than it used to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-00054: Failed to acquire a lock (Type: "TM", Name: "DML", Description:
"Synchronizes accesses to an object") because it is currently held by another
session. The resource being locked can be identified by 73205 ("Table") and 0
("operation")
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The number is the table's &lt;code&gt;OBJECT_ID&lt;/code&gt;. In the lab it matched &lt;code&gt;DBA_OBJECTS.OBJECT_ID&lt;/code&gt; for the table every time. The message doesn't name the session holding the lock. On 21c (XE 21.3, in a hand run) the message was only the classic text, with no object number at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  Find the session
&lt;/h2&gt;

&lt;p&gt;The usual first query, &lt;code&gt;BLOCKING_SESSION&lt;/code&gt; in &lt;code&gt;V$SESSION&lt;/code&gt;, finds nothing here. That column is filled in for a session that is &lt;strong&gt;waiting&lt;/strong&gt;, and a DDL that failed at once isn't waiting. In the lab, while the blocker held its lock and the &lt;code&gt;TRUNCATE&lt;/code&gt; had already failed, no session had &lt;code&gt;BLOCKING_SESSION&lt;/code&gt; set.&lt;/p&gt;

&lt;p&gt;Ask the lock itself instead. &lt;code&gt;V$LOCKED_OBJECT&lt;/code&gt; lists every session holding a lock on an object:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;lo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;session_id&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;sid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;serial&lt;/span&gt;&lt;span class="c1"&gt;#, s.username, s.status, s.machine, s.program,&lt;/span&gt;
       &lt;span class="n"&gt;lo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;locked_mode&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start_time&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;transaction_started&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="nb"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last_call_et&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;idle_minutes&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$locked_object&lt;/span&gt; &lt;span class="n"&gt;lo&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;dba_objects&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;  &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;object_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;object_id&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$session&lt;/span&gt;   &lt;span class="n"&gt;s&lt;/span&gt;  &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;session_id&lt;/span&gt;
&lt;span class="k"&gt;LEFT&lt;/span&gt; &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$transaction&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;addr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;taddr&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;owner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'APP'&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;object_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'ORDERS'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;-- or: lo.object_id = 73205, from the message&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab it returned exactly one row: the blocker, &lt;code&gt;INACTIVE&lt;/code&gt;, &lt;code&gt;LOCKED_MODE&lt;/code&gt; 3 (row exclusive, an open DML transaction), with a transaction start time. That's the shape to look for: a session that isn't running anything, with a transaction that started a while ago. &lt;code&gt;MACHINE&lt;/code&gt; and &lt;code&gt;PROGRAM&lt;/code&gt; tell you whose it is.&lt;/p&gt;

&lt;p&gt;On RAC, query &lt;code&gt;GV$LOCKED_OBJECT&lt;/code&gt; and &lt;code&gt;GV$SESSION&lt;/code&gt; and join on &lt;code&gt;INST_ID&lt;/code&gt; too; the blocker can be on another instance.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then decide what to do with it
&lt;/h2&gt;

&lt;p&gt;You have two options, and they're not equal:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ask the owner to commit or roll back.&lt;/strong&gt; If it's a person with a forgotten window, this is the clean option.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;End the session.&lt;/strong&gt; &lt;code&gt;ALTER SYSTEM KILL SESSION 'sid,serial#' IMMEDIATE&lt;/code&gt;. In the lab the &lt;code&gt;TRUNCATE&lt;/code&gt; then ran at once, and the blocker's uncommitted change was gone: killing a session rolls its transaction back. If that was a real business change that someone thought was saved, it isn't. Find out what it was before you kill it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If it's an application connection, ending it fixes tonight's deploy, but the code path that leaves transactions open is still there. Sessions orphaned by &lt;a href="https://uptimearchitect.com/blog/oracle-ora-03113-idle-connection-firewall/" rel="noopener noreferrer"&gt;a dropped connection&lt;/a&gt; can hold the same lock, with nobody left to commit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Not all DDL fails: some of it waits
&lt;/h2&gt;

&lt;p&gt;Oracle splits DDL into two kinds. Both wait for the transactions already open on the table. They differ in what happens to everyone else while they wait:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Blocking DDL&lt;/strong&gt; makes new DML on the table wait behind it, and gives up with ORA-00054 when &lt;code&gt;DDL_LOCK_TIMEOUT&lt;/code&gt; runs out (at once, with the default of 0). &lt;code&gt;TRUNCATE&lt;/code&gt;, &lt;code&gt;DROP&lt;/code&gt;, &lt;code&gt;CREATE INDEX&lt;/code&gt;, &lt;code&gt;MOVE&lt;/code&gt;, &lt;code&gt;MODIFY&lt;/code&gt; and &lt;code&gt;DROP COLUMN&lt;/code&gt; are blocking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nonblocking DDL&lt;/strong&gt; lets new DML run, and &lt;strong&gt;ignores &lt;code&gt;DDL_LOCK_TIMEOUT&lt;/code&gt;&lt;/strong&gt;. It waits as long as the transaction stays open. The &lt;code&gt;ONLINE&lt;/code&gt; operations are nonblocking (&lt;code&gt;CREATE INDEX ... ONLINE&lt;/code&gt;, &lt;code&gt;ALTER INDEX ... REBUILD ONLINE&lt;/code&gt;, &lt;code&gt;ALTER TABLE ... MOVE ONLINE&lt;/code&gt;), and so, since release 23.3, is a plain &lt;code&gt;ALTER TABLE ... ADD&lt;/code&gt; column.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both behaviors are documented: the &lt;a href="https://docs.oracle.com/en/database/oracle/oracle-database/26/adfns/sql-processing-for-application-developers.html" rel="noopener noreferrer"&gt;Development Guide&lt;/a&gt; explains the two kinds, and the &lt;a href="https://docs.oracle.com/en/database/oracle/oracle-database/26/sqlrf/Automatic-and-Manual-Locking-Mechanisms-During-SQL-Operations.html" rel="noopener noreferrer"&gt;SQL Language Reference&lt;/a&gt; lists the nonblocking statements by release. Here is what 26ai did when I ran each statement by hand against one idle session holding one uncommitted row, with &lt;code&gt;DDL_LOCK_TIMEOUT&lt;/code&gt; at 0. The lab asserts the &lt;code&gt;TRUNCATE&lt;/code&gt;, &lt;code&gt;CREATE INDEX&lt;/code&gt;, &lt;code&gt;ADD&lt;/code&gt; column, &lt;code&gt;CREATE INDEX ... ONLINE&lt;/code&gt; and &lt;code&gt;DROP COLUMN&lt;/code&gt; cases on every run:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Statement&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;TRUNCATE TABLE&lt;/code&gt;, &lt;code&gt;DROP TABLE&lt;/code&gt;, &lt;code&gt;LOCK TABLE ... NOWAIT&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ORA-00054 at once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CREATE INDEX&lt;/code&gt;, &lt;code&gt;ALTER INDEX ... REBUILD&lt;/code&gt;, &lt;code&gt;DROP INDEX&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ORA-00054 at once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ALTER TABLE ... MOVE&lt;/code&gt;, &lt;code&gt;MODIFY&lt;/code&gt;, &lt;code&gt;RENAME COLUMN&lt;/code&gt;, &lt;code&gt;DROP COLUMN&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ORA-00054 at once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ADD CONSTRAINT ... CHECK&lt;/code&gt; (validated), &lt;code&gt;DROP PRIMARY KEY&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ORA-00054 at once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ALTER TABLE ... ADD&lt;/code&gt; a column (nullable, with a default, or &lt;code&gt;NOT NULL&lt;/code&gt; with a default)&lt;/td&gt;
&lt;td&gt;waited for the commit, then ran&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CREATE INDEX ... ONLINE&lt;/code&gt;, &lt;code&gt;REBUILD ONLINE&lt;/code&gt;, &lt;code&gt;DROP INDEX ... ONLINE&lt;/code&gt;, &lt;code&gt;MOVE ONLINE&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;waited for the commit, then ran&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;SET UNUSED COLUMN ... ONLINE&lt;/code&gt;, &lt;code&gt;ADD CONSTRAINT ... ENABLE NOVALIDATE&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;waited for the commit, then ran&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;COMMENT ON TABLE&lt;/code&gt;, &lt;code&gt;GRANT&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ran at once: no table lock needed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;(I also ran the &lt;code&gt;ADD&lt;/code&gt; column and &lt;code&gt;CREATE INDEX ... ONLINE&lt;/code&gt; cases once by hand on the &lt;code&gt;gvenzl/oracle-xe:21-slim&lt;/code&gt; image, XE 21.3, and on Free 23.5. Both waited there too, and &lt;code&gt;TRUNCATE&lt;/code&gt; failed at once; only the message text differed. So the plain &lt;code&gt;ADD&lt;/code&gt; column waited on 21c even though the documentation lists it under 23.3. Don't treat the release number as the boundary: test the statement on your own version.)&lt;/p&gt;

&lt;p&gt;The waiting kind is gentler on the application: in the lab, while &lt;code&gt;ADD&lt;/code&gt; and &lt;code&gt;CREATE INDEX ... ONLINE&lt;/code&gt; waited, another session's &lt;code&gt;UPDATE&lt;/code&gt; on the table ran at once. But it has its own failure mode. With &lt;code&gt;DDL_LOCK_TIMEOUT = 5&lt;/code&gt;, &lt;code&gt;CREATE INDEX&lt;/code&gt; gave up after 5 seconds, as asked, while &lt;code&gt;ADD&lt;/code&gt; and &lt;code&gt;CREATE INDEX ... ONLINE&lt;/code&gt; ignored the timeout and waited the full 28 seconds until the blocker committed. If the blocker never commits, they never finish. A deploy that hangs at an &lt;code&gt;ADD&lt;/code&gt; column for an hour has met an idle transaction, not a slow statement.&lt;/p&gt;

&lt;p&gt;While it waits, the waiting DDL &lt;strong&gt;does&lt;/strong&gt; show up in &lt;code&gt;V$SESSION&lt;/code&gt;: event &lt;code&gt;enq: TX - row lock contention&lt;/code&gt;, with the blocker in &lt;code&gt;BLOCKING_SESSION&lt;/code&gt;. That's the one case where the usual query works.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNplkluP0zAQhf_KkcVDK9JsxcJLgZW6cSoVckElXUAEVW7iJlYdu4od-tDtf8dJWK5PluUzZ745ngspdMnJAuQg9bmoWWuR0VwB2decfOTGCK1wFrYGU-hUoZtGWMtLOK2q-Jt9e3NXa1kaMGQxpC6OcBW25rBsL3lOvmE2uwO95ITS6J-3a9-J9oLHnOz7YqGqBbLNNgmWWegh2ITuxDqh4WdvaEY36QfcIE7pevUFQRpt48Rz14cwJ4-4d9SfmLAG3QlWw7XcRWnwfpet4zDdZoPFZI63UBpnJ5x6PY9CulnO5vP5q5evB4niZ9A4GiQGe14LVUJYN82fxEqr39C-7yNNonUSjqBLSlFo2TUKkxe3_u3zaQ-Y_AI86HaIYozUQ63P_DtvXYSqGiFEpXTLzX9D_I3Ydsr85Eow8wcw051OkjdcWSadYVU5Qhf9kFAvXDmK1TjSAg_PevOQ7tL7d2GQjfRDKE9ojvQopHRqTFotpUuEFcfp099mvWUhmTFYodK6JB5Iw9uGidKt1oU4s2ZYspIfWCctuV5_ALUtx1Q" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNplkluP0zAQhf_KkcVDK9JsxcJLgZW6cSoVckElXUAEVW7iJlYdu4od-tDtf8dJWK5PluUzZ745ngspdMnJAuQg9bmoWWuR0VwB2decfOTGCK1wFrYGU-hUoZtGWMtLOK2q-Jt9e3NXa1kaMGQxpC6OcBW25rBsL3lOvmE2uwO95ITS6J-3a9-J9oLHnOz7YqGqBbLNNgmWWegh2ITuxDqh4WdvaEY36QfcIE7pevUFQRpt48Rz14cwJ4-4d9SfmLAG3QlWw7XcRWnwfpet4zDdZoPFZI63UBpnJ5x6PY9CulnO5vP5q5evB4niZ9A4GiQGe14LVUJYN82fxEqr39C-7yNNonUSjqBLSlFo2TUKkxe3_u3zaQ-Y_AI86HaIYozUQ63P_DtvXYSqGiFEpXTLzX9D_I3Ydsr85Eow8wcw051OkjdcWSadYVU5Qhf9kFAvXDmK1TjSAg_PevOQ7tL7d2GQjfRDKE9ojvQopHRqTFotpUuEFcfp099mvWUhmTFYodK6JB5Iw9uGidKt1oU4s2ZYspIfWCctuV5_ALUtx1Q" alt="An open transaction on the table blocks DDL. Blocking DDL fails with ORA-00054 when DDL_LO" width="456" height="1078"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;An open transaction on the table blocks DDL. Blocking DDL fails with ORA-00054 when DDL_LOCK_TIMEOUT runs out, and makes new DML wait while it waits. Nonblocking DDL (ONLINE operations; ADD column since 23.3) waits for the commit, ignores the timeout, and lets DML through, unless supplemental logging is on.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  DDL_LOCK_TIMEOUT makes everyone else wait too
&lt;/h2&gt;

&lt;p&gt;The common advice for ORA-00054 is to set &lt;code&gt;DDL_LOCK_TIMEOUT&lt;/code&gt; so the DDL waits instead of failing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;SESSION&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;ddl_lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For blocking DDL, that wait has a cost the advice usually leaves out. While a blocking DDL waits for its lock, new DML on the table queues behind it. Not just DML on the blocker's rows: any row.&lt;/p&gt;

&lt;p&gt;The lab ran &lt;code&gt;ALTER TABLE ... DROP COLUMN&lt;/code&gt; with &lt;code&gt;DDL_LOCK_TIMEOUT = 30&lt;/code&gt; while the blocker kept its row for 45 seconds. A third session then updated a &lt;strong&gt;different&lt;/strong&gt; row, one nobody had touched. It waited &lt;strong&gt;27 seconds&lt;/strong&gt;: &lt;code&gt;BLOCKING_SESSION&lt;/code&gt; pointed at the DDL, and &lt;code&gt;FINAL_BLOCKING_SESSION&lt;/code&gt; at the idle session. Then the DDL timed out with ORA-00054 anyway, and the update ran.&lt;/p&gt;

&lt;p&gt;So the timeout turned one failed deploy statement into 27 seconds of stalled writes for the application, and the deploy still failed. On a busy table, every write that arrives during the wait stalls the same way.&lt;/p&gt;

&lt;p&gt;The parameter is still useful. Keep it short, a few seconds, so the stall is short too, and retry the DDL a few times rather than waiting long once. Better still, check for open transactions on the table before running it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Supplemental logging brings the error back
&lt;/h2&gt;

&lt;p&gt;There's a catch in the nonblocking list: some nonblocking DDL is downgraded to blocking when supplemental logging is on. Supplemental logging is what GoldenGate, LogMiner-based capture tools and logical standby need, so it's on in exactly the databases where replication makes deploys delicate.&lt;/p&gt;

&lt;p&gt;The lab turned on minimal database-level supplemental logging (&lt;code&gt;ALTER DATABASE ADD SUPPLEMENTAL LOG DATA&lt;/code&gt;) and ran the same &lt;code&gt;ADD&lt;/code&gt; column against the same idle session. This time it failed at once with ORA-00054. &lt;code&gt;CREATE INDEX ... ONLINE&lt;/code&gt; still waited.&lt;/p&gt;

&lt;p&gt;So the same deploy script can wait politely in test and fail in production, if only production feeds a replication tool. Check before you rely on the waiting behavior:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;supplemental_log_data_min&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$database&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  A safer way to run DDL on a live table
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Look first.&lt;/strong&gt; Run the &lt;code&gt;V$LOCKED_OBJECT&lt;/code&gt; query for the tables the release touches. Any session holding a lock on them will make the DDL fail, or wait.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prefer the online forms&lt;/strong&gt; where they exist: &lt;code&gt;CREATE INDEX ... ONLINE&lt;/code&gt;, &lt;code&gt;REBUILD ONLINE&lt;/code&gt;, &lt;code&gt;MOVE ONLINE&lt;/code&gt;. They don't stall the application while they wait.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Put a time limit on the deploy step&lt;/strong&gt;, not just on the DDL. Nonblocking DDL ignores &lt;code&gt;DDL_LOCK_TIMEOUT&lt;/code&gt;, so a statement that waits on a forgotten transaction will wait until someone notices. Watch for the DDL session in &lt;code&gt;enq: TX - row lock contention&lt;/code&gt; with a &lt;code&gt;BLOCKING_SESSION&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;For blocking DDL, keep &lt;code&gt;DDL_LOCK_TIMEOUT&lt;/code&gt; short and retry&lt;/strong&gt;, so any stall it causes is seconds, not minutes.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Prove it
&lt;/h2&gt;

&lt;p&gt;The lab runs Oracle Database Free in a container. The blocker is a SQL*Plus session that updates one row of &lt;code&gt;APP.ORDERS&lt;/code&gt; and sits idle without committing, then six drills run against it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;drill    case                                                 result
busy     TRUNCATE                                             ORA-00054 at once; message has ORDERS' OBJECT_ID, no session
find     V$LOCKED_OBJECT for that object                      the blocker: INACTIVE, mode 3, open transaction;
                                                              BLOCKING_SESSION empty everywhere
kinds    CREATE INDEX, DDL_LOCK_TIMEOUT 5                     ORA-00054 after 5 s
         ADD column, same timeout                             no error: waited ~28 s for the commit; other DML ran
         CREATE INDEX ONLINE, same timeout                    the same
queue    DROP COLUMN, DDL_LOCK_TIMEOUT 30, blocker holds 45 s an update of another row waited 27 s behind it;
                                                              the DDL still failed with ORA-00054
suplog   ADD column with supplemental logging on              ORA-00054 at once; CREATE INDEX ONLINE still waits
fix      kill the blocker                                     TRUNCATE runs; the blocker's change is rolled back
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't take my word for it — run it.&lt;/strong&gt; The &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/ddl-lock-busy" rel="noopener noreferrer"&gt;ddl-lock-busy lab&lt;/a&gt; &lt;strong&gt;asserts&lt;/strong&gt; every row: the immediate ORA-00054 and the object number in the message; &lt;code&gt;V$LOCKED_OBJECT&lt;/code&gt; finding exactly the blocker, idle, with an open transaction, while no session has &lt;code&gt;BLOCKING_SESSION&lt;/code&gt; set; &lt;code&gt;CREATE INDEX&lt;/code&gt; giving up at the timeout while &lt;code&gt;ADD&lt;/code&gt; and &lt;code&gt;CREATE INDEX ... ONLINE&lt;/code&gt; wait past it with the blocker in &lt;code&gt;BLOCKING_SESSION&lt;/code&gt; and other DML unaffected; the update of an untouched row stuck 20 seconds or more behind a waiting &lt;code&gt;DROP COLUMN&lt;/code&gt;; &lt;code&gt;ADD&lt;/code&gt; failing again under supplemental logging; and the kill rolling the change back. If any of that doesn't reproduce, the run fails. Proven on every CI push.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What teams get wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Looking for a busy session.&lt;/strong&gt; The blocker is usually idle. Look for an open transaction, not activity.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Relying on &lt;code&gt;BLOCKING_SESSION&lt;/code&gt;.&lt;/strong&gt; It's empty when the DDL failed at once. Use &lt;code&gt;V$LOCKED_OBJECT&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Setting a long &lt;code&gt;DDL_LOCK_TIMEOUT&lt;/code&gt;.&lt;/strong&gt; For blocking DDL, every write to the table waits behind it for as long as it waits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Assuming &lt;code&gt;ADD&lt;/code&gt; column always fails fast.&lt;/strong&gt; On 26ai (and on 21c and 23.5 in a hand run) it waited for the commit, and &lt;code&gt;DDL_LOCK_TIMEOUT&lt;/code&gt; didn't stop it. A deploy can hang there indefinitely.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Testing the deploy on a database without supplemental logging.&lt;/strong&gt; With it on, the nonblocking &lt;code&gt;ADD&lt;/code&gt; that waited in test fails in production.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Killing the session without asking what it was doing.&lt;/strong&gt; Its uncommitted work is rolled back.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What does ORA-00054 resource busy and acquire with NOWAIT specified or timeout expired mean?
&lt;/h3&gt;

&lt;p&gt;A DDL statement (or a LOCK TABLE ... NOWAIT) needed a lock on a table that another session holds, and didn't get it in time. Usually the other session has an uncommitted change to the table: even one changed row holds a table-level lock until the transaction commits or rolls back. With the default DDL_LOCK_TIMEOUT of 0 the statement fails at once. On Oracle Database 23ai and 26ai the message reads 'Failed to acquire a lock' and includes the table's object number.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I find which session is causing ORA-00054?
&lt;/h3&gt;

&lt;p&gt;Query V$LOCKED_OBJECT for the table (by owner and name through DBA_OBJECTS, or by the object number in the 26ai message) and join to V$SESSION and V$TRANSACTION. The blocker is usually an INACTIVE session with an open transaction and LOCKED_MODE 3. BLOCKING_SESSION in V$SESSION doesn't help when the DDL failed at once, because nothing is waiting. In the companion lab, V$LOCKED_OBJECT returned exactly the blocking session while no session had BLOCKING_SESSION set.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does DDL_LOCK_TIMEOUT fix ORA-00054?
&lt;/h3&gt;

&lt;p&gt;It makes blocking DDL wait for its lock instead of failing at once, but while it waits, new DML on the table queues behind it. In the companion lab a DROP COLUMN with DDL_LOCK_TIMEOUT = 30 made another session's update of an unrelated row wait 27 seconds, and the DDL still failed with ORA-00054. It also has no effect on nonblocking DDL such as CREATE INDEX ONLINE or, since 23.3, ALTER TABLE ADD column. Keep it short and retry, and find the session holding the lock.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does ALTER TABLE ADD COLUMN hang instead of failing with ORA-00054?
&lt;/h3&gt;

&lt;p&gt;Oracle lists ALTER TABLE ... ADD column as nonblocking DDL from release 23.3 (and in a hand run on XE 21.3 it already behaved that way): it waits for open transactions on the table to finish while letting new DML run, and it ignores DDL_LOCK_TIMEOUT. In the companion lab on 26ai it waited until the blocking session committed. If that session never commits, the ALTER never finishes. The waiting session shows the event 'enq: TX - row lock contention' with the blocker in BLOCKING_SESSION.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does CREATE INDEX ONLINE avoid ORA-00054?
&lt;/h3&gt;

&lt;p&gt;Yes: CREATE INDEX ... ONLINE is nonblocking DDL, so it doesn't fail with ORA-00054 because of an open transaction. Instead it waits for that transaction to end, and lets other DML on the table run meanwhile. In the companion lab it waited about 28 seconds until the blocker committed, even with DDL_LOCK_TIMEOUT set to 5, and kept waiting with supplemental logging on.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does ADD COLUMN fail with ORA-00054 on one database but not another?
&lt;/h3&gt;

&lt;p&gt;Check supplemental logging. Oracle downgrades some nonblocking DDL to blocking when supplemental logging is on, which it is on databases that feed GoldenGate, LogMiner-based tools or a logical standby. In the companion lab, with minimal database-level supplemental logging turned on, ALTER TABLE ADD failed at once with ORA-00054 against the same idle session it had waited for without it.&lt;/p&gt;

&lt;p&gt;ORA-00054 is almost never a busy table. It's one idle session with an open transaction. Find it with &lt;code&gt;V$LOCKED_OBJECT&lt;/code&gt;, decide whether it should commit or be killed, and know which of your release statements will fail fast, which will wait, and which will make the application wait with them. Then prove it with the &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/ddl-lock-busy" rel="noopener noreferrer"&gt;ddl-lock-busy lab&lt;/a&gt;, where one uncommitted row fails a &lt;code&gt;TRUNCATE&lt;/code&gt;, stalls an unrelated update for 27 seconds, and holds an &lt;code&gt;ADD&lt;/code&gt; column until it commits.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://uptimearchitect.com/blog/oracle-ora-00054-resource-busy-ddl/" rel="noopener noreferrer"&gt;uptimearchitect.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>oracle</category>
      <category>database</category>
      <category>devops</category>
      <category>sql</category>
    </item>
    <item>
      <title>ORA-03113 After the Connection Sat Idle: The Firewall Forgot It</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Sat, 14 Nov 2026 15:01:41 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/ora-03113-after-the-connection-sat-idle-the-firewall-forgot-it-5ek0</link>
      <guid>https://dev.to/uptimearchitect/ora-03113-after-the-connection-sat-idle-the-firewall-forgot-it-5ek0</guid>
      <description>&lt;p&gt;The application works all day. Then the first request after a quiet spell fails: the first one in the morning, the first one after lunch, the first one on a rarely used screen. Retry it, and it works.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-03113: end-of-file on communication channel
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-03135: connection lost contact
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The database is fine. No other session noticed anything. The cause is a device between the application and the database: a firewall, a NAT gateway, a cloud load balancer. It tracks every TCP connection that passes through it, and it forgets the ones that stay idle longer than its timeout. A connection pool is designed to keep connections open and idle. The two meet, and the next query on a forgotten connection goes nowhere.&lt;/p&gt;

&lt;p&gt;This post covers which error you get and why, what the database is left holding, the one setting that fixes it, and the settings that look like fixes and aren't. It ends with a lab that puts a firewall between a client and Oracle Database Free 26ai and proves each case.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the device forgets
&lt;/h2&gt;

&lt;p&gt;A stateful firewall, a NAT gateway or a load balancer keeps a table of the TCP connections it lets through. An entry stays as long as packets keep flowing. When a connection sends nothing for longer than the device's idle timeout, the entry is removed. Nothing is sent to either end. The client and the database both still think the connection is open.&lt;/p&gt;

&lt;p&gt;The defaults are short compared with how long a pooled connection can sit idle:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Device&lt;/th&gt;
&lt;th&gt;Default idle timeout&lt;/th&gt;
&lt;th&gt;What happens to the next packet&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-tcp-reset" rel="noopener noreferrer"&gt;Azure Load Balancer&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;4 minutes (configurable 4 to 100)&lt;/td&gt;
&lt;td&gt;dropped silently, unless "TCP reset on idle" is turned on, which resets both ends&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://learn.microsoft.com/en-us/azure/nat-gateway/nat-gateway-resource" rel="noopener noreferrer"&gt;Azure NAT Gateway&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;4 minutes (configurable 4 to 120)&lt;/td&gt;
&lt;td&gt;answered with a reset, toward the side that sent it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html" rel="noopener noreferrer"&gt;AWS Network Load Balancer&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;350 seconds (configurable 60 to 6,000 for TCP listeners)&lt;/td&gt;
&lt;td&gt;answered with a reset to the client&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On-premises firewalls have their own timeouts, often an hour or more, and they're usually set by a network team you'll have to ask. Check each provider's current documentation for your setup; these are the documented defaults at the time of writing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which error, and why
&lt;/h2&gt;

&lt;p&gt;What the application sees depends on what the device does with the first packet of a forgotten connection.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The device sends a reset.&lt;/strong&gt; The client's next query gets a TCP reset straight back, and fails at once. In the lab, on a 26ai client:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-03135: connection lost contact
Process ID: 333
Session ID: 51 Serial number: 3605
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;The device drops the packet silently.&lt;/strong&gt; The client sends the query and gets no answer. Its TCP stack retransmits, waiting longer each time, until it gives up. On Linux that's controlled by &lt;code&gt;net.ipv4.tcp_retries2&lt;/code&gt;, 15 by default. In the lab, with the default, the query hung for &lt;strong&gt;951 seconds&lt;/strong&gt;, almost 16 minutes, and then failed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-03113: end-of-file on communication channel
Process ID: 377
Session ID: 213 Serial number: 13151
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A request that hangs for 16 minutes is often worse than one that fails. Threads and pool connections pile up behind it, and the application's own timeouts may fire first, so the application log can show a generic timeout and no ORA- error at all.&lt;/p&gt;

&lt;p&gt;So: an immediate ORA-03135 after a quiet period points to a device that resets. A long hang, then ORA-03113, points to one that drops silently. Either way the connection was idle longer than something on the path allows.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNptkc1PAjEQxf-VSc98SIgejCFB2Y0ERMA9mLjGlHZ2aey22BaQ8PG3O1sxevAyp_m99-bNngkrkV0DK7TdiiV3AbJBbgCGLzmbWqtRgrDGoAjKGlBS483CtXvamhIdhCU3NBAkbpRACKpCuw45e4VmswcpiaTK4ZZrDW2Y9DOa49uoUFhXYvCR_nUgsnZPIz4jfIKfAT7W6HZQ-4fzwqxeOOTs7OvRSA8cHHqklQPMCX2c95sX3U73EnggWOD_rHR25cErjSboXQ0PCL7npmxAdjclzeAU-kZMfdbsdIEXgQo4da6gUj-xRwQ-zcaTJGslz9PhPHnLhg9JBBdIBcdj_3bUikHeEVdcqw2SV0EnLPG7FgrkYqC0Fheaew8jKK2VrAGsQldxJel5e0bbVXyjxIKvdWDH4xeKPJrg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNptkc1PAjEQxf-VSc98SIgejCFB2Y0ERMA9mLjGlHZ2aey22BaQ8PG3O1sxevAyp_m99-bNngkrkV0DK7TdiiV3AbJBbgCGLzmbWqtRgrDGoAjKGlBS483CtXvamhIdhCU3NBAkbpRACKpCuw45e4VmswcpiaTK4ZZrDW2Y9DOa49uoUFhXYvCR_nUgsnZPIz4jfIKfAT7W6HZQ-4fzwqxeOOTs7OvRSA8cHHqklQPMCX2c95sX3U73EnggWOD_rHR25cErjSboXQ0PCL7npmxAdjclzeAU-kZMfdbsdIEXgQo4da6gUj-xRwQ-zcaTJGslz9PhPHnLhg9JBBdIBcdj_3bUikHeEVdcqw2SV0EnLPG7FgrkYqC0Fheaew8jKK2VrAGsQldxJel5e0bbVXyjxIKvdWDH4xeKPJrg" alt="A pooled connection sits idle past the device's timeout. The device forgets it. If it rese" width="410" height="720"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A pooled connection sits idle past the device's timeout. The device forgets it. If it resets the next packet, the client gets ORA-03135 at once; if it drops it silently, the client hangs until TCP gives up and then gets ORA-03113. SQLNET.EXPIRE_TIME below the timeout keeps the entry alive.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What the database is left holding
&lt;/h2&gt;

&lt;p&gt;The database often never finds out. In the lab, in both the reset case and the silent case, the client's session stayed in &lt;code&gt;V$SESSION&lt;/code&gt; as &lt;code&gt;INACTIVE&lt;/code&gt; after the client had given up on it. The reset went only to the client, and in the silent case nothing reached the database at all. (An Azure Load Balancer with TCP reset turned on resets both ends, so there the database does find out.)&lt;/p&gt;

&lt;p&gt;Those orphan sessions are real sessions. They count against &lt;code&gt;SESSIONS&lt;/code&gt; and &lt;code&gt;PROCESSES&lt;/code&gt;, and if the application had an open transaction, they hold its locks. A pool that replaces each broken connection with a new one leaves one orphan behind per failure.&lt;/p&gt;

&lt;p&gt;The same happens when the client host itself disappears: a VM that's deleted, a network that's cut. In the lab, with no keepalive configured, the session of a client whose network was disconnected was still there after 240 seconds, when the drill stopped checking, and orphans from earlier tests stayed for more than 30 minutes. Nothing was ever going to remove them.&lt;/p&gt;

&lt;p&gt;To look for them, find sessions idle for longer than the device's timeout, from the application's hosts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;sid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;serial&lt;/span&gt;&lt;span class="c1"&gt;#, username, machine, program, status,&lt;/span&gt;
       &lt;span class="nb"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;last_call_et&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;idle_minutes&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$session&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'USER'&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'INACTIVE'&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;last_call_et&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;240&lt;/span&gt;          &lt;span class="c1"&gt;-- the idle timeout on the path, in seconds&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;last_call_et&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A pool's healthy idle connections show up here too, so this is a list to compare, not a list to kill. The signal is many more sessions from an app server than its pool can hold, or sessions from a host that no longer exists.&lt;/p&gt;

&lt;p&gt;The alert log doesn't help with the client's errors: in the lab they left nothing there. The only network entries it had were two &lt;code&gt;Fatal NI connect error 12170&lt;/code&gt; entries with &lt;code&gt;TNS-12535: TNS:operation timed out&lt;/code&gt;. Those are the database's own keepalive giving up on dead connections, which happens once &lt;code&gt;SQLNET.EXPIRE_TIME&lt;/code&gt; is set.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: SQLNET.EXPIRE_TIME on the server
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;SQLNET.EXPIRE_TIME&lt;/code&gt; goes in the &lt;strong&gt;database server's&lt;/strong&gt; &lt;code&gt;sqlnet.ora&lt;/code&gt;, and its value is in &lt;strong&gt;minutes&lt;/strong&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SQLNET.EXPIRE_TIME = 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Where the operating system supports it, Oracle Net uses it to set TCP keepalive on the server side of each connection: after that many minutes of idle time, the server sends a keepalive probe. The probe is traffic, so the firewall, the NAT gateway and the load balancer all see the connection as active and keep their entry. If the client really is gone, the probes go unanswered and the database closes the session itself.&lt;/p&gt;

&lt;p&gt;In the lab, with a firewall that forgot connections after 90 seconds:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;SQLNET.EXPIRE_TIME = 1&lt;/code&gt; (60 seconds): a connection idle for 110 seconds ran its next query normally.&lt;/li&gt;
&lt;li&gt;A client whose network was cut: the database removed its session after about 115 seconds (112 to 117 across runs), instead of never.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Three rules for using it:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;It must be below the smallest idle timeout on the path.&lt;/strong&gt; Behind an Azure Load Balancer or NAT gateway at the 4-minute default, use 1 to 3. Behind an AWS NLB at 350 seconds, up to 5. If you don't know every device on the path, 1 or 2 is a safe choice; the probes are tiny.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It applies to new connections.&lt;/strong&gt; Server processes read &lt;code&gt;sqlnet.ora&lt;/code&gt; when they start. Connections that are already open keep what they had, so recycle the pool after the change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It needs no listener restart&lt;/strong&gt;, for the same reason: each new connection reads the file.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Fixes from the client side
&lt;/h2&gt;

&lt;p&gt;When you can't change the database server's &lt;code&gt;sqlnet.ora&lt;/code&gt;, there are two client-side options. Both worked in the lab; both have a catch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;(ENABLE=BROKEN)&lt;/code&gt; in the connect descriptor&lt;/strong&gt; turns on TCP keepalive on the client's socket:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;(DESCRIPTION=(ENABLE=BROKEN)(ADDRESS=(PROTOCOL=TCP)(HOST=db)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=app)))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On its own, it didn't help in the lab. It switches keepalive on, but the operating system decides when the first probe goes out, and the Linux default (&lt;code&gt;net.ipv4.tcp_keepalive_time&lt;/code&gt;) is 7,200 seconds: two hours. With the client host's keepalive time set to 60 seconds, the connection survived. That's a kernel setting for the whole host, so it's a conversation with whoever owns the app servers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;SQLNET.EXPIRE_TIME&lt;/code&gt; in the client's &lt;code&gt;sqlnet.ora&lt;/code&gt;&lt;/strong&gt; also worked in the lab, where the client is 26ai. The &lt;a href="https://docs.oracle.com/en/database/oracle/oracle-database/26/netrf/parameters-for-the-sqlnet.ora.html" rel="noopener noreferrer"&gt;26ai Net Services Reference&lt;/a&gt; says the parameter is meant primarily for the server but can be used on clients too. I tested only a 26ai client, so check an older one before relying on it.&lt;/p&gt;

&lt;p&gt;The application's pool can help too. Most pools can retire connections that have been idle for longer than a set time, or validate a connection before handing it out. A maximum idle time below the device's timeout avoids the problem. Validation on borrow doesn't avoid it: it finds the broken connection and replaces it, which is better than failing a request, but against a device that drops silently the validation itself can hang, and each replaced connection can leave an orphan session behind. I didn't test pool settings in the lab.&lt;/p&gt;

&lt;h2&gt;
  
  
  What doesn't fix it
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;SQLNET.EXPIRE_TIME&lt;/code&gt; higher than the timeout.&lt;/strong&gt; It's the most common one. Oracle's own &lt;a href="https://docs.oracle.com/en/database/oracle/oracle-database/26/netrf/parameters-for-the-sqlnet.ora.html" rel="noopener noreferrer"&gt;reference&lt;/a&gt; gives 10 as the recommended value, and 10 does nothing behind a 4-minute load balancer. In the lab, &lt;code&gt;EXPIRE_TIME = 2&lt;/code&gt; (120 seconds) behind the 90-second firewall failed exactly like no setting at all: the first probe came after the firewall had already forgotten the connection.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;(ENABLE=BROKEN)&lt;/code&gt; alone.&lt;/strong&gt; See above: with the OS default, the first keepalive comes after two hours.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;SQLNET.RECV_TIMEOUT&lt;/code&gt; on the client.&lt;/strong&gt; It limits how long the client waits for an answer. With it set to 30 seconds, the silent drop in the lab ended in a different error after about 60 seconds (60 to 61 across runs, about two timeouts) instead of 16 minutes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-12609: TNS: Receive timeout occurred
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's better than a 16-minute hang, and worth having, but the connection is still broken and the request still fails. It shortens the failure; it doesn't prevent it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Raising the device's idle timeout.&lt;/strong&gt; It works until a connection sits idle a little longer than the new value. It also costs something: Microsoft's NAT gateway documentation warns that longer idle timeouts hold SNAT ports longer and raise the risk of running out of them. And it only fixes the device you changed, not the next one someone adds to the path.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it
&lt;/h2&gt;

&lt;p&gt;The lab runs Oracle Database Free in a container. There's no firewall box: the database container's network namespace gets connection tracking that forgets an idle TCP connection after 90 seconds, and one iptables rule that drops, or answers with a reset, any packet of a connection it has forgotten. Each client runs in its own container, runs a query, sits idle for 110 seconds, then runs a second query.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;drill     case                                                   second query
reset     firewall sends a reset                                 ORA-03135 at once; the session stays INACTIVE
drop      firewall drops silently, client TCP retries cut to 5   ORA-03113 after ~14 s; the session stays INACTIVE
          same, SQLNET.RECV_TIMEOUT = 30 in the client           ORA-12609 after ~60 s
fix       SQLNET.EXPIRE_TIME = 1 on the server                   runs
toolate   SQLNET.EXPIRE_TIME = 2 on the server                   ORA-03113
client    (ENABLE=BROKEN), OS keepalive time 7,200 s             ORA-03113
          (ENABLE=BROKEN), OS keepalive time 60 s                runs
          SQLNET.EXPIRE_TIME = 1 in the 26ai client's sqlnet.ora runs
dead      client network cut, no EXPIRE_TIME                     session still there after 240 s
          client network cut, EXPIRE_TIME = 1                    session removed after ~115 s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The drills cut the client's TCP retries from 15 to 5 so a silent drop fails in seconds instead of 16 minutes. That changes how long the hang lasts, not which error you get. The 951-second hang with the Linux default was measured once by hand.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't take my word for it — run it.&lt;/strong&gt; The &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/idle-connection-drop" rel="noopener noreferrer"&gt;idle-connection-drop lab&lt;/a&gt; &lt;strong&gt;asserts&lt;/strong&gt; every row: the error for each firewall mode, and that the database still has the session afterwards; ORA-12609 with &lt;code&gt;RECV_TIMEOUT&lt;/code&gt;; the second query running with &lt;code&gt;EXPIRE_TIME = 1&lt;/code&gt; and failing with 2; &lt;code&gt;ENABLE=BROKEN&lt;/code&gt; failing on its own and working with a 60-second keepalive time; client-side &lt;code&gt;EXPIRE_TIME&lt;/code&gt; working; and a vanished client's session kept without &lt;code&gt;EXPIRE_TIME&lt;/code&gt; and removed with it. If any of that doesn't reproduce, the run fails. Proven on every CI push.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What teams get wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Treating ORA-03113 as a database crash.&lt;/strong&gt; It can be one, but when it comes after idle time, from one application, with nothing in the alert log, it's the network path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Setting &lt;code&gt;SQLNET.EXPIRE_TIME&lt;/code&gt; without knowing the timeout.&lt;/strong&gt; It's in minutes and it has to be below the smallest idle timeout on the path. 10 doesn't work behind a 4-minute load balancer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Expecting the change to fix open connections.&lt;/strong&gt; It applies to new ones. Recycle the pool.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trusting &lt;code&gt;(ENABLE=BROKEN)&lt;/code&gt; on its own.&lt;/strong&gt; Without a shorter OS keepalive time, the first probe goes out after two hours.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Calling &lt;code&gt;RECV_TIMEOUT&lt;/code&gt; the fix.&lt;/strong&gt; It makes the failure faster, not less likely.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ignoring the orphans.&lt;/strong&gt; Every failed connection can leave a session on the database. Without &lt;code&gt;EXPIRE_TIME&lt;/code&gt;, nothing removes it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What causes ORA-03113 end-of-file on communication channel after a connection has been idle?
&lt;/h3&gt;

&lt;p&gt;Usually a stateful device between the application and the database, such as a firewall, NAT gateway or cloud load balancer, that removed the idle connection from its table. When the application uses the connection again, the device drops the packets silently, the client's TCP stack retransmits until it gives up, and the client reports ORA-03113. In the companion lab, with Linux's default TCP retries, that took 951 seconds. The database itself is fine.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the difference between ORA-03113 and ORA-03135?
&lt;/h3&gt;

&lt;p&gt;Both mean the client lost its connection to the database. In the companion lab on a 26ai client, a device that answered the forgotten connection with a TCP reset produced ORA-03135 connection lost contact at once, and a device that dropped the packets silently produced a long hang followed by ORA-03113 end-of-file on communication channel. So an immediate ORA-03135 points to a reset, and a long hang followed by ORA-03113 points to a silent drop.&lt;/p&gt;

&lt;h3&gt;
  
  
  What should SQLNET.EXPIRE_TIME be set to?
&lt;/h3&gt;

&lt;p&gt;A number of minutes below the smallest idle timeout of any firewall, NAT gateway or load balancer between the application and the database. Azure Load Balancer and Azure NAT Gateway default to 4 minutes, so use 1 to 3; an AWS Network Load Balancer defaults to 350 seconds, so up to 5. In the companion lab, behind a 90-second timeout, EXPIRE_TIME = 1 kept idle connections working and EXPIRE_TIME = 2 did not.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where do I set SQLNET.EXPIRE_TIME, and does it need a restart?
&lt;/h3&gt;

&lt;p&gt;In the database server's sqlnet.ora. It applies to connections made after the change, because each server process reads the file when it starts; connections that are already open are not affected, so recycle the application's connection pool. No listener or database restart is needed.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does ENABLE=BROKEN fix idle connection timeouts?
&lt;/h3&gt;

&lt;p&gt;Not on its own. (ENABLE=BROKEN) in the connect descriptor turns on TCP keepalive on the client's socket, but the operating system decides when the first keepalive is sent, and the Linux default is 7,200 seconds. In the companion lab it failed on its own and worked when the client host's net.ipv4.tcp_keepalive_time was set to 60 seconds.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why are there INACTIVE sessions left on the database after ORA-03113 or ORA-03135?
&lt;/h3&gt;

&lt;p&gt;Because the database never learned that the client was gone. In the companion lab, the reset went only to the client and the silent drop reached neither end, so the session stayed in V$SESSION as INACTIVE. It holds a process and any locks of an open transaction. With SQLNET.EXPIRE_TIME set, the database's keepalive probes go unanswered and the database removes such a session itself; in the lab that took about 115 seconds with EXPIRE_TIME = 1.&lt;/p&gt;

&lt;p&gt;ORA-03113 after idle time isn't a database fault. It's a connection that sat quiet longer than something on the network allows. Find the smallest idle timeout on the path, set &lt;code&gt;SQLNET.EXPIRE_TIME&lt;/code&gt; on the database server below it, and recycle the pool. If clients can't reach the database at all, rather than losing it after a while, that's a different problem: start with &lt;a href="https://uptimearchitect.com/blog/oracle-ora-12514-listener-service/" rel="noopener noreferrer"&gt;ORA-12514 and the listener&lt;/a&gt;. Then prove it with the &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/idle-connection-drop" rel="noopener noreferrer"&gt;idle-connection-drop lab&lt;/a&gt;, where a 90-second firewall breaks every idle connection and one line in &lt;code&gt;sqlnet.ora&lt;/code&gt; keeps them alive.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://uptimearchitect.com/blog/oracle-ora-03113-idle-connection-firewall/" rel="noopener noreferrer"&gt;uptimearchitect.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>oracle</category>
      <category>database</category>
      <category>devops</category>
      <category>networking</category>
    </item>
    <item>
      <title>ORA-01000 Maximum Open Cursors Exceeded: Don't Raise the Limit, Find the Leak</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Wed, 11 Nov 2026 17:37:32 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/ora-01000-maximum-open-cursors-exceeded-dont-raise-the-limit-find-the-leak-2m0f</link>
      <guid>https://dev.to/uptimearchitect/ora-01000-maximum-open-cursors-exceeded-dont-raise-the-limit-find-the-leak-2m0f</guid>
      <description>&lt;p&gt;The application has been fine for days. Then one request fails, then a few more, all from the same app server:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-01000: maximum open cursors exceeded
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The usual reaction is to raise &lt;code&gt;OPEN_CURSORS&lt;/code&gt;. It's one &lt;code&gt;ALTER SYSTEM&lt;/code&gt;, it needs no restart, and the errors stop. Then they come back a week later.&lt;/p&gt;

&lt;p&gt;ORA-01000 is almost never a limit that's too low. It's an application that opens cursors and doesn't close them: a &lt;strong&gt;cursor leak&lt;/strong&gt;. The limit is just where the leak becomes visible. This post covers what the limit counts, how to find the leaking session and the exact statement, the query that makes a healthy session look like the culprit, and why raising the limit doesn't help the connections that are already failing. It ends with a lab that proves each step on Oracle Database Free 26ai.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the limit counts
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;OPEN_CURSORS&lt;/code&gt; is the maximum number of cursors &lt;strong&gt;one session&lt;/strong&gt; can have open at once. It is not a database-wide pool. A session at the limit gets ORA-01000; every other session carries on. On 26ai the message reads:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-01000: maximum open cursors for session exceeded
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A cursor is the server-side handle for a statement the client is still working with. In application code it's a &lt;code&gt;Statement&lt;/code&gt; or &lt;code&gt;ResultSet&lt;/code&gt; in JDBC, a cursor object in Python, a data reader in .NET. It stays open on the server until the client closes it, or the session ends. If the code never closes it, it stays open for as long as the connection lives. With a connection pool, that can be days.&lt;/p&gt;

&lt;p&gt;That's why ORA-01000 shows up late. A code path that leaks one cursor per call is harmless on the first call. Each pooled connection collects its leaked cursors over hundreds or thousands of requests, and fails when it reaches the limit. The request that fails is just the one that needed the next cursor.&lt;/p&gt;

&lt;p&gt;The parameter's documented default is 50. Many databases run with something higher; the container in this post's lab runs with 300.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the error tells you on 26ai
&lt;/h2&gt;

&lt;p&gt;On 26ai the first ORA-01000 in a session carries a clue:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-01000: maximum open cursors for session exceeded Potential Leaked SQL_ID:
4cgc7syjwjw3y
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab, that SQL_ID was the statement the session held open the most, and &lt;strong&gt;not&lt;/strong&gt; the statement that failed. A session leaked 298 result sets from one procedure and then called a different one. The different call failed, and the message named the &lt;strong&gt;leaked&lt;/strong&gt; statement. In a second test the session leaked 200 result sets from one statement, then 98 from another, and the message still named the one with 200. That's what you want: the application log now points at the code that leaks, not the innocent request that happened to need the next cursor.&lt;/p&gt;

&lt;p&gt;Two things to know:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Only the first error in the session names it.&lt;/strong&gt; In the lab the next 21 errors in the same session said only "maximum open cursors for session exceeded". Find the first one in the application log.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It may be wrapped.&lt;/strong&gt; When the failing statement needs internal SQL first (parsing a new statement, for example), the application sees &lt;code&gt;ORA-00604: Error occurred at recursive SQL level 1&lt;/code&gt; with ORA-01000 underneath. The lab got both forms.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Look the SQL_ID up in &lt;code&gt;V$SQL&lt;/code&gt; (it stays in the shared pool after the session ends, until it ages out), then find that statement in the code.&lt;/p&gt;

&lt;p&gt;The database itself logs nothing. After a run that raised ORA-01000 22 times, the lab found no ORA-01000 in the alert log. If the application swallows its errors, you won't see this one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Find the session
&lt;/h2&gt;

&lt;p&gt;If you don't have the SQL_ID, or want to confirm it, start with the session. The count that the limit applies to is the statistic &lt;code&gt;opened cursors current&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;serial&lt;/span&gt;&lt;span class="c1"&gt;#, s.username, s.module, s.machine, st.value AS open_cursors&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$sesstat&lt;/span&gt; &lt;span class="n"&gt;st&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$statname&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;statistic&lt;/span&gt;&lt;span class="c1"&gt;# = st.statistic#&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$session&lt;/span&gt;  &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sid&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'opened cursors current'&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;span class="k"&gt;FETCH&lt;/span&gt; &lt;span class="k"&gt;FIRST&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="k"&gt;ROWS&lt;/span&gt; &lt;span class="k"&gt;ONLY&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A leaking session sits near &lt;code&gt;OPEN_CURSORS&lt;/code&gt;. A healthy one holds a handful. &lt;code&gt;MODULE&lt;/code&gt; and &lt;code&gt;MACHINE&lt;/code&gt; tell you which application and which app server. Run this on a schedule and you'll see a leak as a number that grows with uptime long before it reaches the limit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Find the statement, and don't count cached cursors
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;V$OPEN_CURSOR&lt;/code&gt; lists cursors per session with their SQL text. It also lists cursors that are &lt;strong&gt;closed&lt;/strong&gt; and kept in the session cursor cache so the next parse of the same statement is cheap. Cached cursors don't count against &lt;code&gt;OPEN_CURSORS&lt;/code&gt;. If you count rows in &lt;code&gt;V$OPEN_CURSOR&lt;/code&gt; without filtering, a busy, healthy session can look like the problem.&lt;/p&gt;

&lt;p&gt;The lab ran two sessions side by side. One leaked 250 result sets. The other ran 450 different queries four times each, closing every one, with &lt;code&gt;SESSION_CACHED_CURSORS&lt;/code&gt; set to 1,000:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;session      opened cursors current    rows in V$OPEN_CURSOR
leaky-app    252                       265
busy-app     2                         458
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;By raw row count, the healthy session has more "open cursors" than the leaking one, and more than the 300 limit. It had two open. 454 of its rows were &lt;code&gt;SESSION CURSOR CACHED&lt;/code&gt;. (The lab set the cache to 1,000 to make the point; at the default of 50, expect a few dozen extra rows per session. Some shops set it much higher.)&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;CURSOR_TYPE&lt;/code&gt; column separates them. Filter on the types that start with &lt;code&gt;OPEN&lt;/code&gt;, and group by statement:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sql_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;MAX&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sql_text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;sql_text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;COUNT&lt;/span&gt;&lt;span class="cm"&gt;(*) AS open_count
FROM   v$open_cursor o
WHERE  o.sid = :sid
AND    o.cursor_type LIKE 'OPEN%'
GROUP  BY o.sql_id
ORDER  BY open_count DESC;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab that returned a single row:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SQL_ID         OPEN_COUNT  SQL_TEXT
4cgc7syjwjw3y  250         SELECT ID, CREATED FROM APP.ORDERS WHERE CUSTOMER_ID = :B1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One statement open 250 times in one session is a leak. Nothing legitimate holds the same query's result open 250 times at once. If instead you see many &lt;em&gt;different&lt;/em&gt; statements, each open a few times, the session may genuinely need more cursors (see the sizing section below).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpd0V1LwzAUBuC_8hKU3XQ4b4dM5lZkWPbRbsqwMmJyaottoknGGNv-u2k7cXh1IMl5n3DOgQktifXBslLvRM6Nw3KcKiB8TdksHnZ7t71eL2Vv6HYHSPzhS16IHJasLbS6v3s3N4PnqyRMkuVwiY7-IkUSYmusNrauhpTrNM8UcYPZPJxuRqs4mcWJj62ppMle_GU77qjybb_pFz0B2rpZruchoslTiE59fd0Sj_FsNcfDGva73BTyDCxq4JgyrQjJItpMxgHyrZKGpIXO4IqKbMqOiPwfRs3XURL_7DeZotSWUDgUCi4n1BMLmhtDYi9KOp8qRcL5mdh_asXV_szaABwZ7UBc5DU49WBMvIQiki1neOG5yzG1mEcUvNg6Z0OU3FpEwRQfWksWgFVkKl5Iv9ID8y1Vs1xJGd-Wjp1OP6sTnzs" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpd0V1LwzAUBuC_8hKU3XQ4b4dM5lZkWPbRbsqwMmJyaottoknGGNv-u2k7cXh1IMl5n3DOgQktifXBslLvRM6Nw3KcKiB8TdksHnZ7t71eL2Vv6HYHSPzhS16IHJasLbS6v3s3N4PnqyRMkuVwiY7-IkUSYmusNrauhpTrNM8UcYPZPJxuRqs4mcWJj62ppMle_GU77qjybb_pFz0B2rpZruchoslTiE59fd0Sj_FsNcfDGva73BTyDCxq4JgyrQjJItpMxgHyrZKGpIXO4IqKbMqOiPwfRs3XURL_7DeZotSWUDgUCi4n1BMLmhtDYi9KOp8qRcL5mdh_asXV_szaABwZ7UBc5DU49WBMvIQiki1neOG5yzG1mEcUvNg6Z0OU3FpEwRQfWksWgFVkKl5Iv9ID8y1Vs1xJGd-Wjp1OP6sTnzs" alt="ORA-01000 is a per-session limit. Find the session by 'opened cursors current', then the s" width="431" height="728"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;ORA-01000 is a per-session limit. Find the session by 'opened cursors current', then the statement by V$OPEN_CURSOR filtered to CURSOR_TYPE LIKE 'OPEN%'. One statement open hundreds of times is a leak to fix in code; many different statements may be a real need.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Raising OPEN_CURSORS: what it does and doesn't do
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;OPEN_CURSORS&lt;/code&gt; can be changed with &lt;code&gt;ALTER SYSTEM&lt;/code&gt;, without a restart, and in a PDB. The lab raised it from 300 to 400 in the PDB while one session was stuck at the limit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;session                                         result after the raise
stuck at the limit before the raise             55 more calls, 55 x ORA-01000
connected before the raise, leaking after it    first ORA-01000 at call 299 (the old limit)
connected after the raise, same leak            first ORA-01000 at call 399
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things went wrong with the "fix":&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;It didn't reach the connections that were already open.&lt;/strong&gt; On 26ai in the lab, every session connected before the change kept the old limit, stuck or not. Only new connections got 400. In a pooled application, that means the raise does nothing until the pool reconnects. When the errors stop after a raise, it's often the app restart that went with it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The leak failed again, just later.&lt;/strong&gt; The new connection hit ORA-01000 at call 399 instead of 299. The leak rate didn't change.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;So raising the limit is a way to buy time while the code is fixed, and it only buys it after the connections are recycled. Each open cursor also holds memory on the server, so a high limit on a leaking app means more memory held before it fails.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Right now:&lt;/strong&gt; the leaked cursors belong to the session. Ending the session releases all of them. Recycle the affected pool's connections or restart that app server. Capture &lt;code&gt;opened cursors current&lt;/code&gt; and the session's &lt;code&gt;V$OPEN_CURSOR&lt;/code&gt; rows first, because they're gone when the session is.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;In the code:&lt;/strong&gt; close what you open, on every path, including the error path. That means try-with-resources in Java for every &lt;code&gt;Statement&lt;/code&gt;, &lt;code&gt;PreparedStatement&lt;/code&gt;, &lt;code&gt;CallableStatement&lt;/code&gt; and &lt;code&gt;ResultSet&lt;/code&gt;, including result sets returned as ref cursors from stored procedures. In Python, use &lt;code&gt;with&lt;/code&gt; blocks for cursors; in .NET, &lt;code&gt;using&lt;/code&gt;. In PL/SQL, close explicit cursors and &lt;code&gt;DBMS_SQL&lt;/code&gt; cursors in the exception handler too. The leak in the lab is a stored procedure that returns a result set to the caller (a &lt;code&gt;SYS_REFCURSOR&lt;/code&gt;). The caller owns that cursor, and the caller never closed it. The same 400 calls with each result set read and closed ran with no error, at the original limit of 300.&lt;/p&gt;

&lt;h2&gt;
  
  
  When the limit really is too low
&lt;/h2&gt;

&lt;p&gt;Some sessions do need more than the default: an application that keeps many statements prepared at once, or a client-side statement cache. Oracle's JDBC driver, for example, can cache statements per connection, and each cached statement keeps its cursor open on the server. That's by design, and the cursors are spread over many different statements, not one.&lt;/p&gt;

&lt;p&gt;Size &lt;code&gt;OPEN_CURSORS&lt;/code&gt; from what the application uses at peak, not from the last error:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;module&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;MAX&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;peak_open_now&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$sesstat&lt;/span&gt; &lt;span class="n"&gt;st&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$statname&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;statistic&lt;/span&gt;&lt;span class="c1"&gt;# = st.statistic#&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="nv"&gt;$session&lt;/span&gt;  &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sid&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'opened cursors current'&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;module&lt;/span&gt;
&lt;span class="nc"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;peak_open_now&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it over a normal business day. If the healthy peak is close to the limit, raise it with headroom and recycle the connections. If one module keeps climbing, it's a leak, and a higher limit only delays the next outage.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it
&lt;/h2&gt;

&lt;p&gt;The lab runs Oracle Database Free in a container. The "application" is SQL*Plus calling a procedure that returns a result set, and holding each result set in its own variable without closing it: the same thing a &lt;code&gt;ResultSet&lt;/code&gt; that's never closed does on the server. Then it breaks and fixes things five ways:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;drill   what happens                                             result
leak    320 calls, no result set closed (OPEN_CURSORS 300)       ORA-01000 from call 299; first error names the SQL_ID
blame   298 leaked, then a different call                        the other call fails, message names the leaker;
                                                                 another session runs that call fine
find    a leaking session next to a busy healthy one             raw V$OPEN_CURSOR rows: 265 vs 458;
                                                                 'OPEN%' filter finds the leak, 250 x one SQL
raise   OPEN_CURSORS 300 -&amp;gt; 400 during the outage                stuck and pre-existing sessions keep 300; new: call 399
fix     400 calls, each result set closed                        no ORA-01000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Why call 299 and not 301? The session already had two cursors open before the first call: SQL*Plus's own statement and one recursive cursor. 298 leaked plus those two is 300.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't take my word for it — run it.&lt;/strong&gt; The &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/open-cursors" rel="noopener noreferrer"&gt;open-cursors lab&lt;/a&gt; &lt;strong&gt;asserts&lt;/strong&gt; every row: the failing call number from the limit; that the first error, and only the first, names the leaked statement's SQL_ID; nothing in the alert log; the leaked statement named even when a different call fails, and a second session unaffected; the healthy session's &lt;code&gt;V$OPEN_CURSOR&lt;/code&gt; row count above the limit while it has almost nothing open; the leaked statement found 250 times by the &lt;code&gt;OPEN%&lt;/code&gt; filter; both sessions opened before the raise keeping the old limit and a new one failing at 399; and no error once each result set is closed. If any of that doesn't reproduce, the run fails. Proven on every CI push.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What teams get wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Raising &lt;code&gt;OPEN_CURSORS&lt;/code&gt; as the fix.&lt;/strong&gt; It moves the failure, and only for new connections. Find the statement first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Counting rows in &lt;code&gt;V$OPEN_CURSOR&lt;/code&gt;.&lt;/strong&gt; That includes cached cursors, which are closed. Use &lt;code&gt;opened cursors current&lt;/code&gt;, or filter &lt;code&gt;CURSOR_TYPE LIKE 'OPEN%'&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Blaming the statement that failed.&lt;/strong&gt; It's whichever call needed the next cursor. On 26ai, read the SQL_ID in the first error; on any release, look at what the session holds open.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Closing the &lt;code&gt;ResultSet&lt;/code&gt; on the happy path only.&lt;/strong&gt; An exception between open and close leaks it just the same. Close in &lt;code&gt;finally&lt;/code&gt;, or let try-with-resources do it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating ORA-01000 as a database problem.&lt;/strong&gt; It's per session, the database logs nothing, and every other session is fine. The fix is in the application.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What does ORA-01000 maximum open cursors exceeded mean?
&lt;/h3&gt;

&lt;p&gt;It means one database session tried to open more cursors at the same time than the OPEN_CURSORS parameter allows. The limit is per session, not for the whole database, so other sessions keep working. It is almost always caused by a cursor leak: the application opens statements or result sets and doesn't close them, so each connection accumulates open cursors until it reaches the limit. Less often, the application genuinely needs more cursors at once than the limit allows.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I find which SQL is causing ORA-01000?
&lt;/h3&gt;

&lt;p&gt;On Oracle Database 26ai the first ORA-01000 in a session includes 'Potential Leaked SQL_ID' followed by the statement the session holds open most; look it up in V$SQL. On any release, find the session with the highest 'opened cursors current' in V$SESSTAT, then query V$OPEN_CURSOR for that SID with CURSOR_TYPE LIKE 'OPEN%', grouped by SQL_ID. One statement open hundreds of times is the leak. In the companion lab that returned one statement open 250 times.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should I just increase OPEN_CURSORS?
&lt;/h3&gt;

&lt;p&gt;Only if the application genuinely needs more cursors at once, which shows up as many different statements each open a few times. For a leak, raising the limit only delays the error: in the companion lab, raising it from 300 to 400 moved the first ORA-01000 from call 299 to call 399. And the new value applied only to new connections; sessions connected before the change kept the old limit, so a connection pool has to reconnect before a raise has any effect.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does V$OPEN_CURSOR show more cursors than OPEN_CURSORS without an error?
&lt;/h3&gt;

&lt;p&gt;Because V$OPEN_CURSOR also lists cursors that are closed and kept in the session cursor cache (CURSOR_TYPE 'SESSION CURSOR CACHED'), and cached cursors don't count against OPEN_CURSORS. In the companion lab a healthy session with SESSION_CACHED_CURSORS set to 1,000 showed 458 rows in V$OPEN_CURSOR against a limit of 300, while its 'opened cursors current' was 2. Filter on CURSOR_TYPE LIKE 'OPEN%' or use the 'opened cursors current' statistic.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does increasing OPEN_CURSORS require a restart?
&lt;/h3&gt;

&lt;p&gt;No. OPEN_CURSORS can be changed with ALTER SYSTEM while the database is running, and in a PDB. But in the companion lab on 26ai the new value applied only to sessions that connected after the change; sessions that were already connected, including the one stuck at the limit, kept the old limit. Recycle the application's connections after changing it.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I fix a cursor leak in Java?
&lt;/h3&gt;

&lt;p&gt;Close every Statement, PreparedStatement, CallableStatement and ResultSet on every path, including when an exception is thrown. try-with-resources does this automatically. Result sets returned from stored procedures as ref cursors are cursors too and must be closed by the caller. To release leaked cursors immediately, recycle the affected connections; ending a session closes all of its cursors.&lt;/p&gt;

&lt;p&gt;ORA-01000 is a leak that took days to fill up, reported by whichever request came last. Before you raise anything, find the session by &lt;code&gt;opened cursors current&lt;/code&gt;, find the statement by &lt;code&gt;V$OPEN_CURSOR&lt;/code&gt; with the cached cursors filtered out, and fix the code that opens it. It's the same family of problem as &lt;a href="https://uptimearchitect.com/blog/oracle-bind-variables-hard-parse/" rel="noopener noreferrer"&gt;the hard-parse storm&lt;/a&gt;: application habits showing up as a database error. Then prove it with the &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/open-cursors" rel="noopener noreferrer"&gt;open-cursors lab&lt;/a&gt;, where one leaked result set per call takes a session down at call 299, and closing it runs 400 calls clean.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://uptimearchitect.com/blog/oracle-ora-01000-maximum-open-cursors/" rel="noopener noreferrer"&gt;uptimearchitect.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>oracle</category>
      <category>database</category>
      <category>performance</category>
      <category>java</category>
    </item>
    <item>
      <title>SYSDATE Is Hours Off After the Move to the Cloud. Nothing Errors.</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Sun, 08 Nov 2026 14:20:34 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/sysdate-is-hours-off-after-the-move-to-the-cloud-nothing-errors-4j06</link>
      <guid>https://dev.to/uptimearchitect/sysdate-is-hours-off-after-the-move-to-the-cloud-nothing-errors-4j06</guid>
      <description>&lt;p&gt;The migration went well. The database moved from the server room to the cloud over the weekend, the row counts matched, the application connected, and nobody saw an error. On Monday afternoon someone asks why the "today's orders" report has stopped showing the evening's orders. On Tuesday the nightly export picks up rows from the wrong day. The audit log and the application log disagree by five hours.&lt;/p&gt;

&lt;p&gt;Nothing is broken, and that's why it takes days to find. Every &lt;code&gt;SYSDATE&lt;/code&gt; in the database now returns &lt;strong&gt;UTC&lt;/strong&gt;, and every column with &lt;code&gt;DEFAULT SYSDATE&lt;/code&gt; is filling with UTC times, in the same column as years of local times.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;SYSDATE&lt;/code&gt; doesn't come from the database. It comes from the &lt;strong&gt;operating system's clock and time zone&lt;/strong&gt;, as the database server process sees them. The on-prem server ran in local time. The new host, like most cloud servers and nearly every container image, runs in UTC. This post shows exactly what moves, what doesn't, the fixes that look right and change nothing, and the two that work. All of it is reproduced in a lab.&lt;/p&gt;

&lt;h2&gt;
  
  
  What moves, and what doesn't
&lt;/h2&gt;

&lt;p&gt;Oracle has several "now" functions, and they don't all read the same clock:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;function&lt;/th&gt;
&lt;th&gt;where its time zone comes from&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;SYSDATE&lt;/code&gt;, &lt;code&gt;SYSTIMESTAMP&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;the &lt;strong&gt;server process's&lt;/strong&gt; OS time zone (&lt;code&gt;TZ&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CURRENT_DATE&lt;/code&gt;, &lt;code&gt;CURRENT_TIMESTAMP&lt;/code&gt;, &lt;code&gt;LOCALTIMESTAMP&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;the &lt;strong&gt;session&lt;/strong&gt; time zone, which the client sets&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;DBTIMEZONE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;a database setting, used for &lt;code&gt;TIMESTAMP WITH LOCAL TIME ZONE&lt;/code&gt; columns&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The first row is the one that matters, because it's what applications use: &lt;code&gt;DEFAULT SYSDATE&lt;/code&gt; on a column, &lt;code&gt;WHERE order_date &amp;gt;= TRUNC(SYSDATE)&lt;/code&gt; in a report, &lt;code&gt;SYSDATE - 1&lt;/code&gt; in a purge job. When the database moves to a host in another time zone, all of it moves with the host.&lt;/p&gt;

&lt;p&gt;An order placed at 8pm in Chicago is stored as 1am the &lt;strong&gt;next day&lt;/strong&gt; (2am in winter). Every report that groups by &lt;code&gt;TRUNC(order_date)&lt;/code&gt; puts the evening's orders on tomorrow.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it
&lt;/h2&gt;

&lt;p&gt;The lab runs Oracle Database Free in a container and "moves" it the only way that matters for &lt;code&gt;SYSDATE&lt;/code&gt;: it starts the instance with &lt;code&gt;TZ=America/Chicago&lt;/code&gt; (the on-prem server), then restarts the same instance in UTC (the cloud host). An application schema has an &lt;code&gt;ORDERS&lt;/code&gt; table with &lt;code&gt;ORDER_DATE DATE DEFAULT SYSDATE&lt;/code&gt;. Before and after the move, it checks what the application's session and a scheduler job see, and places one order:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                                      on-prem (Chicago)    moved (UTC)
app session, SYSDATE                  07:28                12:28
SYSDATE offset from UTC               -5 h                 0 h
scheduler job, SYSDATE offset         -5 h                 0 h

order 1  on-prem         2026-10-31 07:28
order 2  in the cloud    2026-10-31 12:28     &amp;lt;- placed seconds later
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two orders placed seconds apart are stored &lt;strong&gt;5 hours apart&lt;/strong&gt; (6 in winter, when Chicago is UTC-6). The second one is in UTC, in a column full of Chicago times, and nothing in the database records which is which.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't take my word for it — run it.&lt;/strong&gt; The &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/sysdate-timezone" rel="noopener noreferrer"&gt;sysdate-timezone lab&lt;/a&gt; &lt;strong&gt;asserts&lt;/strong&gt; every lab result in this post: the jump to UTC for the app and for a scheduler job, the 5-hour gap, each fix below that doesn't work, the two that do, and the DST trap. Chicago's offset is read from the database at run time, so it holds all year. If any of it stops reproducing, the run fails. Proven on every CI push.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Four fixes that change nothing
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Setting the database time zone.&lt;/strong&gt; The first thing most people try:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;TIME_ZONE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'America/Chicago'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab, after reopening the PDB, &lt;code&gt;DBTIMEZONE&lt;/code&gt; is &lt;code&gt;America/Chicago&lt;/code&gt; and the application's &lt;code&gt;SYSDATE&lt;/code&gt; is &lt;strong&gt;still UTC&lt;/strong&gt;. &lt;code&gt;DBTIMEZONE&lt;/code&gt; only applies to &lt;code&gt;TIMESTAMP WITH LOCAL TIME ZONE&lt;/code&gt; data. &lt;code&gt;SYSDATE&lt;/code&gt; ignores it (unless you turn on the setting in fix 2 below).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Checking from a DBA shell.&lt;/strong&gt; A DBA logs in to the new host, sets &lt;code&gt;TZ=America/Chicago&lt;/code&gt; in their shell, runs &lt;code&gt;sqlplus / as sysdba&lt;/code&gt;, and sees Chicago time. Fixed? In the lab, at the same moment, the application's session was still on UTC. A local &lt;code&gt;/ as sysdba&lt;/code&gt; connection's server process is started by your &lt;code&gt;sqlplus&lt;/code&gt;, so it takes the &lt;code&gt;TZ&lt;/code&gt; of &lt;strong&gt;your shell&lt;/strong&gt;. The application's connections come in over the listener and don't. Always check over the application's connection: &lt;code&gt;SELECT SYSDATE, SYSTIMESTAMP FROM dual&lt;/code&gt; through the same service the app uses.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Setting &lt;code&gt;TZ&lt;/code&gt; on the listener.&lt;/strong&gt; Many runbooks say the listener's environment is what server processes inherit. On 26ai in the lab, restarting only the listener with &lt;code&gt;TZ=America/Chicago&lt;/code&gt; changed nothing; the application's server processes took the &lt;strong&gt;instance's&lt;/strong&gt; &lt;code&gt;TZ&lt;/code&gt;. Older releases may behave differently, and setting it on both is harmless, but the listener alone didn't do it here.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Switching the code to &lt;code&gt;CURRENT_DATE&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;CURRENT_DATE&lt;/code&gt; uses the session's time zone, which the &lt;strong&gt;client&lt;/strong&gt; sets. With an app server in UTC (and in the cloud, the app servers usually are), it's UTC too. In the lab it returned Chicago time only when the client set &lt;code&gt;ORA_SDTZ=America/Chicago&lt;/code&gt;. That works, but it means every client, batch job and tool has to set it, and the first one that doesn't writes UTC again.&lt;/p&gt;

&lt;h2&gt;
  
  
  The scheduler is fine. Except for one thing.
&lt;/h2&gt;

&lt;p&gt;I expected the 2am jobs to start running at 9pm. They didn't. &lt;code&gt;DBMS_SCHEDULER&lt;/code&gt; stores a time zone with each job (in &lt;code&gt;START_DATE&lt;/code&gt;, or the scheduler's &lt;code&gt;default_timezone&lt;/code&gt;, which the database keeps), so after the move the lab's 2am job was still due at 2am Chicago:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NIGHTLY_REGION   next run due 02:00 Chicago (after the move)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The trap is a job whose &lt;code&gt;START_DATE&lt;/code&gt; was written as a &lt;strong&gt;fixed offset&lt;/strong&gt;. &lt;code&gt;SYSTIMESTAMP&lt;/code&gt; returns one (&lt;code&gt;-05:00&lt;/code&gt; in summer), and so does any &lt;code&gt;start_date =&amp;gt; SYSTIMESTAMP&lt;/code&gt;. A fixed offset doesn't know about daylight saving time. In the lab, a 2am job stored with &lt;code&gt;-05:00&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                           next run, Chicago time
                           summer     after the November DST change
start_date ... -05:00      02:00      01:00
start_date ... Chicago     02:00      02:00
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It runs an hour early from November to March, every year, whether or not you move the database. Find them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;job_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;TO_CHAR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start_date&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'TZR'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;tz&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;dba_scheduler_jobs&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;REGEXP_LIKE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;TO_CHAR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start_date&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'TZR'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="s1"&gt;'^[+-]'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and recreate them with a region name, &lt;code&gt;TO_TIMESTAMP_TZ('... America/Chicago', '... TZR')&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fix 1: give the instance its time zone back
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;SYSDATE&lt;/code&gt; follows the instance's &lt;code&gt;TZ&lt;/code&gt;, so set &lt;code&gt;TZ&lt;/code&gt; in the environment that &lt;strong&gt;starts the instance&lt;/strong&gt;, and restart it. In the lab, the instance restarted with &lt;code&gt;TZ=America/Chicago&lt;/code&gt; (the listener left in UTC) put the application's sessions and the scheduler's jobs back on Chicago time.&lt;/p&gt;

&lt;p&gt;Where that environment lives depends on how the database starts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A start script or systemd unit:&lt;/strong&gt; &lt;code&gt;Environment=TZ=America/Chicago&lt;/code&gt; in the unit, or &lt;code&gt;export TZ=...&lt;/code&gt; in the script. Not the DBA's &lt;code&gt;.bash_profile&lt;/code&gt;; the database has to get it when it starts without you.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Grid Infrastructure / Oracle Restart:&lt;/strong&gt; &lt;code&gt;srvctl setenv database -db &amp;lt;db&amp;gt; -env "TZ=America/Chicago"&lt;/code&gt;, then restart it with &lt;code&gt;srvctl&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A container:&lt;/strong&gt; &lt;code&gt;TZ&lt;/code&gt; in the container's environment (&lt;code&gt;environment:&lt;/code&gt; in Compose, &lt;code&gt;env&lt;/code&gt; in Kubernetes).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A managed database service:&lt;/strong&gt; you can't set the OS environment there. Check the provider's time zone setting.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The last three aren't tested in the lab; the lab sets &lt;code&gt;TZ&lt;/code&gt; the way the first does, in the environment of the process that runs &lt;code&gt;STARTUP&lt;/code&gt;. It needs an instance restart. It also has a cost: the database server is now the one server in your cloud that doesn't run in UTC, and everyone who works on it has to know that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fix 2: let the database answer in its own time zone (26ai)
&lt;/h2&gt;

&lt;p&gt;Oracle AI Database 26ai (the release that was 23ai, now version 23.26) added &lt;code&gt;TIME_AT_DBTIMEZONE&lt;/code&gt;; Oracle's reference says it's available "starting with Oracle AI Database 26ai". With it set to &lt;code&gt;USER_SQL&lt;/code&gt;, &lt;code&gt;SYSDATE&lt;/code&gt; and &lt;code&gt;SYSTIMESTAMP&lt;/code&gt; in SQL return the time in &lt;strong&gt;&lt;code&gt;DBTIMEZONE&lt;/code&gt;&lt;/strong&gt;, whatever the OS runs in. The host stays in UTC:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;TIME_ZONE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'America/Chicago'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;SYSTEM&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;time_at_dbtimezone&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;USER_SQL&lt;/span&gt; &lt;span class="k"&gt;SCOPE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SPFILE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;-- then reopen the PDB (or restart the database)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab (run in the PDB, then a PDB close and open), the application's &lt;code&gt;SYSDATE&lt;/code&gt; was Chicago time again with the OS still in UTC, so was a scheduler job's, and a new order's &lt;code&gt;DEFAULT SYSDATE&lt;/code&gt; matched the on-prem one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;app sees: SYSDATE=07:28  SYSTIMESTAMP=07:28 AMERICA/CHICAGO   (OS clock: 12:28 UTC)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things to know before you try it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;ALTER DATABASE SET TIME_ZONE&lt;/code&gt; fails with ORA-30079&lt;/strong&gt; while any &lt;code&gt;TIMESTAMP WITH LOCAL TIME ZONE&lt;/code&gt; column holds data. In the lab one row in such a column was enough; an empty column didn't block it. Check first:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;  &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;column_name&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;dba_tab_columns&lt;/span&gt;
  &lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;data_type&lt;/span&gt; &lt;span class="k"&gt;LIKE&lt;/span&gt; &lt;span class="s1"&gt;'TIMESTAMP(%) WITH LOCAL TIME ZONE'&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="k"&gt;owner&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'SYS'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'SYSTEM'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It's 26ai.&lt;/strong&gt; On 19c, or an older 23ai Release Update without the parameter, you have fix 1. (You may also see &lt;code&gt;SYSDATE_AT_DBTIMEZONE&lt;/code&gt;. Oracle documents it for Autonomous Database Serverless only; on the lab's 26ai Free it can't be changed, ORA-60490.)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The parameter is set per PDB and only takes effect at the next open. Oracle also documents a &lt;code&gt;DATABASE&lt;/code&gt; value, which applies &lt;code&gt;DBTIMEZONE&lt;/code&gt; to all time-dependent operations, not just &lt;code&gt;SYSDATE&lt;/code&gt; and &lt;code&gt;SYSTIMESTAMP&lt;/code&gt; in SQL. The lab tests &lt;code&gt;USER_SQL&lt;/code&gt; only.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpNkWFPwjAQhv_KpZ8ggsJIMBDFDDaUZDhgJUacIXUrsGRrSVchBvjvXgtEv7V39z7ve-2BJDLlpAtklct9smFKA_ViARB9xCR6jzyX-pCVMKeDhy9112MrzRXoDYdC7nhMPqFe78H0EJNQsSTn4LRZZicrTuvWaVdBKsgZip5icjLgqVEcY_LDy5gcIUApHY39iLrjCbyN6AsE4cANLMM0YBG--pAyza6E4EIQ0gCGDib1-mbUTt6AOS5duvwrWtgjzCN_toymQQ0Ul1su7B4Tr49r_Odisi6EM7feajTuO9aj-T-5kFBpdpLquYPudAHZGcbFLlNSFFxovDNtjUuNz1rafibwIhJuEtjyxXrYtO_4jLCBVIon2o4ruS8tYq8yrTHwxWbNtlehcxaac5KzskRUDYtrKVNSA1JwVbAsxR8-EFQW9q9TvmLfuSan0y_napyd" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpNkWFPwjAQhv_KpZ8ggsJIMBDFDDaUZDhgJUacIXUrsGRrSVchBvjvXgtEv7V39z7ve-2BJDLlpAtklct9smFKA_ViARB9xCR6jzyX-pCVMKeDhy9112MrzRXoDYdC7nhMPqFe78H0EJNQsSTn4LRZZicrTuvWaVdBKsgZip5icjLgqVEcY_LDy5gcIUApHY39iLrjCbyN6AsE4cANLMM0YBG--pAyza6E4EIQ0gCGDib1-mbUTt6AOS5duvwrWtgjzCN_toymQQ0Ul1su7B4Tr49r_Odisi6EM7feajTuO9aj-T-5kFBpdpLquYPudAHZGcbFLlNSFFxovDNtjUuNz1rafibwIhJuEtjyxXrYtO_4jLCBVIon2o4ruS8tYq8yrTHwxWbNtlehcxaac5KzskRUDYtrKVNSA1JwVbAsxR8-EFQW9q9TvmLfuSan0y_napyd" alt="Which fix fits: the release decides whether the database can keep its own time zone; other" width="420" height="1060"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Which fix fits: the release decides whether the database can keep its own time zone; otherwise the instance has to start in it. Either way, the rows written before the fix need correcting.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The rows written in between
&lt;/h2&gt;

&lt;p&gt;Neither fix touches data. In the lab, the order placed between the move and the fix was still 5 hours off after both fixes. Every &lt;code&gt;DEFAULT SYSDATE&lt;/code&gt; column, and everything the application wrote with &lt;code&gt;SYSDATE&lt;/code&gt;, is UTC for that window.&lt;/p&gt;

&lt;p&gt;Find the window by key, not by time: the first and last order ID (or audit row, or batch run) after the move and before the fix. Then convert those rows from UTC to local time, letting the database handle daylight saving:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt;    &lt;span class="n"&gt;order_date&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;CAST&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;FROM_TZ&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;CAST&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order_date&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="nb"&gt;TIMESTAMP&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="s1"&gt;'UTC'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                         &lt;span class="k"&gt;AT&lt;/span&gt; &lt;span class="nb"&gt;TIME&lt;/span&gt; &lt;span class="k"&gt;ZONE&lt;/span&gt; &lt;span class="s1"&gt;'America/Chicago'&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="nb"&gt;DATE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;BETWEEN&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;first_id_after_move&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;last_id_before_fix&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(For a UTC value of &lt;code&gt;2026-11-03 01:30&lt;/code&gt; that gives &lt;code&gt;2026-11-02 19:30&lt;/code&gt;, six hours back and on the previous day.) Do it once, for each table, and keep the list of what you changed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before the next move
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Record what the app sees&lt;/strong&gt; on the old server: &lt;code&gt;SELECT SYSDATE, SYSTIMESTAMP, SESSIONTIMEZONE FROM dual&lt;/code&gt; over the application's own service. That's the answer the new server has to give.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Decide the fix before cutover&lt;/strong&gt;: &lt;code&gt;TZ&lt;/code&gt; for the instance, or &lt;code&gt;TIME_AT_DBTIMEZONE&lt;/code&gt; on 26ai. Put it in the build, not in a ticket for after.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check over the app's connection&lt;/strong&gt; after the move, not from a DBA shell on the host.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fix the scheduler's fixed offsets&lt;/strong&gt; while you're there; that one bites every November regardless.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Longer term, the cleanest answer is the one cloud teams already use: store times in UTC, or in &lt;code&gt;TIMESTAMP WITH TIME ZONE&lt;/code&gt;, and convert for display. Then the server's time zone stops mattering. That's an application change, though, and it won't be done by Monday.&lt;/p&gt;

&lt;h2&gt;
  
  
  What teams get wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Testing the move by "does it connect and do the counts match."&lt;/strong&gt; Neither notices a 5-hour shift. Compare a &lt;code&gt;SYSDATE&lt;/code&gt; taken over the application's connection on both sides.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trusting &lt;code&gt;ALTER DATABASE SET TIME_ZONE&lt;/code&gt;.&lt;/strong&gt; It's the setting with "time zone" in the name, and on its own it does nothing for &lt;code&gt;SYSDATE&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checking the fix as SYSDBA from the host.&lt;/strong&gt; Your shell's &lt;code&gt;TZ&lt;/code&gt; is not the application's.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fixing the clock and forgetting the data.&lt;/strong&gt; Everything written between the move and the fix is still wrong, and the first report run over that range will say so.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Moving the problem to every client&lt;/strong&gt; by switching to &lt;code&gt;CURRENT_DATE&lt;/code&gt; and relying on each one to set its session time zone.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Why is Oracle SYSDATE showing the wrong time after a cloud migration?
&lt;/h3&gt;

&lt;p&gt;SYSDATE and SYSTIMESTAMP return the operating system's time in the time zone (TZ) of the database server process, not a database setting. On-prem servers usually run in local time, while cloud hosts and container images usually run in UTC, so after the move SYSDATE, and every column with DEFAULT SYSDATE, returns UTC. In the companion lab, two orders placed seconds apart before and after the move were stored 5 hours apart.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does ALTER DATABASE SET TIME_ZONE change SYSDATE?
&lt;/h3&gt;

&lt;p&gt;No. DBTIMEZONE is used for TIMESTAMP WITH LOCAL TIME ZONE data. In the lab, after setting it to America/Chicago and reopening the PDB, SYSDATE was still UTC. The exception is Oracle AI Database 26ai with TIME_AT_DBTIMEZONE set to USER_SQL, which makes SYSDATE and SYSTIMESTAMP in SQL return the time in DBTIMEZONE.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the difference between SYSDATE and CURRENT_DATE in Oracle?
&lt;/h3&gt;

&lt;p&gt;SYSDATE uses the database server process's operating system time zone. CURRENT_DATE uses the session time zone, which the client sets (for example with the ORA_SDTZ environment variable or ALTER SESSION SET TIME_ZONE). With a client in UTC, CURRENT_DATE is UTC too; in the lab it returned local time only when the client set ORA_SDTZ.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I make SYSDATE return local time on a UTC server?
&lt;/h3&gt;

&lt;p&gt;Either set TZ (for example TZ=America/Chicago) in the environment that starts the database instance and restart it, or, on Oracle AI Database 26ai, set DBTIMEZONE to the local time zone and TIME_AT_DBTIMEZONE to USER_SQL, then reopen the PDB. Changing DBTIMEZONE fails with ORA-30079 while any TIMESTAMP WITH LOCAL TIME ZONE column holds data.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do Oracle scheduler jobs run at the wrong time after moving to a UTC server?
&lt;/h3&gt;

&lt;p&gt;Not if they were created with a time zone region: DBMS_SCHEDULER stores the time zone with the job, and in the lab a 2am America/Chicago job was still due at 2am Chicago after the move. Jobs whose START_DATE has a fixed offset such as -05:00 (what SYSTIMESTAMP returns) don't follow daylight saving time, and in the lab such a job's next run after the November DST change evaluated to 1am instead of 2am.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I fix rows that were stored in UTC after the migration?
&lt;/h3&gt;

&lt;p&gt;Identify the rows written between the move and the fix, preferably by key range, and convert them once with CAST(FROM_TZ(CAST(col AS TIMESTAMP), 'UTC') AT TIME ZONE 'America/Chicago' AS DATE), which also handles daylight saving time. Neither fix changes existing data.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;SYSDATE&lt;/code&gt; being hours off isn't a database bug. It's the database telling you what the new server's clock says, and the new server runs in UTC. Decide before cutover whether the instance keeps its old time zone or the database answers in its own, check it over the application's connection, and correct the rows from the gap. It belongs on the cutover checklist with the other things that pass every count and still break the morning after, like &lt;a href="https://uptimearchitect.com/blog/oracle-sequence-cutover-ora-00001/" rel="noopener noreferrer"&gt;sequences after cutover&lt;/a&gt; and &lt;a href="https://uptimearchitect.com/blog/oracle-character-set-migration-data-loss/" rel="noopener noreferrer"&gt;character-set conversion&lt;/a&gt;, and with choosing the &lt;a href="https://uptimearchitect.com/blog/oracle-cloud-migration-methods/" rel="noopener noreferrer"&gt;migration method&lt;/a&gt; in the first place. Then prove it the way that ends the argument — with the &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/sysdate-timezone" rel="noopener noreferrer"&gt;sysdate-timezone lab&lt;/a&gt;, where the same instance stores two orders seconds apart five hours apart, and the two fixes put them back.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://uptimearchitect.com/blog/oracle-sysdate-timezone-cloud-migration/" rel="noopener noreferrer"&gt;uptimearchitect.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>oracle</category>
      <category>database</category>
      <category>devops</category>
      <category>cloud</category>
    </item>
    <item>
      <title>ORA-01017 After a Password Change: Rotate It With Zero Downtime</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Wed, 04 Nov 2026 15:47:03 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/ora-01017-after-a-password-change-rotate-it-with-zero-downtime-3j3l</link>
      <guid>https://dev.to/uptimearchitect/ora-01017-after-a-password-change-rotate-it-with-zero-downtime-3j3l</guid>
      <description>&lt;p&gt;The security team wants the application account's password rotated. It's a one-line change:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;USER&lt;/span&gt; &lt;span class="n"&gt;app_user&lt;/span&gt; &lt;span class="n"&gt;IDENTIFIED&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="nv"&gt;"Summer_Passw0rd2"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The new password goes into the secrets store, and the app servers pick it up as they restart. Twenty minutes later the application is down, and the error is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-01017: invalid credential or not authorized; logon denied
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;followed by:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-28000: The account is locked
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ORA-01017 on its own just means a wrong username or password. After a password change it means something more specific: &lt;strong&gt;some clients still have the old one.&lt;/strong&gt; Their retries use up the account's failed-login limit, the account locks, and from then on it refuses &lt;em&gt;every&lt;/em&gt; login, including from the servers that already have the new password. The change was correct. The way it rolled out caused the outage.&lt;/p&gt;

&lt;p&gt;Oracle has a feature for exactly this: &lt;strong&gt;gradual password rollover&lt;/strong&gt;, in 19c from Release Update 19.12 and in 21c and later. For a window you choose, the old and the new password both work. This post shows the outage, the feature, the query that tells you which clients are still on the old password, and the two traps that come with it. All of it is reproduced in a lab.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why one password change takes everything down
&lt;/h2&gt;

&lt;p&gt;Three things combine.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Existing connections don't notice.&lt;/strong&gt; A password is checked only at login. Every session that was open before the change keeps working. So the change looks fine at first. The failures start later, when a connection pool recycles, a server restarts, or traffic grows and the pool opens new connections.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Not every client changes at the same moment.&lt;/strong&gt; App servers restart one at a time. A batch job reads its password from a different file. A reporting tool has the password saved in a connection profile nobody remembers. Some of them are on the old password for minutes, some for days.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Failed logins lock the account, not the client.&lt;/strong&gt; The profile's &lt;code&gt;FAILED_LOGIN_ATTEMPTS&lt;/code&gt; (10 in a stock &lt;code&gt;DEFAULT&lt;/code&gt; profile) counts failures for the &lt;em&gt;account&lt;/em&gt;. Once a few clients have retried with the old password, the account is &lt;code&gt;LOCKED(TIMED)&lt;/code&gt; for &lt;code&gt;PASSWORD_LOCK_TIME&lt;/code&gt; (1 day by default), and the correct password gets ORA-28000 too.&lt;/p&gt;

&lt;p&gt;So the incident looks like this: a password change, a quiet period, then ORA-01017 from a few servers, then ORA-28000 from all of them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it
&lt;/h2&gt;

&lt;p&gt;The lab runs Oracle Database Free in a container. It creates &lt;code&gt;APP_USER&lt;/code&gt; on a profile with &lt;code&gt;FAILED_LOGIN_ATTEMPTS 5&lt;/code&gt;, and audits its logins. Then it changes the password while "app servers" still log in with the old one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;password change, no rollover               result
old password, attempts 1-5                  ORA-01017: invalid credential or not authorized
old password, attempt 6                     ORA-28000: The account is locked
new password                                ORA-28000: The account is locked
connection opened before the change         still alive, still querying
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then the same change with a one-day rollover window:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;password change, PASSWORD_ROLLOVER_TIME 1   result
account status                              OPEN &amp;amp; IN ROLLOVER
old password, logins 1-7                    connected (7 logins, limit is 5 failures)
new password                                connected
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't take my word for it — run it.&lt;/strong&gt; The &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/password-rollover" rel="noopener noreferrer"&gt;password-rollover lab&lt;/a&gt; &lt;strong&gt;asserts&lt;/strong&gt; every row: ORA-01017 five times, then ORA-28000 for the old &lt;em&gt;and&lt;/em&gt; the new password without rollover; both passwords working with it; the audit trail telling them apart; a typo still locking the account during rollover; a second change dropping the middle password; and ending the window early. If any of that doesn't reproduce, the run fails. Proven on every CI push.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Gradual password rollover
&lt;/h2&gt;

&lt;p&gt;Rollover is a &lt;strong&gt;profile&lt;/strong&gt; limit, not an option of &lt;code&gt;ALTER USER&lt;/code&gt;. Set it on the profile the application account uses, &lt;em&gt;then&lt;/em&gt; change the password:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="n"&gt;PROFILE&lt;/span&gt; &lt;span class="n"&gt;app_prof&lt;/span&gt; &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="n"&gt;password_rollover_time&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;-- days: 1/24 = one hour&lt;/span&gt;

&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;USER&lt;/span&gt; &lt;span class="n"&gt;app_user&lt;/span&gt; &lt;span class="n"&gt;IDENTIFIED&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="nv"&gt;"Summer_Passw0rd2"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From that moment until the window ends, both passwords log in, and &lt;code&gt;DBA_USERS&lt;/code&gt; shows it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;account_status&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;dba_users&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'APP_USER'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;USERNAME   ACCOUNT_STATUS
APP_USER   OPEN &amp;amp; IN ROLLOVER
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The window is in days, from &lt;strong&gt;one hour&lt;/strong&gt; (&lt;code&gt;1/24&lt;/code&gt;) to &lt;strong&gt;60 days&lt;/strong&gt;; anything outside that is refused with ORA-02377 (the lab checks 30 minutes and 61 days). &lt;code&gt;0&lt;/code&gt;, the &lt;code&gt;DEFAULT&lt;/code&gt; profile's value, turns it off. The window starts at the password change, so a profile changed &lt;em&gt;after&lt;/em&gt; the password doesn't help that change.&lt;/p&gt;

&lt;p&gt;Pick a window longer than your slowest client takes to pick up the new password, including the weekly batch job and the server that only restarts at patch time. A day is short. A week is common.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNplkV1PgzAYhf_Km97okm3eG53BtVMSNkjBr4ghCOUj6foupbgs2_77CjJj9KZXPc855z17kmEuyDWQQuI2q1JtIKKxAgjeYxIKA4EThi8-pwn3Pc9_ZjyJ3CW7-dRXM1RgKgFplmGrzEUDG41FLUVMPmAymcHcIhwvYhyeQvtMp1NwKVtF7sJlFO7feshlh1BiC5u0abao85GVdwHmPYNaRtA2Ffz91osNgvgSegeZrIUyg5L2Smdv3dUObOTOkN9a_4nv0V4osUTVQD00aPPagNFpLe9icuwYTsc4xGQnmpgcYGFjLGqV2--pGdy-43fVHv0wGsPcc225JOD-A3eWycpZstH5FPQ3U-EYCtQgUZUgFLZl1Xmwf-fqHdhr4HL2swOcd4DAFvPp0DmT9i7AoETMyRjIWuh1Wud22T2xFdf9xrko0lYacjyeAIQRm2c" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNplkV1PgzAYhf_Km97okm3eG53BtVMSNkjBr4ghCOUj6foupbgs2_77CjJj9KZXPc855z17kmEuyDWQQuI2q1JtIKKxAgjeYxIKA4EThi8-pwn3Pc9_ZjyJ3CW7-dRXM1RgKgFplmGrzEUDG41FLUVMPmAymcHcIhwvYhyeQvtMp1NwKVtF7sJlFO7feshlh1BiC5u0abao85GVdwHmPYNaRtA2Ffz91osNgvgSegeZrIUyg5L2Smdv3dUObOTOkN9a_4nv0V4osUTVQD00aPPagNFpLe9icuwYTsc4xGQnmpgcYGFjLGqV2--pGdy-43fVHv0wGsPcc225JOD-A3eWycpZstH5FPQ3U-EYCtQgUZUgFLZl1Xmwf-fqHdhr4HL2swOcd4DAFvPp0DmT9i7AoETMyRjIWuh1Wud22T2xFdf9xrko0lYacjyeAIQRm2c" alt="A password rotation with rollover. The window keeps the old password working while clients" width="449" height="990"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A password rotation with rollover. The window keeps the old password working while clients move; the audit trail tells you when they all have; then you end the window instead of waiting for it.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Who is still on the old password?
&lt;/h2&gt;

&lt;p&gt;This is the question the rollover window gives you time to answer, and the database can answer it. With logins audited, every unified audit record for a successful login says which password it used, in &lt;code&gt;AUTHENTICATION_TYPE&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;(TYPE=(DATABASE));(CLIENT ADDRESS=((PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=56796)));
(LOGON_INFO=((VERIFIER=12C-OLD)(CLIENT_CAPABILITIES=O5L_NP,O7L_MR,O8L_LI)));
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;12C-OLD&lt;/code&gt; is the old password, &lt;code&gt;12C-NEW&lt;/code&gt; the new one. (That's what the lab's 26ai database records; check that your release writes &lt;code&gt;LOGON_INFO&lt;/code&gt; before you rely on it. If it doesn't, the same query without the verifier still lists every host and program that logged in since the change, which is your list of clients to check.) Count them since the change:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;regexp_substr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;authentication_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'VERIFIER=([^)]*)'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;verifier&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;userhost&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;client_program_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;logons&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;unified_audit_trail&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;dbusername&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'APP_USER'&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;action_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'LOGON'&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;return_code&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;event_timestamp&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;CAST&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;password_change_date&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="nb"&gt;TIMESTAMP&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                           &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;dba_users&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'APP_USER'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;regexp_substr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;authentication_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'VERIFIER=([^)]*)'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
          &lt;span class="n"&gt;userhost&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;client_program_name&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;verifier&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;userhost&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab (all clients on one host, so grouped by verifier alone), after seven logins with the old password and one with the new:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;VERIFIER=12C-NEW LOGONS=1
VERIFIER=12C-OLD LOGONS=7
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every &lt;code&gt;-OLD&lt;/code&gt; row is a client you still have to update, with its host and program next to it. This needs logins to be &lt;strong&gt;audited&lt;/strong&gt;. The lab creates a policy for the account:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="n"&gt;AUDIT&lt;/span&gt; &lt;span class="n"&gt;POLICY&lt;/span&gt; &lt;span class="n"&gt;app_logons&lt;/span&gt; &lt;span class="n"&gt;ACTIONS&lt;/span&gt; &lt;span class="n"&gt;LOGON&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;AUDIT&lt;/span&gt; &lt;span class="n"&gt;POLICY&lt;/span&gt; &lt;span class="n"&gt;app_logons&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;app_user&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab's database nothing audited logins until that policy existed; check &lt;code&gt;AUDIT_UNIFIED_ENABLED_POLICIES&lt;/code&gt; on yours. Successful logins can be a lot of rows for a busy pool, so scope the policy to the accounts you're rotating, and enable it before the change. The same trail answers the outage question, too: without rollover, the failed logins showed up as &lt;code&gt;ORA-01017 x5&lt;/code&gt; and &lt;code&gt;ORA-28000 x2&lt;/code&gt;, with the client host, and that's how you find the server that locked the account. See &lt;a href="https://uptimearchitect.com/blog/oracle-unified-auditing/" rel="noopener noreferrer"&gt;unified auditing without the noise&lt;/a&gt; for keeping policies like this small.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ending the window
&lt;/h2&gt;

&lt;p&gt;When the audit trail shows no more &lt;code&gt;-OLD&lt;/code&gt; logins for long enough to cover your slowest client, end the window instead of waiting it out:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;USER&lt;/span&gt; &lt;span class="n"&gt;app_user&lt;/span&gt; &lt;span class="n"&gt;EXPIRE&lt;/span&gt; &lt;span class="n"&gt;PASSWORD&lt;/span&gt; &lt;span class="n"&gt;ROLLOVER&lt;/span&gt; &lt;span class="n"&gt;PERIOD&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab the status goes back to &lt;code&gt;OPEN&lt;/code&gt;, the old password gets ORA-01017, the new one connects, and a connection opened earlier with the old password &lt;strong&gt;is still alive and querying&lt;/strong&gt;. Ending the rollover, like the change itself, only affects new logins.&lt;/p&gt;

&lt;p&gt;That's also the reason to end it deliberately. While the window is open, anyone with the old password can still log in. If the rotation was because the old password leaked, a long window keeps the leak open, so pick the window to match the reason for the change.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two traps
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;A wrong password still counts.&lt;/strong&gt; Rollover accepts &lt;em&gt;two&lt;/em&gt; passwords, not any password. In the lab, five logins with a typo during the rollover window got ORA-01017, and then the account was locked: the new password and the old one both got ORA-28000. A client with a mistyped secret locks the account just as it did before. Rollover removes the outage caused by timing; it doesn't remove the lockout.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A second change drops the middle password.&lt;/strong&gt; If the password is changed again inside the window (a rerun of the rotation job, a second ticket, someone "fixing" a typo in the new password), the window continues, but only the &lt;strong&gt;original&lt;/strong&gt; and the &lt;strong&gt;latest&lt;/strong&gt; password work:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;original password (Spring)        connected
first new password (Summer)       ORA-01017
latest password (Autumn)          connected
account status                    OPEN &amp;amp; IN ROLLOVER
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Any client that already moved to the middle password is now failing, and the account status looks exactly the same as it did a minute earlier. If you need a second change, let the first rollover finish (or end it) first.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rotation, step by step
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Know the clients.&lt;/strong&gt; Audit logins for the account for a while before the change. Each distinct &lt;code&gt;USERHOST&lt;/code&gt; / &lt;code&gt;CLIENT_PROGRAM_NAME&lt;/code&gt; is a client that needs the new password.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Set the window&lt;/strong&gt; on the account's profile: &lt;code&gt;ALTER PROFILE ... LIMIT password_rollover_time 7&lt;/code&gt;. If the profile is shared with accounts you don't want to roll over, give this account its own profile.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Change the password once.&lt;/strong&gt; &lt;code&gt;ALTER USER ... IDENTIFIED BY ...&lt;/code&gt;. Check &lt;code&gt;OPEN &amp;amp; IN ROLLOVER&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Push the new password&lt;/strong&gt; to every client, at whatever pace your deployments allow.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch the audit trail&lt;/strong&gt; until there are no &lt;code&gt;-OLD&lt;/code&gt; logins, and chase the ones that remain.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;End the window&lt;/strong&gt;: &lt;code&gt;ALTER USER ... EXPIRE PASSWORD ROLLOVER PERIOD&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If the account is already locked when you start reading this: &lt;code&gt;ALTER USER app_user ACCOUNT UNLOCK&lt;/code&gt;, find the client that's still failing (the audit trail has its host, if failed logins are audited), and fix it before it locks the account again.&lt;/p&gt;

&lt;h2&gt;
  
  
  What teams get wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Changing the password and the clients "at the same time."&lt;/strong&gt; There's no such moment. Some client is always last, and until it moves, it's failing logins against a shared limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Setting the rollover limit after the password change.&lt;/strong&gt; The window starts at the change. Set the profile first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reading &lt;code&gt;OPEN &amp;amp; IN ROLLOVER&lt;/code&gt; as "the old password works."&lt;/strong&gt; After a second change it's the original password that works, not the one most clients were just given.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Raising &lt;code&gt;FAILED_LOGIN_ATTEMPTS&lt;/code&gt; to UNLIMITED&lt;/strong&gt; so rotations stop locking the account. That removes a defence against password guessing for good, to fix a problem that lasts an hour. Use rollover, and keep the limit (see the &lt;a href="https://uptimearchitect.com/blog/oracle-database-hardening-checklist/" rel="noopener noreferrer"&gt;hardening checklist&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Leaving the window open for weeks after everyone has moved.&lt;/strong&gt; It's two valid passwords. End it once the audit trail is clean.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Why do I get ORA-01017 after changing a user's password?
&lt;/h3&gt;

&lt;p&gt;Because some client is still logging in with the old password. Sessions opened before the change keep working, so the failures appear later, when a connection pool opens new connections or a server restarts. Each failed login counts against the profile's FAILED_LOGIN_ATTEMPTS, and once the limit is reached the account locks and every login, including ones with the new password, fails with ORA-28000.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is gradual password rollover in Oracle?
&lt;/h3&gt;

&lt;p&gt;A profile limit, PASSWORD_ROLLOVER_TIME, available in Oracle 19c from Release Update 19.12 and in 21c and later. When an account's password is changed, both the old and the new password are accepted for that many days, so clients can move to the new password one at a time without login failures. The account status shows OPEN &amp;amp; IN ROLLOVER during the window. The window can be from one hour (1/24) to 60 days; 0 turns it off.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I find which clients still use the old password during a rollover?
&lt;/h3&gt;

&lt;p&gt;Audit logins for the account (CREATE AUDIT POLICY ... ACTIONS LOGON and AUDIT POLICY ... BY the user). Each successful login's AUTHENTICATION_TYPE in UNIFIED_AUDIT_TRAIL includes LOGON_INFO with VERIFIER=12C-OLD or VERIFIER=12C-NEW. Group the logins since the password change by verifier, USERHOST and CLIENT_PROGRAM_NAME; every OLD row is a client still to update. This was verified on Oracle 26ai; on other releases, check that LOGON_INFO is recorded.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I end a password rollover period early?
&lt;/h3&gt;

&lt;p&gt;Run ALTER USER username EXPIRE PASSWORD ROLLOVER PERIOD. The account status returns to OPEN and only the new password works for new logins. Sessions that are already connected, including ones that logged in with the old password, are not disconnected.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does password rollover prevent ORA-28000 account locked?
&lt;/h3&gt;

&lt;p&gt;Only for the old password. During the rollover window the old and new passwords both log in, so clients that haven't moved don't fail. A wrong password still fails with ORA-01017 and still counts toward FAILED_LOGIN_ATTEMPTS, and in the companion lab five wrong logins during the window locked the account for both passwords.&lt;/p&gt;

&lt;h3&gt;
  
  
  What happens if I change the password twice during a rollover?
&lt;/h3&gt;

&lt;p&gt;The rollover window continues, but only the original password (from before the first change) and the latest password are valid. The password in between stops working immediately, even though the account status still shows OPEN &amp;amp; IN ROLLOVER. Clients that had already moved to the middle password get ORA-01017.&lt;/p&gt;

&lt;p&gt;ORA-01017 after a password change isn't a mystery about the password. It's a rollout problem: some client is always last, and without a window, the last client locks out the first. Give the account a rollover window, change the password once, watch the audit trail until nobody is on the old one, and close the window yourself. It belongs with the other account controls in the &lt;a href="https://uptimearchitect.com/blog/oracle-database-hardening-checklist/" rel="noopener noreferrer"&gt;hardening checklist&lt;/a&gt; and with the connection errors that look like outages, like &lt;a href="https://uptimearchitect.com/blog/oracle-ora-12514-listener-service/" rel="noopener noreferrer"&gt;ORA-12514&lt;/a&gt;. Then prove it the way that ends the argument — with the &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/password-rollover" rel="noopener noreferrer"&gt;password-rollover lab&lt;/a&gt;, where the same password change locks the account without rollover and causes zero failed logins with it.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://uptimearchitect.com/blog/oracle-ora-01017-password-rollover/" rel="noopener noreferrer"&gt;uptimearchitect.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>oracle</category>
      <category>database</category>
      <category>security</category>
      <category>devops</category>
    </item>
    <item>
      <title>ORA-12514: The Database Is Up. The Listener Just Doesn't Know It.</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Sun, 01 Nov 2026 13:41:46 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/ora-12514-the-database-is-up-the-listener-just-doesnt-know-it-1o5d</link>
      <guid>https://dev.to/uptimearchitect/ora-12514-the-database-is-up-the-listener-just-doesnt-know-it-1o5d</guid>
      <description>&lt;p&gt;The application can't connect. The error is one every Oracle DBA has seen:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Newer clients word it more plainly. The 26ai client in this post's lab says:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-12514: Cannot connect to database. Service FREEPDB is not registered with the listener
at host 127.0.0.1 port 1521.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The database is up. You can log in on the server. So someone restarts the listener, and now &lt;em&gt;nothing&lt;/em&gt; can connect for a minute.&lt;/p&gt;

&lt;p&gt;ORA-12514 is a precise message, and it's good news of a kind: &lt;strong&gt;the listener answered.&lt;/strong&gt; It's running, the host and port are right, and the network path works. What it doesn't have is a service by the name the client asked for. The database behind it may be perfectly healthy. This post covers how services get to the listener, the four common reasons one is missing, and how to tell ORA-12514 from its neighbours. It ends with a lab where restarting the listener &lt;strong&gt;causes&lt;/strong&gt; a minute of ORA-12514 for every service, and one &lt;code&gt;ALTER SYSTEM REGISTER&lt;/code&gt; closes the gap at once.&lt;/p&gt;

&lt;h2&gt;
  
  
  How a service gets to the listener
&lt;/h2&gt;

&lt;p&gt;The listener doesn't know what databases exist. The database tells it. A background process (LREG) in each instance &lt;strong&gt;registers&lt;/strong&gt; the instance's services with the listener: &lt;code&gt;FREEPDB1&lt;/code&gt;, each application service, the CDB's own service. It does that at startup, whenever a service starts or stops, and periodically after that. The listener keeps that list in memory only.&lt;/p&gt;

&lt;p&gt;A client connects by asking for a service: &lt;code&gt;host:1521/FREEPDB1&lt;/code&gt; in EZConnect, or &lt;code&gt;SERVICE_NAME=FREEPDB1&lt;/code&gt; in a &lt;code&gt;tnsnames.ora&lt;/code&gt; descriptor. If the listener has a registered service by that name, it hands the connection to the instance. If not, it refuses with ORA-12514.&lt;/p&gt;

&lt;p&gt;So ORA-12514 has exactly two kinds of cause: &lt;strong&gt;the client asked for the wrong name&lt;/strong&gt;, or &lt;strong&gt;the right name isn't registered&lt;/strong&gt; with &lt;em&gt;this&lt;/em&gt; listener right now. &lt;code&gt;lsnrctl services&lt;/code&gt; (or &lt;code&gt;lsnrctl status&lt;/code&gt;) shows what the listener has:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Services Summary...
Service "FREE" has 1 instance(s).
Service "FREEXDB" has 1 instance(s).
Service "freepdb1" has 1 instance(s).
Service "orders_app" has 1 instance(s).
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the service is in that list, look at the client. If it isn't, look at the database.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which error do you actually have?
&lt;/h2&gt;

&lt;p&gt;Five errors get mixed up in tickets. They fail at different points in the connection, so they point to different fixes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-12154   the alias wasn't found (tnsnames.ora, LDAP)    the client never reached a listener
ORA-12541   no listener at that host and port              wrong host/port, listener down, firewall
ORA-12505   the listener doesn't know that SID             often a PDB name given as a SID
ORA-12514   the listener doesn't know that SERVICE         this post
ORA-12504   no service name given, and no default service  incomplete connect string
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ORA-12505 is the usual developer version. &lt;code&gt;jdbc:oracle:thin:@db01:1521:FREEPDB1&lt;/code&gt; (a colon before the name) asks for a &lt;strong&gt;SID&lt;/strong&gt;, which is an instance, not a PDB. &lt;code&gt;jdbc:oracle:thin:@//db01:1521/FREEPDB1&lt;/code&gt; (a slash) asks for the service, which is what a PDB is reached by.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNptktFKwzAUhl_lkBsVOsZEb4ZM6lZkUN1oiyBWJEuyLS5LSpJuyty7e9La4cXuQsL5_u_85ECY4YIMgSyV2bM1tR6KSakBkreSzLK4N7i-HdyU5B16vRGkh5JMHfi1ACfsTjIBUt8tbH-knLbMq-7a3ZfkGDBpmPspybdwJfmBMVILnGZKCu2Buo3gsDQWqDZItaDpVjTAS7YWbNNEKem80PiozOoKVf5ztYkgjEq9AopApUJMhjFpN_VZOw9WOI_LCT5s6HFaJBnkr3mRPEGWPE7xkJ1Bh_j55AFks7Rt1Ra1D5nNI62qbucQnGNw_tcMl1xfeGhyYS_9uqO1Cu299BHk8UsCeREXCfTB2R32eEaFao4AFOnqaCl1VRnrHQqdug8icxSZUE8X1KGJES6oWLEKsxbCKq3FUn5BOhvH6UcaOng-tcAUdQ7GURbl0RxWxnASAdkKu6WS44c5ENxm23wdLpa0Vp4cj7-i8MJd" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNptktFKwzAUhl_lkBsVOsZEb4ZM6lZkUN1oiyBWJEuyLS5LSpJuyty7e9La4cXuQsL5_u_85ECY4YIMgSyV2bM1tR6KSakBkreSzLK4N7i-HdyU5B16vRGkh5JMHfi1ACfsTjIBUt8tbH-knLbMq-7a3ZfkGDBpmPspybdwJfmBMVILnGZKCu2Buo3gsDQWqDZItaDpVjTAS7YWbNNEKem80PiozOoKVf5ztYkgjEq9AopApUJMhjFpN_VZOw9WOI_LCT5s6HFaJBnkr3mRPEGWPE7xkJ1Bh_j55AFks7Rt1Ra1D5nNI62qbucQnGNw_tcMl1xfeGhyYS_9uqO1Cu299BHk8UsCeREXCfTB2R32eEaFao4AFOnqaCl1VRnrHQqdug8icxSZUE8X1KGJES6oWLEKsxbCKq3FUn5BOhvH6UcaOng-tcAUdQ7GURbl0RxWxnASAdkKu6WS44c5ENxm23wdLpa0Vp4cj7-i8MJd" alt="ORA-12514 means the listener answered and has no service by that name. Compare what the cl" width="764" height="594"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;ORA-12514 means the listener answered and has no service by that name. Compare what the client asked for (the listener log records it) with what the listener has (lsnrctl services). A wrong name is a client fix; a missing registration is a database fix.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it
&lt;/h2&gt;

&lt;p&gt;The lab runs Oracle Database Free in a container. It creates an application service, &lt;code&gt;orders_app&lt;/code&gt;, in the pluggable database &lt;code&gt;FREEPDB1&lt;/code&gt; the way many shops do: with &lt;code&gt;DBMS_SERVICE&lt;/code&gt;, started by hand. Then it breaks the connection four ways, with the database up and healthy every time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;case                                     result                                   fix
FREEPDB instead of FREEPDB1              ORA-12514; FREEPDB1 connects fine        fix the connect string
listener restarted                       ORA-12514 for every service, ~60 s       ALTER SYSTEM REGISTER
PDB closed and opened                    PDB READ WRITE, orders_app ORA-12514     start it + SAVE STATE
second listener on port 1522             "supports no services", ORA-12514        LOCAL_LISTENER
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't take my word for it — run it.&lt;/strong&gt; The &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/listener-service" rel="noopener noreferrer"&gt;listener-service lab&lt;/a&gt; &lt;strong&gt;asserts&lt;/strong&gt; every row: ORA-12514 for the wrong name and the right name still connecting; ORA-12514 right after a listener restart and recovery once the database re-registers; the app service missing after a PDB restart and surviving the next one after &lt;code&gt;SAVE STATE&lt;/code&gt;; and the second listener refusing until &lt;code&gt;LOCAL_LISTENER&lt;/code&gt; names it. It also asserts ORA-12154, ORA-12541 and ORA-12505 for their cases. If any of that doesn't reproduce, the run fails. Proven on every CI push.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Cause 1: the wrong name
&lt;/h2&gt;

&lt;p&gt;The most common cause, and the easiest to miss because the name &lt;em&gt;looks&lt;/em&gt; right. &lt;code&gt;FREEPDB&lt;/code&gt; instead of &lt;code&gt;FREEPDB1&lt;/code&gt;, &lt;code&gt;orders&lt;/code&gt; instead of &lt;code&gt;orders_app&lt;/code&gt;, a domain mismatch (the database has a &lt;code&gt;DB_DOMAIN&lt;/code&gt;, so its service is &lt;code&gt;freepdb1.example.com&lt;/code&gt;, and the client asks for plain &lt;code&gt;freepdb1&lt;/code&gt;, or the other way round), or a connect string still pointing at the old database's service name after a &lt;a href="https://uptimearchitect.com/blog/oracle-cloud-migration-methods/" rel="noopener noreferrer"&gt;migration&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;You don't have to guess which app is asking for what. &lt;strong&gt;The listener log records every refused connect&lt;/strong&gt; with the service the client requested, the client program, and the client host. From the lab:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;(CONNECT_DATA=(SERVICE_NAME=FREEPDB)(CID=(PROGRAM=sqlplus)(HOST=...)(USER=oracle)))
  * (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=59690)) * establish * FREEPDB * 12514
TNS-12514: listener does not currently know of service requested in connect descriptor
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Even on 26ai the log uses the classic text. &lt;code&gt;lsnrctl status&lt;/code&gt; prints the log's location. On a busy listener, grep it for &lt;code&gt;12514&lt;/code&gt; and count by &lt;code&gt;SERVICE_NAME&lt;/code&gt; and &lt;code&gt;HOST&lt;/code&gt;: the typo and the server it lives on fall out in one command.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cause 2: the listener was restarted
&lt;/h2&gt;

&lt;p&gt;This is the one that turns a small problem into an outage. A listener knows only what's been registered with it since it started. Restart it, and for a while it knows nothing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ lsnrctl status
...
The listener supports no services
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every service gets ORA-12514 until the database registers again on its own. In the lab that took &lt;strong&gt;about a minute&lt;/strong&gt;, every time: 61 seconds of refused connections for every service, caused entirely by the restart. Restarting the listener to "fix" ORA-12514 makes it worse.&lt;/p&gt;

&lt;p&gt;The database can be told to register right now:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;SYSTEM&lt;/span&gt; &lt;span class="n"&gt;REGISTER&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab, after the same restart, the connection works immediately after this command. Run it in every instance (and every RAC node) whose services that listener serves. &lt;code&gt;lsnrctl reload&lt;/code&gt; re-reads &lt;code&gt;listener.ora&lt;/code&gt; without a stop/start, but don't count on it keeping the registrations: in the lab it sometimes kept every one and sometimes dropped them all. Follow a reload with &lt;code&gt;ALTER SYSTEM REGISTER&lt;/code&gt; too.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cause 3: the service didn't come back with the database
&lt;/h2&gt;

&lt;p&gt;The PDB is open. &lt;code&gt;FREEPDB1&lt;/code&gt; connects. The application's service doesn't:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;FREEPDB1 is READ WRITE; active services: freepdb1
localhost:1521/FREEPDB1    -&amp;gt; connected
localhost:1521/orders_app  -&amp;gt; ORA-12514: ... Service orders_app is not registered with the listener ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A service created with &lt;code&gt;DBMS_SERVICE&lt;/code&gt; and started by hand is running only until the next time the PDB closes. After a patch, a server reboot or a &lt;a href="https://uptimearchitect.com/blog/oracle-data-guard-switchover-vs-failover/" rel="noopener noreferrer"&gt;switchover&lt;/a&gt;, the PDB comes back and its default service with it, but nothing starts &lt;code&gt;orders_app&lt;/code&gt;. The database is fine. The application is down.&lt;/p&gt;

&lt;p&gt;On a single instance, start the service and save the PDB's state, so it reopens with the same services running:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;SESSION&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;CONTAINER&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;FREEPDB1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;EXEC&lt;/span&gt; &lt;span class="n"&gt;DBMS_SERVICE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;START_SERVICE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'orders_app'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="n"&gt;PLUGGABLE&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="n"&gt;FREEPDB1&lt;/span&gt; &lt;span class="n"&gt;SAVE&lt;/span&gt; &lt;span class="k"&gt;STATE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The lab restarts the PDB again after this, and &lt;code&gt;orders_app&lt;/code&gt; connects. Under Grid Infrastructure or RAC, define the service in Clusterware instead (&lt;code&gt;srvctl add service -db ... -pdb FREEPDB1 -service orders_app&lt;/code&gt;) and it starts with the database. With Data Guard, the same command takes &lt;code&gt;-role PRIMARY&lt;/code&gt;, so the application's service runs only where the primary is, and follows it on switchover. That last case is outside what the lab builds.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cause 4: a listener the database doesn't register with
&lt;/h2&gt;

&lt;p&gt;A new listener on port 1522, for an application tier or a firewall rule, or a listener moved off 1521. Clients pointed at it get ORA-12514, and its status says why:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ lsnrctl status LISTENER2
...
The listener supports no services
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The database registers only with the listeners it knows about. By default that's the local listener on port 1521. Any other address has to be in &lt;code&gt;LOCAL_LISTENER&lt;/code&gt; (or &lt;code&gt;REMOTE_LISTENER&lt;/code&gt;, for RAC and SCAN):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;SYSTEM&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;local_listener&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="s1"&gt;'(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=db01)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=db01)(PORT=1522)))'&lt;/span&gt;
  &lt;span class="k"&gt;SCOPE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;BOTH&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab, both services connect through port 1522 right after this; changing the parameter triggers a registration. (The lab uses &lt;code&gt;SCOPE = MEMORY&lt;/code&gt; so a rerun starts clean.) On a Grid Infrastructure node, the agent manages &lt;code&gt;LOCAL_LISTENER&lt;/code&gt;; register a new listener with &lt;code&gt;srvctl&lt;/code&gt; instead of setting it by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  The checklist
&lt;/h2&gt;

&lt;p&gt;When ORA-12514 comes in, in this order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Read the full message.&lt;/strong&gt; The service name and the host and port the client used are in it (newer clients) or in the listener log (all releases).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;lsnrctl services&lt;/code&gt;&lt;/strong&gt; on that host. Is the service listed? If yes, the client has the wrong name or the wrong listener.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;SELECT name FROM v$active_services&lt;/code&gt;&lt;/strong&gt; in the database (in the PDB). Is the service running? If not, start it, and make sure it starts next time (&lt;code&gt;SAVE STATE&lt;/code&gt; or &lt;code&gt;srvctl&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;SHOW PARAMETER local_listener&lt;/code&gt;&lt;/strong&gt;. Does it include the address the client is using?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;ALTER SYSTEM REGISTER&lt;/code&gt;&lt;/strong&gt;. Re-registers everything now. Harmless, instant, and much better than a listener restart.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What teams get wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Restarting the listener first.&lt;/strong&gt; It refuses every service until the database re-registers, about a minute in the lab. If a restart or a reload really is needed, follow it with &lt;code&gt;ALTER SYSTEM REGISTER&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating ORA-12514 as "the database is down."&lt;/strong&gt; It's a naming or registration problem. The lab's database was up and open through every case.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Starting app services by hand.&lt;/strong&gt; They don't survive a PDB restart. Use &lt;code&gt;SAVE STATE&lt;/code&gt; or Clusterware, and test it by restarting the PDB before the patch weekend does it for you.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Giving a PDB name as a SID.&lt;/strong&gt; That's ORA-12505, not ORA-12514. PDBs are reached by service name: &lt;code&gt;/FREEPDB1&lt;/code&gt;, not &lt;code&gt;:FREEPDB1&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Relying on a default service.&lt;/strong&gt; The lab's container has &lt;code&gt;DEFAULT_SERVICE_LISTENER&lt;/code&gt; set, so a connect string with no service name quietly lands in the CDB root. Without that parameter it fails with ORA-12504. Neither is where the application should be; always name the service.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What does ORA-12514 TNS:listener does not currently know of service requested in connect descriptor mean?
&lt;/h3&gt;

&lt;p&gt;It means the client reached a listener, and the listener has no registered service with the name the client asked for. The listener is up and reachable; the database may be healthy too. Either the client asked for the wrong service name (a typo, a missing or extra domain, an old name after a migration), or the right service isn't registered with that listener: because the listener was just restarted, the service isn't running, or the database doesn't register with that listener. Newer clients word the same error as 'Cannot connect to database. Service X is not registered with the listener at host ... port ...'.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I fix ORA-12514?
&lt;/h3&gt;

&lt;p&gt;Run lsnrctl services on the database host and check whether the service is listed. If it is, fix the client's connect string; the listener log records the exact service name each refused client asked for. If it isn't, check v$active_services in the database: start the service if it's stopped and make it start automatically (SAVE STATE on a single instance, srvctl under Grid Infrastructure). Check that LOCAL_LISTENER includes the listener's address. Then run ALTER SYSTEM REGISTER to register everything with the listener immediately.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why do I get ORA-12514 after restarting the listener?
&lt;/h3&gt;

&lt;p&gt;Because a newly started listener knows no services. The database registers its services again on its own, which took about a minute in the companion lab, and until then every service is refused with ORA-12514. ALTER SYSTEM REGISTER in each instance makes the database register immediately. lsnrctl reload, which re-reads listener.ora without a stop/start, sometimes dropped the registrations in the lab too, so follow it with ALTER SYSTEM REGISTER as well.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the difference between ORA-12514, ORA-12505, ORA-12541 and ORA-12154?
&lt;/h3&gt;

&lt;p&gt;They fail at different steps. ORA-12154 means the client couldn't resolve the alias (in tnsnames.ora or LDAP) and never reached a listener. ORA-12541 means nothing is listening at that host and port. ORA-12505 means the listener doesn't know the SID requested, often because a PDB name was given as a SID (host:1521:PDB instead of //host:1521/PDB). ORA-12514 means the listener answered but has no service by the requested name.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does my PDB's application service disappear after a restart?
&lt;/h3&gt;

&lt;p&gt;A service created with DBMS_SERVICE and started manually is not started automatically when the PDB opens again. The PDB and its default service come back, so the database looks healthy, but connections to the application service fail with ORA-12514. On a single instance, start the service and run ALTER PLUGGABLE DATABASE ... SAVE STATE so the PDB reopens with the same services. Under Grid Infrastructure or RAC, create the service with srvctl add service so Clusterware starts it, with -role PRIMARY for Data Guard.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does ORA-12514 mean the database is down?
&lt;/h3&gt;

&lt;p&gt;No. ORA-12514 is about registration and naming, not database health. In the companion lab the database was up and the PDB open READ WRITE in every case that produced ORA-12514. If the instance were down, a listener with only dynamic registration would also report ORA-12514, so check both: lsnrctl services for what the listener has, and the instance itself for whether it's running.&lt;/p&gt;

&lt;p&gt;ORA-12514 is a listener telling you exactly what's wrong: it's here, and it doesn't have that name. Before you restart anything, compare what the client asked for (the listener log) with what the listener has (&lt;code&gt;lsnrctl services&lt;/code&gt;), and let that point you at the client or the database. Then register, don't restart. It sits with the other Oracle Net settings worth getting right, like &lt;a href="https://uptimearchitect.com/blog/oracle-sqlnet-network-encryption/" rel="noopener noreferrer"&gt;network encryption&lt;/a&gt;, and with the cutover-weekend surprises like &lt;a href="https://uptimearchitect.com/blog/oracle-sequence-cutover-ora-00001/" rel="noopener noreferrer"&gt;the sequence that wasn't moved&lt;/a&gt;. Then prove it the way that ends the argument — with the &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/listener-service" rel="noopener noreferrer"&gt;listener-service lab&lt;/a&gt;, where a listener restart refuses every service for a minute and one &lt;code&gt;ALTER SYSTEM REGISTER&lt;/code&gt; brings them all back.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://uptimearchitect.com/blog/oracle-ora-12514-listener-service/" rel="noopener noreferrer"&gt;uptimearchitect.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>oracle</category>
      <category>database</category>
      <category>devops</category>
      <category>networking</category>
    </item>
    <item>
      <title>ORA-01652: Unable to Extend Temp Segment. Don't Just Add Space.</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Wed, 28 Oct 2026 14:00:23 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/ora-01652-unable-to-extend-temp-segment-dont-just-add-space-njm</link>
      <guid>https://dev.to/uptimearchitect/ora-01652-unable-to-extend-temp-segment-dont-just-add-space-njm</guid>
      <description>&lt;p&gt;The open-orders report has run every morning for a year. Today it fails:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-01652: unable to extend temp segment by 128 in tablespace TEMP
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fix everyone knows is one line: add a tempfile. It takes a minute, it's online, and the next run fails again, a little later. So someone adds a bigger tempfile, turns on autoextend, and the report fails a third time, after filling the disk.&lt;/p&gt;

&lt;p&gt;ORA-01652 says TEMP ran out. It doesn't say &lt;em&gt;why&lt;/em&gt;. Sometimes TEMP really is too small for legitimate work. Often, one query is asking for far more TEMP than any database could give it, and the error is the only thing standing between you and a wrong answer. This post covers what TEMP is for, how to tell the two cases apart, how to find the query, and why TEMP looks full even when it isn't. It ends with a lab where giving TEMP &lt;strong&gt;ten times the space buys ten times the runtime and the same error&lt;/strong&gt;, while restoring one join predicate makes the same report need &lt;strong&gt;no TEMP at all&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What TEMP is for
&lt;/h2&gt;

&lt;p&gt;Sorts, hash joins, &lt;code&gt;DISTINCT&lt;/code&gt;, &lt;code&gt;GROUP BY&lt;/code&gt;, analytic functions and index builds all need a work area. Oracle gives each one memory from the PGA first. When the work area outgrows its share of memory, the operation &lt;strong&gt;spills&lt;/strong&gt; to the temporary tablespace: it writes partial results to TEMP, then reads them back to finish.&lt;/p&gt;

&lt;p&gt;Spilling is normal; a big nightly batch or an index rebuild is supposed to use TEMP. ORA-01652 is raised when a spilling operation asks for another extent and the temporary tablespace has none left to give, and can't grow. The statement fails and its TEMP is released.&lt;/p&gt;

&lt;p&gt;So the question is never just "is TEMP big enough?" It's "is this operation's demand reasonable?" A sort of 200,000 rows that needs 40 MB of TEMP is reasonable. A sort that needs 8 GB to produce 50,000 rows is a bug.&lt;/p&gt;

&lt;h2&gt;
  
  
  The classic runaway
&lt;/h2&gt;

&lt;p&gt;Here is the report as it shipped:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;DISTINCT&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;amount&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'OPEN'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A line went missing in an edit: &lt;code&gt;AND o.customer_id = c.customer_id&lt;/code&gt;. Without it, there is no join condition, so Oracle pairs &lt;strong&gt;every&lt;/strong&gt; open order with &lt;strong&gt;every&lt;/strong&gt; customer, a Cartesian product. The report suddenly showed each order thousands of times, so someone added &lt;code&gt;DISTINCT&lt;/code&gt; to "remove the duplicates."&lt;/p&gt;

&lt;p&gt;Now the database has to deduplicate the whole product. With 50,000 open orders and 2,000 customers, that's 100 million rows into a &lt;code&gt;HASH UNIQUE&lt;/code&gt;, far beyond any PGA work area. It spills, and keeps spilling until TEMP is gone.&lt;/p&gt;

&lt;p&gt;And here's the part that makes adding space the wrong fix: the rows aren't actually duplicates. Each customer name is different, so every one of the 100 million combinations is distinct. &lt;strong&gt;Given enough TEMP, the report would finish, and return 100 million rows instead of 50,000.&lt;/strong&gt; ORA-01652 is the only thing stopping a wrong report from shipping.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpFkVtLAzEUhP_KEHyosMUL6EMRZbEXBcXWVl9ckezmbDeSTUqS1Urb_24SVnwMmfPNzDk7VhlBbARWK_NdNdx6rMaFBiZvBXt6zoenZ5cX56Or0p5cd5qXiuANaOtJC3hqN3C0bkn7gr1jOLzGIsxNZfxsCM7zoAm_aX6wXDzgfpzh9Wg1eZwvJ7OPl2U-m2TIl3fHARB9F4ky3xXs3kF6hyiFoJYHpiXuTEqRgLWxycaa76iFa0ynosp3Vt8U7BCB8wjcF-yH3AilXKPkvmoyhIy0RdlJJbJEa40lVEZXnbUhMjpH1hVsjzxUyoWA2_CKEDx5al7LPsbAaCU1_TX4M9RmhNuwUHKSa3waqXsf6ZzUa2wsCVmFBWUhksBGcR3dpmmB21QsLOw_Wm-v1Q9a80UuKWouVWept64Udw451saI__c08lkG1pJtuRTh3DsWhtt0eEE175Rnh8MvRMCtJg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpFkVtLAzEUhP_KEHyosMUL6EMRZbEXBcXWVl9ckezmbDeSTUqS1Urb_24SVnwMmfPNzDk7VhlBbARWK_NdNdx6rMaFBiZvBXt6zoenZ5cX56Or0p5cd5qXiuANaOtJC3hqN3C0bkn7gr1jOLzGIsxNZfxsCM7zoAm_aX6wXDzgfpzh9Wg1eZwvJ7OPl2U-m2TIl3fHARB9F4ky3xXs3kF6hyiFoJYHpiXuTEqRgLWxycaa76iFa0ynosp3Vt8U7BCB8wjcF-yH3AilXKPkvmoyhIy0RdlJJbJEa40lVEZXnbUhMjpH1hVsjzxUyoWA2_CKEDx5al7LPsbAaCU1_TX4M9RmhNuwUHKSa3waqXsf6ZzUa2wsCVmFBWUhksBGcR3dpmmB21QsLOw_Wm-v1Q9a80UuKWouVWept64Udw451saI__c08lkG1pJtuRTh3DsWhtt0eEE175Rnh8MvRMCtJg" alt="ORA-01652 means a spilling sort or hash asked TEMP for more space than it had. Before addi" width="554" height="870"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;ORA-01652 means a spilling sort or hash asked TEMP for more space than it had. Before adding space, check whether the demand is reasonable. A legitimate big operation needs more TEMP; a runaway query, like a Cartesian join behind a DISTINCT, will fill any amount of TEMP and needs its SQL fixed.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it
&lt;/h2&gt;

&lt;p&gt;The lab runs Oracle Database Free in a container. It creates a &lt;strong&gt;32 MB&lt;/strong&gt; temporary tablespace that cannot autoextend and assigns it to an application user. That user owns &lt;code&gt;ORDERS&lt;/code&gt; (200,000 rows, 50,000 of them &lt;code&gt;OPEN&lt;/code&gt;) and &lt;code&gt;CUSTOMERS&lt;/code&gt; (2,000 rows). Then it runs the broken report three ways:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TEMP            report                     result            elapsed   TEMP used
32 MB           join predicate missing     ORA-01652          1.2 s       31 MB
320 MB (10x)    join predicate missing     ORA-01652 again   10.7 s      318 MB
32 MB           join predicate restored    50,000 rows        0.1 s        0 MB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ten times the space bought ten times the runtime, then the same error. (Elapsed times depend on the hardware; the lab checks the outcomes, not the clock.) A hundred times the space would buy a hundred times the runtime. With the join predicate back, and &lt;code&gt;DISTINCT&lt;/code&gt; left in place to show that the join alone is the fix, the report returns its 50,000 rows in a tenth of a second without touching TEMP.&lt;/p&gt;

&lt;p&gt;The plan says so before anything runs. &lt;code&gt;EXPLAIN PLAN&lt;/code&gt; on the broken report:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;| Id  | Operation               | Name      | Rows  | Bytes |TempSpc|
---------------------------------------------------------------------
|   3 |    HASH UNIQUE          |           |  200M |  6484M|  8445M|
|   4 |     MERGE JOIN CARTESIAN|           |  200M |  6484M|       |
|   5 |      TABLE ACCESS FULL  | ORDERS    |  100K |  1660K|       |
|   6 |      BUFFER SORT        |           |  2000 | 34000 |       |
|   7 |       TABLE ACCESS FULL | CUSTOMERS |  2000 | 34000 |       |
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;MERGE JOIN CARTESIAN&lt;/code&gt; and a &lt;code&gt;TempSpc&lt;/code&gt; estimate of &lt;strong&gt;8,445M&lt;/strong&gt;, about 8 GB of TEMP, for a report that should return 50,000 rows. (The optimizer estimates twice the real row count because, without a histogram, it assumes half the orders are &lt;code&gt;OPEN&lt;/code&gt;. That doesn't change the conclusion.) &lt;a href="https://uptimearchitect.com/blog/oracle-execution-plans-decoded/" rel="noopener noreferrer"&gt;Reading an execution plan&lt;/a&gt; for &lt;code&gt;TempSpc&lt;/code&gt; and Cartesian joins takes seconds, and it would have caught this before the report ever ran.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't take my word for it — run it.&lt;/strong&gt; The &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/temp-space" rel="noopener noreferrer"&gt;temp-space lab&lt;/a&gt; runs the broken report on 32 MB of TEMP and &lt;strong&gt;asserts&lt;/strong&gt; ORA-01652. It asserts the Cartesian join in the plan, then adds a 288 MB tempfile online and asserts the &lt;strong&gt;same&lt;/strong&gt; error again, after catching the report live in &lt;code&gt;V$TEMPSEG_USAGE&lt;/code&gt;. Finally it drops the extra tempfile and asserts that the corrected report returns 50,000 rows using no TEMP. If any of that doesn't reproduce, the run fails. Proven on every CI push.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  How to find the query
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Read the whole message.&lt;/strong&gt; Recent releases put the culprit in the error itself. The lab's Oracle Database Free (26ai) says &lt;em&gt;grow&lt;/em&gt; rather than &lt;em&gt;extend&lt;/em&gt;, and names the SQL ID and how much TEMP the session held:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ORA-01652: unable to grow temp segment in tablespace TEMP_SMALL by 1MB during operation
with SQL ID : 249gkzhhu3cdk, temp space used by session : 31 (MB)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same line goes to the alert log. On 19c the message names only the tablespace, so you have to catch the statement another way.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Live: &lt;code&gt;V$TEMPSEG_USAGE&lt;/code&gt;.&lt;/strong&gt; While the statement is running, this view shows who holds TEMP and for which SQL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;username&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sql_id_tempseg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;segtype&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;blocks&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;block_size&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;temp_mb&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;tempseg_usage&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="k"&gt;session&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;saddr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;session_addr&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;dba_tablespaces&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tablespace_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tablespace&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;blocks&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Use &lt;code&gt;SQL_ID_TEMPSEG&lt;/code&gt;, not &lt;code&gt;SQL_ID&lt;/code&gt;. In the lab, watching the report from a second session, &lt;code&gt;SQL_ID&lt;/code&gt; pointed at SQL*Plus's own login query (&lt;code&gt;SELECT DECODE(USER, 'XS$NULL', ...&lt;/code&gt;), while &lt;code&gt;SQL_ID_TEMPSEG&lt;/code&gt; was the report that was actually holding the TEMP. The other catch: once the statement fails, its TEMP is released and the row is gone. In the lab, &lt;code&gt;V$TEMPSEG_USAGE&lt;/code&gt; is empty a second after the error.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;After the fact: ASH.&lt;/strong&gt; Active Session History samples every active session once a second, and each sample records &lt;code&gt;TEMP_SPACE_ALLOCATED&lt;/code&gt;. The statement that filled TEMP stands out:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;sql_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;samples&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;MAX&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;temp_space_allocated&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;peak_temp_mb&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;active_session_history&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;sample_time&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;SYSTIMESTAMP&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;INTERVAL&lt;/span&gt; &lt;span class="s1"&gt;'30'&lt;/span&gt; &lt;span class="k"&gt;MINUTE&lt;/span&gt;
&lt;span class="k"&gt;AND&lt;/span&gt;    &lt;span class="n"&gt;temp_space_allocated&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;sql_id&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;peak_temp_mb&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab, ASH caught &lt;code&gt;249gkzhhu3cdk&lt;/code&gt; peaking at over 300 MB of TEMP. For older incidents, &lt;code&gt;DBA_HIST_ACTIVE_SESS_HISTORY&lt;/code&gt; has the same column, and an &lt;a href="https://uptimearchitect.com/blog/how-to-read-an-awr-report/" rel="noopener noreferrer"&gt;AWR report&lt;/a&gt; shows the top SQL for the window. (ASH and AWR are part of the Diagnostics Pack on Enterprise Edition.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Then look at its plan.&lt;/strong&gt; With the SQL ID, &lt;code&gt;DBMS_XPLAN.DISPLAY_CURSOR&lt;/code&gt; (if the cursor is still in the shared pool) or &lt;code&gt;EXPLAIN PLAN&lt;/code&gt; on the statement text tells you whether the demand makes sense. Compare the &lt;code&gt;TempSpc&lt;/code&gt; and row estimates with what the statement should return.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why TEMP looks full when it isn't
&lt;/h2&gt;

&lt;p&gt;Right after the failure, the lab looks at the 32 MB tablespace through four views:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;V$TEMPSEG_USAGE        0 rows        the session that caused it is gone
V$TEMP_SPACE_HEADER    32 MB used, 0 MB free
V$SORT_SEGMENT         0 MB in use
DBA_TEMP_FREE_SPACE    31 MB free of 32 MB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Oracle doesn't hand TEMP extents back to the files when a sort finishes. It keeps them allocated in the sort segment and gives them to the next operation that needs them, which is faster than allocating fresh ones. &lt;code&gt;V$TEMP_SPACE_HEADER&lt;/code&gt; keeps counting those extents as used after the sort ends, so a monitoring check built on it reports TEMP at 100% all day after the first big sort. That isn't a leak, and it isn't a reason to add space. &lt;code&gt;V$SORT_SEGMENT&lt;/code&gt; (&lt;code&gt;USED_BLOCKS&lt;/code&gt;) and &lt;code&gt;DBA_TEMP_FREE_SPACE&lt;/code&gt; show what's actually free. Alert on those.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;If the SQL is wrong, fix the SQL.&lt;/strong&gt; A missing join predicate, a &lt;code&gt;DISTINCT&lt;/code&gt; hiding duplicates, a join on the wrong column, a plan that hash-joins two huge row sources because statistics are stale: these need a code or statistics fix. TEMP is just where the damage shows up.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;If the work is legitimate, add space online.&lt;/strong&gt; Big index builds, a nightly aggregation, and more concurrent users than TEMP was sized for are real reasons TEMP is too small:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- Smallfile temporary tablespace: add a tempfile&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="n"&gt;TABLESPACE&lt;/span&gt; &lt;span class="k"&gt;temp&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="n"&gt;TEMPFILE&lt;/span&gt; &lt;span class="s1"&gt;'/u01/oradata/PROD/temp02.dbf'&lt;/span&gt; &lt;span class="k"&gt;SIZE&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="k"&gt;G&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- Bigfile temporary tablespace (one file only): resize it&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="n"&gt;TEMPFILE&lt;/span&gt; &lt;span class="s1"&gt;'/u01/oradata/PROD/temp01.dbf'&lt;/span&gt; &lt;span class="n"&gt;RESIZE&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="k"&gt;G&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both are online, and so is &lt;code&gt;ALTER TABLESPACE ... DROP TEMPFILE&lt;/code&gt; once nothing is using the file; the lab adds a 288 MB tempfile and drops it again without a restart. A bigfile tablespace holds exactly one file, so &lt;code&gt;ADD TEMPFILE&lt;/code&gt; fails with ORA-32771 there. The lab's 26ai database creates new tablespaces as bigfile unless told otherwise; check &lt;code&gt;DEFAULT_TBS_TYPE&lt;/code&gt; in &lt;code&gt;DATABASE_PROPERTIES&lt;/code&gt; on yours.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Contain the damage.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cap autoextend.&lt;/strong&gt; An autoextending tempfile without &lt;code&gt;MAXSIZE&lt;/code&gt; doesn't fix a runaway query. It lets the query fill the filesystem instead of the tablespace, and then everything on that filesystem fails too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Separate temporary tablespaces.&lt;/strong&gt; Give a reporting or batch user its own temporary tablespace (the lab's user has &lt;code&gt;TEMP_SMALL&lt;/code&gt;). Then one application's runaway query can't starve every other session's sorts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Size from evidence.&lt;/strong&gt; The peak of &lt;code&gt;DBA_HIST_ACTIVE_SESS_HISTORY.TEMP_SPACE_ALLOCATED&lt;/code&gt; summed across sessions, over a few busy weeks, tells you what your legitimate workload actually needs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What teams get wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Adding space first, asking later.&lt;/strong&gt; It's the right fix maybe half the time. Check the statement's plan before you add the second tempfile, not after the third.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Turning on unlimited autoextend.&lt;/strong&gt; It converts a failed report into a full filesystem. Use &lt;code&gt;MAXSIZE&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trusting &lt;code&gt;V$TEMP_SPACE_HEADER&lt;/code&gt;.&lt;/strong&gt; It shows extents the sort segment has cached, not space in use. A TEMP that "is always 100% full" is usually fine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Looking for the culprit too late, or in the wrong column.&lt;/strong&gt; &lt;code&gt;V$TEMPSEG_USAGE&lt;/code&gt; empties the moment the statement fails; read the full error text, or go to ASH. While it's running, trust &lt;code&gt;SQL_ID_TEMPSEG&lt;/code&gt;, not &lt;code&gt;SQL_ID&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hiding duplicates with &lt;code&gt;DISTINCT&lt;/code&gt;.&lt;/strong&gt; Unexpected duplicates are a symptom of a wrong join. &lt;code&gt;DISTINCT&lt;/code&gt; makes the report look right on small data and turns into a TEMP bomb on real data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Confusing it with a permanent tablespace.&lt;/strong&gt; &lt;code&gt;ORA-01652 ... in tablespace USERS&lt;/code&gt; (or any non-temporary tablespace) is a different problem: a &lt;code&gt;CREATE TABLE AS SELECT&lt;/code&gt;, index build or &lt;code&gt;MOVE&lt;/code&gt; ran out of room while building its new segment. The tablespace named in the message tells you which case you have.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What does ORA-01652 unable to extend temp segment mean?
&lt;/h3&gt;

&lt;p&gt;It means an operation that spills to disk, such as a sort, hash join, DISTINCT, GROUP BY or index build, asked the temporary tablespace for another extent and none was available, and the tablespace could not grow. The statement fails and its temporary space is released. The error says that TEMP ran out, not why: either the temporary tablespace is genuinely too small for legitimate work, or one statement is demanding far more TEMP than it should, often because of a missing join predicate or a bad plan.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should I just add a tempfile to fix ORA-01652?
&lt;/h3&gt;

&lt;p&gt;Only after checking the statement that caused it. If the work is legitimate, such as a large index build, a nightly batch or more concurrent sessions, adding a tempfile (or resizing the tempfile of a bigfile tablespace) is the right fix and it is online. If the statement is a runaway, such as a Cartesian join behind a DISTINCT, more space only moves the failure. In the companion lab, giving TEMP ten times the space made the same report fail again after ten times the runtime, while restoring one join predicate made it return 50,000 rows using no TEMP.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I find which query caused ORA-01652?
&lt;/h3&gt;

&lt;p&gt;Recent Oracle releases include the SQL ID and the session's temp usage in the error message and the alert log. On 19c, which names only the tablespace, query V$TEMPSEG_USAGE joined to V$SESSION while the statement is running (use its SQL_ID_TEMPSEG column, which identifies the statement that owns the temp segment; SQL_ID can show a different statement the session parsed), or look in Active Session History afterwards: V$ACTIVE_SESSION_HISTORY and DBA_HIST_ACTIVE_SESS_HISTORY record TEMP_SPACE_ALLOCATED for every sample, so grouping by SQL_ID and taking the maximum shows the statement that filled TEMP. Then check its execution plan for a Cartesian join or a TempSpc estimate far larger than the result should need.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does my temporary tablespace always show 100% used?
&lt;/h3&gt;

&lt;p&gt;Because Oracle keeps temporary extents allocated in the sort segment after a sort finishes and reuses them for the next operation, rather than freeing them back to the tempfiles. V$TEMP_SPACE_HEADER counts those cached extents as used, so it can show TEMP as full long after the sorts that used it have finished. V$SORT_SEGMENT (USED_BLOCKS versus FREE_BLOCKS) and DBA_TEMP_FREE_SPACE show the space that is actually free, and are the right views to monitor.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I add or drop a tempfile without downtime?
&lt;/h3&gt;

&lt;p&gt;Yes. ALTER TABLESPACE temp ADD TEMPFILE ... SIZE ... is online, and so is ALTER TABLESPACE temp DROP TEMPFILE ... once no session is using the file. A bigfile temporary tablespace can hold only one tempfile, so adding a file fails with ORA-32771; resize the existing one with ALTER DATABASE TEMPFILE ... RESIZE instead. DATABASE_PROPERTIES.DEFAULT_TBS_TYPE shows whether new tablespaces default to bigfile or smallfile on your database.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is ORA-01652 on a permanent tablespace the same problem?
&lt;/h3&gt;

&lt;p&gt;No. When the message names a permanent tablespace such as USERS rather than a temporary one, the space was needed to build a new permanent segment: a CREATE TABLE AS SELECT, an index build or rebuild, or an ALTER TABLE MOVE, which create their new segment as temporary extents in the target tablespace until they finish. The fix is space in that tablespace, or a smaller operation, not a bigger TEMP.&lt;/p&gt;

&lt;p&gt;ORA-01652 is easy to make go away for a day and hard to make go away for good, because the obvious fix treats the symptom. Before you add space, read the whole error, find the statement, and look at its plan. If the demand is legitimate, add a tempfile online and cap its growth. If it's a Cartesian join behind a &lt;code&gt;DISTINCT&lt;/code&gt;, no amount of TEMP will help, and enough of it would let a wrong report finish. It belongs with the other errors that freeze or fail work for a reason the message doesn't spell out, like &lt;a href="https://uptimearchitect.com/blog/oracle-ora-01555-snapshot-too-old/" rel="noopener noreferrer"&gt;ORA-01555&lt;/a&gt; and &lt;a href="https://uptimearchitect.com/blog/oracle-ora-00257-archiver-stuck/" rel="noopener noreferrer"&gt;ORA-00257&lt;/a&gt;. Then prove it the way that ends the argument — with the &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/temp-space" rel="noopener noreferrer"&gt;temp-space lab&lt;/a&gt;, where ten times the TEMP buys ten times the runtime and the same error, and one restored join needs none.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://uptimearchitect.com/blog/oracle-ora-01652-temp-space/" rel="noopener noreferrer"&gt;uptimearchitect.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>oracle</category>
      <category>database</category>
      <category>performance</category>
      <category>sql</category>
    </item>
    <item>
      <title>I couldn't even read one row. The session that locked it was gone. So I built a ghost hunter for Oracle</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Wed, 28 Oct 2026 01:52:57 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/i-couldnt-even-read-one-row-the-session-that-locked-it-was-gone-so-i-built-a-ghost-hunter-for-a99</link>
      <guid>https://dev.to/uptimearchitect/i-couldnt-even-read-one-row-the-session-that-locked-it-was-gone-so-i-built-a-ghost-hunter-for-a99</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-week4-2026-10-26"&gt;Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine&lt;/a&gt;&lt;/em&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="n"&gt;amt&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;gh_ledger&lt;/span&gt; &lt;span class="k"&gt;where&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="n"&gt;ORA&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;01591&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;lock&lt;/span&gt; &lt;span class="n"&gt;held&lt;/span&gt; &lt;span class="k"&gt;by&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;doubt&lt;/span&gt; &lt;span class="n"&gt;distributed&lt;/span&gt; &lt;span class="n"&gt;transaction&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;71483&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Not an update. A plain &lt;code&gt;select&lt;/code&gt;. Oracle wouldn't even let me &lt;em&gt;look&lt;/em&gt; at the row.&lt;/p&gt;

&lt;p&gt;Two minutes earlier, a transfer of 40 between two of my lab databases had crashed halfway through its commit. (Oracle commits across two databases in two steps: both sides promise, then both commit. This one died between the steps.) I crashed it on purpose, with a test hook Oracle ships for exactly this (&lt;code&gt;COMMIT COMMENT 'ORA-2PC-CRASH-TEST-7'&lt;/code&gt;), because I wanted to see what a DBA is left with afterwards.&lt;/p&gt;

&lt;p&gt;One side committed. The other had promised to commit, and then the session that made the promise was gone. Until somebody decides how that story ends, the row stays locked for everyone, and no session holds the lock, so there's nothing to kill. Oracle's recovery process would normally settle it once both databases can talk again. In my lab I switched that off, so the row waits. Nobody is coming back for it.&lt;/p&gt;

&lt;p&gt;DBAs have a quiet name for things like this. Ghosts. For this week's theme I took the word literally: things that are still in the database after whatever owned them is gone.&lt;/p&gt;

&lt;p&gt;Día de los Muertos isn't about getting rid of the dead. It's about saying who they were. So nothing here is removed quietly: every ghost gets its name, its date and an epitaph written into the report, and the report stays on disk after the ghost is laid to rest.&lt;/p&gt;
&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Ghost Hunt&lt;/strong&gt; finds what's still haunting your Oracle databases, and lays it to rest only when you say so.&lt;/p&gt;

&lt;p&gt;Six kinds so far:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🕯️ a session whose user went home with a transaction open, while someone else waits on it;&lt;/li&gt;
&lt;li&gt;⛓️ an in-doubt distributed transaction, like the one above;&lt;/li&gt;
&lt;li&gt;📜 a database file on disk that the database stopped using;&lt;/li&gt;
&lt;li&gt;⚰️ a dropped table still holding its space in the recycle bin;&lt;/li&gt;
&lt;li&gt;🪦 an object left invalid by something it depended on;&lt;/li&gt;
&lt;li&gt;🗝️ a contractor's locked account whose nightly job is still enabled.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;SQL finds every ghost. &lt;strong&gt;Gemma 4&lt;/strong&gt;, running locally on Ollama, writes the field notes: two lines on top, and an epitaph for each ghost. A &lt;strong&gt;Temporal&lt;/strong&gt; workflow then waits, for up to twelve hours, for a person to approve each exorcism. Only then does it touch anything, and afterwards it hunts again to prove the ghost is gone.&lt;/p&gt;

&lt;p&gt;This is the in-doubt ghost from the opening, as the report shows it:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;### 2. ⛓️ Distributed transaction 5.30.71483 is prepared and nobody is coming back for it
lab-se2 · ORCLPDB1. Since 2026-10-27 23:48.

Here lies 5.30.71483, a prepared distributed transaction from 23:48.
It waits for a master that will never return.

The other side: lab-26ai/FREEPDB1 committed. Decision: commit force: the other side committed.

Exorcism, if you approve it:
commit force '5.30.71483';
begin dbms_transaction.purge_lost_db_entry('5.30.71483'); commit; end;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The epitaph is Gemma's. Everything else, including &lt;strong&gt;the decision&lt;/strong&gt;, is code. Ghost Hunt reads both databases, matches the two halves of the transaction by its global id, and sees that the other side committed. So the only correct ending is to force this side to commit too. If no side has an outcome written down, it refuses to guess and tells you what to look up.&lt;/p&gt;
&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/TWNPsXcCrMk" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;The video is one real run, start to finish:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;the row nobody can read;&lt;/li&gt;
&lt;li&gt;the hunt and its report;&lt;/li&gt;
&lt;li&gt;I &lt;code&gt;kill -9&lt;/code&gt; the worker while the hunt waits for me;&lt;/li&gt;
&lt;li&gt;I approve the exorcism anyway, while no worker is running;&lt;/li&gt;
&lt;li&gt;I start the worker again. Temporal picks the workflow up where it was and lays three ghosts to rest: the session blocking someone, the prepared side of the transfer, and the committed side's leftover entry. Then it hunts again.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Then the row reads &lt;code&gt;60&lt;/code&gt;, which is 100 minus the 40 that left. The other database reads &lt;code&gt;140&lt;/code&gt;. The money that was in limbo is accounted for on both sides.&lt;/p&gt;
&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/pyaroslav" rel="noopener noreferrer"&gt;
        pyaroslav
      &lt;/a&gt; / &lt;a href="https://github.com/pyaroslav/ghost-hunt" rel="noopener noreferrer"&gt;
        ghost-hunt
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      Finds the ghosts in your Oracle databases and lays them to rest only when you approve. Gemma + Temporal + Sentry agent tracing.
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;Ghost Hunt&lt;/h1&gt;
&lt;/div&gt;

&lt;p&gt;&lt;a href="https://github.com/pyaroslav/ghost-hunt/actions/workflows/tests.yml" rel="noopener noreferrer"&gt;&lt;img src="https://github.com/pyaroslav/ghost-hunt/actions/workflows/tests.yml/badge.svg" alt="tests"&gt;&lt;/a&gt;
&lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/f8df3091bbe1149f398a5369b2c39e896766f9f6efba3477c63e9b4aa940ef14/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4d49542d677265656e"&gt;&lt;img src="https://camo.githubusercontent.com/f8df3091bbe1149f398a5369b2c39e896766f9f6efba3477c63e9b4aa940ef14/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4d49542d677265656e" alt="license: MIT"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Ghost Hunt finds what's still haunting your Oracle databases, and lays it to rest only when you say so.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;A ghost is something still in the database after whatever owned it is gone: a session whose user went home
with a transaction open, a distributed transaction that crashed halfway and now holds a row nobody can even
read, a control file the database stopped using two weeks ago, a dropped table still holding its space, a
contractor's locked account whose nightly job still runs.&lt;/p&gt;
&lt;p&gt;SQL finds every ghost. Gemma, running locally, writes the field notes. A Temporal workflow waits, for hours if
need be, for a person to approve each exorcism, and then hunts again to prove the ghost is gone.&lt;/p&gt;
&lt;div class="snippet-clipboard-content notranslate position-relative overflow-auto"&gt;
&lt;pre class="notranslate"&gt;&lt;code&gt;### 2. ⛓️ Distributed transaction 10.9.199680 is prepared and nobody is coming back for it
`lab-se2` · `ORCLPDB1`. Since 2026-10-27 22:37
_Here lies 10.9.199680, a prepared distributed&lt;/code&gt;&lt;/pre&gt;…&lt;/div&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/pyaroslav/ghost-hunt" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;h2&gt;
  
  
  The Spirit of It: the one ghost I didn't plant
&lt;/h2&gt;

&lt;p&gt;To have something to hunt, I planted ghosts in two lab databases running in Docker: Oracle 26ai Free and 12.2 Standard Edition 2. The first time I ran Ghost Hunt, it found one I hadn't planted. Here it is in the demo run's report:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;### 5. 📜 control02.ctl (18 MB) is on disk but the database stopped using it
lab-se2 · CDB$ROOT. Since 2026-10-13 16:08.

Here lies control02.ctl, an 18 MB file from 2026-10-13.
It rests on the disk, though the database no longer needs it.

The hunter won't touch this one. Move it aside, keep it a week, then delete it.
Never delete a file the database still lists.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I went back through that database's alert log. On October 13 at 16:07, while the container was still creating the database, its setup script ran this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;SYSTEM&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;control_files&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'/opt/oracle/oradata/ORCLCDB/control01.ctl'&lt;/span&gt; &lt;span class="k"&gt;SCOPE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;SPFILE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From that restart on, the database had &lt;strong&gt;one&lt;/strong&gt; control file. The second copy, the one that exists in case the first is lost, was last written a minute later. It has sat on the disk for two weeks, looking exactly like a live control file: a spare kept faithfully for a database that stopped counting on it. It's the same database whose crash started my &lt;a href="https://dev.to/uptimearchitect/my-database-died-on-tuesday-i-found-out-on-sunday-so-now-a-local-model-reads-my-oracle-logs-while-4oma"&gt;Week 3 entry&lt;/a&gt;, and I'd looked at it every day since without noticing.&lt;/p&gt;

&lt;p&gt;Ghost Hunt doesn't touch files. Some ghosts you bury yourself.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. SQL finds the ghosts, with an account that can only look
&lt;/h3&gt;

&lt;p&gt;The hunter connects as &lt;code&gt;c##ghost&lt;/code&gt;, a common user with &lt;code&gt;CREATE SESSION&lt;/code&gt; and &lt;code&gt;SELECT ANY DICTIONARY&lt;/code&gt;, and nothing else. There's one query per kind of ghost, run in each PDB (each database inside the container database). For example, a session that is idle, has an open transaction, and has others waiting on it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;serial&lt;/span&gt;&lt;span class="o"&gt;#&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;username&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last_call_et&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;used_urec&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="k"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="k"&gt;session&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="k"&gt;where&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;blocking_session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;blocked&lt;/span&gt;
  &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="k"&gt;session&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;join&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;transaction&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;on&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ses_addr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;saddr&lt;/span&gt;
 &lt;span class="k"&gt;where&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'USER'&lt;/span&gt; &lt;span class="k"&gt;and&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'INACTIVE'&lt;/span&gt; &lt;span class="k"&gt;and&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last_call_et&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;idle&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The queries never select &lt;code&gt;machine&lt;/code&gt;, &lt;code&gt;osuser&lt;/code&gt; or SQL text (on my lab, &lt;code&gt;machine&lt;/code&gt; is my real hostname). Files are compared by listing the data directory (through &lt;code&gt;docker exec&lt;/code&gt;) against &lt;code&gt;v$datafile&lt;/code&gt;, &lt;code&gt;v$tempfile&lt;/code&gt;, &lt;code&gt;v$logfile&lt;/code&gt; and &lt;code&gt;v$controlfile&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Code decides; the model doesn't
&lt;/h3&gt;

&lt;p&gt;Ghosts are numbered by harm. Whatever is blocking someone right now comes first: the session others wait on, the row nobody can read. The in-doubt decision follows only an outcome that's already written down somewhere:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;if a side committed, force the prepared side to commit;&lt;/li&gt;
&lt;li&gt;if a side was already rolled back, force a rollback;&lt;/li&gt;
&lt;li&gt;otherwise, don't decide at all. The database that decides (the commit point) never sits in &lt;code&gt;prepared&lt;/code&gt;. So if every side the hunter can see is prepared, the decider isn't in the hunt, and any choice would be a guess.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The settled side's entry is the only written record of how the transaction ended, so it's purged only &lt;em&gt;after&lt;/em&gt; its prepared partner has been forced, in the same run. Approve the committed side (#3 in the demo) without the prepared one (#2) and the hunter refuses: "approve #2 with it, so its partner is forced first."&lt;/p&gt;

&lt;p&gt;Only two kinds of ghost can be exorcised by the hunter, because only for those can the right move be read off the database: a session that &lt;strong&gt;someone is waiting on&lt;/strong&gt;, and an in-doubt transaction with a written outcome. A session that's idle with an open transaction but blocks nobody gets a note for its owner, not a kill. Everything else gets the SQL and a sentence, for a person to run.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Hours later, is it still the same ghost?
&lt;/h3&gt;

&lt;p&gt;The approval can come six hours after the hunt. By then the user who went home may be back, may have committed, and may be halfway through something new in the very same session. So right before anything runs, the exorcist checks the ghost again:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;for a session: the same sid and serial# (the session's number and which incarnation of it), the same &lt;strong&gt;transaction id&lt;/strong&gt;, still idle, still blocking someone;&lt;/li&gt;
&lt;li&gt;for an in-doubt transaction: the same entry, still in the same state.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If anything changed, nothing is run, and the report says "not touched: it changed since the hunt, so the approval no longer applies." Afterwards the databases are hunted again. If one can't be reached for that check, the report says the exorcism ran but couldn't be verified. It doesn't claim success it didn't see.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Gemma writes the words, and every line is checked
&lt;/h3&gt;

&lt;p&gt;Gemma gets each ghost's name, a one-sentence description, the date, and the decision. It never sees SQL. It returns JSON: two summary lines and one epitaph per ghost. The prompt asks for "an old caretaker who has buried many databases: dry, kind, a little spooky."&lt;/p&gt;

&lt;p&gt;Every line is checked before it goes into the report:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the epitaph must name its own ghost;&lt;/li&gt;
&lt;li&gt;every number in it must appear in that ghost's facts;&lt;/li&gt;
&lt;li&gt;every identifier (anything with an underscore or a dot, like &lt;code&gt;GH_AUDIT_2019&lt;/code&gt; or &lt;code&gt;LAB.GH_RATES_TODAY&lt;/code&gt;) must appear in that ghost's facts;&lt;/li&gt;
&lt;li&gt;the summary must name both databases, and its second line must name the ghost the &lt;em&gt;code&lt;/em&gt; ranked first.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A line that fails gets one retry. Then a plain sentence from the facts replaces it, and the report says so. This is the pattern from my Night Shift entry, applied to every line instead of two.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Temporal waits, and survives
&lt;/h3&gt;

&lt;p&gt;The workflow is &lt;code&gt;hunt (one per database) → examine → write_story → deliver → wait → exorcise → hunt again → deliver&lt;/code&gt;. The interesting step is the wait:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;workflow&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;wait_condition&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;approval&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                              &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hours&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;wait_hours&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An approval is a Temporal signal, so the server keeps it even while no worker is running. That's exactly what the demo does: the worker is killed during the wait, the approval arrives anyway, and the restarted worker carries on from the wait. If nobody answers in twelve hours, the workflow ends and touches nothing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcv6cozobjwhx81pmf6x9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcv6cozobjwhx81pmf6x9.png" alt="Temporal timeline: two hunts, Gemma, the approval timer, the approve signal, three exorcisms, two hunts to check" width="798" height="180"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The demo run's whole event history is in the repo, &lt;a href="https://github.com/pyaroslav/ghost-hunt/blob/main/results/demo-history.json" rel="noopener noreferrer"&gt;&lt;code&gt;results/demo-history.json&lt;/code&gt;&lt;/a&gt;. The hunt and the report ran on worker &lt;code&gt;3143315&lt;/code&gt;. The three exorcisms and the second hunt ran on worker &lt;code&gt;3144782&lt;/code&gt;, the one I started after the kill.&lt;/p&gt;

&lt;p&gt;The exorcist is a second account, &lt;code&gt;c##exorcist&lt;/code&gt;, with just enough to kill a session and force or purge a transaction (the exact grants are in the README). That includes &lt;code&gt;ALTER SYSTEM&lt;/code&gt;, which is not a small privilege, and exactly why the exorcist isn't the hunter. I found one of the grants the hard way: on 26ai the purge fails with &lt;code&gt;ORA-41900: missing DELETE privilege on "SYS"."PENDING_TRANS$"&lt;/code&gt;, which at least says exactly what it wants.&lt;/p&gt;

&lt;p&gt;The tests run the whole workflow on Temporal's time-skipping test server, in CI, without a database:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;only approved ghosts are touched;&lt;/li&gt;
&lt;li&gt;a ghost that changed since the hunt is left alone;&lt;/li&gt;
&lt;li&gt;twelve hours with no answer touch nothing. That test passes in under a second.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Four more cover the purge order, an unreachable database, a check that can't run, and the model being down.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Sentry sees the shape, not the ghosts
&lt;/h3&gt;

&lt;p&gt;Every activity that touches a database or the model is a Sentry &lt;code&gt;gen_ai.invoke_agent&lt;/code&gt; transaction, with &lt;code&gt;execute_tool&lt;/code&gt; spans for the hunts and exorcisms and a &lt;code&gt;chat&lt;/code&gt; span for Gemma. Each one carries a trace id derived from the workflow id, so a hunt that waits, loses its worker and resumes is still &lt;strong&gt;one trace&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Feblbm132o31q9eif4r69.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Feblbm132o31q9eif4r69.png" alt="The hunt in Sentry: one trace, eight activities, across the worker restart" width="800" height="162"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;What leaves the machine: the model name, token counts, timings, how many ghosts of each kind, and errors cut down to their &lt;code&gt;ORA-&lt;/code&gt; code. No SQL, no object or user names, no prompt, no answer. (Sentry's servers see the IP address an event comes from, like any web server. Left alone, Sentry turns it into a city. So Ghost Hunt sends &lt;code&gt;0.0.0.0&lt;/code&gt; as the user's address, and the project has "Prevent Storing of IP Addresses" on.) The span says it plainly:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0f6bpg4enebheskuyodq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0f6bpg4enebheskuyodq.png" alt="Sentry span for the Gemma call: gemma4:26b, 967 in + 425 out tokens, &amp;quot;No input for this span&amp;quot;" width="800" height="301"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I didn't get that right the first time. The first trace I opened had a &lt;code&gt;subprocess&lt;/code&gt; span I never wrote. Sentry instruments the standard library by default, and its description was the full command line, &lt;code&gt;docker exec w2-se2 find /opt/oracle/oradata …&lt;/code&gt;. Harmless in a lab, and exactly the kind of thing I'd promised not to send. Now that integration is off, a filter drops any span that isn't &lt;code&gt;gen_ai.*&lt;/code&gt;, error stack frames lose their file paths (a home directory names its user), and tests check all three.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which Gemma? I read every line
&lt;/h2&gt;

&lt;p&gt;The checks catch an epitaph that names the wrong ghost or invents a number. They can't catch a sentence that uses the right names and numbers and still says something untrue.&lt;/p&gt;

&lt;p&gt;So I took one real hunt (7 ghosts) and asked two sizes of Gemma 4 for its report 20 times each. Then I read all 280 epitaphs and 40 summaries against the facts:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Same hunt, 20 runs each&lt;/th&gt;
&lt;th&gt;gemma4:e4b&lt;/th&gt;
&lt;th&gt;gemma4:26b&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Epitaphs and summaries passing the automatic checks&lt;/td&gt;
&lt;td&gt;160 / 160&lt;/td&gt;
&lt;td&gt;160 / 160&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Epitaphs with a false claim&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;24 / 140&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1 / 140&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Summaries (two lines each) with a false claim&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;5 / 20&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0 / 20&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Median seconds per report (one call)&lt;/td&gt;
&lt;td&gt;1.05&lt;/td&gt;
&lt;td&gt;1.32&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Both models passed every check on every run. Measured only by the checks, they'd be a tie. Read by a DBA, e4b's mistakes are the kind that sound right:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Here lies GH_AUDIT_2019, a table dropped on 2026-10-27. It still holds 7.0 MB &lt;strong&gt;of memory&lt;/strong&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It's disk space.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Here lies 4.4.159581, who finished a distributed transaction on 2026-10-27 22:37. It rests, &lt;strong&gt;having already been purged&lt;/strong&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It hadn't been. Purging it &lt;em&gt;is&lt;/em&gt; the exorcism, and nobody had approved anything.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Here lies session 217, which began its work at 2026-10-27 22:37. It &lt;strong&gt;still waits for another session&lt;/strong&gt; to finish its task.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Backwards. It's the one being waited on.&lt;/p&gt;

&lt;p&gt;The 26B model made one mistake in 140 (the same "of memory"). It costs about a quarter of a second per report once loaded, so it's the default. Every line from all 40 runs is in the repo (&lt;a href="https://github.com/pyaroslav/ghost-hunt/blob/main/results/eval.json" rel="noopener noreferrer"&gt;&lt;code&gt;results/eval.json&lt;/code&gt;&lt;/a&gt;), with my count and criteria in &lt;a href="https://github.com/pyaroslav/ghost-hunt/blob/main/results/claims.md" rel="noopener noreferrer"&gt;&lt;code&gt;results/claims.md&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;What I take from it: a validator tells you the model stayed inside the facts. It doesn't tell you the model understood them. That's also why, in Ghost Hunt, a wrong epitaph can make a report wrong in tone, but never makes the hunter touch the wrong thing. The ghosts, their order, the decisions and the SQL never come from the model.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;p&gt;Everything Ghost Hunt reads is what a DBA can't paste into a hosted chatbot: session lists, transaction ids, user names, file paths. Here, Gemma runs on the same machine as the databases. Temporal is one binary with a SQLite file. The only thing that can leave is the Sentry metadata above, and that's optional. A hunt has no API bill.&lt;/p&gt;

&lt;p&gt;Open weights mattered in a second way, too. The model choice above came from running the same input 40 times and reading every answer. With an open model on my own GPU, that's a few minutes and no bill. I can pin the exact model and rerun the comparison whenever a new Gemma comes out. Anyone with Ollama can repeat it from &lt;code&gt;results/ghosts.json&lt;/code&gt;, no Oracle needed.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it doesn't do (yet)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;It hunts two lab databases. The queries are standard dictionary views, and I'd expect them to work on any 12.2+ CDB, but I've only run them on these two.&lt;/li&gt;
&lt;li&gt;The in-doubt ghost needed two lab settings to stay put:

&lt;ul&gt;
&lt;li&gt;distributed recovery is disabled on both databases, or Oracle's recovery process (RECO) quietly resolves it within seconds;&lt;/li&gt;
&lt;li&gt;the database link goes from 12.2 to 26ai, because the other direction fails with &lt;code&gt;ORA-28267&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;The file check needs to list the data directory, through &lt;code&gt;docker exec&lt;/code&gt; here.&lt;/li&gt;
&lt;li&gt;The false-claim count is one fixture, read by one person: me.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The row
&lt;/h2&gt;

&lt;p&gt;It reads 60 now. The 40 that left arrived on the other side, the transaction that carried it has an epitaph in a report on my disk, and the control file is still waiting for me to bury it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Best Use of Gemma.&lt;/strong&gt; Gemma 4 26B, local on Ollama, writes the summary and every epitaph inside line-by-line checks, and was chosen over e4b by reading 280 epitaphs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Best Use of Temporal.&lt;/strong&gt; The exorcism waits on a signal for up to twelve hours, survives its worker being killed, and touches only approved ghosts, re-checked, in a safe order.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Best Use of Sentry Agent Tracing.&lt;/strong&gt; One metadata-only &lt;code&gt;gen_ai&lt;/code&gt; trace per hunt, across a worker restart, including the leak I found in my own trace and the test that keeps it out.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Happy Halloween, and to everyone keeping an ofrenda this weekend: may your databases remember only what they should.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
      <category>oracle</category>
      <category>opensource</category>
    </item>
    <item>
      <title>My database died on Tuesday. I found out on Sunday. So now a local model reads my Oracle logs while I sleep</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Mon, 26 Oct 2026 00:09:57 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/my-database-died-on-tuesday-i-found-out-on-sunday-so-now-a-local-model-reads-my-oracle-logs-while-4oma</link>
      <guid>https://dev.to/uptimearchitect/my-database-died-on-tuesday-i-found-out-on-sunday-so-now-a-local-model-reads-my-oracle-logs-while-4oma</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-week3-2026-10-19"&gt;Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;On Tuesday afternoon at 18:39 UTC, a background process in one of my lab databases died:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-10-20T18:39:10 Instance Critical Process (pid: 7, ospid: 3437, MMAN) died unexpectedly
                    PMON (ospid: 3425): terminating the instance due to error 822
2026-10-20T18:39:13 Instance terminated by PMON, pid = 3425
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;MMAN manages the database's memory. When it dies, Oracle takes the whole instance down with it, and that's what happened. Then nothing happened for five days. No alert, no email. The alert log doesn't write anything once the database is gone. I found out on Sunday evening, by accident, while looking for something else.&lt;/p&gt;

&lt;p&gt;Every DBA knows the cure, and nobody actually does it: read the alert log every morning. It's three hundred lines a night of log switches, resizes and scheduler windows. Somewhere in there might be the one line that matters. Some mornings there isn't one, and the morning there is, you've stopped reading.&lt;/p&gt;

&lt;p&gt;So I built a sloth to do it for me.&lt;/p&gt;
&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Night Shift&lt;/strong&gt; reads your Oracle alert and listener logs overnight and leaves you &lt;strong&gt;two lines&lt;/strong&gt; in the morning. Everything else stays in a list underneath, in case you want it:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gh"&gt;# Night Shift&lt;/span&gt;
&lt;span class="gt"&gt;
&amp;gt; lab-26ai had an out of space issue; lab-se2 is down. lab-26ai also saw 37 checkpoint issues and 1 deadlock.&lt;/span&gt;
&lt;span class="gt"&gt;&amp;gt; Start lab-se2 immediately, then check the alert log entries from before it stopped.&lt;/span&gt;

&lt;span class="gu"&gt;## lab-26ai: up&lt;/span&gt;
2026-10-24 23:01 to 2026-10-25 23:01 UTC · 303 log lines, 263 routine
&lt;span class="p"&gt;
-&lt;/span&gt; 🔴 Out of space in a tablespace (ORA-01653), 2026-10-25 22:47 UTC. Add or extend a datafile, or find what grew.
  FREEPDB1(3):ORA-01653: unable to increase tablespace NS_TINY [FREEPDB1] by 1MB during insert or update on table LAB.NS_FILL
&lt;span class="p"&gt;-&lt;/span&gt; 🟡 Checkpoint not complete, 37× 2026-10-25 01:00 UTC → 2026-10-25 21:06 UTC. Redo logs are too small or too few for the load.
&lt;span class="p"&gt;-&lt;/span&gt; 🟡 Deadlocks (ORA-00060), 2026-10-25 22:47 UTC. Read the trace file: it names both sessions and the SQL.

&lt;span class="gu"&gt;## lab-se2: DOWN&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; 🔴 Database is not running now. no pmon process for ORCLCDB; last alert log entry 2026-10-20 18:39 UTC

Night Shift · Temporal run night-20261025-230121 · 1 s · summary by gemma4:e4b
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;That's a real run from tonight, trimmed a little, against two lab databases on my desktop: Oracle 26ai Free and 12.2 Standard Edition 2. To give it something to find, I filled a 16 MB tablespace and caused a deadlock on the 26ai one. The SE2 one was still dead from Tuesday (more on that at the end).&lt;/p&gt;

&lt;p&gt;It's set up once and then left alone. A Temporal Schedule runs it at 6 AM. The brief lands in a Markdown file and a desktop notification. The model runs locally in Ollama. Night Shift never connects to the database: it reads the log files and checks the process list, and nothing leaves the machine.&lt;/p&gt;
&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/wHePK3i-vds" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;
&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/pyaroslav" rel="noopener noreferrer"&gt;
        pyaroslav
      &lt;/a&gt; / &lt;a href="https://github.com/pyaroslav/night-shift" rel="noopener noreferrer"&gt;
        night-shift
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      Reads your Oracle alert and listener logs overnight and leaves a two-line brief for the morning. Gemma 4 on Ollama + Temporal, all local.
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;Night Shift&lt;/h1&gt;
&lt;/div&gt;

&lt;p&gt;&lt;a href="https://github.com/pyaroslav/night-shift/actions/workflows/tests.yml" rel="noopener noreferrer"&gt;&lt;img src="https://github.com/pyaroslav/night-shift/actions/workflows/tests.yml/badge.svg" alt="tests"&gt;&lt;/a&gt;
&lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/f8df3091bbe1149f398a5369b2c39e896766f9f6efba3477c63e9b4aa940ef14/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4d49542d677265656e"&gt;&lt;img src="https://camo.githubusercontent.com/f8df3091bbe1149f398a5369b2c39e896766f9f6efba3477c63e9b4aa940ef14/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4d49542d677265656e" alt="license: MIT"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Night Shift reads your Oracle alert and listener logs overnight and leaves a two-line brief for the morning.&lt;/strong&gt; It runs on your own machine: Gemma 4 on Ollama writes the brief, and a Temporal workflow makes sure it arrives.&lt;/p&gt;
&lt;p&gt;I built it because my lab database died on a Tuesday afternoon and I found out on Sunday night.&lt;/p&gt;
&lt;div class="snippet-clipboard-content notranslate position-relative overflow-auto"&gt;
&lt;pre class="notranslate"&gt;&lt;code&gt;# Night Shift
&amp;gt; lab-26ai has an out of space tablespace; lab-se2 is down; lab-offsite logs are unreadable.
&amp;gt; Start lab-se2 and review its logs from before it stopped.

## lab-26ai: up
2026-10-24 22:56 to 2026-10-25 22:56 UTC · 302 log lines, 262 routine

- 🔴 Out of space in a tablespace (ORA-01653), 2026-10-25 22:47 UTC. Add or extend a datafile, or find what grew.
- 🟡 Checkpoint not complete, 37× 2026-10-25 01:00 UTC → 2026-10-25 21:06 UTC. Redo logs are too small or too few for the load.
-&lt;/code&gt;&lt;/pre&gt;…&lt;/div&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/pyaroslav/night-shift" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;p&gt;The obvious version of this project is one line of code: paste last night's alert log into a model and ask for a summary. I built that version too, as the baseline, and it's the reason the real one looks the way it does.&lt;/p&gt;

&lt;h3&gt;
  
  
  Rules sort the night, the model writes two lines
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://github.com/pyaroslav/night-shift/blob/main/nightshift/logs.py" rel="noopener noreferrer"&gt;&lt;code&gt;logs.py&lt;/code&gt;&lt;/a&gt; splits the alert log into timestamped entries. It handles 12c+ ISO timestamps and the old 11g &lt;code&gt;Tue Oct 20 18:39:10 2026&lt;/code&gt; style, plus the listener log. Each entry goes through about twenty plain patterns, written so a DBA can read the file and agree with every line:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Critical:&lt;/strong&gt; instance terminations, dead background processes, ORA-600/7445, block corruption, a stuck archiver, a full tablespace, missing online redo logs, ORA-4031, running out of processes, the listener losing the service.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look today:&lt;/strong&gt; deadlocks, checkpoint not complete, ORA-1555, failed scheduler jobs, connection timeouts, failed logins, swapping.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Routine:&lt;/strong&gt; log switches, resizes, scheduler windows, service updates. These are counted, never reported.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Repeats are folded together. A job that fails every ten minutes is one finding with a count. One crash that writes four alert log entries ("terminating", two state dumps, "terminated") counts as one incident, not four. An ORA- code that matches no rule still shows up as "Other Oracle error", so a problem I didn't think of can't quietly disappear. And if you've decided to live with something, like the Statistics Advisor job that fails every ten minutes on a fresh 12.2 CDB, there's a &lt;code&gt;mute&lt;/code&gt; list.&lt;/p&gt;

&lt;p&gt;A deadlock line in the alert log only says "see the trace file", and nobody opens trace files at 7 AM. So for ORA-00060, Night Shift opens the trace itself. It reads the deadlock graph to tell the kind of deadlock apart: TX locks held and wanted in exclusive mode mean two sessions updated the same rows in opposite order; a TM lock usually means a foreign key without an index; share-mode waits point at duplicate-key inserts or bitmap indexes. Then it pulls each session's user and SQL. The client machine and program name stay out of the brief:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="p"&gt;-&lt;/span&gt; 🟡 Deadlocks (ORA-00060), 2026-10-25 22:47 UTC. Row lock deadlock: two sessions updated the same rows
  in opposite order. Make the application touch rows in one order (e.g. by key), or commit sooner.
&lt;span class="p"&gt;  -&lt;/span&gt; session 230 (LAB): update ns_dl set v = 2 where id = 1
&lt;span class="p"&gt;  -&lt;/span&gt; session 42 (LAB): update ns_dl set v = 2 where id = 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two checks look at what &lt;em&gt;isn't&lt;/em&gt; in the log. Is pmon running? And did the alert log write anything at all? A database that crashed on Tuesday writes nothing on Wednesday, so a silent night is a critical finding, not a quiet one. My first run of the pmon check told me lab-26ai was down. It wasn't: since 23ai, the process is called &lt;code&gt;db_pmon_FREE&lt;/code&gt;, not &lt;code&gt;ora_pmon_FREE&lt;/code&gt;. Both names are checked now.&lt;/p&gt;

&lt;p&gt;Then &lt;strong&gt;Gemma 4 (e4b)&lt;/strong&gt; gets the night as a small JSON digest (about 500 characters per database), not the log, and writes exactly two lines. Its answer is checked before anyone sees it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;exactly two lines;&lt;/li&gt;
&lt;li&gt;every number in them has to appear in the digest;&lt;/li&gt;
&lt;li&gt;every database with a critical finding has to be named.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It gets three tries, and then a plain template writes the lines instead. The list under the two lines never comes from the model, so even a bad summary can't hide an outage.&lt;/p&gt;

&lt;h3&gt;
  
  
  Temporal makes sure it arrives
&lt;/h3&gt;

&lt;p&gt;A morning brief that silently doesn't arrive is worse than no tool at all, because by then you've stopped checking yourself. That's why it's a &lt;a href="https://temporal.io" rel="noopener noreferrer"&gt;Temporal&lt;/a&gt; workflow and not a cron job:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Each database is read in its own activity&lt;/strong&gt;, with retries at 15 s, 30 s and 60 s. A host that stays unreachable doesn't crash the run. It becomes a finding: "Couldn't read the logs. The night is unknown."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;If Ollama is down all night&lt;/strong&gt;, the summary activity fails three times and the template writes the brief instead.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;If the worker dies halfway&lt;/strong&gt;, the run picks up where it stopped as soon as a worker comes back. Steps that already finished aren't repeated.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A Temporal Schedule&lt;/strong&gt; starts it at 06:00 America/Chicago, with the overlap policy set to skip.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All of it runs locally. &lt;code&gt;night-shift server&lt;/code&gt; starts Temporal's single-binary dev server with a SQLite file, so the history survives a reboot, and two systemd user units in the repo bring the server and worker back up afterwards. If the machine is asleep at 6 AM, the Schedule's catch-up window starts the missed run when it wakes up.&lt;/p&gt;

&lt;p&gt;Two of those promises (an unreachable host, and no model all night) have a test on Temporal's time-skipping test server. It runs in about a second in CI. For the rest, I tested by hand: I added a third "database" whose container doesn't exist. I started a run, killed the worker with &lt;code&gt;kill -9&lt;/code&gt; twenty seconds in, and started it again thirty seconds later:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1ppfesrhibvahcksisxa.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1ppfesrhibvahcksisxa.png" alt="Temporal timeline: two databases read in seconds, the missing one retried four times across the worker restart, then the summary and delivery"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The two real databases were read in under a second. The missing one was retried four times over 1 m 45 s, across the restart. Then Gemma wrote the summary and the brief was delivered, with the missing host as the third critical finding. The first time this happened wasn't planned. My first worker crashed on startup: Temporal's workflow sandbox doesn't allow &lt;code&gt;Path.resolve()&lt;/code&gt; at import time. The run I'd started just waited in the queue for ten minutes. When I fixed the worker, the run finished as if nothing had happened.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does the brief say what matters?
&lt;/h3&gt;

&lt;p&gt;I replayed &lt;strong&gt;22 nights&lt;/strong&gt; from the two lab databases' logs (Oct 13 to tonight), each one the 24 hours up to 6 AM Central. I compared Night Shift with the one-line version: the same night's raw alert log, the same model, the same two-line instructions. The events that &lt;em&gt;must&lt;/em&gt; be in the two lines were found with plain grep on the raw log, independent of my rules: an instance died, a tablespace filled, a deadlock, a missing online redo log, or an alert log silent all night. That gave 8 events across 5 nights.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;gemma4:e4b, 22 nights&lt;/th&gt;
&lt;th&gt;Night Shift&lt;/th&gt;
&lt;th&gt;Raw log, same model&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Must-report events named in the two lines&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;7 / 8&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;6 / 8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kept to two lines&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;21 / 21&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;17 / 21&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nights with numbers that aren't in the log&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;7 (4 of them wrong counts)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;What the model reads (whole prompt)&lt;/td&gt;
&lt;td&gt;~1,200 characters&lt;/td&gt;
&lt;td&gt;up to 1.7 M characters before the cut&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;What the numbers hide:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The raw log is good on small nights.&lt;/strong&gt; On the night lab-se2 died, there were only four entries, and it wrote the better line: &lt;em&gt;"Instance terminated at 18:39:13 due to MMAN dying unexpectedly, triggering PMON error 822."&lt;/em&gt; Night Shift's line said the instance terminated, but not why. The why is in the list underneath.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On big nights it stops following instructions.&lt;/strong&gt; Four nights didn't fit the 32K-token context I gave it, so Ollama cut them down to about 16K tokens and most of the night was lost. On all four it ignored "two lines" and wrote a multi-page essay with headings and a priority table. On one of them, the night's only deadlock was in the part that got cut, so the model never saw it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It can't count.&lt;/strong&gt; On four different nights it told me how many log switches there had been: 54, 33, 87 and 45. The real numbers were 61, 69, 58 and 77. That's four for four wrong, and each one sounded completely sure of itself. The other three nights' numbers are rounded file sizes it converted from KB, values from SQL it wrote itself, and the next log sequence after the window. Night Shift's counts come from Python. The model only gets to repeat them, and the number check rejects anything it makes up.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It can't see silence.&lt;/strong&gt; The night after the crash, there was nothing to send it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Night Shift's miss:&lt;/strong&gt; on lab-se2's first night, the two lines named the shutdown, the missing redo log and 17 job failures, but not the 2 deadlocks. They're in the list, but not in what you'd read at 7 AM.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The honest caveats:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;These are lab logs, and the must-report events are few.&lt;/li&gt;
&lt;li&gt;One of the two "instance died" nights is a &lt;code&gt;shutdown abort&lt;/code&gt; I ran during setup.&lt;/li&gt;
&lt;li&gt;The tablespace was filled on purpose.&lt;/li&gt;
&lt;li&gt;I changed the prompt once after the first replay. The two lines kept skipping deadlocks, and Gemma was writing "as per the finding's 'do' field". That first run scored 6/8 and is in the repo next to the second. I also loosened one scoring pattern from "silent" to "silen" after Gemma wrote "alert log silence", which gave Night Shift one of its 7.&lt;/li&gt;
&lt;li&gt;A rerun with the final code, which adds the restart rule, scored 8/8 (&lt;code&gt;results/replay-v3.json&lt;/code&gt;). The two lines vary a little from run to run, so the table keeps the lower number.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;p&gt;An alert log is a map of someone's infrastructure. It has host names, file paths, user names, SQL fragments, and the exact times their systems fell over. A DBA can't paste a client's into a hosted chatbot. I wouldn't want to give a SaaS agent a login to production just to read it, either.&lt;/p&gt;

&lt;p&gt;The open pieces are what make this possible:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Gemma 4 on Ollama&lt;/strong&gt; runs on the same desktop as the databases, at no cost per night, with no internet needed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Temporal&lt;/strong&gt; is open source, so the durable part runs locally as one binary and a SQLite file. Nothing is hosted anywhere, and you still get retries, resume and schedules.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;I can see why the model said what it said.&lt;/strong&gt; The exact digest it read is right there. When it's wrong, I fix a rule or the prompt instead of hoping the next model version is better. A closed API wouldn't have let me run the raw-log baseline on 22 nights for free either, and that baseline is what told me where the model needs help.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The Lazy Payoff
&lt;/h2&gt;

&lt;p&gt;Reading two alert logs properly takes a good ten minutes on a busy night, and nothing at all on the nights you skip it, which is how I lost five days. Now it's two lines with my coffee, and the full list only when a line is red.&lt;/p&gt;

&lt;p&gt;Its first real run told me what I'd missed all week: &lt;strong&gt;lab-se2 is DOWN, last alert log entry Tuesday 18:39 UTC.&lt;/strong&gt; So I started it. The next brief had moved on by itself:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;## lab-se2: up
2026-10-25 00:03 to 2026-10-26 00:03 UTC · 90 log lines, 85 routine

- 🟡 Instance started, 2026-10-26 00:02 UTC. Check the restart was planned; the entries just before it say why it stopped.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It also points at the listener's "address already in use" lines from the restart, which are worth a glance and nothing more. The Temporal Schedule fires for the first time at 6:00 tomorrow morning, and this is what will be waiting:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flf5s5g8czjqu0zq5ewjy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flf5s5g8czjqu0zq5ewjy.png" alt="Desktop notification from Night Shift with the first line of the brief"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Tonight, the sloth is on shift.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Best Use of Temporal:&lt;/strong&gt; the brief is a Temporal workflow with one activity per database, retry policies, and a fallback when the model is down. A host that can't be reached turns into a finding instead of a crash. A Temporal Schedule runs it at 6 AM. I tested it by killing the worker mid-run and pointing it at a host that doesn't exist (timeline above). Everything runs locally on Temporal's open-source dev server.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Best Use of Gemma:&lt;/strong&gt; Gemma 4 (e4b) runs locally in Ollama and writes the two lines from a compact digest, checked for invented numbers and missing databases. The replay compares it with the same model reading the raw log, night by night.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
      <category>oracle</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Checkpoint Not Complete: Every Commit Waits, and the SQL Is Fine</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Sun, 25 Oct 2026 20:49:26 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/checkpoint-not-complete-every-commit-waits-and-the-sql-is-fine-3g41</link>
      <guid>https://dev.to/uptimearchitect/checkpoint-not-complete-every-commit-waits-and-the-sql-is-fine-3g41</guid>
      <description>&lt;p&gt;The nightly batch used to finish in twenty minutes. Now it takes an hour, and during that hour the application stutters: every few seconds, every session that writes anything (an order, a login timestamp, a session audit row) freezes for a second or two, then carries on. The SQL hasn't changed. The plans haven't changed. CPU is fine, the storage team says the disks are fine, and the alert log has the same two lines over and over:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Thread 1 cannot allocate new log, sequence 23
Checkpoint not complete
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing is broken, and no query is slow. The database is waiting for itself: the &lt;strong&gt;online redo logs are too small for the rate at which the workload changes data&lt;/strong&gt;, so the log writer keeps running out of logs it is allowed to reuse. This post covers what that message means, why it freezes every writer at once, how to spot it before users do, and how to fix it online. It ends with a lab where the same update takes &lt;strong&gt;14.1 seconds&lt;/strong&gt; on two 20 MB redo logs, 11.9 of them waiting, and &lt;strong&gt;1.9 seconds&lt;/strong&gt; after the fix.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the redo log circle works
&lt;/h2&gt;

&lt;p&gt;Every change in an Oracle database is first written as &lt;strong&gt;redo&lt;/strong&gt; to the online redo logs, a small, fixed set of files (&lt;em&gt;groups&lt;/em&gt;) that the log writer (LGWR) fills one after another in a circle. When the current group is full, LGWR &lt;strong&gt;switches&lt;/strong&gt; to the next one, and when it reaches the last group it wraps around and reuses the first.&lt;/p&gt;

&lt;p&gt;Reusing a group is only safe once nothing still needs the redo in it for crash recovery. The redo protects changes to data blocks that may still exist only in memory. So before a group can be overwritten, the &lt;strong&gt;checkpoint&lt;/strong&gt; that covers it must finish: the database writer (DBWR) has to write every dirty block whose changes are recorded in that group to the datafiles. Until that happens, &lt;code&gt;V$LOG&lt;/code&gt; shows the group as &lt;code&gt;ACTIVE&lt;/code&gt;, meaning still needed for recovery, and LGWR cannot touch it.&lt;/p&gt;

&lt;p&gt;Normally the circle is big enough that DBWR finishes long before LGWR comes back around, and the next group is always &lt;code&gt;INACTIVE&lt;/code&gt; and ready.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why it freezes everyone
&lt;/h2&gt;

&lt;p&gt;When the logs are small relative to the write rate, LGWR fills the whole circle faster than DBWR can complete the checkpoints. It comes back to a group that is still &lt;code&gt;ACTIVE&lt;/code&gt;, and it has nowhere to write. The alert log records it as &lt;em&gt;"cannot allocate new log … Checkpoint not complete"&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;LGWR is a single point of serialisation. Until DBWR frees that group, &lt;strong&gt;no session in the database can generate redo&lt;/strong&gt;: no insert, no update, no delete, no commit. They all wait on the same event, &lt;code&gt;log file switch (checkpoint incomplete)&lt;/code&gt;, and then all resume at once. That's the stutter. It's not one slow query; it's the whole instance pausing every time the log writer laps the checkpoint.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpVUl1rwjAU_SuXPE1Q9l6GY7NzCLKHVaaw7iE2t20wzS1Juk7U_74krbi9JTk5H_ckJ1aQQJYAKxX1Rc2Ng02aa4DtZ84ytFaStuABXSEI7vjD3tzP76S2aNwUutaf4RQKahrpJjn7gtlsDmtPXr9u36E30qEFg4IiUWpH4GqEojMGtQNFFVSGutZTg-068pennC3GGxGFslMqGSRsFND4c8VWb0-Lzerj5TFnl6CxDBrnnB3RJiBdiI_FoSVvDp4tSGPOzpCFCXvpihq4FnBAbGNeqasxzFVIUwLWSaVgMArsnWcvbrqaXCihVegwxsRvNMdhfgM9DzFIRySMXEqFYAfzu7_x9FVkMmbYxUJS75Y--0JLqaWtfaVj0tBFVBXSuCPsFRUHCyUZD3D3r9o0KmVhmQ2vFJaF4tZ6lz0Xt20GFZFgU2ANmoZL4X_IiXmrJv4VgSXvlGOXyy_AZsDY" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpVUl1rwjAU_SuXPE1Q9l6GY7NzCLKHVaaw7iE2t20wzS1Juk7U_74krbi9JTk5H_ckJ1aQQJYAKxX1Rc2Ng02aa4DtZ84ytFaStuABXSEI7vjD3tzP76S2aNwUutaf4RQKahrpJjn7gtlsDmtPXr9u36E30qEFg4IiUWpH4GqEojMGtQNFFVSGutZTg-068pennC3GGxGFslMqGSRsFND4c8VWb0-Lzerj5TFnl6CxDBrnnB3RJiBdiI_FoSVvDp4tSGPOzpCFCXvpihq4FnBAbGNeqasxzFVIUwLWSaVgMArsnWcvbrqaXCihVegwxsRvNMdhfgM9DzFIRySMXEqFYAfzu7_x9FVkMmbYxUJS75Y--0JLqaWtfaVj0tBFVBXSuCPsFRUHCyUZD3D3r9o0KmVhmQ2vFJaF4tZ6lz0Xt20GFZFgU2ANmoZL4X_IiXmrJv4VgSXvlGOXyy_AZsDY" alt="LGWR writes redo into a circle of online log groups. Before it can reuse a group, DBWR mus" width="450" height="1030"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;LGWR writes redo into a circle of online log groups. Before it can reuse a group, DBWR must finish the checkpoint covering it. With two 20 MB groups and a heavy write burst, LGWR comes back around to a group that is still ACTIVE, and every session that needs to write redo waits until DBWR frees it. With groups large enough to absorb the burst, the next group is always INACTIVE and nobody waits.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;This is a close cousin of &lt;a href="https://uptimearchitect.com/blog/oracle-ora-00257-archiver-stuck/" rel="noopener noreferrer"&gt;ORA-00257, archiver stuck&lt;/a&gt;. There, a log group can't be reused because the &lt;strong&gt;archiver&lt;/strong&gt; hasn't copied it yet, and the wait is &lt;code&gt;log file switch (archiving needed)&lt;/code&gt;. Here it's because the &lt;strong&gt;checkpoint&lt;/strong&gt; hasn't finished. Same circle, same freeze, different fix.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it
&lt;/h2&gt;

&lt;p&gt;The lab uses the community Oracle Database Free container image, which trims the redo logs to &lt;strong&gt;two groups of 20 MB each&lt;/strong&gt; to keep the image small: exactly the layout that causes this. It loads a table of 100,000 rows of about 1 KB, then runs one statement that rewrites every row and commits, a little over 200 MB of redo. It measures the elapsed time, the waits from &lt;code&gt;V$SYSTEM_EVENT&lt;/code&gt;, the switches from &lt;code&gt;V$LOG_HISTORY&lt;/code&gt;, and the &lt;code&gt;Checkpoint not complete&lt;/code&gt; lines in the alert log:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;redo logs              elapsed   switches   waits (time waited)   alert lines
2 x 20 MB (image)       14.1 s         12        12 (11.9 s)              12
3 x 512 MB               1.9 s          0         0                        0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the default logs, &lt;strong&gt;84% of the update's elapsed time was waiting&lt;/strong&gt; for a redo log to become reusable: 11.9 of 14.1 seconds. The statement itself was never the problem. After the logs were resized, the identical update ran in 1.9 seconds with no switches and no waits.&lt;/p&gt;

&lt;p&gt;Elapsed times depend on the hardware and vary from run to run; on my machine the small-log update took between 8 and 14 seconds. The waits and the alert-log lines appear every time, and disappear every time after the resize.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't take my word for it — run it.&lt;/strong&gt; The &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/checkpoint-incomplete" rel="noopener noreferrer"&gt;checkpoint-incomplete lab&lt;/a&gt; runs the burst on the default 2 x 20 MB logs and &lt;strong&gt;asserts&lt;/strong&gt; that &lt;code&gt;log file switch (checkpoint incomplete)&lt;/code&gt; waits and &lt;code&gt;Checkpoint not complete&lt;/code&gt; alert-log lines appear. It then resizes the logs online and asserts the same burst produces &lt;strong&gt;zero&lt;/strong&gt; of both. If the stall doesn't reproduce, or the fix doesn't remove it, the run fails. Proven on every CI push.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  How to spot it
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The alert log&lt;/strong&gt; is the giveaway. Search it for &lt;code&gt;Checkpoint not complete&lt;/code&gt;; one occurrence during a once-a-year data load is noise, but dozens a day mean the logs are undersized for your real workload.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The wait event.&lt;/strong&gt; &lt;code&gt;log file switch (checkpoint incomplete)&lt;/code&gt; shows up in &lt;code&gt;V$SYSTEM_EVENT&lt;/code&gt;, in AWR's Top Events and in ASH. In an &lt;a href="https://uptimearchitect.com/blog/how-to-read-an-awr-report/" rel="noopener noreferrer"&gt;AWR report&lt;/a&gt; it's often near the top during batch windows, and it's easy to misread as an I/O problem, because the time is spent waiting on DBWR's writes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How often the logs switch.&lt;/strong&gt; &lt;code&gt;V$LOG_HISTORY&lt;/code&gt; has one row per switch:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;TO_CHAR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;TRUNC&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;first_time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'HH24'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="s1"&gt;'YYYY-MM-DD HH24'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;hour&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                                             &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;switches&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;log_history&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;first_time&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;SYSDATE&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;7&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;TRUNC&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;first_time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'HH24'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A common rule of thumb is a few switches per hour at peak, roughly one every 15 to 20 minutes. The lab's update switched 12 times in 14 seconds. If your busiest hours show switches &lt;em&gt;per minute&lt;/em&gt;, the logs are far too small for that workload, whether or not you've seen the alert yet.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Oracle's own sizing advice.&lt;/strong&gt; &lt;code&gt;V$INSTANCE_RECOVERY.OPTIMAL_LOGFILE_SIZE&lt;/code&gt; suggests a redo log size in megabytes, but only when &lt;code&gt;FAST_START_MTTR_TARGET&lt;/code&gt; is set. With it at 0, as on the lab database, the column is empty. In the lab, with a 60-second target, it suggested 827 MB. Treat it as a starting point, not a law: it's derived from the recovery target, not from your peak write rate.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;You can't resize a redo log file in place. Instead, you &lt;strong&gt;add new groups at the size you want, move onto them, and drop the old ones&lt;/strong&gt;, all online:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- 1. Add bigger groups (use your own paths, and two members each on separate disks in production)&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="n"&gt;LOGFILE&lt;/span&gt; &lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'/u01/oradata/PROD/redo03.log'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;SIZE&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="k"&gt;G&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="n"&gt;LOGFILE&lt;/span&gt; &lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'/u01/oradata/PROD/redo04.log'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;SIZE&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="k"&gt;G&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="n"&gt;LOGFILE&lt;/span&gt; &lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'/u01/oradata/PROD/redo05.log'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;SIZE&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="k"&gt;G&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- 2. Switch until one of the new groups is CURRENT, then checkpoint&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;SYSTEM&lt;/span&gt; &lt;span class="n"&gt;SWITCH&lt;/span&gt; &lt;span class="n"&gt;LOGFILE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;SYSTEM&lt;/span&gt; &lt;span class="k"&gt;CHECKPOINT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- 3. Wait until the old groups show INACTIVE in V$LOG, then drop them&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;group&lt;/span&gt;&lt;span class="o"&gt;#&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bytes&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;mb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="k"&gt;DROP&lt;/span&gt; &lt;span class="n"&gt;LOGFILE&lt;/span&gt; &lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="k"&gt;DROP&lt;/span&gt; &lt;span class="n"&gt;LOGFILE&lt;/span&gt; &lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Order matters.&lt;/strong&gt; A &lt;code&gt;CURRENT&lt;/code&gt; group can't be dropped, and neither can an &lt;code&gt;ACTIVE&lt;/code&gt; one, because it's still needed for recovery. Switch off it, checkpoint, and wait for &lt;code&gt;INACTIVE&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dropping a group doesn't delete its file.&lt;/strong&gt; Remove the old files at the operating-system level afterwards (with Oracle Managed Files, the database does it for you).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Size for the peak, not the average.&lt;/strong&gt; The goal is that, at your busiest, a log lasts long enough for DBWR to finish the checkpoint long before LGWR needs it again. In the lab, 512 MB groups hold the entire burst, so there is no switch at all. In production, aim for that rule of thumb of a few switches an hour at peak.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;More groups help too.&lt;/strong&gt; A third or fourth group gives DBWR more time before LGWR comes back around. Two groups is the bare minimum Oracle allows, and it leaves no slack.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;All groups the same size.&lt;/strong&gt; Mixed sizes make switch timing unpredictable and complicate standby redo logs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The other lever is DBWR itself: if it can't keep up because the storage is slow or asynchronous I/O is off, checkpoints take longer at any log size. (The lab container runs with &lt;code&gt;FILESYSTEMIO_OPTIONS=none&lt;/code&gt;, which doesn't help DBWR either.) But undersized logs are far more common, and resizing them is a five-minute, fully online change.&lt;/p&gt;

&lt;h2&gt;
  
  
  What teams get wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Treating it as a storage problem.&lt;/strong&gt; The waits look like I/O, so the first instinct is faster disks. Faster writes help DBWR a little; logs sized for the workload remove the wait entirely.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never changing the default.&lt;/strong&gt; Many databases still run on the redo logs they were created with, sized for the workload of the day they were built, or trimmed for a small footprint the way the lab's container image trims them to two 20 MB groups. Fine for a laptop, not for a batch window. Check yours.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Forgetting the standby.&lt;/strong&gt; With Data Guard, standby redo logs should be the same size as the online redo logs, with one more group per thread. Resize the online logs and leave the standby redo logs small, and the standby can't use them for real-time apply; it falls back to applying from archived logs. Resize both.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Confusing it with the archiver.&lt;/strong&gt; &lt;code&gt;log file switch (archiving needed)&lt;/code&gt; is &lt;a href="https://uptimearchitect.com/blog/oracle-ora-00257-archiver-stuck/" rel="noopener noreferrer"&gt;ORA-00257&lt;/a&gt;'s wait: the archive destination is full or slow. Bigger logs don't fix that one; freeing the destination does.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fearing longer recovery, or ignoring it.&lt;/strong&gt; With &lt;code&gt;FAST_START_MTTR_TARGET&lt;/code&gt; at 0, how far the checkpoint lags is tied to the size of the smallest log, so bigger logs can mean more redo to replay after a crash. When you enlarge the logs, set &lt;code&gt;FAST_START_MTTR_TARGET&lt;/code&gt; (say, 60 to 300 seconds) and the database paces its checkpoint writes to keep recovery time bounded. Undersized logs cost you every busy day; properly sized ones cost a little disk.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dropping the wrong group.&lt;/strong&gt; Run &lt;code&gt;SELECT group#, status FROM v$log&lt;/code&gt; before every &lt;code&gt;DROP LOGFILE&lt;/code&gt;. A &lt;code&gt;CURRENT&lt;/code&gt; or &lt;code&gt;ACTIVE&lt;/code&gt; group can't be dropped, and in a hurry it's easy to type the wrong number.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What does 'Checkpoint not complete' mean in the Oracle alert log?
&lt;/h3&gt;

&lt;p&gt;It means the log writer (LGWR) filled the current online redo log and wanted to switch to the next group in the circle, but that group was still ACTIVE: the checkpoint covering it had not finished, so the database writer (DBWR) had not yet written all the dirty blocks whose changes are recorded in that log. The log cannot be overwritten until that happens, so LGWR waits, and while it waits no session can generate redo. The usual cause is online redo logs that are too small for the database's write rate, so LGWR laps the checkpoint during busy periods.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the 'log file switch (checkpoint incomplete)' wait event?
&lt;/h3&gt;

&lt;p&gt;It is the wait that sessions experience when they need to write redo but the log writer cannot switch to the next online redo log, because that log is still needed for crash recovery until a checkpoint completes. Every session that inserts, updates, deletes or commits waits on it at the same time, which is why it shows up as the whole database stalling rather than one slow query. In the companion lab, a 100,000-row update spent 11.9 of its 14.1 seconds on this wait with two 20 MB redo logs, and zero seconds after resizing them.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I resize Oracle online redo logs?
&lt;/h3&gt;

&lt;p&gt;Online redo log files cannot be resized in place. Add new log groups at the desired size with ALTER DATABASE ADD LOGFILE GROUP n ... SIZE, run ALTER SYSTEM SWITCH LOGFILE until one of the new groups is CURRENT, run ALTER SYSTEM CHECKPOINT, wait until V$LOG shows the old groups as INACTIVE, then drop them with ALTER DATABASE DROP LOGFILE GROUP n and delete their files at the operating-system level (Oracle Managed Files are removed automatically). The whole procedure is online. Keep all groups the same size, and in a Data Guard configuration resize the standby redo logs to match.&lt;/p&gt;

&lt;h3&gt;
  
  
  How big should Oracle redo logs be?
&lt;/h3&gt;

&lt;p&gt;Big enough that, at peak load, a log lasts long enough for the checkpoint covering the previous logs to finish well before the log writer needs them again. A common rule of thumb is a few log switches per hour at the busiest time, roughly one every 15 to 20 minutes. Check V$LOG_HISTORY for switches per hour; switches per minute mean the logs are far too small. V$INSTANCE_RECOVERY.OPTIMAL_LOGFILE_SIZE also suggests a size in megabytes, but only when FAST_START_MTTR_TARGET is set. Use at least three groups, all the same size.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is 'Checkpoint not complete' the same as ORA-00257 archiver error?
&lt;/h3&gt;

&lt;p&gt;No, although both freeze every session that writes. With 'Checkpoint not complete', the next online redo log cannot be reused because its checkpoint has not finished, and sessions wait on log file switch (checkpoint incomplete); the fix is usually larger or more redo log groups. With ORA-00257, the database is in ARCHIVELOG mode and the next log cannot be reused because the archiver has not been able to copy it, typically because the archive destination or Fast Recovery Area is full; sessions wait on log file switch (archiving needed), and the fix is to free or enlarge the archive destination.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do larger redo logs make crash recovery slower?
&lt;/h3&gt;

&lt;p&gt;Not in a way that should stop you from sizing them properly. Crash recovery time depends on how much redo must be applied since the last checkpoint position, and modern Oracle uses incremental checkpointing to write dirty blocks continuously rather than only at log switches. With FAST_START_MTTR_TARGET at 0, how far that checkpoint may lag is tied to the size of the smallest redo log, so larger logs can mean more redo to replay. Setting FAST_START_MTTR_TARGET makes the database pace its checkpoint writes to keep the estimated recovery time within the target, so set it when you enlarge the logs. Undersized logs, on the other hand, stall every writer on every busy day.&lt;/p&gt;

&lt;p&gt;Redo sizing is one of those settings that is decided once, at database creation, and then forgotten until the workload outgrows it. The symptoms point everywhere else: the &lt;a href="https://uptimearchitect.com/blog/oracle-wait-events-decoded/" rel="noopener noreferrer"&gt;wait events&lt;/a&gt; look like I/O, the batch looks slow, and the SQL looks guilty. Check the alert log for &lt;code&gt;Checkpoint not complete&lt;/code&gt;, count your log switches per hour, and if the logs are lapping the checkpoint, give them room: add bigger groups, switch, checkpoint, drop the old ones, and resize the standby redo logs to match. Then prove it the way that ends the argument — with the &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/checkpoint-incomplete" rel="noopener noreferrer"&gt;checkpoint-incomplete lab&lt;/a&gt;, where the same update drops from 14.1 seconds, 11.9 of them waiting, to 1.9.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://uptimearchitect.com/blog/oracle-checkpoint-not-complete/" rel="noopener noreferrer"&gt;uptimearchitect.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>oracle</category>
      <category>database</category>
      <category>devops</category>
      <category>sre</category>
    </item>
    <item>
      <title>Same SQL, Fast for One Value, Slow for Another</title>
      <dc:creator>Uptime Architect</dc:creator>
      <pubDate>Wed, 21 Oct 2026 13:46:44 +0000</pubDate>
      <link>https://dev.to/uptimearchitect/same-sql-fast-for-one-value-slow-for-another-22he</link>
      <guid>https://dev.to/uptimearchitect/same-sql-fast-for-one-value-slow-for-another-22he</guid>
      <description>&lt;p&gt;The order-history screen is fast for every customer except one. Or it's fast all morning and crawls all afternoon. Or it was fine for months and became slow the night after a restart, with nobody deploying anything. You pull the statement and run it yourself with the problem value, and it comes back instantly. The SQL is identical. The data hasn't changed. The plan in the execution history is the same plan it always uses.&lt;/p&gt;

&lt;p&gt;That last sentence is the clue. The statement uses a &lt;strong&gt;bind variable&lt;/strong&gt;, which is the right thing to do, and the cursor for it holds &lt;strong&gt;one plan&lt;/strong&gt;. That plan was chosen for whichever value happened to arrive first. On a column where values are wildly unevenly distributed, that's a plan built for one kind of caller and handed to every other. This post is how that happens, why Oracle's safety net for it, adaptive cursor sharing, doesn't always catch it, how to see it in the cursor cache, and how to fix it, with a lab that reproduces it on demand.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bind peeking: the first caller picks the plan
&lt;/h2&gt;

&lt;p&gt;When a statement with a bind variable is hard-parsed, the optimizer &lt;strong&gt;peeks&lt;/strong&gt; at the actual bind value and costs the plan as if that value were a literal. That's usually good: a plan built for a real value beats a plan built for an average. The catch is what happens next. The finished cursor goes into the shared pool and &lt;strong&gt;every later execution reuses it&lt;/strong&gt;, whatever value it binds. The plan is decided once, by the first caller.&lt;/p&gt;

&lt;p&gt;On a column with evenly spread values, that doesn't matter: one value costs about the same as another. It matters on a &lt;strong&gt;skewed&lt;/strong&gt; column, where a few values are rare and one is everywhere: a status code where 99.9% of rows are &lt;code&gt;CLOSED&lt;/code&gt;, a country column dominated by one country, a flag that's almost always &lt;code&gt;N&lt;/code&gt;. If the column has a &lt;strong&gt;histogram&lt;/strong&gt;, the optimizer knows about the skew and builds very different plans for the rare value and the common one. Peeking means it builds &lt;strong&gt;one&lt;/strong&gt; of them, and everyone gets that one.&lt;/p&gt;

&lt;p&gt;The classic shape is a join. For a rare value, a few hundred driving rows, the best plan is &lt;strong&gt;nested loops&lt;/strong&gt;: for each order, look up its customer by primary key. For the common value, hundreds of thousands of rows, the best plan is a &lt;strong&gt;hash join&lt;/strong&gt;: read both tables once and match them in memory. Nested loops for the common value means one index probe per row, hundreds of thousands of times.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpNk1Fv0zAUhf_KlV9WpKwUBohFMLS16boJ1qkdQ4jswY1vGtMkjnyddVPb_75rN4W9RFFyfb5zzpU3IjMKRQwiL806K6R1cDdKa4DzP6mYSKugkZYQTA79fh9-TZJZAuSkawm-QkypeIDj4zO42KTiXpYtQoO4QvVlYd-emRpcgZBrSw7wCbPWaVN_S8XOEy78wW0qrLQYw9H0Nrk5gt77wQCsWdObVGxhyCZukBwqKI1pKIbGmgXC8Of8bvojmc07TOaxFoxV_OxZvSwc5MYCi7HOw2taZqrK1Mwbfp_OkxET352eRp_D4BZGITUV8NdoHrIoFSyMK8DJRYkEHtUJDkPuhA8MW0sMs9gSG_Xc5D6Z_YZH30c3nYTpMU__A9u2JqhfpwtpOjv7pBTBp8FJdPLxQ_BCndo4qF1y5-dKNk4_ImR7E8Qr1PUyKBGyYwlK5zlaZOeH5i-7LmoTA8kKYW3sqjQ-a5ut0PkmJmx17uSzzxy2uODfTSnrKGizeZC547r5rXN1kH1G8gpXYXlryApdqs7fPmJxaDiUdSikU5mEbNd8eKyfYqZjwIYFh0aDRK_UDJdlUHCFJu4Llc6kw8PGs1ISwTiaeOv_P4yiq-galsYoEYGo0FZSK74BG8Epq3AXFOayLZ3Y7V4A3Qz-PQ" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNpNk1Fv0zAUhf_KlV9WpKwUBohFMLS16boJ1qkdQ4jswY1vGtMkjnyddVPb_75rN4W9RFFyfb5zzpU3IjMKRQwiL806K6R1cDdKa4DzP6mYSKugkZYQTA79fh9-TZJZAuSkawm-QkypeIDj4zO42KTiXpYtQoO4QvVlYd-emRpcgZBrSw7wCbPWaVN_S8XOEy78wW0qrLQYw9H0Nrk5gt77wQCsWdObVGxhyCZukBwqKI1pKIbGmgXC8Of8bvojmc07TOaxFoxV_OxZvSwc5MYCi7HOw2taZqrK1Mwbfp_OkxET352eRp_D4BZGITUV8NdoHrIoFSyMK8DJRYkEHtUJDkPuhA8MW0sMs9gSG_Xc5D6Z_YZH30c3nYTpMU__A9u2JqhfpwtpOjv7pBTBp8FJdPLxQ_BCndo4qF1y5-dKNk4_ImR7E8Qr1PUyKBGyYwlK5zlaZOeH5i-7LmoTA8kKYW3sqjQ-a5ut0PkmJmx17uSzzxy2uODfTSnrKGizeZC547r5rXN1kH1G8gpXYXlryApdqs7fPmJxaDiUdSikU5mEbNd8eKyfYqZjwIYFh0aDRK_UDJdlUHCFJu4Llc6kw8PGs1ISwTiaeOv_P4yiq-galsYoEYGo0FZSK74BG8Epq3AXFOayLZ3Y7V4A3Qz-PQ" alt="A bind-variable statement is optimized once, for the first value it sees, and the cursor i" width="708" height="1483"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A bind-variable statement is optimized once, for the first value it sees, and the cursor is reused for every value after. On a skewed column with a histogram, a rare first value produces a nested-loops plan that is right for a few rows; the common value then inherits it and pays for one index probe per row. Adaptive cursor sharing can split the cursor into one plan per kind of value, but only once it has seen executions that differ enough; until then every caller gets the first caller's plan.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Adaptive cursor sharing: the safety net, with holes
&lt;/h2&gt;

&lt;p&gt;Oracle has known about this since 11g, and &lt;strong&gt;adaptive cursor sharing&lt;/strong&gt; (ACS) is its answer. A cursor that peeked a bind on a column with a histogram is marked &lt;strong&gt;bind-sensitive&lt;/strong&gt;. Oracle then watches its executions, sorting each one into a small number of buckets by how much work it did. If executions with different bind values land in clearly different buckets, the cursor becomes &lt;strong&gt;bind-aware&lt;/strong&gt;: from then on, each new value is checked against the existing child cursors, and a value that needs a different plan gets its own child cursor with its own plan.&lt;/p&gt;

&lt;p&gt;That works well when the difference is visible. The hole is in "clearly different". If the rare value's first execution and the common value's executions land in the &lt;strong&gt;same bucket&lt;/strong&gt;, ACS sees nothing to react to, and the cursor stays exactly as it was. And whether you're protected can depend on &lt;strong&gt;the order in which values arrive&lt;/strong&gt; after each restart, shared pool flush, or statistics refresh, which nobody controls.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it
&lt;/h2&gt;

&lt;p&gt;The lab builds &lt;code&gt;ORDERS&lt;/code&gt; with 200,000 rows and a skewed &lt;code&gt;STATUS&lt;/code&gt;, &lt;strong&gt;200 &lt;code&gt;OPEN&lt;/code&gt; and 199,800 &lt;code&gt;CLOSED&lt;/code&gt;&lt;/strong&gt;, with a frequency histogram on the column, and joins it to a 1,000,000-row &lt;code&gt;CUSTOMERS&lt;/code&gt; table. It flushes the shared pool, runs the statement once with &lt;code&gt;'OPEN'&lt;/code&gt;, then five times with &lt;code&gt;'CLOSED'&lt;/code&gt;, each in a fresh session, measuring session logical reads from &lt;code&gt;V$MYSTAT&lt;/code&gt; and the child cursor each execution ran:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;&amp;gt;&amp;gt; Fresh shared pool. The SAME statement, bound:  ... where o.status = :s
   OPEN    rows=200     logical reads=2624     child cursor=0  plan=NESTED LOOPS
   CLOSED  rows=199800  logical reads=603391   child cursor=0  plan=NESTED LOOPS
   CLOSED  rows=199800  logical reads=603354   child cursor=0  plan=NESTED LOOPS
   CLOSED  rows=199800  logical reads=603354   child cursor=0  plan=NESTED LOOPS
   CLOSED  rows=199800  logical reads=603354   child cursor=0  plan=NESTED LOOPS
   CLOSED  rows=199800  logical reads=603354   child cursor=0  plan=NESTED LOOPS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The optimizer peeked &lt;code&gt;'OPEN'&lt;/code&gt;, estimated 199 rows from the histogram, and chose nested loops, which is right for 200 rows. Every &lt;code&gt;'CLOSED'&lt;/code&gt; run then reused child cursor 0 and its nested loops: &lt;strong&gt;603,354 logical reads, five times in a row&lt;/strong&gt;. Adaptive cursor sharing never marked the cursor bind-aware. The first &lt;code&gt;'OPEN'&lt;/code&gt; execution, including its hard parse, and every &lt;code&gt;'CLOSED'&lt;/code&gt; execution fell into the same middle bucket, so the safety net never saw a difference. (The plan was even an adaptive plan, with a statistics collector that can switch join methods, but an adaptive plan settles on its first execution and the settled plan is what gets reused.)&lt;/p&gt;

&lt;p&gt;Give &lt;code&gt;'CLOSED'&lt;/code&gt; its own plan and the cost collapses:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;&amp;gt;&amp;gt; The fix for a skewed, low-cardinality column: the predicate as a literal -&amp;gt; one cursor + plan per value
   CLOSED  rows=199800  logical reads=34716    child cursor=0  plan=HASH JOIN
   OPEN    rows=200     logical reads=806      child cursor=0  plan=NESTED LOOPS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;34,716 logical reads instead of 603,354&lt;/strong&gt;: the same rows, 17 times less work, because the plan was built for the value it runs. And &lt;code&gt;'OPEN'&lt;/code&gt; keeps its cheap nested loops at 806.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't take my word for it — run it.&lt;/strong&gt; The &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/bind-peeking" rel="noopener noreferrer"&gt;bind-peeking lab&lt;/a&gt; builds the skewed tables on Oracle Database Free, peeks the rare value, and &lt;strong&gt;asserts&lt;/strong&gt; that every common-value run inherits nested loops on the same cursor at 10 times or more the work of a hash join, without adaptive cursor sharing splitting it. Then it gives each value its own plan and asserts the common value drops to 10% or less of that work. If the trap stops reproducing or the fix stops working, the run fails. Proven on every CI push.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  See it in your own system
&lt;/h2&gt;

&lt;p&gt;Start from the statement, not the plan. A query whose cost swings by value shows up as &lt;strong&gt;high variance in buffer gets per execution&lt;/strong&gt; for one SQL_ID. Then look at its cursors:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- one row per child cursor: which ones are bind-sensitive / bind-aware, and what each costs&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;child_number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;plan_hash_value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;is_bind_sensitive&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;is_bind_aware&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;executions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buffer_gets&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="k"&gt;NULLIF&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;executions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;gets_per_exec&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="k"&gt;sql&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;sql_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'&amp;amp;sql_id'&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;child_number&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Bind-sensitive &lt;code&gt;Y&lt;/code&gt;, bind-aware &lt;code&gt;N&lt;/code&gt;, one child, huge gets per execution:&lt;/strong&gt; the lab's situation. The cursor knows the bind matters, but it never split.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Several children with different plan hash values, bind-aware &lt;code&gt;Y&lt;/code&gt;:&lt;/strong&gt; ACS is doing its job.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;V$SQL_CS_HISTOGRAM&lt;/code&gt;&lt;/strong&gt; shows how its executions were bucketed; all executions in one bucket means ACS has nothing to react to.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To see which value the plan was built for, ask for the peeked binds:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;DBMS_XPLAN&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DISPLAY_CURSOR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'&amp;amp;sql_id'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'TYPICAL +PEEKED_BINDS'&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the lab that prints &lt;code&gt;:S ... 'OPEN'&lt;/code&gt; above a nested-loops plan, the whole diagnosis in one screen. In AWR, the same statement appearing with &lt;strong&gt;more than one plan hash value&lt;/strong&gt; over time, with very different elapsed times, is the historical fingerprint: the plan changes whenever the cursor is rebuilt with a different first value.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;Pick by how the column behaves. In order of preference:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;For a skewed, low-cardinality column, use a literal for that predicate.&lt;/strong&gt; A status code with five values makes five cursors, each with the right plan. That's no hard-parse storm: the storm comes from literal values with high cardinality, like IDs. Keep binds for everything else in the statement. This is the exception to "bind everything" that the &lt;a href="https://uptimearchitect.com/blog/oracle-bind-variables-hard-parse/" rel="noopener noreferrer"&gt;bind variables post&lt;/a&gt; ends on, and the lab's fix.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Split the query in the application&lt;/strong&gt; when it already knows which kind of value it has: a separate statement for the "open orders" screen and the "history" screen. Two statements, two plans, no guessing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A SQL plan baseline with more than one accepted plan.&lt;/strong&gt; Baselines cooperate with adaptive cursor sharing: accept the nested-loops plan and the hash-join plan, and each value can use the one that fits. See &lt;a href="https://uptimearchitect.com/blog/oracle-sql-plan-management-baselines/" rel="noopener noreferrer"&gt;SQL plan management&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Remove the histogram&lt;/strong&gt;, if one plan is genuinely acceptable for every value. Without it the optimizer stops distinguishing values and picks one middle-of-the-road plan. It's a legitimate choice when no value is catastrophic, but a bad one when, as here, the wrong plan costs 17 times the work.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What not to do: don't "fix" it by flushing the shared pool or purging the cursor when it goes bad. That just rolls the dice again on which value arrives first, and the problem returns on its own schedule.&lt;/p&gt;

&lt;h2&gt;
  
  
  What teams get wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Blaming the data or the database for "random" slowness.&lt;/strong&gt; It isn't random: it's the plan built for whichever value arrived first, and it changes when the cursor is rebuilt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Testing by hand.&lt;/strong&gt; Paste the statement into a SQL tool with the slow value typed in and it's different SQL text, so it gets its &lt;em&gt;own&lt;/em&gt; hard parse and the right plan, and it's fast. The problem only shows on the shared, bind-variable cursor the application uses.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trusting adaptive cursor sharing to handle it.&lt;/strong&gt; It helps when executions differ enough to land in different buckets, and quietly does nothing when they don't, as the lab shows five times in a row.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Flushing the shared pool as the fix.&lt;/strong&gt; It resets the lottery; it doesn't change the odds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Disabling bind peeking database-wide.&lt;/strong&gt; It's a hidden parameter, and switching it off trades one wrong plan for a plan built for no real value at all, for every statement in the database.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Turning a whole application to literals to escape it.&lt;/strong&gt; That's the &lt;a href="https://uptimearchitect.com/blog/oracle-bind-variables-hard-parse/" rel="noopener noreferrer"&gt;hard-parse storm&lt;/a&gt; all over again. Use literals only for the skewed, low-cardinality predicate.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is bind peeking in Oracle?
&lt;/h3&gt;

&lt;p&gt;Bind peeking is the optimizer's behavior of looking at the actual value of a bind variable when it hard-parses a statement, and optimizing the plan as if that value were a literal. It produces a plan suited to a real value rather than to an average. The resulting cursor is then stored in the shared pool and reused by every later execution of the same statement, regardless of the value those executions bind. So the plan is effectively chosen by whichever value happened to arrive first when the cursor was built. On columns with evenly distributed values this rarely matters; on skewed columns with histograms it can make the same SQL fast for one value and very slow for another.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why is the same Oracle query fast for one value and slow for another?
&lt;/h3&gt;

&lt;p&gt;Usually because the statement uses a bind variable on a skewed column and the cursor's single plan was built for a different kind of value. If the first execution peeked a rare value, the optimizer may have chosen nested loops or an index-driven plan that is ideal for a few rows; every later execution reuses that plan, and a common value matching hundreds of thousands of rows then pays for one index probe per row. In the companion lab the same join costs 2,624 logical reads for a rare value and 603,354 for the common value on the inherited plan, versus 34,716 with its own hash-join plan. Running the query by hand with the slow value often looks fast, because a manual run with different text gets its own hard parse and the right plan.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is adaptive cursor sharing?
&lt;/h3&gt;

&lt;p&gt;Adaptive cursor sharing, introduced in Oracle 11g, lets a statement with bind variables have more than one plan. A cursor that peeked a bind on a column with a histogram is marked bind-sensitive, and Oracle monitors how much work its executions do, sorting them into buckets. If executions with different bind values land in clearly different buckets, the cursor is marked bind-aware, and new bind values are then checked against the existing child cursors; a value that needs a different plan gets a new child cursor with its own plan. You can see it in V$SQL (IS_BIND_SENSITIVE, IS_BIND_AWARE and multiple child cursors) and in V$SQL_CS_HISTOGRAM.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why didn't adaptive cursor sharing fix my plan?
&lt;/h3&gt;

&lt;p&gt;Because adaptive cursor sharing only reacts when it sees executions that differ enough to fall into different workload buckets. If the rare value's first execution and the common value's executions land in the same bucket, the cursor stays bind-sensitive but never becomes bind-aware, and every execution keeps using the original plan. In the companion lab the first execution with the rare value, including its hard parse, and all five executions with the common value fell into the same bucket, so the cursor never split even though the common value did 17 times the necessary work each time. Whether a given workload is protected can also depend on the order in which values arrive after a restart, a shared pool flush, or a statistics refresh.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I find which bind value a plan was built for?
&lt;/h3&gt;

&lt;p&gt;Use DBMS_XPLAN.DISPLAY_CURSOR with the PEEKED_BINDS format option, for example SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR('sql_id', child_number, 'TYPICAL +PEEKED_BINDS')). The output includes a Peeked Binds section showing the value the optimizer saw when it built that child cursor's plan. Combine it with V$SQL for the same SQL_ID: one row per child cursor, its plan hash value, whether it is bind-sensitive and bind-aware, and buffer gets per execution. A single child with huge gets per execution and a peeked value from the rare end of a skewed column is the bind-peeking signature.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should I use literals instead of bind variables to avoid bind peeking problems?
&lt;/h3&gt;

&lt;p&gt;Only for the specific predicate on a skewed, low-cardinality column, such as a status code or a flag with a handful of values. Each distinct literal gets its own cursor and its own plan, and with only a few distinct values that means only a few cursors, which is no burden on the shared pool. Keep bind variables for high-cardinality values like IDs, names, and dates: using literals there causes a hard-parse storm, with thousands of nearly identical cursors, shared pool pressure, and latch contention. The rule is bind by default, literal by exception, and the exception is low-cardinality skew.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does flushing the shared pool fix a bad bind-peeked plan?
&lt;/h3&gt;

&lt;p&gt;Only by accident, and only temporarily. Flushing the shared pool, or purging the one cursor with DBMS_SHARED_POOL.PURGE, forces a new hard parse, and the new plan will be built for whichever bind value arrives first afterwards. If that is the common value, things look fixed until the next time the cursor is rebuilt with the rare value first; if it is the rare value, nothing changes. It resets the lottery without changing the odds. Fix the cause instead: give the skewed predicate a literal, split the statement in the application, or use a SQL plan baseline with more than one accepted plan.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should I disable bind peeking?
&lt;/h3&gt;

&lt;p&gt;No. Disabling bind peeking requires a hidden parameter, which should only be changed under Oracle Support's guidance, and it applies to every statement in the database. Without peeking, the optimizer builds plans using average selectivity rather than any real value, so instead of a plan that is right for some values you get a plan that is right for none of the skewed ones. Address the specific statements that suffer: literals for low-cardinality skewed predicates, separate statements per case, SQL plan baselines, or removing a histogram that does more harm than good.&lt;/p&gt;

&lt;p&gt;Bind peeking sits right at the meeting point of two other posts in this series. &lt;a href="https://uptimearchitect.com/blog/oracle-bind-variables-hard-parse/" rel="noopener noreferrer"&gt;Bind variables&lt;/a&gt; are why the cursor is shared in the first place, and that's the right default; &lt;a href="https://uptimearchitect.com/blog/oracle-indexes-help-vs-hurt/" rel="noopener noreferrer"&gt;indexes&lt;/a&gt; are why the same predicate deserves an index for one value and a full scan for another. The plan itself is something you can read in &lt;a href="https://uptimearchitect.com/blog/oracle-execution-plans-decoded/" rel="noopener noreferrer"&gt;the execution plan&lt;/a&gt;, the histogram that sets the trap comes from &lt;a href="https://uptimearchitect.com/blog/oracle-optimizer-statistics-demystified/" rel="noopener noreferrer"&gt;optimizer statistics&lt;/a&gt;, and &lt;a href="https://uptimearchitect.com/blog/oracle-sql-plan-management-baselines/" rel="noopener noreferrer"&gt;SQL plan management&lt;/a&gt; is one way to pin more than one good plan. When one statement is fast for some values and slow for others, look at the peeked bind before anything else, and prove it the way that ends the argument — with the &lt;a href="https://github.com/pyaroslav/oracle-labs/tree/main/bind-peeking" rel="noopener noreferrer"&gt;bind-peeking lab&lt;/a&gt;, where the same query costs 2,624 reads for one value and 603,354 for another until each value gets its own plan.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://uptimearchitect.com/blog/oracle-bind-peeking-adaptive-cursor-sharing/" rel="noopener noreferrer"&gt;uptimearchitect.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>oracle</category>
      <category>database</category>
      <category>performance</category>
      <category>sql</category>
    </item>
  </channel>
</rss>
