The Worker Kept Hitting the Old Database After the Config Was Changed

The failure I keep on a throwaway server has no traceback. The API reports the new database, the background job writes to the old database, and every status check still says green. It is the kind of bug that can survive a code review because the diff is correct but the process is stale.

This is a controlled reproduction, not a production incident report, so the database URLs are fake and the services are intentionally tiny. I am walking through it because the debugging sequence matters more than the specific stack.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option let me run this two-process reproduction in a throwaway environment instead of guessing from a laptop.

The misleading symptom

The first clue was that the API and a one-off script reported different database targets. A fresh python api.py printed postgres://new-db/app, but a scheduled job that had been running for hours kept appending postgres://old-db/app to its log. No error appeared, and the job exit code remained zero, so the usual symptoms were absent.

That gap is what made the failure instructive. A config diff would have shown the new URL, a lint check would have passed, and a manual API request would have returned the expected result. The broken path was hidden inside a process that no one thought to restart.

Before reaching for a fix, I wrote down the observation that any successful explanation had to satisfy. The API must see the new URL, the worker must see the old URL, and both must use the same config function.

A two-process reproduction

The reproduction uses three tiny files. The config function reads the environment at call time, so the bug is not caused by import-time caching.

# config.py
import os

def get_database_url():
    return os.getenv('DATABASE_URL', 'postgres://local/app')

# api.py
from config import get_database_url

print(f'api sees {get_database_url()}')

# worker.py
import signal
import time
from config import get_database_url

def run_job(signum, frame):
    with open('/tmp/jobs.log', 'a') as f:
        f.write(f'{time.time():.0f} {get_database_url()}\n')

signal.signal(signal.SIGUSR1, run_job)
print('worker ready', flush=True)
while True:
    time.sleep(1)

Start the worker with the old value, then run the API with the new value, and finally signal the still-running worker.

DATABASE_URL=postgres://old-db/app python worker.py &
echo $! > worker.pid

DATABASE_URL=postgres://new-db/app python api.py
# api sees postgres://new-db/app

kill -USR1 $(cat worker.pid)
cat /tmp/jobs.log
# 1750000000 postgres://old-db/app

The worker is not reading a stale config file or a cached module attribute. It is running with a different environment than the freshly launched API process.

What the debugger showed

On Linux, /proc/<pid>/environ exposes the environment that a process received when it was started. A quick check made the root cause visible instead of theoretical.

cat /proc/$(cat worker.pid)/environ | tr '\0' '\n' | grep '^DATABASE_URL='
# DATABASE_URL=postgres://old-db/app

Compare that with the process start time. If the worker began before the deployment changed the environment, the mismatch is easy to confirm.

ps -o pid,lstart,cmd -p $(cat worker.pid)

This evidence changed the question. The problem was not whether the configuration was correct, but whether every running process had received the corrected environment.

Why the model's first answer was wrong

I fed the symptom and the two-process code into MonkeyCode's free model access as a debugging partner. The first suggestion was plausible: check the config file and the connection string. It was also insufficient, because a config diff would not explain why the API worked while the worker did not.

That is the trap with any fast model answer. A model will often return the most common cause, not the cause that is consistent with all the evidence. The config file did contain the new URL, yet the worker still wrote to the old database, so the model's first hypothesis failed the test I had written down before asking for help.

I used the free server option to rerun the reproduction after each suggestion. When the model proposed a config cache, I changed the function to read the environment at call time and showed the same result. When it proposed a stale file handle, the log append still worked. The only explanation that survived was the worker's process environment.

The fix and the guardrail

The durable fix in the reproduction is not a code change. It is a process lifecycle change: terminate the worker and start a new one with the updated environment.

kill $(cat worker.pid)
DATABASE_URL=postgres://new-db/app python worker.py &
echo $! > worker.pid
kill -USR1 $(cat worker.pid)
cat /tmp/jobs.log
# 1750000120 postgres://new-db/app

In a real deployment, that may mean restarting the worker service, recreating the container, or rolling the process group after a config change. The key guardrail is to compare process start time with the deployment time before declaring the change complete.

A simple decision table helps separate the likely causes.

Symptom	Likely cause	Fast verification	Fix
Fresh process sees new URL	Deployment changed the environment	Run api.py once	Start and request normally
Running worker sees old URL	Worker process started before the change	ps -o lstart plus /proc/<pid>/environ	Restart the worker
Both processes see old URL	Config source was not actually updated	Diff the config file	Correct the source, then restart
Both see new URL but job still fails	A different bug lives in the job path	Check job logs and return codes	Debug the job itself

What I would repeat

The reusable technique is to require every hypothesis to explain both observations at the same time. A config diff explained one observation and ignored the other, so it was not a complete root cause. The environment check explained both observations with one mechanism.

That discipline matters when using a free model. A model can generate many plausible hypotheses quickly, but it cannot inspect a running process unless I give it the right evidence. My job was to collect the process start time and environment dump, feed those back into the conversation, and reject any suggestion that contradicted them.

Limitations and who should skip this

This approach assumes a Linux host or a container where /proc/<pid>/environ is available. On macOS or Windows, the same evidence must come from a process inspector instead of a direct filesystem read.

A free model can be rate-limited, may not know your exact orchestration layer, and will occasionally give confident but wrong process advice. The reproduction is also only a teaching scaffold: do not treat it as a substitute for a controlled incident response procedure on a production system.

Skip this workflow if your team already has a mandatory restart policy after every environment change. The bug will not reproduce there because the stale process never survives long enough to hide the mismatch.

Keep a small two-process script like this for the next time a successful deploy leaves one code path behind. If a quick answer points only at the config file, ask it to explain why the other process was not affected.