DEV Community

Lily
Lily

Posted on Originally published at dev.to

A Dead PID Held My Lock for 2 Hours: One Missing Line, Zero Output, exit 0 Every Time

For 30 straight days as a college student earning ¥100k/month, I posted to Instagram by hand, and then I burned out and stopped. Today the same job runs on a Claude Code autonomous environment, I touch nothing, and it holds up ¥1.2M/month in revenue. Except for the two hours when it quietly stopped: three consecutive launchd runs, zero pieces of content generated, last exit=0 every single time, and not one alert. The cause was a process that had already been killed, holding a lock file nobody would take away from it.

Why this setup works

From "doing the work" to "building the environment"

The problem with updating social media by hand is that it burns willpower. No matter how motivated you are, sleep, health, and mood all fluctuate. During the period when I was laid off and my income went to zero, I had no mental slack for posting at all.

The autonomous environment I spent six months building with Claude Code runs regardless of my emotional state. launchd calls a script, the script generates content with claude -p (MAX plan quota; paid APIs are off-limits), the output is queued for auto-posting, and it goes out to Instagram every day at 19:30. As long as this machinery keeps working, ¥1.2M/month in sales holds up without me lifting a finger.

The mental model I want to hand you

A lot of people think "automation = writing scripts," and that's only half right. A script is correct at the moment you write it. Given time, external dependencies break, processes die for reasons you didn't anticipate, and lock files turn into debris that blocks every future run.

An autonomous environment that actually works is one that assumes breakage and carries a layer that repairs it. The lock story here is a textbook case.

~/dev/brand-404/sns/gen_feature.py is a script launched on a schedule by launchd that auto-generates Instagram feature articles. A single run takes a long time (up to three claude -p calls, plus image generation, adding up to tens of minutes), so it has a lock mechanism to prevent the next run from overlapping with one that hasn't finished.

Mishandling that lock file (~/dev/brand-404/sns/gen_work/.lock) meant that pid 94799 held the lock without releasing it even though it had already been killed, every subsequent run was skipped with "lock held — 終了", and generation stopped for about two hours.

Why "2 hours"

The lock check has a constant LOCK_STALE_SEC = 2 * 3600 (2 hours) (line 57 of gen_feature.py). It's a safety valve: "even if the lock exists, steal it if the mtime is older than 2 hours."

But the original implementation only looked at mtime, without checking whether the pid was alive. Even with a killed pid sitting in .lock, as long as the mtime was within 2 hours it kept deciding "still running" and skipping. launchd could fire a third and a fourth time — all of them "lock held — 終了" until the two hours elapsed.

The worst possible state — automation that is "running" but produces nothing — continued silently for two hours.

What the "build the environment" side requires

With manual work, you notice: "huh, nothing got generated today," and you run it by hand. But building an autonomous environment means taking on responsibility for it continuing to work correctly while nobody is watching.

A pid liveness check looks like belt-and-suspenders, but in practice "the script gets killed and the lock stays behind" happens routinely. If you kill it instantly with SIGKILL, the release_lock() in the finally block never runs at all. Same thing when you stop it by hand during development. If you're scheduling a long-running script with launchd, a pid liveness check is mandatory.


The overall flow

System architecture

launchd (毎日定時 + 毎日19:30)
  │
  ├─ gen_feature.py(毎日定時)
  │     acquire_lock()         ← 今回の話
  │     ↓
  │     キュー残数チェック
  │     QUEUE_TARGET=3 に不足があれば
  │     ↓
  │     brand-catalog.json からブランド選定
  │     ↓
  │     Shopify /products.json 取得
  │     ↓
  │     claude -p #1: コピー生成 (copy.md)
  │     ↓
  │     claude -p #2: 画像役割選定 (images.md)
  │     ↓
  │     スライド生成 (build-post-from-json.mjs)
  │     ↓
  │     claude -p #3: セルフQA (qa.md)
  │     ↓ QAを通過
  │     content/sns/feature-XX-{slug}/ に出力
  │     release_lock()
  │
  └─ ig_autopost.py(毎日19:30)
        content/sns/feature-* のキューから1本取り出し
        ↓
        Instagram Graph API で投稿
        ↓
        state/ig_posted.jsonl に記録
Enter fullscreen mode Exit fullscreen mode

The queue targets a standing inventory of 3 items (gen_feature.py line 60, QUEUE_TARGET = 3; line 61, MAX_GEN_PER_RUN = 2). With 3 in stock, posting doesn't break even if generation fails for one or two days in a row.

What the lock is responsible for

When you manage a script with a long single run (up to CLAUDE_TIMEOUT = 600 seconds × 3 calls, plus image downloads) via scheduled launchd starts, these problems appear:

  • The next instance starts before the previous run has finished and double-generates the same brand
  • Concurrent writes to the shared brand-catalog.json corrupt it

gen_work/.lock is what prevents that. The process writes its own PID into the lock file, and on the next start the script checks whether that PID is alive before deciding whether to run.

The acquire_lock() implementation (after the fix)

Lines 225–249 of gen_feature.py are the whole thing.

LOCK_STALE_SEC = 2 * 3600       # 57行目

def acquire_lock() -> bool:
    GEN_WORK.mkdir(parents=True, exist_ok=True)
    if LOCK_FILE.exists():
        age = time.time() - LOCK_FILE.stat().st_mtime
        try:
            pid = int(LOCK_FILE.read_text(encoding="utf-8").strip())
            if pid <= 0:
                raise ValueError
        except (OSError, ValueError):
            log("stale lock 奪取: PIDが空または非数値")
        else:
            try:
                os.kill(pid, 0)          # ← プロセス生死チェック
            except ProcessLookupError:
                log(f"stale lock 奪取: pid={pid} は不在")
            except PermissionError:
                if age <= LOCK_STALE_SEC:
                    return False
                log("stale lock 奪取: mtime 2h超")
            else:
                if age <= LOCK_STALE_SEC:
                    return False
                log("stale lock 奪取: mtime 2h超")
    LOCK_FILE.write_text(str(os.getpid()), encoding="utf-8")
    return True
Enter fullscreen mode Exit fullscreen mode

The decision logic before the fix (the broken code) only looked at mtime.

# ❌ 修正前: pid の生死を見ない
def acquire_lock() -> bool:
    GEN_WORK.mkdir(parents=True, exist_ok=True)
    if LOCK_FILE.exists():
        age = time.time() - LOCK_FILE.stat().st_mtime
        if age <= LOCK_STALE_SEC:
            return False          # ← kill済みpidでも2h以内は全部ここで返る
        log("stale lock 奪取: mtime 2h超")
    LOCK_FILE.write_text(str(os.getpid()), encoding="utf-8")
    return True
Enter fullscreen mode Exit fullscreen mode

What os.kill(pid, 0) means

os.kill(pid, 0) sends signal 0 (the null signal). Signal 0 doesn't actually send anything. The kernel just checks the pid and returns: success if the process exists and you have permission to signal it, ProcessLookupError if the process doesn't exist, and PermissionError if the process exists but belongs to another user.

Using this behavior, you can safely check whether a process is alive.

Result of os.kill(pid, 0) Meaning acquire_lock's decision
No exception The process for that pid exists and is running If mtime is within 2h, treat as "running" and skip
ProcessLookupError The process for that pid is gone (killed, etc.) Stale lock — steal it immediately
PermissionError The pid exists but is owned by another user mtime fallback (steal if older than 2h)

In this incident, where pid 94799 had been killed, the fixed code would raise ProcessLookupError, immediately log "stale lock 奪取: pid=94799 は不在", and let the next runner take the lock.

The shell-side kill -0 idiom

The same pattern is used in the multi-start guard of ~/.claude/scripts/automation-health.sh (lines 20–30).

_ah_lock="${TMPDIR:-/tmp}/automation-health.lock"
if ! mkdir "$_ah_lock" 2>/dev/null; then
  if kill -0 "$(cat "$_ah_lock/pid" 2>/dev/null)" 2>/dev/null; then
    echo "automation-health: 別インスタンス稼働中のためスキップ" >&2
    exit 0
  fi
  rm -rf "$_ah_lock"
  mkdir "$_ah_lock" 2>/dev/null || { echo "lock取得失敗・スキップ" >&2; exit 0; }
fi
echo $$ > "$_ah_lock/pid"
trap 'rm -rf "$_ah_lock"' EXIT
Enter fullscreen mode Exit fullscreen mode

The comments (lines 17–20) spell out the reasoning: "--deep scans everything, so I/O piles up and load average spikes if multiple instances run. An atomic mkdir lock narrows it to one. The PID liveness check auto-steals stale locks, and trap EXIT guarantees release on both normal and abnormal termination."

The shell's kill -0 <pid> has exactly the same semantics as Python's os.kill(pid, 0). 2>/dev/null discards stderr so the "no such process" error message isn't shown to the user. If the process exists, the exit code is 0; if not, it's 1 — so the success or failure of if kill -0 ... is the liveness check.

On top of that, trap 'rm -rf "$_ah_lock"' EXIT guarantees lock release on normal exit, error exit, or signal receipt. It's the same idea as calling release_lock() from Python's try/finally.

What you see when you put the two implementations side by side

Python (gen_feature.py)          Shell (automation-health.sh)
─────────────────────            ─────────────────────────────
os.kill(pid, 0)                  kill -0 <pid>
  ProcessLookupError → 奪取        終了コード1 → 残骸 → 奪取
  PermissionError    → mtime fb    終了コード≠0→ 同上
  例外なし           → 実行中       終了コード0 → 実行中

try/finally release_lock()       trap 'rm -rf lock' EXIT
Enter fullscreen mode Exit fullscreen mode

Different languages, same three-step pattern for correct lock acquisition.

  1. Read the PID from the lock file
  2. Check whether the process is alive with kill -0 / os.kill(pid, 0)
  3. If it's gone, steal immediately; if it exists, use mtime as a secondary safety valve

mtime is strictly the last resort for when the pid check isn't usable — it must never be the primary check.

Implementation details

Combining release_lock() with try/finally

I've talked about acquire_lock(), but the correct lock pattern has its crux on the release side too.

release_lock() at lines 252–256 of gen_feature.py is simple.

def release_lock() -> None:
    try:
        LOCK_FILE.unlink()
    except FileNotFoundError:
        pass
Enter fullscreen mode Exit fullscreen mode

Swallowing FileNotFoundError is deliberate. It's a guard against crashing when the finally block and a manual call overlap, and it prevents the absurdity of "the main work raises an exception because releasing the lock failed."

What matters is the structure of the caller, main() (lines 995–1020).

if not acquire_lock():
    log("lock held(他プロセスが実行中 or 2h以内)— 終了")
    return 0

try:
    made = 0
    attempts = 0
    max_attempts = need + 3
    attempted_brands: set[str] = set()
    while made < need and attempts < max_attempts:
        result = run_pipeline(attempted_brands)
        ...
    log(f"生成完了: {made}/{need}本 試行{attempts}回 (queue残 {count_queue_remaining()}本)")
finally:
    release_lock()
Enter fullscreen mode Exit fullscreen mode

Everything after acquiring the lock is wrapped in try/finally. Even if an exception is raised in the middle of run_pipeline(), the finally block always runs, so the lock is reliably released.

There's one exception where this doesn't work: SIGKILL. If you force-terminate the process with kill -9 <pid>, the Python runtime itself dies instantly, so the finally block never runs. That was the direct cause of this incident.

The same goes for trap EXIT (automation-health.sh line 30).

trap 'rm -rf "$_ah_lock"' EXIT
Enter fullscreen mode Exit fullscreen mode

The EXIT trap runs on SIGTERM (a normal kill), normal exit, and error exit — but not on SIGKILL. In either language, lock debris from a forced termination can't be prevented by try/finally and trap EXIT alone. That's exactly why you need the liveness check via os.kill(pid, 0).

The three exception branches and the reason for each

The os.kill(pid, 0) call in acquire_lock() has three branches (lines 235–247).

try:
    os.kill(pid, 0)
except ProcessLookupError:
    log(f"stale lock 奪取: pid={pid} は不在")
except PermissionError:
    if age <= LOCK_STALE_SEC:
        return False
    log("stale lock 奪取: mtime 2h超")
else:
    if age <= LOCK_STALE_SEC:
        return False
    log("stale lock 奪取: mtime 2h超")
Enter fullscreen mode Exit fullscreen mode

ProcessLookupError is the case where the process doesn't exist. This incident (a killed pid) falls here. Steal immediately. There's no need to look at mtime.

PermissionError is the case where the pid exists but belongs to another user's process. Since you lack permission to send a signal, you can confirm existence but not whether it's "a previous instance of my own script." In that situation you have no choice but to fall back on mtime, so it leans toward "treat as running" if it's within two hours.

No exception (the else block) means the process exists and you do have permission to signal it — that is, it is definitely still running. Here too, if mtime is within two hours, it's treated as a legitimate in-progress run.

LOCK_STALE_SEC = 2 * 3600 is purely a last-resort safety valve. It only kicks in for the PermissionError case where the pid check doesn't apply, and for the else case where the process is running but is taking unexpectedly long for some reason. This connects back to what I said earlier about never making mtime the primary check. Whether two hours is the right threshold isn't the essence of the problem; doing the pid check first is.

Handling an invalid PID value

The case where the lock file exists but its contents are corrupt is handled too (lines 229–234).

try:
    pid = int(LOCK_FILE.read_text(encoding="utf-8").strip())
    if pid <= 0:
        raise ValueError
except (OSError, ValueError):
    log("stale lock 奪取: PIDが空または非数値")
Enter fullscreen mode Exit fullscreen mode

OSError is when the file couldn't be read (permission issues, etc.), and ValueError is when the int conversion fails or the pid is 0 or below. Both are treated as "undecidable = debris" and the lock is stolen.

The pid <= 0 check is validation that leans on the POSIX guarantee that a PID is always a positive integer. int("0") and int("-1") convert successfully but aren't valid values for a real process PID. If strip() on an empty file yields an empty string, int("") raises ValueError, which falls into the same path.

The atomicity of the shell's mkdir lock

automation-health.sh implements its lock with mkdir (line 22) because that's more atomic than writing a file.

if ! mkdir "$_ah_lock" 2>/dev/null; then
  if kill -0 "$(cat "$_ah_lock/pid" 2>/dev/null)" 2>/dev/null; then
    echo "automation-health: 別インスタンス稼働中のためスキップ" >&2
    exit 0
  fi
  rm -rf "$_ah_lock"
  mkdir "$_ah_lock" 2>/dev/null || { echo "lock取得失敗・スキップ" >&2; exit 0; }
fi
echo $$ > "$_ah_lock/pid"
Enter fullscreen mode Exit fullscreen mode

mkdir is atomic at the kernel level. If the directory doesn't exist it's created and succeeds; if it exists it fails. Even if two processes call mkdir simultaneously, the kernel lets only one of them succeed.

Python's write_text() carries no such guarantee. There is a theoretically possible race where A writes first and B overwrites immediately after. In the case of gen_feature.py, launchd starts it on an interval so the probability of concurrent execution is extremely low — but not zero. If you want the same atomicity in Python, one approach is os.mkdir(). The reason gen_feature.py deliberately doesn't use that approach is that the high-frequency problem for this script wasn't "simultaneous start races" but "killed pids left behind." The simple implementation was chosen to match the priority of the problem that actually needed solving.


Where I got stuck

Symptom: nothing is generated, but there's no "error" either

The first sign of trouble was in my Discord #brand-404 channel. The "🆕 o81 IG特集生成: ..." notification that had arrived daily up to the previous day didn't come.

Nothing came into the Discord alert channel either. The symptom wasn't "an error occurred" — it was "nothing happened."

Thinking the launchd job had failed, I ran automation-health.sh, and com.lily.gen-feature showed "loaded / last exit=0". exit 0 is a normal exit. But no generation was coming.

Digging through the logs, I found three hours' worth of these lines.

[gen] lock held(他プロセスが実行中 or 2h以内)— 終了
Enter fullscreen mode Exit fullscreen mode

All three launchd starts ended right there. Someone was holding the lock.

Root cause: I looked at the lock file directly

cat-ing ~/dev/brand-404/sns/gen_work/.lock gave me:

94799
Enter fullscreen mode Exit fullscreen mode

pid 94799 was written in it. ps aux | grep 94799 returned nothing. Not a live process.

Checking the mtime with ls -la ~/dev/brand-404/sns/gen_work/.lock showed a timestamp about 1.5 hours old. Against LOCK_STALE_SEC = 2 * 3600 (2 hours), that was still 0.5 hours short. That's why it kept deciding "within 2h = running" and skipping.

pid 94799 had died because, during debugging the night before, I ran kill -9 $(cat ~/dev/brand-404/sns/gen_work/.lock) in another terminal. Normally release_lock() would run and .lock would disappear. But -9 (SIGKILL) skips the finally block. The lock stayed behind and swallowed every launchd start the next morning.

The first, wrong fix: I shortened LOCK_STALE_SEC

The first thing I thought of was "2 hours is too long, let's make it 30 minutes."

I actually changed it to LOCK_STALE_SEC = 30 * 60 and committed. That was wrong.

The problem wasn't that 2 hours is long — it was that a killed pid was being judged alive. Even at 30 minutes, the same problem persists for 30 minutes after the kill. Worse, it introduces the risk of stealing the lock in the middle of a legitimate long run (three claude -p calls = up to CLAUDE_TIMEOUT = 600 seconds × 3, plus image downloads).

The correct fix is "check whether the pid is alive before looking at mtime," not changing the mtime threshold. Tuning the threshold isn't a cure; it just softens the symptom. I reverted that commit and switched to the current approach of checking os.kill(pid, 0) first.

The second snag: I missed PermissionError

After putting the first fixed version — which only caught ProcessLookupError — into production, I got stuck again during testing.

I ran sudo python3 ~/dev/brand-404/sns/gen_feature.py --dry-run once for verification purposes, and when I then ran it as a normal user, the leftover-lock symptom appeared again.

A process run under sudo is owned by root. Calling os.kill(root_pid, 0) as a normal user returns PermissionError. My first fix only looked at ProcessLookupError, so PermissionError propagated out and produced a stack trace.

# ❌ PermissionError を見落とした中間バージョン
try:
    os.kill(pid, 0)
except ProcessLookupError:
    log(f"stale lock 奪取: pid={pid} は不在")
# PermissionError は捕まえておらず、外へ伝播する
else:
    if age <= LOCK_STALE_SEC:
        return False
Enter fullscreen mode Exit fullscreen mode

In Python's try/except/else structure, else runs when no exception at all was raised in the try block. Since PermissionError wasn't caught, that case entered neither except ProcessLookupError nor else, and the exception propagated straight to the caller. In the end I caught PermissionError explicitly and routed it to the mtime fallback, arriving at the current form (lines 241–243).

except PermissionError:
    if age <= LOCK_STALE_SEC:
        return False
    log("stale lock 奪取: mtime 2h超")
Enter fullscreen mode Exit fullscreen mode

Lesson: "exited normally" is not "worked correctly"

What this burned into me hardest is that exit 0 and "generation completed" are different things.

log("lock held — 終了"); return 0 is exit 0. The launchd job history records it as "success." automation-health.sh returns green too. No Discord alert arrives. From the outside everything is fine — and yet the actual goal, content generation, didn't happen at all.

This kind of silent failure can only be detected if you design monitoring around "did the intended side effect occur?" rather than "did the process die?"

I now have separate monitoring that fires a Discord alert if the remaining queue count (count_queue_remaining()) stays at zero for a certain period. Process health checks and output health checks need to be designed separately. Making an autonomous environment "designed on the assumption of breakage" includes monitoring, in a separate layer, not only whether the process is alive but whether the environment keeps producing the output it's supposed to.

Gotchas

The earlier sections covered "the design flaw of only looking at mtime," "the fix via os.kill(pid, 0)," and "missing PermissionError." Here I'll add the points I actually got stuck on while running scripts with this same structure.

PID reuse causes a false "alive" verdict

PIDs on Linux/macOS are finite and wrap around. On macOS they cycle around a maximum near 99999. In a long-running environment, it's possible that the PID of a process you kill -9'd yesterday is being used by an unrelated process today. Call os.kill(pid, 0) in that state and you get "the process is alive." gen_feature.py currently writes only the PID into .lock (line 248, LOCK_FILE.write_text(str(os.getpid()), ...)), but the combination of launchd running it once a day and LOCK_STALE_SEC = 2 * 3600 (line 57) means there's virtually no real harm. For scripts invoked at high frequency, consider writing the PID paired with a start timestamp and treating it as "running" only when both match.

Forget GEN_WORK.mkdir(parents=True, exist_ok=True) and the first run dies instantly

Line 226 of acquire_lock() starts with GEN_WORK.mkdir(parents=True, exist_ok=True). Without that one line, on a first start where the gen_work/ directory doesn't exist, an operation before LOCK_FILE.exists() raises FileNotFoundError. launchd starts the script, it dies immediately, and you're left with last exit=1 and nothing running. Keep the order: create before releasing.

write_text() is not atomic (a theoretical race exists)

With Python's Path.write_text(), other processes can potentially read the mid-write state. If the next instance calls read_text() before the LOCK_FILE.write_text(str(os.getpid()), ...) write completes, it reads an empty string, int("") raises ValueError → the lock is stolen as debris → two instances run simultaneously. That race is theoretically valid. Since gen_feature.py is started intermittently by launchd, simultaneous starts essentially don't happen. But for a high-frequency script called every minute, use an atomic write with os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY). This is exactly why automation-health.sh implements its lock with mkdir (line 22, if ! mkdir "$_ah_lock" 2>/dev/null;).

SIGKILL isn't the only thing that skips finally

The middle section covered how SIGKILL (kill -9) skips try/finally. One more thing to watch for is a C extension module calling os._exit() internally. sys.exit() raises SystemExit, so finally runs; os._exit() terminates the process along with the runtime immediately, so finally does not run. The more you depend on external libraries, the higher the chance that finally won't run. Designing the pid liveness check on the assumption that finally won't run is the only fundamental countermeasure.

Building lock file paths from relative paths makes them environment-dependent

If the launchd plist has a WorkingDirectory key set, the script's starting directory changes to it. Build the lock file path from relative paths and the lock file ends up in a different place depending on the plist's WorkingDirectory. gen_feature.py builds every path as an absolute path from ROOT = Path(__file__).resolve().parent.parent (line 36). Scripts that run as launchd jobs should use __file__-based absolute paths.

_draft-feature-XX-{slug} directories accumulate

On QA failure or slide-generation failure, run_pipeline() leaves the _draft-feature-XX-{slug} directory in place instead of deleting it. That's an intentional design decision so failed content can be inspected later (lines 954–959). count_queue_remaining() excludes anything with the _draft- prefix from its count (lines 203–218), so inventory calculation isn't affected. But if they keep piling up, content/sns/ fills with junk. A monthly cleanup with find content/sns -name '_draft-*' -mtime +30 -type d is needed.

Interpreting exit 0 as "normal" lets silent failures slip through

The main() function also ends with return 0 when lock acquisition fails (lines 996–997). launchd records that as a normal exit, and the launchd check in automation-health.sh displays last exit=0 as . "exit 0 because the queue was stocked and no generation was needed" and "exit 0 because lock acquisition failed and nothing was done" look identical to launchd. That's why the actual two-hour stoppage was invisible.

The trap of the "no error means everything's fine" design philosophy

Discord alerts only arrive for failure cases where discord_alert() is called (lines 140–142). A failed lock acquisition is a skip, not an error, so no alert arrives. If you design automation monitoring purely around "notify me when something bad happens," silent skips will never be detected. What made me notice this incident in the first place was the absence of the success notification "🆕 o81 IG特集生成: ..." (discord_post_review(), line 973) in the #brand-404 channel.

Discord's default User-Agent gets blocked by Cloudflare

As the comment in _discord_post() notes (lines 93–94), Cloudflare rejects the Python-urllib/x.y User-Agent that Python's urllib sends by default with 403/1010. It's worked around by explicitly setting ua_header = "lily-o81-gen/1.0". If an automation script hits an external service with the default settings of urllib or the requests library, it can get blocked without warning.

Misjudging the balance between curl timeouts and retries

fetch_products() retries Shopify's products.json with attempts=3 (lines 503–522). The design of waiting 15 seconds and retrying on failure comes from real damage: "some stores temporarily return empty results under consecutive access" (Beyond The Vines, 2026-07-26). Make the timeout extremely short and you get "a brand that actually has products is judged to have 0 and demoted to skip." Make it too long and a single failure stalls things for minutes. Scripts that hit external APIs should design three things as a set: the per-attempt timeout, how many times to retry, and the wait between retries.


Best practices

Building on the earlier sections and the gotchas, here are the practices for lock management in long-running launchd scripts, extracted from real code.

1. Fix the pid liveness check as the primary decision and mtime as the secondary one

try:
    os.kill(pid, 0)
except ProcessLookupError:
    pass  # 即奪取
except PermissionError:
    if age <= LOCK_STALE_SEC:
        return False  # mtime fallback
else:
    if age <= LOCK_STALE_SEC:
        return False  # mtime fallback
Enter fullscreen mode Exit fullscreen mode

The shell version has the same semantics with kill -0 "$(cat pid_file)" (automation-health.sh line 23).

2. Always implement the three exception branches as a set

If you catch only ProcessLookupError, PermissionError propagates out and crashes the script. Write all three as a set. Miss even one and you get "a bug specific to that case."

3. Always wrap lock release in try/finally

if not acquire_lock():
    return 0

try:
    # メイン処理
finally:
    release_lock()
Enter fullscreen mode Exit fullscreen mode

In shell, trap 'rm -rf "$lock"' EXIT is the equivalent. It releases on normal exit, on an exception, and on sys.exit().

4. Have release_lock() swallow FileNotFoundError

It's a guard so a second unlink() doesn't crash when finally and a manual call overlap. It prevents the absurdity of "the main work raises an exception because releasing the lock failed" (the pattern at lines 252–256).

5. Derive LOCK_STALE_SEC from the maximum run time

The longest run of gen_feature.py is CLAUDE_TIMEOUT = 600 seconds (line 66) × up to 3 calls + image downloads, roughly 35–40 minutes. LOCK_STALE_SEC = 2 * 3600 (line 57) gives plenty of margin against that. Shortening the threshold does not solve the "a killed pid releases the lock within 2h" problem. Threshold tuning is symptomatic treatment; the pid liveness check is the cure.

6. Build the lock file path as an absolute path based on Path(__file__).resolve()

An implementation that doesn't depend on launchd's WorkingDirectory is mandatory. The pattern of building every path from ROOT = Path(__file__).resolve().parent.parent (line 36) in gen_feature.py works as a template as-is.

7. Always run GEN_WORK.mkdir(parents=True, exist_ok=True) before acquiring the lock

It's required so the first start doesn't crash when the directory doesn't exist. Put it at the top of acquire_lock() (line 226).

8. In shell, ensure atomicity with a mkdir lock

mkdir is atomic at the kernel level. Even if two processes call it simultaneously, only one succeeds (automation-health.sh lines 22–27). If you need atomicity in Python too, use os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY).

9. Design monitoring around "the intended output," not the process

The launchd check in automation-health.sh treats last exit=0 as normal, but a lock-acquisition skip is also exit 0. The right answer is positive monitoring: alert if the success notification "content generation completed" (discord_post_review(), line 973) doesn't arrive within a certain window.

10. Have success notifications (don't rely on error notifications alone)

A design that only notifies on errors misses silent skips. The design where the absence of the "🆕 o81 IG特集生成: ..." success notification signals a problem was the only signal that made me notice this failure. Having a "something good happened" notification is what makes anomaly detection effective.

11. Keep several days' worth of queue inventory

The combination of QUEUE_TARGET = 3 (line 60) and MAX_GEN_PER_RUN = 2 (line 61) is an inventory design where "posting doesn't break even if generation fails one or two days in a row." Even if a lock failure stops things for a day, 3 items in stock keep the posting going. Having a buffer eliminates the single point of failure in automation.

12. Make it possible to check state without taking the lock, via a --dry-run option

gen_feature.py --dry-run (lines 981–993) shows only the remaining queue count without taking the lock. It's safe to call even while production is running. During incident investigation you can check state without worrying about "going to grab the lock and causing contention."

13. Make it possible to run E2E tests against production logic with a --force option

Normally it only generates until QUEUE_TARGET is met, but --force generates one item regardless of the remaining queue count (lines 980–984). You can test the whole pipeline without changing production logic, and use it to verify a fix.

14. Always set an explicit User-Agent on urllib calls to external APIs

Python's default Python-urllib/x.y gets blocked by Cloudflare and some CDNs. Specify an identifiable string like "lily-o81-gen/1.0". It's mandatory when hitting Discord, Shopify, or any other API behind Cloudflare.


Summary

pid 94799 occupied the lock for two hours because of one missing line. The check that confirms whether a process is alive with os.kill(pid, 0) was absent, so a killed pid kept being misjudged as "still running." The fix itself is a few lines.

But what really needs to be understood is the "why." SIGKILL skips finally. launchd's exit 0 doesn't mean "it worked correctly," only "the process terminated." Monitoring should be designed around "is the intended output occurring?" rather than process liveness. Those three points are the substance of what it means to make an autonomous environment "designed on the assumption of breakage."

The first, wrong fix (shrinking LOCK_STALE_SEC from 2 hours to 30 minutes) was a textbook mistake of trying to soften the symptom while ignoring the root cause. Shrinking it to 30 minutes doesn't solve "a killed pid releases the lock within 2h." If anything, it creates the risk of stealing the lock from a legitimate long run. For a script that can take more than 30 minutes at CLAUDE_TIMEOUT = 600 seconds (line 66) × 3 calls, a 30-minute threshold is too short.

Building an autonomous environment means taking on responsibility for it continuing to work correctly while nobody is watching. The social media I updated by hand every day back when I was a college student earning ¥100k/month now runs on a launchd + Claude Code autonomous environment. What sustains ¥1.2M/month in sales isn't the content itself — it's the accumulated fixes I've stacked into "a design that doesn't stop." This pid liveness check is one of them.


The full picture of the system, the breakdown of the ¥1.2M/month, and the 30-day playbook are collected in a paid note article
📕 Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート


Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Top comments (0)