DEV Community: Lily

5 Safety Guards for Auto-Archiving Another AI Agent's Conversations into My Vault: Designing a Manus API Bridge

Lily — Sun, 27 Sep 2026 00:00:03 +0000

My Claude Code conversations were already saved to my knowledge vault automatically, but anything I gave to Manus stayed inside its browser UI. I couldn't search those tasks or find them later. Now a small Python bridge checks the Manus API every 5 minutes and saves each task to the same shelf as my Claude Code logs. It uses five safety guards to keep it quiet, self-healing and free of secrets.

Last time, I wrote about how a weekly batch job in my skills CLI had been failing silently. This post continues with the same vault setup. It covers how I automatically collect conversations from an AI agent other than Claude Code (Manus) through its API and store them as primary sources in a shared vault.

The problem: Manus tasks weren't part of the primary sources

The vault has a location called raw/conversations/, and Claude Code conversation logs pile up there automatically. Tasks I sent to Manus were different. The only way to see their contents was to open the task screen in the browser. When I later tried to remember what I had asked for in a task, it didn't show up in vault search or in the shared hot cache.

Manus has a public API that lets you read your own task list and messages. That was the starting point for this bridge: fetch those tasks on a schedule and put them on the same shelf (the raw layer) as the Claude Code conversation logs. launchd runs ~/Documents/claude-obsidian/bin/manus-task-archive.py every 5 minutes, and the script writes its output to ~/Documents/my-knowledge-base/raw/manus-conversations/.

Note
This script only fetches and saves. A separate, safe reflection worker handles summaries and "proposals to the vault." This script never does that work. A core rule of how I run the vault is to keep raw primary sources separate from the layer that interprets them.

Design: credentials live only in the Keychain

I don't want the API key in the code or in any file, so the script reads it from the macOS Keychain on every run.

KEYCHAIN_SERVICE = "manus-vault-memory-archiver"

def keychain_key() -> str:
    result = subprocess.run(
        ["security", "find-generic-password", "-s", KEYCHAIN_SERVICE, "-w"],
        text=True,
        capture_output=True,
        check=False,
    )
    if result.returncode != 0 or not result.stdout.strip():
        raise RuntimeError("Manus archive API key is unavailable in macOS Keychain")
    return result.stdout.strip()

The docstring also states that this worker never passes the key to print or write. If the key can't be fetched, the function raises an exception. The except in main() logs it and the run stops. A broken key doesn't affect other jobs or conversations.

Incremental sync: only process tasks whose revision changed

The script fetches Manus tasks by paging through task.list → task.listMessages. Fetching every message on every run would be wasteful. Instead, the script saves each task's update time as its "revision" in a state file and skips the task if that value hasn't changed.

def task_revision(task: dict[str, Any]) -> str:
    for key in ("updated_at", "updatedAt", "modified_at", "timestamp", "created_at", "createdAt"):
        if task.get(key) not in (None, ""):
            return str(task[key])
    return "unknown"

for task in accessible:
    ident = task_id(task)
    if not ident:
        continue
    revision = task_revision(task)
    if state["tasks"].get(ident) == revision:
        continue
    messages = list_messages(credential, ident)
    archive_task(task, messages)
    state["tasks"][ident] = revision

Here is the actual state file (AI/.runtime/manus-task-archive-state.json):

{
  "tasks": {
    "rRkPkzEkUrImaYRXsI3sLY": "1790174319",
    "7hBaAc4dhs9hycZaAnC8z5": "1789628528",
    "5VXpxPKZkzify3yp9FA5LX": "1790077323"
  },
  "accessible_task_count": 3
}

The launchd log (~/.claude/logs/manus-task-archive.log) shows that even though the job runs every 5 minutes, most runs find nothing new:

[2026-09-26 07:43:32 +0900] ok: accessible=3 updated=0
[2026-09-26 07:48:33 +0900] ok: accessible=3 updated=0
[2026-09-26 07:53:34 +0900] ok: accessible=3 updated=0
[2026-09-26 07:58:51 +0900] ok: accessible=3 updated=0

A run of accessible=3 updated=0 lines means the API returns 3 tasks each time, but none of their revisions changed, so nothing gets written. Of all my measurements, this was the clearest proof that the diff check works correctly.

Self-healing: stale locks, and leaving nothing behind after a crash

The launchd StartInterval is fixed at 300 seconds (5 minutes), set in ~/Library/LaunchAgents/com.shun.manus-vault-archive.plist.

<key>StartInterval</key>
<integer>300</integer>
<key>ThrottleInterval</key>
<integer>60</integer>
<key>LowPriorityIO</key>
<true/>
<key>Nice</key>
<integer>12</integer>

Overlapping runs on a 5-minute schedule would cause trouble, so the script uses a lock directory created with mkdir to prevent more than one run at a time. The case I paid attention to was a previous run crashing and leaving its lock behind.

def acquire_lock() -> bool:
    STATE_DIR.mkdir(parents=True, exist_ok=True)
    if LOCK_DIR.exists():
        try:
            age = time.time() - LOCK_DIR.stat().st_mtime
            if age > 1800:
                LOCK_DIR.rmdir()
        except OSError:
            pass
    try:
        LOCK_DIR.mkdir()
        return True
    except FileExistsError:
        return False

If a lock is older than 30 minutes (1800 seconds, or six 5-minute cycles), the next run removes it. The system recovers on a later cycle before a human notices and runs rm. Releasing the lock also sits in a finally block, so the lock is always removed, whether the run crashes with an exception or exits normally.

The log shows this working in practice:

[2026-09-26 06:42:45 +0900] ok: accessible=3 updated=0
[2026-09-26 06:48:15 +0900] error: RuntimeError: API request failed: <urlopen error [Errno 8] nodename nor servname provided, or not known>
[2026-09-26 06:53:16 +0900] ok: accessible=3 updated=0

At 06:48, DNS resolution failed briefly and the run crashed with an exception. The finally block had already released the lock, so the next run 5 minutes later (06:53) finished normally with nothing blocking it. The lock design was meant to do exactly this: fix itself before a human notices.

Redacting secrets and meeting links

This raw layer becomes "primary source material for future AI agents to read." To keep secrets out of it, the script scrubs them with regular expressions.

SECRET_PATTERNS = [
    (re.compile(r"\bsk-[A-Za-z0-9_\-]{16,}\b"), "[REDACTED_API_KEY]"),
    (re.compile(r"(?i)\b(?:api[_ -]?key|token|password|secret|bearer)\s*[:=]\s*[^\s`'\"]{8,}"), "[REDACTED_SECRET]"),
    (re.compile(r"https?://(?:meet\.google\.com|zoom\.us/j|teams\.microsoft\.com)/[^\s)\]>]+", re.I), "[REDACTED_MEETING_LINK]"),
]

The comment in the code explains the design intent:

# Deliberately conservative. The archive is a private raw source, but it must
# not become an accidental secret store shared with future agents.

The important part is the order of reasoning. I didn't start from "this raw store is personal, so it can be a little loose." I started from "future agents will read it, so keep it conservative." Zoom, Meet and Teams meeting URLs get the same treatment as API key patterns. Conversation logs often contain lines like "join from this link," and I don't want those saved permanently in the primary sources.

Backdating mtime: keeping the index timeline intact

Each archived Markdown file's modification time (mtime) is set to the time the task was last updated in Manus, not the time it was archived.

output = RAW_DIR / f"manus_{ident}.md"
atomic_write(output, "\n".join(parts).rstrip() + "\n")
stamp = updated.timestamp()
os.utime(output, (stamp, stamp))

Without this, a task that finished 9 days ago but was archived for the first time today would appear as "something that just happened" in the shared hot cache and other indexes that sort by mtime. In fact, one of the files from the first archive run had an update date of 9/17:

# 会話ログ: Manus / Agent Manus / 7hBaAc4d
日時: 2026-09-17 07:02 UTC
更新: 2026-09-17 07:02 UTC
Task ID: `7hBaAc4dhs9hycZaAnC8z5`
Status: `waiting`

The file was written today, but its mtime stays at 9/17, so it doesn't appear in the list of "today's events." Writes also go through atomic_write, which writes a temp file and then calls os.replace. If the process dies partway through, no half-written file is left behind.

After writing, if at least one task was updated, the script also runs the shared hot-cache update script:

if updated_files and UPDATE_HOT.exists():
    subprocess.run([sys.executable, str(UPDATE_HOT)], check=False,
                    stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

This way, Manus tasks reach the vault's "recent sessions" list through the same path as the other raw conversation logs.

The 5 safety guards at a glance

Guard	What it protects against	Implementation
Keychain-only credential	Key leaks	Call `security find-generic-password` on every run; never write the key in code
Incremental sync by revision diff	Wasted API calls and duplicate writes	Compare `state["tasks"][ident]` with the revision
Stale-lock self-healing	Permanent skips after a crash	Automatically `rmdir` a lock dir older than 1800 seconds on the next run
Secret / meeting-link redaction	Secrets leaking into primary sources	Replace with 3 regex patterns
mtime backdating	Breaking the index timeline	Pin the file to the task's update time with `os.utime`

Pitfalls I hit

Key names for task IDs and revisions vary across API responses → I made the implementation defensive: task.get("task_id") or task.get("id"), and for revisions, check keys in order from updated_at through createdAt.
Message paging might never end → I added a safety valve that raises RuntimeError once paging goes past MAX_MESSAGE_PAGES (100). I chose "fail with an error I'll notice" over an infinite loop.
Brief network outages surface as exceptions → The log shows a real urlopen error failure (2026-09-26 06:48). Because lock release is in finally, the next 5-minute cycle recovered normally with no special handling.
Leaving mtime as "now" breaks time-ordered indexes → Until I backdated files to the task's update time, old tasks kept showing up as "today's events."
Loose redaction would cause problems once future agents read the archive → As the code comment says, I made the patterns conservative, assuming this private raw source will be read by other agents in the future.

Summary

The goal is to put Manus tasks on the same raw/ shelf as Claude Code conversation logs, as primary sources.
Credentials are read from the Keychain on every run and never stored in code or logs.
Changes are detected only by comparing revision strings. accessible=3 updated=0 is the normal, most common result.
A stale lock is removed automatically with rmdir, so duplicate runs and post-crash lockups get fixed without waiting for a human.
Secrets and meeting links are redacted with regular expressions, and the patterns are kept conservative because future agents will read the archive.
Each file's mtime is set to the time of the original event, not the fetch time, so the vault's timeline indexes stay intact.

Next time, I plan to write about how the reflection worker safely summarizes the Manus conversations in this raw layer and promotes them into the wiki.

If you archive conversations from more than one AI agent, how do you keep secrets and outdated timestamps out of the shared record?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

3 Weekly Runs, 3 Failures: My "Fixed" Claude Code Skill Auto-Update Failed Silently for 3 Weeks

Lily — Sat, 26 Sep 2026 00:00:08 +0000

The fix had comments. It had logs. It even had an audit note saying "measured 4m00s, exit 0." It still failed every Sunday for three weeks straight, and nobody noticed.

Last time, I wrote about handling external drive mounts and file identity. This one is less flashy but harder to deal with. It's about an automation I thought I had fixed that kept failing silently for three weeks, while the comments and logs saying it was fixed stayed in place.

The job itself is boring. It's npx skills@latest update -g, the CLI that updates Claude Code skills, run weekly by launchd. I had stacked three layers on top of it: a fix for unauthenticated GitHub API rate limits, a guard against hangs, and failure detection. After that I assumed it was fixed. Then I opened the actual logs, and the last three Sunday runs had all failed for the same reason.

The problem: `npx skills@latest update -g` is fragile against the unauthenticated GitHub API

~/Library/LaunchAgents/com.lily.skills-update.plist is a plist whose only job is to launch the skill update script every Sunday at 4:40.

<key>Label</key><string>com.lily.skills-update</string>
<key>ProgramArguments</key><array><string>/bin/zsh</string><string>-lc</string><string>~/.claude/scripts/skills-auto-update.sh</string></array>
<key>StartCalendarInterval</key><dict><key>Weekday</key><integer>0</integer><key>Hour</key><integer>4</integer><key>Minute</key><integer>40</integer></dict>

The skills-auto-update.sh it calls basically just runs npx skills@latest update -g --yes. The catch is that this CLI routinely hits GitHub's API rate limit when unauthenticated, and depending on network conditions, its fetch can hang. The more skills you have, the more often this happens. Leave it alone and one day you notice nothing has updated in months.

Design: three layers of self-healing

To handle this, the script has three defenses. Here are the relevant parts of the actual file.

# skills CLI は未認証で GitHub API を叩くため、スキル数が増えるとレート上限に当たる。
# gh は認証済み(5000/h)なのでトークンを渡す。取れなくても従来どおり動く。
if command -v gh >/dev/null 2>&1; then
  GH_TOKEN="$(gh auth token 2>/dev/null)"
  if [[ -n "$GH_TOKEN" ]]; then export GH_TOKEN GITHUB_TOKEN="$GH_TOKEN"; fi
fi

  # timeout 上限: npx の network fetch がハングするのを gtimeout で頭打ち（async hook なので非ブロッキング）。
  # 45s だった頃はスキル数が増えて毎回タイムアウトし、5,833行の失敗ログを吐きながら
  # 一度も更新できていなかった（2026-08-08 監査で実測4分00秒・exit 0 を確認）。
  # STAMP は成功時のみ更新する。旧実装は失敗でも touch していたため、
  # ネットワーク不調時に「更新が静かに失敗し続ける」状態になっていた（2026-06-11 監査）。
  if command -v gtimeout >/dev/null 2>&1; then
    if gtimeout 600 "$NPXBIN" skills@latest update -g --yes >/dev/null 2>>"$ERRLOG"; then
      touch "$STAMP"
    else
      echo "[$(date '+%Y-%m-%d %H:%M:%S')] skills update failed (timeout or error) — STAMP not updated, will retry next session" >> "$ERRLOG"
    fi

The three layers work like this:

Pass gh auth token: moves from unauthenticated (60/h) to authenticated (5000/h) to avoid the rate limit.
gtimeout 600: 45 seconds had already proven too short, and runs stalled more as the skill count grew, so the job is capped at 10 minutes.
Update STAMP only on success: the old implementation ran touch even on failure, so "last run" kept advancing when runs failed and problems stayed invisible. To fix this, STAMP is now updated only on success, and failures are written to skills-update.err with a reason.

According to the comments, the 2026-08-08 audit confirmed "measured 4m00s, exit 0," so this design was verified to work at least once.

Note
It's good design that the code comments record why each fix was made (the failed 45-second timeout, the 5,833-line log, the 2026-08-08 measurement). But those are only facts verified at that point in time. They don't prove the job keeps working afterward. Forget that difference and you'll read the comments and assume it's fixed.

Reading the real logs: "timeout or error" three weeks in a row

That's the design intent. So what actually happened? Here is ~/.claude/logs/skills-update.err:

[2026-09-06 04:40:38] skills update failed (timeout or error) — STAMP not updated, will retry next session
npm warn exec The following package was not found and will be installed: skills@1.5.26
[2026-09-13 05:27:06] skills update failed (timeout or error) — STAMP not updated, will retry next session
npm warn exec The following package was not found and will be installed: skills@1.7.0
[2026-09-20 04:41:35] skills update failed (timeout or error) — STAMP not updated, will retry next session

9/6, 9/13, 9/20: all Sundays, all around the plist's start time (about 4:40). So for these three weeks, the Sunday job started every time, failed every time, and logged the same reason every time. The npm warn exec lines in between show that npx's own bootstrap (fetching skills@1.5.26 → skills@1.7.0) was getting through. So the failure isn't before npx launches. It's further in, in skills update itself, exactly where you'd suspect rate limiting or a network hang.

A contradiction in the STAMP file

Now another real file: ~/.claude/skills-last-update, which should only be updated on success.

birth: Aug  8 11:54:13 2026 / mtime: Sep 10 17:02:10 2026

It was created on 2026-08-08, the same date as the audit behind the "measured 4m00s, exit 0" design comment. That part adds up.

The mtime is the problem: 2026-09-10 17:02. That doesn't match any slot in the plist's schedule (every Sunday at 4:40). It's not a Sunday, and it's not the morning. So the run that last updated this STAMP most likely did not come from launchd's weekly automated run.

Every automated run recorded after that (9/6, 9/13, 9/20) failed. So for at least these three weeks, the automated path alone never succeeded once. The STAMP had stopped moving three weeks earlier, and nobody went to look.

Why the "fix" didn't fix anything

This is the part I most wanted to write about. Each of the three self-healing layers is working as intended.

gtimeout 600 works: the exit code came back within 600 seconds and the script fell into the else branch. If it were truly hanging, the timing of the [timeout] entries would be more irregular, but all three were logged at regular points within 1 to 47 minutes of launch.
Success-only STAMP works: nothing succeeded, so the STAMP didn't advance. That's the correct, designed behavior.
echo ... >> "$ERRLOG" works: three weeks of entries are all there.

So none of the safety mechanisms is broken. Yet updates failed three weeks running. What the design changed was going from "a failure tells you nothing" to "you can see when a failure happened and why." It never fixed the failure itself. All the "should work" code really did was turn a silent failure into a logged one. The root cause was untouched.

The most suspicious variable left is gh auth token. The script comment says "it still works as before even if the token can't be obtained." Read the other way, that means nobody checks whether the token was obtained. Running gh auth status locally, I confirmed that besides the authenticated account, there is an account whose keyring access times out. gh auth token itself currently exits normally (exit 0), but nothing guarantees it resolves the same way every time in launchd's non-interactive, 4:40 a.m. context. If keychain access stalls there and GH_TOKEN is passed empty, the script gives no warning, falls back to the unauthenticated path, and hits the same rate limit it was meant to avoid. These logs alone can't prove that, so I'm leaving it as the next hypothesis to dig into.

How this ties to the macos-hook-path-timeout skill

The gtimeout dependency has some history. ~/.claude/skills/auto/macos-hook-path-timeout/SKILL.md is an existing skill that was auto-generated based on this very script.

### 症状2: スクリプトの `timeout 60 ...` が効かない（`timeout: command not found`）
macOSは GNU coreutils の `timeout` を**標準搭載しない**。ECCのhooks.md等が前提にしていても不発。

**修正**: `brew install coreutils` で `gtimeout` が入る。スクリプトは
`command -v gtimeout >/dev/null && TIMEOUT_CMD="gtimeout 60"` の形で優先検出させる
（`post_tsc_check.sh` / `skills-auto-update.sh` はこの形。coreutils導入だけで設計通りに起動する）。

(Translation: Symptom 2: a script's timeout 60 ... doesn't work (timeout: command not found). macOS does not ship GNU coreutils' timeout, so it fails even when docs like ECC's hooks.md assume it exists. Fix: brew install coreutils provides gtimeout. Scripts should detect it first with command -v gtimeout >/dev/null && TIMEOUT_CMD="gtimeout 60". post_tsc_check.sh and skills-auto-update.sh use this form, so installing coreutils alone makes them start as designed.)

This skill's Verification section only goes as far as checking that the gtimeout command exists.

command -v gtimeout && echo "post_tsc_check 等の timeout 分岐が起動する"

So all this skill guarantees is that "if gtimeout is installed, the timeout branch runs." Whether the job actually succeeds once it's in that branch is out of scope. The three-week failure streak happened in exactly that gap: gtimeout started correctly, capped correctly, and logged correctly, and skills update itself still didn't get through.

Pitfalls I hit

Raising the 45-second timeout to 600 seconds "should have" fixed it, but it didn't fix the network or rate-limit problem itself → A longer timeout only helps with hangs. If the cause is something else (rate limiting, or GH_TOKEN not being obtained), it will fail in the same place however far you extend it.
Success-only STAMP didn't fix failures. It only made them visible → Unless something regularly reads the logs (cron monitoring, a statusline integration), nobody reads the visible failures and you're back to "nothing has updated in months."
Whether npm warn exec shows up tells you where the failure is → If that line is there, npx's bootstrap succeeded, so the cause is inside skills update (network/API side).
"It still works as before even if the token can't be obtained" is the weakest fallback → An empty GH_TOKEN quietly drops to the unauthenticated path. If you don't log whether auth succeeded, you can't tell when a supposed fix has quietly stopped doing anything.
Compare the STAMP mtime against the cron schedule → If the update time doesn't match the job's start time, the success probably didn't come from the automated path.
An existing auto-skill's Verification section may only check that a command exists → "Does the command exist?" and "Does the job keep succeeding afterward?" are separate things to verify.

Summary

All three self-healing layers (GH_TOKEN injection, gtimeout, success-only STAMP) behaved as designed, both in the code and in the run logs.
Even so, the last three weekly runs (9/6, 9/13, 9/20) all failed for the same reason, and the STAMP stayed stuck at 2026-09-10.
The safety mechanisms only turned a "silent failure" into a "logged failure." They did not fix the root cause.
"Should work" (design, past measurements) and "actually worked" (raw logs, STAMP mtime) are different things, and you need to check them against each other regularly.
Whether gh auth token resolves every time in launchd's non-interactive context is still unverified. That's the next hypothesis to rule out.

Next time, I plan to check whether gh auth token really resolves every time in launchd's context (non-interactive, early morning, keychain not unlocked) and report whether that turned out to be the root cause.

What about you: how do you check that your "fixed" background jobs are still succeeding, and not just logging their failures neatly?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

6 Pitfalls Syncing AI Memory to Google Drive With No API: Keeping File IDs Intact

Lily — Fri, 25 Sep 2026 00:00:04 +0000

Use the obvious "write a temp file, then os.replace()" pattern on a Google Drive for Desktop mount and the file keeps its name but loses its Drive identity. Drive sees the original file disappear and a brand-new one show up in its place, with a new file id. I switched to overwriting the same file descriptor in place. The file id now survives every push, and the sync path still needs no gws auth and no network calls.

Last time I wrote about isolating untrusted external output. This post covers the sync path behind it: memory-drive-mount-sync.py, which treats the local Google Drive for Desktop mount folder as nothing more than a place to drop files.

The problem: add a sync path without adding another API

I already have one path that puts vault memory in the cloud: memory-cloud-sync.py, built on gws (Google Workspace CLI). It's thorough: OAuth, the Drive API, Docs API batchUpdate, and CAS-style conflict detection via requiredRevisionId. The cost of that thoroughness is a heavy set of dependencies.

My Mac also already runs Google Drive for Desktop, which mounts Drive contents as local files under ~/Library/CloudStorage/GoogleDrive-<google-account>/マイドライブ/ (My Drive). Writing there should sync without any extra auth or API calls.

But the naive "write to a temp file and rename with os.replace()" approach breaks things. os.replace() swaps the inode, and Drive for Desktop ties each Drive file id to the local file object (the inode). Swap that object out with a rename and Drive sees the original file vanish and a new one appear. The file id the file carried is lost.

Design: push overwrites the same file, pull reads only a narrow set of notes

The module docstring of memory-drive-mount-sync.py lays out exactly what the script is allowed to do:

"""One bounded, single-run local sync against a Google Drive for Desktop mount.

This is deliberately separate from memory-cloud-sync.py: it never calls ``gws``,
never touches network credentials, and does nothing beyond one push (of the
locally built curated snapshot) and one bounded pull (of a handful of plain
dated notes already sitting under the mounted folder). It only ever reads and
writes: the vault's own AI/.runtime build cache, a single named file inside
the mount directory (in place), and AI/INBOX/cloud inside the vault.
"""

It has exactly three jobs:

push: take the SHARED-MEMORY.txt built from the vault by memory-bridge.py build and overwrite the file of the same name in the mount, as the same file
pull: pick up only date-named plain-text notes in the mount (.txt/.md/.markdown, max 50 files, 256KB per file) and stage them in quarantine under the vault's AI/INBOX/cloud
block concurrent runs with a single-run lock

Note: With no gws and no Drive API, there's no competing-write detection like requiredRevisionId. Instead, TOCTOU (check-then-use) gaps are closed with POSIX fstat/inode matching, O_NOFOLLOW, and flock. With no auth layer, the safety comes from being meticulous at the filesystem layer.

Push: overwriting without breaking the file id

This is the core function. Instead of os.replace(), it does seek(0) → write → truncate on the existing file descriptor.

def overwrite_shared_memory(path: Path, content: bytes) -> None:
    """Overwrite an existing regular file in place, preserving its identity.

    Deliberately never os.replace(): the mount file's identity (e.g. a Google
    Drive file id behind the local inode) must be preserved rather than
    swapped for a new one.
    """
    initial = m.check_regular_non_symlink(path)
    flags = os.O_RDWR
    if hasattr(os, "O_NOFOLLOW"):
        flags |= os.O_NOFOLLOW
    fd = os.open(path, flags)
    try:
        opened = os.fstat(fd)
        if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != (initial.st_dev, initial.st_ino):
            raise MountSyncError("mount_target_changed_during_write")
        handle = os.fdopen(fd, "r+b", closefd=False)
        try:
            handle.seek(0)
            handle.write(content)
            handle.truncate()
            handle.flush()
            os.fsync(handle.fileno())
        finally:
            handle.close()
    finally:
        os.close(fd)

check_regular_non_symlink first runs an lstat-based safety check (not a symlink, is a regular file). Then the file is open()ed, and the (st_dev, st_ino) from an immediate fstat is compared with the lstat result taken before the open. If they differ, the file was swapped between the check and the open (TOCTOU), and the write is aborted.

The push itself also has a guard before writing:

mount_path = mount_dir / "SHARED-MEMORY.txt"
# Missing / symlink / non-regular mount target: refuse to write, surface a
# non-sensitive error code, and never fall back to creating a new file
# (that would mint a new Drive file id in place of the shared one).
m.check_regular_non_symlink(mount_path)

If the target is missing, is a symlink, or isn't a regular file, the script does not fall back to creating a new file. Creating one would make Drive issue a different file id. Content is compared with the updated: line stripped out, and if nothing else changed, the write is skipped entirely. That way a timestamp-only difference doesn't set off a Drive sync event on every run.

Pull: only date-named plain-text notes

The pull targets are explicitly narrowed:

MAX_PULL_FILES = 50
MAX_PULL_FILE_BYTES = 256 * 1024
NAME_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}")
PULL_EXTENSIONS = {".txt": "text/plain", ".md": "text/markdown", ".markdown": "text/markdown"}

def classify_pull_candidate(name: str) -> Optional[str]:
    if not NAME_DATE_RE.match(name):
        return None
    if name.upper().startswith("SHARED-MEMORY"):
        return None
    lower = name.lower()
    for suffix, mime in PULL_EXTENSIONS.items():
        if lower.endswith(suffix):
            return mime
    return None

Only names starting with YYYY-MM-DD are picked up, and the push target itself (anything starting with SHARED-MEMORY) is explicitly excluded. Without that exclusion, the next pull would pick up the file you just pushed as an "unknown external note," creating a loop.

for entry in entries:
    info = entry.stat(follow_symlinks=False)
    if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
        counts["skipped"] += 1
        continue
    ...
    candidates.append((entry, mime))
    if len(candidates) >= MAX_PULL_FILES:
        break

Symlinks and non-regular files are rejected here, and the candidate list stops at 50. Even with thousands of files in the folder, scandir keeps the scan cheap, and at most 50 files are actually read.

Reusing secret screening and symlink safety checks

Each note picked up by pull goes straight into m.stage_note(), which already exists in memory-cloud-sync.py. The hyphen in the filename rules out import memory-cloud-sync, so the module is loaded by path with importlib.

_CLOUD_SYNC_PATH = Path(__file__).with_name("memory-cloud-sync.py")
_spec = importlib.util.spec_from_file_location("memory_cloud_sync_for_mount_sync", _CLOUD_SYNC_PATH)
if _spec is None or _spec.loader is None:  # pragma: no cover - defensive only
    raise ImportError("memory-cloud-sync.py could not be loaded")
m = importlib.util.module_from_spec(_spec)
sys.modules[_spec.name] = m
_spec.loader.exec_module(m)

This lets the script reuse without duplicating a single line of code:

m.secure_read: reads with an fstat match check before and after open
m.has_secret / m.is_unsafe: a regex bank covering AWS keys, GitHub PATs, sk--style API keys, JWT shapes, Slack tokens, Bearer headers, emails, phone numbers, and medical/whistleblowing terms
m.stage_note: saves safe notes to AI/INBOX/cloud/<sha256>.md with provenance, and writes unsafe ones to quarantine/ as JSON that leaves out the body

Split that logic across two places and eventually you fix one copy and forget the other, so keeping it in one place is deliberate.

A single-run lock to prevent concurrent runs

launchd starts the script every 5 minutes, so if the previous run hasn't finished, the new one has to skip immediately.

lock_path = vault / "AI" / ".runtime" / "memory-drive-mount-sync.lock"
m.ensure_directory_no_symlinks(lock_path.parent)
lock_fd = acquire_lock_fd(lock_path)
...
try:
    fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
    os.close(lock_fd)
    print("locked")
    return 0

acquire_lock_fd refuses to proceed if the lock file itself is a symlink, then opens it with O_NOFOLLOW. If the non-blocking flock can't be acquired, the script prints "locked" and exits cleanly with exit 0 rather than raising an error. The key point: a concurrent start is expected, not a failure.

An unusable mount (Drive app not running, not mounted, etc.) is handled separately too:

def check_mount_dir(path: Path) -> None:
    try:
        info = os.stat(path)
    except OSError as exc:
        raise MountUnavailableError() from exc
    if not stat.S_ISDIR(info.st_mode):
        raise MountUnavailableError()
    if not os.access(path, os.R_OK | os.X_OK):
        raise MountUnavailableError()

This returns mount_unavailable with exit 3, a different code from the lock case ("locked", exit 0) and from general errors (exit 2). That lets you tell three cases apart mechanically in the logs: "still running," "mount is gone," and "broke mid-run."

launchd config: every 5 minutes, low priority

<key>StartInterval</key>
<integer>300</integer>
<key>LowPriorityIO</key>
<true/>
<key>Nice</key>
<integer>12</integer>
<key>ThrottleInterval</key>
<integer>60</integer>

It starts every 5 minutes (StartInterval=300), and LowPriorityIO plus Nice 12 lower its disk I/O and CPU priority. Since the sync uses no network, the only real costs are local disk I/O and the CPU time for memory-bridge.py build, so a priority that stays out of the way of foreground work is enough. ThrottleInterval=60 is launchd's minimum restart interval, so even a crash loop can't restart more often than once a minute.

Pitfalls I hit

Overwriting via rename with os.replace() breaks the Drive file id → switched to seek(0) + write + truncate on the same fd
TOCTOU: the file gets swapped between check and open → compare (st_dev, st_ino) from lstat and fstat, and abort on mismatch
The next pull picks up the file you just pushed → classify_pull_candidate explicitly excludes names starting with SHARED-MEMORY
A diff in the updated: line alone triggers a write every run → strip the updated: line before checking whether content changed
Hyphens in the filename make import impossible → load by path with importlib.util.spec_from_file_location
Write failures when the mount is unavailable get buried among "errors" → split mount_unavailable out into its own exit code

Wrap-up

The Google Drive for Desktop local mount can serve as a plain file drop for syncing, with no auth layer added
Drive tracks file ids by inode, so never use os.replace(). Overwrite through the same fd to keep the file object intact
With no auth layer, POSIX-level care (fstat/inode matching, O_NOFOLLOW, flock) provides the safety instead
Pull is narrowed by date-based name, extension, file count, and size, and secret screening plus symlink safety checks are reused from the existing implementation so there's only one copy to maintain
Concurrent runs, an unavailable mount, and general errors each get their own exit code, so logs can be triaged mechanically

Next time, I plan to write about extending this single-run lock and staging quarantine approach to the other sync paths.

Have you run into file identity problems when writing to a synced folder like Drive, Dropbox, or iCloud? How did you work around them?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

I Don't Trust the LLM That Writes My Shared Memory: 3 Regex Gates Before Claude Code Auto-Commits to My Obsidian Vault

Lily — Thu, 24 Sep 2026 00:00:02 +0000

Every 10 minutes, an unattended job has an LLM summarize my AI conversation logs, then commits and pushes the result to my Obsidian Vault. No human reviews it first. The obvious risk: the model's output ends up in git history, and other AI sessions read it later as memory. In the naive version, whatever the model returned went straight into the Vault. Now I treat that output as untrusted input. The model runs with no write tools, and the output goes through the same regex checks at three separate points before anything reaches a commit.

Last time, I wrote about how I misread a quota-limit message as "content too short" and failed three times in a row. This post covers something that comes one step earlier: a design that treats the AI's own output as untrusted input.

The problem: unattended commits of conversation-log summaries to my Vault

Conversation logs from Claude Code and Codex keep piling up under ~/Documents/my-knowledge-base/raw/ (conversations/, codex-conversations/, manus-conversations/). Every 10 minutes a job picks them up, has a model summarize them, writes the summary back to AI/SESSION-STATE.md in my Obsidian Vault, and runs git commit and push. Nobody checks it along the way.

The scariest part is that the LLM doing the summarizing can itself return untrustworthy output. A conversation log might contain an API key or auth URL I pasted in. It might also contain prompt-injection text that came in through another AI or a web page. If I write that into the Vault without checking it, secrets end up in git history, or the next AI session that reads the Vault follows fake instructions.

The previous post was about misreading a quota-limit message as real body text, but the root cause is the same: if you treat whatever text an AI returns as meaningful content by default, things break. This post covers the countermeasures: a sandbox on the generation side and a regex gate on the receiving side.

Overview: isolate generation, distrust everything on receipt

The pipeline is two files: memory-reflect.sh (a bash script launched by launchd) and normalize-memory-reflection.py (the receiving gate).

Layer	What it does	Owner
Generation	Have a maximally de-privileged Claude produce the summary	`memory-reflect.sh`
Validation	Re-validate the returned JSON without trusting any of it	`normalize-memory-reflection.py`
Double check	Re-scan the written files and the git diff with regex	`memory-reflect.sh` (second half)

Note: The key is not to assume that restricting the generation side makes things safe. Even with the tools limited to --tools Read,Grep,Glob, you can't control the content of the text the model returns. So the receiving side validates it again, zero-trust.

Layer 1: Summarize with a Claude that has no privileges

Here's the invocation:

"$TIMEOUT_BIN" "$MODEL_TIMEOUT_SECONDS" "$CLAUDE_BIN" --print --no-session-persistence --model sonnet \
  --setting-sources project \
  --strict-mcp-config --mcp-config '{"mcpServers":{}}' \
  --tools 'Read,Grep,Glob' \
  --output-format json --json-schema "$SCHEMA" \
  --system-prompt 'You are a read-only structured memory extractor. Follow the user prompt exactly and return only schema-compliant JSON.' \
  "$PROMPT" > "$RESULT" 2>>"$LOG"

--tools 'Read,Grep,Glob': no Bash, Edit, Write, or MCP. The model can't write files and can't reach outside.
--strict-mcp-config --mcp-config '{"mcpServers":{}}': forces MCP server connections to zero in code, instead of relying on me having cleaned up my config.
--output-format json --json-schema "$SCHEMA": no free-form text. If the output doesn't match the schema, the call itself fails.
timeout 120 (MODEL_TIMEOUT_SECONDS): a wall-clock cap in case it hangs.

Here's the schema (reformatted for readability):

{
  "type": "object",
  "additionalProperties": false,
  "required": ["session_state", "proposals", "sources_reviewed", "limitations"],
  "properties": {
    "session_state": { "type": "string", "maxLength": 5000 },
    "proposals": {
      "type": "array",
      "maxItems": 8,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["kind", "title", "content", "sources", "confidence", "expiry"],
        "properties": {
          "kind": { "enum": ["durable-candidate", "provisional-candidate", "wiki-update-candidate", "do-not-store"] },
          "confidence": { "enum": ["high", "medium", "low"] }
        }
      }
    },
    "sources_reviewed": { "type": "array", "maxItems": 16 },
    "limitations": { "type": "array", "maxItems": 12 }
  }
}

The prompt adds a second layer of instructions:

Never include or paraphrase: credentials, API keys, tokens, passwords, cookies,
auth URLs/codes, financial account or card data, medical detail, direct contact
data, private messages, meeting links, detailed private schedules, unconfirmed
legal/financial terms, or unverified allegations.

However, this is only a request to the model. Telling it "don't write secrets" in the prompt doesn't guarantee it will comply. That's why the layers that follow do the real work.

If the call fails, the script tries a local Qwen fallback (ollama-memory-json.py). If that also fails, it writes {} and treats the result as empty. This is an unattended job, so I'd rather it fail safely with no output than halt when something breaks.

Layer 2: The regex gate, zero-trust validation of model output

The docstring of normalize-memory-reflection.py says:

"""Normalize untrusted local LLM reflection output into a safe schema.

On malformed or unsafe output, emit a provenance-only fallback. The fallback
never summarizes transcript contents and therefore cannot promote invented or
sensitive details into the shared session state.
"""

It states outright that LLM output is untrusted. If validation fails, it returns a fallback that never summarizes anything from the transcripts. Most of the work happens in three stages.

① Don't trust JSON parsing itself

def unique_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
    result: dict[str, Any] = {}
    for key, value in pairs:
        if key in result:
            raise ReflectionError("duplicate-json-key")
        result[key] = value
    return result

With object_pairs_hook=unique_object, any JSON object with duplicate keys is rejected. The standard json.loads silently keeps the last value, so this closes off parser-differential attacks where something like {"kind":"do-not-store","kind":"durable-candidate"} slips past the check.

Envelope unwrapping is tightly restricted here too:

def unwrap_structured_output(value: Any, depth: int = 0) -> Any:
    ...
    for key in ("structured_output", "result"):
        nested = value.get(key)
        if isinstance(nested, dict):
            return unwrap_structured_output(nested, depth + 1)
        if isinstance(nested, str):
            match = FENCED_JSON_RE.fullmatch(nested.strip())
            # A result string may be exactly JSON or a single explicit JSON fence.
            # Do not scrape an object embedded in arbitrary untrusted prose.
            return unwrap_structured_output(parse_json_text(match.group(1) if match else nested), depth + 1)
    raise ReflectionError("structured-output-missing")

As the comment says, it deliberately does not fish JSON-looking fragments out of surrounding prose, where prompt injection could have planted them. It only accepts the --output-format json CLI envelope (the structured_output/result keys) or exactly one Markdown JSON fence. Nesting is cut off at depth > 3.

② Reject content with regex

SECRET_RE = re.compile(r"(?i)(?:\bsk-[A-Za-z0-9_-]{16,}\b|\b(?:api[_ -]?key|password|secret|bearer|token)\s*[:=]\s*[^\s`'\"]{8,})")
MEETING_RE = re.compile(r"https?://(?:meet\.google\.com|zoom\.us/j|teams\.microsoft\.com)/[^\s)\]>]+", re.I)
AUTH_LINK_RE = re.compile(r"https?://[^\s)\]>]*(?:oauth|auth|signin|login)[^\s)\]>]*|https?://[^\s)\]>]*[?&](?:code|token|access_token|id_token)=[^\s&#)\]>]+", re.I)
FINANCIAL_RE = re.compile(r"(?i)\b(?:card\s*number|credit\s*card|bank\s*account|routing\s*number|iban)\b")
HEALTH_RE = re.compile(r"(?i)\b(?:medical\s+record|diagnosis|medication|patient\s+detail|health\s+condition)\b")
CONTACT_RE = re.compile(r"(?i)\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b|\b(?:\+?\d[\d .()/-]{7,}\d)\b")
INSTRUCTION_RE = re.compile(
    r"(?i)\b(?:ignore|disregard|override)\b.{0,80}\b(?:previous|prior|system|developer)\b"
    r"\b(?:instructions?)\b"
    r"|\b(?:system\s+prompt|developer\s+message|jailbreak)\b"
)

The interesting part is how CONTACT_RE is used. A regex that rejects phone-number-like digit sequences will almost certainly flag dates like 2026-09-23 too. So the actual check looks like this:

def is_safe(value: str) -> bool:
    contact_candidate = ISO_DATE_RE.sub("[DATE]", value)
    if CONTACT_RE.search(contact_candidate):
        return False
    return not any(pattern.search(value) for pattern in (
        SECRET_RE, MEETING_RE, AUTH_LINK_RE, FINANCIAL_RE, HEALTH_RE, INSTRUCTION_RE,
    ))

Replacing ISO dates with [DATE] before the phone-number check prevents those false positives. INSTRUCTION_RE targets the stock phrases of prompt injection, like "ignore previous instructions", "system prompt", and "jailbreak". It's there so that even if instructions injected in another session show up in a conversation log, they never get adopted as session_state.

③ Restrict citations to the list of files actually provided

raw_reviewed = candidate["sources_reviewed"]
...
# The model may only cite the source manifest generated by the wrapper.
if any(item not in sources for item in raw_reviewed):
    raise ReflectionError("sources-reviewed-not-in-manifest")

sources is the file list that memory-reflect.sh built with find and passed in as an argument (SOURCE_LIST). If the model claims "I also read this file" and that file isn't in the list, the output is rejected immediately. Each proposal's sources must likewise be a subset of the files provided. This closes off the trick where the model fabricates sources to sneak its own "facts" in as durable-candidates.

If validation fails, say nothing

def fallback(sources: list[str], reason: str) -> dict[str, Any]:
    return {
        "status": "fallback",
        "session_state": (
            "## Automatic archive update\n\n"
            f"- {len(sources)}件の新しいlocal transcriptを取得した。"
            "structured reflectionの出力が検証できなかったため、内容は共有memoryへ昇格していない。\n"
            ...
        ),
        "proposals": [],
        ...
    }

(The Japanese strings read roughly: "Fetched N new local transcripts. The structured reflection output could not be validated, so its content was not promoted to shared memory.")

When validation fails, the fallback session_state only says how many transcripts were fetched and summarizes nothing from their contents. The safe option here isn't to produce a harmless-looking summary. It's to say nothing about the content at all.

Layer 3: Scan the files and the git diff again

Even after normalization passes, memory-reflect.sh runs two more regex checks of the same kind.

# Last line of defense: never copy likely secret values to shared state.
if grep -E -q -i '(api[_ -]?key|password|secret|bearer|token)[[:space:]]*[:=][[:space:]]*[^[:space:]]{8,}' "$CANDIDATE"; then
  echo 'reflection blocked: secret-like value in candidate state'
  exit 1
fi
mv "$CANDIDATE" "$STATE_FILE"

This grep runs separately on the candidate file for SESSION-STATE.md and on the proposal file for AI/INBOX/auto/. Then, right before committing, it re-scans the staged diff itself:

git add -- "$STATE_FILE" "$INBOX_FILE"
git diff --cached --check -- "$STATE_FILE" "$INBOX_FILE"
git diff --cached -- "$STATE_FILE" "$INBOX_FILE" | grep -E -i '(api[_ -]?key|password|secret|bearer|token)[[:space:]]*[:=][[:space:]]*[^[:space:]]{8,}' >/dev/null && {
  echo 'reflection blocked: secret-like value in staged diff'
  git restore --staged -- "$STATE_FILE" "$INBOX_FILE" || true
  exit 1
}
if ! git diff --cached --quiet -- "$STATE_FILE" "$INBOX_FILE"; then
  git commit --only -m "chore(memory): reflect local sessions" -- "$STATE_FILE" "$INBOX_FILE" >/dev/null
  git push origin main >/dev/null
fi

Running the same regex three times (inside the normalizer, on the candidate file, and on the staged diff) looks redundant, but each check sees different input. The normalizer sees the model's raw JSON. The second check sees the file after it's been rendered as Markdown. The third sees the exact diff going into git. Formatting and Markdown conversion can change how escapes and newlines are handled, which could let something slip past the first check. So the last line of defense has to look at the actual bytes being committed.

git commit --only matters too. I also edit this Vault repo by hand, so the job might run while I have other changes that I forgot to stage. --only limits the commit to the two files added in this run, so unrelated changes never get swept in.

The launchd configuration

<key>StartInterval</key>
<integer>600</integer>
<key>ThrottleInterval</key>
<integer>60</integer>
<key>LowPriorityIO</key>
<true/>
<key>Nice</key>
<integer>12</integer>
<key>EnvironmentVariables</key>
<dict>
  <key>MEMORY_REFLECT_MIN_INTERVAL_SECONDS</key>
  <string>300</string>
</dict>

StartInterval=600 launches the job every 10 minutes. Two guards on the script side prevent overlapping runs even if a previous one takes longer: a debounce (MIN_INTERVAL, overridden to 300 seconds in the plist) and a lock directory (if mkdir "$LOCK" fails, the run skips immediately). LowPriorityIO and Nice 12 keep this background job from taking CPU and I/O away from foreground work.

Pitfalls I hit

Here are the problems I actually ran into, as recorded in the code comments:

The phone-number regex flagged dates as false positives → replace dates with [DATE] via ISO_DATE_RE before applying CONTACT_RE
Duplicate JSON keys could bypass the check → reject duplicate keys as soon as they're found, using object_pairs_hook
Trusting the model's own claims about its sources isn't validation → force sources_reviewed/proposal.sources to be a subset of the files actually provided
ValueError was only caught after ad-hoc extraction paths, so the script could crash partway through → this past bug is noted in the code. It's now fixed: errors are normalized to ReflectionError, which returns a deterministic failure code
With the regex check in only one place, things could slip through during formatting or Markdown conversion → repeat the same check in three places: normalizer output, the written files, and the staged diff
Commits could sweep in unrelated changes → git commit --only limits the commit to the two files

Summary

On the generation side, --tools Read,Grep,Glob, disabled MCP, and an enforced JSON Schema mean the model has no write capability at all.
On the receiving side, output is treated as fully untrusted and validated in three stages: hardened JSON parsing, regex detection of secrets, contact info, and prompt injection, and restricting citations to the manifest.
When validation fails, the fallback chooses to summarize nothing, erring on the side of safety.
The same regex checks run again when writing the files and right before the git commit. Validation shouldn't stop at one checkpoint.
git commit --only keeps the unattended job from sweeping up a human's uncommitted changes.

Next time, I plan to write about what happens after SESSION-STATE.md and the INBOX proposals pass these layers: how a human actually reviews them and promotes them into durable memory.

If you let an LLM write into shared memory or a repo unattended, where do you put your trust boundary: on the generation side, the receiving side, or right before the commit?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

4 Days, 3 Wasted Calls Per Run: My Retry Loop Mistook a Quota-Limit Message for a 'Too Short' Article

Lily — Mon, 21 Sep 2026 05:00:05 +0000

Last time I wrote about using a probe to decide which jobs to stop first when the quota runs dry. This post is about the problem sitting right next to it: my script could tell when the quota was exhausted, but it treated the limit message as if it were real article body text, then kept retrying a failure that no rewrite could ever fix. From September 13 to 16, every run burned 3 claude -p calls to produce nothing. After the fix, it burns 1 and bails out.

The problem: 4 straight days (9/13–9/16) of 3 failures per run

note2-daily-stock.sh auto-generates product review articles for note2 (bokumolily) using claude -p. For each product it calls claude -p up to 3 passes to produce the body. Each pass is judged by a quality check (article_quality_ok) with criteria like "does the character count exceed MIN_CHARS (7,000)?" and "are there at least 2 affiliate links?"

For 4 days, 9/13 through 9/16, this script failed 3 times in a row with the exact same symptom. The cause: when claude -p hits the Max plan's limit, it returns a one-line limit-reached message (no TITLE: line), and the script treated that as "an ordinary generation failure where the body is too short." The comment in the actual code records exactly what happened:

  # 2026-09-16: claude -pがクォータ上限メッセージ(1行・TITLE:行なし)を返すと
  # 本文が空のまま「文字数不足」として3回とも空振りし、上限中に3回分のclaude呼び出しを浪費していた
  # (note2-daily 9/13-9/16 4日連続で同一症状を実測)。上限メッセージは即検出して1回で中断する。

(note2-daily-stock.sh:565-567)

While the limit is in effect, no amount of rewriting will produce a body. Yet article_quality_ok only ever said "quality NG reason: not enough characters," so the script concluded "maybe one more pass will fix it" and threw away 2 more pointless claude -p calls during the limit window.

Note: A retry loop needs more than the monolithic fact that "it failed." If you don't distinguish "a failure a rewrite can fix" from "a failure a rewrite cannot fix," you'll keep walking into the same wall.

The fix: grep for the limit message and abort after 1 pass

The fix is to grep for the limit-message pattern immediately after each pass's generation, and if it matches, abort right there instead of spending the remaining passes.

  for pass in 1 2 3; do
    RAW=$(generate_article_raw "$PRODUCT_JSON" "$IMG_COUNT")
    if printf '%s' "$RAW" | grep -qiE "hit your (weekly |5-hour |usage |session )?limit|usage limit reached|rate limit|resets? (at |in |on )?[0-9]{1,2}(:[0-9]{2})? *[ap]m"; then
      QUOTA_HIT=1
      log "ABORT: quota-message検出のため中断 (pass $pass) $SLUG raw先頭=[$(printf '%s' "$RAW" | tr '\n' ' ' | cut -c1-160)]"
      break
    fi
    write_article_file "$RAW" "$OUT"
    ...
  done

(note2-daily-stock.sh:563-572, excerpt)

When QUOTA_HIT is set, the script deletes the half-written output file and any images downloaded along the way, notifies Discord, and exits immediately. Instead of 3 calls, it consumes exactly 1 at that point and retreats.

  if [ "$QUOTA_HIT" -eq 1 ]; then
    [ -e "$OUT" ] && rm -f "$OUT"
    rm -f "$ROOT/${IMG_BASENAME}-p"*.jpg
    discord_notify "⚠️ note2-daily: claudeクォータ上限のため中断（3回消費せず中止）: $ITEM_NAME"
    exit 1
  fi

(note2-daily-stock.sh:590-595)

"Not enough characters" and "limit reached" look alike on the surface (the body is nearly empty), but the correct response is completely different. The former might be fixed by a rewrite; the latter won't be fixed no matter how many times you retry until the reset time. Separating those two cases up front with a single grep is the entire fix.

The pitfall I stepped in: the sibling scripts still don't have it

There are 3 other scripts that share the same shape as note2-daily-stock.sh — "have claude -p write the body, retry if the quality check fails": article-daily-stock.sh (the script that writes this very Zenn article series), maker-daily-stock.sh, and series-daily-stock.sh. I grepped all 3, and none of them contained the limit-message detection pattern.

$ grep -n "hit your\|usage limit\|QUOTA_HIT\|rate limit" \
    article-daily-stock.sh maker-daily-stock.sh series-daily-stock.sh
(該当なし)

In particular, run_pass() in series-daily-stock.sh has almost exactly the same structure that note2 tripped over.

run_pass(){ # $1=prompt $2=outfile $3=label
  local prompt="$1" outfile="$2" label="$3" attempt wait
  for attempt in 1 2 3; do
    rm -f "$outfile"
    timeout 600 "$CLAUDE" ... -p "$prompt" ... > "$outfile" ...
    if [ -s "$outfile" ]; then
      ...
    fi
    log "WARN: pass $label 空出力(attempt $attempt/3)"
    if [ "$attempt" -lt 3 ]; then
      wait=$((attempt * 30))
      sleep "$wait"
    fi
  done

(series-daily-stock.sh:254-279, excerpt)

Waiting 30 seconds, then 60 seconds, on "empty output (attempt N/3)" and hammering the same call 3 times for the same reason is essentially the same structure note2 hit on 9/13–9/16. If this script runs during a limit window, it should waste calls in exactly the same way — and it hasn't been fixed yet.

maker-daily-stock.sh only has a single generation pass, so it doesn't fail 3 in a row, but the same dates show up in its comments too.

  # 失敗時もキューを1件進める。進めないと同じ題材が先頭に居座り続け、翌日以降も
  # 同じ理由で落ち続ける（2026-09-13〜16 に同一題材が7回連続で失敗した実測）。

(maker-daily-stock.sh:412-413)

In other words, the Max plan limit on 9/13–9/16 didn't just hit note2 — it dragged the maker pipeline down with it. On the maker side, the underlying problem of "misdiagnosing the limit message as an incomplete body" is still there, exactly as in note2; it's being held together by a different band-aid: "even on failure, at least advance the topic queue." The same root cause has a different emergency patch in each script, applied whenever someone happened to notice.

I confused a one-line limit message with a "body too short" quality failure → grep for it right after generation and abort before the quality check
Judging retries only by "empty/non-empty" or "character count" can't tell failure types apart → branch on the failure reason (retry the fixable ones, exit immediately on the unfixable ones)
Even when one limit drags down multiple pipelines at once, the fix doesn't propagate → note2 detects the cause, maker advances the queue, article/series are untouched — the responses are still all over the place
Fixing one script doesn't automatically carry over to sibling scripts that copied the same structure → don't assume "it's fixed" until you've confirmed it with grep

Summary

For 4 days (9/13–9/16), note2-daily-stock.sh misdiagnosed the limit-reached message as "not enough characters" and burned all 3 pointless claude -p calls during the limit window
The fix greps for the limit-message pattern immediately after generation and, on a match, aborts after 1 pass, ahead of the quality check
During the same 4 days, maker-daily-stock.sh also failed 7 times in a row on the same topic — the limit hit multiple pipelines simultaneously. But maker's fix is only "advance the queue," and the underlying misdiagnosis remains unfixed there as well as in article-daily-stock.sh and series-daily-stock.sh
A retry loop that doesn't distinguish failure types (fixable vs. unfixable) will step in the same hole every time. A lesson learned in one place only pays off once it's propagated to other scripts with a similar structure

Next time I plan to write about whether that propagation itself can be automated — a mechanism to mechanically carry a fix found in one script over to its "structurally similar siblings."

How do your retry loops tell a fixable failure from one that no retry will ever fix?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

One Line of mv Kills the 4x Duplicate Run After Recovery: Making launchd Queue Processing Idempotent

Lily — Mon, 21 Sep 2026 00:00:06 +0000

Six months ago my side business made ¥0 a month. This month it's ¥1.2M, running on an autonomous setup built with Claude Code. And here's the thing about automation: the moment it breaks is exactly when the sloppiness in your design gets exposed.

Why This Design Works

The Fork Between "Fixing by Habit" and "Fixing the Environment"

When automation breaks, the reflex is to reach for a procedural fix: "next time I'll do this instead." Check the logs more often. Retry by hand. Space things out a bit more. All of these are bandages meant to shrink the wound the next time the problem recurs. They don't remove the cause, so once the same conditions line up, it will happen again.

After six months of running an automated system, one thing I'm certain of: a scheduler combined with a shared directory will always collide unless you explicitly implement mutual exclusion. Not "it might collide if you're unlucky," but "it will collide whenever the conditions for collision line up." This isn't a probability problem; it's a structural one.

On September 17, 2026, Codex was down for an entire day with usage_limit_exceeded. Under ~/.codex/sessions/2026/09/17/ there were 54 sessions piled up, every one of them terminated with a limit error. Throughout that time, three workflows—note-autolike, ai-portraits-fragments, and social-autolike—kept firing jobs, and unprocessed JSON files accumulated in ~/dev/note-autolike/done/. Measured the next morning at 05:20, the JSON files flagged needs_imagegen_thumbnail had swelled from 43 to 57. Fourteen new files had stacked up, all of them left untouched.

The instant Codex came back, five launchd lanes fired at once. Each lane's pickup script grabs the JSON files under done/ with ls or a glob and starts processing. With no mutual exclusion, multiple processes look at the same JSON at the same time. The result: the funnel-pm lane processed the same JSON 4 times, one minute apart. As the numbers show—8 of 11 files were hit 2 or more times—this wasn't a freak accident. It was an inevitable consequence of the design.

launchd Doesn't Know About "Fighting Over Files"

Look at StartCalendarInterval in com.lily.codex-note-funnel.plist and you'll see two entries: 10:40 and 16:40.

<key>StartCalendarInterval</key>
<array>
    <dict>
        <key>Hour</key>
        <integer>10</integer>
        <key>Minute</key>
        <integer>40</integer>
    </dict>
    <dict>
        <key>Hour</key>
        <integer>16</integer>
        <key>Minute</key>
        <integer>40</integer>
    </dict>
</array>

launchd simply invokes run-codex-funnel.sh according to this schedule. Whether the previous run finished, or whether another lane is currently working on the same file—launchd has no interest in any of it. And that's correct, by-design behavior. launchd is a scheduler, not a mutex. Mutual exclusion over a file queue is the application layer's responsibility—in other words, it's a feature the pickup script itself has to implement.

Backlog Plus Simultaneous Recovery Is What Surfaces the Race

In normal operation, each lane's timing is slightly offset, which naturally keeps collisions rare. Target JSON files also arrive one at a time in sequence, so there are few opportunities for multiple lanes to see the same file simultaneously.

But a long outage like Codex's usage_limit_exceeded changes the situation completely. All lanes start at once with 57 JSON files backed up from the outage period.

通常運転:
  レーンA → job_001 処理 → 完了
  レーンB → job_002 処理 → 完了  （タイミングがズレており競合しない）

Codex 復帰直後:
  レーンA ─┐
  レーンB ─┤──→ job_001 を同時取得 → 4連射
  funnel-pm ─┘
  （57本が溜まっており、全レーンが先頭ファイルに殺到）

A procedural fix like "adjust the intervals" can't handle this "burst release" pattern. No matter how tightly you tune the intervals, the instant after recovery, every lane will fire simultaneously.

Why a Lock File Isn't Enough

"Just create a lock file" is the natural idea. But look closely at the implementation and there's a problem.

# NG: check-then-act は原子的でない
LOCKFILE=~/dev/note-autolike/.processing.lock
if [ ! -f "$LOCKFILE" ]; then
    touch "$LOCKFILE"
    process_file "$json"
    rm "$LOCKFILE"
fi

Between the if [ ! -f ] check and the touch, there's a "Time of Check to Time of Use" (TOCTOU) window where another process can slip in. Two processes both observe "no file" and both execute touch—that's the race. Under high load and high frequency, this window really does open.

On top of that, if the process crashes mid-run, the lock file stays behind. Every subsequent run sees the lock and exits immediately, so processing stalls completely. You need a separate crash-recovery procedure, and operational cost goes up.

File locking via the flock command is another option, but since file descriptors aren't inherited each time launchd spawns a new process, it doesn't work for exclusion between lanes.

The Guard Philosophy Shown by autolike-plist-reconcile.sh

The existing ~/.claude/scripts/autolike-plist-reconcile.sh contains a guard built with the same concern in mind. That script does a reload when the AUTOLIKE_TIMEOUT_SEC value in the plist has drifted from the value launchd has loaded, but the core is this code:

pid=$(launchctl list | awk -v l="$L" '$3==l{print $1}')
if [ -n "$pid" ] && [ "$pid" != "-" ]; then
    echo "[$(date '+%F %T')] $L 実行中(pid=$pid) のため見送り ($have -> $want)" >>"$LOG"
    continue
fi
launchctl bootout "$D/$L" 2>/dev/null

The comment at the top of the script says that booting out a running job wipes out that run's likes entirely, so it must always wait. It fetches the PID via launchctl list and skips the bootout unless the value is - (not running). Before starting work, check "is someone else already processing this?"—and if so, exit immediately. That's the same idea as the mv pattern in this article.

That said, a PID check has limits. You can't reduce to zero the chance that a new process starts between the "check" and the "bootout." In the context where this script runs, that's acceptable, but for claiming exclusive ownership of a file queue, it's insufficient. You need a stronger atomic operation.

The Overall Flow

Architecture Diagram: From Race to Resolution

Codex usage_limit_exceeded から復帰
              │
              ▼
    ~/dev/note-autolike/done/ に 57 本が滞留
    ┌────────────────────────────────────────┐
    │  job_20260917_001.json                │
    │  job_20260917_002.json                │
    │  ...                                  │
    │  job_20260917_057.json                │
    └────────────────────────────────────────┘
              │
    5 レーンが launchd により一斉起動
    ┌────────────────────────────────────────┐
    │ com.lily.codex-note-funnel   (10:40)  │
    │ com.lily.autolike.note1               │
    │ com.lily.autolike.note2               │
    │ com.lily.autolike.funnel-pm  ← 4連射  │
    │ com.lily.autolike.social              │
    └────────────────────────────────────────┘
              │
    【修正前】各レーンが done/ を glob → 先頭ファイルを取得
              │  排他なし → 全レーンが job_001 を同時に read
              ↓
    funnel-pm が 1分差で同一 JSON を 4 回処理
    ─────────────────────────────────────────
    【修正後】mv による原子的所有権取得
    ┌────────────────────────────────────────┐
    │  mv done/job_001.json processing/     │
    │  ├─ 成功: 自プロセスのみが処理を継続   │
    │  └─ 失敗: 他プロセスが取得済み → exit 0│
    └────────────────────────────────────────┘
              │
    「1ファイル = 1プロセスのみ処理」が保証される

Why mv Is Atomic

On the same filesystem, mv executes a single POSIX rename(2) system call. Because the "rename this file" operation completes indivisibly at the kernel level, even if two processes simultaneously run mv done/job_001.json processing/job_001.json, exactly one succeeds and the other is guaranteed to fail. No window opens between "check" and "acquire."

This is the fundamental difference between a PID check and mv. With a PID check, "check" and "start processing" are separate operations. With mv, "check" and "acquire" are completed inside one system call.

# 2プロセスが同時に実行した場合の挙動
# プロセスA: mv done/job_001.json processing/job_001.json → 成功（終了コード 0）
# プロセスB: mv done/job_001.json processing/job_001.json → 失敗（No such file or directory, 終了コード 1）

Wiring It into the Pickup Script

Here's the implementation to insert at the top of run-codex-funnel.sh.

#!/bin/bash
set -uo pipefail

DONE_DIR=~/dev/note-autolike/done
PROCESSING_DIR=~/dev/note-autolike/processing
mkdir -p "$PROCESSING_DIR"

# done/ から処理対象を1本選ぶ
TARGET=$(ls "$DONE_DIR"/*.json 2>/dev/null | head -1)
[ -z "$TARGET" ] && exit 0          # 対象なし → 正常終了

BASENAME=$(basename "$TARGET")

# アトミックに所有権を取得。失敗 = 他プロセスが取得済み
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null || exit 0

WORKING="$PROCESSING_DIR/$BASENAME"
# ここから先は自プロセスだけが処理する権利を持つ

There's effectively only one situation in which mv fails: another process has already mv'd the file and it no longer exists. That is precisely the race condition itself, so exiting immediately with exit 0 is the correct response. You don't even need to write an error log.

A cross-device mv (spanning different mount points) is not atomic, so be careful—but in this setup, both done/ and processing/ live under ~/dev/note-autolike/, which guarantees the same filesystem.

Connecting with com.lily.codex-note-funnel.plist

The ProgramArguments in com.lily.codex-note-funnel.plist are structured like this:

<key>ProgramArguments</key>
<array>
    <string>~/.claude/scripts/claude-quota-guard.py</string>
    <string>--job</string>
    <string>com.lily.codex-note-funnel</string>
    <string>--priority</string>
    <string>--</string>
    <string>~/dev/note-autolike/run-codex-funnel.sh</string>
</array>

claude-quota-guard.py runs first as a quota-control wrapper, and if there's headroom, it launches run-codex-funnel.sh. Since the pickup logic lives inside run-codex-funnel.sh, inserting the mv ownership claim into the first few lines of the script is all it takes. No plist changes required.

launchd will keep calling run-codex-funnel.sh at 10:40 and 16:40. Even with 57 JSON files backed up right after Codex recovers from usage_limit_exceeded, every time run-codex-funnel.sh starts it repeats the same behavior: "first claim one file with mv; if that fails, exit immediately." Even if five processes, including the funnel-pm lane, run at the same time, exactly one process handles each file.

Implementation Details

The Full Script: The Order for Stacking Defensive Layers

The first half showed only the single mv line, but to actually run it in production you need layers before and after. Looking at ProgramArguments in com.lily.codex-note-funnel.plist, claude-quota-guard.py runs as the leading wrapper, and then run-codex-funnel.sh is called. The full script looks like this:

#!/bin/bash
# launchd 環境の PATH は最小限のため先頭で上書きする
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
export PATH

set -uo pipefail

DONE_DIR=~/dev/note-autolike/done
PROCESSING_DIR=~/dev/note-autolike/processing
PROCESSED_DIR=~/dev/note-autolike/processed
LOG=~/dev/note-autolike/logs/codex-funnel.log

mkdir -p "$PROCESSING_DIR" "$PROCESSED_DIR"

# 起動時リカバリ: 前回クラッシュで processing/ に残ったファイルを done/ へ戻す
for stale in "$PROCESSING_DIR"/*.json; do
    [ -f "$stale" ] || break
    age=$(( $(date +%s) - $(stat -f %m "$stale") ))
    if [ "$age" -gt 3600 ]; then
        mv "$stale" "$DONE_DIR/" 2>/dev/null
        echo "[$(date '+%F %T')] recovered stale: $(basename "$stale")" >> "$LOG"
    fi
done

# done/ から処理対象を1本選ぶ
TARGET=$(ls "$DONE_DIR"/*.json 2>/dev/null | head -1)
[ -z "$TARGET" ] && exit 0

BASENAME=$(basename "$TARGET")

# ─── ここが核心 ─────────────────────────────────────────
# アトミックに所有権を取得。失敗 = 他プロセスが取得済み → 即退出
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null || exit 0
# ────────────────────────────────────────────────────────

WORKING="$PROCESSING_DIR/$BASENAME"

# クラッシュ時に processing/ へファイルが残るのを防ぐ EXIT トラップ
# 正常完了時は processed/ へ移動、異常時は done/ へ差し戻す
_cleanup() {
    local code=$?
    if [ $code -eq 0 ]; then
        mv "$WORKING" "$PROCESSED_DIR/$BASENAME" 2>/dev/null
    else
        mv "$WORKING" "$DONE_DIR/$BASENAME" 2>/dev/null
        echo "[$(date '+%F %T')] ERROR exit=$code, returned: $BASENAME" >> "$LOG"
    fi
}
trap '_cleanup' EXIT

# ── ここから先は自プロセスだけが $WORKING を処理する ──
echo "[$(date '+%F %T')] processing: $BASENAME" >> "$LOG"
# ... 本体処理 ...

Why Override PATH at the Top

Look at EnvironmentVariables in com.lily.codex-note-funnel.plist: the key exists, but the content is an empty <dict/>.

<key>EnvironmentVariables</key>
<dict>
</dict>

When launchd starts a process, its PATH is only the minimal set /usr/bin:/bin:/usr/sbin:/sbin. If you echo $PATH in a terminal you'll see /usr/local/bin included, but that's added by .zshrc or the nvm initialization script. launchd doesn't run any such shell initialization. Tools installed in /usr/local/bin like jq, node, and python3 fail silently as "command not found" in scripts launched from a plist. The fix is one of two choices: write it in the plist's EnvironmentVariables, or hardcode it at the top of the script. The latter is more convenient because you can verify the script's behavior on its own, so I've standardized on export PATH at the top.

Why `set -uo pipefail` and `|| exit 0` Must Go Together

When set -uo pipefail is active, the whole script aborts the moment a command returns a non-zero exit code. mv returns exit code 1 when the source file doesn't exist. That maps exactly onto the normal race scenario of "another process already claimed it."

# NG: set -uo pipefail 環境では mv 失敗でスクリプトが非ゼロ終了する
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null

# OK: 失敗を明示的に「正常な退出」に変換する
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null || exit 0

2>/dev/null is needed as its counterpart. With || exit 0 alone, "No such file or directory" is printed to stderr. Since StandardErrorPath in com.lily.codex-note-funnel.plist points to ~/dev/note-autolike/logs/codex-funnel.error.log, every race piles a line that isn't actually an error into the error log. When you monitor logs, this becomes the cause of false alarms like "the error log is growing."

The "Double Check" Idiom Demonstrated by autolike-plist-reconcile.sh

The PID-check portion of autolike-plist-reconcile.sh has a subtlety that's easy to miss.

pid=$(launchctl list | awk -v l="$L" '$3==l{print $1}')
if [ -n "$pid" ] && [ "$pid" != "-" ]; then

The first column returned by launchctl list is either a PID or -. - means the job is registered but not currently running. With [ -n "$pid" ] alone, a pid of - would also be judged as "has a PID." That's why [ "$pid" != "-" ] is joined with AND.

This idiom of checking "the variable is non-empty" and "the value is meaningful" separately can be applied to the mv pattern too. If you want to add a defensive layer that verifies the WORKING file actually exists rather than looking only at the exit code of mv, write it like this:

mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null || exit 0
# 念のため: mv が成功したはずなのにファイルがない（同一FSでない等）
[ -f "$WORKING" ] || { echo "FATAL: mv succeeded but file missing" >> "$LOG"; exit 1; }

This extra check is normally never reached, but it makes the symptom explicit if you accidentally run a cross-device mv. In an environment with set -uo pipefail active, having the cause in the log beats dying silently—debugging afterward is dramatically faster.

Design Rationale for Startup Recovery

The step at the top of the script that "returns processing/ files older than one hour to done/" is a dead-letter countermeasure.

age=$(( $(date +%s) - $(stat -f %m "$stale") ))
if [ "$age" -gt 3600 ]; then
    mv "$stale" "$DONE_DIR/" 2>/dev/null
fi

If the process crashes after claiming ownership with mv, the file stays in processing/. The EXIT trap is designed to return it to done/, but the trap won't run in cases like a forced kill with signal 9. The symptom of "I check done/ the next morning and it's empty, yet nothing got processed" is caused by this pattern. By running recovery at startup, the file automatically re-enters the queue on the next launchd invocation. The threshold is 3600 seconds (one hour) so that legitimately long-running jobs don't get returned by mistake. Since StartCalendarInterval in com.lily.codex-note-funnel.plist has the two entries 10:40 and 16:40, the maximum schedule gap is six hours. One hour is a sufficient safety margin within that range.

Where I Got Stuck

Stuck #1: The Race Was Gone, but the Error Log Kept Growing

The day after implementing mv || exit 0, I looked at codex-funnel.error.log and the line count had gone from 1 to 32. The processed count was increasing normally. The 4x duplicates were gone. And yet the error log was growing.

At first I assumed "there must be an exception somewhere" and started debugging the main processing. After about 30 minutes of chasing it, I realized 2>/dev/null was missing. The code I first wrote was this:

mv "$TARGET" "$PROCESSING_DIR/$BASENAME" || exit 0

|| only handles the exit code. It doesn't stop output to stderr. launchd keeps writing stderr to StandardErrorPath. As a result, "No such file or directory" accumulated at every moment of contention.

The correct form is this:

mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null || exit 0

The cause was that I hadn't built the habit of always writing these two together. My preconception that "error log growing = bug in the main body" sent my debugging in the wrong direction. When the error log grows, the first thing to check is "am I silencing the normal failures?"—I should have made that order routine from the start.

Stuck #2: Files Kept Piling Up in processing/ While done/ Looked Empty Forever

This happened in the version before I implemented the EXIT trap. Codex crashed overnight several times in a row, and when I checked the next morning, done/ was empty but processing wasn't happening.

Tracing the symptom, five JSON files had accumulated in processing/. Each time the script started, it checked done/, found TARGET empty, and did exit 0. But the "files that should be processed" were stuck in processing/.

done/       ← 空（次の仕事はここに来る）
processing/ ← job_001.json, job_002.json ...  ← ゾンビ
processed/  ← 完了済み

This state does double damage. First, processing stops. Second, "the fact that it's stuck isn't visible from outside." Because done/ is empty, the automation looks healthy. But even when new files arrive in done/, nobody ever touches the zombies in processing/.

The fix combined two things: returning files via the EXIT trap and recovering stale processing/ files at startup. Either one alone is insufficient; to handle forced kills where the trap doesn't run, startup recovery is required. After implementing this, I only need to periodically ls -la the processing/ directory to confirm nothing is lingering, and this type of stall resolves itself by the next morning.

Stuck #3: Works When Run from the Terminal, Silently Does Nothing from launchd

This was the failure that burned the most time. Running run-codex-funnel.sh directly in the terminal processes JSON files normally. Via launchd, "processing: ..." never appears in codex-funnel.log. The error log is empty too. The script is starting, but doing nothing.

What I used to pin down the cause was reproducing the shell launchd starts as faithfully as possible.

env -i HOME="$HOME" PATH=/usr/bin:/bin:/usr/sbin:/sbin bash -l ~/dev/note-autolike/run-codex-funnel.sh

Running it with env -i to minimize the environment variables, jq: command not found appeared partway through. The main processing used jq to read the needs_imagegen_thumbnail flag from the JSON, but jq was installed at /usr/local/bin/jq. With set -uo pipefail active, the script exits non-zero the moment jq isn't found. The exit code should have been written to codex-funnel.error.log, but since the error message itself never went to stderr (no jq means no error text either), the log was empty.

Had I read that EnvironmentVariables in com.lily.codex-note-funnel.plist was empty, I would have noticed sooner. The cause was skipping verification on the assumption that "PATH is probably fine" without actually reading the file. Since then, writing PATH= at the top has been an absolute rule for every script run via launchd.

PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
export PATH

Whether this one line is present is now the first thing I check in code review.

Stuck #4: `ls *.json` Returned Exit Code 2 on "No Files" and Crashed the Script

When done/ is empty, ls "$DONE_DIR"/*.json fails shell glob expansion and returns exit code 2. In a set -uo pipefail environment, this causes the script to terminate abnormally.

# NG: done/ が空のとき ls が exit 2 → pipefail でスクリプト全体が落ちる
TARGET=$(ls "$DONE_DIR"/*.json | head -1)

# OK: 2>/dev/null でエラーを捨て、変数が空かどうかで判断する
TARGET=$(ls "$DONE_DIR"/*.json 2>/dev/null | head -1)
[ -z "$TARGET" ] && exit 0

This problem wouldn't occur if the design treated an empty done/ as "abnormal." But this setup is designed so that "in normal operation, done/ is empty more often than not," so empty is the normal path. To "treat empty as normal," you need a line that explicitly absorbs the command's failure.

Looking at how autolike-plist-reconcile.sh is written, there are three places where it uses continue to skip out of the loop.

want=$($PB -c "Print :EnvironmentVariables:AUTOLIKE_TIMEOUT_SEC" "$P" 2>/dev/null) || continue
have=$(launchctl print "$D/$L" 2>/dev/null | ...) | head -1)
[ -n "$have" ] || continue
[ "$want" = "$have" ] && continue

|| continue appears throughout. The decision "if we can't get this lane's info, skip it" is written as moving on to the next iteration without stopping on the error. This idea of "converting a failure into an instruction for the next step" is the same philosophy as mv ... || exit 0. Errors aren't something to leave in the log—they're a signal for the next action. Once I switched to this viewpoint, my scripts became dramatically clearer.

Stuck #5: It Took a Full Day to Notice `funnel-pm` Was Firing 4x

The last "stuck" is less about technology and more about observability. I only noticed the 4x duplicate on September 17 when I counted the files in processed/ at 05:20 the next morning, on the 18th. Looking at the log, there were four entries for the same JSON, one minute apart.

For the 17 hours the problem was happening, I knew nothing. The logs accumulate in ~/dev/note-autolike/logs/codex-funnel.log, but nobody monitors them in real time. The fact that "processing was duplicated" could only be detected from duplicate filenames in processed/.

I now use this duplicate-detection one-liner in my morning check:

ls ~/dev/note-autolike/processed/ | sed 's/_[0-9]*$//' | sort | uniq -d

It strips the timestamp portion of the filenames and checks for duplicate base names. If it returns anything, that day's processing had a double execution.

Structural improvement that drives duplicates to zero (the mv pattern) and, independently, monitoring that lets you notice duplicates happened—these are problems on different layers. Build a design that doesn't break, and also have a mechanism that lets you notice a break the same day rather than the next. In an environment where automation feeds real revenue, this two-tier approach is indispensable.

Pitfalls

Environment and Configuration

launchd does not inherit PATH. Running with an empty EnvironmentVariables block will get you stuck. EnvironmentVariables in com.lily.codex-note-funnel.plist is empty, and the effective PATH at launchd startup is only /usr/bin:/bin:/usr/sbin:/sbin. The codex installed via nvm is not on this PATH. Even though run-codex-funnel.sh has set -uo pipefail, in a "command not found" situation there are cases where -u reacts late (which codex returns an empty string). The fastest diagnosis is to start it manually with launchctl start com.lily.codex-note-funnel and check the logs.
Nice: 10 and LowPriorityIO: true conflict with burst processing. Both plist settings deprioritize the script's execution when the system is under heavy load. When you want to churn through 57 JSON files right after Codex recovers, Nice: 10 is a "yield CPU to other user operations" setting, so processing is delayed. "Run quietly in the background" and "process at full speed after recovery" are in conflict. Decide which to prioritize and write the intent in a comment. The current plist chooses the former.
The default ThrottleInterval depends on the launchd version. Accumulated non-zero exits can cause StartCalendarInterval scheduled starts to be skipped. This ties directly into failure #3 in part 2 (throttling on exit 1), but to add: if 10:40 is skipped due to throttling, the next scheduled run is 16:40, six hours later. The reliable diagnosis is to go back through days where nothing was written to codex-funnel.error.log (StandardErrorPath). If you consistently use exit 0, the default ThrottleInterval setting is fine, but if you're "leaving it to the default," record that intent in a plist comment or documentation.
An mv failure due to a missing claimed/ directory and an mv failure due to race avoidance both return the same exit 0. I touched on this in failure #5 of part 2, but let me restate it as the underlying design trade-off. Giving "normal race avoidance" and "abnormal environment misconfiguration" the same exit code prevents launchd's throttle, but delays discovery of problems. Since you're paying the price of this silence, external monitoring is mandatory.

Traps Specific to the mv Claim Pattern

Cross-device mv is not atomic. POSIX rename(2) is serialized by the kernel only "on the same filesystem." If queue/ and claimed/ span different mount points (e.g., NFS, Docker volumes, external drives), the operation decomposes internally into copy + delete, and a TOCTOU window opens. This setup has both directories under ~/dev/note-autolike/, so there's no issue, but whenever you change the directory layout, verify the same filesystem with df -h.
Filename sort order becomes unstable when files are created in the same second. The lexicographic sort in ls -1 "$QUEUE_DIR"/*.json | sort | head -1 assumes the timestamp portions of the filenames differ. When batch submission creates multiple files within one second, same-name prefixes collide, and five lanes grab different files as the "first." As a result, no claim contention occurs, all lanes process simultaneously, and five Codex sessions start at once. The workaround is to include epoch seconds + PID suffix in the filename: job_$(date +%s)_$$_$(uuidgen | cut -d- -f1).json.
Leftover files in claimed/ become a silent dead-letter queue. If processing crashes midway, the file stays in claimed/. The next launchd start looks at queue/, so files in claimed/ are never picked up again. In fact, after Codex recovered on September 17, 2026, there was a day when several files were stuck in claimed/. Since adding monitoring that alerts when a file in claimed/ has gone more than an hour without an update, the lag before noticing this state has dropped to zero.
The claude-quota-guard.py fd-inheritance issue remains even after the mv claim. The first entry in ProgramArguments in com.lily.codex-note-funnel.plist is ~/.claude/scripts/claude-quota-guard.py. While this Python wrapper launches run-codex-funnel.sh via subprocess.Popen, the job appears "running" to launchd. The PID check in autolike-plist-reconcile.sh (verifying via launchctl list | awk -v l="$L" '$3==l{print $1}' that the PID is not -), which skips reload on the judgment "codex-funnel is running," correctly captures this "wrapper is alive" state as well. The mv claim eliminated the race, but you need to stay aware of how PIDs are read given that the wrapper launch is a precondition.
The glob in reconcile.sh excludes codex-note-funnel. autolike-plist-reconcile.sh targets com.lily.autolike.*.plist. com.lily.codex-note-funnel.plist is a separate file and doesn't match this glob. Since the automatic timeout-value reload that reconcile performs doesn't apply to codex-funnel, changing codex-funnel's timeout setting requires a manual launchctl bootout + bootstrap. If this is intentional, state it explicitly in a comment so that six months from now you won't be confused.

Operations and Monitoring Traps

No log rotation configured for codex-funnel.log. The plist's StandardOutPath points to ~/dev/note-autolike/logs/codex-funnel.log. launchd doesn't rotate logs. With dozens of lines per run accumulating continuously, it reaches tens of MB in a year. You need explicit rotation via newsyslog or logrotate, or line-count cap management inside the script.
Without a mechanism to automatically detect "log not updating," you won't notice a stall for a week. I've separately added monitoring that fires an alert if the update timestamp of claimed/ hasn't changed within 15 minutes after the 10:40 start. The exit 0 design maximizes compatibility with launchd, but in exchange carries the silent risk that "normal exit and silent failure are indistinguishable." This monitoring script isn't optional—it should be installed as a pair with the mv claim pattern.

Best Practices

Here are the rules that solidified after six months of running an autonomous environment at the ¥1.2M/month scale, each grounded in real code.

1. Put the mv ownership claim at the very top of the script

Run mv before argument validation, log initialization, or directory checks. If you can't get the file, there's no need to run any subsequent processing at all. This minimizes CPU and memory usage per lane while resolving contention as fast as possible.

TARGET=$(ls -1 "$QUEUE_DIR"/*.json 2>/dev/null | sort | head -1)
[ -z "$TARGET" ] && exit 0
BASENAME=$(basename "$TARGET")
mv "${QUEUE_DIR}/${BASENAME}" "${CLAIMED_DIR}/${BASENAME}" 2>/dev/null || exit 0
# ここより下は自プロセスだけが実行する

2. Log successful claims; keep failures (race avoidance) silent

With "success = logged, race avoidance = silent," grep "claimed:" codex-funnel.log | wc -l becomes today's processed count. A count log that needs no filtering dramatically reduces the implementation cost of periodic monitoring scripts. Pairing 2>/dev/null with exit 0 is a consistent pattern.

3. Return exit 0 only for "normal race avoidance" and "empty queue"

For "directory doesn't exist," "permission error," and "unexpected mv error," return exit 1. The more silent exits you have, the later you discover failures. Set up the environment with an initial deploy script, then write the script body on the premise that "only normal exits are exit 0," and maintenance cost goes down.

4. Make state visible with the three directories queue/, claimed/, done/

When the directory itself represents state, ls ~/dev/note-autolike/claimed/ | wc -l tells you "the number of stuck files" at a glance. No flag management or DB needed—ls alone is your monitoring tool.

5. Put epoch seconds + PID in filenames to stabilize sort order

FILENAME="job_$(date +%s)_$$_$(uuidgen | cut -d- -f1).json"

$$ is the PID of the generating process. Even when multiple files are created in the same second, differing PIDs make the lexicographic order deterministic. This stabilizes the premise of the mv claim pattern: all five lanes select the same file as first.

6. Standardize the "record the reason for skipping" pattern from reconcile.sh across all scripts

Lines 18–22 of autolike-plist-reconcile.sh log the reason and continue when a running job is found:

pid=$(launchctl list | awk -v l="$L" '$3==l{print $1}')
if [ -n "$pid" ] && [ "$pid" != "-" ]; then
  echo "[$(date '+%F %T')] $L 実行中(pid=$pid) のため見送り ($have -> $want)" >>"$LOG"
  continue
fi

The pattern of logging "why nothing was done" makes later failure tracing dramatically easier. Keep this habit even after the mv claim eliminates the race.

7. Separate post-crash recovery from the main script

If you build the "return leftover claimed/ files to queue/" step into the main script, a new race reappears where two processes simultaneously "return the leftover file → try to re-claim it." Stick to a design where recovery is delegated to an independent monitoring script that runs at low frequency.

8. Record the start time in a sidecar file right after claiming

mv "${QUEUE_DIR}/${BASENAME}" "${CLAIMED_DIR}/${BASENAME}" 2>/dev/null || exit 0
echo "{\"claimed_at\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}" \
  > "${CLAIMED_DIR}/${BASENAME}.claim"

Writing the start time to a .claim sidecar lets you write a recovery script that "returns files whose .claim is older than one hour to queue/." Aging detection for leftover files becomes precise.

9. Make the initial deploy script idempotent and re-runnable

mkdir -p ~/dev/note-autolike/{queue,claimed,done,logs}

Prepare a script that runs this before calling launchd's bootstrap. This structurally prevents a recurrence of failure #5 from part 2, where mv fails silently because a directory doesn't exist.

10. Separate log files between lanes to prevent interleaved lines

The plist settings StandardOutPath: codex-funnel.log and StandardErrorPath: codex-funnel.error.log point to separate files per lane, and that's correct. In the post-recovery situation where five lanes start at once, if a shared log gets interleaved, reconstructing what happened becomes impossible. Including the label name (com.lily.codex-note-funnel) in the log filename is the minimum bar.

11. Be aware of ThrottleInterval behavior and make the setting's intent explicit

If you consistently use exit 0, launchd's default ThrottleInterval causes no problems. But consciously choose to "leave it at the default" and write the reason in the plist or operations docs. This prevents the confusion of "I didn't change any settings, so why?" when trouble hits six months later.

12. Verify the same filesystem with df -h before placing queue/ and claimed/

The biggest risk to the atomicity premise is a cross-device mv. Build the habit of checking whenever you change directories into the deploy script:

queue_dev=$(df -P "$QUEUE_DIR" | awk 'NR==2{print $1}')
claimed_dev=$(df -P "$CLAIMED_DIR" | awk 'NR==2{print $1}')
[ "$queue_dev" = "$claimed_dev" ] || { echo "ERROR: cross-device mv"; exit 1; }

Summary

The problem that occurred on September 17, 2026, after Codex recovered from usage_limit_exceeded—"the funnel-pm lane processed the same JSON 4 times, one minute apart"—was an inevitability of the design. When five lanes fire at once with 57 JSON files backed up, a pickup script without mutual exclusion collides structurally, not probabilistically. Because launchd is a scheduler, not a mutex.

The fix was one line.

mv "$TARGET" "$CLAIMED_DIR/$BASENAME" 2>/dev/null || exit 0

POSIX rename(2) is serialized by the kernel on the same filesystem. Fundamentally unlike a "check, then acquire" PID check, the check and the acquisition complete within a single system call. Unlike flock with its fd-inheritance problem, the lock is decoupled from the life and death of the process. Since the file itself becomes the claim token, a leftover file after a crash remains in a state that's human-readable.

The "do nothing if running" PID-check pattern in autolike-plist-reconcile.sh and the mv claim pattern here were born from the same philosophy. The only difference is "whether there's a TOCTOU window." Both are designs that make the choice "exit if someone else is processing" in the very first operation.

To keep an autonomous environment stable over the long term, the only way is to keep stacking up these kinds of "structures that don't permit contention in the first place." Procedural fixes shrink the wound; environmental fixes eliminate the event. The reason I could go from ¥0 to ¥1.2M a month in six months is that on the days things broke, I repeatedly chose "fix the structure" over "next time I'll do it differently."

The full picture of the system, the breakdown of the ¥1.2M/month, and the 30-day setup guide are compiled in a paid note (Japanese).
📕 How to actually earn with a Claude Code autonomous environment — the system, real examples, getting started, and support

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

2 Pitfalls in Priority Probes: Letting One Real Request Through an Open Circuit Breaker

Lily — Sun, 20 Sep 2026 00:00:06 +0000

A circuit breaker that stops everything is easy to reason about — until the one job that can't afford to wait gets stopped along with everything else. In my setup, that job was posting: it fell to a 43% execution rate (231 runs vs. 308 skips) because engagement jobs had burned through the quota first. This post is about the fix inside claude-quota-guard.py — claim_priority_probe and run_job, which let a single real request through while the circuit is still open — and the two pitfalls I hit along the way, one of which silently dropped a daily job for four days (2026-09-13 to 16).

Last time, I wrote about how the gate blocks one row, not the whole batch. Before I get to quota-catchup.py — the script that re-runs everything once it detects quota recovery — I want to look at how the SKIPPED markers it reads are actually produced.

The problem: the circuit stops every job equally, but posting can't wait

When claude-quota-guard.py detects the quota limit, it returns EXIT_CIRCUIT_OPEN (75) until open_until and stops all 15 guarded jobs uniformly. A comment in the code explains why that wasn't enough:

# 🔴 2026-08-21: circuit が開くと全ジョブが一律で止まるため、消費の大半を占める
# 返信/エンゲージ系がクォータを使い切った巻き添えで「投稿」まで停止していた。
# 実測(launchd.log 累計): xpilot.autopost は 231実行/308スキップ＝実行率43%で、
# threadspilot.engage(64%) より優先度が低い扱いになっていた。投稿はその時間帯を逃すと
# 二度と埋まらないので、--priority を付けたジョブだけは circuit が開いていても
# この間隔で1回だけ試行を許す。試行が通ればクォータ回復の早期検知にもなる
# (従来は open_until まで盲目的に待つだけだった)。

(claude-quota-guard.py:18-24)

The measured result: stopping everything uniformly meant the posting job (xpilot.autopost) got caught in the crossfire of the quota-hungry engagement jobs and dropped to a 43% execution rate — effectively lower priority than threadspilot.engage (64%). On top of that, open_until is determined either by "the reset time parsed from the limit message" or by "a 6-hour cooldown" (record_claude_result), so even if the actual quota comes back earlier, the circuit dutifully stays open until that time. For a job like posting, where "if you miss the time slot, it never gets filled," that's not something you can ignore.

The design: give `--priority` jobs one attempt every 30 minutes

The fix is: "even while the circuit is open, let priority jobs — and only priority jobs — send one real request at a fixed interval." The caller passes --priority when handing a command to run_job.

def run_job(label: str, command: list[str], priority: bool = False) -> int:
    if not command:
        print("claude quota guard: --job requires a command after --", file=sys.stderr)
        return 2
    status = circuit_status()
    probe_claimed = False
    if status["is_open"]:
        if not (priority and claim_priority_probe()):
            print(
                "CLAUDE_QUOTA_JOB_SKIPPED "
                f"job={label} reason={status['reason']} remaining={status['remaining_seconds']}s ts={now()}",
                file=sys.stderr,
            )
            return 0
        probe_claimed = True
        print(
            "CLAUDE_QUOTA_JOB_PRIORITY_PROBE "
            f"job={label} reason={status['reason']} remaining={status['remaining_seconds']}s ts={now()}",
            file=sys.stderr,
        )

(claude-quota-guard.py:471-490)

Even with priority set, whether the job can actually attempt anything is decided by claim_priority_probe().

def claim_priority_probe() -> bool:
    """circuit が開いている間、優先ジョブに試行権を1つ渡す。

    間隔は全優先ジョブで共有する(=1本が使ったら次の枠まで他も待つ)。上限に本当に
    達している間に何本も叩いてもクォータは戻らないため、叩く回数自体を絞る。
    """
    if PRIORITY_PROBE_INTERVAL <= 0:
        return False
    with locked_state() as state:
        last = int(state.get("last_priority_probe") or 0)
        current = now()
        if current - last < PRIORITY_PROBE_INTERVAL:
            return False
        state["last_priority_probe"] = current
        return True

(claude-quota-guard.py:454-468; PRIORITY_PROBE_INTERVAL defaults to 1800 seconds at claude-quota-guard.py:25)

last_priority_probe is a single timestamp stored in locked_state() (a JSON-persisted state guarded by fcntl.flock). The key point is that this value lives per circuit, not per job: it doesn't care who probed, only "has it been 30 minutes since anyone last probed?"

The side that wins the attempt passes CLAUDE_QUOTA_PRIORITY_PROBE=1 in the environment of the child process it launches via subprocess.run.

    guard = str(Path(__file__).resolve())
    env = os.environ.copy()
    env["CLAUDE_AUTOMATION_GUARD"] = "1"
    env["CLAUDE"] = guard
    env["CLAUDE_BIN"] = guard
    if probe_claimed:
        # 内側の run_claude に「プローブとして走っている」ことを伝える(これが無いと circuit で即 75 になる)
        env["CLAUDE_QUOTA_PRIORITY_PROBE"] = "1"

(claude-quota-guard.py:491-498)

The receiver of this flag is run_claude. When the job command internally invokes the real claude binary, PATH has been rewired so the call goes through the guard itself — which means a second circuit_status() check runs inside the child process.

def run_claude(arguments: list[str]) -> int:
    status = circuit_status()
    if status["is_open"]:
        # run_job が優先プローブを claim した子プロセスだけは circuit を素通りして実 claude を叩く。
        # 結果は record_claude_result に入るので、上限文なら circuit が延び、成功なら閉じる。
        if os.environ.get("CLAUDE_QUOTA_PRIORITY_PROBE") != "1":
            print(
                "CLAUDE_QUOTA_CIRCUIT_OPEN "
                f"reason={status['reason']} remaining={status['remaining_seconds']}s",
                file=sys.stderr,
            )
            return EXIT_CIRCUIT_OPEN
        print(
            "CLAUDE_QUOTA_PRIORITY_PROBE_PASS "
            f"reason={status['reason']} remaining={status['remaining_seconds']}s",
            file=sys.stderr,
        )

(claude-quota-guard.py:416-432)

If you forget to propagate the environment variable, run_job wins the attempt, but the inner run_claude checks the circuit again and immediately returns EXIT_CIRCUIT_OPEN. The one line that gets you through both layers of the gate is env["CLAUDE_QUOTA_PRIORITY_PROBE"] = "1".

And the result of the actual call flows into record_claude_result as usual. If the limit message still comes back, open_until is extended; if the call succeeds, the circuit closes. Whether the probe fails or succeeds, that single result directly determines the circuit's next state — which is what makes this "one real request let through."

Pitfall 1: "one every 30 minutes" is a counter shared by all priority jobs

As the docstring on claim_priority_probe says, PRIORITY_PROBE_INTERVAL is shared across all jobs, not tracked per job. Even if several --priority jobs are scheduled inside the same 30-minute window, the moment the first one passes claim_priority_probe(), last_priority_probe is updated. Every subsequent job evaluates to False at priority and claim_priority_probe() and falls through to SKIPPED without ever touching the real client.

This is intentional. While the limit is genuinely in effect, hammering it with multiple requests won't bring the quota back, so the design throttles the number of attempts themselves. Operationally, though, if you forget that "having multiple priority jobs does not mean each gets its own 30-minute opportunity," you will lose time wondering "why does this one job never get a turn to verify recovery?"

Pitfall 2: when the probe fails, it silently lands as `RAN exit≠0`

This is the main subject of this post. After subprocess.run, run_job checks whether the probe came up empty and emits a different marker accordingly.

    try:
        result = subprocess.run(command, env=env, check=False)
    except OSError as exc:
        print(f"claude quota guard job={label}: {exc}", file=sys.stderr)
        return 127
    if probe_claimed and result.returncode != 0 and circuit_status()["is_open"]:
        # 優先プローブが上限のまま空振りした。RAN exit≠0 のまま残すと quota-catchup.py
        # （最新マーカー=SKIPPED だけを再実行）から漏れ、復帰後も当日分が欠番になる
        # （2026-09-13〜16 の note2-daily / codex-note-funnel 実測）。SKIPPED として記録する。
        print(
            "CLAUDE_QUOTA_JOB_SKIPPED "
            f"job={label} reason=priority-probe-quota exit={result.returncode} ts={now()}",
            file=sys.stderr,
        )
        return result.returncode
    print(
        f"CLAUDE_QUOTA_JOB_RAN job={label} exit={result.returncode} ts={now()}",
        file=sys.stderr,
    )
    return result.returncode

(claude-quota-guard.py:499-518)

Before this branch existed, the code only looked at the fact that the probe had won the attempt and actually launched a child process, and fell straight through to the trailing CLAUDE_QUOTA_JOB_RAN. Even when the probe hit the limit again and ended with exit≠0, what remained in the log was CLAUDE_QUOTA_JOB_RAN job=... exit=1.

The problem is that this RAN marker means "already done" as far as quota-catchup.py is concerned. Here is latest_job_marker, which quota-catchup.py uses to narrow down re-run candidates:

def latest_job_marker(paths: list[Path], label: str) -> Optional[Tuple[str, int]]:
    """Return the newest timestamped skip/run marker for one launchd label."""
    marker_pattern = re.compile(
        r"CLAUDE_QUOTA_JOB_(SKIPPED|RAN)\s+job="
        + re.escape(label)
        + r"(?=\s|$).*\bts=(\d+)(?=\s|$)"
    )
    ...


def latest_marker_is_today_skip(paths: list[Path], label: str, today: datetime.date) -> bool:
    marker = latest_job_marker(paths, label)
    if marker is None:
        return False
    marker_type, timestamp = marker
    return marker_type == "SKIPPED" and datetime.fromtimestamp(timestamp).astimezone().date() == today

(quota-catchup.py:160-193)

As you can see, this check only looks at the marker type (SKIPPED or RAN) and never inspects the exit code on the RAN side. Even if the probe failed with exit=1, as long as the last log line is CLAUDE_QUOTA_JOB_RAN, latest_marker_is_today_skip returns False and the job quietly drops out of find_candidates' re-run set.

The consequence: even after the circuit really closes, the job sits there with the wrong record — "already RAN today" — until its next scheduled time (the following morning, for instance). The comment records the real-world impact: note2-daily and codex-note-funnel both lost their daily run through this path between 2026-09-13 and 16.

The fix is simple. Check the condition "the probe came up empty while the circuit was open" (probe_claimed and result.returncode != 0 and circuit_status()["is_open"]) first, and only in that case emit CLAUDE_QUOTA_JOB_SKIPPED reason=priority-probe-quota instead of CLAUDE_QUOTA_JOB_RAN. The function's return value stays result.returncode, unchanged. launchd's LastExitStatus still records the actual failure correctly, while only the log marker that quota-catchup.py reads gets relabeled as "still needs a retry." That's the separation of concerns.

Note: The exit code and "should this be retried?" are separate axes. This bug happened because the two had been crammed into a single RAN marker, and the quota-catchup.py side never anticipated the exit≠0 case. When designing log markers, it's safer not to let "what actually happened" and "what the downstream batch should do next" share the same string.

Summary

The circuit stops everything uniformly, but jobs that can't recover a missed opportunity — like posting — can hold one attempt every 30 minutes via --priority
The attempt right is managed by claim_priority_probe through last_priority_probe in locked_state, and it is shared across all priority jobs, not per job. If multiple priority jobs land in the same window, there's only one opportunity
The child process that wins the attempt must receive CLAUDE_QUOTA_PRIORITY_PROBE=1. Forget it, and the inner run_claude rejects the call at the second circuit check
When the probe comes up empty, record it as CLAUDE_QUOTA_JOB_SKIPPED reason=priority-probe-quota, not CLAUDE_QUOTA_JOB_RAN exit≠0. The re-run check in quota-catchup.py only looks at the marker type and never at the exit code
Design log markers so that "what actually happened" and "what the downstream batch should do" are kept separate

Next time, I'll cover how quota-catchup.py picks up these SKIPPED markers and decides how much to re-run after recovery.

If you run a circuit breaker in front of your own scheduled jobs: does your downstream retry logic distinguish "ran and failed" from "never really got a chance"?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

13 Days of Silence: How One Bad Draft Froze an Entire DM Lane

Lily — Sat, 19 Sep 2026 11:00:05 +0000

Last time I wrote about a Terminal-watching daemon that auto-clicks Computer Use approval dialogs. This post is the same genre—an unattended job dying quietly—but the culprit wasn't a gap in monitoring. It was the design itself. founder-scout's automated DM lane sent zero messages for 13 days because of a single bad record, and nobody noticed.

The problem: the exit code was there, but nobody saw it

founder-scout has a followers lane that sends a first DM to new followers of @bokuwalily. When I checked on 2026-09-10, I found that the last successful send was 8/28 at 16:35. Thirteen days, not a single DM.

The queue (state/followers_queue.jsonl) kept growing with every run, from 336 entries to 390. In other words, drafts were being generated daily and the send script was launching daily. Still zero sends. The logs recorded process.exit(3) day after day, but that abort path was never wired to a Discord alert, so it never crossed anyone's eyes.

Root cause: turning "one bad record" into "everything stops"

On 8/28 I wired a number-gate (checkColdOutbound) into all seven outbound paths to stop drafts from leaking my actual download counts or revenue figures. At that time I designed followers_send.mjs and dm.mjs so that if even one draft in the batch leaked a number, the script would immediately process.exit(3) and halt everything.

The leaking draft was addressed to @kouAI_work and contained the phrase "downloads on August 25…". That one record was never written to the ledger, so every run picked the same six entries from the head of the queue, and every run stopped on the same one. The sendable drafts behind it were never evaluated once—they just kept piling up in the queue.

Note
The same day, a different incident of the same shape happened on the dm.mjs side. Composer timeouts for the same three people counted as three consecutive failures and tripped the stop guard, so the four people behind them didn't get sent for two runs in a row. Different cause, but the same pattern: the head of the queue gets pinned → the job stops there every time.

Fail-closed is for stopping that one record

This is the lesson. Pre-send checks like the number-gate or composer validation exist to keep one dangerous message from going out. But if you implement them as "any anomaly → exit the whole thing," then unless that anomaly heals itself, the entire lane stops permanently. An unrecorded anomaly stays parked at the head of the queue, so it never heals itself.

The fix has two parts.

1. Record the dropped row in the ledger and treat it as exhausted

Here's the relevant section of src/followers_send.mjs.

// src/followers_send.mjs:84-97
// 送信前ゲート: 自分のDL数・売上の実数が入っている下書きは「その1件だけ」外して台帳に記録する。
// 1件で全体を止めると、その下書きがキュー先頭に残り続けてレーンごと止まる
// （実測2026-08-28〜09-10: kouAI_work 1件で13日間0通）。
{
  const gateAt = new Date().toISOString();
  const leaks = drafts
    .map((t) => ({ who: t.handle || '?', bad: checkColdOutbound(t.dm || '') }))
    .filter((x) => x.bad.length);
  for (const l of leaks) {
    console.error(`number gate: 除外 @${l.who} — ${l.bad.map((b) => b.why).join('/')}`);
    appendSync(SENT, { handle: l.who, ok: false, reason: 'number-gate: ' + l.bad.map((b) => `${b.why}(${b.hit})`).join('/'), follow: 'skipped', at: gateAt, lane: LANE });
    exhausted.add(l.who);
  }
}
const sendable = drafts.filter((d) => !exhausted.has(d.handle));
const batch = sendable.slice(0, Math.min(perRun, laneLeft, accountLeft));

process.exit(3) is gone. Instead, it writes ok:false, reason:'number-gate: …' to the SENT ledger, marks only that one entry as exhausted, and the rest proceed to the batch as usual. dm.mjs got exactly the same treatment.

// src/dm.mjs:82-94
// 送信前ゲート: 数字漏れの下書きは「その1件だけ」外して台帳に記録する。全体を止めると
// 先頭に残り続けてレーンごと止まる（followers レーンで13日間0通の実測あり）。
{
  const gateAt = new Date().toISOString();
  const leaks = queue
    .map((t) => ({ who: t.handle || '?', bad: checkColdOutbound(t.dm || '') }))
    .filter((x) => x.bad.length);
  for (const l of leaks) {
    console.error(`number gate: 除外 @${l.who} — ${l.bad.map((b) => b.why).join('/')}`);
    recordSent({ handle: l.who, ok: false, reason: 'number-gate: ' + l.bad.map((b) => `${b.why}(${b.hit})`).join('/'), follow: 'skipped', at: gateAt });
    exhausted.add(l.who);
  }
}
const sendable = queue.filter((r) => !exhausted.has(r.handle));

Because it's now in the ledger, from the next run onward this entry is excluded at the point where the exhausted set is built (followers_send.mjs:49-51 and dm.mjs:50-52, where the string number-gate is picked out of reason and added to exhausted). The moment it was recorded, this anomaly became one that "automatically disappears on the next run."

2. Move anyone with a failure history to the back of the queue

The other fix is a reorder so that the three-consecutive-failures guard (break when consecutiveFailures >= 3) doesn't keep hitting the same faces every time.

// src/followers_send.mjs:81-83
// 一度失敗した相手は後ろへ回す。先頭の同じ数人が毎run失敗すると「3連続失敗で停止」に
// 当たり、後ろの送れる人まで道連れになる（実測2026-09-10: 同じ3人で2run連続0通）。
.sort((a, b) => (attempts.get(a.handle) || 0) - (attempts.get(b.handle) || 0));

It just sorts by attempts (the number of past send attempts) in ascending order. Now people who are structurally destined to keep failing—DMs not open, composer timing out—no longer monopolize the head of the queue, and the sendable people behind them no longer get dragged down every run.

Monitoring: an exit code alone doesn't mean "it's working"

The reason this went unnoticed for 13 days is that the logs from the broken abort path only lived locally. Seeing "loaded" in launchctl list or pm2 list only means the job is executing; whether it's producing results is a separate question.

You need to check three things together: ① is the job loaded, ② what's the distribution of exit codes in recent logs, ③ what does the actual platform show (in this case, the real count of DMs sent). In this incident, ① and ② looked fine, and it was only when I checked ③ that "zero DMs in 13 days" came to light. On top of that, I made every run log laneToday (the lane's send count for the day) and added an operational rule: three consecutive days at 0 is an anomaly. The log line at followers_send.mjs:101 is what feeds that.

console.log(`queue=${sendable.length} laneToday=${laneToday}/${perDay} accountToday=${sentToday.length}/${cfg.limits.dmPerDay} thisRun=${batch.length} held=${held}`);

Pitfalls I hit

Designing process.exit(3) on a single anomaly → an unrecorded anomaly stays parked at the head of the queue and the lane stops permanently
Forgetting to wire a Discord alert to the abort path → an exit code in a log nobody reads is the same as silence
Skipping the dropped row without writing it to the ledger → the next run re-evaluates the same draft in the same position and gets stuck on the same one every time
The three-consecutive-failures guard hitting the same pinned people every run → simply moving anyone with a failure history to the back keeps the sendable people behind them from being taken down with them
Judging "running" from the loaded state in launchctl list alone → it's only healthy once you've checked the exit codes and the real counts on the platform side

Takeaways

A fail-closed gate is written to stop that one record, not to stop the lane
Halt-everything plus not-recorded equals permanent stop unless it clears on its own next run. Whenever you write a halt, ask: "Will this stop condition naturally disappear on the next run?"
The fix was a two-line prescription: write the dropped row to the ledger with ok:false, reason and mark it exhausted, and sort anyone with a failure history to the back of the queue by ascending attempts
The loaded state in launchctl list or an exit 0 isn't enough for monitoring. Only when you've looked at the distribution of exit codes plus the real counts on the platform can you say "it's working"

Do you have an unattended job right now whose "healthy" status you've only ever verified from the process list—and when did you last check the actual output on the other end?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

3 Weeks of Silent Failure: How One Missing `--model` Flag Drained My Interactive Quota Across 4 Jobs

Lily — Sat, 19 Sep 2026 05:00:06 +0000

I went from ¥100k a month as a student to ¥600k juggling side gigs, then to ¥0 after a layoff—and six months of building an autonomous Claude Code environment brought me to ¥1.2M a month in revenue. The difference wasn't working harder. It was betting on "grow the environment" instead of "do the work."

Why This Setup Works

Look up how to make money on the side and you'll always land on the same three options: sell your skills, make content, or build a service. They're all correct. But they share one ceiling: your time is finite.

Back when I was juggling gigs at ¥600k a month, I genuinely hit the wall on working hours. Every article, every DM, every proposal needed my own hands. The more clients I had, the more quality slipped. I couldn't add more, and I couldn't let quality drop—and that's exactly when the layoff came. I decided I wasn't going to do it the same way again.

For the restart, I chose a different approach: build the environment that runs without me first. Today, launchd jobs fire at 8:00 and 10:35 every morning, generate a stock of articles, and post a notification to Discord. Whether my Mac is open or I'm asleep, the article stock keeps growing.

There are three reasons this setup is strong.

Time accumulates. Human work consumes time. Environment work produces it. Every job run stacks up as inventory that doesn't disappear. Even when I'm sick and can't do anything for three days, the jobs quietly keep growing the stock. "As long as they're running correctly," that is—and that "if" is the whole subject of this post.

The scale changes by an order of magnitude. A human can write a few articles a day at most. Jobs run in parallel, as many as you like. Right now, four lanes—article generation, daily PDCA, structuring external data, and triaging incoming DMs—all run during the same night. I passed the throughput of a single person because agents were lined up side by side.

How deeply you understand quota structure changes. Someone who uses Claude Code as a "tool to ask the AI" and someone who wires it in as "part of the infrastructure" have fundamentally different monthly throughput. To use it as infrastructure, you have to know exactly how quota is shared and how it's consumed, or you will get stuck. This outage happened precisely in that gap of knowledge.

The Overall Flow

Here's the skeleton of the content-generation pipeline up front. I'll dissect the outage in the second half, so having the big picture first will make it easier to follow.

launchd
 ~/Library/LaunchAgents/com.shun.article-daily.plist
 起動タイミング: 08:00 / 10:35（毎日）
     │
     ▼
 ~/.claude/scripts/claude-quota-guard.py
     --job com.shun.article-daily
     │  ← クォータ残量のガード層
     ▼
 ~/.discord/run-and-notify.sh
     "zenn" "Zenn記事ストック生成"
     │  ← Discord通知ラッパー
     ▼
 ~/.claude/scripts/article-daily-stock.sh  apply
     │  ← 実際の生成ロジック
     ▼
 claude -p "プロンプト..." [--model ???]
     │
     ├─ --model を明示した場合
     │       指定モデルの独立した枠で実行
     │       → 記事テキストが正常に返る ✅
     │
     └─ --model を省略した場合
             対話セッションの「既定モデル枠」を共有消費
             → 枠が切れると以下の1行だけ stdout に吐いて終了:
                 "You've reached your Fable limit."
             → exit code は非0
             → ただしログには「1行返ってきた」としか残らない
             → [ -s output ] 等の素朴なチェックを通り抜ける ❌
             → 何週間でも誰も気づかない ← 実際に3週間気づかなかった

The bottom of the diagram is the heart of the outage. claude -p does not error when you omit --model. It quietly goes off and consumes the interactive session's quota. When that quota runs out, it returns a single-line message and exits. That one line gets mistaken for "there was output," and detection slips through.

The plist Implementation

Here's an excerpt from the actual com.shun.article-daily.plist (real paths replaced with ~).

<key>ProgramArguments</key>
<array>
  <string>~/.claude/scripts/claude-quota-guard.py</string>
  <string>--job</string>
  <string>com.shun.article-daily</string>
  <string>--</string>
  <string>/bin/bash</string>
  <string>~/.discord/run-and-notify.sh</string>
  <string>zenn</string>
  <string>Zenn記事ストック生成</string>
  <string>/bin/bash</string>
  <string>~/.claude/scripts/article-daily-stock.sh</string>
  <string>apply</string>
</array>

<key>StartCalendarInterval</key>
<array>
  <dict>
    <key>Hour</key><integer>8</integer>
    <key>Minute</key><integer>0</integer>
  </dict>
  <dict>
    <key>Hour</key><integer>10</integer>
    <key>Minute</key><integer>35</integer>
  </dict>
</array>

<key>StandardOutPath</key>
<string>~/.claude/logs/article-daily.out.log</string>
<key>StandardErrorPath</key>
<string>~/.claude/logs/article-daily.err.log</string>

<key>ProcessType</key><string>Background</string>
<key>LowPriorityIO</key><true/>
<key>Nice</key><integer>10</integer>

ProcessType: Background, LowPriorityIO: true, and Nice: 10 go in as a set. By explicitly telling the OS I/O scheduler "this is a low-priority background task," they keep the job from interfering with foreground work (opening Figma, looking something up in Safari).

The call chain is four levels deep: launchd → quota-guard → run-and-notify → article-daily-stock. claude -p is invoked inside article-daily-stock.sh, and whether --model is present there is the whole question.

Four Lanes Surfaced at Once on September 3, 2026

In my case, the same omission surfaced in four jobs simultaneously. Quoting the record left in my knowledge base:

Lane	Symptom	Damage window
`gen-column.mjs` (LINE column generation)	exit 1 every morning with the one-line "Fable limit". Stock dwindled to 2 remaining	2026/8/13–9/3 (3 weeks)
`triage` (incoming DM classification)	Every quota-exhausted run fell through to "unclassifiable"; not a single incoming DM had been auto-replied since 9/2	9/2–
`brand-404` (IG daily PDCA)	Zero output due to `Fable limit`. That day's action items and daily report never generated	9/2
`hosei-grad-planner` (PDF structuring)	All 3 runs at 00:21 / 04:30 / 10:30 produced zero structured data	9/3

For three weeks, gen-column.mjs burned through its stock every morning while "looking like it was working fine." It was only discovered when the stock hit 2 remaining. The job's log recorded "1 line returned." Nobody was reading what that one line said.

"Four lanes at once" is an important fact too. If just one job breaks, it's easy to notice. When several go silent at the same time, the unease of "everything's too quiet" fades. If anything, it pulls you toward the misreading that "automation is working and my hands are free."

The root cause is a code defect: not writing --model. Combined with lax log monitoring that treats "one line = success," it went undetected for three weeks. Each hole on its own would have been easy to spot, yet stacked together they created a blind spot.

I'll dissect that structure in the second half: why omitting --model is so hard to notice, how to build one script that makes detection reliable, and how to implement a grep gate that mechanically rejects any existing plist with the flag missing.

Implementation Details

Where `--model` Goes and How to Write It

Looking at article-daily-stock.sh after the post-outage fix, there are two places where claude -p is called: topic planning (auto-generating the next topic when the queue is empty) and article body generation. Both have --model.

# キューが空のとき: 実作業からネタを自動立案（article-daily-stock.sh 327-330行目）
TOPIC_JSON=$(run_to 600 "$CLAUDE" -p "$REPLENISH_PROMPT" \
  --strict-mcp-config --mcp-config '{"mcpServers":{}}' \
  --model "${ARTICLE_MODEL:-sonnet}" --effort high \
  --output-format text --allowedTools "Read,Grep,Glob,Bash" --max-turns 20 2>>"$LOG")

# 記事本文の生成（article-daily-stock.sh 397-401行目）
run_to 1500 "$CLAUDE" -p "$PROMPT" \
  --strict-mcp-config --mcp-config '{"mcpServers":{}}' \
  --model "${ARTICLE_MODEL:-sonnet}" --effort high \
  --output-format text --allowedTools "Read,Grep,Glob,Write,Bash" --max-turns 40 >> "$LOG" 2>&1 \
  || log "WARN: claude -p exit非0（検証で弾く）"

The key is the "${ARTICLE_MODEL:-sonnet}" form. Even if the ARTICLE_MODEL environment variable is undefined, it falls back to sonnet, so the "forgot it and fell through to the default quota" accident can't happen. It also satisfies a separate operational rule: "don't hardcode model IDs in code." If you embed a model ID as a string, you end up rewriting every script when the vendor retires an alias. Make it switchable through a single variable and you can flip every lane with one line: launchctl setenv ARTICLE_MODEL haiku.

One more thing: I always include the combination --strict-mcp-config --mcp-config '{"mcpServers":{}}'. Launching from launchd doesn't go through a login shell, so the .mcp.json in the home directory gets read. If it contains external MCP server configuration, the batch may hit external networks unintentionally. Explicitly passing an empty server config completely blocks external MCP calls inside the batch.

Designing `strict_quota_message`—The 400-Character Wall

Inside claude-quota-guard.py there's a function called strict_quota_message(). This is the single most important piece of detection logic for this incident.

# claude-quota-guard.py 41-47行目
STRICT_QUOTA_PATTERNS = (
    r"(?:you(?:'ve| have) )?hit your (?:weekly |5-?hour |usage |session )?limit",
    r"claude (?:ai )?usage limit reached",
    r"approaching your (?:weekly |usage )?limit",
)
STRICT_QUOTA_MAX_CHARS = 400

# claude-quota-guard.py 159-170行目
def strict_quota_message(text: str) -> bool:
    """成功終了(exit 0)の本文が「上限メッセージそのもの」かを判定する。

    上限本文は短く、その一文だけが返る。記事本文にたまたま同じ語が出ても発火させない。
    """
    trimmed = text.strip()
    if not trimmed or len(trimmed) > STRICT_QUOTA_MAX_CHARS:
        return False
    return any(
        re.search(pattern, trimmed, flags=re.IGNORECASE)
        for pattern in STRICT_QUOTA_PATTERNS
    )

The STRICT_QUOTA_MAX_CHARS = 400 cap is the linchpin of the design. When Claude hits a limit, the string it returns is a single sentence like "You've reached your Fable limit." or "You've hit your weekly limit."—a few dozen characters at most. A properly generated article body, on the other hand, is almost never under 400 characters. This length asymmetry is what mechanically separates "this is the limit message itself" from "this is an article body that mentions limits."

The regex patterns are designed with the same thinking. The loose QUOTA_PATTERNS (used when exit is non-zero) and the strict STRICT_QUOTA_PATTERNS (used when exit is 0) are kept separate to prevent a false positive where an article whose subject happens to be "quota" trips my own circuit breaker. In fact, during the period when this check was still loose, I had an incident where "an article written about quota tripped the circuit breaker," which is how I settled on the current implementation that switches detection logic based on exit 0 vs. non-zero.

The Postcondition Gate—"There Was Output" Is Not Success

article-daily-stock.sh includes a function called article_ok() (lines 133–181). Even if claude -p returns a healthy exit code, no article is accepted unless it passes this function.

# article-daily-stock.sh 133行目
MIN_ARTICLE_BYTES=1200

# article-daily-stock.sh 173-181行目
article_ok() {
  local f="$1"
  [ -s "$f" ] || return 1
  [ "$(stat -f%z "$f" 2>/dev/null || echo 0)" -ge "$MIN_ARTICLE_BYTES" ] || return 1
  grep -qE '^title:' "$f" || return 1
  frontmatter_title_ok "$f" || return 1
  grep -qiE 'request timed out|不明な商品|TODO: *本文|\(生成失敗\)' "$f" && return 1
  return 0
}

At least 1200 bytes, a title: frontmatter key present, a title of 70 characters or fewer, and no timeout phrasing mixed in—only when all four conditions are met is the run recorded as "generation succeeded." This validation block sits immediately after claude -p, and if the file doesn't meet the bar, it's deleted and the script exits (lines 411–415). So as not to waste the day's slot, the marker file (~/.claude/logs/.article-daily-done-${TODAY}) is designed to be created "only after validation passes" (line 509).

A postcondition is evidence that "generation completed correctly." Not whether a file exists, but whether the file's contents have the expected quality, verified mechanically. Since switching to this mindset, the misreading of "the job ran = output was stocked" has gone away.

A grep Gate That Mechanically Catches the Omission

It's hard for a human to find a missing --model. The more files you have, the more visual inspection breaks down. The following one-liner sweeps across ~/.claude/scripts/ and ~/Library/LaunchAgents/ and extracts "lines that have claude -p but no --model."

# plist配下のジョブスクリプトで --model を書き忘れている箇所を抽出する
grep -rn "claude -p" ~/.claude/scripts/ \
  | grep -v -- '--model' \
  | grep -v '^Binary'

Ideally this would go into CI or a pre-commit hook, but launchd jobs aren't connected to normal CI. Instead I use the following approach. During the weekly launchctl list | grep com.shun check that confirms all jobs exist, I run the grep above alongside it. Zero lines back means everything's safe. Even one line back means something's missing somewhere.

In cases of multi-level calls—a script called from a plist calls a script which calls another—you need to follow the chain recursively down to the leaf script.

# 多段呼び出しも含めて再帰的に検索
grep -rn "claude -p\|claude --print" \
  ~/.claude/scripts/ \
  ~/dev/ \
  --include="*.sh" --include="*.mjs" --include="*.js" --include="*.py" \
  | grep -v -- '--model' \
  | grep -v '#.*claude -p'   # コメント行は除外

Without narrowing target extensions via --include, the results balloon. The four types .sh .mjs .js .py covered 90% of my environment.

Where I Got Stuck

Three Weeks Without Noticing, Until Stock Hit 2

gen-column.mjs is a job that generates one LINE column every morning. It was supposed to raise an alert once stock dropped below 5, but for some reason it never fired. When stock hit 2, I was manually looking at Discord notifications and finally thought, "something's wrong."

Digging through the logs, every morning looked like this.

[2026-08-14 08:03:12] claude -p "..." 実行
[2026-08-14 08:03:19] 出力: "You've reached your Fable limit."
[2026-08-14 08:03:19] 出力文字数: 32
[2026-08-14 08:03:19] ジョブ完了

The job "completed" and returned exit 0, so quota-guard's tally recorded it as "success." The stock-count logic was adding "normal completion = 1 article generated," so the stock count appeared to be increasing. In reality, not a single one was added.

What exposed it was counting the actual number of files in stock directly with ls. It was 2. The gap between the job's success count and the real file count was the evidence.

After that, I made two changes: stock monitoring uses the real file count instead of a counter, and a postcondition check runs after generation. A counter is a declaration of intent to "generate"; it is not evidence that generation "succeeded."

The Wording Changed and the Circuit Breaker Let It Straight Through

On September 2, 2026, it came to light that QUOTA_PATTERNS in claude-quota-guard.py did not include session limit.

That day's error message looked like this.

You've hit your session limit · resets 6:50pm (Asia/Tokyo)

This is the wording for the "5-hour session limit." It's phrased differently from the earlier "weekly limit" or "Fable limit." Because QUOTA_PATTERNS at the time didn't have this pattern, quota detection slipped through. As a result, every job that day—including 3 note lanes—kept firing blanks against the limit, and 306 lines of wasted Claude calls piled up over 4 hours.

The fix was adding r"resets?\s+(?:at\s+)?\d{1,2}(?::\d{2})?\s*[ap]m" to QUOTA_PATTERNS (claude-quota-guard.py line 37). It's a pattern that catches the combination of the verb "resets" and a time in am/pm format. That line is still in the code today, with a comment recording the live account: "2026-09-02: the 5-hour limit comes back in the form of ~."

The lesson here is the principle that quota detection must not depend on the vendor's wording. Vendors change wording. They don't announce it. Anthropic has already used multiple phrasings: "You've reached your limit," "You've hit your Fable limit," "You've hit your session limit." Pattern matching needs to be written on the "structural features of the wording," not "exact match of the wording."

The Circuit Breaker Stopped the Posting Jobs Too

On August 21, 2026, when the circuit breaker opened (= detected quota exhaustion), a problem arose where not only generation jobs but posting jobs stopped as well.

There's a record in the comments of claude-quota-guard.py.

# 🔴 2026-08-21: circuit が開くと全ジョブが一律で止まるため、消費の大半を占める
# 返信/エンゲージ系がクォータを使い切った巻き添えで「投稿」まで停止していた。
# 実測(launchd.log 累計): xpilot.autopost は 231実行/308スキップ＝実行率43%で、
# threadspilot.engage(64%) より優先度が低い扱いになっていた。投稿はその時間帯を逃すと
# 二度と埋まらないので...

Timing is everything for posting. A post that misses its 8 a.m. slot won't have the same effect if it goes up at night. Its priority is fundamentally different from a generation job's.

The cause was a design in which "the circuit breaker stops all jobs without distinction." The generation jobs were the ones that exhausted the quota, yet the posting jobs got taken down as collateral.

The remedy was adding a --priority flag (it remains in the current code as the priority: bool argument of run_job()). Priority jobs are allowed, via claim_priority_probe(), to attempt one run per PRIORITY_PROBE_INTERVAL (default 30 minutes) even while the circuit is open (claude-quota-guard.py lines 454–468). If the attempt succeeds, it's judged that "quota has recovered" and the circuit closes. With this design, one mechanism achieves two goals: "posting doesn't stop, and recovery is detected early."

Exit 0, but the Output Was a Quota Message

The hardest pattern to find was "exit code 0 with a limit message on stdout." This happens not only when --model is omitted, but also under certain conditions where claude returns the quota message as a "success."

The original quota-guard recorded exit code 0 as "success" and leaned toward closing the circuit. So when a limit message came back with exit 0, the circuit didn't open, the job was recorded as "success," and the actual output was just the one limit line—a state that could persist for cycle after cycle.

Introducing strict_quota_message() (the 400-character check described earlier) was the direct fix for this. Even with exit 0, if stdout meets that function's conditions, it's treated as quota detection and the circuit opens (claude-quota-guard.py lines 187–190).

def is_quota_response(returncode: int, text: str) -> bool:
    if returncode != 0:
        return quota_message(text)        # ゆるいパターン
    return strict_quota_message(text)     # 厳しいパターン（400字以下＋構造マッチ）

Judge by both the exit code and the output content. Since this design went in, there have been no more misses from "exit 0 false successes."

Looking back now, all four failures grew from the same single root: the property that "claude -p doesn't error even with incomplete arguments." Incomplete arguments, unexpected behavior—Claude returns something regardless. Because that "something" gets recorded as output, everything looks normal if monitoring is lax.

Rather than designing jobs on the assumption of perfection, assume imperfect execution and layer detection and recovery on top—since moving to that philosophy, accidents like a four-lane simultaneous stall haven't happened.

Pitfalls

Listed in the order I actually hit them. For every one, I thought "I'll notice eventually"—and didn't.

Omitting --model doesn't make the command error
claude -p runs silently even with incomplete arguments. When omitted, it shares the interactive session's quota, so the problem surfaces "when the interactive quota runs out," not the moment you write the batch. It doesn't break the day you forget it; it dies quietly weeks later at some unrelated moment. The farther apart cause and effect are, the harder it is to spot the cause.

A limit message comes back with exit 0
When --model is omitted and the interactive quota runs out, claude -p writes "You've reached your Fable limit." to stdout and exits. There are cases where the exit code is 0 here. A script that judges success by $? can't distinguish the limit message from normal output. [ -s output.txt ] passes too, as long as the file isn't empty. Even 32 characters counts as "there was output."

The stock counter and the real file count drift apart
In the gen-column.mjs case, every time the job "completed" with exit 0 the success counter went up, and stock "appeared to be increasing." In reality, not one was added. A counter is a declaration that "I tried to generate," not evidence that "I generated." The reason it took three weeks to notice, until stock was actually at 2, was that the monitoring metric was the counter.

Four lanes stopping at once gets misread as "automation is working"
If one job stops, it's easy to notice "something's off." When four go silent at once, the unease of "everything's too quiet" fades. If anything, it pulls you toward "my hands are free = automation is working." The four-lane stall that surfaced on 9/3 was missed for three weeks by exactly this mechanism.

Vendors change quota wording without notice
The 9/2 error came in the form "You've hit your session limit · resets 6:50pm (Asia/Tokyo)," phrased differently from the earlier "weekly limit" and "Fable limit." QUOTA_PATTERNS at the time didn't include this pattern, and it slipped past quota detection. The result was 306 lines of wasted Claude calls over 4 hours. Write detection as "exact match of the wording" and it's neutralized the instant the vendor changes the phrasing.

The circuit breaker stops posting jobs uniformly
This was the 8/21 incident. When generation jobs exhausted the quota, posting jobs stopped as collateral. Measured, xpilot.autopost's execution rate had dropped to 43%. Generation jobs can retry if they fail. A post that misses its window can never be filled in. Stopping jobs with fundamentally different priorities under the same circuit breaker was the design flaw.

launchd doesn't go through a login shell
A script started from a plist runs in a different environment than when you run it manually from the terminal. The PATH in ~/.zshrc isn't read. node and claude commands managed by nvm can't be found. That's why com.shun.article-daily.plist explicitly sets PATH under EnvironmentVariables. This is the cause of the vast majority of "works locally, dies under launchd."

.mcp.json gets read and the batch hits external networks
Under launchd, the .mcp.json in the home directory is read automatically. If there's MCP server configuration for local development, the batch goes out to connect externally without meaning to. Without --strict-mcp-config --mcp-config '{"mcpServers":{}}', a batch can do things you can't reproduce locally.

Hardcoding model IDs in scripts causes mass breakage
When the vendor retires or renames a model alias, you end up rewriting every script with the string embedded. Write it with the version baked in, like --model claude-sonnet-4-5, and every model update means a tour through all your scripts.

The "alert when stock drops below 5" never fired
Because the alert logic pulled the stock count from the counter, the alert didn't fire while the counter was "increasing" (it actually wasn't). The monitoring metric has to be the real file count on the filesystem, not the generation job's success count.

Best Practices

Distilled from the four-lane simultaneous stall and three weeks of unnoticed loss, these are the rules I still apply to every job.

1. Always specify --model on claude -p
This is the starting point for everything. Omission is forbidden. Before writing a job, make it a rule that "a claude -p without --model does not exist."

2. Write it with an environment-variable fallback

--model "${ARTICLE_MODEL:-sonnet}"

Don't write model IDs directly into scripts. The :-sonnet fallback for the undefined case prevents "forgot it and fell to the default quota," and one line—launchctl setenv ARTICLE_MODEL haiku—switches every job.

3. Put a postcondition gate immediately after claude -p

MIN_ARTICLE_BYTES=1200
article_ok() {
  [ -s "$f" ] || return 1
  [ "$(stat -f%z "$f")" -ge "$MIN_ARTICLE_BYTES" ] || return 1
  grep -qE '^title:' "$f" || return 1
  grep -qiE 'request timed out|TODO: *本文' "$f" && return 1
  return 0
}

"There was output" is not success. Minimum byte count, presence of frontmatter, no timeout phrasing mixed in—only when all three conditions are met is it recorded as "generation succeeded."

4. Monitor stock by real file count
A counter is intent; the real file count is evidence. Base the alert metric on the actual number from ls ~/.claude/stock/ | wc -l. If the counter goes up but the files don't, it's a failure.

5. Write quota detection on the "structure of the wording"
Instead of an exact match like "Fable limit", use structural patterns like these.

r"(?:you(?:'ve| have) )?hit your (?:weekly |5-?hour |usage |session )?limit"
r"resets?\s+(?:at\s+)?\d{1,2}(?::\d{2})?\s*[ap]m"

Even if the vendor changes the wording, any phrasing with the structure of "you've hit a limit" gets caught.

6. Judge by both exit code and output content

def is_quota_response(returncode: int, text: str) -> bool:
    if returncode != 0:
        return quota_message(text)        # ゆるいパターン
    return strict_quota_message(text)     # 厳しいパターン（400字以下＋構造マッチ）

To prevent exit 0 false successes, check the exit code and stdout independently.

7. Use a 400-character cap to separate quota messages from article bodies
The message Claude returns at a limit is a few dozen characters. A normal article body is almost never 400 characters or less. If len(trimmed) > 400, don't treat it as a limit message—this one line prevents the false positive where "an article about quota trips my own circuit breaker."

8. Put --strict-mcp-config --mcp-config '{"mcpServers":{}}' into batches as a pair
Under launchd, .mcp.json gets read. Explicitly passing an empty server config shuts down the accident of a batch unintentionally calling external MCP. These two flags are one setting together. One alone doesn't work.

9. Change circuit-breaker behavior by job priority
Don't stop posting jobs and generation jobs with the same circuit breaker. Mark posts with priority: true and allow one attempt per PRIORITY_PROBE_INTERVAL (every 30 minutes) even while the circuit is open. A successful attempt detects quota recovery and closes the circuit—one mechanism achieves both keeping posts alive and detecting recovery early.

10. Run the grep gate the moment you write a plist

grep -rn "claude -p\|claude --print" \
  ~/.claude/scripts/ ~/dev/ \
  --include="*.sh" --include="*.mjs" --include="*.js" --include="*.py" \
  | grep -v -- '--model' \
  | grep -v '#.*claude -p'

Zero lines means every job is safe. Even one line back means something's missing. Always run it when adding a plist. Pairing it with the weekly launchctl list check turns it into a habit.

11. The three-piece set: ProcessType: Background / LowPriorityIO: true / Nice: 10
Put this combination into every batch job. By explicitly telling the OS "low-priority background task," it won't interfere with foreground work like having Figma open or researching in a browser. The plist default sets nothing (Standard priority), so if you don't specify, your foreground gets squeezed.

12. Set the launchd PATH explicitly via EnvironmentVariables
node and claude managed by nvm exist only in the login shell's PATH. Since launchd launches don't go through a login shell, specify a PATH containing nvm's binary path under the EnvironmentVariables key. In the com.shun.article-daily.plist implementation, ~/.nvm/versions/node/v24.13.0/bin goes at the front of PATH.

13. For multi-level calls, grep recursively down to the leaf
In the chain plist → quota-guard.py → run-and-notify.sh → article-daily-stock.sh → claude -p, applying the grep gate only to the plist never inspects the leaf script. Specify .sh .mjs .js .py via --include and search recursively to the end of the call chain.

14. Leave dated incident comments in the script
Line 37 of claude-quota-guard.py still has the comment # 2026-09-02: 5時間上限は〜の形式で返る. Writing down why a pattern was added and when means that six months later, you (or Claude Code) don't have to guess "why is this pattern here" when reading the code. Incident history gets read more when it sits next to the code than in a commit message.

Summary

Omitting --model is a one-line oversight. It produced an outage that lasted three weeks, hit four lanes simultaneously, and drained stock down to 2. The command doesn't error, it returns exit 0, and the log says "there was 1 line"—when those three overlap, a human eye won't catch it for weeks.

To use automation as infrastructure, you can't design jobs on the premise that they "work perfectly." The right philosophy is to assume imperfect execution and layer detection and recovery on top. Explicit --model is the starting point; the postcondition gate, real-file-count monitoring, structural pattern detection, and the priority-aware circuit breaker are its extensions.

Today, accidents where four lanes go silent at once no longer happen. The same principles apply as jobs grow, so even with 171 jobs running, the added mental cost is close to zero. "The environment works in my place" only holds because there's a mechanism in place for it to notice when it breaks and recover on its own.

Have you ever had a scheduled job that "succeeded" every single day while producing nothing—and what was the check that finally caught it?

The full picture of the system, the breakdown of the ¥1.2M/month, and the 30-day procedure are compiled in a paid note.
📕 How to actually make money with an autonomous Claude Code environment — system, real examples, getting started, and support

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

5 Pitfalls I Hit Building a Daemon That Auto-Approves Claude Code's Computer Use Dialog

Lily — Sat, 19 Sep 2026 00:00:05 +0000

Last time I wrote about the morning my Wi-Fi was dead for five hours. This post is in the same "stop babysitting an unattended machine" vein: a small daemon that presses Enter on Claude Code's Computer Use approval dialog for you. My first attempt, launched from launchd, produced nothing but nine osascript timeout lines at 22-second intervals and never sent a single keystroke. Once I moved it to run as a child of Terminal, it went from zero approvals to actually delivering \r to the dialog within seconds.

The problem: pressing "Allow for this session" by hand, every time

When you use Computer Use (screen control) in Claude Code, a confirmation dialog appears: Computer Use wants to control these apps. Selecting "Allow for this session" and pressing Enter gets you through, but it comes back every single time you cross a session boundary. I wanted to know whether it could be made permanent via settings, so I ran strings on the binary to check.

computerUseMcpState.allowedApps（セッション内メモリ）にしか積まれず初期値は空
settings.json / ~/.claude.json に事前付与キーは無い
dialog kind=computer_use_approval（requestDialog系）で PermissionRequest hookも通らない
bypassPermissionsModeAccepted: true でも出る

That's my field log from tracing the claude 2.1.266 binary with strings. The approval state lives only in session memory, in computerUseMcpState.allowedApps, and vanishes the moment you cross a process boundary. The same investigation did turn up settings that can be made permanent: bypassPermissionsModeAccepted: true in ~/.claude.json and projects[*].hasTrustDialogAccepted: true (I set 85 of them in one pass) both work, but they only suppress the "dangerous operation" confirmation, which is a different thing from the Computer Use approval dialog. In other words, there's no room to fix this on the settings side. That was the conclusion strings nailed down.

Note: If you put a * in the middle of an allow rule string (for example grep's [^}]* regex), you get a "wildcard before the rest" warning and the rule is rejected. Putting the wildcard at the end is the safe option.

If settings won't do it, the only option left is to look at the screen and press Enter on the human's behalf. So I wrote ~/.claude/scripts/cu_dialog_autoallow.py.

Implementation: poll every Terminal tab, and don't misfire

What it does is simple: poll every tab in Apple Terminal every 2 seconds, and when the dialog string shows up, send \r. But "see it, press Enter immediately" causes accidents. The implementation leans heavily toward misfire prevention.

READ = f'''
tell application "Terminal"
  set out to ""
  repeat with w in windows
    set k to count of tabs of w
    repeat with i from 1 to k
      set c to contents of tab i of w
      set out to out & (tty of tab i of w) & "{SEP}" & c & "{SEP}"
    end repeat
  end repeat
  return out
end tell
'''

The history property is heavy, around 600KB per tab, so I read only the visible screen (contents of tab i of w). There's also a trap: if you iterate with repeat with t in tabs and try to grab contents of t, you get the tab object itself back. You have to index explicitly in the form tab i of w.

The core of misfire prevention is checking which option the cursor is pointing at before firing.

def selected_is_allow(text):
    # the highlighted option line starts with the pointer glyph
    tail = text.split("Enter to confirm")[0]
    lines = [l.strip() for l in tail.splitlines() if l.strip()]
    for l in reversed(lines[-6:]):
        if l.startswith("❯") or l.startswith(">"):
            return "Allow for this session" in l
    return False

The line starting with ❯ (or > for ASCII-only environments) is the pointer line, and the only thing checked is whether it contains "Allow for this session". If the dialog's default cursor is on the Deny side and you send Enter anyway, you get a denial, so skipping this step makes things actively worse.

if "Computer Use wants to control these apps" in text and "Enter to confirm" in text:
    now = time.time()
    if now - last_sent.get(tty, 0) < 8:
        continue
    if not selected_is_allow(text):
        log(f"{tty}: dialog visible but cursor not on Allow; skipping")
        last_sent[tty] = now
        continue
    res = press_enter(tty)

The way Enter is sent is also designed so it doesn't steal focus.

def press_enter(tty):
    scpt = f'''
tell application "Terminal"
  repeat with w in windows
    repeat with t in tabs of w
      if tty of t is "{tty}" then
        do script "" in t
        return "sent"
      end if
    end repeat
  end repeat
  return "notfound"
end tell'''

do script "" in t just writes a single newline byte to that tab's tty; it doesn't activate the window. Sends are deduplicated per tty with an 8-second window, and fcntl.flock prevents multiple instances from running.

The trap I fell into: launching from launchd hangs silently

My first attempt was to launch it from a recurring launchd job. The result: AppleEvents from python3 to Terminal hang silently under TCC (Automation). No prompt, nothing. It just freezes. Here's the actual log.

2026-09-12 15:32:35 osascript timeout
2026-09-12 15:32:57 osascript timeout
2026-09-12 15:33:19 osascript timeout
2026-09-12 15:33:41 osascript timeout
2026-09-12 15:34:03 osascript timeout
2026-09-12 15:34:25 osascript timeout
2026-09-12 15:34:47 osascript timeout
2026-09-12 15:35:09 osascript timeout
2026-09-12 15:35:31 osascript timeout

You can see osascript timeout repeating endlessly at 22-second intervals. A process launched from launchd sits outside the user's TCC context, so its control request to Terminal can't even surface an approval dialog and gets swallowed whole.

Note: TCC (Automation) is nastier than "a permission dialog appears and gets denied": it hangs silently without ever showing a prompt. If you're sending AppleEvents via launchd, suspecting this failure mode is the fastest route to a fix.

The workaround: launch as a child process of Terminal

The workaround was to launch it as a child process of Terminal itself. AppleEvents from a child process of the same app don't trigger a TCC permission request at all. Here's what's at the end of my ~/.zshrc.

# Claude Code computer-use「Allow for this session」を自動Enter（Terminal起点で起動=TCC不要・launchd起点はTCCで固まる）
if [[ "$TERM_PROGRAM" == "Apple_Terminal" ]] && ! pgrep -qf cu_dialog_autoallow.py; then
  (nohup /usr/bin/python3 ~/.claude/scripts/cu_dialog_autoallow.py >/dev/null 2>&1 &)
fi

TERM_PROGRAM restricts this to shells started from Apple Terminal, pgrep stops a second instance, and nohup keeps it resident in the background. This block runs every time you open a new tab, but thanks to the pgrep guard only one process ever actually starts. After the switch, start pid=... shows up in the log and \r is actually being delivered to the dialog.

2026-09-12 15:36:16 /dev/ttys010: CU dialog -> Enter (sent)
2026-09-12 15:36:18 /dev/ttys011: CU dialog -> Enter (sent)
2026-09-12 15:36:26 /dev/ttys010: CU dialog -> Enter (sent)

Pitfalls I hit

Launched from launchd, AppleEvents hang silently under TCC → launch as a child of Terminal via nohup at the end of .zshrc
The history property is heavy at 600KB per tab → read only the visible screen with contents of tab i of w
repeat with t in tabs; contents of t returns the tab itself → use explicit index access with tab i of w
Sending Enter without checking the cursor position misfires (you press it on the Deny side too) → parse the ❯ pointer line and send only when Allow is confirmed
Transient AppleScript connection drops and syntax errors show up in the log (実行エラー: 接続が無効です (-609) / syntax error ... (-2741)) → design the loop to ignore one-off timeouts and keep polling (a lost cycle is retried 2 seconds later)

That last point is backed by real logs. On the morning of 09-13 and around midday on 09-16, connection drops and syntax errors came in bursts, but every one of them recovered on its own in the restart cycle a few minutes later, and an actual dialog approval (CU dialog -> Enter (sent)) was also recorded at 23:30 on 09-16. One caveat: I haven't yet measured \r reaching a real dialog at the moment another session is holding the CU lock; the end-to-end confirmation so far is against a fake dialog screen.

Summary

Per the strings investigation, there is no settings-based route to make Computer Use approval permanent (session memory only; the PermissionRequest hook doesn't fire either)
The workaround is a design that polls the visible screen and checks the cursor position to press Enter on your behalf without misfiring
Launching from launchd hangs silently under TCC. Running as a child process of Terminal is the only workaround
One-off connection drops and syntax errors are swallowed by the polling loop and self-heal

Next time I'll write about a sibling of this AppleScript watcher daemon: tiling Terminal windows in one shot.

Have you found any other way to get past a TCC-gated AppleEvent from a background process without going through the app itself?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

4 Automation Chrome Processes Were Killing My Real Browser: Bundle-ID Isolation and a 6-Hour Self-Repair Loop

Lily — Fri, 18 Sep 2026 11:00:06 +0000

At 13:31 JST on September 1, 2026, open -a "Google Chrome" returned error code -600 and nothing happened. Three headless Chrome processes and one headful one were already running on my Mac — all spawned by the same automation that had grown from ¥0 to ¥1.2M a month in six months. That afternoon, the whole setup got redesigned from the ground up.

Why This Setup Works

Once a solo developer's automation grows past a certain size, there's a problem you will inevitably hit: "the jobs are running fine, but at some point my own working environment broke."

On the morning of September 1, 2026, Lily's Mac had three headless Chrome processes and one headful one — four in total — running at the same time. Auto-liking on social media, thumbnail generation, per-account profile management: each was an independent job calling Playwright. And every one of them was directly exec-ing /Applications/Google Chrome.app/Contents/MacOS/Google Chrome. That was all it took.

macOS LaunchServices treats apps with the same CFBundleIdentifier as a single instance of the same app. Google Chrome's bundle ID is com.google.Chrome. If an automation job has already started a process with that bundle ID, then when you run open -a "Google Chrome" from the Dock or Spotlight, the OS decides "that app is already running, so I'll just bring the existing frontmost window forward." But the automation process usually has no window. The Chrome window a human expects never comes back. That is what -600 (procNotFound) really is.

The nasty part is that you can't notice this until it happens. The automation jobs are running normally. Exit code is 0. Logs are clean. Only when a human tries to use Chrome does nothing happen.

In general, this problem has the property that "the more automation jobs you add, the higher the probability." With one job, Chrome is held for a short time. As you go to three, five, ten jobs, Chrome being permanently held by one job or another becomes the normal state. The more automation you stack on to grow revenue, the longer your own Chrome is unusable. The causality is inverted.

There were two possible directions for a fix. One: make every automation job kill Chrome when it's done. Two: separate the automation browser from the human's Chrome at the level of its name. The former requires managing the shutdown timing of every job, and each new job creates a new gap. The latter, once built, eliminates the interference at the OS level. Only the latter is a permanent fix.

Create a separate bundle named /Applications/Chrome Automation.app and change its bundle ID to com.google.ChromeAutomation. From the OS's perspective it's a completely different app. LaunchServices manages com.google.Chrome and com.google.ChromeAutomation separately. No matter how many Chrome Automation instances the automation launches, a human's open -a "Google Chrome" looks for a different bundle ID and is unaffected.

If I had to explain in one line why this works: "Resource contention disappears the moment you separate the names." As long as they share a name, you have a structure where killing one kills the other. Split the name, and the OS isolates them for you.

There is, however, one awkward implementation wall.

Playwright's channel: 'chrome' option writes the path of the browser it launches into an internal file, playwright-core/lib/coreBundle.js. This file lives inside node_modules. Every time npm install runs, it gets overwritten and reverts to /Applications/Google Chrome.app.

In other words, even if you rewrite the path, it periodically reverts. Designing around the fact that "it reverts" is the key to stable operation. "Anticipate the revert and re-apply periodically." That's the reason the 6-hour self-repair script exists.

The Overall Flow

Here's the system architecture as a diagram.

  ┌──────────────────────────────────────────────────────────┐
  │  npm install（いつ走るかわからない）                          │
  │     ↓                                                     │
  │  playwright-core/lib/coreBundle.js                        │
  │     "...Google Chrome.app/Contents/MacOS/Google Chrome"  │ ← 元に戻る
  └──────────────────────────────────────────────────────────┘
                    ↑最大6時間以内に検知
  ┌──────────────────────────────────────────────────────────┐
  │  launchd  com.shun.chrome-automation-repair              │
  │  StartInterval: 21600（6時間ごと）RunAtLoad: true          │
  │     ↓                                                     │
  │  ~/.claude/scripts/chrome-automation-repair.sh           │
  │                                                           │
  │  1. keychain_gate() ─ --use-mock-keychain 未設定を警告      │
  │                                                           │
  │  2. バージョン比較                                           │
  │     src_ver（Google Chrome.app）                           │
  │     dst_ver（Chrome Automation.app）                      │
  │        一致 → スキップ                                      │
  │        不一致 or 不在 → rebuild                            │
  │            cp -R /Applications/Google Chrome.app         │
  │                   /Applications/Chrome Automation.app    │
  │            PlistBuddy: CFBundleIdentifier                │
  │                        → com.google.ChromeAutomation     │
  │            PlistBuddy: CFBundleName                      │
  │                        → Chrome Automation               │
  │            codesign --force --deep --sign -              │ ← ad-hoc 再署名
  │                                                           │
  │  3. playwright-core を glob で検索（find より速い）           │
  │     ~/dev/*/node_modules/playwright-core/lib/coreBundle.js│
  │     ~/content/*/node_modules/playwright-core/...         │
  │     ※ネスト4階層まで対応                                     │
  │        sed: Google Chrome.app のパス → AUTO_BIN に置換      │
  │        patched=N  already=M                               │
  │        patched+already==0 → exit 1（無音成功を防ぐ）          │
  │                                                           │
  │  ログ: ~/.claude/logs/chrome-automation-repair.log        │
  └──────────────────────────────────────────────────────────┘
                    ↓ パッチ済み
  ┌──────────────────────────────────────────────────────────┐
  │  playwright channel:'chrome' の解決先                      │
  │  /Applications/Chrome Automation.app/                    │
  │              Contents/MacOS/Google Chrome                │ ← 自動化バンドル
  └──────────────────────────────────────────────────────────┘
         ↕ 完全に独立
  ┌──────────────────────────────────────────────────────────┐
  │  open -a "Google Chrome"                                 │
  │  com.google.Chrome ← 人間の Chrome（干渉なし）               │
  └──────────────────────────────────────────────────────────┘

Let's look at the core parts of the script in real code.

Bundle Creation and Version Management

SRC="/Applications/Google Chrome.app"
DST="/Applications/Chrome Automation.app"
BUNDLE_ID="com.google.ChromeAutomation"
AUTO_BIN="$DST/Contents/MacOS/Google Chrome"

ver() { /usr/libexec/PlistBuddy -c "Print :CFBundleShortVersionString" \
        "$1/Contents/Info.plist" 2>/dev/null; }

src_ver="$(ver "$SRC")"
dst_ver="$(ver "$DST")"

if [ ! -d "$DST" ] || [ "$src_ver" != "$dst_ver" ]; then
  cp -R "$SRC" "$DST"
  /usr/libexec/PlistBuddy -c "Set :CFBundleIdentifier $BUNDLE_ID" \
      "$DST/Contents/Info.plist"
  /usr/libexec/PlistBuddy -c "Set :CFBundleName Chrome Automation" \
      "$DST/Contents/Info.plist"
  codesign --force --deep --sign - "$DST"
fi

The ver() function reads CFBundleShortVersionString (e.g. 127.0.6533.120) via PlistBuddy. When Google Chrome auto-updates, src_ver and dst_ver drift apart, so the next 6-hour cycle automatically rebuilds Chrome Automation.app.

The - (minus) in codesign --force --deep --sign - means an ad-hoc signature. The moment you rewrite the bundle ID in Info.plist, Google's signature becomes invalid. If you try to launch without re-signing, the helper processes die instantly with SIGKILL (exit 137). An ad-hoc signature is a self-signature with no trust chain, but for local-only execution that's fine.

Glob Search and Patching of playwright-core

SCAN_ROOTS=("${HOME}/dev" "${HOME}/content")

list_core_bundles() {
  local root sub
  shopt -s nullglob
  for root in "${SCAN_ROOTS[@]}"; do
    for sub in \
      "$root"/*/node_modules/playwright-core/lib/coreBundle.js \
      "$root"/*/node_modules/playwright/node_modules/playwright-core/lib/coreBundle.js \
      "$root"/*/*/node_modules/playwright-core/lib/coreBundle.js \
      "$root"/*/*/node_modules/playwright/node_modules/playwright-core/lib/coreBundle.js
    do
      [ -f "$sub" ] && echo "$sub"
    done
  done
  shopt -u nullglob
}

This sweeps the repositories under ~/dev and ~/content with glob patterns up to four levels deep. Walking the entire tree with find . -name "coreBundle.js" takes 4 minutes; hitting the layouts that actually exist (the two patterns */node_modules/playwright-core/... and */node_modules/playwright/node_modules/playwright-core/...) directly with globs brings the runtime down to 0.95 seconds.

The patch step is an in-place substitution with sed -i '':

patched=0; already=0
while IFS= read -r f; do
  if /usr/bin/grep -q "$SRC/Contents/MacOS/Google Chrome" "$f" 2>/dev/null; then
    /usr/bin/sed -i '' \
      "s|$SRC/Contents/MacOS/Google Chrome|$AUTO_BIN|g" "$f" \
      && patched=$((patched+1))
  elif /usr/bin/grep -q "$AUTO_BIN" "$f" 2>/dev/null; then
    already=$((already+1))
  fi
done < <(list_core_bundles)

if [ $((patched + already)) -eq 0 ]; then
  log "ERROR: playwright-core が1件も見つからない。SCAN_ROOTS を確認せよ。"
  exit 1
fi

patched is the number of files rewritten this run; already is the number already patched. If both are 0, the search path is broken and nothing was found. To prevent "zero results but treated as success," it explicitly fails with exit 1. This fail-loud design prevents the later question, "why did it go six hours unfixed without anyone noticing?"

Scheduled Execution via launchd

<key>StartInterval</key><integer>21600</integer>
<key>RunAtLoad</key><true/>
<key>LowPriorityIO</key><true/>
<key>Nice</key><integer>10</integer>
<key>ProcessType</key><string>Background</string>
<key>StandardOutPath</key>
  <string>~/.claude/logs/chrome-automation-repair.log</string>

StartInterval: 21600 is 21600 seconds = 6 hours. With RunAtLoad: true, it also runs once the moment you launchctl load it. Nice: 10 and LowPriorityIO: true lower CPU and disk priority so it doesn't interfere with human work in the background.

The meaning of the 6-hour interval is an SLA: "auto-repair within at most 6 hours after an npm install runs." In practice the automation jobs rarely run npm install, so in most cases the patch doesn't revert before the next cycle. Even if it does, it's fixed within 6 hours. This design accepts the "length of the repair window" as a known risk.

Keychain Gate Detection

At the top of the script there's a detection routine called keychain_gate():

keychain_gate() {
  local hits
  hits=$(grep -rIl --include='*.sh' --include='*.py' --include='*.js' \
      --exclude-dir=node_modules ... \
      -E 'Chrome Automation\.app/Contents/MacOS|Google Chrome\.app/Contents/MacOS' \
      "${SCAN_ROOTS[@]}" ~/.claude/scripts 2>/dev/null \
    | xargs -I{} grep -l -e '--headless' {} 2>/dev/null \
    | xargs -I{} grep -L -e 'use-mock-keychain' {} 2>/dev/null \
    | grep -v 'scent-media/scripts/ensure_chrome.sh' || true)
  if [[ -n "$hits" ]]; then
    log "WARNING: --use-mock-keychain 無しでChromeをheadless起動している..."
    echo "$hits" | sed 's/^/  /'
  fi
}

Because Chrome Automation.app is ad-hoc signed, even if you grant "Always Allow" on the "Chrome Safe Storage" ACL in macOS Keychain, the code hash changes every time a Chrome update triggers a rebuild, and the permission is invalidated. Any script that launches headless without --use-mock-keychain --password-store=basic causes a Keychain access dialog to pop up every time. Since this leads to "silent failures" in automation, there's a gate that detects and warns — but does not fix.

In this round of work, the repositories fixed came to 16 locations where Playwright's channel:'chrome' resolves, 10 files with direct shell and Python invocations, and 8 repositories committed. The larger the scale, the less "I fixed one place" is the end of it. I needed to propagate the fix across every repository, and have a mechanism that automatically restores the fix even when npm install wipes it out. That's why this setup exists.

Implementation Details

The pgrep Guard — Don't Rebuild While Running

The bundle-creation logic shown earlier has one more safety valve. When a version mismatch is detected, if Chrome Automation.app is running, skip the rebuild.

if pgrep -f "Chrome Automation.app/Contents/MacOS" >/dev/null 2>&1; then
  log "SKIP rebuild: 自動化Chromeが稼働中 (src=$src_ver dst=${dst_ver:-none})"
else
  log "rebuild: src=$src_ver dst=${dst_ver:-none}"
  rm -rf "$DST" || { log "ERROR: 旧バンドル削除に失敗"; exit 1; }
  cp -R "$SRC" "$DST" || { log "ERROR: コピー失敗"; exit 1; }
  ...
fi

cp -R takes 15–30 seconds. If Chrome Automation.app's Contents/MacOS/Google Chrome is read mid-copy, a half-written binary gets executed and the whole job breaks. pgrep -f "Chrome Automation.app/Contents/MacOS" looks for running processes, and if any exist, the design is to "wait until the next 6-hour cycle."

Even when the rebuild is skipped, the patched/already counting still runs afterward. The bundle itself may be stale, but as long as the path in coreBundle.js is correct, jobs keep working.

launchd's Shell Doesn't Have a Human's PATH

The plist contains an EnvironmentVariables block.

<key>EnvironmentVariables</key>
<dict>
  <key>HOME</key><string>~</string>
  <key>LANG</key><string>en_US.UTF-8</string>
  <key>PATH</key>
    <string>~/.local/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
</dict>

The shell launchd starts only inherits its environment from /etc/launchd.conf; it reads neither ~/.zshrc nor ~/.zprofile. Unless you set PATH explicitly, only /usr/bin:/bin is available, and codesign and PlistBuddy in /opt/homebrew/bin can't be found. HOME is set explicitly so that the ${HOME}/dev expansion is resolved via EnvironmentVariables rather than as a shell variable. LANG is insurance so grep doesn't mangle filenames containing Japanese.

The log destinations being split into StandardOutPath and StandardErrorPath is for the same reason. launchd writes stdout and stderr to separate files, both landing in ~/.claude/logs/. If you only look at one, it can appear as though "nothing is happening."

Reading the keychain_gate Pipeline

The keychain_gate() implementation has an unusual structure: three chained greps.

grep -rIl ... \
  -E 'Chrome Automation\.app/Contents/MacOS|Google Chrome\.app/Contents/MacOS' \
  "${SCAN_ROOTS[@]}" "${HOME}/.claude/scripts" \
| xargs -I{} grep -l -e '--headless' {} \
| xargs -I{} grep -L -e 'use-mock-keychain' {}

Stage 1 finds "files that directly write the Chrome binary path." node_modules, .git, profiles, logs, venv, and tests are excluded from the search.

Stage 2 keeps only files containing --headless. Scripts that launch with a GUI are fine even if the Keychain dialog appears.

Stage 3 keeps only files that do not contain use-mock-keychain (-L is the "list non-matching files" flag).

Files that pass all three stages are in the state of "using Chrome headless but without mock-keychain." Each time they run, the Keychain "Allow?" dialog pops up. When a dialog appears mid-automation, every process stalls until the next click. The one file excluded via grep -v, scent-media/scripts/ensure_chrome.sh, is deliberately designed to use the real Keychain for a different reason.

The script does not auto-fix on this detection. Persistent profiles like Instagram encrypt their session cookies with a Keychain key, and if the key changes, the login is wiped out. Log a warning and leave the decision to a human — that's the design choice.

Final Verification Commands

Once the setup is in place, confirm it actually works.

# 1. launchd の登録確認
launchctl list | grep chrome-automation-repair

# 2. バンドルが生きているか
open -a "Chrome Automation" --args --version

# 3. playwright が正しいパスを見ているか
grep -r "Chrome Automation" ~/dev/*/node_modules/playwright-core/lib/coreBundle.js 2>/dev/null | head -5

# 4. 人間の Chrome は独立して起動できるか
open -a "Google Chrome"

Number 2's --args --version passes the --version flag to Chrome, making it print the version string at startup (e.g. Google Chrome 127.0.6533.120). If you get the string back without a window opening, the bundle is in a launchable state. Number 4 is the final confirmation: if the human's Chrome opens without returning -600, the isolation is working.

Where I Got Stuck

`find` Was Taking 4 Minutes

The first script I wrote searched with find.

find "${HOME}/dev" "${HOME}/content" \
  -name "coreBundle.js" \
  -path "*/playwright-core/lib/*"

When I ran it, it took 4 minutes 15 seconds to complete. There are several hundred node_modules directories under ~/dev, and find walks every one of them. A 4-minute process every 6 hours wastes CPU and disk even in the background, and worse, I hadn't noticed the "it's slow when you actually run it" fact until the first implementation.

The layouts where coreBundle.js actually exists are just two patterns. For a directly installed playwright-core: */node_modules/playwright-core/lib/coreBundle.js. For a nested install via playwright: */node_modules/playwright/node_modules/playwright-core/lib/coreBundle.js. Hitting these two patterns directly with globs down to two levels of repository depth brought the runtime to 0.95 seconds.

Globs are fast because filesystem traversal collapses to OS directory-entry lookups. Where find walks every node, a glob only checks "does an entry matching this pattern exist?" If your search space fits a known structure, glob is clearly superior as a find replacement.

Forgot the Ad-Hoc Signature, Got exit 137

My first implementation had cp -R and the PlistBuddy rewrite, but no codesign. When I tried to launch Chrome Automation.app in a test, the helper processes died instantly with exit 137.

Exit 137 is SIGKILL (128 + 9). The process didn't exit on its own; the OS forcibly killed it. When macOS Library Validation determines that "the binary exists but its signature doesn't match the bundle ID in Info.plist," it SIGKILLs the helper. Google's signature was issued for com.google.Chrome, and the moment you change CFBundleIdentifier in Info.plist to com.google.ChromeAutomation, it's invalid.

Adding codesign --force --deep --sign - and re-running made it launch. --force overwrites the existing signature, --deep re-signs not just the main binary but all frameworks, helpers, and plugins, and - (minus) specifies an ad-hoc signature without an Apple certificate. Gatekeeper doesn't "trust" this signature, but it will launch for local execution. The first time, you get a "developer cannot be verified" confirmation dialog, but after pressing "Open Anyway" once under System Settings → Privacy & Security, it's no longer needed.

Zero Results Still Exited 0 as Success

Early on, I misconfigured SCAN_ROOTS. I wrote ~/Development instead of ~/dev, and since the directory didn't exist, list_core_bundles returned nothing.

The script ran to completion and returned exit 0. patched=0 and already=0, but neither condition tripped, and the log just said playwright patched=0 already=0 rebuilt=0. launchd only looks at the job's exit code, so it kept recording "success" every 6 hours while nothing was being fixed.

This is where the worst case happens: after npm install reverts the path, it can sit unrepaired for up to 6 hours. The code that explicitly fails with exit 1 when patched + already == 0 was born from this failure.

if [ $((patched + already)) -eq 0 ]; then
  log "ERROR: playwright-core が1件も見つからない。SCAN_ROOTS を確認せよ。"
  exit 1
fi

When the exit code is non-zero, launchd records it to StandardErrorPath and re-runs on the next cycle. Make failures loud. That's the correct design.

One Deeply Nested Repo Slipped Through

The first glob pattern only covered one level.

"$root"/*/node_modules/playwright-core/lib/coreBundle.js

This single level doesn't catch ~/dev/social-autolike/node_modules/playwright/node_modules/playwright-core/lib/coreBundle.js. That's the case where playwright is installed containing playwright-core inside it. social-autolike had installed playwright directly, so this nesting occurred.

As a result, while the other 15 locations pointed at Chrome Automation.app, social-autolike alone kept directly hitting Google Chrome.app. When the same problem happened later in dokuji-rosen-sns (described below), the habit of first checking "has a new nesting pattern appeared?" came from this failure.

preflight Was `pkill -9`-ing the Human's Chrome Too

metrics-hub had a script called preflight_chrome.sh whose job was to clean up stale Chrome processes before a job started. It was implemented like this.

pkill -9 "Google Chrome"

pkill does partial matching on the process name. The string Google Chrome appears in the process names of both com.google.Chrome and com.google.ChromeAutomation. Every time an automation job ran preflight, every Chrome window the human had open was SIGKILLed.

Worse, this breaks in a way that's hard to trace back to a cause. Jobs using Chrome Automation.app run normally. The only remaining symptom is that the human's Chrome suddenly crashes. Nothing shows up in the job logs. Only after the vague awareness of "Chrome keeps crashing" accumulates do you realize preflight is the culprit.

The fix is to filter by binary path instead of process name.

pkill -9 -f "Chrome Automation.app/Contents/MacOS"

-f matches against the process's full command line. By targeting only processes whose command line contains the path Chrome Automation.app/Contents/MacOS, the implementation never touches the human's Chrome.

A 4th Repo Fell Into the Same Hole a Month Later

In the September 1 fix, I changed 16 coreBundle.js locations and 10 direct-invocation files, and committed 8 repositories. At that point, I believed "everything is covered."

On September 11, it came out that tools/lib/browser.mjs in dokuji-rosen-sns was still on channel:'chrome'. Even after social-autolike migrated to the bundled chromium on August 12, dokuji-rosen-sns remained on channel:'chrome'. Only between 8:00 and 10:40 in the morning, when it competed with other jobs for Chrome, startup exceeded 180 seconds and timed out — and this continued for 5 days. 39 hangs in 5 days, and the 125 likes in the 8:00 slot vanished every day, yet the alert was skimmed past as a single line: "will auto-recover on the next cycle."

This lesson overlaps with the general form I left in the wiki: "A fix to a shared function means nothing unless it propagates to every call site that uses it." Even if you fix the script, as long as another repo written in the same pattern remains, the fix isn't complete.

Having both ~/dev and ~/content in SCAN_ROOTS of chrome-automation-repair.sh is also to avoid missing a repo that exists in only one of them. Checking "does this repo use channel:'chrome'?" every time a new repo is created is a human task. But if the script runs every 6 hours and patched becomes 1 or more, at least the scenario of "a month passes without noticing a miss" is prevented. For now, as long as the script keeps returning patched=0 already=N, I judge that everything is pointing the right way.

Pitfalls

Here's a complete list of the landmines I actually stepped on. Most of "I set it up but it doesn't work" is one of these.

Forgetting codesign after rewriting Info.plist. The moment you change CFBundleIdentifier to com.google.ChromeAutomation, Google's signature is invalid. Launch without re-signing and Library Validation rejects it; helper processes die instantly with exit 137 (SIGKILL). The --deep in codesign --force --deep --sign - is mandatory. With --force alone, only the main binary is updated, leaving frameworks, helpers, and plugins with the old signature.
Full-walking node_modules with find. find ~/dev ~/content -name "coreBundle.js" took 4 minutes 15 seconds to finish, because there are several hundred node_modules under ~/dev. coreBundle.js only exists in two layouts: */node_modules/playwright-core/lib/ and */node_modules/playwright/node_modules/playwright-core/lib/. Hit those directly with globs and the runtime drops to 0.95 seconds. There's no need to full-walk a known structure.
Writing only one nesting pattern. Projects that install playwright directly have */node_modules/playwright-core/lib/coreBundle.js. Projects where playwright-core comes in via playwright have the nested form */node_modules/playwright/node_modules/playwright-core/lib/coreBundle.js. Write only one in the glob and you miss the other kind of repo every time. I hit this in social-autolike: I thought I'd fixed 16 locations, but one was still pointing at the old path.
Zero results still exits 0 as success. Put a nonexistent path in SCAN_ROOTS and list_core_bundles returns nothing, exiting normally with patched=0 already=0. launchd records exit 0 as "success," and a "do-nothing job" quietly runs every 6 hours. Only by explicitly failing with exit 1 when patched + already == 0 does an ERROR land in the log and the job re-run on the next cycle.
pkill -9 "Google Chrome" kills the human's Chrome too. Partial matching on the process name hits both com.google.Chrome and com.google.ChromeAutomation. preflight_chrome.sh in metrics-hub was exactly this: every time it tried to clean up the automation bundle, it SIGKILLed every tab the human had open. Filter on the full command line with pkill -f "Chrome Automation.app/Contents/MacOS" and the implementation never touches the human's Chrome.
launchd's shell doesn't read ~/.zshrc. The PATH of the shell launchd starts is only /usr/bin:/bin. Neither codesign nor PlistBuddy in /opt/homebrew/bin is visible. Unless you explicitly set PATH, HOME, and LANG in the plist's EnvironmentVariables, the script silently dies with "command not found." If LANG isn't en_US.UTF-8, grep may also mangle filenames containing Japanese. It ends with exit 127 and nothing in the log, so discovery is delayed.
Chrome Automation.app's binary gets read mid-rebuild. cp -R /Applications/Google\ Chrome.app /Applications/Chrome\ Automation.app takes 15–30 seconds in the real environment. If an automation job tries to exec Chrome Automation.app/Contents/MacOS/Google Chrome during that window, it grabs a half-copied binary and crashes. Check for running processes with pgrep -f "Chrome Automation.app/Contents/MacOS", and if any exist, defer to the next 6-hour cycle.
Keychain ACLs are invalidated by Chrome updates. Chrome Automation.app is ad-hoc signed, so its code hash changes with every rebuild. Even if you set "Always Allow" on "Chrome Safe Storage" in macOS Keychain, when Chrome auto-updates and a rebuild runs, the ACL is invalidated and a "Allow password access?" dialog appears on every headless launch. The right answer is to add --use-mock-keychain --password-store=basic to every headless launch. However, persistent profiles like Instagram encrypt session cookies with the Keychain key, so switching to mock wipes the login. That's why keychain_gate() only detects and warns without auto-fixing.
Fixing just one repository means the neighbor hits the same hole a month later. Scripts that directly specify channel:'chrome' are scattered across every repository in ~/dev and ~/content. Fix one, and another repository's browser.mjs keeps pointing at the old path. dokuji-rosen-sns actually hit this: only between 8:00 and 10:40 in the morning, competing with other jobs for Chrome, it hung for over 180 seconds — for 5 days straight. For the record, 39 times in 5 days, with the 125 likes in the 8:00 slot vanishing every day, and the alert was skimmed as a single line: "auto-recovers on next tick." A fix is only one complete unit once you've cross-checked every repository in SCAN_ROOTS.
Unless you look at both StandardOutPath and StandardErrorPath, it looks like "nothing is happening." launchd writes stdout and stderr to separate files. If an error only appears in .err.log and you only check .log, you won't notice. Build the habit from the start of checking both tail -f ~/.claude/logs/chrome-automation-repair.log and tail -f ~/.claude/logs/chrome-automation-repair.err.log, and discovering silent failures shrinks from hours to minutes.
A Gatekeeper dialog appears on Chrome Automation.app's first launch. Since an ad-hoc signature isn't on Apple's trust chain, the first time you get "cannot be opened because the developer cannot be verified." Press "Open Anyway" once under System Settings → Privacy & Security and it won't appear again. Easy to forget right after moving the environment to a new Mac or after a major Chrome update (when the bundle is rebuilt).

Best Practices

Here's the setup above distilled into general principles. Not limited to Chrome bundle isolation — these are design guidelines for any long-running automation on macOS.

1. Separate human resources from unattended-job resources at the level of their "name."
Bundle ID, profile path, port number, PID file. The moment any one of these is shared, you get a structure where "killing one kills the other." Just split the names and the OS isolates them for you. The design cost is one-time; the OS manages it afterward.

2. Make "npm install will always revert it" a design premise.
Don't rely on the fact that a patch was applied. Build "anticipate the revert and re-apply periodically" into the design from the start. A window of up to 6 hours where it reverts to the old path remains, but that's an acceptable known risk. "Fixed" and "stays fixed" are different states.

3. Count patch results and make zero fail-loud.
Keep two variables, patched=N already=M, and exit 1 if patched + already == 0. "Searched but found nothing" is equivalent to "did nothing." Silently exiting 0 means a misconfiguration quietly repeats every 6 hours.

4. Ensure idempotency with version comparison and skip unnecessary rebuilds.
Compare src_ver and dst_ver via /usr/libexec/PlistBuddy -c "Print :CFBundleShortVersionString" and implement "same version, do nothing," and the 6-hour cycle becomes nearly free. A rebuild only runs when Google Chrome auto-updates.

5. Put a pgrep guard before rebuilding.
Check for running processes with pgrep -f "Chrome Automation.app/Contents/MacOS", and if any exist, defer to the next cycle. The problem of a binary being grabbed mid-copy is completely prevented by this guard alone. A mechanism that can decide "don't touch it right now" is what creates long-running stability.

6. Always write PATH, HOME, and LANG in the launchd plist's EnvironmentVariables.
launchd reads none of your interactive shell's configuration. Most cases of "the script works locally but silently dies under launchd" are this. Just writing them in the plist from the start brings this class of trouble to zero.

7. Use glob over find and hit known layouts directly.
If you control the structure of what you're searching, a full walk is always overkill. playwright-core lives in only two patterns. Hitting them directly with globs finishes in 1/270th the time of find.

8. Use codesign with the three-option set --force --deep --sign -.
--force overwrites the existing signature, --deep applies to all frameworks, helpers, and plugins, - is an ad-hoc signature. Miss even one and Library Validation SIGKILLs. Remember them as a set of three and you won't get stuck.

9. Add --use-mock-keychain --password-store=basic to headless Chrome.
Only jobs with persistent profiles (Instagram etc.) are the exception: using mock wipes the session, so keep the real Keychain for them. Know which category applies, and for the majority of headless launches, adding mock is the correct setting.

10. Use -f with pkill for full-path matching. Don't use partial process-name matching.
pkill "Google Chrome" kills both even though the bundle IDs differ. Filter on the full command line with -f "Chrome Automation.app/Contents/MacOS" and the implementation never touches the human's Chrome.

11. After a fix, cross-check every repository in SCAN_ROOTS.
Fixing one repository isn't complete while another repository written in the same pattern remains. Surface the misses with grep -r "channel:'chrome'" ~/dev ~/content --include="*.mjs" --include="*.js" 2>/dev/null, and only after handling all of them is it done.

12. Keep the final verification commands together with the script.
Decide up front what you'll verify with after setup. For this setup, it's these four commands.

# 1. launchd への登録確認
launchctl list | grep chrome-automation-repair

# 2. バンドルが起動できる状態か
open -a "Chrome Automation" --args --version

# 3. playwright が正しいパスを見ているか
grep -r "Chrome Automation" ~/dev/*/node_modules/playwright-core/lib/coreBundle.js 2>/dev/null | head -5

# 4. 人間の Chrome は独立して起動できるか
open -a "Google Chrome"

If number 2's --args --version returns Chrome's version string (e.g. Google Chrome 127.0.6533.120) and number 4 opens without -600, the isolation is working correctly. Thinking of "the setup plus its verification commands as one unit" lowers the cost of reproducing the same environment six months later.

13. Collect logs in ~/.claude/logs/ and check both stdout and stderr.
Look at only one and you'll keep not noticing that an error is in .err.log. Set things up so tail -f ~/.claude/logs/*.log ~/.claude/logs/*.err.log gives you every job's status at a glance, and the cost of discovering silent failures drops.

14. Choose the repair window (SLA) deliberately and record the rationale.
StartInterval: 21600 (6 hours) is an SLA: "even if npm install runs, it's repaired within at most 6 hours." Shorter raises certainty but increases CPU and disk impact. It's important to deliberately choose the maximum your environment can tolerate and leave the rationale in a comment in the plist.

Summary

If I had to state in one line what remained as a design from that September 1 afternoon when open -a "Google Chrome" returned -600: "Contention only happens while resources are shared. Split the name, and the OS isolates them for you."

With one automation, the problem is invisible. As you stack up two, five, ten, human resources and automation resources occupy the same namespace, and a structure where killing one breaks the other quietly forms. The more automation you add to grow revenue, the more your own working environment breaks — noticing that paradox was the starting point of this design.

The mechanism stabilized the moment I stopped viewing "npm install reverts it" as a "problem that must be fixed" and accepted it as "periodic repair on the premise that it reverts." Even if the patch reverts, it's fixed within 6 hours. Building that repair window into the design as a known risk is the key to long-running operation.

Now that chrome-automation-repair.sh runs every 6 hours and keeps returning patched=0 already=16 rebuilt=0, the automation jobs use Chrome Automation.app and the human's Chrome runs independently. Mornings like September 1 don't come anymore.

What's the one shared resource in your automation setup that you and your jobs are still fighting over — and how long has it been silently losing?

The full picture of the system, the breakdown of the ¥1.2M/month, and the 30-day procedure are compiled in a paid note (Japanese).

📕 How to actually earn with a Claude Code autonomous environment — mechanism, real examples, getting started, and support

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Wi-Fi Was Dead for 5 Hours While Chrome Kept Working: 4,960 DNS Failures and the 5-Minute Probe That Catches Them

Lily — Fri, 18 Sep 2026 05:00:06 +0000

Last time I wrote about how many jobs you should re-run the moment your quota comes back. This post goes back a few days earlier, to a quieter and far more annoying problem: the morning my home Wi-Fi was effectively dead for five hours — while the browser kept loading pages as if nothing were wrong.

The problem: the browser works, but only the jobs die

On the morning of 2026-09-12, four unattended jobs failed one after another.

Job	Time
`com.lily.es-daily-rows`	05:01
`com.lily.paid-note-pin-guard`	07:20
`com.lily.line-column-gen`	08:40
`com.lily.line-pdca`	08:41

The errors were Node fetch failed and python getaddrinfo failures. Yet during that same window, browsing in Chrome worked perfectly. This is the worst kind of failure: "the network is up, but some processes die anyway."

Digging into the unified log, airportd had been emitting SlowWiFiDnsFailure at an abnormal rate. In the last 24 hours there were 4,960 of them. They ramped up around 05:43, ran at 1,000–1,700+ per hour through the 06:00–11:00 window, peaked at 1,720 in the 10 o'clock hour, and stopped cold after 12:38. The human action that ended it came at 13:09: I manually switched from home Wi-Fi to my phone's tethering. After the switch, 37 probe runs showed failed: 0, with lookups under 60ms and connects under 100ms on both IPv4 and IPv6.

The culprit was that only the DNS path was dead. Chrome has its own DNS resolution (DoH plus a cache), so it sails on even when the home router's DNS is rotten. Node and Python go straight to the system resolver — the home router's DNS — and get stuck. The intuition that "the browser works, so the network is fine" was itself the diagnostic trap.

Pitfalls I hit (diagnosis edition)

I almost misread a symlink's mtime as "we've been on tethering for three weeks" → /etc/resolv.conf is a symlink, and its mtime still said Aug 15. Checking the real file, /var/run/resolv.conf, its mtime matched the switchover time (13:09:39) exactly, so the first reading was wrong. You always have to know whether you're looking at the symlink's mtime or the target's.
Unified log retention windows differ per facility → configd only had entries from 05:54 onward, while airportd went back to 19:00 the previous day. I thought I was comparing two facilities on the same timeline, but one of them had already dropped data.
The name SlowWiFiDnsFailure is misleading → It's logged as "slow," not "dead," so even grepping for anomalies requires a threshold decision.
Node fetch failed appeared 343 times in that same 24 hours. The apparent gap between the job failure times (05:01–08:41) and the start of the airportd burst (05:43 onward) is also explained by the difference in facility retention.

Note
On that day I was able to dig through the unified log afterward and pin down "roughly 5,000 events between 06:00 and 12:38." But that only worked because the incident happened to be large enough to stand out. If the same thing happens next time for a shorter period or at lower frequency, it may be past the unified log's retention window and leave nothing behind. What I needed was a way to trace root causes that doesn't depend on "digging through the log after the fact" — and that's the real subject of this post.

Never chasing it after the fact again: net-probe.sh

What I built is a script that does one thing: every 5 minutes it appends a one-line JSONL "snapshot of the network state." It uses neither Chrome nor claude, and the target is to finish in under 20 seconds.

#!/bin/bash
# 5分毎の軽量ネットワークプローブ（Chrome/claude 不使用・目標20秒以内)。
# gateway / SSID / nameserver / dig 3ホスト / curl -4,-6 2ホスト / python getaddrinfo を
# 1行JSONで ~/.claude/logs/net-probe.jsonl へ追記する。
# 目的: Node fetch failed / python getaddrinfo 失敗が「どの回線・どのDNSで」起きたかを後から突合する。

The key is to fire everything in parallel. Fetching the SSID via system_profiler alone takes 5 seconds, so it gets pushed into the background alongside the other collection.

# --- SSID は system_profiler が5秒かかるので並列 ---
( system_profiler SPAirPortDataType 2>/dev/null | awk '/Current Network Information:/{getline; gsub(/^ +| *:$/,""); print; exit}' > "$TMP/ssid" ) &

# --- DNS: システム順序で3ホスト + 各NSで oauth2 (すべて並列, 2秒×1回) ---
for h in oauth2.googleapis.com discord.com note.com; do
  ( dig +time=2 +tries=1 "$h" A > "$TMP/dig.$h" 2>&1 ) &
done
i=0
for ns in ${NS//,/ }; do
  ( dig +time=2 +tries=1 @"$ns" oauth2.googleapis.com A > "$TMP/digns.$i" 2>&1; echo "$ns" > "$TMP/digns.$i.ns" ) &
  i=$((i+1))
done

dig uses +time=2 +tries=1 so it always gives up at 2 seconds, curl uses --max-time 6, and Python's getaddrinfo runs in a thread with join(5) to force a 5-second cutoff. Every timeout has to be explicit, because when DNS is truly dead, the probe itself would hang and miss the next 5-minute slot.

def f():
    t = time.time()
    try:
        r = socket.getaddrinfo('oauth2.googleapis.com', 443)
        res.update(ok=True, n=len(r), ms=round((time.time()-t)*1000))
    except Exception as e:
        res.update(ok=False, err=type(e).__name__+': '+str(e), ms=round((time.time()-t)*1000))
th = threading.Thread(target=f, daemon=True); th.start(); th.join(5)
if not res: res.update(ok=False, err='timeout>5s', ms=5000)

Once wait confirms all the fragments are in, the one-line JSON is assembled in Python rather than bash. Hand-building JSON via string concatenation breaks the moment an SSID contains a space or a special character, so this one part is left to json.dumps.

gw = os.environ['GW']
fail = sum(1 for d in dns.values() if d['status'] != 'NOERROR') + sum(1 for h in http.values() if h['http'] == 0) + (0 if py.get('ok') else 1)
rec = {
  'ts': os.environ['TS'], 'gateway': gw, 'gateway6': os.environ['GW6'], 'iface': os.environ['IFACE'],
  'ip4': os.environ['IP4'], 'ssid': rd(f'{T}/ssid').strip(), 'security': os.environ['SEC'],
  'isIphoneHotspot': gw == '172.20.10.1', 'resolvers': [x for x in os.environ['NS'].split(',') if x],
  'utun': int(os.environ['UTUN'] or 0), 'dns': dns, 'dnsPerResolver': per_ns, 'http': http,
  'pyGetaddrinfo': py, 'failures': fail,
}

isIphoneHotspot is determined by whether the gateway IP is 172.20.10.1 (the default address for iPhone tethering). Instead of eyeballing the gateway IP every time, this single field tells you at a glance which connection you're on. failures is the sum of dns / http / pyGetaddrinfo failures, so pulling anomalous lines out of the JSONL is just grep '"failures":[1-9]'.

launchd: run it quietly every 5 minutes

StartInterval gives a fixed-interval launch, and the priority is lowered so it stays strictly in the background.

<key>StartInterval</key>
<integer>300</integer>
<key>LowPriorityIO</key>
<true/>
<key>Nice</key>
<integer>10</integer>
<key>ProcessType</key>
<string>Background</string>

Nice 10 plus LowPriorityIO ensures it never steals CPU or IO from the other automation jobs. A job whose purpose is to watch whether the network is alive should not itself become a source of resource contention.

Four days of real measurements

From deployment until today, it has accumulated 662 lines.

$ wc -l ~/.claude/logs/net-probe.jsonl
662

Broken down by gateway: home Wi-Fi (192.168.3.1) accounts for 594 entries, iPhone tethering (172.20.10.1) for 68. Lines matching "failures":[1-9] number 50 — about 7.6% of 662. Some days had zero, while the hour 2026-09-13T09 alone had 5 failures clustered together — unevenness that would have been flattened out without 5-minute granularity.

A single line looks like this (excerpt from the actual log).

{"ts":"2026-09-16T20:28:45+0900","gateway":"192.168.3.1","iface":"en0",
 "security":"WPA2_PSK","isIphoneHotspot":false,
 "dns":{"oauth2.googleapis.com":{"status":"NOERROR","ms":17,"answers":5,"err":null},
        "discord.com":{"status":"NOERROR","ms":17,"answers":11,"err":null},
        "note.com":{"status":"NOERROR","ms":15,"answers":8,"err":null}},
 "http":{"v4:discord.com":{"http":200,"lookup":0.004285,"connect":0.013422,"total":0.115357}, ...},
 "pyGetaddrinfo":{"ok":true,"n":4,"ms":107},"failures":0}

Cross-referencing: reconstructing when, which connection, and which DNS after the fact

The network-flap-probe-2026-09-12.json I put together right after the incident still holds verdict: "dns-resolver-flap" along with the explanation of the cause, exactly as written that day (in Japanese).

"verdictDetail": "症状の発生源は自宅Wi-Fiの DNS 経路。airportd が 06:00-12:38 に
SlowWiFiDnsFailure を約5,000件（毎時1,000-1,700件）記録しており、failing 4 ジョブと
fetch failed 343件はこの回線上で発生。13:09に人間が iPhone テザリングへ手動切替した後は
fault 0・全プローブ成功。Chrome 経由ジョブが通っていたのは Chrome が独自 DNS
(DoH/非同期リゾルバ+キャッシュ)を使い、Node/pythonはシステムリゾルバ
（自宅ルータ DNS）を直撃するため"

In short, it says: the source of the symptoms was the home Wi-Fi's DNS path; airportd recorded roughly 5,000 SlowWiFiDnsFailure events (1,000–1,700 per hour) between 06:00 and 12:38; the 4 failing jobs and the 343 fetch failures all occurred on that connection; after the manual switch to iPhone tethering at 13:09, faults dropped to 0 and every probe succeeded; and the Chrome-based jobs kept working because Chrome uses its own DNS (DoH / async resolver + cache) while Node and Python hit the system resolver (the home router's DNS) directly.

That day I dug all the way there by hand. With net-probe.jsonl, the same conclusion can be reached mechanically, at 5-minute granularity. The gateway field records the connection, and the dnsPerResolver field records how each individual DNS server responded to its query, so "when, on which connection, against which DNS server" can be reconstructed just by filtering fields. Whether a manual reproduction step (rebooting the router, pinning DNS to 1.1.1.1/8.8.8.8) is even needed can be decided case by case from this record.

Summary

Wi-Fi can be "connected" while only the DNS path is dead. Chrome carrying on as normal is because it has its own DNS cache — it is not evidence of health.
If root-cause tracing depends on "digging through the unified log after the incident," you run into traps like per-facility retention differences and misreading a symlink's mtime.
net-probe.sh is a thin mechanism that fires gateway / SSID / DNS / HTTP (v4, v6) / getaddrinfo all in parallel with timeouts and appends one JSONL line in under 20 seconds.
With launchd's StartInterval=300 + Nice 10 + LowPriorityIO, it runs quietly every 5 minutes without competing with other jobs.
662 lines in 4 days, 50 of them with failures>0 (7.6%). With this accumulated, the next time the same thing happens, reaching the cause means "filter the fields," not "dig after the fact."

Using this JSONL, I'd like to grow it to the point where a detected anomaly automatically notifies Discord and, if needed, switches connections automatically. That's for another post.

Have you ever had a job fail while the browser insisted everything was fine — and how long did it take you to find out DNS was the reason?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

DEV Community: Lily

5 Safety Guards for Auto-Archiving Another AI Agent's Conversations into My Vault: Designing a Manus API Bridge

The problem: Manus tasks weren't part of the primary sources

Design: credentials live only in the Keychain

Incremental sync: only process tasks whose revision changed

Self-healing: stale locks, and leaving nothing behind after a crash

Redacting secrets and meeting links

Backdating mtime: keeping the index timeline intact

The 5 safety guards at a glance

Pitfalls I hit

Summary

3 Weekly Runs, 3 Failures: My "Fixed" Claude Code Skill Auto-Update Failed Silently for 3 Weeks

The problem: npx skills@latest update -g is fragile against the unauthenticated GitHub API

Design: three layers of self-healing

Reading the real logs: "timeout or error" three weeks in a row

A contradiction in the STAMP file

Why the "fix" didn't fix anything

How this ties to the macos-hook-path-timeout skill

Pitfalls I hit

Summary

6 Pitfalls Syncing AI Memory to Google Drive With No API: Keeping File IDs Intact

The problem: add a sync path without adding another API

Design: push overwrites the same file, pull reads only a narrow set of notes

Push: overwriting without breaking the file id

Pull: only date-named plain-text notes

Reusing secret screening and symlink safety checks

A single-run lock to prevent concurrent runs

launchd config: every 5 minutes, low priority

Pitfalls I hit

Wrap-up

I Don't Trust the LLM That Writes My Shared Memory: 3 Regex Gates Before Claude Code Auto-Commits to My Obsidian Vault

The problem: unattended commits of conversation-log summaries to my Vault

Overview: isolate generation, distrust everything on receipt

Layer 1: Summarize with a Claude that has no privileges

Layer 2: The regex gate, zero-trust validation of model output

① Don't trust JSON parsing itself

② Reject content with regex

③ Restrict citations to the list of files actually provided

If validation fails, say nothing

Layer 3: Scan the files and the git diff again

The launchd configuration

Pitfalls I hit

Summary

4 Days, 3 Wasted Calls Per Run: My Retry Loop Mistook a Quota-Limit Message for a 'Too Short' Article

The problem: 4 straight days (9/13–9/16) of 3 failures per run

The fix: grep for the limit message and abort after 1 pass

The pitfall I stepped in: the sibling scripts still don't have it

Summary

One Line of mv Kills the 4x Duplicate Run After Recovery: Making launchd Queue Processing Idempotent

Why This Design Works

The Fork Between "Fixing by Habit" and "Fixing the Environment"

launchd Doesn't Know About "Fighting Over Files"

Backlog Plus Simultaneous Recovery Is What Surfaces the Race

Why a Lock File Isn't Enough

The Guard Philosophy Shown by autolike-plist-reconcile.sh

The Overall Flow

Architecture Diagram: From Race to Resolution

Why mv Is Atomic

Wiring It into the Pickup Script

Connecting with com.lily.codex-note-funnel.plist

Implementation Details

The Full Script: The Order for Stacking Defensive Layers

Why Override PATH at the Top

Why set -uo pipefail and || exit 0 Must Go Together

The "Double Check" Idiom Demonstrated by autolike-plist-reconcile.sh

Design Rationale for Startup Recovery

Where I Got Stuck

Stuck #1: The Race Was Gone, but the Error Log Kept Growing

Stuck #2: Files Kept Piling Up in processing/ While done/ Looked Empty Forever

Stuck #3: Works When Run from the Terminal, Silently Does Nothing from launchd

Stuck #4: ls *.json Returned Exit Code 2 on "No Files" and Crashed the Script

Stuck #5: It Took a Full Day to Notice funnel-pm Was Firing 4x

Pitfalls

Environment and Configuration

Traps Specific to the mv Claim Pattern

Operations and Monitoring Traps

Best Practices

Summary

2 Pitfalls in Priority Probes: Letting One Real Request Through an Open Circuit Breaker

The problem: the circuit stops every job equally, but posting can't wait

The problem: `npx skills@latest update -g` is fragile against the unauthenticated GitHub API

Why `set -uo pipefail` and `|| exit 0` Must Go Together

Stuck #4: `ls *.json` Returned Exit Code 2 on "No Files" and Crashed the Script

Stuck #5: It Took a Full Day to Notice `funnel-pm` Was Firing 4x

The design: give `--priority` jobs one attempt every 30 minutes

Pitfall 2: when the probe fails, it silently lands as `RAN exit≠0`

Where `--model` Goes and How to Write It

Designing `strict_quota_message`—The 400-Character Wall

`find` Was Taking 4 Minutes

preflight Was `pkill -9`-ing the Human's Chrome Too