This is a continuation of my "Claude Code environment" series. Last time I wrote about making Google Calendar the ground truth for my Vault. This time it's about the timeout hell that hit the same vault-auto-ingest job — and how I climbed out of it with "sub-step idempotency."
When hot.md failed to update three mornings in a row, the cause wasn't "launchd never fired" — it was "launchd fired but the job died partway through." I'd already put the outer retry in place (firing across multiple slots). The problem was on the inside.
The problem: step2 timed out every day and froze hot.md for three days
step2 of vault-auto-ingest.sh is the step that digests the last 28 hours of conversation logs and updates the Vault's wiki/. During busy periods, both Claude Code logs and Codex logs would pile up heavily in this step, and once you added the full-article rewrites, it stopped fitting inside the 40-minute window.
# vault-auto-ingest.sh L26-28 のコメント（実際の記述）
# step2 は活動多発期に28h分のClaude+Codexログ消化＋全記事リライトが40分枠に収まらず
# 連日timeout（hot.md凍結の真因, 2026-06-11〜13）。
The plist has four slots: 4:55 / 8:20 / 10:45 / 12:15. This is the outer retry — a design where, if today's success marker (DONE_MARKER) is present, later slots are skipped. But the premise for DONE_MARKER being set is that step2 passes — and since that step2 timed out every single time, the marker was never set, and all four slots kept failing in a row.
Note
The outer retry (multiple slots) is a mechanism for "getting it to succeed somewhere today"; it has no ability to "preserve the part that succeeded partway." That's exactly why an inner resume mechanism is needed.
The design: split by source and give each sub-step its own independent marker
The fix was to split step2 into two — "Claude log digestion" and "Codex log digestion" — and give each of them its own independent marker file.
STEP2_MARKER="$HOME/.claude/logs/.vault-ingest-step2-done-${TODAY}"
STEP2A_MARKER="$HOME/.claude/logs/.vault-ingest-step2a-claude-${TODAY}"
STEP2B_MARKER="$HOME/.claude/logs/.vault-ingest-step2b-codex-${TODAY}"
The markers form a three-layer structure.
| Marker | Meaning |
|---|---|
STEP2A_MARKER |
Claude log digestion done for today |
STEP2B_MARKER |
Codex log digestion done for today |
STEP2_MARKER |
Both done (composite marker) |
And these three layers are handled uniformly through the ingest_src() function.
ingest_src() { # $1=マーカー $2=ソースdir $3=ソース名 $4=timeout秒 $5=追加指示
local marker="$1" src="$2" name="$3" to="$4" extra="$5"
[ -f "$marker" ] && { echo "[$(date '+%F %T')] step2($name) は本日実施済み — skip" >> "$LOG"; return 0; }
cd "$VAULT" && run_to "$to" "$CLAUDE" -p \
"... ${src} に直近28時間で追加・更新されたファイル（${name}由来の会話ログ）だけを対象に ... ${extra}" \
--dangerously-skip-permissions >> "$LOG" 2>&1 \
&& { touch "$marker"; echo "[$(date '+%F %T')] step2($name) 完了" >> "$LOG"; return 0; } \
|| { echo "[$(date '+%F %T')] WARN: step2($name) 失敗/timeout（次スロットで再試行）" >> "$LOG"; return 1; }
}
The crux is the second line: [ -f "$marker" ] && return 0. If the marker exists, return immediately; if not, run claude -p and set the marker. That alone gives you the idempotency of "don't repeat an already-completed sub-step on a same-day retry."
The calls look like this.
ingest_src "$STEP2A_MARKER" "$KB/raw/conversations/" "claude" 1500 ""
ingest_src "$STEP2B_MARKER" "$KB/raw/codex-conversations/" "codex" 1500 "Codex由来でも既存記事に統合し、Claude側と重複する話題は新記事を作らず追記でまとめろ。"
# 両サブが済んだ時だけ合成マーカーを立てる
[ -f "$STEP2A_MARKER" ] && [ -f "$STEP2B_MARKER" ] && touch "$STEP2_MARKER"
Each timeout is 1500 seconds (25 minutes). By changing the original "40-minute window for both combined" into "25 minutes each × 2 runs, independent," even if one times out, only that one gets retried in the next slot.
What granularity to draw the markers at
There's a spectrum of choices from "one marker for everything" to "one marker per step." My criterion this time was this:
The boundary for drawing a marker = "a unit of work that can be independently re-run."
Claude logs and Codex logs are different sources; when one finishes, the other can be processed independently of it. That's the condition for being an independent unit. Conversely, "the final reconciliation of hot.md" is work that should be done once, looking at the results of both — so the right form is to place it once, independently, after the two ingests.
Splitting too finely raises management cost (marker files proliferate, and they're hard to follow when debugging). This time I used "source type" as the granularity, but if digesting a single source grew to take over 30 minutes, subdividing into "source × time window" would be the next move.
The composite-marker logic: use an AND condition to prevent regression
[ -f "$STEP2A_MARKER" ] && [ -f "$STEP2B_MARKER" ] && touch "$STEP2_MARKER"
Setting STEP2_MARKER with an AND condition is important — you must not use OR. With OR, you fall into the vicious cycle where "even on a day the Codex logs timed out and weren't digested, step2 is treated as complete, and the next day's Codex log volume swells further."
If all four of the day's slots finish with STEP2_MARKER still unset, DONE_MARKER won't be set either (just before setting DONE_MARKER there's a freshness check on today-brief.md, and if the ingest is incomplete the brief stays stale → the design deliberately doesn't write DONE_MARKER due to insufficient freshness). The next day's 4:55 slot doesn't run from scratch — only the incomplete ones among STEP2A/B run. This is the behavior of "inner resume."
Note
Only when multi-slot-launchd-retry (the outer retry) and substep-idempotent-marker (the inner resume) are combined is "it will definitely complete somewhere today" guaranteed. With only one of the two, all four slots can still fail.
Pitfalls I hit
- Cutting the timeout too short so it always failed → moved to 1500 seconds with room to spare for the source volume. On low-activity days it finishes in a minute, so there's no downside to setting it long.
-
If
STEP2_MARKERis already set, it bails out before the two sub-steps' skip checks → I placed theSTEP2_MARKERcheck outside (above) theingest_srccalls, returning early at the entrance. -
The marker date is fixed with
TODAY=$(date +%Y%m%d), but for a run that crosses midnight it would write to the next day's marker →TODAYis determined just once at the top of the script, so in practice it doesn't cross over. Still, you'll get bitten if you're not aware of it. -
On days when the Codex log source (
raw/codex-conversations/) is empty, ingest_src runs with nothing to do, exits normally, and STEP2B_MARKER gets set → since a no-op finishes normally, it's correct for the marker to be set. No problem. - The case where sleep-freezing leaves a marker half-set → handled with caffeinate -s (only effective on AC power) + resume in the next slot. A marker left behind by a frozen run is correctly treated as that day's accomplishment.
Summary
- The root cause of step2 timing out was "cramming two sources into one window." Splitting comes first.
- The
ingest_src()function encloses "if the marker exists skip, otherwise run and set the marker" into a single function. - Draw marker granularity at "a unit of work that can be independently re-run." Too fine and management cost rises.
- The composite marker uses an AND condition. With OR, incomplete digestion becomes permanent, a vicious cycle.
- The outer retry (multiple slots) and the inner resume (sub-step idempotency) have different roles. You need both.
Next time: the receiving side of the ingest results — the problem where hot.md's contents grew too large and started to pressure Claude's context, and a design that uses Obsidian's [[WikiLink]] structure to resolve references lazily.
Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*
Top comments (0)