DEV Community

Lily
Lily

Posted on • Originally published at dev.to

Turning Conversation Logs into Obsidian Long-Term Memory: A Pipeline That Distills Them Every Night

#obsidian #automation #ai #claudecode

This is part of my "Claude Code environment" series. In the post about splitting memory into four layers I described "conversation logs = layer 1" and "the Obsidian Vault = layer 4." This time I'll write about the implementation of the plumbing that distills from layer 1 into layer 4, automatically, every night.

Raw conversation logs are enormous and unreadable. The human-authored Obsidian wiki, on the other hand, is structured long-term memory. I run an unattended job that appends "just the last 28 hours, every night, while preserving the domain structure" from the former into the latter. What this article is really about is the way to kill silent failures and the mechanisms that prevent fabrication — things I only figured out by actually doing it.

The big picture: harvest → distill → back up

launchd fires it every morning, and it's roughly a three-step process.

  1. Harvest: refresh the conversation logs (layer 1) to the latest state (extract_conversations.py)
  2. Distill: with headless claude -p, append only the last 28 hours of diffs to the relevant domain in the Vault
  3. Back up: every run does git commit + push to a private repo (so a mess can be reverted)

The distillation is split into two runs by source (Claude conversations and Codex conversations). I'll explain why below, but the short version is that combining them into one run overran the time window and timed out every single night.

Trap 1: launchd can't touch protected folders (TCC)

This is where everything got stuck first. The Vault lives under ~/Documents, which is synced by iCloud and protected by macOS TCC (privacy protection). Even running git from launchd fails because it can't write into the protected area.

The nasty part is that this tends to become a "silent failure." So I detect it early in a preflight check and log it loudly plus send a notification.

# launchd配下では ~/Documents(保護領域) に触れない場合がある。
# ここで早期検知し、サイレント失敗(exit 0偽装)を防ぐ。
if ! ( cd "$VAULT" && git rev-parse --git-dir >/dev/null 2>&1 ); then
  echo "❌ FDA未付与: launchdから '$VAULT' にアクセス不可(TCC保護)" >> "$LOG"
  notify_fail "FDA未付与（設定→フルディスクアクセス→/bin/bash を許可）"
  exit 1
fi
Enter fullscreen mode Exit fullscreen mode

The fix is to "grant Full Disk Access to /bin/bash under System Settings → Privacy & Security → Full Disk Access." The script itself lives outside the protected area (~/.claude/scripts/). If you put it in ~/Documents, launchd can't exec it.

I also tried escaping to ~/ via a symlink, but that doesn't work because iCloud treats the symlink as a conflict and breaks it. When a protected area overlaps with iCloud sync, the constraints pile up in counterintuitive ways. Keeping the script outside protection and the data inside protection — drawing that line firmly — is the stable choice.

Trap 2: sleep and double execution → make it self-recovering

If you close the lid and it sleeps, the nightly job freezes. caffeinate -s only works on AC power, so on battery it just stops. So I made it self-recovering: "retry across multiple slots until it succeeds, and quit immediately once it does."

  • The plist fires on multiple slots such as 4:55 / 8:15 / 10:15 / 12:15
  • If today's run has already succeeded, later slots see the done marker and immediately exit 0 (a no-op that doesn't even dirty the log)
  • Double execution is excluded with an mkdir lock (stale locks are auto-reclaimed based on process liveness) 
DONE_MARKER="$HOME/.claude/logs/.vault-ingest-done-${TODAY}"
[ -f "$DONE_MARKER" ] && exit 0   # 本日成功済みなら即終了
Enter fullscreen mode Exit fullscreen mode

Splitting the distillation into two runs (Claude/Codex) was for the same reason. On busy days, digesting 28 hours of logs plus rewriting articles didn't fit in the 40-minute window, so it timed out day after day, and that in turn froze the hot cache. By giving each sub-step its own independent marker, even if one times out the next slot retries just the remainder.

No fabrication: keep the ground truth separately

This is the single most important lesson. When I threw "write today's brief from the conversation logs" at claude -p, it would sometimes plausibly invent commitment hours or durations for events that aren't in the notes. Seeing prose like "M/D–M/D," it would inflate it into "I'm tied up for this entire period."

There are two countermeasures.

① Make an actual snapshot of the calendar the canonical source. Instead of prose, I dump the time-stamped events pulled from the Google Calendar API into a separate file as ground truth, and instruct: "read this as the one and only canonical schedule."

② On fetch failure, preserve the last-known-good. Even if the API fails, don't wipe the canonical file to empty. I keep the previously successful values with a stale mark.

if [ -n "$CAL_SRC" ]; then
  cp "$CAL_TMP" "$CAL_SNAPSHOT"; cp "$CAL_SNAPSHOT" "$CAL_LASTGOOD"
elif [ -s "$CAL_LASTGOOD" ]; then
  # 取得失敗: 前回goodを温存し ⚠️stale 印で再構築（last-known-good）
  { echo "⚠️ 本日取得失敗。以下は前回成功時点の値(stale)。"; tail -n +4 "$CAL_LASTGOOD"; } > "$CAL_SNAPSHOT"
fi
Enter fullscreen mode Exit fullscreen mode

And I constrain it hard on the prompt side too: "Don't add information that isn't in the notes by guessing." "For date and time claims, explicitly separate confirmed information from speculation." If you're going to let an AI write your long-term memory, the only option is to structurally eliminate room for invention.

Freshness judgment: don't mistake an old artifact for "today's output"

When archiving a brief with a date stamp, I only accept files newer than this run's start time as today's output.

START_STAMP=$(mktemp ...)          # ラン開始時刻スタンプ
# ...生成...
if [ -s "$BRIEF_SRC" ] && [ "$BRIEF_SRC" -nt "$START_STAMP" ]; then
  cp "$BRIEF_SRC" "$ARCH_FILE"; touch "$DONE_MARKER"   # 新しい時だけ完了扱い
else
  notify_fail "ブリーフ未完 — 次スロットで自動再試行"   # 古いまま＝失敗、再試行へ
fi
Enter fullscreen mode Exit fullscreen mode

Without this, even when generation times out, it would copy "yesterday's old brief" and record it as "success." By judging freshness with mtime, I make sure the job never disguises a failure as a success.

Pitfalls I hit

  • Can't write to the protected area from launchd, silent failure → detect early with a TCC preflight, grant FDA
  • Escaping outside protection via symlink → iCloud breaks it with conflict handling. Not viable
  • Sleep freeze / double execution → multiple-slot retry + mkdir lock + done marker
  • 28h in one run overruns the window and times out day after day → two runs by source + independent sub-markers
  • Inventing an event's commitment hours → make the actual calendar canonical + last-known-good + no-guessing prompt
  • Mistaking an old artifact for today's output → only treat as done when newer than the run start stamp

Summary

  • Auto-distill conversation logs (layer 1) → Obsidian (layer 4), every night, just the last 28h of diffs
  • The protected area is blocked by TCC. Detect early, be loud, allow no silent failure
  • Handle sleep and double execution with multiple-slot retry + lock + done marker for self-recovery
  • If you let an AI write memory, keep the ground truth separately and structurally forbid guessing
  • Use mtime freshness judgment so it never disguises a failure as a success

Next time I'll write about the boss of this unattended job — running Claude Code's autonomous loop 24 hours a day and stopping runaways with cost.

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Top comments (0)