DEV Community

Lily
Lily

Posted on • Originally published at dev.to

How I Auto-Generate Claude Skills From Conversation Logs at 3:30 AM

#claude #ai #productivity #automation

Every night while I sleep, a batch job reads yesterday's conversation logs, distills the reusable procedures out of them, and writes them to disk as SKILL.md files. In my previous post I covered the lifecycle management of those auto-generated skills (the two-stage stale → archive retirement). This time I'm exposing the whole internal structure of the step that comes before that: the harvest batch that runs at 3:30 every night, reads the conversation logs, extracts only the reusable procedures, and writes them out as SKILL.md.

Right now ~/.claude/skills/auto/ has 104 skills stacked up in it. Almost all of them were generated unattended by skill-harvest.sh.

The problem: reading all 3,349 logs every night would torch the quota

~/Documents/my-knowledge-base/raw/conversations/ has accumulated 3,349 conversation logs as of today. If I fed all of them to an LLM every night, I could certainly harvest skill candidates. But at what cost?

3,349 logs × 10k tokens/log (average estimate) = 33 million tokens. Burning that on the Max quota every night would wipe out every other job.

I kill this with two mechanisms.

  1. Watermark method: never re-read a file that has already been processed
  2. Budget cap: fix the per-run LLM spend at a ceiling of --max-budget-usd 1.20

Combining the two keeps each night's harvest to a fixed "3 diff logs, 45KB."

The full pipeline: 5 phases

~/.claude/scripts/skill-harvest.sh runs in this flow.

1. ウォーターマーク読み込み → 差分ログを 3 本だけ取得
2. system-reminder ノイズを除去 → バイト上限でダイジェスト生成
3. claude -p ヘッドレス → ステージングに SKILL.md を書かせる
4. author:auto を保証 → AUTO ディレクトリへシェルがコピー
5. ウォーターマーク更新 → ログ追記して終了
Enter fullscreen mode Exit fullscreen mode

There are only four parameters (from the top of skill-harvest.sh).

MAX_LOGS=3              # 1回で扱うログ本数
PER_LOG_BYTES=15000     # ログ1本あたりの取り込み上限バイト
BUDGET_USD=1.20         # 暴走防止コストキャップ
TIMEOUT_SEC=600         # 壁時計タイムアウト
Enter fullscreen mode Exit fullscreen mode

The watermark method prevents duplicate processing

~/.claude/skills/auto/.harvest-watermark records the timestamp from when the previous harvest completed.

# 前回以降に更新されたログを収集（初回は最新 MAX_LOGS 件）
if [[ -f "$WM" ]]; then
  newlogs=("${(@f)$(find "$LOGS" -name '*.md' -newer "$WM" 2>/dev/null)}")
else
  newlogs=("${(@f)$(ls -t "$LOGS"/*.md 2>/dev/null)}")
fi

# mtime 降順で上位 MAX_LOGS 件に絞る
newlogs=("${(@f)$(ls -t "${newlogs[@]}" 2>/dev/null | head -$MAX_LOGS)}")
Enter fullscreen mode Exit fullscreen mode

find -newer "$WM" extracts only the files that are "newer than the watermark." Of the 3,349 logs, typically only 3–5 were updated between last night and this morning — so the set to process is smaller by orders of magnitude.

Right before exiting, the watermark is always updated.

touch "$WM"
exit 0
Enter fullscreen mode Exit fullscreen mode

You can verify this in the actual logs.

[2026-07-05 03:30:05] harvest start: 3 log(s), digest=   18255B
[2026-07-05 03:35:54] harvest done (exit 0, created=0)
Enter fullscreen mode Exit fullscreen mode

It processes about 3 logs and 18KB every night.

Controlling runaway cost with a $1.20 budget cap

The ceiling is passed to the --max-budget-usd flag of claude -p.

"$CLAUDE" -p "$PROMPT" \
  --model sonnet \
  --permission-mode acceptEdits \
  --allowedTools "Write Edit Read" \
  --max-budget-usd "$BUDGET_USD" >> "$LOG" 2>&1 < /dev/null
Enter fullscreen mode Exit fullscreen mode

The moment it exceeds $BUDGET_USD=1.20, claude aborts automatically. This runs on the flat-rate Claude Max quota, so $1.20 doesn't translate directly into out-of-pocket cost — but it does serve to fix how much of the quota is consumed and limit the impact on other jobs like autopilot.

The budget cap alone can't bound wall-clock time. The TIMEOUT_SEC=600 wall-clock timeout is implemented with perl -e 'alarm N; exec @ARGV'.

( cd "$STAGING" && perl -e 'alarm shift @ARGV; exec @ARGV' "$TIMEOUT_SEC" \
  "$CLAUDE" -p "$PROMPT" ... )
Enter fullscreen mode Exit fullscreen mode

On timeout it dies with exit 142 (SIGALRM = 128+14). It's there in the actual logs.

[2026-07-08 03:30:05] harvest start: 3 log(s), digest=   35159B
[2026-07-08 03:40:05] harvest done (exit 142, created=0)
[2026-07-09 03:30:06] harvest start: 3 log(s), digest=   35159B
[2026-07-09 03:40:15] harvest done (exit 142, created=0)
Enter fullscreen mode Exit fullscreen mode

It's force-killed at 10 minutes. Because a timeout is still recorded as created=0, the next day's harvest keeps running normally.

Note: macOS's stock coreutils doesn't ship GNU's timeout command. perl -e 'alarm N; exec @ARGV' is a portable alternative that works on macOS. gtimeout (Homebrew coreutils) works too, but it depends on the plist's PATH, which is more fragile.

Pre-filtering system-reminder noise to lighten the prompt

Conversation logs contain Claude Code's <system-reminder> tags and skill/tool listing lines written in verbatim. Shoving those into the prompt lets noise unrelated to procedural knowledge take up the bulk of the tokens.

grep -v -e 'system-reminder' -e '^- [a-z0-9].*:' "$f" 2>/dev/null | head -c $PER_LOG_BYTES
Enter fullscreen mode Exit fullscreen mode

There are two kinds of exclusions.

  • system-reminder: removes entire lines containing <system-reminder>
  • '^- [a-z0-9].*:': removes the - skill-name: description style lines of the skill/tool listing

This pulls in at most $PER_LOG_BYTES=15000 bytes from a single log. Combined with MAX_LOGS=3, what gets handed to the LLM in one harvest is fixed at a maximum of 45,000 bytes (about 45KB). No matter how many logs pile up, the prompt size doesn't change.

Writing files securely via a staging directory

Everything under ~/.claude/ can be write-protected while Claude Code is running. If you let claude -p write directly to ~/.claude/skills/auto/, it dies with permission denied.

STAGING=$(mktemp -d -t skill-harvest-stg)
( cd "$STAGING" && perl -e 'alarm shift @ARGV; exec @ARGV' "$TIMEOUT_SEC" \
  "$CLAUDE" -p "$PROMPT" \
  --permission-mode acceptEdits \
  --allowedTools "Write Edit Read" \
  --max-budget-usd "$BUDGET_USD" >> "$LOG" 2>&1 < /dev/null )
Enter fullscreen mode Exit fullscreen mode

mktemp -d creates a temporary directory under /tmp, and claude -p is run while cd "$STAGING". The prompt explicitly says "create ./kebab-name/SKILL.md directly under the current directory (absolute paths forbidden)."

- ファイル: ./<kebab-name>/SKILL.md （カレントディレクトリ直下に作る。絶対パス禁止）
Enter fullscreen mode Exit fullscreen mode

After claude finishes writing, the shell side scans $STAGING and copies into the AUTO directory.

for sd in "$STAGING"/*(/N); do
  [[ -f "$sd/SKILL.md" ]] || continue
  name="${sd:t}"
  [[ "$name" == .* ]] && continue
  # author: auto を保証（無ければ frontmatter 直後に挿入）
  grep -q '^author:[[:space:]]*auto' "$sd/SKILL.md" || python3 - "$sd/SKILL.md" <<'PY'
import sys,re
p=sys.argv[1]; s=open(p).read()
if s.startswith('---'):
    s=re.sub(r'^---\n', '---\nauthor: auto\n', s, count=1)
else:
    s='---\nname: %s\nauthor: auto\nversion: 1.0.0\n---\n'%__import__("os").path.basename(__import__("os").path.dirname(p))+s
open(p,'w').write(s)
PY
  [[ -e "$AUTO/$name" ]] || cp -R "$sd" "$AUTO/$name"
done
rm -rf "$STAGING"
Enter fullscreen mode Exit fullscreen mode

If author: auto isn't in the frontmatter, a python3 one-liner inserts it before transferring. This author: auto is the key used to decide what curate targets, so it's mandatory.

launchd registration

The schedule portion of the plist (~/Library/LaunchAgents/com.lily.skill-harvest.plist).

<key>StartCalendarInterval</key>
<dict>
    <key>Hour</key><integer>3</integer>
    <key>Minute</key><integer>30</integer>
</dict>
Enter fullscreen mode Exit fullscreen mode

It launches at 3:30 every day. It runs with ProcessType: Background, and logs are emitted to ~/.claude/logs/com.lily.skill-harvest.log.

Registration and an immediate test:

launchctl load ~/Library/LaunchAgents/com.lily.skill-harvest.plist
launchctl start com.lily.skill-harvest
Enter fullscreen mode Exit fullscreen mode

Pitfalls I hit

  • launchd's PATH is minimal: in a bare launchd environment, claude isn't found. Unless you explicitly add nvm's bin and ~/.local/bin to the plist's EnvironmentVariables/PATH, it exits immediately
  • Direct writes to ~/.claude get permission denied: it's protected while Claude Code is running. Worked around with staging + shell copy
  • The timeout command isn't standard on macOS: substituted with perl -e 'alarm N; exec @ARGV'
  • Forget to touch the watermark and it re-scans all logs every time: the touch "$WM" right before exit must always be placed immediately before exit 0
  • exit 142 is recorded looking the same as created=0: since a timeout can't be distinguished from a normal exit, get in the habit of telling them apart by the time gap between start and done in the log (normal: a few minutes, timeout: exactly 10 minutes)
  • A skill with the same name already existing can't be overwritten: because [[ -e "$AUTO/$name" ]] skips the copy, updating an existing skill (patch) is handled on the prompt side by instructing "if it's a duplicate, Read the existing one and patch it"

"A miss is normal," as seen in the real logs

Here's this week's harvest log.

[2026-07-05 03:30:05] harvest start: 3 log(s), digest=   18255B
この会話ログ抜粋を確認しました。INDEX.md はファイル一覧のみで
手順的知識なし。tsuzuke-ios の 2 件は既存の
xcodegen-project-yml-security-review スキルでカバー済み。
該当なし
[2026-07-05 03:35:54] harvest done (exit 0, created=0)

[2026-07-06 03:30:05] harvest start: 3 log(s), digest=   18255B
一回性レビュー＋抽出基準に非該当。よってファイルは作成しませんでした。
[2026-07-06 03:30:45] harvest done (exit 0, created=0)

[2026-07-07 03:30:05] harvest start: 3 log(s), digest=   35008B
[2026-07-07 03:32:28] CREATED: note-bokumolily-factual-review
[2026-07-07 03:32:28] harvest done (exit 0, created=1)
Enter fullscreen mode Exit fullscreen mode

Four out of five days are created=0. A miss isn't an anomaly — it's proof the extraction criteria are working correctly. Duplicates of existing skills, one-off personal circumstances, and idle chat all get rejected. Because the LLM makes the rejection call autonomously, it doesn't churn out junk.

Note: A created=0 harvest consumes almost no cost (Max quota). It's just enough to launch claude -p, read the prompt, and return "nothing applicable." "Misses being cheap" is part of why I can run this nightly batch every single night.

Summary

  • Watermark method: extract only the diff logs with find -newer, keyed off the mtime of .harvest-watermark. Even with 3,349 logs, it processes only 3 per night
  • Double budget cap: --max-budget-usd 1.20 + perl alarm 600 puts a ceiling on both LLM spend and execution time
  • Noise filter up front: adding grep -v 'system-reminder' as preprocessing compresses the prompt to a fixed 45KB
  • Writing via staging: mktemp -d lets the shell mediate, avoiding write contention against ~/.claude
  • A miss is normal: with tight extraction criteria, a run of created=0 days is healthy behavior

I wrote about how skills crossing the 100 mark raised the importance of the curate side in the previous piece. Next time I plan to write about how these two batches piling up skills and memory started to make Claude Code slow to launch — auditing and reducing the amount of context injection.

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Top comments (1)

Collapse
 
fromzerotoship profile image
FromZeroToShip •

"A miss isn't an anomaly — it's proof the extraction criteria are working correctly." I want that line tattooed on every alerting system I own. Most people tune a pipeline until it always produces something, which is exactly how you train it to manufacture noise. You built the opposite: silence as a positive signal.

I do the manual, artisanal version of what you automated. I'm not an engineer (physical therapist who builds hospital tools with AI), and I run a persistent memory system by hand — after a session I distill what was actually learned into a rule or a memory file the next session inherits. Reading yours, the part that hit hardest wasn't the automation, it was your discipline about what NOT to keep. My failure mode is the opposite of a missed extraction: I over-save, and every "individually justified" note quietly taxes every future load until the index itself is the problem. Your created=0 four days out of five is the exact restraint I lack.

The noise pre-filter is the piece I'm stealing tonight. I strip system-reminder tags too, but by hand and inconsistently — moving that to a deterministic grep before the model ever sees the log is obviously right the moment you say it. Cheaper, and it stops the boilerplate from diluting the signal the extractor is hunting for.

One question, since you've clearly thought about it: how do you keep 104 auto-skills from colliding or going stale? The generation problem I've mostly solved; it's the garbage collection I never do.