DEV Community: oji - building AI in public

300問の問題集をAIで自動生成！ 250問目でAPIエラー、75分が水の泡に。チェックポイント機構で“途中再開”を実装した話

oji - building AI in public — Sun, 04 Oct 2026 23:30:18 +0000

最近、ある資格試験の勉強をしてて、公式の問題集だけじゃ足りないなと思ってた。じゃあ作るか、と。GenAIを使えば、類題なんていくらでも作れる時代だし。

早速、試験のシラバスを食わせて、章ごとに問題と解説を300問分生成するスクリプトを書いた。
python generate_questions.py を叩いて、あとは待つだけ。プログレスバーがぐんぐん伸びていくのを見ながら、「いやー、便利な時代になったもんだ」なんて思ってた。

実行開始から75分後。プログレスバーは8割を超えたあたり。そろそろ終わるかな、とコンソールを覗き込んだら、見たくない赤い文字が表示されてた。

requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

まあ、API呼び出しのエラー自体はよくあること。問題はその後。成果物が出力されるはずのディレクトリを見たら、ファイルが空っぽ。

……え？

そう。250問近くまで生成したはずのデータが、どこにもない。75分という時間と、それなりの額のAPIトークンが、一瞬で溶けた。これは、やばい。

原因：ナイーブすぎた全件一括処理

原因は完全に自分の設計ミス。実装がナイーブすぎた。
書いたコードは、ざっくりこんな感じだった。

空のリスト all_questions = [] を用意。
for ループで300回、GenAIのAPIを叩く。
生成された問題を all_questions.append(new_question) でリストに追加していく。
ループが全部終わったら、all_questions の中身をまとめてJSONファイルに書き出す。

この設計の何がまずいか。それは、途中で一度でも処理がコケたら、メモリ上にあった all_questions の中身が全部消し飛ぶこと。

外部APIを叩くような長時間バッチ処理で、この実装は致命的だった。ネットワークは気まぐれだし、APIサーバーだってたまには機嫌が悪くなる。そういう「中断」を全く想定してなかったのが敗因。

修正：キリのいいところで保存・再開する

このままだと、生成が終わるまで神に祈るゲームになる。エンジニアとしてそれはありえない。
そこで、「チェックポイント機構」を実装することにした。

考え方はシンプルで、「キリのいいところまで処理が進んだら、その時点での成果をファイルに保存しておく」というもの。そして、もしスクリプトが中断しても、次に起動したときに「どこまで終わってたか」をファイルから読み込んで、続きから再開できるようにする。

具体的には、章ごとの問題生成が終わるたびに、結果をファイルに追記していく方式にした。ファイル形式はJSONL（JSON Lines）がこういう用途に便利。1行が1つの独立したJSONオブジェクトになってるから、追記がめちゃくちゃ楽。

修正後のコードはこんな感じ。

import os
import json

# exam_slug は 'shikaku_xyz' みたいな試験の識別子
def generate_with_checkpoint(exam_slug):
    # 試験のシラバス（章のリスト）を読み込む
    chapters = load_syllabus(exam_slug)
    # 進捗を保存するチェックポイントファイル
    checkpoint_file = f"app_factory/exams/{exam_slug}/generated.jsonl"

    # すでに完了した章のIDを保持するセット
    completed_chapters = set()
    # チェックポイントファイルが存在すれば、中身を読んで完了済みリストを作る
    if os.path.exists(checkpoint_file):
        with open(checkpoint_file, "r", encoding="utf-8") as f:
            for line in f:
                # 1行ずつJSONをパースして、chapter_id をセットに追加
                completed_chapters.add(json.loads(line)["chapter_id"])

    # シラバスの全チャプターをループ
    for chapter in chapters:
        # もし、この章がすでに完了済みならスキップ
        if chapter["id"] in completed_chapters:
            print(f"Skipping completed chapter: {chapter['id']}")
            continue

        # 未処理の章なので、AI生成APIを呼び出す
        print(f"Generating for chapter: {chapter['id']}...")
        new_items = call_genai_api(chapter["prompt"])

        # 生成が終わったら、結果をチェックポイントファイルに「追記」する
        with open(checkpoint_file, "a", encoding="utf-8") as f:
            for item in new_items:
                # どの章の成果物かわかるように chapter_id を付与
                item_with_meta = {"chapter_id": chapter["id"], **item}
                # JSON文字列に変換して、改行をつけて書き込む
                f.write(json.dumps(item_with_meta, ensure_ascii=False) + "\n")

このコードのポイントは3つ。

起動時に進捗を確認: スクリプトが始まったら、まず checkpoint_file が存在するかチェック。あれば中身を読んで、どの章が完了済みかを completed_chapters セットに記録する。
処理済みタスクのスキップ: for ループの中で、処理対象の章が completed_chapters に含まれていたら、APIを叩かずにスキップする。
進捗の永続化: 1つの章の処理が終わるたびに、open(..., "a") で追記モードでファイルを開き、結果を書き込む。これで、たとえ次の章でエラーが起きても、そこまでの成果はファイルに残る。

この修正で、スクリプトは何度実行しても大丈夫な「冪等性（べきとうせい）」を持つようになった。途中で止まっても、もう一度同じコマンドを叩けば、何事もなかったかのように続きから再開してくれる。

学び：中断は必ず起こる前提で設計する

今回の失敗で学んだのは、長時間実行されるバッチ処理、特に外部APIみたいな不確定要素が絡むスクリプトは、「中断は必ず起こる」という前提で設計しなきゃいけない、ということ。

「全部終わったらまとめて保存」はシンプルだけど、脆い。
「少しずつ進めて、都度保存」は少し手間だけど、頑健。

この「チェックポイントを設けて途中再開できるようにする」という考え方は、AIのコンテンツ生成だけじゃなくて、大規模なデータ処理、Webスクレイピング、インフラのプロビジョニング（Terraformとかも内部的にはstateファイルで似たようなことやってる）とか、いろんな場面で応用が効く。

個人開発のツールなんて、動けばいいと思いがちだけど、こういう一手間を加えておくだけで、運用がえぐいほど安定する。75分を無駄にしたのは痛かったけど、おかげで今後の開発で同じ轍を踏むことはなくなるはず。良い勉強代だったと思うことにした。

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

If a provider-agnostic RAG Q&A API is useful to you, mine is MIT-licensed on GitHub: rag-faq-api. It runs and passes its full test suite **with no API key* (offline stub LLM + hashing embedder), swaps to Claude / Gemini / OpenAI via one env var, and ships a retrieval-quality harness (Hit@k / MRR / Recall@k) with a chunking sweep.*

My Python patch failed silently: the `\n` in a triple-quoted string became a literal newline!

oji - building AI in public — Sat, 03 Oct 2026 23:30:20 +0000

Hey everyone, it's your friendly neighborhood dev, 38, building AI trading bots as a side hustle.

One evening after my main job, I was updating library dependencies for my custom bot. I needed to tweak a specific library's behavior, so I wrote some code to directly patch its source.

The usual drill: I stored the diff content for the patch command in a Python triple-quoted string (like a here-document), then executed it via a subprocess.

But it just wouldn't work.

No errors, nothing. The library's behavior remained completely unchanged after the "patch." The worst kind of bug: the silent failure. I probably spent about two hours debugging this.

Long story short, the culprit was a specific detail of Python's string literal behavior. The \n I wrote inside the here-document was being converted into an actual newline character, instead of being treated as the literal two-character sequence "backslash and n."

The Symptom: Patch Fails Silently

Here’s a simplified version of what I was doing. Preparing a diff text as a string and piping it to subprocess for the patch command.

# The problematic code (conceptual)
import subprocess

# Path to the library file to be patched
target_file = "/path/to/some/library/file.py"

# Dynamically generated patch (defined as a here-document)
patch_content = """
--- a/file.py
+++ b/file.py
@@ -123,7 +123,7 @@
 class SomeClass:
     def some_method(self):
-        return "old_string\n"
+        return "new_string_with_patch\n"
"""

# Execute the patch command
proc = subprocess.run(
    ['patch', target_file],
    input=patch_content,
    text=True,
    capture_output=True
)

# Check the result
if proc.returncode != 0:
    print("Patch application failed!")
    print(proc.stderr)
else:
    print("Patch applied (or so I thought...)")

Running this code, returncode was 0. stderr was empty. It printed "Patch applied." But when I actually checked target_file, its content was untouched.

Initially, I suspected issues with the patch command's path or permissions. But creating a .patch file with the exact same content and applying it manually from the command line worked perfectly.

This meant the patch_content being passed from the Python script was somehow malformed.

The Cause: `\n` became a literal newline

With the hypothesis that "the string content is wrong," I started by printing patch_content to the console.

--- a/file.py
+++ b/file.py
@@ -123,7 +123,7 @@
 class SomeClass:
     def some_method(self):
-        return "old_string
"
+        return "new_string_with_patch
"

At first glance, nothing seemed wrong. The diff format looked correct.

This is where I got stuck. But on closer inspection, something felt off. The line that should have been return "old_string\n" was instead return "old_string followed by a newline, with the closing " on the next line.

That's when it clicked. The \n was being interpreted as an escape sequence and converted into an actual newline character (LF).

Python's triple-quoted strings """...""" interpret escape sequences like \n and \t just like regular strings. In this case, I wanted to include the literal string literal return "old_string\n" as part of the patch diff. But Python's interpreter, being helpful, converted \n to a newline, resulting in an invalid diff format that the patch command couldn't understand.

This is a nasty one. It's a spec I know, yet when you're tired, it's easy to fall into this trap.

The Fix: Use raw strings `r"""..."""`

Once the cause was identified, the fix was simple: prevent backslashes within the string literal from being escaped.

There are two ways:

Escape the backslash itself (\\n)
Use a raw string

Since I didn't want any escape sequences interpreted in the entire patch content, using a raw string made the intention clearest.

# Corrected code
import subprocess

target_file = "/path/to/some/library/file.py"

# Use a raw string (r"""...
""") to disable escape sequences
patch_content = r"""
--- a/file.py
+++ b/file.py
@@ -123,7 +123,7 @@
 class SomeClass:
     def some_method(self):
-        return "old_string\n"
+        return "new_string_with_patch\n"
"""

# Rest of the code is the same
proc = subprocess.run(...)

Just add an r before the opening quotes of the string. Now, \n inside patch_content is no longer a newline character, but simply the two-character string "backslash and n."

With this fix, the patch applied successfully. What a journey...

Lesson Learned: Use `repr()` when dealing with code as strings

The takeaway from this experience is simple:

When treating text that contains meaningful escape sequences (like code or configuration files) as a string, either use raw strings or pay extreme attention to how escape sequences are handled.

Here-documents are especially convenient for multi-line text, which also makes them prone to this specific trap.

Another lesson learned is about debugging. Because I used print() to output the string, the visual difference was subtle, delaying discovery. In such cases, repr() should have been my go-to.

print(repr(patch_content)) would instantly show how Python interprets the string.

# Example output of print(repr(patch_content))
'--- a/file.py\n+++ b/file.py\n...return "old_string\\n"\n...'

This way, it would be crystal clear whether it was \\n (as intended) or \n (unintended). Ugh, I should have done this from the start.

In personal projects, especially side hustles, these minor gotchas can easily eat up several hours. But accumulating these failure logs, one by one, is ultimately the fastest way to improve.

Hope this helps anyone else who might fall into a similar trap.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My AI Agent Kept Missing Deadlines and Double-Posting — Here's How I Fixed It with JSON Interfaces and Idempotency Checks

oji - building AI in public — Fri, 02 Oct 2026 23:30:19 +0000

Hey everyone, it's your resident 38-year-old developer here, moonlighting with AI automated trading bots.

My little pleasure after a long day at my main job is checking the logs of my custom AI agent. Every night, it analyzes a specific market and records its preliminary trading decisions in JSON format. Pretty smart little helper, usually.

But one morning, checking the logs as usual, I found a mess:

Three identical trading decision records for the same date.
Decisions that should have been logged by 23:59 the previous day were somehow being attempted at 2:30 AM, failing with a "deadline exceeded" error.

For a moment, I panicked, thinking the AI had bugged out and gone rogue. An agent designed for autonomous operation was repeating tasks and missing critical deadlines. This was a pretty serious situation.

Today, I want to share the nitty-gritty of how I prevented this "AI meltdown."

The Problem: AI Is Too Obedient, and Unaware of Execution Environment Quirks

Upon closer investigation, the AI itself hadn't gone crazy. The root cause lay in the overall "mechanism" that ran the AI agent.

My system was a simple setup: a Windows Task Scheduler would kick off a Python script at a fixed time every night, and that script would call an LLM (Claude) to generate the decision.

The problem arose when, for some reason, the Task Scheduler re-executed the task. For example, a brief network outage, or a slight delay in response due to machine load. In such cases, the OS might "helpfully" retry the task, thinking, "Oh, did that task fail? I'll just run it again."

The AI, however, knows nothing about these OS-level nuances. It simply received the command to "make a decision and record it," and dutifully went through its thought process from scratch each time, writing down the results. Even if I added "execute only once" to the prompt, it wouldn't prevent OS-level retries.

In essence, entrusting the entire "make a decision and write it down" process to the AI was the fundamental design flaw.

The Fix: Separating "Context Gathering" from "Decision Writing"

To address this, I clearly separated the responsibilities of the Python script (decide.py), which served as the interface to the AI, into two distinct parts.

Step 1: The AI retrieves "context" as JSON for its decision-making.

First, I created a --context mode, specifically for retrieving the information the AI needs to think.

py -3.12 C:/path/to/my/project/decide.py --context

Executing this command returns a JSON object to standard output, summarizing current market information, historical data, and so on. This command never alters the system's state, no matter how many times it's run. It just reads data. So, it's safe.

The AI agent first understands the current situation using this command.

Step 2: The AI writes its "decision result" in JSON format.

Next, I prepared a --write mode that simply accepts the AI-generated decision JSON and saves it to a file or database.

py -3.12 C:/path/to/my/project/decide.py --model claude-fable-5-1 --write '{"session_date": "2024-05-20", "decision": "buy", "reason": "...", "confidence": 0.85}'

The crucial part is that I implemented rigorous pre-write checks within this --write script.

# Part of decide.py (conceptual)
import argparse
import json
from datetime import datetime

def write_decision(data):
    # 1. Deadline Check
    session_date_str = data.get("session_date")
    session_date = datetime.strptime(session_date_str, "%Y-%m-%d").date()

    # If the deadline has passed, don't write and raise an error
    if datetime.now().hour > 23: # Example: 23:00 is the deadline
        raise ValueError(f"Deadline passed for session {session_date_str}")

    # 2. Idempotency Check (prevent duplicate writes)
    if record_exists(session_date):
        print(f"Record for {session_date_str} already exists. Skipping.")
        return # If it exists, do nothing and exit successfully

    # Only if all checks pass, execute the write operation
    save_to_database(data)
    print(f"Successfully wrote record for {session_date_str}")

# ... argparse handling for --context and --write ...

With this modification, even if the Task Scheduler mistakenly executes the command multiple times:

First execution: Passes checks and writes successfully.
Subsequent executions: The record_exists check identifies "already recorded" and finishes without doing anything.

The deadline check also blocks the write operation if the deadline has passed.

This property, where executing an operation multiple times yields the same result, is called "idempotency." By implementing this mechanism, I gained safe control over the AI agent's execution.

Lesson Learned: Designing AI Autonomy and Its Control Boundaries

The key takeaway from this failure is that if you're giving an AI agent autonomy, you absolutely must design "safe execution boundaries" from the human side.

Clearly separate read operations (reading state) from write operations (changing state). This is fundamental to Web API design, and it applies perfectly to AI agent integration.
All write operations must be idempotent. AI and its operating environment might not always behave as expected and execute only once. It's essential for the receiving end to control the system, ensuring it doesn't break no matter how many requests it receives.
Let the AI focus on "thinking." Procedural tasks like deadline checks or duplicate submission prevention are more reliable and cost-effective when handled by external programs rather than trying to force them into AI prompts.

Delegating tasks to an AI might be similar to assigning work to a smart new hire. They're capable, but they don't know all the operational rules or company culture. So, before allowing critical operations, always establish double-check and permission-setting mechanisms.

Developing AI agents isn't just about clever prompt engineering; it's about building up these subtle interface designs and error handling practices that are key to stable operation. This incident truly drove that point home for me.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My Monitoring Bot Went "Silent": 3 Task Scheduler Pitfalls and How Negative Testing with API Snapshots Saved Me

oji - building AI in public — Tue, 29 Sep 2026 23:30:17 +0000

Hey there, it's OJ, a part-time AI agent and algo-trading bot developer.

Today, I want to share a common pitfall in personal development: a web monitoring bot I built suddenly went "silent." No error logs, no alerts, just complete silence. These silent failures are the worst kind.

The culprit? Classic Windows Task Scheduler gotchas and a flaw in my monitoring system's design. I eventually solved it by implementing a robust mechanism to detect when the monitoring itself failed. Here's how it went down.

What Happened: The Weekly Bot's Silence

I had built a weekly bot to check for unintended changes to my content on an online learning platform. Rather than dealing with a heavy, unstable browser, I leveraged their public API to fetch titles and descriptions. Simple and elegant.

I tested it locally, everything looked good, and I registered it with Task Scheduler. Set it to run every Sunday night, and mostly forgot about it.

Then one day, it hit me: "Wait, I haven't received any notifications from the bot recently." Not even a success log. Checking Task Scheduler's history showed either no execution records or cryptic statuses like "Task was denied by an operator or administrator."

If it's not running, fine, throw an error! But silence? That's what really kills you.

The Cause: 3 Task Scheduler Traps

My investigation revealed several traps in Windows Task Scheduler's default settings, especially when developing and running on a laptop.

1. Won't Run on Battery Power

Open the task properties, go to the "Conditions" tab, and you'll find "Start the task only if the computer is on AC power." This is checked by default.

Meaning, if I unplugged my laptop on the weekend and moved it to the living room, the task's conditions wouldn't be met. Consequently, the bot wouldn't launch at its scheduled time.

2. Won't Run if PC is Asleep

Another common one. In the "Settings" tab, there's an option: "Run task as soon as possible after a scheduled start is missed." If this isn't checked, and your PC is asleep or shut down at the scheduled time, the task is just skipped.

Leave your laptop closed over the weekend, and your task might never get a chance to run.

3. Console Character Encoding (cp932) Issue

While not the direct cause this time, this is a hotbed for silent failures. When running Python scripts via Task Scheduler, standard output can sometimes be processed with cp932 (Shift_JIS).

If your script prints UTF-8 characters, you'll instantly hit a UnicodeEncodeError. The catch? This error output might not be logged anywhere, leaving you clueless that the task even failed. Essential to guard against.

The Fix: 2 Steps to Reliable Monitoring

First, I reviewed all my Task Scheduler settings based on these pitfalls:

Unchecked "Start the task only if the computer is on AC power."
Enabled "Run task as soon as possible after a scheduled start is missed."
As a precaution, I added chcp 65001 to my batch file to switch to UTF-8, or configured it via Python environment variables.

This largely solved the "task not executing" problem.

However, a fundamental issue remained: I couldn't distinguish between "the monitored content hasn't changed" and "the monitoring bot failed for some reason."

So, to detect "monitoring failure" itself, I integrated negative testing using approved snapshots.

Here's the refined flow:

Hit the monitoring target's API to fetch the JSON response.

GET /api-2.0/courses/7347629/?fields[course]=title,headline,description,...

Compare the fetched JSON with a pre-saved "golden JSON file" (the snapshot).
If there's a difference, notify, of course.
Crucially: Even if there's no difference, always send a log indicating "comparison process completed successfully."

Now, if the bot doesn't run, I won't receive the "success log," and I'll know immediately.

Furthermore, to ensure the reliability of this mechanism itself, I perform "negative testing." I intentionally corrupt the snapshot file or mock the API response to generate differences, and then verify that notifications are indeed sent when an anomaly occurs.

Updating snapshots is made easy with a simple command:

# Save the current API response as the new "golden" version
python -m app_factory.udemy.published_check --bless genai_passport

This also helps in adapting to changes in the monitored API's structure.

The Lesson: How to Know If Your Creation Isn't "Dead"

What I learned from this experience is that personal automation tools, if left as "set it and forget it," can silently die without you ever knowing.

Task Scheduler is handy, but question its default settings. Always double-check power conditions and behavior during sleep.
To prevent silent failures, designing for "proof of success" is crucial. "No errors = success" is not enough. Think: "Success log present = success."
Monitoring system reliability is guaranteed through negative testing. Regularly testing "will I get a notification if it breaks?" is incredibly important, even for personal projects.

Flashy feature development is fun, but this kind of gritty operational improvement ultimately saves your time. It's a small defense against being betrayed by your own automation tools.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My PC Screamed! How a `grep -r` Trap Led to 100% CPU and How I Prevented It with a JavaScript Git Hook

oji - building AI in public — Wed, 23 Sep 2026 23:30:17 +0000

Hey there, it's your resident "old man" developer, 38 years old and tinkering with AI trading bots in my spare time.

One evening, as I was routinely checking my bot's logs, disaster struck. I wanted to search the entire repository for a specific log output, so I mindlessly typed this into my terminal:

grep -r "some_keyword" .

The next second, my PC's fans roared like a Formula 1 car, and my mouse cursor started lagging severely. I opened Task Manager, and sure enough, CPU usage was pegged at 100%. I genuinely thought my machine was on its last legs. The culprits? grep.exe and, consequently, Windows Defender's real-time scan, which went haywire.

Ultimately, I had to force-restart my PC. Who knew a simple grep could cause such a catastrophe? Today, I'm sharing the story of this "grep scream incident" and the mechanism I built to prevent it from ever happening again.

The Problem: `.gitignore` Is Just a Text File

Why did grep go rogue? The reason became clear quickly.

My repository contained several massive log files generated by the bot. These were temporarily placed there for testing, with the largest one clocking in at 23GB. Naturally, I had these files properly listed in .gitignore to exclude them from Git's tracking.

But here's where my huge misconception lay: the shell's grep command knows absolutely nothing about .gitignore. To grep, .gitignore is just ignore.txt—nothing more, nothing less.

grep -r recursively, and naively, reads all files under the current directory. So, it plunged headfirst into that 23GB log file. No wonder the CPU screamed.

This same issue can occur with find . | xargs grep "..." or PowerShell's Select-String -Path . -Pattern "..." -Recurse. Essentially, any command that recursively scans the file system is susceptible to this trap.

Especially when working with automated trading bots, gigabyte-sized logs and data files are a daily occurrence. It seems personal development environments are particularly prone to these kinds of accidents.

The Solution: "Being Careful" Isn't Enough, So I Automated It

Even if I vow to use rg (ripgrep) or ag—which respect .gitignore—I can already foresee myself accidentally typing grep out of habit. Humans forget, and humans make mistakes.

So, instead of relying on "being careful," I opted for an approach that would "physically prevent execution."

Specifically, I leveraged a "hook" mechanism that kicks off a specific script right before a command is executed in the terminal. If a dangerous grep command is about to run, the hook detects and blocks it.

I wrote this hook script in JavaScript. What it does is simple: it checks the command string using a regular expression.

// Check the command line about to be executed with a regular expression

// Roughly detect grep-like command call patterns
const GREP_CALL = /(?:^|[\s(])(?:[^\s|;&]*[\\/])?(r|e|f)?grep(?:\.exe)?(?:\s+(.*))?$/s;

// Detect short recursive flags like -r or -R
const SHORT_RECURSIVE_FLAG = /(?:^|\s)-(?!-)[A-Za-z]*[rR][A-Za-z]*(?=\s|$)/;

// findstr /s on Windows is equally dangerous, so detect it too
const FINDSTR_SUBDIRS = /\bfindstr(?:\.exe)?\b[^|;\n]*\s\/s\b/i;

function shouldBlock(toolName, command) {
  // Only target Bash or PowerShell execution
  if (toolName !== 'Bash' && toolName !== 'PowerShell') return false;

  // findstr /s is an immediate block
  if (FINDSTR_SUBDIRS.test(command)) return true;

  // Split the command by pipes or semicolons and check if any segment
  // includes a recursive grep call
  return command.split(/\|\||&&|[|;\n]/).some(isRecursiveGrepSegment);
}

function isRecursiveGrepSegment(segment) {
  const match = segment.match(GREP_CALL);
  if (!match) return false;

  const args = match[2] || '';
  // Check for long flags like --recursive or --directories=recurse
  const hasLongRecursiveFlag = /\s--(recursive|directories=recurse)\b/.test(args);
  // Check for short flags like -r, -R
  const hasShortRecursiveFlag = SHORT_RECURSIVE_FLAG.test(args);

  return hasLongRecursiveFlag || hasShortRecursiveFlag;
}

With this script integrated into a hook, if I accidentally type grep -r ., the command will be blocked before execution with a message like: "Dangerous command! Please use rg instead." Peace of mind achieved.

The regular expression is a bit messy because I wanted to detect not just grep -r but also -R, --recursive, and grep commands connected via pipes. findstr /s causes equally nasty accidents, so I included it in the block list as well.

Summary and Lessons Learned

What I learned from this incident is, while obvious, crucial: you'll get burned if you don't properly understand what your tools are doing implicitly.

.gitignore is Git's rulebook, not the shell's.
Mechanical prevention is more robust and reliable than relying on self-discipline.
In personal development, especially with automation and bot development, huge files can be generated unintentionally. Safety measures in the development environment, though seemingly trivial, are incredibly important.

Just grep, but grep can be a beast. To keep my PC from screaming, I need to properly understand the characteristics of my tools. With this, I've made my development environment safer for those late-night coding sessions. 👍

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

When AI hallucinated 'systemd' and 'cron' on my Windows machine: The 'context blindness' trap of LLMs

oji - building AI in public — Tue, 22 Sep 2026 23:30:17 +0000

Hey, it's Grandpa Oji (@oji_ai_dev) here.

I'm a 38-year-old developer who tinkers with AI bots in my side hustle and works as a full-time engineer during the week. Recently, I decided to start a simple dev blog – separate from my notes – to properly document my personal projects.

Naturally, I wanted to leverage AI for some of the article generation to boost efficiency. As a first step, I asked it to write a setup guide for a simple bot I run periodically via Windows Task Scheduler.

I fed it the logs and source code, then vaguely prompted: "Explain how to set up this bot." When I read the first draft, the opening lines immediately threw me for a loop.

"First, register it as a service using systemd. Place the following unit file in /etc/systemd/system/..."

...Huh? systemd?

My bot runs on a plain old Windows 11 machine, not Linux. systemd doesn't exist there. Reading further, it continued: "For periodic execution, it's convenient to register a schedule using crontab -e." It had completely mistaken the operating system.

This was rough. It strung together plausible-sounding technical terms, but the content was pure fantasy. It was a spectacular moment of AI 'context blindness.'

The Cause: Implicit Assumptions Don't Translate

Why did this happen? One hundred percent, it was because my prompt was too vague.

My instruction to the AI was literally just, "Write a blog post explaining this script." I completely omitted crucial, seemingly obvious information like the OS being Windows and that it was scheduled with Task Scheduler.

When an LLM receives incomplete information, it attempts to fill in the gaps with the "most probable, general knowledge" from its vast training data. When the topic shifts to demonizing or periodically executing scripts on a server, the vast majority of technical articles in the world assume a Linux environment.

So, the AI reasoned with a ridiculously simple inference: "Okay, this person wants to run a bot on a server. Servers typically mean Linux, right? Therefore, I should talk about systemd and cron."

From my perspective, I'm thinking, "No, it's my Windows machine humming in the corner of my room," but the AI doesn't pick up on that 'vibe.' Any context that I consider 'obvious' must be explicitly stated in words, or the AI won't grasp it.

The Fix: Give AI a Map in the Form of 'Constraints'

The way to prevent this 'context blindness' is simple: clearly define "constraints" in your prompt.

For example, in this case, my initial vague prompt looked like this:

# Bad example
Write a blog post explaining how to periodically run the following Python script.

# --- Script ---
import time
from pathlib import Path

def main():
    log_file = Path("log.txt")
    with log_file.open("a", encoding="utf-8") as f:
        f.write(f"Executed at {time.time()}\n")

if __name__ == "__main__":
    main()

With this, the AI felt free to interpret and started talking about Linux.

So, I gave it a "map" in the form of the execution environment:

# Good example
Write a blog post explaining how to periodically run the following Python script.

# Constraints
- OS: Windows 11
- Execution Method: Task Scheduler
- Audience: Beginners using Python on Windows

# --- Script ---
import timerom pathlib import Path

def main():
    log_file = Path("log.txt")
    with log_file.open("a", encoding="utf-8") as f:
        f.write(f"Executed at {time.time()}\n")

if __name__ == "__main__":
    main()

By explicitly stating the OS and execution method under "# Constraints," the accuracy of the AI's output dramatically improved. When I regenerated with this prompt, it correctly provided sensible steps for a Windows environment, starting with "Open Task Scheduler, then 'Create New Task'...".

The Lesson: AI is Smart, But Not Psychic

This incident reinforced my view that LLMs are like "incredibly competent, but utterly clueless, new hires."

If your instructions are ambiguous, the AI, with good intentions, will fill in the gaps with general knowledge. But if that knowledge doesn't align with the actual situation (i.e., the execution environment), the output won't just be useless – it can be actively harmful.

Especially when having AI handle technical content, you must communicate the "execution environment context" upfront, even if it feels tedious. This includes the OS, library versions, frameworks, and hardware configuration. Neglecting this will result in it confidently generating non-existent commands or code that won't run due to version mismatches.

The generated content often looks plausible, which means there's a serious risk that someone less knowledgeable might believe it. Tragedies like debugging incorrect systemd configuration files for hours could easily happen.

Delegating tasks to AI is convenient, but you can't escape the need for precise "prompt engineering" and the "human oversight" of checking the final output. Slack on these fundamentals, and you'll only end up causing yourself trouble.

This was a good failure log that made me rethink how I interact with AI.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My Bot Went Silent: The Case of the Missing "Always-On" Process

oji - building AI in public — Mon, 21 Sep 2026 23:30:18 +0000

It's your friendly neighborhood AI dev, oji (@oji_ai_dev)! Today I want to share a recent, pretty critical mistake I made with my side-hustle AI trading bot. A newly implemented feature completely failed to run after deployment – no errors, no logs, just a complete, silent death.

The root cause? A fundamental misalignment in assumptions about design and operation within our small team (which includes me!). A classic, basic oversight.

The New Feature: Totally Mute After Deployment

It all started when I added new periodic tasks to the bot. Up until then, most tasks ran daily. This time, I was adding weekly and monthly report generation features.

A remote team member handled the logic, and my role was to deploy it to production. I reviewed the code, confirmed it ran fine in my local dev environment, and thought, "Great, we'll start getting those reports next week."

A few days passed.

Suddenly, I wondered, "Is it actually running?" I checked the production server logs. ...Nothing. There should have been logs like "Weekly task started," but there was no trace.

At first, I thought, "Maybe it's not time yet?" But checking the calendar, it was clearly overdue. It had been completely skipped. No error logs anywhere. This was bad.

The Shocking Truth Behind the Search

My first suspicion was the code. But it worked in dev. Next, I checked Python library dependencies on the production environment. No issues there either.

Completely stumped, I decided to deep-dive: check process status on the production server, cron jobs, task scheduler settings – everything.

Midway through documenting my investigation, I found the smoking gun:

`scheduler.py` has never run on this machine. There are 0 persistent processes and 0 task scheduler registrations. All periodic executions are handled by `Oji*` tasks (e.g., `OjiNotePost`).

The moment I read that, everything clicked. It felt like a punch to the gut.

The new feature was designed around a scheduler.py script that would run as a persistent process, launching tasks at specified times. It's a common implementation using Python's schedule library.

Naturally, the development team assumed this scheduler.py would be running as a daemon in the production environment.

But on my home server, no one had ever started such a process.

A Twist Between Design Philosophy and Operational Reality

How did this happen?

My operational policy for my home server was to avoid persistent processes wherever possible to conserve memory and prevent zombie processes. That's why all existing periodic tasks were executed by the OS task scheduler (Windows Task Scheduler, in my case), which directly kicked off individual Python scripts at scheduled times.

So, in summary:

Design Assumption: scheduler.py runs persistently to manage tasks (persistent process model).
Operational Reality: The OS launches individual scripts on demand (event-driven model).

This architectural premise was completely misaligned between myself and the dev team. I assumed, "Just drop the code, and I'll register it with the OS scheduler." The team assumed, "It will be deployed to a server where scheduler.py is already running."

This "assumption gap" silently killed the new feature. We hadn't documented anything, and the most crucial part – "how it runs" – was completely overlooked. Brutal.

The Fix and the Lesson

The fix was simple. Instead of launching a persistent process, I adapted to the existing operational setup.

I refactored the new feature's logic into an independent script and registered it with the OS task scheduler. Now it runs using the same mechanism as all other tasks.

I learned some big lessons from this failure:

Shared understanding of architecture is critical. Especially in solo dev or small teams, it's easy to assume "everyone gets it," but this leads to fatal accidents.
"How it runs (operations)" is as important as "what it does (features)." This kind of issue frequently arises when development and operations are separated.
The only reliable sources of truth are code and configuration files. Even a simple README.md with a production process diagram or a list of launch commands could have prevented this incident.

A feature that just "doesn't run" without throwing errors is the hardest to detect and the scariest. I've now firmly committed to implementing a mechanism to regularly check if expected artifacts (log files, DB records, etc.) are actually being generated.

This incident was a stark reminder that side-hustle dev is a continuous series of gritty failures and learnings.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My PC was Slow as Molasses: The Culprit? Orphaned `grep` Processes After a Shell Timeout.

oji - building AI in public — Sun, 20 Sep 2026 23:30:17 +0000

Hey there, oji_ai_dev here.

Working on AI agents and bots as a side hustle often brings a different flavor of technical challenge compared to my day job. This time, my development PC suddenly ground to a halt, and I spent several hours debugging the issue.

Long story short, the culprit wasn't Windows Defender (though it looked like it) but rather a bunch of "orphaned grep processes" hiding behind it.

The Symptom: PC Stuck at 100% CPU Usage

One evening, as I was setting up my dev environment and coding, my PC's fans suddenly started roaring. Typing in VSCode became sluggish, almost unworkable. Clearly, something was very wrong.

I opened Task Manager, and sure enough, the CPU usage was pegged at 100%. At the top of the list was the familiar MsMpEng.exe – Windows Defender.

"Not you again..."

Any developer has probably experienced this. When you generate or read a large number of files, Defender's real-time scanning can go haywire and bring your PC to its knees. "Oh, it's just one of those days," I thought, and temporarily disabled Defender.

...But the situation didn't change. CPU was still at 100%. This was bad.

The Investigation: The Real Culprit Behind Defender

If Defender wasn't the culprit, what was eating up all that CPU? Staring at the Task Manager's "Details" tab, and looking at the processes consuming CPU time, I noticed several unfamiliar processes:

grep.exe

Why was this here? And multiple instances of it? This was GNU grep bundled with Git for Windows. I didn't remember starting it directly.

Then, I recalled my actions just prior. I had used an AI assistant (Claude) to search the contents of a massive monorepo. This repository was a chaotic mess of accumulated logs and build artifacts from years of operation, totaling about 34GB.

The command I fed to the AI assistant was something simple, like this:

grep -r "some_legacy_function" .

The shell environment used by the AI assistant is configured to time out after a certain period (e.g., 300 seconds) for safety. Most likely, searching such a huge repository took too long, and the parent shell died due to a timeout.

Here's where the problem started.

In a Linux environment, when a parent shell dies, its child processes often terminate along with it. However, grep.exe running within Git Bash on Windows behaved differently. Even after its parent was gone, the child processes continued to run.

These are "orphan processes."

Multiple grep.exe instances, having lost their parent, continued to relentlessly scan 34GB of files, unmanaged. No wonder the CPU was at 100%. What appeared to be Defender going berserk was merely a secondary phenomenon: these orphaned processes were generating an insane amount of file I/O, and Defender's scan was reacting to it. I had completely misidentified the root cause.

The Fix and Prevention

Once the cause was known, the fix was simple. I manually force-terminated all suspicious grep.exe processes from Task Manager. The roaring noise immediately subsided, and my PC quieted down, with CPU usage dropping to around 5%. Peace was restored.

However, this left me vulnerable to the same mistake again. A more fundamental solution was needed.

First, I decided to stop indiscriminately using grep -r on huge repositories. It completely ignores .gitignore and diligently searches through node_modules, large binaries, and everything else, which is terrible for performance. Instead, I've standardized on ripgrep (rg), which intelligently searches while respecting .gitignore and is much faster.

Furthermore, I decided to implement a hook in my AI assistant's execution environment to prevent potentially dangerous commands from being run in the first place. This simple mechanism checks the command's content before the tool (in this case, Bash) is executed, and if it matches certain patterns, it throws an error and stops.

The concept is a script similar to this:

// .claude/hooks/block_recursive_grep.js (conceptual hook script)
const tool = process.env.CLAUDE_TOOL_NAME;
const params = JSON.parse(process.env.CLAUDE_TOOL_PARAMS);

if (tool === 'Bash') {
    const command = params.command;
    // Define dangerous command patterns that trigger recursive searches
    const recursivePatterns = [
        /grep\s+.*\s-r\b/,          // grep -r
        /grep\s+.*\s--recursive\b/, // grep --recursive
        /find\s+.*\s\|\s*xargs\s+grep/, // find | xargs grep
        /findstr\s+.*\s\/s\b/       // findstr /s (Windows)
    ];

    // If any pattern matches, block the command and exit with an error
    if (recursivePatterns.some(p => p.test(command))) {
        console.error(`ERROR: Recursive grep is blocked due to performance risk on large repos. Use the 'Grep' tool (ripgrep) instead, which respects .gitignore and is much faster.`);
        process.exit(1); // Block the command
    }
}

process.exit(0); // Allow other commands

With this hook in place, if I accidentally tell the AI to use grep -r in the future, it will be blocked before execution and prompted to "use ripgrep instead because it's dangerous." For someone as forgetful as me, these kinds of physical guardrails are the most effective.

Summary

I learned two main lessons from this incident:

Shell timeouts don't guarantee child processes will be killed. Especially in Windows environments, it's crucial to be aware of the risk of orphaned processes consuming resources.
Don't be fooled by superficial symptoms (like Defender running wild). Always suspect that there might be a true culprit generating massive I/O behind the scenes.

Time for side projects is limited, so losing half a day to environment issues like this is truly painful. But every failure makes my environment more robust. I hope this incident log helps save someone else's precious time. 👍

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

Oops! Almost Leaked User Data with My AI Feature – A Wake-Up Call from API Terms of Service

oji - building AI in public — Sat, 19 Sep 2026 23:30:17 +0000

Hey there, it's your friendly neighborhood dev, 38 years old, slinging code as an engineer by day and tinkering with AI trading bots on weekends.

Recently, I decided to spice up a personal tool I'm building on the side by adding a little AI magic. The idea was simple: users type some text, and the AI generates something nice based on it. I spun it up quickly, hitting the Gemini API from the backend, and it worked like a charm. "Oh, this is actually pretty neat!" I thought, grinning to myself.

Once the feature felt stable, I figured it was time to write a user manual. That's when I started digging into the official documentation, not just the API reference I'd skimmed during implementation, but also the terms of service and FAQs.

And then, I stumbled upon a single sentence that made my blood run cold.

Free Tier API Input Data Can Be Used for Product Improvement

I found it on Google's policy page regarding data usage for their AI services. It clearly stated:

(Paraphrased) Content submitted through the free tier APIs (including prompts and responses) may be used to improve Google's products and services, such as generative AI models.

This was a huge red flag.

My tool allows users to input any text they want. What if a user accidentally types in confidential information – like customer data from work, or a snippet from a top-secret project proposal?

That data would then travel through my tool, to Google's servers, and potentially be used as training data for their AI models. Sure, they probably anonymize it and process it in various ways, but the fact that the terms explicitly state it "may be used" is a serious concern.

Users wouldn't know any of this background. They'd just be using my tool as a "handy text generator." My tool could inadvertently become a pipeline for leaking user's confidential information to an external entity. Realizing this possibility sent shivers down my spine. This was a nightmare scenario.

My Immediate Fixes

It was a blessing in disguise that I caught this before release. I immediately took several steps.

First, I added a prominent disclaimer about the AI feature to my tool's manual and FAQ. It roughly said something like this:

[Important] Caution Regarding AI Feature Use

This feature utilizes an external AI service (Google Gemini API).

Per the terms of service, data entered into this feature may be used by Google, the service provider, for product improvement (e.g., training AI models).

Therefore, absolutely DO NOT input any personal information, company confidential information, or any other data that should not be shared externally.

By using this feature, you agree to the above.

Next, I updated the UI. Right below the input form, I added a link to this FAQ page with a label like "Important Usage Notes." I felt it was crucial to issue a warning right where the feature is used, not just burying it in the manual.

Finally, on the implementation side, I made sure to document this risk in the docstring of the function that calls the API. This serves as a reminder for future me, or anyone I might collaborate with.

# WARNING: This function sends data to an external AI service.
# The data sent via the free tier API may be used for service improvement.
# Do NOT send personal, confidential, or sensitive information.
# See our FAQ for more details.

import google.generativeai as genai
import os

# Best practice: get API key from environment variables
genai.configure(api_key=os.environ.get("GEMINI_API_KEY"))

# Select the model
model = genai.GenerativeModel('gemini-1.5-flash')

def generate_text_from_prompt(prompt_text: str) -> str:
    """
    Generates text from a user prompt.

    WARNING: This function sends data to an external API.
    Data submitted via the free tier API may be used by the service provider
    for service improvement. DO NOT send confidential information.
    Refer to the FAQ for more details.
    """
    if not prompt_text:
        return "Input text is empty."

    try:
        response = model.generate_content(prompt_text)
        return response.text
    except Exception as e:
        # In a real app, you'd log this more robustly
        print(f"An error occurred: {e}")
        return "An error occurred. Please try again later."

Embedding warnings directly into the code is a form of defense. It makes the risk visible even at the source code level.

My Key Takeaway

The lesson I learned from this experience is simple but incredibly important:

When integrating external APIs, read not just the functional specifications, but also the terms of service and privacy policy, thoroughly.

This is especially true for AI services, where data handling is core to their operation. Free tiers versus paid tiers can have vastly different implications, not just for feature limits, but also for data privacy. This was precisely the case here.

Being satisfied with "it works" and immediately pushing it to users is genuinely risky. As developers, we bear the responsibility for how what we build impacts our users. Since this feature involved sending user-entered data externally, I should have been extra cautious, and I regret not having been so initially.

I'm sure more fantastic AI services will emerge. My desire to leverage them to build cool things hasn't changed. But this incident hammered home the importance of truly understanding the underlying "contract" and fulfilling our responsibility to protect users while using these powerful tools.

Personal projects offer freedom, but that also means you're solely responsible. Every time I experience a close call like this, it's a stark reminder to stay sharp.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

"API Limit 21KB!" ...Then 32KB of data caused a mysterious error. The culprit? Confusing bytes with characters.

oji - building AI in public — Fri, 18 Sep 2026 23:30:15 +0000

Hey, it's your resident Old Man Developer here. One evening, while checking the logs for my self-made bot as usual, I noticed a recurring, inexplicable error in a section that interacts with an external API.

The logs showed messages like Payload too large. "Ah, typical," I thought. But something felt off. The API documentation clearly stated a "21KB payload limit," and my implementation was supposed to have a logic check to verify data size before sending it.

Yet, it was still being rejected. My local check passed, but the API screamed "too big!" This contradiction baffled me for quite a while.

The Cause: `len()` was the Root of All Evil

Initially, I suspected API capriciousness, or perhaps a temporary network glitch. But the error consistently reproduced with specific data patterns. Okay, this was on my end.

I pulled out the problematic data and measured its size locally.

# Assuming data_string holds the problematic data
print(len(data_string))

It was clearly smaller than 21 * 1024 = 21504. Still, the API was rejecting it. I just couldn't understand why.

After about half a day of scratching my head, it suddenly hit me: the data I was handling contained a fair amount of Japanese text. "Could it be... byte count?"

To test this hypothesis, I wrote a simple piece of code.

text = "こんにちは世界"

char_count = len(text)
byte_count_utf8 = len(text.encode('utf-8'))

print(f"String: '{text}'")
print(f"Character count: {char_count}") # => 7
print(f"Byte count (UTF-8): {byte_count_utf8}") # => 21

Seeing this, everything clicked.

"こんにちは世界" is only 7 characters. But encoded in UTF-8, it becomes 21 bytes. That's a 3x difference! For English (ASCII) text, 1 character is basically 1 byte, so len() roughly matches the byte count. But when multibyte characters like Japanese come into play, the story changes entirely.

In essence, I was checking the "character count" with len(), but the API was looking at the "byte count" calculated by len(str.encode('utf-8')). The "21KB" in the documentation had, in my mind, been unconsciously translated to something like "21k characters." Damn, it was a complete assumption on my part.

When I re-measured the actual error-causing data by byte count, it easily exceeded 21KB, coming in at nearly 32KB. No wonder the API was upset!

The Fix: Switch to Byte-Based Check Logic

Once the cause was known, the fix was simple. Just change the size check before sending to the API from character-based to byte-based.

def check_api_limit(data_string, limit_in_kb):
    # Encode the string in UTF-8 and calculate byte count
    byte_size = len(data_string.encode('utf-8'))

    # Convert KB limit to bytes for comparison
    limit_in_bytes = limit_in_kb * 1024

    if byte_size > limit_in_bytes:
        # Raise an error here to prevent the API call
        raise ValueError(f"Data size ({byte_size} bytes) exceeds limit ({limit_in_bytes} bytes)")

# Calling side
try:
    check_api_limit(my_data, 21)
    # Logic to send to API
except ValueError as e:
    print(f"Error: {e}")
    # Handle splitting, error handling, etc.

I implemented a function like this and placed it right before the API call. Now, I can accurately detect size overages locally before the API even gets a chance to complain. Since deploying this fix, the mysterious error has completely disappeared.

Lessons Learned: Question Units, Measure Empirically

The lessons I learned from this failure are quite simple yet crucial:

Question Unit Definitions: When documentation uses ambiguous terms like "size" or "KB," I need to make it a habit to accurately confirm whether it refers to "character count" or "byte count." If it's not specified, I should have tested with minimal data to verify behavior.
Don't Forget the Multibyte Trap: This issue constantly arises, especially when building AI agents that handle natural language data like Japanese. Always keep in mind that character count and byte count are completely different.
Assumptions are the Biggest Enemy: Python's ease of use, like getting "size" with len(), ironically became the pitfall this time. Doubting my own "I know this" assumptions is a subtle but incredibly important practice.

So, that was my story of wasting half a day due to confusing "character count" and "byte count" – a slightly embarrassing but common mistake anyone can make.

Squashing these small, unassuming errors one by one is the reality of personal development, especially when you're working on side projects. I'll share again if I screw up in another interesting way.

X: @oji_ai_dev
note: oji_ai_dev

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My custom filter showed "zero false positives"! Then I realized my validation logic was flawed, letting all the garbage through.

oji - building AI in public — Thu, 10 Sep 2026 23:30:18 +0000

Hey there, it's your "Oji" (38-year-old AI/quant dev on the side). During the week, I'm a regular company employee; on weekends, I tinker with AI agents and automated trading bots.

Recently, I was building a filter to automatically extract "AI-related companies" from a market data list, based on their business descriptions. The mechanism is simple: it scores companies based on hits of predefined positive keywords (e.g., "machine learning," "natural language processing") and negative keywords (e.g., "gaming," "entertainment").

To improve the filter's accuracy, I set up a test to tune the threshold (how many positive keyword hits qualify a company). I prepared lists of positive examples (actual AI companies) and negative examples (non-AI companies) to see how well the filter classified them—standard stuff.

Then I ran the test, and the results were mind-blowing.

"Zero false positives, no matter the threshold I tried."

For a moment, I thought, "Is my filter a genius?" Zero false positives is usually impossible. But seconds later, I cooled down. Results that are too perfect usually mean something is wrong.

Sure enough, when I manually reviewed the list of companies that passed the filter, I found obvious non-AI companies like gaming studios mixed in. My filter was letting garbage through, yet the test reported "no issues." The despair I felt realizing this late on a weekday night was quite something.

Why "Zero False Positives"?

The root cause wasn't a bug in the code itself, but a design flaw in the validation logic.

Basic accuracy validation involves looking at two metrics:

False Positive (FP): A non-AI company incorrectly classified as an AI company.
False Negative (FN): An AI company incorrectly classified as a non-AI company.

The issue I encountered was with FP. To count FPs, the definition of negative examples (companies that are "not AI companies") is crucial.

And here was my definition of a negative example:

"A company with 0 positive keyword hits AND 2 or more negative keyword hits."

This was the source of all evil. This definition, seemingly sound at first glance, contained a self-contradiction.

My filter's rule for classifying a company as "passing (AI-related)" was: "2 or more positive keyword hits."

You probably see it now.

Filter's passing condition: positive_hits >= 2
Test's negative example condition: positive_hits == 0

These two conditions can never be true simultaneously. A company classified as "passing" by the filter already has positive_hits >= 2, so it can never fit the negative example definition of positive_hits == 0.

In essence, when trying to count "negative examples that were incorrectly classified as passing (i.e., false positives)," the structure itself made it impossible for any company to be both a "negative example" and "passing." Of course, false positives would always be zero.

Translating the concept to code:

// Filter rule: Pass if positive keyword count >= 2
function is_theme_company(positive_hits) {
  return positive_hits >= 2;
}

// Validation's negative example definition (buggy)
function is_negative_example(positive_hits, negative_hits) {
  // Only consider companies with zero positive hits as negative examples
  return positive_hits === 0 && negative_hits >= 2;
}

// Validation process
let false_positives = 0;
for (const company of all_companies) {
  const is_selected = is_theme_company(company.pos_hits);

  // If it's a negative example but selected, count as FP
  if (is_selected && is_negative_example(company.pos_hits, company.neg_hits)) {
    false_positives++;
  }
}
// With this logic, is_selected=true (pos_hits>=2) and is_negative_example=true (pos_hits==0)
// can never both be true, so false_positives will always be zero.

The test was too tightly coupled to the logic it was supposed to validate, preemptively deciding the outcome. It was a brutal rookie mistake.

How I Fixed It

Once I understood the cause, the fix was simple.

I changed the definition of negative examples to be independent of keyword hit counts. Specifically, I manually selected dozens of companies that were unequivocally not AI companies (e.g., food manufacturers, apparel brands, construction companies) and created a fixed "negative example list."

Running the test again with this list, false positives, as expected, came pouring out. Finally, the numbers showed that the filter was indeed picking up many irrelevant companies. This was the real starting point for tuning.

My Takeaway

The lessons from this failure are significant.

"The test passed" only means something if the test itself is correct.

Finding code bugs with tests is fundamental, but you must always question the possibility that the test logic itself is flawed. Especially when validating rules you've created yourself, there's a risk that the validation logic gets dragged by the tested logic, unconsciously leading to "conclusion-first" tests.

When you get "too perfect results," first question your own assumptions. This is a crucial reminder I'll engrave into my mind.

This story also applies to backtesting automated trading bots. When a backtest shows unusually good performance, it's rarely because you've discovered a brilliant strategy; it's usually because you're looking at future data (leakage) or not accounting for fees and slippage.

I'm logging embarrassing failures like this as part of building in public. Hopefully, it helps someone else in their solo dev journey.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

米国企業の分析bot、外国企業だけ“黙って”分析をスキップしてた話 — 10-Kと20-Fで「リスクの章立て」が違う罠

oji - building AI in public — Tue, 08 Sep 2026 23:30:19 +0000

どうも、おじいです。平日夜と休日に、AI エージェントとか自動売買 bot をちまちま作ってる 38 歳。

今日は、副業で開発してる企業分析 bot でやらかした、結構えぐいバグの話。動いてるように見えて、実は大事なデータをごっそり見逃してた。データ処理系の個人開発やってる人には、あるあるかもしれない。

何が起きたか

作ってる bot の一つに、米国上場企業の年次報告書（SEC ファイリング）を読み込んで、事業リスクを LLM で要約・評価するやつがある。企業の健全性をざっくり把握するためのツールだ。

こいつが、ある特定の企業群について、やけに「クリーン」な分析結果を出してくることに気づいた。「リスク要因: 特になし」みたいな。最初は「お、超優良企業か？」なんて思ってたんだけど、何社か続くとさすがにおかしい。そんなわけない。

ログを掘ってみたら、衝撃の事実が判明した。bot が、特定の企業の分析だけ「黙って」スキップしてた。エラーも吐かずに、ただ空っぽのデータを返してた。そのせいで、後段の LLM は「リスクに関する記述がなかった」と判断して、「リスクなし」という結論を出してたわけだ。これはやばい。

原因: 「10-K」と「20-F」という様式の違い

原因は、米国証券取引委員会（SEC）に提出される年次報告書の「様式」の違いにあった。

bot は当初、米国企業が提出する「Form 10-K」という書類だけを想定して作ってた。この 10-K では、事業リスクは「Item 1A. Risk Factors」という項目に記載されるのがお作法。だから、bot は機械的に「Item 1A」のセクションを引っこ抜くように実装してた。

ところが、分析結果が空になってた企業を調べたら、全部「外国企業」だった。米国市場に上場してる外国企業は、10-K の代わりに「Form 20-F」という別の様式で報告書を提出する。

そして、この 20-F では、リスク要因は「Item 3.D. Risk Factors」に書かれている。

つまり、bot は 20-F の書類に対して、存在しない「Item 1A」を探しに行ってた。当然、見つかるわけがない。

一番の問題は、その後の処理だった。セクションが見つからなかった時のエラーハンドリングを、こんな風に書いてた。

# 抽出失敗と「該当なし」を混同する危険なコード
try:
    risk_text = doc.get_section("Item 1A")
except SectionNotFound:
    risk_text = "" # これでは失敗したのか、元々空なのか分からない

一見、例外をキャッチしてて安全そうに見える。でも、これが罠だった。「セクションが見つからなくて抽出に失敗した」ケースと、「セクションはあったけど中身が空だった」ケースが、どっちも risk_text = "" になってしまう。

このせいで、データ欠損という重大な異常が、正常な「リスク記載なし」として処理され、静かにバグが進行してたわけだ。

修正: フォームタイプをちゃんと見て、失敗を区別する

対策はシンプル。
まず、処理対象のドキュメントが 10-K なのか 20-F なのかをちゃんと判別する。その上で、フォームタイプに応じて読み込むべきセクション ID を切り替えるようにした。

# フォームタイプに応じて処理を分岐する改善コード
if doc.form_type == "10-K":
    section_id = "Item 1A"
elif doc.form_type == "20-F":
    section_id = "Item 3.D"
else:
    # 未対応のフォームタイプなら、処理を中断
    log.warning(f"Unsupported form type: {doc.form_type}")
    return None

# 抽出を試みる。失敗したらNoneが返るようにget_section側を修正
risk_text = doc.try_get_section(section_id)

if risk_text is None:
    # セクション自体が見つからなかった場合（＝異常）
    log.error(f"Section not found: {section_id}")
    # ... エラー処理 ...
elif risk_text == "":
    # セクションはあったが空だった場合（＝正常）
    log.info(f"Risk factors section is empty: {section_id}")

get_section メソッドも修正して、セクションが見つからない場合は空文字じゃなく None を返すように変更した。

これで、呼び出し側は戻り値を見るだけで、

None → セクションが見つからなかった（＝想定外のエラー）
"" （空文字）→ セクションはあったが、中身が空だった（＝データ上、リスク記載なし）
(str) → 正常に抽出できた

と、明確に区別できるようになった。この修正後、これまでスキップされてた外国企業の分析も、無事に動き出した。

学び: エラーを握りつぶすと、静かに死ぬ

今回の失敗から学んだことは２つ。

ドメイン知識はマジで大事。 今回で言えば、SEC ファイリングに 10-K と 20-F の違いがある、という知識。技術的な実装力だけじゃなくて、扱ってるデータそのものへの理解がないと、こういう根本的な見落としをする。特に金融系は、こういう「お作法」の塊みたいな世界。
「失敗」と「空」を区別する設計。 エラーを安易に握りつぶして、空文字や 0 みたいな「無害そうな」デフォルト値を返すのは、本当に危険。サイレントなデータ欠損は、気づいた時には手遅れ、なんてこともあり得る。自動売買 bot なら、資産を溶かす原因に直結する。None を使ったり、専用の例外を投げたりして、異常は異常としてちゃんと後続の処理に伝える設計がいかに重要か、身をもって知った。

副業の個人開発だと、ついドキュメントを斜め読みして「動いたからヨシ！」で進めがちだけど、こういう地味な仕様の確認こそ、一番時間をかけるべき部分なのかもしれない。

この失敗ログが、同じようにデータ処理系の bot を作ってる誰かの参考になれば嬉しい。

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

DEV Community: oji - building AI in public

300問の問題集をAIで自動生成！ 250問目でAPIエラー、75分が水の泡に。チェックポイント機構で“途中再開”を実装した話

原因：ナイーブすぎた全件一括処理

修正：キリのいいところで保存・再開する

学び：中断は必ず起こる前提で設計する

My Python patch failed silently: the `\n` in a triple-quoted string became a literal newline!

The Symptom: Patch Fails Silently

The Cause: \n became a literal newline

The Fix: Use raw strings r"""..."""

Lesson Learned: Use repr() when dealing with code as strings

My AI Agent Kept Missing Deadlines and Double-Posting — Here's How I Fixed It with JSON Interfaces and Idempotency Checks

The Problem: AI Is Too Obedient, and Unaware of Execution Environment Quirks

The Fix: Separating "Context Gathering" from "Decision Writing"

Lesson Learned: Designing AI Autonomy and Its Control Boundaries

My Monitoring Bot Went "Silent": 3 Task Scheduler Pitfalls and How Negative Testing with API Snapshots Saved Me

What Happened: The Weekly Bot's Silence

The Cause: 3 Task Scheduler Traps

1. Won't Run on Battery Power

2. Won't Run if PC is Asleep

3. Console Character Encoding (cp932) Issue

The Fix: 2 Steps to Reliable Monitoring

The Lesson: How to Know If Your Creation Isn't "Dead"

My PC Screamed! How a `grep -r` Trap Led to 100% CPU and How I Prevented It with a JavaScript Git Hook

The Problem: .gitignore Is Just a Text File

The Solution: "Being Careful" Isn't Enough, So I Automated It

Summary and Lessons Learned

When AI hallucinated 'systemd' and 'cron' on my Windows machine: The 'context blindness' trap of LLMs

The Cause: Implicit Assumptions Don't Translate

The Fix: Give AI a Map in the Form of 'Constraints'

The Lesson: AI is Smart, But Not Psychic

My Bot Went Silent: The Case of the Missing "Always-On" Process

The New Feature: Totally Mute After Deployment

The Shocking Truth Behind the Search

A Twist Between Design Philosophy and Operational Reality

The Fix and the Lesson

My PC was Slow as Molasses: The Culprit? Orphaned `grep` Processes After a Shell Timeout.

The Symptom: PC Stuck at 100% CPU Usage

The Investigation: The Real Culprit Behind Defender

The Fix and Prevention

Summary

Oops! Almost Leaked User Data with My AI Feature – A Wake-Up Call from API Terms of Service

Free Tier API Input Data Can Be Used for Product Improvement

My Immediate Fixes

By using this feature, you agree to the above.

My Key Takeaway

"API Limit 21KB!" ...Then 32KB of data caused a mysterious error. The culprit? Confusing bytes with characters.

The Cause: len() was the Root of All Evil

The Fix: Switch to Byte-Based Check Logic

Lessons Learned: Question Units, Measure Empirically

My custom filter showed "zero false positives"! Then I realized my validation logic was flawed, letting all the garbage through.

Why "Zero False Positives"?

How I Fixed It

My Takeaway

米国企業の分析bot、外国企業だけ“黙って”分析をスキップしてた話 — 10-Kと20-Fで「リスクの章立て」が違う罠

何が起きたか

原因: 「10-K」と「20-F」という様式の違い

修正: フォームタイプをちゃんと見て、失敗を区別する

学び: エラーを握りつぶすと、静かに死ぬ

The Cause: `\n` became a literal newline

The Fix: Use raw strings `r"""..."""`

Lesson Learned: Use `repr()` when dealing with code as strings

The Problem: `.gitignore` Is Just a Text File

The Cause: `len()` was the Root of All Evil