DEV Community: oji - building AI in public

My custom filter showed "zero false positives"! Then I realized my validation logic was flawed, letting all the garbage through.

oji - building AI in public — Thu, 10 Sep 2026 23:30:18 +0000

Hey there, it's your "Oji" (38-year-old AI/quant dev on the side). During the week, I'm a regular company employee; on weekends, I tinker with AI agents and automated trading bots.

Recently, I was building a filter to automatically extract "AI-related companies" from a market data list, based on their business descriptions. The mechanism is simple: it scores companies based on hits of predefined positive keywords (e.g., "machine learning," "natural language processing") and negative keywords (e.g., "gaming," "entertainment").

To improve the filter's accuracy, I set up a test to tune the threshold (how many positive keyword hits qualify a company). I prepared lists of positive examples (actual AI companies) and negative examples (non-AI companies) to see how well the filter classified them—standard stuff.

Then I ran the test, and the results were mind-blowing.

"Zero false positives, no matter the threshold I tried."

For a moment, I thought, "Is my filter a genius?" Zero false positives is usually impossible. But seconds later, I cooled down. Results that are too perfect usually mean something is wrong.

Sure enough, when I manually reviewed the list of companies that passed the filter, I found obvious non-AI companies like gaming studios mixed in. My filter was letting garbage through, yet the test reported "no issues." The despair I felt realizing this late on a weekday night was quite something.

Why "Zero False Positives"?

The root cause wasn't a bug in the code itself, but a design flaw in the validation logic.

Basic accuracy validation involves looking at two metrics:

False Positive (FP): A non-AI company incorrectly classified as an AI company.
False Negative (FN): An AI company incorrectly classified as a non-AI company.

The issue I encountered was with FP. To count FPs, the definition of negative examples (companies that are "not AI companies") is crucial.

And here was my definition of a negative example:

"A company with 0 positive keyword hits AND 2 or more negative keyword hits."

This was the source of all evil. This definition, seemingly sound at first glance, contained a self-contradiction.

My filter's rule for classifying a company as "passing (AI-related)" was: "2 or more positive keyword hits."

You probably see it now.

Filter's passing condition: positive_hits >= 2
Test's negative example condition: positive_hits == 0

These two conditions can never be true simultaneously. A company classified as "passing" by the filter already has positive_hits >= 2, so it can never fit the negative example definition of positive_hits == 0.

In essence, when trying to count "negative examples that were incorrectly classified as passing (i.e., false positives)," the structure itself made it impossible for any company to be both a "negative example" and "passing." Of course, false positives would always be zero.

Translating the concept to code:

// Filter rule: Pass if positive keyword count >= 2
function is_theme_company(positive_hits) {
  return positive_hits >= 2;
}

// Validation's negative example definition (buggy)
function is_negative_example(positive_hits, negative_hits) {
  // Only consider companies with zero positive hits as negative examples
  return positive_hits === 0 && negative_hits >= 2;
}

// Validation process
let false_positives = 0;
for (const company of all_companies) {
  const is_selected = is_theme_company(company.pos_hits);

  // If it's a negative example but selected, count as FP
  if (is_selected && is_negative_example(company.pos_hits, company.neg_hits)) {
    false_positives++;
  }
}
// With this logic, is_selected=true (pos_hits>=2) and is_negative_example=true (pos_hits==0)
// can never both be true, so false_positives will always be zero.

The test was too tightly coupled to the logic it was supposed to validate, preemptively deciding the outcome. It was a brutal rookie mistake.

How I Fixed It

Once I understood the cause, the fix was simple.

I changed the definition of negative examples to be independent of keyword hit counts. Specifically, I manually selected dozens of companies that were unequivocally not AI companies (e.g., food manufacturers, apparel brands, construction companies) and created a fixed "negative example list."

Running the test again with this list, false positives, as expected, came pouring out. Finally, the numbers showed that the filter was indeed picking up many irrelevant companies. This was the real starting point for tuning.

My Takeaway

The lessons from this failure are significant.

"The test passed" only means something if the test itself is correct.

Finding code bugs with tests is fundamental, but you must always question the possibility that the test logic itself is flawed. Especially when validating rules you've created yourself, there's a risk that the validation logic gets dragged by the tested logic, unconsciously leading to "conclusion-first" tests.

When you get "too perfect results," first question your own assumptions. This is a crucial reminder I'll engrave into my mind.

This story also applies to backtesting automated trading bots. When a backtest shows unusually good performance, it's rarely because you've discovered a brilliant strategy; it's usually because you're looking at future data (leakage) or not accounting for fees and slippage.

I'm logging embarrassing failures like this as part of building in public. Hopefully, it helps someone else in their solo dev journey.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

If a provider-agnostic RAG Q&A API is useful to you, mine is MIT-licensed on GitHub: rag-faq-api. It runs and passes its full test suite **with no API key* (offline stub LLM + hashing embedder), swaps to Claude / Gemini / OpenAI via one env var, and ships a retrieval-quality harness (Hit@k / MRR / Recall@k) with a chunking sweep.*

米国企業の分析bot、外国企業だけ“黙って”分析をスキップしてた話 — 10-Kと20-Fで「リスクの章立て」が違う罠

oji - building AI in public — Tue, 08 Sep 2026 23:30:19 +0000

どうも、おじいです。平日夜と休日に、AI エージェントとか自動売買 bot をちまちま作ってる 38 歳。

今日は、副業で開発してる企業分析 bot でやらかした、結構えぐいバグの話。動いてるように見えて、実は大事なデータをごっそり見逃してた。データ処理系の個人開発やってる人には、あるあるかもしれない。

何が起きたか

作ってる bot の一つに、米国上場企業の年次報告書（SEC ファイリング）を読み込んで、事業リスクを LLM で要約・評価するやつがある。企業の健全性をざっくり把握するためのツールだ。

こいつが、ある特定の企業群について、やけに「クリーン」な分析結果を出してくることに気づいた。「リスク要因: 特になし」みたいな。最初は「お、超優良企業か？」なんて思ってたんだけど、何社か続くとさすがにおかしい。そんなわけない。

ログを掘ってみたら、衝撃の事実が判明した。bot が、特定の企業の分析だけ「黙って」スキップしてた。エラーも吐かずに、ただ空っぽのデータを返してた。そのせいで、後段の LLM は「リスクに関する記述がなかった」と判断して、「リスクなし」という結論を出してたわけだ。これはやばい。

原因: 「10-K」と「20-F」という様式の違い

原因は、米国証券取引委員会（SEC）に提出される年次報告書の「様式」の違いにあった。

bot は当初、米国企業が提出する「Form 10-K」という書類だけを想定して作ってた。この 10-K では、事業リスクは「Item 1A. Risk Factors」という項目に記載されるのがお作法。だから、bot は機械的に「Item 1A」のセクションを引っこ抜くように実装してた。

ところが、分析結果が空になってた企業を調べたら、全部「外国企業」だった。米国市場に上場してる外国企業は、10-K の代わりに「Form 20-F」という別の様式で報告書を提出する。

そして、この 20-F では、リスク要因は「Item 3.D. Risk Factors」に書かれている。

つまり、bot は 20-F の書類に対して、存在しない「Item 1A」を探しに行ってた。当然、見つかるわけがない。

一番の問題は、その後の処理だった。セクションが見つからなかった時のエラーハンドリングを、こんな風に書いてた。

# 抽出失敗と「該当なし」を混同する危険なコード
try:
    risk_text = doc.get_section("Item 1A")
except SectionNotFound:
    risk_text = "" # これでは失敗したのか、元々空なのか分からない

一見、例外をキャッチしてて安全そうに見える。でも、これが罠だった。「セクションが見つからなくて抽出に失敗した」ケースと、「セクションはあったけど中身が空だった」ケースが、どっちも risk_text = "" になってしまう。

このせいで、データ欠損という重大な異常が、正常な「リスク記載なし」として処理され、静かにバグが進行してたわけだ。

修正: フォームタイプをちゃんと見て、失敗を区別する

対策はシンプル。
まず、処理対象のドキュメントが 10-K なのか 20-F なのかをちゃんと判別する。その上で、フォームタイプに応じて読み込むべきセクション ID を切り替えるようにした。

# フォームタイプに応じて処理を分岐する改善コード
if doc.form_type == "10-K":
    section_id = "Item 1A"
elif doc.form_type == "20-F":
    section_id = "Item 3.D"
else:
    # 未対応のフォームタイプなら、処理を中断
    log.warning(f"Unsupported form type: {doc.form_type}")
    return None

# 抽出を試みる。失敗したらNoneが返るようにget_section側を修正
risk_text = doc.try_get_section(section_id)

if risk_text is None:
    # セクション自体が見つからなかった場合（＝異常）
    log.error(f"Section not found: {section_id}")
    # ... エラー処理 ...
elif risk_text == "":
    # セクションはあったが空だった場合（＝正常）
    log.info(f"Risk factors section is empty: {section_id}")

get_section メソッドも修正して、セクションが見つからない場合は空文字じゃなく None を返すように変更した。

これで、呼び出し側は戻り値を見るだけで、

None → セクションが見つからなかった（＝想定外のエラー）
"" （空文字）→ セクションはあったが、中身が空だった（＝データ上、リスク記載なし）
(str) → 正常に抽出できた

と、明確に区別できるようになった。この修正後、これまでスキップされてた外国企業の分析も、無事に動き出した。

学び: エラーを握りつぶすと、静かに死ぬ

今回の失敗から学んだことは２つ。

ドメイン知識はマジで大事。 今回で言えば、SEC ファイリングに 10-K と 20-F の違いがある、という知識。技術的な実装力だけじゃなくて、扱ってるデータそのものへの理解がないと、こういう根本的な見落としをする。特に金融系は、こういう「お作法」の塊みたいな世界。
「失敗」と「空」を区別する設計。 エラーを安易に握りつぶして、空文字や 0 みたいな「無害そうな」デフォルト値を返すのは、本当に危険。サイレントなデータ欠損は、気づいた時には手遅れ、なんてこともあり得る。自動売買 bot なら、資産を溶かす原因に直結する。None を使ったり、専用の例外を投げたりして、異常は異常としてちゃんと後続の処理に伝える設計がいかに重要か、身をもって知った。

副業の個人開発だと、ついドキュメントを斜め読みして「動いたからヨシ！」で進めがちだけど、こういう地味な仕様の確認こそ、一番時間をかけるべき部分なのかもしれない。

この失敗ログが、同じようにデータ処理系の bot を作ってる誰かの参考になれば嬉しい。

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My Bot's KPI: Trades 2 > Opportunities 1?! When Your Numerator and Denominator Don't Observe the Same Time.

oji - building AI in public — Fri, 04 Sep 2026 23:30:20 +0000

Hey everyone, it's your friendly neighborhood "old man" developer here. I'm a 38-year-old part-time engineer, spending my evenings and weekends tinkering with AI bots.

Today, I want to share a particular moment from my bot development log that had me face-palming – one of those "how could I make such a dumb mistake?" moments. In short, a KPI I built to measure my bot's performance started spitting out logically impossible values. Specifically, my "fulfillment rate" exceeded 100%.

Imagine a scenario like, "there was only one trading opportunity, but somehow two trades were executed." It made no sense. And no, my bot hadn't gone rogue and started placing infinite orders. The cause was a simple, elementary error in my KPI calculation logic.

What Happened: My Tests Turned Red with an `AssertionError`

I've set up a decent suite of unit and integration tests for my bots. Last weekend, after adding some new logic, I ran the tests as usual, and they went red with an unfamiliar error:

AssertionError: fulfillment_rate > 1.0

fulfillment_rate is a custom KPI I use to see how many of the trading opportunities my bot identifies actually result in a successful trade. The formula is simple: actual_trades / total_opportunities.

For this to exceed 1.0 (or 100%) is impossible. Digging into the logs, I found bizarre entries like trades: 2 against total_opportunities: 1. For a moment, I thought there was a bug in my aggregation logic, double-counting trades. But a direct check of the DB confirmed two actual trades. And at that moment, only one opportunity was visible. What was going on...?

The Cause: The End of the Month Hadn't Arrived Yet

Stepping through the code with a debugger, the cause became immediately clear. The "observation time" for calculating the numerator and denominator was out of sync.

The problematic code, conceptually, looked something like this:

# Problematic logic (conceptual)

# Denominator: Fetch all trading opportunities as of the end of the month
opportunities = get_opportunities_as_of(end_of_month_date)

# Numerator: Fetch all trades that occurred within a specific period
trades = get_trades_in_period(start_date, end_date)

# Calculate fulfillment rate
fulfillment_rate = len(trades) / len(opportunities)

At first glance, this code might seem fine. But running it before the end of the month arrives leads to a nasty problem.

Let's say I ran the test on August 26th.

Numerator Calculation: get_trades_in_period would retrieve all trades from August 1st to August 26th. If a new trade was entered and executed on August 26th, the numerator len(trades) would be incremented.
Denominator Calculation: get_opportunities_as_of would try to fetch trading opportunities that should be visible as of end_of_month_date, i.e., August 31st. But it's only August 26th. Since future data isn't available yet, the trading opportunities that occurred on August 26th hadn't been included in the denominator calculation.

The result was a discrepancy: the numerator increased in real-time, while the denominator wouldn't update until the end of the month. If trades were concentrated near month-end, the fulfillment rate would easily exceed 100%. It's obvious in hindsight, but I didn't catch it until the test failed.

The Fix: Unifying the Observation Point to the "Entry Date"

Once I pinpointed the cause as a "mismatch in observation time," the fix was straightforward: align the numerator and denominator to be calculated from data observed at the same point in time.

Specifically, I changed the logic to re-calculate trading opportunities based on the actual "entry date" when the bot made a trade.

# Corrected logic (conceptual)

# Base calculation on the day the trade occurred
trade_entry_date = get_trade_entry_date(...)

# Denominator: Get opportunities as of "that specific day" the trade occurred
opportunities = get_opportunities_as_of(trade_entry_date)

# Numerator: Similarly, get trades that occurred "on that specific day"
trades = get_trades_on(trade_entry_date)

# Now, the observation points are aligned
fulfillment_rate = len(trades) / len(opportunities)

Now, without waiting for month-end, I can correctly calculate, for each trade, "what percentage of opportunities existing at that specific moment were executed." By leveraging the bot's own business day determination logic, I was able to align the observation points.

The Lesson: "Numerator and Denominator on the Same Playing Field" - A Fundamental Principle

This mistake of "misaligned aggregation criteria for numerator and denominator" isn't actually my first time. Since I started personal development, this is probably the fourth time.

User registration counts and active user counts had different aggregation periods, causing active rates to exceed 100%.
API total request counts and error counts had different time zones, leading to skewed error rates.
In backtesting profit/loss calculations, the timing of fee accrual was off, overstating profits.

All of them were the same type of mistake.

"When calculating KPIs, always obtain the numerator and denominator from the same observation point, the same definition, and the same playing field."

This is a fundamental principle, yet it's easy to overlook when you're deep in the code. It's especially easy to fall into this trap when pulling numbers from different tables or data sources.

This latest failure was detected by an AssertionError in my tests, which was great. If I hadn't had tests and remained oblivious, I might have mistakenly thought, "Wow, this bot is performing incredibly well!" and made bad decisions based on that false impression.

This past weekend was another stark reminder that humble testing and strict adherence to basic principles are paramount. I hope sharing this failure log can be helpful to someone out there.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

One Windows Task Scheduler Setting Made the Difference Between Bot Apocalypse and Resiliency

oji - building AI in public — Thu, 03 Sep 2026 23:30:14 +0000

Hey, it's oji_ai_dev here. I'm a 38-year-old developer building AI agents and automated trading bots on the side.

Today, I want to talk about a super subtle, yet critically important, infrastructure setting. My home server PC crashed overnight, and when I woke up, some of my bots were completely dead. I initially thought it was just a typical PC hiccup, but after digging in, I found it was a silent failure caused by my own human error. These kinds of silent failures are truly brutal.

What Happened: My Dashboard Was Too Quiet This Morning

It all started when I woke up and checked my custom monitoring dashboard. Several bot logs, which should have been running, had stopped dead around 3 AM.

"Ah, the PC crashed again."

This happens occasionally. Sure enough, the Event Viewer showed an unexpected shutdown record. Probably a power flicker or something. But here's the kicker: I have the PC set to auto-restart, yet out of my 11 running bots, 4 simply hadn't started back up.

No error logs. They just weren't executing. This is the worst kind of "silent failure." It leads to data loss and missed opportunities. What was the difference between the running bots and the dead ones?

The Cause: One Tiny Checkbox

I started comparing the Task Scheduler settings for the 7 bots that restarted successfully and the 4 that remained silent.

I found it almost immediately. The culprit was this setting in the "Conditions" tab:

"Start the task only if the computer is on AC power"

The working bots had this box unchecked, meaning they would start even if the PC was on battery. The 4 dead bots, however, all had this box checked.

When the PC shut down overnight and restarted, for some fleeting moment, the OS must have detected it as "on battery power." Any tasks scheduled to start at that precise moment were skipped because they weren't on AC power. That was the truth.

Why did I miss this setting? Tracing back my memory, a few months ago, I had wanted to ensure all bots would reliably restart after a power outage. I went through and updated the settings for all my active bots. I changed 7 of them correctly, but completely forgot the remaining 4. A classic manual rollout mistake. It really drove home how dangerous assumptions can be.

Fix and Prevention: Auditing Settings with Code

Once I knew the cause, the fix was simple: open the settings for the 4 problematic tasks and uncheck the box.

But that's not a fundamental solution. I could make the same mistake again. So, I decided to build a system to verify and audit settings using commands.

First, individual task settings can be exported as XML using the schtasks command:

schtasks /Query /TN "MyBotTask" /XML > task.xml

Then, I could open this XML file and check for <DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>.

But doing this for all 11 bots is tedious, and I'd surely forget again. This is where PowerShell comes in handy. I wrote a simple script to iterate through all tasks in a specific folder and list any that don't have the desired setting.

# Detect tasks under the '\MyBots\' path that are configured to NOT start when on battery power (true)
Get-ScheduledTask | Where-Object { $_.Settings.DisallowStartIfOnBatteries -eq $true -and $_.TaskPath -like '\MyBots\*' }

Running this periodically, or whenever I deploy a new bot, will prevent this specific configuration oversight. Ideally, I'd use something like Ansible for configuration management, but for personal side projects, even a simple auditing script like this is incredibly effective.

Lesson Learned: Manual Configuration Rollouts Always Lead to Accidents

The lesson from this incident is simple:

Manual rollouts of configurations will inevitably lead to accidents.

This is a given for large-scale systems in my day job, but with personal projects, I get complacent, thinking "only I touch this." However, as the number of bots grows, my memory becomes unreliable. I learned firsthand how dangerous it is to assume "I've done everything."

System robustness isn't just about fancy algorithms or the latest AI models. It's supported by every single, mundane infrastructure setting. A single checkbox can bring your system's availability to zero.

Time is limited for side projects. That's why I should have built a system to manage and audit these "set-and-forget" parts of the infrastructure with code from the beginning. Spending half a day troubleshooting is far less productive than spending that time building new features.

Personal development offers freedom, but it also means you're solely responsible for your infrastructure. This failure was a good opportunity to re-emphasize that responsibility.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My AI Agent Was Serving Stale Data for 30 Days: A Silent Failure Rooted in State File Freshness

oji - building AI in public — Wed, 02 Sep 2026 23:30:17 +0000

Hey everyone, it's Grandpa Dev. I'm a 38-year-old side-hustle engineer building AI agents and automated trading bots in my evenings and weekends.

Today, I want to share a pretty nasty silent failure I encountered with one of my AI agents. It took me a full month to notice, and honestly, it was a chilling discovery. I'm documenting this as a failure log in hopes it helps other solo developers running their own automation systems.

What Happened: My Agent Served the Same Content for 30 Days Straight

One of the agents I run is responsible for summarizing market information and generating a daily briefing every morning. Last week, I finally realized it had been serving the exact same content, day after day, for an entire month.

Specifically, a logic fix I made in late August was never actually reflected. The agent diligently delivered briefings generated with old information and outdated logic from August 28th, for almost the entire month of September. Ouch.

Digging deeper, I found that a weekly review process was also running with the same old logic. This genuinely sent shivers down my spine. It looked like it was working, but it was effectively dead—a true zombie process.

The Investigation: A Compound Failure from Two Factors

After a deep dive into the logs, I finally pinpointed the causes. There were two main issues at play.

1. Configuration "Drift"

First, at a fundamental level, there was a "drift" where a configuration file change didn't propagate to all the code that referenced it.

In late August, I modified the path and content of a configuration file to tweak the briefing generation logic. Of course, I updated the main generation code that referenced this setting. But I missed something: I forgot to update another operational script (operator.py) that indirectly referenced the same configuration file.

As a result, the new code used the new settings, while the old code continued to look for the old settings. This created a twisted state. The initial misstep was that the design change hadn't fully permeated the entire codebase.

2. Silent Failure Ignoring State File "Freshness"

However, if it were just configuration drift, I probably would have noticed sooner. The real problem was that the operational code completely ignored the "freshness" of the state file.

My agent involves several interconnected processes:

generator.py creates the raw material for the briefing from the latest information (a state file like staged_brief.json).
↓
operator.py reads that state file and performs the final delivery.

The problematic operator.py, if staged_brief.json existed, would read and process it without any regard for when it was created.

So, due to the configuration drift, there were days when generator.py failed and couldn't generate a new staged_brief.json. But operator.py didn't care. It found the old staged_brief.json from the previous day and thought, "Oh, a file exists!" and just reused it, continuing to deliver stale content.

The logs clearly showed this evidence:

// operator.py ignores staged_brief freshness -> FABLE for 8/31 and 9/1 are both "(2026-08-28)" and run_count is identical = 8/28 version was delivered unnoticed for 3 business days.

Even when file updates failed, no errors were thrown. Old data was silently used. This was the true nature of the silent failure. Scary stuff.

The Fix: Adding "Freshness Guards" to All State-Handling Code

The fix was simple yet thorough. I added guard mechanisms to every part of the code that reads "time-sensitive data" like state files or caches, checking their modification timestamps.

Conceptually, the code looks something like this:

from datetime import datetime, timedelta
import os

class StaleDataError(Exception):
    """Custom exception for stale data"""
    pass

def get_file_mtime(file_path: str) -> datetime:
    """Get file modification time as datetime"""
    return datetime.fromtimestamp(os.path.getmtime(file_path))

# --- Revised process ---
state_file = "path/to/staged_brief.json"

# Freshness guard: If the file is older than 1 day, raise an exception to stop processing
if (datetime.now() - get_file_mtime(state_file)).days > 1:
    raise StaleDataError(f"State file {state_file} is older than 1 day. Aborting.")

# Subsequent processing will now execute with the guarantee that the data is fresh
print("State file is fresh. Proceeding with the operation.")
# ...

Thanks to this guard, if the state file ever stops updating for any reason, the consuming process will now immediately throw a StaleDataError and terminate, preventing a silent failure.

Of course, I also addressed the original configuration drift problem by refactoring to unify the configuration file reference paths.

Lessons Learned: Fail-Safes Are the Lifeline for Personal Automation

I learned a lot from this failure:

Configuration changes are as dangerous as code changes. Don't just update documentation or config files and call it a day. Be aware that it affects every corner of the code that references it. You need to be as meticulous as doing a full grep sweep to check for impacts.
When dealing with stateful files, always check for "freshness." Just checking for file existence isn't enough. Skipping that extra step of verifying the modification time creates a breeding ground for fatal zombie processes like mine.
Robustness is paramount, especially for side projects. Unlike a full-time job where you might have 24/7 monitoring, personal projects need fail-safe designs that immediately stop and notify you of errors instead of silently chugging along incorrectly. This is literally a lifeline.

While developing cool new features is fun, building a robust, albeit unsung, system is ultimately the key to sustained personal development. This incident truly drove that home for me.

I hope this failure log helps someone out there prevent a similar issue in their own projects.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

副業応募botが受注ゼロ！犯人は『毎週クラッシュして成果を捨てる』1行のバグだった話

oji - building AI in public — Sat, 29 Aug 2026 23:30:17 +0000

どうも、おじいです。

本業の傍ら、AI エージェントとか自動売買 bot を個人開発してる。最近、自分の副業探し自体を自動化しようと思って「副業応募 bot」を作って動かしてた。

こいつの仕事はシンプル。

週末に『ある求人サイト』とか複数のマーケットを巡回して、良さげな案件をリストアップする
リストアップした案件に対して、俺のスキルセットに合わせた提案文を AI で自動生成して応募する

これを1ヶ月ほど動かしてみたんだけど、結果は「受注ゼロ」。まあ、そんなに甘くないよな、と。最初は「提案文の質が悪いのかな」とか「そもそもスキルが足りてないのかも」とか、結構まともな反省をしてた。

でも、念のためログをちゃんと見てみたら、えぐい事実が発覚した。

そもそも、提案を1件も送ってなかった。

いや、マジかよと。630件も提案文を「下書き」は作ってたのに、応募ボタンを押すフェーズまで到達したのがゼロ件。これは提案文の質とかそういうレベルの話じゃない。完全にシステムの問題。マジでやばいと思った。

原因：エラー1つで全データを吹き飛ばす脆すぎる設計

なんでこんなことになったのか。原因を深掘りしたら、問題は応募スクリプトじゃなくて、その前段の「案件収集スクリプト」にあった。

こいつは毎週日曜の深夜に動くように設定してたんだけど、どうやら処理の途中でクラッシュしてたらしい。そして、このクラッシュの仕方が最悪だった。

当時のコードは、だいたいこんな感じ。

// Before: A single error crashes the whole script
def scrape_all():
    browser = pw.chromium.launch()
    results = []
    # 複数のサイトをループで巡回
    for site in SITES:
        # このへんでサイトの仕様変更とかでエラーが出ると…
        page = browser.new_page() 
        try:
            # 収集した結果をいったんメモリ上のリストに追加
            results.extend(scrape_site(page))
        except Exception as e:
            log.error(f"Failed on {site}: {e}")
            # エラーが出てもループを止めないように try-except は入れてたつもりだった

    # ループが全部正常に終わらないと、ここにはたどり着かない
    # つまり、途中でクラッシュすると、それまでの成果は全部消える
    save_results(results)

このコードの問題点は2つ。

エラー処理の範囲がデカすぎる: ループの外側でブラウザを起動してるから、browser.new_page()みたいな処理で何かあると、try-exceptブロックの外で例外が発生して、スクリプト全体が即死する。
最後にしか保存しない: 収集した案件データは、全部のサイトを回りきるまでメモリ上の results リストに溜め込んでる。だから、最後の1サイトでコケたとしても、それまでに集めた全データが失われる。

実際、ログを追ったら『某マーケット』の HTML 構造が微妙に変わってて、要素が見つからずにタイムアウトエラーを起こしてた。たったこれだけのことで、毎週日曜の深夜、bot は数時間かけて集めたデータを全部ドブに捨てて、黙って死んでたわけだ。そりゃ応募できるわけがない。

修正：死ぬことを前提にしたエラー処理と中間保存

この手の長時間動くスクリプトは、「絶対にどこかでコケる」という前提で設計しないとダメだった。反省して、こう書き直した。

// After: Robust error handling and intermediate saves
def scrape_all():
    all_results = []
    # サイトごとに、ブラウザの起動から全部やる
    for site in SITES:
        try:
            # エラーが起きる可能性のある処理を全部 try ブロックに入れる
            browser = pw.chromium.launch()
            page = browser.new_page()
            site_results = scrape_site(page)
            all_results.extend(site_results)

            # 成功したサイトの結果は、その都度ファイルに書き出す
            save_intermediate_results(site, site_results) 
        except Exception as e:
            # ここでエラーをキャッチすれば、次のサイトの処理には進める
            log.error(f"Failed on {site}, skipping: {e}")
        finally:
            # ちゃんとブラウザを閉じる処理も忘れずに
            if 'browser' in locals() and browser:
                browser.close()

    # 最後に全部の結果をマージして保存
    save_final_results(all_results)

ポイントは2つ。

try-exceptをループの内側に入れる: これで、あるサイトの処理で失敗しても、そのエラーはループ内で完結する。次のサイトの処理には影響が出ない。
中間保存: 1サイトのデータが取れたら、その時点ですぐにファイルに保存する。これで、たとえスクリプトが最後の最後でクラッシュしたとしても、それまでに成功した分のデータは手元に残る。

この修正で、bot はようやく毎週安定して案件データを集めてこれるようになった。

学び：本当の課題はコードの先にあった

この一件で学んだことは多い。

まず技術的な話。長時間動くバッチ処理は、とにかく堅牢性が命。エラー処理は「念のため」じゃなくて「必須の設計要素」。そして、状態（この場合は収集データ）はメモリに置かず、こまめに永続化する。基本だけど、サッと作ると忘れがちだ。

次に、問題解決のアプローチ。最初、「受注ゼロ」という結果だけ見て「提案文が悪い」とか「スキルが足りない」とか、表層的な仮説に飛びついちゃったのが良くなかった。「そもそも応募はされているのか？」という、一番基本的なファクトの確認を怠ってた。ログは嘘をつかない。何か問題が起きたら、まずデータを深く掘るのが鉄則。

そして一番大きかったのが、このバグを直したことで見えてきた、さらに本質的な課題だ。

安定して案件を収集できるようになった結果、わかったことがある。

「俺がやりたいレベルの副業案件、市場にほとんど出てねえわ」

結局、技術的な問題を解決したら、その先にある事業戦略（この場合は副業戦略）の課題が浮き彫りになっただけだった。まあ、これも大きな一歩か。

コードのバグを直すのは楽しいけど、そのコードが解決しようとしてる問題そのものがズレてないか？っていう視点も、個人開発では大事なんだなと、改めて痛感した。

さて、次は何を自動化しようかな。

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My monitoring tool lied to me and secretly deleted all past data integrity issues

oji - building AI in public — Fri, 28 Aug 2026 23:30:18 +0000

Hey there, it's your resident grandpa coder here. I'm 38 and I spend my evenings and weekends tinkering with AI agents and automated trading bots.

Today, I want to talk about how my homemade monitoring tool betrayed me. And I mean, a pretty catastrophic betrayal. My watchdog, designed to detect data integrity issues in my bots, quietly switched sides, reported "everything's fine!", and silently wiped out all records of past, unresolved issues.

These kinds of "silent failures" are genuinely terrifying. By the time you notice, it's often too late. This is a record of how overconfidence in my own code led to my monitoring tool actively concealing the very problems it was supposed to monitor—a truly gnarly self-contradiction.

The Discovery: Slack Went Quiet

Each of my bots collects and stores specific data daily. Gaps in this data throw off all downstream analysis and trading logic. So, I had a run_watch script constantly running, designed to instantly notify me via Slack if any data gaps were detected.

This script was quite effective. When I first implemented it, I'd frequently get notifications like "Data missing for 2023-10-26!". This helped me quickly catch API changes or my own bugs.

But then, at some point, the notifications just… stopped.

"Oh, my bots have been stable lately. Looks like I've fixed all the data integrity issues," I thought, blissfully unaware. I even started to wonder if my coding skills had improved.

Then one day, I happened to peek directly into the database. And there they were: data gaps, plain as day. Some were from a week ago, others much older, all unresolved. And Slack? Not a peep.

This was bad. It was like my watchdog wasn't just neglecting its duties, but actively colluding with the burglars.

The Cause: Self-Incriminating Logic in the Monitoring Tool

I immediately dug into the run_watch code. Its logic was roughly like this:

Scan each bot's data store to get a list of missing dates (missing_dates).
Load a list of previously notified but unresolved missing data issues, saved in unresolved_issues.json.
Compare the current scan results (missing_dates) with the unresolved_issues.json list.
If new missing data is found that's not in unresolved_issues.json, notify Slack and add it to the list.
Conversely, if an issue is in unresolved_issues.json but not found in the current scan, assume it's "resolved" and remove it from the list.

The problem lay in logic point #5 and a specific function supporting it.

Right at the heart of the problem was this _oldest_expected() function:

// Problematic code: Supposed to return the 'oldest expected date', but returned the 'oldest currently missing date'
def _oldest_expected(report, bot):
    ms = e.get("missing") or []
    return min(ms) if ms else "0000-00-00"

Based on its name, this function should return the oldest date for which a bot is expected to have data. For instance, if a bot started collecting data from 2022-01-01, it should return that date.

But look at the implementation. All it does is e.get("missing"), which means it's returning the oldest date from the list of missing data found in the current scan. If no missing data is found at all, it returns "0000-00-00".

This discrepancy between the name and the implementation created a critical bug.

If, for some reason, the bot's data source API was temporarily unstable, or the network was flaky, the scan process might return "zero missing items."

At that moment, run_watch would interpret it like this:

"Oh, no missing data found in this scan. That means all those past, unresolved issues stacked up in unresolved_issues.json must be resolved! Alright, I'll clear them all out!"

And just like that, records of past data gaps, still very much present in the DB, were completely wiped from the monitoring list. The monitoring tool was actively deleting its own evidence. The tool meant to prevent silent failures was creating the most silent and insidious kind of failure. Irony, indeed.

The Fix: When in Doubt, Do Nothing (Fail Safe)

The fix was simple.

I changed the responsibility of the scan process itself to return information about "how far back it actually scanned." The monitoring tool then uses this "scan range" to determine if an issue is resolved.

Specifically, each bot's scan function scan() now returns expected_first (the earliest date it actually scanned) along with the list of missing data. If the scan range can't be confirmed due to API issues or similar, it returns a future date like "9999-99-99".

The monitoring tool now checks this expected_first. Only if an unresolved issue date in unresolved_issues.json is newer than expected_first does it consider it "resolved."

If expected_first is a future date (meaning the scan range is uncertain), it resolves nothing. It defaults to the safe side. This simple change prevents temporary scan failures from wiping out all historical logs.

Lessons Learned: Don't Over-Trust Your Own Tools

I learned three things from this incident:

Function names must perfectly match their implementation. _oldest_missing_found_in_this_scan would have been better than _oldest_expected, even if it's a ridiculously long name. When a name lies, you end up deceiving yourself.
A monitoring tool is itself a single point of failure. If it goes silent, everything goes dark. Perhaps I need a meta-monitoring system that considers "no notifications" an anomaly. Something like a DIY Dead Man's Snitch.
Proving a negative is hard. A report of "no missing data" needs to distinguish between "there genuinely is no missing data" and "we failed to detect any missing data." A fail-safe design that "does nothing" when uncertain is especially critical for monitoring systems like this.

Ultimately, the person you should trust least with the code you write is yourself. This incident was a stark reminder that when things feel "stable lately," that's precisely when you should be most suspicious that some silent time bomb is ticking away.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My Trading Bot Silently Ignored Signals for 41 Days: The "Opportunity vs. Execution" Logging That Saved Me

oji - building AI in public — Wed, 26 Aug 2026 23:30:17 +0000

Hey everyone, it's your resident grandpa-developer here.

I develop and run AI-powered automated trading bots in my spare time, evenings, and weekends. Recently, one of my production bots went a full 41 days without making a single trade.

No error logs. My watchdog process monitor was chirping happily. It just looked like it was perpetually "waiting." If the market was quiet and no entry conditions were met, that would be normal. But what if something was broken internally, causing it to ignore signals it should have acted on?

This "running but doing nothing" state is the absolute scariest. It's impossible to distinguish normal waiting from a silent failure. It genuinely gave me heart palpitations.

Digging into Logs Revealed "Ignored Signals"

41 days of silence felt definitively off, so I decided to pull all the detailed logs and investigate. Usually, I only glance at errors and summaries, but this time, I grep'd through every single debug-level raw log.

And then, a stark truth emerged.

Internally, the bot was generating "signals" — the triggers for trades. Meaning, its logic had identified moments when it should've thought, "Now's the time, enter!" But in the subsequent logs, there was no record of an order being placed.

In essence, after generating a signal, the bot was silently suppressing it.

The reason this went undetected for 41 days was a flaw in my log design. I wasn't recording the fact that "a signal occurred" in an easily monitorable way. I was logging "actions" like orders and errors, but not the "opportunities" where an action should have been taken. This made it impossible to tell if the continuous inaction was normal or abnormal.

The Fix: Log "Opportunities" and "Executions"

To prevent these kinds of silent failures, the only way is to log both what the bot intended to do and what it actually did, then monitor the discrepancy. I've dubbed this an "Implementation Fidelity Check."

Specifically, I modified the log output to count the progression from signal generation to execution across different stages.

# Example of newly instrumented log items

# For bot A:
# seen:      number of times the monitored event was observed
# qualified: number of times it met signal conditions
# taken:     number of times a position was actually taken
# log -> insider_paper: seen=120 qualified=3 taken=2

# For bot B:
# fired:    total number of signals generated
# opened:   number of orders actually placed
# rejected: number of orders rejected for any reason
# log -> ny_stack: [SIGNALS] fired=5 opened=5 rejected=[]

With this in place, I can immediately spot anomalies like "qualified is increasing but taken remains zero" or "the count for fired and opened doesn't match."

I rolled out this system to all my bots and set up daily aggregations. And lo and behold, a host of problems became visible.

Bot	Opportunities	Executions	Fulfillment Rate	Notes
trend_exec	1	0	0%	🔴 1 opportunity unfulfilled
gotobi_paper	6	4	67%	⚠️ Missed end-of-month signals

For trend_exec, an API spec change had added a new mandatory parameter to requests, which I hadn't followed up on. Order requests were silently failing every time. 0% fulfillment rate. This was terrible.

For gotobi_paper, I had a gap in my calendar definition for specific trading days (like 'Gotobi' days at month-end), leading to signals being generated but not executed. 67% fulfillment rate. Another missed opportunity.

Conclusion: The Scariest Thing in Bot Dev is "Quiet Failure"

What I learned from this incident is that "no errors ≠ normal."

Failures where a process dies or throws an exception and stops are detectable, so they're manageable. The truly terrifying ones are silent failures that don't throw errors, simply bleeding opportunity in the background.

To prevent this, a system that records the bot's "intent (opportunities)" separately from its "actions (executions)" and monitors the fulfillment rate is indispensable. Without it, you can't tell the difference between normal waiting and a critical bug.

In personal dev environments, it's easy to push off setting up monitoring like this. But if you're running bots in production, I've painfully realized it's one of the first features you absolutely must implement. 41 days of lost opportunities was, frankly, quite painful.

From now on, when building a new bot, I'll prioritize integrating this "Implementation Fidelity Check" system. If you're building similar bots, I hope this experience proves useful.

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

My AI Agent Recommended a Non-Existent Investment Product — Exposing Information Gaps Between Fund Distributors and Asset Manage

oji - building AI in public — Tue, 25 Aug 2026 23:30:20 +0000

Hey everyone, it's your average 38-year-old developer here, hacking away at AI agents and automated trading bots in my evenings and weekends, just like many of you.

Today, I want to share a recent blunder with a financial information AI agent I built. What started as a chuckle-worthy mistake turned into a serious lesson about "information reliability" — a critical factor that can be fatal for any automated system design.

The Problem: AI Recommends a "Non-Existent Investment Trust"

With the new NISA (Japan's tax-advantaged investment program) kicking off, I was building an AI agent to list investment trusts tracking specific indices, primarily to rebalance my own portfolio. I started by asking it to compare products linked to the trending "FANG+" index.

Within seconds of my prompt, the AI confidently replied:

"For FANG+ linked investment trusts, considering trust fees and performance, I recommend the following two:

eMAXIS Slim FANG+: This series aims for the lowest operational costs in the industry and is highly recommended.
iFreeNEXT FANG+: It has a strong track record and offers stable management."

Hmm, interesting. I knew about iFreeNEXT FANG+, but "eMAXIS Slim FANG+"?

"Wait, is that even a thing?"

I know the eMAXIS Slim series is popular for things like S&P500 and global index funds, but FANG+? I'd never heard of it. Skeptical, I quickly Googled it.

...And indeed, it didn't exist.

The AI had confidently fabricated a plausible-sounding product and recommended it. This was a red flag.

Deeper Dive: The Trap of Information Hierarchy

AI "hallucinations" are common. It likely combined the popular "eMAXIS Slim" brand with the popular "FANG+" index to generate a seemingly plausible answer.

It would be easy to just dismiss AI as "unreliable" and move on. But as an engineer, I wanted to dig deeper. What if the agent had only returned real product names? Would I have blindly trusted it and moved on to the next step?

Curious, I decided to cross-reference multiple sources for the legitimate "iFreeNEXT FANG+". Specifically, I checked the website of "Company A" (a distributor selling the product) and the official website of "Daiwa Asset Management" (the asset manager that operates the fund).

What I found was even more concerning:

The NISA growth investment category eligibility status conflicted between the two sources.

Asset Manager's (Daiwa Asset) Official Site: Clearly stated as eligible.
Distributor's (Company A) Site: Seemed to indicate it was ineligible, or the information was outdated and not updated.

Which one is correct? Unsurprisingly, it's the "asset manager" who creates the product. The distributor is essentially a "retailer" that procures and sells the product. They might have delayed updates or simple transcription errors.

This incident made me realize that information has a clear "hierarchy":

Primary Information: The source of the information, like the asset manager's official website.
Secondary Information: Information that processes or reposts primary information, such as distributor websites, news articles, or blogs.
AI Generation: Information learned, re-synthesized, and generated by AI from these sources.

My agent had completely ignored this hierarchy. It treated all information gathered from the internet as flat data, simply summarized by the AI. This inherent risk meant it could recommend non-existent products or be misled by outdated secondary data.

The Fix: Embedding Fact-Checking into Code

This incident fundamentally changed my approach to designing information gathering agents. It's not enough to just have AI "research"; you need to build in mechanisms to "verify" for it to be practical.

The solution was simple:

Designate Reliable Sources as "Truth": In this case, data obtained from the "asset manager's official website" is defined as the master data (primary information).
Automate Cross-Checking: Any data obtained from AI or other secondary sources must be cross-referenced against the master data for fact-checking.

Specifically, I implemented logic like this using Python (pandas):

import pandas as pd

# Primary data obtained from the asset manager's official website (master data)
primary_data = {
    'Fund Name': ['iFreeNEXT FANG+', 'iFreeNEXT NASDAQ100'],
    'Asset Manager': ['Daiwa Asset Management', 'Daiwa Asset Management'],
    'NISA Growth Eligible': [True, True],
    'Source': ['Asset Manager Official', 'Asset Manager Official']
}
df_primary = pd.DataFrame(primary_data)

# AI's recommendation list (including the fabricated product)
ai_recommendation = {
    'Fund Name': ['eMAXIS Slim FANG+', 'iFreeNEXT FANG+'],
    'Reason': ['Lowest trust fees', 'Strong track record']
}
df_ai = pd.DataFrame(ai_recommendation)


# --- Fact-checking logic ---
# 1. Check if AI's recommendation exists in primary data (master)
for fund_name in df_ai['Fund Name']:
    if fund_name not in df_primary['Fund Name'].values:
        print(f"[WARNING] AI recommended '{fund_name}' which does not exist in master data.")

# 2. Merge distributor information with primary information to detect discrepancies
# (Skipped here, but involves merging distributor data with df_primary and checking for differences)

This code doesn't blindly trust the AI's output. It first checks if the fund names recommended by the AI exist in our defined df_primary (primary information) list. Anything not found is flagged as a "warning".

Furthermore, by merging specifications from secondary sources (like distributor sites) with the primary information, we can automatically detect issues like the "NISA eligibility discrepancy."

The Lesson: Master Your Primary Sources

What I learned from this failure is that AI is a highly capable assistant, but not the ultimate decision-maker. Especially in domains requiring accuracy, such as finance or technical information, a system to verify AI output is the lifeline.

And the foundation of that verification is the ability to discern "which information is primary."

When building automated systems, it's easy to gravitate towards readily accessible secondary information or APIs. But we must constantly ask: where did this information come from? Is its "freshness" and "reliability" guaranteed?

Ultimately, the biggest leverage comes from automating the tedious process of verification. Even if the AI lies, the overall system can still produce correct outputs. This incident reaffirmed my commitment to building robust systems that can handle such challenges. ✍️

When I Asked AI to Analyze "FANG+" and It Started Investigating Oil Companies — The Ticker Symbol Trap

oji - building AI in public — Tue, 25 Aug 2026 03:26:48 +0000

Hey there, it's your friendly neighborhood old guy. By day, I'm doing my regular gig, but by night, I'm tinkering with AI agents and algo-trading bots – I'm 38, for context.

Last weekend, while trying to integrate a new strategy into my investment analysis bot, I stumbled into a rather amusing (and frankly, a bit dangerous) pitfall. Consider this a self-admonition, meticulously documented.

Asked for FANG+ Analysis, Got an Oil Price Report

The genesis of the problem was simple. I was curious about recent tech stock movements, so I instructed my custom AI agent: "Analyze the FANG+ index and report on its future outlook."

This agent is designed to gather relevant data based on a given theme, analyze it, and generate a summary. As usual, I kicked off the task, and it started processing immediately.

Minutes later, I looked at the generated report and scratched my head.

"...Something's off, isn't it?"

The report indeed stated "FANG analysis." But the content had absolutely no mention of tech companies. Instead, it was all about the energy sector: crude oil price trends, shale oil extraction costs, OPEC production volumes.

It felt like reading an earnings call transcript for an oil company.

"Wait, why? Did I mess up the prompt?"

No, I double-checked the logs; the instruction was clearly "Analyze FANG+." Did the AI bug out? I wondered, but tracing its thought process revealed that the AI itself had acted with extreme logic. The problem lay in the "environment" I had provided.

The Culprit: "FANG" Lurking in the S&P 500 List

I quickly pinpointed the cause.

When this AI agent searches for analysis targets, it refers to an investment universe (a list of target stocks) that I've prepped. In this case, I had fed it an S&P 500 constituent list I had on hand, verbatim.

Here’s an excerpt from that list:

...
VRSK XEL CTSH TTWO LULU FANG CEG TEAM AZN ZS DXCM
...

You probably see it now.

When the AI received the instruction "FANG+", it found the string "FANG" within this list. It then interpreted this as "This must be the FANG the user is referring to," and commenced its analysis.

This ticker symbol, FANG, has, of course, absolutely nothing to do with the tech index. It's the ticker for "Diamondback Energy, Inc.," an oil and natural gas company based in Texas.

No wonder I got a report on crude oil prices. The AI wasn't wrong. If anything, it made the most "logical" decision given the dataset. This misunderstanding, frankly, was wild. A human would infer "contextually, they mean the tech index," but AI purely deals with data. Its purity, in this case, backfired.

The Fix: Eliminate Ambiguity, Define Universe Strictly

To prevent these kinds of incidents, there's only one way: rigorously eliminate ambiguity from the instructions and data given to the AI.

1. Be Specific with Prompts
A vague instruction like "Analyze FANG+" was the problem. If I had specified the ticker symbol, such as "Analyze the NYSE FANG+ Index (^NYFANG)", there would have been no room for the AI to err.

2. Separate Data Universes
Fundamentally, treating individual stocks (Stock) and indices (Index) or ETFs within the same list was a mistake.

Data source for stocks
Data source for ETFs
Data source for indices These should have been clearly separated, and the AI agent's search scope should have been limited. For example, if the task is "index analysis," it should be constrained to only refer to the index list.

For now, I've revised the prompt template and added a UI where the user (me) explicitly specifies the type of analysis target (stock, ETF, index). This should significantly reduce the risk of ticker symbol collisions.

The Lesson: Financial Data is a Minefield of "Name Collisions"

The lesson from this failure is simple:

In the world of financial data, similar strings frequently mean entirely different things.

Just as "FANG" is a colloquial term for a tech index and simultaneously the ticker for an oil company, these kinds of "name collisions" are everywhere. For example, AMZN is Amazon, but AMZ might be a JPMorgan index-linked security.

You cannot expect AI to perform the "contextual reading" that humans do unconsciously. When developing AI agents, the "Garbage In, Garbage Out" principle is absolute. No matter how advanced the LLM, if the reference data sources or underlying assumptions are flawed, the output will be unreliable.

Squashing these mundane bugs one by one, that's the reality of solo dev work, I think. More than flashy new features, this foundational data hygiene is far more crucial for a bot's stable operation.

Now, which bug to squash next? I'll report back if I mess up again. 👍

Predicting Shareholder Meeting Failure: How AI Quantified Risk from a Single Sentence in Past Minutes and a 5x Increase in Share

oji - building AI in public — Mon, 24 Aug 2026 00:28:11 +0000

Hey, it's your friendly neighborhood dev-grandpa here, still chugging along building AI agents on weeknights and weekends.

Today, I want to share a story about how one of my custom analysis bots dug up a significant risk from a completely unexpected angle. I'll walk through how the AI quantitatively assessed the probability of a seemingly unpredictable event—whether a shareholder meeting would successfully pass a critical resolution—by analyzing past meeting minutes and other disclosure documents.

The Trigger: Will This Reverse Stock Split Actually Pass?

I had a particular U.S. company under monitoring by my bot. This company was planning a reverse stock split, a common move to boost share price, usually approved without issue at shareholder meetings.

But something felt off. This company had become incredibly popular with retail investors, and its shareholder base was growing rapidly. I started to wonder: if the shareholders were too dispersed, would it be difficult to reach the required quorum for the meeting to proceed?

To test this hypothesis, I instructed my AI agent: "Read all of this company's past SEC filings (like EDINET in Japan) and identify any risks related to shareholder meetings."

The AI's Discovery: A Single Sentence Revealing Past Failures

The agent accessed the EDGAR database and started downloading years' worth of proxy statements (DEF 14A/C). Hundreds of megabytes. A human would spend days just reading through it.

Using NLP, it began parsing the text, extracting relevant passages based on keywords suggesting meeting delays or failures, like "quorum," "adjourn," and "postpone."

A few minutes later, the agent pinged me on Slack:

"Found a history of the Special Meeting being adjourned twice due to a lack of a quorum several years ago."

No way. Seriously?

The discovery was a single sentence buried in the minutes from that time:

... the Special Meeting was adjourned on two occasions due to a lack of a quorum. At the reconvened meeting, approximately 45.9% of the outstanding shares were present...

In essence: "The meeting was postponed twice because they couldn't get a quorum. Even when reconvened, only 45.9% of shares were present." That's below the majority. This is a big deal; usually, this doesn't happen.

This qualitative information – the fact that they'd screwed up before – was already a significant red flag.

Combining Qualitative Insights with Quantitative Data

But the AI's job wasn't done. The mere fact that they "failed in the past" is too crude for proper analysis. The crucial next step was to compare: "How different are the conditions now compared to then?"

So, I gave the agent the next set of tasks:

Retrieve the number of outstanding shares at the time of the quorum failure.
Retrieve the current number of outstanding shares.
Compare the two to estimate the degree of shareholder dispersion.

The results came back quickly:

Outstanding shares then: approx. 180 million
Outstanding shares now: approx. 950 million

Astoundingly, since the failed shareholder meeting, the number of outstanding shares had ballooned by 5.1 times.

This is where the dots finally connected:

Qualitative (Past Information): This company inherently struggles with proxy solicitation for shareholder meetings.
Quantitative (Current Data): The number of shareholders has increased by 5.1 times since then, making further dispersion extremely likely.

Combining these two pieces, I could form a highly accurate hypothesis: "The risk of this upcoming shareholder meeting failing due to lack of a quorum is considerably high."

In code, the logic looks something like this:

import re

def extract_governance_risks(document_text: str) -> dict:
    """Extracts governance risks from disclosure document text"""
    risks = {}

    # History of adjournment due to lack of quorum
    if re.search(r'adjourned .* due to a lack of a quorum', document_text, re.IGNORECASE):
        risks["QUORUM_FAILURE_HISTORY"] = True

    # Extract attendance rate
    match = re.search(r'(\d+\.\d+)% of the outstanding shares were present', document_text)
    if match:
        attendance_rate = float(match.group(1))
        if attendance_rate < 50.0:
            risks["LOW_ATTENDANCE_RATE"] = attendance_rate

    return risks

# --- Evaluation Logic ---
# Past document text (sample)
past_document_text = """
... the Special Meeting was adjourned on two occasions due to a lack of a quorum.
At the reconvened meeting, approximately 45.9% of the outstanding shares were present...
"""

# Past and current outstanding shares (in millions)
shares_past = 186.8
shares_current = 949.7

risks_found = extract_governance_risks(past_document_text)
share_increase_factor = shares_current / shares_past

if risks_found.get("QUORUM_FAILURE_HISTORY") and share_increase_factor > 2:
    print(f"High Risk: History of quorum issues AND shareholder base dispersed by {share_increase_factor:.1f}x.")

The Final Piece: Reading the Company's Countermeasures

However, it's still too early to conclude "high risk!" The company must have learned from its past mistakes.

I had the agent re-read the latest proxy statement in detail. Sure enough, they had taken action.

Among the proposals for the current meeting was an agenda item to "amend the Bylaws to reduce the quorum for shareholder meetings from a majority to one-third."

Smart move. They were lowering the bar themselves.

This significantly offsets the risk of a quorum failure. Still, the facts of past failures and the rapid increase in shareholders remain. The final assessment landed somewhere around: "Risk still exists, but it's controlled to a non-fatal level."

Lessons Learned: Combining Text and Numbers Changes Everything

This whole experience offered significant lessons:

Combining qualitative information (text) with quantitative data (numbers) explodes the resolution of analysis. AI agents excel at processing both at high speed.
Even for seemingly unpredictable events, digging into past logs (meeting minutes, error logs, anything) allows for some degree of quantitative risk prediction for the future.
Ignoring countermeasures or changes made by the other party (company, system, market) leads to incomplete analysis.

This isn't just about automated trading. It applies to predicting failures in your own services, measuring the effectiveness of marketing campaigns, and many other scenarios.

Ultimately, AI isn't a magic wand. It's a tool for automating the grunt work of diligent data collection and then interpreting that data to form hypotheses. But because of it, you can sometimes uncover insights that a human would never find. That's why this side hustle is so engaging.

Alright, which bot should I tinker with next? I'll share if I stumble upon any more interesting failure logs. See ya.

My AI Lied to Me About a Stock Crash — Adversarial Testing Revealed Its Limits (and How to Use AI Correctly)

oji - building AI in public — Sat, 22 Aug 2026 23:30:17 +0000

Hey, it's OJ. I'm 38 and dabble in AI agents and algorithmic trading bots as a side gig.

Recently, I had a pretty "brutal" experience: an AI agent I built straight-up lied to me. As part of my build-in-public journey, I felt this was a crucial lesson to share, especially for anyone trying to automate information gathering with AI. So, here's the log.

The Trigger: A Stock I Held Dropped -33%

One day, a stock I was watching plummeted -33%. Normally, I'd immediately jump to official disclosure sites or the company's IR page, frantically searching for timely disclosures.

But this time, I saw it as a perfect opportunity to test a research AI agent I was developing. Its supposed job: give it a company name and a time period, and it would collect related news and disclosures from the web, then analyze and report on stock price movement factors.

So, I threw it a prompt: "Report the reason for XX company's sudden stock drop, citing specific sources." Honestly, I expected a faster, more accurate answer than I could get manually.

The AI's "Plausible" Analysis Report

A few minutes later, the AI generated a report that looked legitimate. Its conclusion:

"The stock price drop is not due to a company-specific issue, but rather a sector-wide risk-off event."

It even provided supporting evidence:

// AI's plausible but incorrect explanation
const ai_response = `{
  "conclusion": "The drawdown is sector-wide, NOT company-specific.",
  "evidence": "Peers like Company A (-10.7%) and Company B (-14.1%) also dropped on the same day.",
  "reasoning": "It is unlikely that a Japan-specific catalyst would cause a US-based peer to fall 10.7%."
}`

It cited Company A falling -10.7% and Company B falling -14.1% on the same day, arguing that "it's unlikely a Japan-specific catalyst would cause a US-based peer to drop over 10%." On the surface, it made sense.

For a moment, I almost accepted it: "Ah, right, the market sentiment must have been bad." But -33%? That's an extreme drop. It was too abnormal to simply dismiss as a sector-wide issue.

This gut feeling was what allowed me to uncover the AI's lie.

The Answer Was in the Primary Source

In the end, I did what I always do: I dug into the primary sources myself. I went to the company's IR site, and the answer was immediately there.

"Notice Regarding Issuance of New Shares Through Third-Party Allotment."

It was a public offering. And the dilution rate was a significant 15.79%. This was announced after market close. No wonder the stock almost hit its limit down the next day. It was entirely a company-specific factor.

// The ground truth found in primary sources
const ground_truth = `{
  "catalyst": "Public stock offering announced after market close.",
  "impact": "15.79% dilution of shares.",
  "result": "Stock price dropped -15.02% the next day (limit down)."
}`

The AI completely ignored this crucial primary information (the PDF disclosure) and instead picked up only secondary data (other companies' stock prices) from web searches, fabricating a plausible narrative.

AI doesn't say "I don't know." It fabricates the most coherent story from the fragmented information it has. This was my personal experience with hallucination.

The Limits of AI, and How to Use It Correctly

This incident drastically changed my perspective on AI.

In other cases, the AI had correctly identified reasons for stock price changes. But even then, the evidence it presented was always news articles reported after the stock had moved. In other words, AI can offer post-hoc explanations, but it cannot predict. Obvious, but a critical distinction.

Blindly relying on AI for "answers" is too dangerous. So, how should we use it?

My conclusion: use it as an "aid" for human primary source research.

The role of my AI agent now isn't to "give answers." It's to "accelerate the process of finding answers."

For example, I use it like this:

"From this 220,000-character financial report PDF, extract all mentions of 'impairment loss' and 'goodwill,' then summarize them chronologically."
"From 5 years of timely disclosure data, list the dates and overviews of any announcements regarding 'new share issuance' or 'stock splits.'"

Tasks like extracting keywords and organizing vast amounts of data, which would take a human several hours, are delegated to the AI. Based on the output, I perform the final analysis and make judgments myself. The AI is merely an excellent research assistant. I'm the driver.

Never take AI-generated text at face value; always verify with primary sources. Skip this crucial step, and you'll eventually be tripped up by a "plausible lie" like I was.

As someone who develops AI myself, I need to understand its limitations better than anyone. This failure was a valuable lesson that reinforced that understanding.

DEV Community: oji - building AI in public

My custom filter showed "zero false positives"! Then I realized my validation logic was flawed, letting all the garbage through.

Why "Zero False Positives"?

How I Fixed It

My Takeaway

米国企業の分析bot、外国企業だけ“黙って”分析をスキップしてた話 — 10-Kと20-Fで「リスクの章立て」が違う罠

何が起きたか

原因: 「10-K」と「20-F」という様式の違い

修正: フォームタイプをちゃんと見て、失敗を区別する

学び: エラーを握りつぶすと、静かに死ぬ

My Bot's KPI: Trades 2 > Opportunities 1?! When Your Numerator and Denominator Don't Observe the Same Time.

What Happened: My Tests Turned Red with an AssertionError

The Cause: The End of the Month Hadn't Arrived Yet

The Fix: Unifying the Observation Point to the "Entry Date"

The Lesson: "Numerator and Denominator on the Same Playing Field" - A Fundamental Principle

One Windows Task Scheduler Setting Made the Difference Between Bot Apocalypse and Resiliency

What Happened: My Dashboard Was Too Quiet This Morning

The Cause: One Tiny Checkbox

Fix and Prevention: Auditing Settings with Code

Lesson Learned: Manual Configuration Rollouts Always Lead to Accidents

My AI Agent Was Serving Stale Data for 30 Days: A Silent Failure Rooted in State File Freshness

What Happened: My Agent Served the Same Content for 30 Days Straight

The Investigation: A Compound Failure from Two Factors

1. Configuration "Drift"

2. Silent Failure Ignoring State File "Freshness"

The Fix: Adding "Freshness Guards" to All State-Handling Code

Lessons Learned: Fail-Safes Are the Lifeline for Personal Automation

副業応募botが受注ゼロ！犯人は『毎週クラッシュして成果を捨てる』1行のバグだった話

原因：エラー1つで全データを吹き飛ばす脆すぎる設計

修正：死ぬことを前提にしたエラー処理と中間保存

学び：本当の課題はコードの先にあった

My monitoring tool lied to me and secretly deleted all past data integrity issues

The Discovery: Slack Went Quiet

The Cause: Self-Incriminating Logic in the Monitoring Tool

The Fix: When in Doubt, Do Nothing (Fail Safe)

Lessons Learned: Don't Over-Trust Your Own Tools

My Trading Bot Silently Ignored Signals for 41 Days: The "Opportunity vs. Execution" Logging That Saved Me

Digging into Logs Revealed "Ignored Signals"

The Fix: Log "Opportunities" and "Executions"

Conclusion: The Scariest Thing in Bot Dev is "Quiet Failure"

My AI Agent Recommended a Non-Existent Investment Product — Exposing Information Gaps Between Fund Distributors and Asset Manage

The Problem: AI Recommends a "Non-Existent Investment Trust"

Deeper Dive: The Trap of Information Hierarchy

The Fix: Embedding Fact-Checking into Code

The Lesson: Master Your Primary Sources

When I Asked AI to Analyze "FANG+" and It Started Investigating Oil Companies — The Ticker Symbol Trap

Asked for FANG+ Analysis, Got an Oil Price Report

The Culprit: "FANG" Lurking in the S&P 500 List

The Fix: Eliminate Ambiguity, Define Universe Strictly

The Lesson: Financial Data is a Minefield of "Name Collisions"

Predicting Shareholder Meeting Failure: How AI Quantified Risk from a Single Sentence in Past Minutes and a 5x Increase in Share

The Trigger: Will This Reverse Stock Split Actually Pass?

The AI's Discovery: A Single Sentence Revealing Past Failures

Combining Qualitative Insights with Quantitative Data

The Final Piece: Reading the Company's Countermeasures

Lessons Learned: Combining Text and Numbers Changes Everything

My AI Lied to Me About a Stock Crash — Adversarial Testing Revealed Its Limits (and How to Use AI Correctly)

The Trigger: A Stock I Held Dropped -33%

The AI's "Plausible" Analysis Report

The Answer Was in the Primary Source

The Limits of AI, and How to Use It Correctly

What Happened: My Tests Turned Red with an `AssertionError`