DEV Community: Hidenari Fujiwara

Your Browser Automation Clicks Might Be Landing 25% Off — And Nothing Will Tell You

Hidenari Fujiwara — Sun, 16 Aug 2026 01:52:34 +0000

I spent a day pointing an AI agent at a browser to publish one product across four marketplaces. Most of it worked. The parts that didn't work failed in the worst possible way: silently, with no error, no exception, and no log line.

Here are the five failure modes I hit, in the order I hit them, and the probe that turned the worst one from "this site is broken" into a two-line fix.

Everything below is from one working session. No hypotheticals.

The symptom: clicks that do nothing

I asked the agent to click a menu item. The tool reported success:

[computer:left_click] Clicked at (383, 734)

Nothing happened. No menu opened, no network request fired, no console error. I re-read the DOM — the element was there, visible, not disabled, not covered by an overlay:

const el = document.elementFromPoint(383, 734);
// => <span>削除</span>   ← the exact element I wanted

So the coordinate was right, the element was right, and the click "succeeded". And nothing happened.

I burned close to an hour assuming the site was blocking synthetic input. It wasn't.

The probe: make the page answer for itself

Instead of reasoning about it, I made the page report whether a click ever arrived. Inject a button at a known position with a counter:

const b = document.createElement('button');
b.id = '__clicktest';
b.textContent = 'CLICK TEST';
b.style.cssText =
  'position:fixed;top:200px;left:200px;width:220px;height:60px;' +
  'z-index:2147483647;background:#f0f;';
b.addEventListener('click', (e) => {
  window.__clickOK = (window.__clickOK || 0) + 1;
  window.__trusted = e.isTrusted;
});
document.body.appendChild(b);
window.__clickOK = 0;

const r = b.getBoundingClientRect();
`center=${r.x + r.width / 2},${r.y + r.height / 2}`;
// => center=310,230

Then click that exact center and read the counter:

window.__clickOK;  // => 0

Zero. The click was not landing on a button that occupied 220×60 pixels at a coordinate I had just measured. That rules out "the site blocks synthetic events" — a blocked event still arrives; it just gets ignored by the handler. Mine never arrived.

So the coordinate space was wrong.

The math: two coordinate spaces, one number

The automation tool takes coordinates in screenshot space. The page reports coordinates in CSS pixel space. If the screenshot is captured at a different scale than the viewport, every coordinate you read from the DOM is wrong by a constant factor — and the failure is invisible, because a click still happens, just somewhere else.

I measured the factor by bisecting. Device-pixel space (×2, since devicePixelRatio was 2) missed. Then:

window.__clickOK;  // after clicking (232, 172) instead of (310, 230)
// => 1

That gives the ratio directly:

k = 232 / 310 = 0.7484

Viewport was 1702 CSS px wide; the screenshots came back ~1274 px. 1274 / 1702 ≈ 0.7485. It matched.

From then on, every click went through one conversion:

const K = 0.7484;  // measure this yourself; do not copy mine

function clickPoint(el) {
  const r = el.getBoundingClientRect();
  return {
    x: Math.round((r.x + r.width / 2) * K),
    y: Math.round((r.y + r.height / 2) * K),
  };
}

Every interaction that had failed for the previous hour started working on the first try — dropdowns, confirmation dialogs, menu items.

The lesson isn't the number. k depends on your window size, DPI, and tool. The lesson is that you should never trust a coordinate you didn't verify with a probe, because this class of bug produces no error at any layer.

Run the probe once at the start of a session. It costs three seconds.

Failure mode 2: the menu that was below the fold

With clicks fixed, one menu item still did nothing. Same probe logic applied — elementFromPoint returned null for it.

const r = item.getBoundingClientRect();
r.y;                  // => 981
window.innerHeight;   // => 876

The dropdown extended past the bottom of the viewport. getBoundingClientRect() happily returns coordinates for content that isn't on screen, and a click at y=981 in a 876px viewport goes nowhere.

Scroll first, then re-measure. Never cache coordinates across a scroll:

el.scrollIntoView({ block: 'center' });
await new Promise(r => setTimeout(r, 300));
const r2 = el.getBoundingClientRect();  // re-read, always

Failure mode 3: file inputs that don't exist yet

One site had an upload drop zone with no file input in the DOM:

document.querySelectorAll('input[type=file]').length;  // => 0

The input is created on demand when you click the zone. So: click the zone, then look. To catch the element before the app calls .click() on it (which opens a native dialog you can't drive), patch createElement first:

const orig = document.createElement.bind(document);
window.__created = [];
document.createElement = function (tag, ...rest) {
  const el = orig(tag, ...rest);
  if (String(tag).toLowerCase() === 'input') {
    setTimeout(() => {
      if (el.type === 'file') window.__created.push(el);
    }, 0);
  }
  return el;
};

Now click the zone and the input is waiting for you in window.__created. Populate it with a DataTransfer and dispatch change:

const dt = new DataTransfer();
dt.items.add(file);
input.files = dt.files;
input.dispatchEvent(new Event('change', { bubbles: true }));

This worked on two of the sites I tried. On the third it didn't — see below.

Failure mode 4: synthetic drops that React ignores

Some drop zones aren't inputs at all. The obvious move is to synthesize the drop:

const dt = new DataTransfer();
dt.items.add(file);
for (const type of ['dragenter', 'dragover', 'drop']) {
  const ev = new DragEvent(type, { bubbles: true, cancelable: true });
  Object.defineProperty(ev, 'dataTransfer', { value: dt });
  zone.dispatchEvent(ev);
}

I tried this against every plausible target — the drop zone, its ancestors, document, window. Nothing. The editor never registered a file.

To be sure the file wasn't the problem, I generated one in-page so there was no transfer step at all:

const c = document.createElement('canvas');
c.width = 1200; c.height = 800;
c.getContext('2d').fillRect(0, 0, 1200, 800);
const blob = await new Promise(r => c.toBlob(r, 'image/jpeg', 0.8));
const file = new File([blob], 'test.jpg', { type: 'image/jpeg' });

Same result. That's a real boundary: for that editor, drag-and-drop needs genuine OS-level input, and no amount of event synthesis substitutes for it. I stopped and did those uploads by hand.

Knowing where the wall is is worth more than another hour of clever attempts.

Failure mode 5: localhost is still mixed content

To avoid shipping image bytes through a JS payload, I served them locally with CORS enabled and fetched them from the page:

class H(http.server.SimpleHTTPRequestHandler):
    def end_headers(self):
        self.send_header('Access-Control-Allow-Origin', '*')
        super().end_headers()

curl confirmed the header. From an https:// page, the fetch hung until it timed out:

await fetch('http://127.0.0.1:8941/img1.png');  // never resolves

An https page fetching http://127.0.0.1 is mixed content. The CORS header is irrelevant — the request never gets far enough to matter. Terminate that path early instead of debugging your server.

The preflight I now run first

Five minutes of setup that would have saved me most of a day:

// 1. Coordinate probe — is my click space the page's space?
//    (inject the test button above, click its center, read window.__clickOK)

// 2. Is the target actually on screen?
const onScreen = (el) => {
  const r = el.getBoundingClientRect();
  return r.top >= 0 && r.bottom <= window.innerHeight && r.width > 0;
};

// 3. Does this page even have the input I assume it has?
document.querySelectorAll('input[type=file]').length;

// 4. Verify before every destructive click.
//    Hover first, then confirm what is under the cursor:
document.elementFromPoint(x / K, y / K)?.textContent?.trim();

Step 4 matters more than it looks. In one menu, "Delete" sat directly below "Unpublish" — 24 pixels apart. With a 25% coordinate error, "unpublish" lands on "delete". Hovering and reading back the element text before committing turns a destructive misfire into a no-op.

What I'd tell my past self

The productive shift wasn't a better selector strategy. It was refusing to reason about why a click "didn't work" and instead making the page report what it actually received.

Every one of these five failures is invisible from the outside: success-shaped tool output, no exception, no console error. The only reliable signal came from instrumenting the page and reading a counter.

If your automation is mysteriously doing nothing, don't start with the selector. Start with the probe.

All measurements here come from a single session driving Chrome against live marketplace sites. The scaling factor k is specific to that window and tool — measure your own.

Building an Incremental Financial Data Pipeline on a Free Tier (¥0 Infrastructure)

Hidenari Fujiwara — Fri, 14 Aug 2026 12:59:58 +0000

My stock research pipeline ingests prices and financials for every listed company in Japan, runs event-study backtests, and tracks a live paper portfolio — on delayed free-tier data, a laptop, and an infrastructure bill of exactly zero yen. This is the architecture, and the design decisions that made "free" actually work.

When people hear "financial data pipeline," they picture streaming infrastructure and four-figure data subscriptions. My entire setup is: one official free API, Python, parquet files, and a macOS scheduler. It has processed 276 small-cap stocks and 1,385 earnings events through backtests, and it runs unattended every trading day.

The unlock wasn't a clever tool. It was accepting one constraint and designing around it.

1. Why delayed data + a free tier is actually practical

My data source is J-Quants, the Japan Exchange Group's official API, whose free tier delays data by 12 weeks. That sounds disqualifying — until you separate what a research pipeline actually does into two jobs:

Research (backtesting, screening, factor analysis): operates on history. A 12-week delay is irrelevant when you're studying six years of earnings events.
Execution (what's the price right now?): needs today. But it needs today for only ~20 tickers in my portfolio, not the whole market — and a free quotes library (yfinance) covers that.

Keeping these two concerns in separate modules means the expensive requirement (fresh data) shrinks to a tiny surface area, and the bulky requirement (deep history, full market) lands on the free tier. Most hobby-quant projects die by paying real-time prices for a research problem.

2. Ingestion: assume the API hates you

The retrieval layer makes three assumptions that turned out to be load-bearing:

Responses are paginated — follow the pagination_key until it's gone.
You will get rate-limited — exponential backoff on HTTP 429.
Even polite clients should throttle — a minimum 250ms between requests, on top of backoff, because a full-universe pull is thousands of calls and you want to be a good citizen of an API you depend on.

I walked through the client code in the J-Quants guide, so here I'll just note the architectural point: ingestion is its own layer with no business logic in it. It takes an endpoint and parameters, returns rows, and knows nothing about stocks. Every time I've blurred that boundary in past projects, changing a research question forced me to touch (and break) download code.

3. The cache layer: misses are data too

Everything the API returns is written to parquet, keyed by request. The interesting decision — the one that halved my backtest runtime — was what to do when the API returns nothing:

def fetch_cached(key: str, fetch_fn):
    path = CACHE_DIR / f"{key}.parquet"
    if path.exists():
        df = pd.read_parquet(path)
        return None if df.empty else df    # a cached "nothing" is an answer
    df = fetch_fn()
    df.to_parquet(path)                    # persist even when empty
    return None if df.empty else df

Financial data is full of legitimate emptiness: delisted tickers, market holidays, companies that didn't exist yet, quarters with no disclosure. My first backtest run spent roughly half its wall-clock time re-asking the API questions whose true answer was "nothing," every single run. Caching the misses — persisting empty frames instead of only successful payloads — cut runtime in half, with zero change to results.

The general lesson: "no data" is a fact about the world, and facts are cacheable. Most caching tutorials only cache hits.

4. Incremental updates — and what tools like Airbyte formalize

With a cold cache, a full-universe pull takes a long evening. With a warm one, the daily update is minutes, because the pipeline only fetches what it doesn't already have: the cache key encodes (endpoint, ticker, period), so yesterday's history is never re-downloaded, and each day appends only the new slice.

That's an incremental sync — hand-rolled. If you've used a data-integration platform like Airbyte, you'll recognize everything above as things it formalizes: sources with pagination and rate-limit handling, destinations, cursor fields ("what did I fetch last?"), incremental vs. full-refresh modes. Building it by hand taught me exactly what those abstractions are for — and where my version is weaker (my "cursor" is a filename convention; state lives in a directory listing; there's no schema migration story). If this pipeline ever outgrows one laptop, the migration path is clear: my ingest layer becomes a declarative source config, parquet becomes DuckDB or Postgres, and the cache logic I hand-wrote disappears into the platform's sync engine. The architecture translates one-to-one — because it's the same problem.

For a personal-scale project, though, the file-based version has a real virtue: you can ls your state. Debugging a sync issue is open cache/8035_daily.parquet, not spelunking through a job orchestrator.

5. Two pitfalls that produce beautiful, wrong results

Both of these produced exciting "findings" that died under inspection:

Stock splits make fundamentals look broken. Splits arrive in the disclosure stream; if you join fundamentals to prices naively, a 1:3 split reads as earnings collapsing by two-thirds — or, joined the other way, as a stock suddenly trading at a third of its "fair value." My event studies had to adjust for split factors before computing any surprise or valuation metric. If your backtest finds that stocks crater on earnings and then recover perfectly, check your split handling before you check your genius.

One-off gains make garbage look cheap. Company-issued forecasts make forward-valuation screens easy — and easily fooled. A company that sold its headquarters building shows a spectacular earnings yield for exactly one year. My screen excludes implausible earnings yields on the theory that if a small-cap looks too cheap, it's usually an asset sale, not a misunderstood business. That single guard removed most of the false positives from the cheapest quintile.

What it all costs

J-Quants free tier: ¥0
Quotes for ~20 live tickers: ¥0
Storage: a few GB of parquet on my laptop: ¥0
Scheduling: launchd, which macOS already runs: ¥0
Compute: the laptop I already own

The constraint that made this possible wasn't budget discipline. It was the architectural split in section 1 — once research and execution stopped sharing a data requirement, every remaining piece had a free-tier-shaped hole it fit into.

Whether the strategies this pipeline tests actually make money is a different story — most of mine didn't survive their own backtests — but that's the point of building the pipeline first: cheap, fast falsification beats expensive conviction.

Data is used under J-Quants' personal-use license. Nothing here is investment advice — the pipeline exists precisely because I don't trust my own hunches.

launchd: The macOS Scheduler You Already Have (A Practical Tutorial)

Hidenari Fujiwara — Fri, 14 Aug 2026 12:32:46 +0000

Cron exists on macOS, but Apple has been quietly steering everyone toward launchd for twenty years. Here's a hands-on guide to running your scripts on a schedule — with two real jobs I run every week, and the mistakes I made so you don't have to.

I run two scheduled jobs on my Mac that quietly do work for me:

Every weekday at 15:45 (after the Tokyo market close), a job records prices for a paper-trading portfolio and sends me a macOS notification with the day's P&L.
Every Monday at 09:00, a job pulls my dev.to article stats into a CSV so I can see trends without checking dashboards.

Total infrastructure cost: ¥0. No cloud, no serverless, no Docker. Just launchd, the init system that is already running on your Mac. This tutorial builds up a working job from scratch.

Step 1: Understand the one file that matters

A launchd job is defined by a single XML file (a "property list" / plist) in ~/Library/LaunchAgents/. The naming convention is reverse-DNS: com.yourname.yourjob.plist.

Here is a minimal, real one — my weekly stats job:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.example.devto-stats</string>

    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/python3</string>
        <string>/Users/you/projects/writing/scripts/devto_stats.py</string>
    </array>

    <key>StartCalendarInterval</key>
    <dict>
        <key>Weekday</key><integer>1</integer>  <!-- Monday -->
        <key>Hour</key><integer>9</integer>
        <key>Minute</key><integer>0</integer>
    </dict>

    <key>StandardOutPath</key>
    <string>/Users/you/projects/writing/data/launchd.log</string>
    <key>StandardErrorPath</key>
    <string>/Users/you/projects/writing/data/launchd.log</string>
</dict>
</plist>

Four things to notice:

Label must be unique on your machine and should match the filename.
ProgramArguments is an array: binary first, then each argument as its own <string>. This is the #1 source of silent failures — python3 script.py as a single string will not work.
StartCalendarInterval is cron's schedule, but readable. Omit Weekday to run daily; use an array of dicts for multiple times.
Always set the log paths. launchd failures are invisible without them.

Step 2: Load it, run it, prove it works

# Load the job (modern syntax; "load" is the deprecated spelling)
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.example.devto-stats.plist

# Is it registered?
launchctl list | grep devto-stats

# Don't wait for Monday — force a run right now:
launchctl kickstart gui/$(id -u)/com.example.devto-stats

# Watch what happened:
tail -f ~/projects/writing/data/launchd.log

To unload: launchctl bootout gui/$(id -u)/com.example.devto-stats.

kickstart is the debugging superpower. Edit script → kickstart → read log, in a tight loop, instead of waiting for the schedule to come around.

Step 3: Write scripts that survive being run by a robot

A script that works in your terminal will betray you under launchd, because launchd gives it a minimal environment: no shell profile, no PATH additions, no working directory you expect, no API keys you exported once in .zshrc. Rules I follow after learning each one the hard way:

Absolute paths for everything.

#!/bin/zsh
REPO="/Users/you/projects/trading"     # not ~/projects, not $PWD
PY="$REPO/.venv/bin/python"            # the venv's python, not "python3"
cd "$REPO" || exit 1

Read secrets from files, not from the environment. My stats script reads its API key from a git-ignored file and exits politely when it's missing, so the job is safe to install before it's configured:

KEY_FILE = ROOT / ".devto_api_key"

def main() -> None:
    if not KEY_FILE.exists():
        print("API key not set. Skipping.")
        return
    key = KEY_FILE.read_text().strip()
    ...

Make re-runs harmless (idempotency). Scheduled jobs get run twice — by you while debugging, by launchd after a missed schedule. My market-tracking job records positions only if they haven't been recorded already:

existing = {p["ticker"] for p in journal.open_positions()}
new_rows = rows[~rows["ticker"].isin(existing)]   # re-running adds nothing

Append to logs with a timestamp header, so the log file reads as a history:

{
  echo "===== $(date +%F) $(date +%H:%M) ====="
  "$PY" tracker.py report
  echo ""
} >> "$LOG"

Step 4: Close the loop with notifications

A scheduled job you never hear from is a job you'll forget exists — until it's been broken for a month. The cheapest feedback channel on macOS is a notification, one line of AppleScript away:

summary=$(grep "Portfolio" "$LOG" | tail -1)
osascript -e "display notification \"${summary:-Job failed, check log}\" \
  with title \"Daily tracker\"" 2>/dev/null || true

Note the fallback text: if the report failed and $summary is empty, the notification says so instead of silently not appearing. The || true keeps a notification failure from failing the whole job.

Step 5: Know when not to use launchd

Honest limitations, from running these jobs for a while:

Your Mac must be awake. launchd runs missed StartCalendarInterval jobs when the machine wakes — but only the most recent miss, and "sometime after wake" may be too late for time-sensitive work. My 15:45 market job assumes the laptop is open in the afternoon; that assumption holds for me, but a server it is not.
No retries, no alerting, no history UI. You are the monitoring system (hence Step 4).
If the job must run at an exact minute regardless of your laptop's mood, use a cloud scheduler. The point of launchd is the enormous class of personal automation where "today, roughly on time, for free, with my local files and credentials" is exactly right.

The part where this compounds

Each job is small. The pattern is what compounds: every repetitive check in your life — a dashboard you open, a number you copy into a spreadsheet, a daily record you keep meaning to keep — is a candidate for a 30-line script and a plist. I now write the script, wire the plist, watch it via notifications for a week, and forget about it.

My laptop does the remembering. That's the whole trick.

Both jobs in this article are simplified from real ones I run: a paper-trading tracker for a rule-based stock research project (build log) and a writing-stats collector. Simplifications don't change behavior.

Japan's Stock Exchange Has an Official API for Individuals. Almost Nobody Outside Japan Knows About It.

Hidenari Fujiwara — Fri, 14 Aug 2026 12:05:31 +0000

A practical developer's guide to J-Quants — the Japan Exchange Group's official market data API — from someone who has been running backtests and a daily pipeline on it for months.

If you've ever tried to analyze the Japanese stock market programmatically from outside Japan, you know the data situation looks hostile. Bloomberg is enterprise money. yfinance gives you prices but its Japanese fundamentals are patchy. Most tutorials just tell you to trade the US market instead.

Here's what almost nobody writes about in English: the Tokyo Stock Exchange's parent company (JPX) runs an official data API for individual investors, and the free tier is genuinely useful. It's called J-Quants, it has an English version of the site, and I've been using it for months as the backbone of a rule-based research pipeline (the build log is here).

This is the guide I wish had existed when I started.

What you get, and what it costs

Plan	Price/month	History	Catch
Free	¥0	2 years	12-week delay on data
Light	¥1,650 (~$11)	5 years	—
Standard	¥3,300 (~$22)	10 years	—
Premium	¥16,500 (~$110)	20 years	—

The 12-week delay on the free tier sounds disqualifying until you realize what it's actually for: backtesting. Historical research doesn't care about the delay. For live tracking I fill the gap with yfinance — delayed official data for the heavy lifting, free real-ish-time quotes for marking to market. That combination costs ¥0.

One thing to check before you build anything: the license is personal use. No redistribution, no commercial resale of the data. Build your own research tools, not a public data product.

The API in three endpoints

The current version (V2) uses plain API-key auth — a x-api-key header, no token-refresh dance. The three endpoints I use daily:

/v2/equities/master — all listed companies: code, name, market segment, sector, size classification
/v2/equities/bars/daily — daily OHLCV
/v2/fins/summary — financial statements, including company-issued forecasts (more on why that's special below)

A minimal client that handles the two things the docs won't hold your hand through — pagination and rate limits:

import time, requests

API = "https://api.jquants.com/v2"
HEADERS = {"x-api-key": YOUR_KEY}

def get_all(path: str, params: dict | None = None) -> list[dict]:
    """Follow pagination_key until exhausted; back off on 429s."""
    params, rows, data_key = dict(params or {}), [], None
    while True:
        for attempt in range(5):                      # 429 -> exponential backoff
            r = requests.get(f"{API}{path}", headers=HEADERS,
                             params=params, timeout=60)
            if r.status_code == 429:
                time.sleep(1.5 * (attempt + 1) ** 2)
                continue
            break
        r.raise_for_status()
        j = r.json()
        if data_key is None:                          # payload key varies by endpoint
            data_key = next(k for k in j if k != "pagination_key")
        rows.extend(j[data_key])
        if not j.get("pagination_key"):
            return rows
        params["pagination_key"] = j["pagination_key"]

Two production notes baked into that snippet:

The payload key changes per endpoint (bars, summary, …), so detect it instead of hardcoding.
Throttle yourself. I keep a minimum 250ms between requests on top of the backoff; full-universe pulls run for a while and the API will 429 you without it.

The feature US-market people don't believe: company-issued guidance

Here's the structurally interesting part. Japanese listed companies publish their own earnings forecasts as part of standard disclosure — next fiscal year's revenue, operating profit, EPS, straight from management, machine-readable in /v2/fins/summary.

If you've only worked with US data, let that sink in: no scraping analyst consensus, no paying a vendor for estimates. The company itself tells you what it expects, updates it when things change, and the exchange serves it to you in JSON.

This enables research that's genuinely hard to do cheaply elsewhere. My own use: post-earnings drift studies keyed on guidance revisions (did management raise the forecast?) and value screens using forward EPS from the source rather than trailing numbers. Whether those strategies survive scrutiny is a separate question — mine mostly didn't — but the data access is remarkable for ¥0.

Gotchas from months of production use

Cache aggressively, including empty responses. My first full-universe backtest spent half its runtime re-requesting data that legitimately didn't exist (delisted names, holidays, pre-IPO dates). The fix that mattered wasn't caching hits — it was caching misses:

def fetch_cached(key: str, fetch_fn):
    path = CACHE_DIR / f"{key}.parquet"
    if path.exists():
        df = pd.read_parquet(path)
        return None if df.empty else df       # cached "nothing" is an answer too
    df = fetch_fn()
    df.to_parquet(path)                       # persist even when empty
    return None if df.empty else df

Corporate actions will bite your event studies. Stock splits show up in disclosures; if you join fundamentals to prices naively, a 1:3 split looks like earnings collapsing by two-thirds. Adjust before you celebrate a "signal."

Guard against one-off earnings spikes. Company forecasts make forward-valuation screens easy, but a company selling its headquarters building looks spectacularly "cheap" for exactly one year. My screens exclude implausible earnings yields — if it looks too good, it's usually an asset sale, not a business.

Free-tier delay shapes your architecture. Twelve weeks of delay means the free tier can't tell you about this quarter. Design around it: J-Quants for research and screening, a real-time source for execution-time prices. Keeping those two concerns in separate modules made this painless.

Is it worth it?

For anyone who wants to do serious, reproducible research on a major market without a data budget: yes, emphatically. An official exchange API with fundamentals and management guidance, a usable free tier, pandas-friendly JSON, and an English site — I don't know of an equivalent offer from any other major exchange.

The Japanese market itself is having a moment with international investors. The tooling to study it properly is sitting right there, mostly undocumented in English. Now you know.

I use J-Quants under its personal-use license for my own research. Nothing here is investment advice. If you build something with it, I'd genuinely like to hear about it in the comments.

I Asked Claude Code to Find Me a Trading Edge. It Killed Three Strategies First.

Hidenari Fujiwara — Fri, 14 Aug 2026 07:26:03 +0000

How I built a fully automated, rule-based Japanese stock research pipeline with Claude Code, J-Quants, and a macOS cron job — and why "the AI is not allowed to predict anything" turned out to be the most useful constraint in the whole project.

Everyone's first idea for "AI + stocks" is the same: ask the model whether a stock will go up. That idea has been tested, and it fails — LLMs guessing post-event price direction land around coin-flip accuracy. I'm a freelance web developer in Japan, not a quant, and I knew that if I let an LLM "predict" anything I would just be laundering my own wishful thinking through a chatbot.

So I gave Claude Code a different job description. In my project's CLAUDE.md — the standing instructions file the agent reads every session — there's a hard rule:

No predictions. Data only. The AI's role is limited to four things: (1) structuring data, (2) computing factual metrics, (3) checking facts against pre-defined rules, (4) explaining results with sources. Trading decisions are rule-based. Overfitting to past data is a hidden prediction — prefer economically sensible rules.

Claude Code's job was to be the engineer: build the data pipeline, implement backtests I specified, and then — this is the important part — kill my ideas with evidence. Over a few weeks it killed three of them. Here's the honest record.

The stack

Nothing exotic. The point was cheap and reproducible:

J-Quants API — the official Japan Exchange data service. The free personal tier has a 12-week delay on prices, which is fine for backtesting; I fill in current prices with yfinance for live tracking.
Python + pandas for backtests, with a local file cache so repeated runs don't hammer the API. (One lesson learned: cache empty responses too. Half my early runtime was re-requesting data that legitimately didn't exist.)
Claude Code as the pair engineer. I describe the rule, it writes the backtest, I interrogate the output.
launchd (macOS's cron) for the daily tracking job. No cloud, no server bill.

Round 1: technical strategies — dead

First I had Claude Code implement the classics on Nikkei-universe data: breakout entries, RSI mean-reversion, with realistic costs and taxes included.

Ten-year portfolio result: +15.4% total. Sounds okay until you put it next to the benchmark: buy-and-holding the Nikkei over the same period returned +138.8% (with a −26% max drawdown). My "strategy" wasn't a strategy; it was an expensive way to sit out a bull market.

Verdict: dead. Next.

Round 2: earnings drift (PEAD) — killed by a single filter

Post-earnings announcement drift is one of the best-documented anomalies: stocks that beat guidance keep drifting up for weeks. On large caps, my backtest found nothing — no monotonicity across surprise sizes, slightly negative drift everywhere. Institutional money has eaten that edge.

Small caps looked genuinely exciting: 276 stocks, 1,385 earnings events, 423 trades. The threshold sweep was beautifully monotonic — bigger guidance beats, bigger drift: +1.6% → +2.1% → +2.7% → +3.4% per trade as the threshold rose.

Then I asked Claude Code to add one boring, adult filter: only keep stocks with at least ¥100M in daily trading value — i.e., stocks I could actually buy without moving the price.

The edge evaporated: −0.34% per trade on the 33 surviving trades. At ¥300M/day it got worse (−2.48%).

The anomaly was real, but it lives exclusively in stocks too illiquid to trade at size. This was the single most valuable chart the pipeline ever produced, and it's a negative result. If your backtest doesn't include an executability filter, it isn't a backtest — it's fan fiction.

Round 3: small-cap value × quality — the survivor

Third idea: a quarterly-rebalanced screen. Rank small caps by valuation (with quality guards: equity ratio ≥ 25%, positive forecast EPS, and a filter against one-off earnings spikes), split into quintiles, hold the cheapest.

This one behaved differently. The cheapest quintile (Q1) returned +8.89% per quarter, returns were fully monotonic down the ranks, the cheap-vs-expensive spread was +8.96% per quarter, and Q1 beat the universe average in all six backtest quarters.

And the crucial difference from PEAD: the edge survived the liquidity filter. Restricted to ≥¥100M/day stocks, Q1 still returned +10.06% per quarter (+78.7% cumulative vs +42.2% for the universe). The sign didn't flip.

One strategy out of three survived its own audit. That's the pipeline working as intended.

Automating the paper trade

A backtest that survives in-sample is still just a hypothesis, so the surviving strategy went into "Phase 0": a paper portfolio tracked automatically every trading day, with a pre-committed bar to clear — beat the benchmark over 2–3 months without touching the parameters — before real money scales beyond pocket change.

The automation is deliberately low-tech. A launchd job fires at 15:45 JST on weekdays (after the Tokyo close) and runs a shell script:

#!/bin/zsh
# Runs from launchd on weekdays at 15:45 JST.
# 1) Record any not-yet-entered positions at today's opening price
# 2) Append the daily report to the log, push a summary to macOS notifications
REPO="$HOME/work/makemoney"
PY="$REPO/.venv/bin/python"
{
  echo "===== $(date +%F) ====="
  "$PY" phase0_track.py open --date "$(date +%F)"
  report=$("$PY" phase0_track.py report)
  echo "$report"
} >> "$REPO/data/phase0_daily.log"

summary=$(echo "$report" | grep "Portfolio" | head -1)
osascript -e "display notification \"$summary\" with title \"Phase 0\""

The Python side is a small CLI with two subcommands. open records entries at the day's opening price (same conditions as the fractional-share "opening auction" orders I'd use with real money) with an idempotency guard, so re-running never double-records:

# double-entry guard
existing = {p["ticker"] for p in journal.open_positions()
            if p["strategy"] == STRATEGY}
lst = lst[~lst["ticker"].isin(existing)]

report marks the portfolio to market and compares it against two benchmarks (Nikkei 225 and a TOPIX ETF) measured from the same entry date — the comparison that Round 1 taught me never to skip. The result lands in a log file and a macOS notification. Total infrastructure cost: ¥0.

One month in: the honest numbers

As of 2026-08-14, one month after entry (18 positions, a small paper portfolio):

Portfolio: +4.32%
Nikkei 225: +2.55%
TOPIX ETF: +4.66%

Beating the Nikkei, narrowly losing to TOPIX. Inside the portfolio the dispersion is exactly what a small-cap quintile bet looks like: the best position is +46%, the worst is −33%. The daily report ends with a line I wrote for my own discipline: "The pass bar is 2–3 months against the benchmark. Do not react to daily noise."

Maybe it clears the bar in October. Maybe it doesn't and the strategy joins the graveyard with the other three. Either outcome is the system working.

What Claude Code was actually good at

A few workflow notes for anyone trying something similar:

Put your constraints in CLAUDE.md, not in your willpower. "No predictions", "never commit to main", "every research claim needs a source URL" — encoded as standing project rules, the agent enforces them even when I'm tempted not to.
Make the agent a strategy killer, not a strategy generator. LLMs are dangerously good at producing plausible-sounding trading ideas. They are actually useful at implementing the boring falsification machinery — cost models, liquidity filters, benchmark alignment — fast enough that you'll really run it.
Commit each negative result. My git log reads like a lab notebook: "large-cap PEAD: no edge", "PEAD edge concentrates in illiquid names", "value×quality survives liquidity filter". Claude Code picks up this context every session and stops me from re-testing yesterday's dead idea with today's optimism.
The agent writes the pipeline; the human owns the rules. Every threshold in the system (quality guards, liquidity floor, rebalance cadence) was decided by me and frozen before the evaluation window. The fastest way to fool yourself with an eager coding agent is to let it "just try a few more parameter values."

Nothing here is investment advice — it's a build log of a personal research tool, running on pocket-money stakes precisely because the evidence isn't in yet. J-Quants data is used under its personal-use license.