DEV Community: Mashi Mashi

Why "Accuracy" Is the Wrong Metric for Probabilistic Prediction Models

Mashi Mashi — Mon, 03 Aug 2026 00:29:09 +0000

If you build a model that predicts the probability of an outcome — a race result, a match winner, a medical risk score, anything where the output is meant to be a probability rather than a hard label — "accuracy" is almost always the wrong number to optimize for, and often the wrong number to even report. I run ai-race.jp, which produces probabilistic predictions for horse races, and the gap between "the model is accurate" and "the model is useful" shows up constantly in that domain. This isn't a post about picking winners; it's about why the standard classification metric misleads you once your output is a probability instead of a category, and what to look at instead.

Why accuracy breaks down for probabilistic output

Accuracy answers one question: for the class you predicted as most likely, how often were you right? That's a fine question when your model outputs a single label — spam or not spam, cat or dog. It's a bad question when your model's actual product is a probability distribution over many possible outcomes, because accuracy throws away everything except whether the top pick happened to win.

Take a race with twelve runners. A model that assigns the eventual winner a 9% probability, barely ahead of eleven other horses clustered near 8%, gets full credit under "accuracy" the same as a model that assigned that horse 60% and put daylight between it and the field. Both models "got it right." But the second model expressed real, calibrated confidence and the first one barely distinguished the winner from noise. If you only track accuracy, you cannot tell these two models apart, even though one of them is obviously doing something much more useful. Worse, a model can improve its calibration and honesty — start admitting more uncertainty in genuinely uncertain races — and accuracy won't move, or can even look slightly worse, because admitting uncertainty sometimes means not concentrating probability mass on the single most likely outcome.

Log loss: the metric that actually rewards honest probabilities

Log loss (cross-entropy) scores a prediction by how much probability mass the model assigned to the outcome that actually happened, penalizing confident wrong answers heavily and rewarding confident right answers, with a smooth gradient in between for everything less certain. If the true outcome gets assigned probability p, the loss contribution is -log(p). A model that says "5% chance" for the actual winner pays a much steeper penalty than one that said "25% chance," even if both technically ranked that horse outside their top pick.

This matters because log loss can't be gamed by just picking a favorite and hoping. To score well on log loss across many races, a model has to spread probability sensibly across plausible outcomes and reserve high confidence for situations that genuinely warrant it. It directly rewards the property you actually want from a probabilistic model: does the number mean what it says it means. Two models can have identical accuracy — same set of "correct" top picks — and meaningfully different log loss, and the one with lower log loss is the one whose probabilities you should trust more when making any downstream decision that depends on the actual probability value, not just the ranking.

Calibration: does 30% actually happen 30% of the time

Log loss is a single aggregate number; calibration is what you check to understand why that number is good or bad. A model is calibrated if, among all the predictions where it said "30% chance," the outcome actually happened close to 30% of the time. You check this with a reliability diagram: bucket predictions by their stated probability, and for each bucket plot the stated probability against the observed frequency. A perfectly calibrated model sits on the diagonal.

Systematic miscalibration shows up in predictable shapes. Overconfidence looks like a curve that sags below the diagonal at high stated probabilities — when the model says 70%, the true rate is more like 55%. Underconfidence does the opposite. Favorite-longshot bias, a well-documented pattern in betting markets broadly, tends to show up as heavy favorites being slightly underpriced in probability terms while longshots are systematically overpriced — bettors and models alike tend to overvalue a small chance at a big outcome. Checking calibration separately from log loss matters because it tells you where a model is wrong, not just how wrong it is on average: a single log loss number can look mediocre for a model that's excellent everywhere except one badly miscalibrated segment.

Why probability, not a hard pick, is the honest product

There's a specific reason this distinction is not academic in horse racing. Every race has a fixed pool of money and a payout structure — the "控除率" (deduction rate, commonly around 20–30% depending on bet type in Japan's pari-mutuel system) — meaning the pool structurally pays back less than it takes in, before you even get to prediction quality. Any tool that frames its output as "the winner" rather than "here is the estimated probability distribution over outcomes" is overstating what a model can honestly claim, given that even a well-calibrated model is operating inside a negative-sum structure. Reporting probabilities, log loss, and calibration is the honest framing: it tells you how good the estimates are as estimates, without implying that following them is a path to guaranteed profit. Nothing here is a suggestion to bet, and a low log loss does not offset the structural deduction rate — it just means the probabilities themselves are trustworthy as probabilities.

What to check instead of accuracy

If you're building or evaluating a probabilistic model, a reasonable checklist looks like: report log loss (or Brier score, which is similar but uses squared error instead of log) as your primary aggregate metric; plot a reliability diagram and check for systematic over- or under-confidence at different probability bands; and if you must report something accuracy-like for a general audience, pair it with the calibration data so a "70% confident and right" claim can actually be verified against the model's track record at that confidence level. None of this is exotic — it's standard practice in weather forecasting and quantitative finance, both fields that learned the accuracy trap decades ago because the cost of a badly calibrated probability is direct and measurable. Prediction modeling for any real-world stochastic event deserves the same discipline.

Shipping a Next.js App on Cloudflare Workers with OpenNext + D1

Mashi Mashi — Mon, 03 Aug 2026 00:26:19 +0000

I recently shipped souzoku-baton.jp, a Next.js app, on Cloudflare Workers using OpenNext instead of Vercel. The motivation was mostly cost and consolidation — I already run a handful of other properties on Cloudflare and didn't want a second billing relationship and a second set of DNS quirks just to host one more app. OpenNext's Cloudflare adapter has matured a lot, but there were three areas where the defaults quietly did the wrong thing, and I want to write them down before I forget the exact symptoms.

Why OpenNext instead of just Pages

Cloudflare Pages can serve a Next.js static export directly, but the moment you need server-side rendering, API routes with real backend logic, or middleware that does more than redirect based on a cookie, a plain static export stops being enough. OpenNext (@opennextjs/cloudflare) compiles a standard Next.js app into a Cloudflare Worker, mapping the Next.js server runtime onto Workers' request/response model and giving you access to bindings — D1, KV, R2, Durable Objects — directly inside your Next.js server code. That binding access was the actual reason I picked it: the app needed a real relational store for inheritance/estate-planning records, and D1 was the natural fit given everything else was already on Cloudflare.

The setup itself is close to the documented quick start: create-cloudflare scaffolds the project, wrangler.jsonc declares the D1 binding, and open-next.config.ts controls how the adapter builds the output. Where things got interesting was after the first deploy actually worked.

Problem 1: the D1 migrations ledger silently diverges from reality

D1 tracks applied migrations in a d1_migrations table that it manages itself, and wrangler d1 migrations apply is supposed to be the only thing that touches it. Early on, while iterating quickly, I ran a couple of schema changes with wrangler d1 execute --file directly against the remote database — faster to type than generating a proper migration file when I was just testing a column rename. That was the mistake. d1_migrations only gets a row inserted when you go through migrations apply, so after those direct executes, the ledger no longer matched the actual schema: the table structure had my changes, but the migrations system had no record of them.

The failure mode was confusing rather than loud. wrangler d1 migrations list reported every migration as unapplied, including ones whose effects were clearly already live in production. Running apply at that point would have tried to re-run migrations whose CREATE TABLE and ALTER TABLE statements would immediately fail against a schema that already had those columns. The fix was to manually reconcile the ledger: for each migration file whose changes were already present in production, I ran an INSERT OR IGNORE INTO d1_migrations (id, name, applied_at) VALUES (...) matching the migration's expected id and filename, effectively telling D1 "trust me, this one is already done." After that, migrations list and migrations apply behaved correctly again. The lesson was simple in hindsight: never touch a D1 database with raw execute once migrations are in play, even for something that feels like a one-line fix — the ledger has no way to know about it.

Problem 2: KV-backed rate limiting has real latency and consistency limits

I used Workers KV to implement a basic per-IP rate limiter on a couple of public form endpoints — read the current count, check it against a threshold, write the incremented count back. This is a well-known anti-pattern for KV specifically because KV is eventually consistent and optimized for high-read/low-write workloads, not for fast read-modify-write cycles. Under light traffic it worked fine. Under a short burst — someone refreshing a form repeatedly, or a bot hammering an endpoint — the count reads and writes raced against each other enough that the limiter under-counted requests by a meaningful margin, letting more through than the configured threshold.

I didn't need to rearchitect the whole thing; I moved just the rate-limiting counter to a Durable Object, which gives you a single consistent point of coordination per key (per IP, in this case) instead of KV's distributed eventual-consistency model. The Durable Object holds the counter in memory and persists it to its own storage, and because each DO instance is single-threaded with respect to its own state, the race condition disappears by construction rather than by careful timing. KV is still fine for things like feature flags or cached config that tolerate staleness; it's just the wrong tool for a counter that needs read-then-write correctness under concurrent hits.

Problem 3: Next.js metadata's `openGraph` fields don't inherit the way you'd expect

Next.js's metadata API lets you define an openGraph object in a parent layout and have child routes inherit and extend it, which works nicely for things like title and description. What surprised me is that openGraph.images does not merge the way nested metadata fields usually do when a child route defines its own partial openGraph object — if a page-level generateMetadata returns an openGraph key at all, Next.js treats it as a full replacement for the object rather than a shallow merge on top of the layout default, and any field you don't re-specify (including images) is simply gone rather than falling back to the parent's value.

In practice this meant several inner pages exported an openGraph object to set a page-specific title, and as a side effect silently lost the default sharing image the layout had defined, so social previews for those pages showed a blank or default browser card instead of the intended graphic. The fix was to stop treating the layout's openGraph as something child routes could partially override, and instead centralize a buildOpenGraph(overrides) helper that always spreads the full default object first and applies only the specific keys a page needed to change. That guarantees images and other shared fields survive even when a page customizes just the title.

Net take

None of these are OpenNext bugs exactly — they're gaps between how the underlying Cloudflare primitives behave and what you'd naively assume coming from a more traditional Node/Vercel deployment model. D1's migration ledger, KV's consistency model, and Next.js's metadata merging rules are all documented, just not in a way that surfaces itself until you've already hit the edge case. Running on Workers with OpenNext has been solid otherwise — cold starts are noticeably better than serverless functions I've run elsewhere, and having D1, KV, and Durable Objects available in the same request without extra network hops made a few things simpler than they would have been split across separate services.

Migrating 10 WordPress Sites to Cloudflare Pages: What Broke

Mashi Mashi — Mon, 03 Aug 2026 00:23:58 +0000

A few months ago I moved a batch of WordPress sites off a shared LAMP host and onto Cloudflare Pages as static exports. The pitch is obvious: no PHP process to keep patched, no MySQL to babysit, effectively free hosting, and a CDN in front of everything by default. What the pitch doesn't tell you is how many small, boring things break on the way there. This post is a rundown of what actually went wrong migrating a set of ten WordPress sites — one of them is burningtribe.tokyo, which I'll use as the concrete example — and how I fixed each issue.

The approach

The migration itself is conceptually simple: crawl the live WordPress site, save every URL as a static HTML file plus its assets, and serve that tree from Cloudflare Pages. I used a combination of wget --mirror and a custom crawler for a couple of sites where wget choked on query-string-based pagination. The static output then gets pushed with wrangler pages deploy. No build step, no framework, just files.

That simplicity is exactly why it seemed low-risk. It was not.

Problem 1: relative canonical tags pointed everything at the homepage

The first thing I noticed after deploying was that Google Search Console started reporting most inner pages as "duplicate, Google chose different canonical" — and the canonical it picked was the homepage. The cause was almost funny once I found it: the WordPress theme emitted <link rel="canonical" href="/"> as a relative path in a few cached page fragments, instead of an absolute URL like https://burningtribe.tokyo/some-post/. On the original WordPress install this didn't matter because the page itself resolved the relative reference correctly at the point of caching. Once the HTML was frozen and served statically from a different origin structure (Pages serves everything from the apex), that relative canonical collapsed to the site root for every single page that had it.

The fix was a straightforward but tedious pass: grep every exported HTML file for rel="canonical", and rewrite any relative or root-relative href into a fully qualified absolute URL matching that page's own path. I wrote a small script that walked the export directory, parsed the canonical tag, and replaced it based on the file's own location relative to the site root. Running that across all ten sites turned up the same bug in three of them — it wasn't a burningtribe.tokyo-specific quirk, it was a plugin combination that a few sites shared.

Problem 2: no sitemap.xml survived the export

WordPress sitemap plugins (I was using a common SEO plugin's built-in XML sitemap) generate their sitemap dynamically from a PHP endpoint, not as a static file anywhere in the theme or uploads directory. A mirror crawl that only follows <a href> links will never discover /sitemap.xml unless it's explicitly linked from a page, which it usually isn't — it's referenced from robots.txt and submitted directly to Search Console instead.

The result: after cutover, sitemap.xml on the new static site was either missing entirely or, worse, still referencing the old dynamic URLs that no longer existed as generated endpoints. For a couple of sites this went unnoticed for almost two weeks because the sites still ranked on already-indexed pages; the failure mode is silent. I ended up regenerating sitemaps after the fact by walking the exported file tree directly (every .html file becomes a <url> entry) rather than trying to preserve the plugin's output, since the plugin's output no longer had anything running to produce it. That also meant re-submitting fresh sitemaps in Search Console for each property, which is worth doing explicitly rather than assuming the crawler will pick up the change organically.

Problem 3: enqueued JS and CSS 404s

WordPress uses wp_enqueue_script and wp_enqueue_style to load most theme and plugin assets, and those functions frequently append a version query string, like theme.js?ver=6.4.2. A naive mirror tool that treats theme.js and theme.js?ver=6.4.2 as different resources will save one and reference the other, or save the query-string version and leave the extensionless base path unresolved depending on crawl order. I saw a batch of these appear as 404s in the Pages deployment logs for image lazy-load scripts and one analytics snippet, which quietly broke lazy-loaded images on a couple of pages until I caught it in a manual QA pass.

The fix was to strip version query strings during the export step and de-duplicate on the base filename, then verify with a link checker (linkchecker worked fine for this) run against the deployed Pages URL rather than the local export — some of these only showed up once served over HTTPS with Cloudflare's own caching layer in front, because local file serving during testing was more forgiving about missing query strings than the production edge was.

Problem 4: internal redirects had nowhere to live

The original WordPress sites had a handful of accumulated 301 redirects — old permalink structures, a couple of merged categories, one full domain change years back that was still being redirected from an old slug. Those redirects lived in a redirection plugin's database table, which obviously doesn't come along in a static export at all.

Cloudflare Pages supports a _redirects file for exactly this case, and that's where these rules ended up. The process was: export the plugin's redirect rules (most redirect plugins have a CSV or JSON export), then transform that list into the _redirects file format (/old-path /new-path 301) with a short script, and commit that file at the root of the Pages project. It's a five-minute fix once you know it's needed, but it's easy to forget entirely if you're focused on "does the site render" rather than "does the old URL still resolve," and a couple of these old redirects were still receiving meaningful traffic from long-lived backlinks.

What I'd do differently

If I were starting this migration over, I'd build the sitemap and _redirects file generation into the export pipeline from the start rather than patching them in reactively after Search Console complained. I'd also run the canonical-tag check as an automated assertion in CI before any deploy, since it's cheap to check and expensive to have silently wrong for weeks. None of these problems are exotic — they're all a version of the same root cause: WordPress plugins generate output dynamically at request time, and a static export only captures what happened to be requested during the crawl, not the underlying logic that produced it. Once you internalize that, most of the migration checklist writes itself.

The sites are stable now and load meaningfully faster than they did on the old host, so the migration was worth it. But "just export it as static files" undersells how much of WordPress's behavior is implicit and request-time-dependent rather than baked into the files themselves.

What I Learned Shipping 90+ Mobile Apps with AI Coding Agents

Mashi Mashi — Sun, 02 Aug 2026 21:40:35 +0000

Over the last year I've shipped more than ninety mobile apps to the App Store and Play Console, almost all of them built with heavy help from AI coding agents (Claude Code, mostly, with a mix of other tools along the way). That number sounds absurd until you understand what most of these apps actually are: small, focused utilities and niche games built by a team of one, not ninety separate startups with ninety roadmaps.

I want to write down what actually changed in my workflow, because most of the "AI coding agent" content I read online is either breathless hype or reflexive skepticism. The reality, at least for me, was messier and more useful than either extreme.

The unlock wasn't code generation

The first thing people assume is that AI agents are valuable because they type code faster than you. That's true, but it's not the interesting part. The interesting part is that a coding agent can hold an entire small app's context in working memory and act on it consistently — rename a variable across twelve files, update a schema and every place that touches it, or run a build, read the error, and fix it without you copy-pasting a stack trace into a chat window.

Once I stopped treating the agent as an autocomplete engine and started treating it as a junior engineer who never gets tired of grunt work, the portfolio approach became possible. A single person cannot manually maintain ninety codebases. A single person plus an agent that can be pointed at a repo and told "the build is broken, fix it" can.

Store submissions are where the real complexity lives

Writing the app was never the bottleneck. App Store Connect and Google Play Console submissions were. Both platforms have dozens of small, easy-to-miss requirements that have nothing to do with whether your code works:

A subscription app needs an EULA link in its description, or it gets auto-rejected by a script before a human ever looks at it.
Free trials that are only configured for some territories will pass review in one region and throw a purchase error in another.
Deleting an app record on App Store Connect doesn't free up the bundle ID — it's reserved permanently, which matters a lot when you're iterating fast.
Google Play's classifier will flag things like "reads Android system logs" language even when the underlying permission usage is completely benign, and the appeal path is not obvious.

None of this shows up in a tutorial about building your first app. It only shows up after you've been rejected a few dozen times and started keeping a checklist. I eventually had the agent maintain that checklist as a literal pre-submission gate script, because I stopped trusting myself to remember eighteen rules across ninety apps.

Agents are excellent at finding the boring bug you'd never look for

The bugs that actually hurt were never the ones in business logic. They were things like: a retired ad SDK dependency left behind after a plugin was removed, causing a crash only on cold start on specific Android versions. Or a locale check that worked on every simulator I tested but returned the wrong result on real Japanese devices because the JS engine's Intl implementation doesn't localize the way you'd expect.

What made agents genuinely useful here wasn't cleverness — it was persistence and breadth. I could ask an agent to scan all ninety repos for a specific anti-pattern (an unreferenced dependency, a hardcoded staging URL, a duplicate icon mismatch between the launcher and the store listing) and get a real answer in minutes instead of spending a weekend grepping by hand. The value compounds specifically because the portfolio is large — the same technique on three apps would just be "using find-and-replace with extra steps."

Where I still don't trust the agent alone

I want to be honest about the limits, because overclaiming here helps no one:

Anything involving real money — in-app purchase configuration, subscription pricing, Stripe webhooks — gets a manual review pass every time. Agents are good at pattern-matching prior mistakes, not at catching a wrong price point unless you tell them exactly what to check.
Destructive operations (force pushes, database migrations, deleting review-blocking records) are never delegated fully. I treat "the agent proposes, I approve" as a hard rule for anything irreversible.
Long-running or flaky external processes — a build that intermittently fails because of a shared temp directory, or a CDN cache that lies about propagation — still need a human to notice the pattern isn't random.

The honest summary is that AI coding agents didn't replace the parts of shipping software that require judgment. They replaced the parts that required patience: reading every line of a hundred-line diff, remembering which of ninety apps still uses a deprecated permission, re-running the same fifteen-step submission checklist without skipping step eleven because you're tired.

What actually made this sustainable

Two habits mattered more than any specific tool:

Writing down failure modes as I hit them, in plain files the agent could read back later, rather than trusting memory. The second time a submission failed for the same reason, that was a process bug, not a one-off.
Treating verification as non-negotiable. An agent reporting "build succeeded" is not proof of anything except that a compiler didn't complain. I learned to require an actual artifact check — did the binary get uploaded, does the store listing reflect it, does the app open — before calling anything done.

If you're exploring this space and want a more structured way to actually practice building with an AI coding agent rather than just reading about it, I've been collecting what I've learned into a small practical resource at AI Craft Campus — it's aimed at people who want to go from "I asked an AI to write some code" to "I run a small, working software portfolio with an AI agent doing the repetitive parts." It's a work in progress, same as the rest of this.

Ninety apps is not a finish line, it's just where the counter happens to be right now. The workflow that got me here — small repos, aggressive automation of the boring parts, and a healthy distrust of "it built successfully" as a definition of done — is the part I'd actually recommend adopting, independent of which specific tool you use.