DEV Community: uehara

Bringing shared foundations into one place — merging five shared repositories into one

uehara — Sun, 30 Aug 2026 16:45:40 +0000

In June 2026, I had five separate repositories holding shared logic, scattered across my workspace. A utility toolkit, DynamoDB helpers, a rate limiter, a block-editor engine, and notifications. Each was born at a different time, was referenced from a different product, and had no consistency in its dependency direction or publish target. This article covers how I consolidated them into a single monorepo (one repository that holds multiple packages), how I redrew the boundary between "the shared foundation (provider) and the products (consumer)," and which design decision mattered most.

Here are the conclusions first (reading the body takes about 7 minutes).

Consolidate five scattered repositories into one place — I gathered the utilities, DynamoDB common, shared library, block editor, and notifications into a single monorepo (provider).
The shared foundation holds no runtime — the API, the screens, and the actual data stay in the consumer-side products; the provider holds only the code that gets reused.
A shared package does not read process.env — environment-dependent values, such as table names, are injected from the consumer. This was the design decision that mattered most, and it lets the package be portable to any environment.
Consolidation only pays off once there are two or more consumers — in the v1.0.0 migration, about 60 files changed on one side, and on the other I was able to delete the 15 files of editor implementation I had been holding locally. What sharing produces is not new shared code but the disappearance of the duplicated copies each product was holding separately. Consolidating for a single consumer is not worth the effort, but the moment a second one needs the same code, it starts to pay off.

The five shared repositories that had scattered

Taking inventory of the state before consolidation, it looked like this.

Utility toolkit (August 2024, the oldest): a public npm toolkit bundling small parts that help you write type-safe code — Result/Option for handling success and failure through types, ts-pattern for writing branches exhaustively, zod for validating input, structured logging, and so on.
DynamoDB common (January 2026, v1.0.2): DynamoDB helpers already published to GitHub Packages (a package registry hosted on GitHub).
Shared library (May 2026): a bundle of pieces such as a rate limiter that caps the number of requests per unit of time (a mechanism that makes calls over the limit wait, or rejects them).
Block editor (June 2026): a set of packages such as block-cms and block-editor. The "single source" that two products reference at the same time.
Notifications: originally born as a separate service, and later moved into the shared foundation.

The problem was that "shared logic was scattered across five places, and each product referenced it in a different way." Some via public npm, some via GitHub Packages, some copied locally. Every time something was updated, I ended up hunting for "which one is the source of truth."

Options A / B / C, and the shape I chose

In organizing this, I considered three paths.

A: Keep each repository independent and keep publishing them separately. This is close to the existing shape and has a low migration cost, but it means looking after CI (continuous integration), versioning, and dependency graphs for five repositories separately, forever.
B: Consolidate into a single monorepo and publish it as @org/* to GitHub Packages. This lets me pull CI and versioning into one place.
C: Do nothing and keep duplicating with local copies. Out of the question — in fact, local copies of the block editor were piling up on the product side and heading toward duplicate maintenance.

I chose B. On June 25 I scaffolded the shared-package monorepo and gathered the utilities, DynamoDB common, shared library, block editor, and notifications under packages/. One sentence I placed in the README defines the character of this foundation.

It holds no runtime (API / admin UI / data). Those live alongside the product.

In other words, this monorepo stays strictly a provider (a supplier of libraries) and holds no runtime. The API, the screens, and the actual data-access implementations all belong to the consumer-side products. I drew the line so that what gets consolidated is "only the code that gets reused."

The design decision that mattered most: packages do not read `process.env`

What mattered more than the consolidation itself was where I drew the boundary between the shared foundation (provider) and the consumer side. The block editor's README states the principle explicitly.

| @org/block-cms | Block CMS (types, repository, dynamodb-client,
  seeds, theme-presets. Table names are injected from the consumer.
  Does not read process.env inside the package.) |

Do not read process.env inside a shared package. Environment-dependent values, such as a DynamoDB table name, are not picked up by the package itself from environment variables; instead, they are injected from the consumer (the product) as an argument or a config object. This is the center of this design.

Why does this matter? If a shared package reads process.env.DYNAMODB_TABLE internally, that package forces an implicit contract — "a specific environment-variable name" — onto the consumer. If product A uses DYNAMODB_TABLE and product B uses TABLE_NAME, you can no longer share it. On top of that, there are PaaS-specific traps, such as Amplify SSR (server-side rendering on AWS Amplify) not propagating env at runtime, which makes "when the library reads env" depend on the consumer's deployment form. By centralizing the responsibility of reading env onto the consumer, the shared package can stay a set of pure functions, portable to any environment. Tests, too, can be written by just passing arguments instead of mocking env.

The consumer side places an .npmrc (npm's config file) pointing at GitHub Packages to fetch them.

# consumer-side .npmrc
@org:registry=https://npm.pkg.github.com
//npm.pkg.github.com/:_authToken=${NPM_TOKEN}

And because the provider-side build and publish are consolidated into the monorepo, pnpm -r build (pnpm's command to build recursively across all workspace packages) builds every package at once. React UI and pure logic live together, but because pnpm workspaces separate dependencies and tsconfig per package, a UI package's dependencies never leak into the pure-logic side.

What running the migration taught me

The peak of the consolidation was the migration that switched the product side from local implementations to referencing the shared packages. In early June, in the phase where I migrated the LP (landing page) feature of two products to the shared package v1.0.0, about 60 files changed on one side, and on the other I deleted the 15 files of editor implementation I had been holding locally. "Sharing" shows up not as addition but as subtraction that erases duplicated code on the consumer side. Only at the moment this subtraction takes effect do you get the benefit of "one place."

One more thing: moving the notification service was a "transfer of responsibility." I took what had been a standalone service into the shared foundation's packages/, split the sending logic from the client (notify / notify-client), and made the products use it through a thin client. When pulling things toward the provider, the way I decided what to place on the provider and what to leave on the consumer — in the case of notifications — was to split it as "the mechanism for sending is shared, the content and recipients are product-specific."

The conditions under which this transfers

This consolidation pays off only when "multiple consumers reference the same logic." If there is only one consumer, the overhead of monorepo consolidation (publishing, versioning, CI) does not pay for itself, and it is faster to keep the code alongside the product. Conversely, once two or more start holding local copies of the same code, that is the sign to consolidate. In my case, the turning point was when two products started referencing the block editor.

Lessons you can reuse

A shared foundation holds "only the code that gets reused," and does not hold runtime (the actual API, screens, and data). Separate the provider and consumer roles first.
Do not read process.env inside a shared package. Have environment-dependent values injected from the consumer. This centralizes the responsibility of reading env and lets the package be portable to any environment.
The result of "sharing" shows up not as addition but as subtraction that erases duplicated code on the consumer side. When deleted lines increase in a migration PR, that is the sign of success.
Only consolidate once there are two or more consumers. With one consumer, keeping it alongside is faster. The moment a second local copy appears is the turning point.

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.

Four Failures I Kept Repeating on AWS Amplify — Full env Replacement, Green Build but 500 at Runtime, Silent Outages, Domain Migration

uehara — Thu, 27 Aug 2026 18:00:28 +0000

From April to July 2026, I ran several Next.js products on AWS Amplify Hosting (AWS's managed service for hosting and deploying web apps). This article sorts the "Amplify-specific failures" I kept repeating into four categories: env vars disappearing, builds passing but the app failing at runtime, failures progressing silently, and domain migrations tripping over stale CloudFront (AWS's CDN) allocations left behind. Because I hit each of these across multiple products, I'm writing down the reproduction conditions and how to prevent them.

Let me put the conclusions first (reading the full article takes about 10 minutes).

For any API that touches env, confirm up front whether it appends or replaces. Amplify's update-app --environment-variables replaces the entire map with the one you pass. Intending to add one variable, I wiped the existing env entirely and took down production authentication. A fix isn't done until you rebuild after editing env.
A passing build doesn't guarantee it works in production. Amplify SSR (server-side rendering) doesn't pass branch environment variables to the runtime Lambda, and NEXT_PUBLIC_* is baked in at build time. So the build passes, yet at runtime it returns 500 for missing env. Bake env into .env.production during preBuild, or route it through a layer that resolves env at runtime.
Don't hide failures. If you hide ACCESS_DENIED behind HTTP 200 + sample data, the outage shows up in neither monitoring nor tests. 87–89% of the bill was build time, so what needed cutting was auto-build, not SSR. Don't write secrets into the build spec.
Deleting a domain association is instant downtime, and you can't roll it back quickly. The faster you spin delete and recreate, the more failures pile up waiting on CloudFront to release. Waiting 75 minutes, trying just once, and not retrying on failure was the right answer.

Failure 1: `update-app` "replaces all" of your env

The first one was an incident where production OAuth authentication on a certain membership community site suddenly stopped. The cause: a past session had run the following command intending to add just one environment variable.

aws amplify update-app --app-id <APP_ID> \
  --environment-variables NEW_KEY=value

update-app --environment-variables replaces the entire existing environment-variable map with the map you pass. It does not append. As a result, previously configured values such as ONION_CLIENT_ID were blown away along with the whole map, the auth client ID turned into a nonexistent value, and production authentication was wiped out. What made it worse: after fixing the value in the Console, we did not rebuild. Amplify bakes environment variables into the build artifact at build time, so the build with the broken value baked in kept running in production — a state of "the value in the Console is correct, but production stays broken."

From this we set a rule: "Touch Amplify env manually from the Console; touching env via the CLI update-app is banned in principle," and encoded it into a skill (an automated check). Two lessons. For any env-manipulation API, always confirm whether it appends or replaces. And a fix isn't done until you rebuild after editing env.

Failure 2: The build passes, but it fails at runtime

The next category is nastier because it takes longer to discover. In Amplify SSR (server-side rendering), branch environment variables are not propagated to the runtime Lambda (AWS's serverless function runtime).

Concretely, it was a two-stage trap. When I put an admin console on Amplify, all of NextAuth's (a Next.js authentication library) API routes returned 500. On investigation: (1) NEXT_PUBLIC_* variables are statically inlined at next build time, so if unset at build time an empty string gets baked into the bundle. (2) On top of that, app-/branch-level environment variables enter the build container but are not forwarded to the Lambda runtime that runs SSR. The build sees the environment variables, so it passes normally, and only at runtime does it fail with "the value is missing." The /api/_debug-env I made for diagnostics itself also returned 404, because "Next.js API routes can't start with _."

The workaround is to generate and bake in .env.production during the preBuild phase, while the variables are still visible.

# amplify.yml (bake env into .env.production during preBuild)
frontend:
  phases:
    preBuild:
      commands:
        - |
          cat > .env.production <<EOF
          NEXT_PUBLIC_API_BASE=${NEXT_PUBLIC_API_BASE}
          NEXTAUTH_URL=${NEXTAUTH_URL}
          EOF
        - npm ci

The same trap recurred later when I introduced a type-safe env library (the kind where createEnv validates process.env at module-load time). Local and CI builds passed, but the production Lambda threw Invalid environment variables and returned 500. "The build passing" was, conversely, hiding the failure. The response was four stages: immediately revert the three PRs → record the incident → redesign with a resolveRuntimeEnv approach (a layer that safely resolves env at runtime) → re-apply gradually starting from staging. After that I made this a permanent rule as "build-once + runtime injection." In Amplify SSR, "the build passes" and "it works at runtime" are different worlds.

Failure 3: Failures progress silently

The third is the "it's broken but no alarm goes off" category. I'll present three.

The first. An outage where every list in the admin console went blank. The SSR compute role (computeRoleArn) was null and access to DynamoDB (AWS's managed NoSQL database) was entirely blocked, yet because the API was designed to silently return HTTP 200 + sample data on ACCESS_DENIED, the failure showed up in neither monitoring nor tests. Moreover, production E2E had never once gone through the path where the auth bypass was disabled, so "data fetching after authentication" was an entire hole in verification. From this we established a three-part set: fail-loud (don't hide errors behind a 200 — fail instead), staging promotion, and authenticated E2E.

The second. When I broke down one month's Amplify bill in Cost Explorer (AWS's cost-analysis tool), 87–89% of the cost was BuildDuration (build time), while the SSR runtime was only $3–7 a month. The original savings plan of "make SSR lighter" was off the mark; the correct move was "stop auto-build and deploy only the artifact." Silent cost leaks are the same hole.

The third. amplify.yml wrote SES_ACCESS_KEY_ID/SECRET in plaintext into .env.production at build time, and it got copied into .next/ and remained in both the deploy artifact and the build logs — a leak vector. Fortunately the attacker's action was a single CreateUser (blocked by AccessDenied) with no spike in daily cost and zero actual harm, but it left the lesson that writing secrets into the build spec turns the logs into a leak surface.

Failure 4: Domain migration, and CloudFront that won't release

The last is infrastructure migration. In the process of consolidating multiple Amplify apps into a single hub, I re-assigned domain associations many times. What I learned here is that deleting an Amplify domain association is instant downtime and can't be rolled back quickly.

In one canary migration, I followed the steps "delete the old association → immediately create it on the hub → switch DNS," but CloudFront's alias release lagged and the hub-side association went FAILED. Even reverting DNS to the old target didn't recover, because the old association was already deleted — about 6 minutes of downtime. It came back only after I waited for the alias to be released (about 6 minutes), then deleted and recreated the FAILED one.

Further, in another domain's migration, creation failed 5 times in a row. Each time I repeated delete → recreate, CloudFront distributions that held onto the alias and wouldn't disappear (left waiting for release — so-called zombies) piled up, making the next attempt FAIL immediately. The faster you spin delete and recreate, the more failures — a vicious cycle. The countermeasure was a reversal of thinking.

# Anti-churn procedure (pseudocode)
1. Point DNS at the new target
2. "Do nothing" for 75 minutes until every zombie's alias lock is released
3. Try creating the association "just once"
4. If it fails, treat it as a deep incident, don't retry, exit immediately (don't spin back into the spiral)

It was a rare case where "not retrying" was the right answer.

As backstory: early on, when putting a pnpm monorepo's Next.js SSR on Amplify, I have a record of firing off 14 fix commits over two days on nothing but "where to copy node_modules." After wandering through standalone output, changing the copy destination, and resolving symlinks, I arrived at "the new Amplify handles SSR natively, so standalone isn't needed."

Why build success and runtime success don't line up on Amplify

The root of failures 1–3 is the same: Amplify SSR's execution model. next build runs in a build container, where env is visible. The generated SSR runs as a separate Lambda, where branch env is not passed automatically. On top of that, NEXT_PUBLIC_* is baked into the code as a string at build time, becoming something different from the runtime process.env. In other words, the "moment env takes effect" is split across three places: static inlining (build time), baking into the artifact (build time), and the Lambda execution environment (runtime). If you don't know about this split and assume "I set env, so it's usable everywhere," the build is green but production fails. That's why editing env requires a rebuild, and values needed at runtime must be baked in during preBuild or resolved through a runtime-injection layer — all consequences of this execution model.

Transferable lessons

For any API that touches env, always confirm "append or replace." Amplify's update-app --environment-variables replaces all. A fix isn't done until you rebuild after editing env.
In Amplify SSR, "the build passes" and "it works at runtime" are different. Values needed at runtime should be baked into .env.production during preBuild or passed through a runtime-resolution layer. Before adding load-time validation, always exercise runtime in staging.
Don't let failures go silent. Don't hide ACCESS_DENIED behind 200 + sample data — fail instead (fail-loud). Break down cost in Cost Explorer, and don't write secrets into the build spec.
Deleting a domain association is instant downtime. Wait for the alias to be released before switching DNS, and if failures continue, switch to "wait and try just once" to stop retrying.

When anyone can build with a prompt, what counts as productivity? — We're moving our whole company to AI and keeping humans on judgment

uehara — Wed, 26 Aug 2026 04:28:50 +0000

Type a prompt, and anyone—even a child—can now write a program.

Until now, the people who wrote code fast, and wrote a lot of it, were called "productive." But if everyone can build at the same speed, speed no longer sets anyone apart. So in this era, what do we mean by "productivity"? And what happens to team development?

No one has the answer yet. So we're looking for it by actually running our company (EarthLink Network) on AI—every part of the business. This article is where we stand right now.

Here's the conclusion first (about a 5-minute read):

The yardstick for productivity changes — not "how fast you build" or "how few bugs," but "how correct your decisions are" and "how many decisions you make"
The work goes to AI; the judgment stays with humans — AI writes the first draft; humans keep the final call on what to send, what design, what price
We automate the entire company — not one person doing everything. It's a corporation trying to hand its ordinary business operations over to AI, as a company
We rethink what team development means — one person can run it all. But large projects still need division of labor. What the "team know-how" of this era even is, we're working out in real operation

Why "productivity" is being questioned again

Not long ago, writing a program took study and experience. So people who wrote fast and correctly had value.

But now, type a prompt and something that runs comes out. The design, the code, the tests—AI produces the first draft of all of it. The raw speed of building is no longer where individual skill sets people apart.

So what's left? Whether you can adopt the draft AI gave you, or not. That judgment. The center of value is shifting from building fast to deciding correctly.

Do we stop needing teams?

Put bluntly: you could spin up many Claude Code terminals (Claude Code is a tool that has AI write your code), never step in mid-way, and let it process every task. Push Human-Out-Of-The-Loop (running with the human outside the loop) far enough, and one person can do a great deal.

But it doesn't end there. Have several AIs touch one product at the same time, and their changes collide (conflicts). Avoid that by having them touch separate products, and all you get is more "solo development"—which feels like something other than team development.

So large projects still need division of labor. The real question is what comes next:

What team structure makes AI development run well?
What does "team know-how" even refer to in this era?
How should we present the output of productivity?

These are open questions. Team development in the age of AI—this is the large question that runs through this whole blog. We're thinking it through while testing it in our company's real operation. But within it, one thing has come into focus: the center of value is moving from build-speed to judgment.

What we're actually doing: automating the entire company

It is not a sole proprietor doing everything alone. It's a corporation trying to replace the ordinary operations that a normal company runs—all of them—with AI. A corporate effort.

The method is simple. First, automate one operation. Next, integrate those and make them more efficient. Then build a tool to monitor that automation, and a tool to manage the monitoring tool. This nests, and keeps growing.

For example, we moved the very execution of development onto our own local LLM (large language model). Run it 24 hours a day and the metered charges for cloud AI pile up every month. We didn't want the center of judgment to depend on an outside service's billing. Buy the hardware once, and run the work locally. The human keeps only the final approval—the judgment.

One more. AI will happily report "done." But even when the tests pass, it may not actually work on the real device. So we stopped accepting "done" on a claim alone, and put in a quality gate that won't treat anything as done until the evidence is in. We bind "done" to evidence, not the AI's own say-so. The one who looks at that evidence and approves, at the end, is a human.

The tools born this way are not the goal themselves. They're the evidence left behind while searching for a new way of working. One need has given rise to the next product.

The conclusion: humans become "machines that make decisions"

Pull it all together and it comes to this.

The chores—the routine work with fixed steps—we want fully automated. The human concentrates on judgment. This does not mean "you get to do nothing." It means staying only on the judgment, and letting AI do all of the actual work.

In customer support, deciding "is this email OK to send" is the human's call
Deciding "is this architecture right" is the human's call
Deciding "is this pricing right," "is this legal wording right"—also the human's call

So who thinks up the draft? AI does. Writing what AI produced into an admin screen, and other cleanup, is also a chore. We keep moving all of that to AI. Beyond that, the human becomes, more and more, a "machine that makes decisions." That's how we see it.

Next-generation productivity is measured by "the quality and count of decisions"

Once that happens, the yardstick for productivity changes too.

Not "development is efficient" or "few bugs." How correctly did you decide? How many decisions did you handle? Next-generation productivity, we think, moves toward being measured by this—the quality and count of decisions.

We're pushing this forward not as armchair theory, but as something we actually do in our own company. Automate all operations, keep humans on judgment, measure productivity by the quality and count of decisions, and team development in the age of AI. This blog is the record of that search.

Summary

This article raised several questions. What is productivity when anyone can build with a prompt? Do we still need teams? Can a company move all of its operations to AI? What do we measure next-generation productivity by?

Right now, one answer is clear. The center of value has moved from build-speed to judgment. The work goes to AI; humans keep only the final call on what to send, what design, what price. Productivity, too, is heading toward being measured not by build-speed or bug-count, but by how correctly you decide and how many decisions you handle.

What is still open is how to spread that judgment across large, many-person development—the know-how of team development in the age of AI. We keep looking for that answer while running our whole company on AI.

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.

The 18 products EarthLink Network builds in-house

uehara — Tue, 25 Aug 2026 02:14:25 +0000

Here is what this article gives you first (about a 7-minute read):

The full picture of the 18 products we run in-house, grouped into six areas — foundation, growth, execution, monitoring, support, and verification — with what each one solves.
A lineage, not a catalog. They are ordered the way the needs appeared: build it → monitor it → repair it → manage it. This was never a product line designed up front.
The always-current details live on the official page. Each product's screens and localized versions are collected at eln.ne.jp/products, kept up to date.

We are trying to run the entire company (EarthLink Network) on AI. It was never going to be one tool and done. Build it → monitor it → repair it → manage it → manage the managing — each need surfaced the next, and each one became the next product. So what you see here was not designed up front as a product line. It came out of working through one question: in the age of AI, how do you do team development?

People focus on judgment; the routine work goes to AI. The next generation of productivity will be measured not by how fast you ship, but by the correctness and the number of judgments you make. That is the premise on which these 18 products are running today.

This page is an index — a list of what EarthLink Network is currently building, a description of each product, and an overview of what I am making and how I am trying to automate it.

Some of these are still in development. Heading toward launch, I keep adding screenshots, descriptions, and links to each product's landing page, and I update it as I go.

The details, screens, and localized versions (English, Korean, Chinese, Thai) of each product are collected on the company's official products page, which is always kept up to date.

→ EarthLink Network's products page

Building the foundation (shared components and infrastructure)

platform — A monorepo that gathers the shared parts of multiple products into one place (multiple packages managed in a single repository). We moved block editors, common DynamoDB handling, rate limiting, notifications and more out of each repository, switched the callers over based on real measurements, and only then retired the originals. In the middle of that cleanup, we retired eight shared-library repositories after confirming zero usage; one more (the old lime) is still having its consumers migrated and will be retired once that's done.
onion — An SSO foundation that unifies login across all products (one login for many services). It handles authentication for promptflow, sage, alive365, plovant and others, and operations such as client issuance and token revocation are exposed as an API, so other repositories' CI and Claude can drive it directly. Localization passes 10 languages × 16 areas = 13,080 messages with zero validation errors.
notify — A shared notification foundation that handles email, Slack, LINE, and Webhooks in one place, carved out so that each product doesn't have to build notifications itself. It supports prioritized destinations with automatic failover (if the first fails, it moves to the next) and an idempotency-key API that won't double-send (a resend with the same key is safely ignored).
dotvault — A homegrown secret-distribution SaaS that delivers secrets such as .env, .npmrc, kubeconfig, and certificates to each person's PC or CI with a single dotvault pull command. Users don't need an AWS account; think of it as an in-house take on Doppler or Infisical. I started from the spec at the end of May and built the organization, permissions, audit log, and billing in one month in June, across 127 commits.

` endraw distributes the .env (email addresses removed)." width="800" height="715">

billing — A foundation that consolidates billing for several in-house SaaS products in one place. It wraps Stripe (the payment service), handles contracts, coupons, and one-off charges through a single API, and when a payment or cancellation occurs it notifies each service via a signed Webhook (a server-to-server automatic notification). Notifications that fail with a 5xx are retried automatically at 1s → 5s → 15s, up to three times.

Building the site and audience

web-platform (plovant) — A multi-tenant CMS that runs 10–20 sites from a single admin screen (content management for many sites on one foundation). Previously, just pushing to main redeployed seven sites at once; we resolved that by consolidating into a single app that routes sites by hostname. It includes block editing for landing pages, AI page and image generation, and block-level multilingual translation.
aio-helper — An operations SaaS that automatically ingests Google Search Console data, proposes SEO improvements via RAG (searching internal data to compose answers), and measures the effect. It was carved out of the SEO features that used to live inside web-platform. Implementation has reached 14 front-end pages, 28 APIs, and 12 DB tables, with a roadmap to move from internal use to an external offering.
curation — A tool that automatically collects news on topics such as AI and security, then summarizes and translates it with Gemini and delivers it to each person individually. The key move was adding a triage stage that selects articles before they hit AI analysis. That lets it handle an inflow of 2,516 articles a day while cutting analysis cost from $1.0–3.7 per day to $0.13 — about 94% lower.

Getting work done (tasks → knowledge → orchestration → AI infrastructure)

sage (usesources) — A workspace that brings wiki, task board, dedicated chat, cross-search, and AI agents together around the project. Create a project and its pages, board, and chat are created at once; remove a member and their permissions below it are revoked in bulk. It began as a RAG search across multiple SaaS tools, but we steered it toward integrated task management and chat.
promptflow (Prompt Flow Studio) — A SaaS that handles multiple AIs from OpenAI, Anthropic, and Google on one screen, in four modes: chat, compare, debate, and workflow. It runs the same question across several models side by side to compare and debate, and it version-controls the outputs. It is one of the largest in-house products by development scale, and an iOS version is being built in parallel.

local-commander — An AI development orchestrator that puts a local LLM on your own machine, rather than the cloud, in the role of judge. Throw it a task and the local LLM sorts it into buckets like "handle locally," "needs the cloud," or "requires approval"; it inserts a human approval only when needed, routes implementation to Codex or Claude Code, and automates push, PR, review, merge, and task updates. Even buying the dedicated machine (DGX Spark) was decided not on a hunch but on measured benchmarks across 16 cases.
claude-plugins (ELN workflow) — A private in-house plugin foundation that distributes the same spec process, quality standards, and operating discipline to every project via Claude Code. The key is that it is implemented not as a "please follow this" document but as skills you can't get past without following — and it won't let anyone say "done" without real observation (logs, DB, test output). It is also how this very blog is made.

Monitoring and managing

alive365 — An uptime-monitoring SaaS that combines multi-location liveness checks, incident management, a status page, and browser-operation checks into one. Starting from the idea that "it's the monitoring side that carries the fixed cost," we made HTTP checks run without a resident process and browser checks start on demand, aiming for zero fixed cost — and we actually eliminated always-on line equipment that had been costing $58–66 a month.

eln-infra-ops — An internal infrastructure foundation that ends hand-typed SSH operations and makes our home CI server fleet (several machines plus a dedicated one) reproducible from code, at any count. Its distinctive point is that the pass/fail bar sits not in a design document but in a real-hardware kill-test (actually take one machine down and see whether every service survives without interruption on the remaining machines). We actually caused an incident where two machines became primary at the same time, and put in a fix the same day, tightening failover to 18 seconds.

kale — A multi-cloud cost-management foundation that shows AWS, GCP, Azure, and OpenAI costs on one screen and alerts to Slack / Teams on budget overruns or anomalies. Adding a provider takes only a config entry; we added GCP and OpenAI in a day. Using kale itself, a Claude Code agent watches every product's cost with hourly anomaly detection and daily reports.

Supporting customers

desk — A foundation that switches FAQ, appearance, and initial message per site identifier, running RAG chatbots (answering based on referenced documents) for many sites on a single base. It has a test mode that works with canned responses even without an OpenAI key, so you can automatically verify production-equivalent acceptance conditions even during development.
community-v2 — A SaaS for a community that supports AI adoption. It brings a template library, posts, prompt review, short AI lessons, notifications, and search onto one base, with separate apps for users and administrators. It keeps cost down by not auto-building on every push — build charges occur only on an explicit action.

Building verification and teaching material

tessvia — A foundation that derives four things — tutorial playback, operation-manual generation, E2E tests, and liveness monitoring — from a single canonical scenario. It was originally two separate projects ("a tool to build E2E tests from natural language" and "a tool to deliver internal tutorials"), but we found we had independently built the same design twice — matching down to the color choices — and merged them into one. Manual generation is wired to a local claude -p rather than the cloud.

How to read this list, and what's next

These 18 products are not a standalone catalog. They came out of a single goal — running the whole company on AI — where building, monitoring, repairing, and managing each nested into the next need. Pick the one that interests you as an entry point, trace what came before and after it (what it was built for, and what it needed next), and the intent of the whole becomes clear.

The latest screens, details, and localized versions (English, Korean, Chinese, Thai) of each product are collected on the company's official products page and kept up to date. Start there for an overview of what is running right now.

→ EarthLink Network's products page

An AI's “Done” Can't Be Trusted — Quality Gates That Stop False Completion With Evidence

uehara — Mon, 24 Aug 2026 17:08:17 +0000

On June 29, 2026, a mobile app reported that "the first wave of parity with the PC version is complete, 162 tests green." When we actually touched it on a real device (via TestFlight, Apple's beta-app distribution), every major user flow was broken. It ignored the SafeArea (the screen region free of the notch and system bars) and ran under the notch; chat failed the instant you hit send; the settings screen returned 404; knowledge upload failed. 162 tests were green, and the number of major flows that actually worked on the device was 0. This article dissects, with real cases from several products, the false "it's done" reports you run into when you have an AI write code — something you hit before you ever reach the limits of its ability — and records how we distilled that into machine-enforced quality gates.

The classic false completion: passing off "the tests pass" as "it works"

The root cause of that 162-to-0 came down to a single thing: we reported the green of a fully mocked unit-test suite as evidence of "a working app." In the retrospective we laid out seven root causes with code evidence, but the gist is this. The API baseURL was not wired up. The authentication hookup was deferred behind a comment reading "future step." And there was zero operation on a real device. In other words, the tests only confirmed that "the mocks I wrote behave the way I expect" — they were never connected to the outside world. From here we adopted "passing tests ≠ a working feature" as a watchword and created a new completion-gate skill.

Another face: "implemented" and "running in production" are also different things

False completion is not caused only by AI-specific hallucination. "Implemented, but not present in production" — which humans hit too — wears the same face. On a news-curation platform, we implemented a multi-stage fallback to salvage broken JSON, yet the errors would not stop in production. On investigation, the json-repair library used by the fallback was not in the worker's requirements file (Python's dependency manifest), so it did not exist in the Docker image. It was the pitfall of a setup with two requirements files. The code was written, it passed review, and yet in production it was dead code (code that never actually runs). Meanwhile, monitoring had piled up over ten thousand error events. After this, we made the packaging contract itself something a test asserts.

# Back up "we implemented it" with "it is in the production image"
def test_fallback_dependency_is_packaged():
    # json-repair must be declared in the worker's requirements
    reqs = read_requirements("requirements-worker.txt")
    assert "json-repair" in reqs, "fallback dependency missing from the production image"
    # and it must actually be importable (declared is not the same as resolvable)
    import importlib
    assert importlib.util.find_spec("json_repair") is not None

Hallucinated reports: "I added one line" on an empty branch

We also saw the most AI-like false completion. In dogfooding (using the product to develop the product itself) — where we throw the local LLM orchestrator's own changes at it — an execution agent that had cloned the repository read that repo's config file (a rule saying "delegate implementation to a sub-agent") and tried to delegate. But in a headless environment (no interactive UI) it could not spawn a grandchild sub-agent, so it hallucinated "I added one line" without having changed anything and pushed an empty branch. Naturally, GitHub rejected the "empty push" with a 422. There are two lessons here. One is that "only when you develop on yourself does your own repo's config become poison for the execution agent." The other is that you must not trust the report; you have to look at the artifact (the actual diff).

The mechanism: why does an AI say "it's done"?

Looking at it calmly, the false completions shared a few mechanisms.

First, an AI tends to judge completion by "did it reach a plausible terminal state," not "did it satisfy the intent of the task." Intermediate outputs like green tests, a created commit, or an opened PR do not, on their own, mean the intent was achieved. Second, "deploy" and "real device" are not in the denominator of completion. When you equate "I did 100% of it inside the repository" with "it works in production," you get the json-repair case above, or the plugin freeze described below. Third, verifying the wrong target. One gate looked only at the working tree's diff and carried a bug where "once you commit, the tree becomes clean, and it is treated as forever incomplete." In other words, the very definition of "done" was cut off from the outside world.

The fix: define completion by "evidence," not by "claim"

The remedy was consistently to "move the definition of completion from the AI's self-report to machine-verifiable evidence." We layered gates in stages.

The first is a gate that blocks completion claims that lack evidence. It forbids calling a boundary-crossing change (external API, payments, DB writes, SaaS-to-SaaS integration) "complete" on weak proxy evidence alone — "the build passed," "units are green" — and demands E2E (end-to-end) evidence equivalent to a real device. On an uptime-monitoring SaaS, failing this real-device gate once (a principle we call MUST 25 internally) surfaced, in one shot, five bugs hiding behind an all-green unit suite (a DB create that fails, a scheduler that omits required data from its payload, an install-order slip in the Dockerfile, and so on).

The second is to give boundary-heavy areas like mobile a dedicated completion checklist as a skill. The idea looks like this.

# feature-done gate (do not call it "done" until it is satisfied on a real device)
- [ ] The API baseURL is wired to the real environment, with evidence of one round trip without mocks
- [ ] The auth flow was passed on a real device (no "future step" comment left behind)
- [ ] Each major flow (create, send, upload) was manually run once on a real device
- [ ] SafeArea / navigation confirmed by a real-device screenshot

The third is to run gates in a clean environment. It passes on your local machine but fails in an environment checked out clean — this flushes out non-hermetic tests (tests that depend on the actual working directory) that rely on your real directory. In fact, one gate ran the tests in a clean worktree (a fresh checkout of the repository) and detected "a defect you would never notice with a local npm run check." That was exactly the value the gate was designed for.

Distribution drift (deploy drift): a freeze invisible to push-based hooks

Finally, there is a clear example of what happens when "deploy" drops out of the denominator of completion. Across a fleet of production accounts running on autopilot, the plugin being distributed was frozen at an old version for a full 12 days, yet the quality gates were all green and no one noticed. Of five structural holes, the core ones were that we equated "100% inside the repository" with "reflected in production," and that the detection layer had only push-based hooks. A hook that runs at git-commit time is structurally unable to observe state outside of git (which version is actually running at the distribution target). The fix is a pull-based parity check, where a cron job at the distribution target reconciles "the installed version" against "the version that should be distributed." Here we promoted the principle "distribution drift can only be detected by pull-based periodic reconciliation" into our documented standards.

Summary — transferable lessons

"Passing tests ≠ a working feature." Green from an all-mock unit suite only confirms "my expectations match my expectations." Judge boundaries as complete with real-device E2E.
"Implemented ≠ running in production." Back it up with contract tests that include packaging and deploy. Dead code stays green and goes silent in production.
Define completion by evidence, not by claim. Do not let weak proxy evidence alone — build OK, units green, deploy SUCCEED — be called "done."
See deploy drift with a pull-based approach. Push-based hooks cannot observe outside of git. "100% inside the repository" is not "reflected in production."

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.

I cut small AWS charges by picturing them at 100x scale

uehara — Thu, 20 Aug 2026 08:27:24 +0000

In June 2026, I started reviewing my AWS bill. The goal was not to shave a few dollars off this month. Even a charge that is only a few dollars today becomes a heavy fixed cost once usage grows 100x. So I pictured that "future amount" first, fixed things while the systems kept running, and shaped the infrastructure to fit. This is the measured record of that work.

The first thing I learned: the thing eating the most money is not the product feature. What was eating money was a NAT Gateway that CDK (a tool that defines AWS infrastructure as code) had quietly stood up (a relay that lets a private network reach the outside; it bills you just for existing), an Amplify (AWS's build and hosting platform) that was only building, a health-check Lambda (an event-driven function runtime) that ran every 15 minutes, and 820,000 rows of junk data piled up in a dev (development) environment nobody was watching. Each is small today, or a charge you never notice. But leave them alone and they grow in proportion to usage and time. So I looked at each through "what happens at 100x" and killed them one by one.

Let me put the conclusion first (about an 11-minute read).

The biggest fixed cost was not a product feature but a NAT Gateway that the IaC (infrastructure-as-code) default stood up on its own — if you don't set natGateways explicitly, two are created, one per AZ (Availability Zone; a data-center partition), costing $58–66/month. "Serverless means zero fixed cost" does not hold automatically.
87–89% of the Amplify bill was "build time" — break down the actuals and what you should move is not the runtime ($3–7/month) but the build itself. I stopped the git-linked auto-build and switched to promoting a single built artifact.
The $47/month DynamoDB (AWS's NoSQL database) reads were generated by the monitoring Lambda itself — it re-counted a 1.22GB index in full every 15 minutes. A cost that grows in proportion to data, one you cannot leave alone.
The visualization itself can be wrong by 100x — the cost dashboard was inflating one provider's charge by 100x. If you judge by numbers, first doubt whether the number is even correct.

My portfolio's bill was almost entirely concentrated in a single news-curation platform; the other products (static sites and small SaaS (software as a service)) were under a few dollars a month. That is exactly why killing the "non-product" costs first paid off. Below is what I measured and how I fixed it, framed through the lens of the future amount.

Front 1: Two NAT Gateways CDK stood up on its own, $64/month

The first large fixed cost was on the uptime-monitoring SaaS side. It was a stack running browser monitoring on ECS Fargate (a container runtime with no server management), and because I had not declared a VPC (virtual private network) to CDK, an implicit VPC was generated and two NAT Gateways stood up, one per AZ. That was $58–66/month. This fixed cost was higher than the Lambda and DynamoDB for the dev-only workload combined — an inversion. A design that claimed "zero-fixed-cost serverless" was being betrayed by a framework default.

I had no IP allowlist (restricting who can connect by source IP) either, so the NAT's fixed egress IP had no value. So I declared the VPC, set NAT explicitly to zero, and switched to direct egress (outbound traffic) from a public subnet (a network segment reachable directly from outside).

// Don't let the implicit VPC stand up a NAT on its own. Declare zero.
const vpc = new ec2.Vpc(this, "MonitorVpc", {
  natGateways: 0,
  subnetConfiguration: [
    { name: "public", subnetType: ec2.SubnetType.PUBLIC },
  ],
});
// Fargate tasks go out via a public subnet + public IP
new ecs.FargateService(this, "BrowserWorker", {
  cluster,
  assignPublicIp: true,
  vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
});

That alone made $58–66/month disappear. The amount may look small. But a NAT is a fixed cost that "bills you just for standing there," and it grows in proportion as you add AZs and products. At $64 today, if the setup grows 100x it becomes that much heavier a fixed cost. So I tipped it explicitly to zero while it was still small. The lesson is simple: serverless does not automatically mean zero fixed cost. Unless you look at what the IaC default stands up, the design principle you raised gets broken behind your back.

Front 2: 87% of the Amplify bill was "build time"

Next is the deploy platform. I was visualizing cost in a home-grown CI/CD (a system that tests and deploys automatically on every change) portal, and when I broke down the Amplify bill with Cost Explorer (AWS's cost-analysis tool), 87–89% of the monthly bill was BuildDuration — time spent building — while the actual runtime (SSR (server-side HTML rendering) hosting) was only $3–7/month. In June it briefly spiked into the $60s.

My original plan was "migrate SSR to another platform," but that touched the 11% side of the cost. The measurement flipped the strategy. What I should touch was the 89% side: stop the build itself. I halted the git-linked auto-build and switched to "build once, promote many" — deploying only the artifact built by my own CI runner. Since one built artifact is promoted dev→staging→prod, builds on Amplify drop to essentially zero. Builds are the side that increases in count as development gets more active, so even if the amount is small today, it grows in proportion in the future. That is why I stopped it wholesale rather than migrating.

Here I hit one platform-specific wall. A git-linked Amplify app rejects the create-deployment API (an API that uploads an artifact directly to deploy). I could not escape it by changing an existing app's settings, so I worked around it by standing up a repo-unlinked "manual-deploy-only app" per environment.

Front 3: The cause of the $47/month was "a health check every 15 minutes"

The third was the 15-minute monitor. DynamoDB read cost on the curation platform had reached $47/month, and following the breakdown, the cause was not a product feature but that a Lambda monitoring the pipeline's health was COUNTing (a full scan that counts every row) a 1.22GB GSI (Global Secondary Index; a secondary index for lookups) across all statuses every 15 minutes. And most of that was re-counting even terminal, excluded statuses (parse errors and anti-bot (bot-blocking) exclusions) that never change once they land. A textbook case of the monitoring itself being the cost source.

The stopgap was to lower the aggregation frequency for the near-static terminal statuses.

# Stop COUNTing all statuses every 15 min;
# count the changeable ones often, thin the terminal ones to every 2 hours
ACTIVE = {"RAW", "CRAWLED", "ANALYZING"}          # the moving ones
TERMINAL = {"EXCLUDED_PARSE_ERROR", "EXCLUDED_ANTIBOT", "PUBLISHED"}

def collect_counts(now):
    statuses = ACTIVE.copy()
    if now.minute < 15 and now.hour % 2 == 0:      # terminal ones only once / 2h
        statuses |= TERMINAL
    return {s: count_by_status(s) for s in statuses}

That cut read cost by about 87% (roughly $47 → $5–6/month). But this is a stopgap. What is scary about this design is that as articles grow, the COUNT target grows too, and cost rises in proportion to data volume. At $47 today, if the number of articles grows 100x the reads grow roughly 100x too. So as the permanent fix I stopped counting the status aggregation on the GSI every time and moved to a counter table that updates counts incrementally via DynamoDB Streams (a mechanism that streams table changes), making the aggregation read O(1) (a constant amount of work regardless of row count). I implemented that the next month. A "periodically count everything" design is worth removing at the root.

Front 4: monitoring, backups, and the dev environment — the "non-product"

Stacking up the fronts, what emerged was that the cost sources gather, without fail, in the "non-product." A backup engine was full-scanning the dev table every day, and the 820,000+ crawl-job traces and article junk piled up there were driving up both the scan volume and the backup volume. I resolved it by purging dev down to 665 operational rows. Even small junk becomes a daily scan cost once 820,000 of them accumulate. On the uptime-monitoring SaaS side too, I dropped 22 dev monitors from a 5-minute to a daily interval ($18→$4/month) and disabled detailed container metrics in prod to cut CloudWatch (AWS's monitoring and metrics platform) cost. Reducing the analysis target by about 94% with pre-ingest triage, holding the per-day analysis cost to $0.13, was part of this same flow.

What worked across the board was a dashboard that gathers the cost of multiple providers onto one screen. Here I also hit a bug that was not funny. One day the Anthropic cost alone displayed as absurdly large, and it turned out that the Admin API (Anthropic's management API) returns cost in cents (the smallest unit) while the script summed it as dollars. The raw value "58.3135" should be $0.58; it had ballooned 100x. A 2-line fix of ÷100 corrected it, and as a lesson I noted "the currency unit differs across all three providers (one is cents, one is dollars, the bill is yen)." To picture the future amount, the premise is that today's number is displayed correctly first. If the visualization is off by 100x, the entire cost judgment goes wrong.

The mechanism: why the "non-product" becomes a future cost

What eventually clicked is that the places where cost tends to appear share a common structure. A product feature is billed only when a request arrives (elastic), whereas NAT, builds, periodic monitoring, and backups are steady costs that "keep running even when nothing is moving and nothing has changed." Steady costs grow quietly in proportion to scale, time, and data volume. And they slip in as unconscious settings — an IaC default, or "let's just do every 15 minutes" — so they never enter the field of view of feature development. That is why you look not at "how much now" but at "how much at 100x." On top of that, I fixed "measure then fix, fix then measure" into the operation, and watch cost changes daily on a dashboard rather than the bill. The bill arrives a month late, so if you only watch that, it is too late. The right infrastructure was not about features working — it was shaping these steady costs so they will not break in the future either.

Takeaways you can reuse

Look not at today's amount but at "when it's 100x." The smaller the steady cost, the more quietly it grows in proportion to scale and data. Kill it — structure and all — while it's small.
Doubt the IaC default. Without setting natGateways explicitly, a NAT stands up on its own. "Serverless = zero fixed cost" does not hold automatically.
Break the bill into its parts before fixing it. Only once I knew 87% of Amplify was build time did I see that the fix was stopping the 89% side, not migrating the 11% side.
Periodic execution gets more expensive as it grows. The full COUNT every 15 minutes became O(1) with an incremental counter. Suspect that monitoring and backups can themselves be a cost source.
Build the visualization correctly. The currency unit differs by provider. If the dashboard is off by 100x, not just the future amount but today's judgment goes wrong.

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.

I looked at the GitHub Actions bill and built a CI + security base at home

uehara — Wed, 19 Aug 2026 19:41:04 +0000

In late June 2026, my org's GitHub Actions stopped at the billing cap. Every PR (pull request) CI (continuous integration; a system that runs tests automatically on each change) check FAILED in 2 seconds with an empty log. Around the same time, plugin release notifications stopped arriving in Slack. Not a single line of code was wrong. An external service had, one day, stopped my infrastructure over billing.

At the time I was running GitHub Actions on the free tier. But I hit the free-tier cap, CI was blocked, and to keep developing I had no choice but to start paying by usage (metered charges for anything beyond the free minutes). "Free" ended right there. From that point on, money was going out no matter what — that is the starting point. So the question was not "cut cost by testing less" but "which is the better deal: keep paying GitHub, or build my own base?" GitHub's cost is metered, so tests run on every commit and the monthly amount grows the more products and the more active the development gets. A home CI server (ci1/ci2 class, about ¥135,799), by contrast, is a one-time outlay plus electricity, and it does not grow no matter how much you use it. A cost that keeps growing versus a one-time cost that does not — that structural difference was the deciding factor. And the per-run time of CI grows as the code grows. Measured: one product's GitHub Actions run time grew from about 1 minute on average in March this year to about 16 minutes on average — with the longest over an hour — by August. Commits across the products I actively run come to roughly 2,400 a month. Keep paying for that volume and that growth on GitHub's metered pricing (Linux $0.008/min), and the over-the-free-tier portion alone lands around $150–200/month, swelling every month as tests get longer. A home CI server for about ¥135,799 overtakes that in under half a year, and is cheaper forever after (the commit counts and run times are measured; the monthly dollar figure is an estimate from the free tier and average minutes). On top of that, the security-scanning tools I wanted to add (DefectDojo and so on) also need somewhere to run. So the best move was to build a single infrastructure base at home and run tests, security scanning, and monitoring all there. This is the record of acting on that decision. Per the records at the time, this base's monorepo (a layout that bundles multiple repositories into one) grew from its first commit to 330 commits and 35 ADRs in about a month.

Let me put the conclusion first (about an 11-minute read).

Every run failing instantly (2 seconds) with an empty log was the fingerprint of a GitHub Actions billing block — an external service stops your infrastructure one day, silently. Don't make the mechanism that reports failure depend on the mechanism that fails.
I built a base of my own that doesn't stop, and didn't throw away the old one either — I switched to self-hosted (owned by me) runners while keeping the GitHub Actions-side workflows, so I could go back with no changes if billing returned.
A distributed security workflow stopped every PR — a full-history scan by gitleaks (a tool that detects secrets slipped into code) drops even healthy PRs. A gate must be verified not only for "can it block" but for "does it pass correctly," or it stops everyone's work.
The pass/fail bar for HA (High Availability; a setup that doesn't stop even if one node dies) is actually cutting power and network on real hardware — even with all unit tests green, there were 3 bugs that only surfaced when I dropped it on real hardware.

Symptom: every run fails instantly and the log is empty

At first, I suspected our own workflows. But both CI checks FAILED in a consistent 2 seconds. Opening the log, there was nothing inside. To isolate it, I first suspected the Slack notification path and hit the webhook directly.

# Is the webhook itself alive? → HTTP 200, delivered to Slack = the code is innocent
curl -s -o /dev/null -w "%{http_code}\n" -X POST "$SLACK_WEBHOOK" \
  -H 'content-type: application/json' \
  -d '{"text":"probe from local"}'
# => 200

The webhook was 200. The local notification script was fine too. So the notification logic was innocent. Next, peeking at the CI side with gh run view, every run was ending in 2 seconds with an empty log. Here the picture came together. The runner itself is not starting. The cause was a GitHub Actions billing block; the org's GitHub Actions were fully stopped. Every run failed instantly (2 seconds) before it could even start, with nothing left in the log — that behavior itself, in hindsight, was the fingerprint of a billing block. And the nasty part was that the stopped mechanism was silent. Because the "mechanism that reports failure" — Slack notifications — itself depended on Actions and went quiet, it took several days to notice.

The turn: build a CI that doesn't depend on hosted, "in code"

An external SaaS (software as a service) can, over its billing situation, stop your infrastructure one day out of nowhere. Then the only choice is to hold a base of your own that doesn't stop. And if I'm building a base at home, putting only CI on it would be a waste. Tests, the security scanning I plan to add, and monitoring — I'd consolidate them all onto the same single infrastructure. That, I decided, was best. The plan was: "stop the practice of SSHing (remotely logging into a server to run commands) in to type commands directly, and manage the home server fleet declaratively with Ansible / Docker Compose / Terraform (all tools that define and apply configuration as code)." I harvested the state of the single existing CI server into a monorepo, and codified (defined as code) the self-hosted runners, a vulnerability ledger, observability (making runtime state visible from outside) via Grafana/Loki/Prometheus (a monitoring stack for visualization, logs, and metrics), and the log platform — almost all in one go over the first day or two.

Each product's workflow is mechanically re-pointed from a hosted premise to self-hosted runners.

# Drop the hosted (ubuntu-latest) dependency; point at the home runner labels
jobs:
  test:
    runs-on: [self-hosted, linux, docker]   # labels for the home runner fleet
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install --frozen-lockfile && pnpm test

I kept the Actions-side workflows rather than deleting them, so I could go back with no changes if billing came back. "If it breaks, switch to another base, and don't throw away the broken one either" — that became the basic policy.

Trap 1: the distributed security workflow stopped PRs across the whole org

Soon after migrating, the commonly distributed security workflow caused an incident that blocked every PR at the destinations. At the time this workflow was distributed to 25 of the org's 102 non-archived repositories, so the blast radius was wide. The real cause was how gitleaks was used.

gitleaks detect --source . scans the target across all refs (effectively git log --all). That means it picks up test dummy secrets left on old branches and old leak traces, so even a healthy PR always fails. On top of that, workspace pollution on the self-hosted runners produced false positives detecting "a file that shouldn't exist in that PR." The fix is to limit the scan scope to the PR diff.

# Stop scanning all refs and always failing PRs. Look only at the diff.
gitleaks detect --source . --redact \
  --log-opts="origin/${BASE_REF}..HEAD"

To be safe, I even prepared a self-test workflow that runs E2E (end-to-end) verification on both a branch that should PASS and a branch that should FAIL. The lesson: a security gate is no good if it can only "block" — unless you also verify it "passes correctly," it becomes a weapon that stops everyone's work.

Trap 2: going self-hosted broke the Docker build as a side cost

The migration came with debts too. Right after fully removing the hosted runners, CDK (a tool that defines cloud infrastructure as code) Docker bundling stopped working on several products. In a "runner inside a container + the host's docker socket" setup, the volume mount the bundling uses came up empty. On one product the deploy workflow was wiped out for a while, and I lived on manual deploys from a local Mac. The remedy was to replace Docker bundling with local dependency expansion via pip install --target on the runner, and I finally closed it after confirming 5 consecutive successful deploy workflows by measurement. Move the base and the build premise on top of it quietly collapses. Here too, the policy of confirming "it works" only by measurement paid off.

HA: caused a split-brain and fixed it the same day by dropping real hardware

With a single runner, if it dies, CI stops again. So I went to a 2-node setup, and here the biggest incident happened. After failing over from ci1 to ci2 to add an NVMe (a fast storage standard), rebooting ci1 caused both nodes to become active and uncoordinatedly fight over the same job queue — a split-brain (both systems becoming primary at once), which I observed late at night. Jobs were submitted from the web to ci2 but executed on ci1 — an unsettling state.

The cause was a structural problem: "the starting system has no gate to decide whether it is the designated active." ci1 was auto-starting without knowing ci2 was active. The same day I implemented atomic lease fencing (a mechanism that reliably locks out the old primary to prevent double activation) — a fence-agent resident on both nodes with heartbeat 2s / TTL (time-to-live) 6s / grace 6s — and passed a real-hardware kill-test (a test that actually cuts power and network). Auto failover 18 seconds; self-fence + takeover 17 seconds on an NFS (a network-shared filesystem) partition; data divergence exactly zero. The biggest lesson here is failover ≠ fencing. Manual failback gets forgotten. And even with unit tests (bats; a testing framework for bash) all green, there were 3 bugs that surfaced only when I actually cut power and network on real hardware (a livelock where heartbeats yield to each other during grace and make no progress; infinite I/O blocking from a hard-mount; a blind steal that takes the primary without checking the other side's lease). This was the day I wrote into the Mission that the pass line is the real-hardware kill-test, not a design document.

Infra Portal: consolidate all products onto one screen

Finally, I grew a home-grown portal (I call it the Infra Portal) that consolidates onto one screen: CI status, deploy promotion, multi-provider cost, backup sufficiency, and the liveness of servers and the local LLM (large language model). There are two design ideas. One: "don't trust registration; auto-detect capability from the repo and the live environment by measurement." The other: "don't treat HTTP 200 as alive — read the meaning of the actual response." The former catches gaps in self-reporting; the latter catches "the server responds but the contents are broken." Here too there was a laughable failure: one node's card was frozen at 62 hours ago because of an asymmetric HA install — I had forgotten to put the reporting script needed on the passive side. I fixed it by making both nodes symmetric.

Takeaways you can reuse

Tests can't be skipped. So I looked at the cost and consolidated onto one base of my own. Instead of fitting inside a free tier, I estimated cost on the premise of running continuously, and drew CI, security, and monitoring onto a single piece of infrastructure.
Every run failing instantly with an empty log is the fingerprint of a billing block. An external service stops your infrastructure one day, silently. Don't make the mechanism that reports failure depend on the mechanism that fails.
Build the base yourself, and don't throw away the old one. Switch to self-hosted while keeping the hosted workflows, so you can go back with no changes.
A distributed gate must be verified down to "passing correctly." Like gitleaks' all-refs scan, a gate that can only block ends up stopping everyone's work.
The HA pass/fail bar is the real-hardware kill-test. failover ≠ fencing. Even with bats all green, there are bugs that only surface when you actually drop it on real hardware.

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.

When a Provider Retires Your LLM Model: Two Products, the Root Cause, and Preventing Recurrence

uehara — Mon, 17 Aug 2026 09:22:04 +0000

On the morning of July 10, 2026, the design-generation feature of promptflow — our multi-LLM workflow SaaS — went completely dark. The cause wasn't our code. Google had discontinued the gemini-2.5 model family the same day, with no advance notice. And this "the model just vanished" incident wasn't the first. On July 5 of the same month, the image model dall-e-3 disappeared from OpenAI's API, and one month before that, on June 3, we had just finished scrambling to handle the retirement of gemini-2.0-flash-lite in curation, our news-curation platform.

This article puts these back-to-back "provider retires a model" incidents from two separate products side by side and records, from the logs and code at the time, (1) why they become an outage instantly, and (2) what to monitor and what to prepare as a fallback to contain the damage. We hadn't anticipated this risk well enough. Model retirement hit three times in one month. The first, in curation (6/3), was handled by simply swapping in the successor model; but in promptflow it struck twice in July (7/5 and 7/10), and that is where we stopped "fixing it when it breaks" and built the defense into the system instead. The individual incidents are written up separately — the promptflow edition (PF-002) and the curation edition (MISC-002). This article focuses on the cross-cutting lessons from putting the two together.

Here is the conclusion up front (about a 6-minute read).

The cause wasn't our code — it was the provider retiring a model. A model that worked yesterday simply doesn't exist today. It's the kind of change code review can never catch.
Hard-coding model IDs into code and prompts is what spread the outage across everything. The moment a version disappears, the whole generation path goes down with it. Pin versions to aliases, and hold declared fallback candidates for when one is retired.
The response format changed too, and parsing caused collateral damage. Gemini 3 started returning its "thinking" part separately, and the generated HTML lost its doctype. Don't let parsing implicitly depend on one model's output format.
The same incident happened three times in one month, across two products. Rather than leaving it as a patch, we promoted it into model-catalog governance and automatic switch-over on retirement.

The symptoms: "does not exist," and a trap where the whole response format changes

The first sign was on the curation side. On June 3, gemini-2.0-flash-lite, which we used for article analysis, entered its retirement-notice window and responses became unstable. That one was solved just by moving to the successor gemini-2.5-flash-lite; while we were at it, we merged the two analyze and long_insight calls into one and cut roughly 16% of the tokens per article. At that point it felt like nothing more than a routine "rename the model" chore.

The real trouble came the next month, on the promptflow side. On July 5, the image-generation E2E test failed like this:

POST /v1/images/generations  model=gpt-image-1  -> HTTP 504 Gateway Timeout
POST /v1/images/generations  model=dall-e-3
  -> 400  { "error": { "code": "model_not_found",
            "message": "The model `dall-e-3` does not exist" } }

Our primary, gpt-image-1, returned a 504, and the fallback candidate dall-e-3 was reported as "does not exist." We restored service on the spot by switching to gpt-image-1-mini. But five days later, on July 10, Google discontinued the gemini-2.5 family without notice and design generation went completely dark. The nasty part was that it wasn't just a name disappearing. The migration target, the Gemini 3 family, had changed its response format itself — it misread the "thinking" part as body text, and the generated HTML lost its <!doctype html> entirely, a second-order failure.

The first hypothesis, and how it was wrong

On July 10, the first thing I suspected was that our own prompt engineering had broken. Design generation had been in constant quality tuning around that time, so I thought, "did I break the prompt again?" But rolling the prompt back to the last stable version still produced empty output. Next I suspected the API key had expired — but other models responded fine.

The split became clear the moment I pinned the model ID and threw a bare call at it:

curl -s "$GEN_ENDPOINT/models/gemini-2.5-flash:generateContent" \
  -H "x-goog-api-key: <REDACTED>" \
  -d '{"contents":[{"parts":[{"text":"ping"}]}]}' | jq '.error // .candidates[0]'

What came back wasn't our bug — it was an error from the provider meaning "this model is no longer offered." That is where the hypothesis shifts from prompt to provider. In other words, what worked yesterday simply doesn't exist today. It's the kind of change code review can never catch.

The mechanism: why model retirement becomes a total outage so fast

Once I stepped back and organized it, there were three reasons the damage spread across everything at once.

We had hard-coded model IDs into code and prompt contracts. When a generation path is pinned to a specific version, the moment that version disappears the whole path goes down.
The response parsing implicitly depended on one model's output format. Once Gemini 3 returned the thinking part in a separate field, the old parser mixed it into the body and the HTML broke.
We had no way to know about retirements in advance. Provider announcements are buried in email and changelogs; you cannot assume a human tracks them every day.

In short, against a model disappearing for reasons outside our control, the system had no room at all to switch to an alternative.

The fix: stop pinning versions, make parsing tolerate broken responses, and detect retirements

As an immediate response, we moved model references from fixed IDs to latest-equivalent aliases, and added thinking-part exclusion and HTML salvage to the response parser.

// Gemini 3 mixes in a thinking part. Extract only the body, and
// if the doctype is missing, add it to rescue the "broken HTML."
function extractHtml(parts: Part[]): string {
  const body = parts
    .filter((p) => p.kind !== "thought")   // don't misread the thinking part as body
    .map((p) => p.text ?? "")
    .join("");
  const html = body.trim();
  if (!/^<!doctype/i.test(html) && /<html[\s>]/i.test(html)) {
    return `<!doctype html>\n${html}`;   // add the doctype if it's missing
  }
  return html;
}

On top of that, we turned recurrence prevention into a mechanism. One part is model-catalog governance: hold the usable models as a versioned set, run an automatic new-model scanner on a cron (scheduled job), and surface additions and retirements to a human via notifications. The other is graceful fallback on retirement: declare alternative candidates per model, and when a call fails with model_not_found, switch automatically to the next candidate. The picture is a map like this:

# Declare each generation model's version and where to switch on retirement
image_generation:
  primary: gpt-image-1
  fallback: [gpt-image-1-mini]     # drop the retired dall-e-3 from the candidates
design_generation:
  primary: gemini-flash-latest   # don't pin the version
  fallback: [gemini-flash-stable, sonnet-html-fill]
  parser: defensive-v2           # thinking exclusion + doctype salvage

On the curation side too, we stopped treating a model migration as a mere "rename" and paired it with logic that drops unexpected responses into a terminal status to prevent infinite retries. The lesson from curation fed straight into the design of promptflow.

Afterward, and the limits

After this July double-hit, model retirement went from "an incident you fix when it happens" to "an operation that assumes it will happen and switches to an alternative." The scanner cron now tells us about new models ahead of time, and at least we have reduced how often "everything suddenly stops one morning" happens. But the limits are clear too. The fallback's quality isn't the same as the primary's; for a feature like design generation, where output quality is the selling point, a "doesn't go down but degrades" state remains. The scanner can't detect a response-format change — in the end you don't know until you throw a real request at it. And fundamentally, as long as you depend on a model you don't own, you cannot reduce the risk of supply cutoff to zero. All you can do is prepare so that when the day comes, it is a partial degradation rather than a total stop.

Takeaways you can reuse

Don't hard-code model IDs. Pin versions to aliases, and hold declared fallback candidates to switch to on retirement. "It worked yesterday" doesn't guarantee "it exists today."
Don't let response parsing implicitly depend on one model's format. Write it assuming you'll salvage the body even from a broken response — for format changes like a mixed-in thinking part or a missing doctype.
Don't track retirements by hand. Manage the usable models as a versioned set, and detect additions and retirements with a scanner cron that surfaces them to a human.
Turn the first occurrence into a mechanism. The same incident will inevitably recur in another product. Don't leave it as a patch; promote it into fallback and governance, and you'll greatly reduce the damage the second time around.

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.

Cloud bills kept climbing from 24/7 AI development — I moved the decisions and the implementation to my own local LLMs and cut the cost

uehara — Sat, 15 Aug 2026 15:05:26 +0000

From late June into July 2026, I rebuilt the execution backbone of my development onto my own local large language models (LLMs). The trigger was cost.

The more you hand development over to AI, the more every task-routing decision and every actual code generation gets sent to a cloud AI, each and every time. Run it 24 hours a day without stopping, and that usage-based billing piles up every month, exactly in proportion to how much you used. And this is something you keep paying, month after month, forever.

But, I thought. If I buy the hardware once and shift the execution beyond that point onto my own local LLMs, couldn't I erase most of this ongoing cost? Couldn't I replace usage-based billing — which grows with every bit you use — with a one-time hardware cost?

Here is what I did. I moved the task-routing decisions (which I'll call the "orchestrator" in this article) and much of the hands-on work from the cloud AI to my own local LLMs. To do that, I bought one NVIDIA DGX Spark and combined it with the four Macs I already had to build an execution backbone that development tasks flow through. Leave the hands-on work to the AI, and keep only the decisions for the human. Push that division far enough and you arrive at Human-Out-Of-The-Loop (HOOTL), where the human steps outside the loop.

Let me say this up front. This is not a bragging-rights story about "I cut ¥X." What I want to convey here is not a proof of some dollar amount, but the way you wire things so you erase as much of the recurring cost — the usage-based billing — as the design lets you erase.

Let me put the conclusion first (reading the body takes about 8 minutes).

Erased as much of the recurring billing as the design allowed — took both the decisions and the hands-on work locally first, and sent to the cloud only what genuinely needed it
Locked the buy/no-buy criteria with an ADR (a record of design decisions) before purchasing — ruled out "buy it because I want it," and decided the purchase on measured speed alone
Decided which model goes where, per role, from measured speed and cost — routing decisions on 14B, hands-on code generation on the zero-billing 72B lane
Ran one model per host — avoided swap costs and made the split between local and cloud visible in numbers

When you actually do this replacement, the things to decide narrow down to three. ① Whether to buy your own hardware, and if so, which. ② Which model to put on which job. ③ How to line up multiple machines. I'll write them in order.

The period in question breaks down like this. The purchase decision itself was an immediate call from the benchmark results alone (late June 2026). From there the DGX Spark arrived (July 10), and it took about two weeks to get the basic form working. After that, in late July, an incident occurs. From the purchase decision to here is about one month of records.

Before buying: to erase the recurring billing, I first decided "whether to buy my own hardware" by measurement

Why did I want to shift the decisions onto my own local LLMs in the first place? There are two reasons. One is the recurring cost. The other is that I don't want the decision-making itself to depend on external billing.

The routing decision is the central process that runs 24 hours a day. Leave this to the cloud, and the cost of the central process piles up every month in proportion to use, and if it stops for reasons on the external service's side, all of development stops with it. So I put no cloud LLM on the orchestrator. This is a design principle of local-commander (that's the name of this development orchestrator).

Cloud costs stop development in two forms. On a free tier you hit the usage limit and stop; on usage-based billing it piles up without limit, in proportion to use. Neither is a fantasy for us. Our organization's GitHub Actions stopped from April 2026 on suspicion of hitting the free-tier ceiling.

I ran the numbers, too. Running CI on GitHub Actions' standard runners (Linux, 2 cores, $0.008/min) for two machines' worth, even estimating utilization at half (12 hours a day), comes to about ¥55,000/month. Against that, a CI server is ¥130,000 each, ¥260,000 for two, one time. It pays for itself in about five months, and beyond that there is no recurring billing.

That said, shifting the decisions local requires speed. Running a 31B-class model on the MacBook Pro (M3) at hand gives an effective 5 tok/s, 60–200 seconds per decision. That doesn't reach the speed needed for 24-hour autonomous operation.

Here, so that "I want a DGX Spark" as a want-it impulse wouldn't come first, I bound myself before buying with an ADR (Architecture Decision Record — a document that records design decisions).

ADR-0003: Gate the DGX purchase decision on PoC (proof-of-concept) benchmark results only

On top of that, I ran a 16-case classification benchmark. The results are these.

The system's final classification matched 16 of 16 (100%), with 0 dangerous-side misclassifications (qwen3.6-35b)
But this 100% does not mean "the LLM alone is perfect." The raw LLM's JSON-formatting success was 69% (31% fallback), and broken output was absorbed by deterministic safety overrides on the code side, lifting the final decision to 100%. The 100% is not the model's cleverness but the result of a mechanism that fails toward the safe side even when it breaks
The only bottleneck is speed (60–200 seconds per decision on M3)

The correctness and safety of the final decision can be guaranteed by safety rules on the code side even when the LLM breaks. What's missing is only speed, and speed can be solved with hardware. The day after this measurement lined up, I decided to buy.

This connects to the recurring-cost story. Hardware is done once you buy it. The execution beyond that carries no per-token billing. Replace the monthly usage-based billing with a one-time expense. I positioned it as a purchase for exactly that. The PoC report and ADR I used for the decision are all kept in the repository.

The day it arrived: the first thing I did after ssh

2026-07-10, the DGX Spark went live. This is the first session where I ssh dgx'd in.

aarch64 on GNU/Linux 6.17, 3.67TB of storage. From the day it arrived, I'd already started running model-selection benchmarks

The terminal and dashboard images in this article have only the text — product names, hostnames, IPs, and so on — replaced with anonymous dummies from the actual screens (layout and numbers are real data; only verified numbers are shown). Some screens are anonymized diagrams with the layout reconstructed.

I bought this machine intending to place it at the center of the execution backbone that replaces the monthly usage-based billing. So from day one, I began setup on the premise of running local-commander (the first thing I placed was a marker called .lc-bootstrap-allow that permits its initial startup).

Selection: "bigger models are smarter" did not hold for classification

The DGX has 121GiB of memory. It loads a 72B class with room to spare. Naturally the urge comes up: "I want the biggest one as the orchestrator." Here too I chose by measurement. Here is the result of comparing 14B / 32B / 72B on the same cases for the orchestrator's job (task classification).

Model	Classification accuracy	Time per decision
qwen2.5-coder:14b	100%	4.4 s
qwen2.5-coder:32b	88%	(intermediate)
qwen2.5:72b	100%	22.8 s

14B and 72B tie on accuracy, and 72B is 5× slower. And only the intermediate 32B drops to 88% from broken JSON formatting — a counterintuitive result. For a task like classification, "bigger is smarter" does not hold. So I decided on 14B for the orchestrator.

So is 72B wasted? No. I turned 72B into the "zero-billing work lane" that runs the hands-on code generation. This is where the recurring-cost story bites. Throwing the hands-on work at the cloud incurs usage-based billing, but running it on the 72B already loaded on the DGX incurs no per-token billing.

Just to be safe, though — I put 72B on the hands-on work not because "72B is smarter at code generation." In the generation benchmark, 14B and 72B tie at a 57% success rate, and 72B is about 6× slower, and even the primary source concludes "promotion to 72B is shelved." I still wired up the 72B lane because I can reuse an asset already loaded on the hardware at zero per-token billing. The reason I put 72B on the work even though 14B is faster is that the four 14B Macs are always full with classification and have no spare, while the DGX has 72B loaded from the start (how these hosts are split is the next section). I limit its use to low-risk tasks classified as local-ok (judged able to complete safely on local), and if it fails, escalate to the cloud (Claude). On that premise, 72B's slowness is acceptable, and I gain one more lane that carries no billing.

Configuration: the DGX is a 72B-only machine, the Macs are 14B classifiers

The final configuration is one model, one host.

DGX Spark: a qwen2.5:72b (111.4GB) dedicated machine. As a "work lane" that carries no per-token billing, it processes low-risk (local-ok) code-implementation tasks 6 in parallel, and escalates to the cloud on failure
Four Macs (M4 / M3 and others): a pool of qwen2.5-coder:14b classifiers, bundled with a weighted load balancer (a mechanism that distributes load by weight). Apple Silicon, with its GPU and unified memory, is inherently suited to LLM inference. On top of that they double as CI runners, saving the pricey cloud-build (macOS runner) billing for iPhone apps. They pull many roles, so if I was going to buy, they were the optimal choice
Cloud (Claude / Codex): only tasks that don't complete locally flow through, passing a human approval gate
CI server: separately from the above, I've also introduced a CI server to keep the orchestrator (local-commander) from stopping (the cost estimate is above). Availability (HA) and configuration details I'll write in a separate article

The aim of this split is to flow to the cloud, which carries recurring billing, only the tasks that truly need the cloud. Both routing decisions and hands-on work are taken locally first. Only the part that doesn't complete locally goes out to the cloud.

Seen on the admin screen, it looks like this.

Execution lanes: the cloud (Claude) is idle, classification 14B has all four Macs busy, and the work 72B (DGX) is also idle. Only the routing decisions run without pause

The reason for not loading multiple models on one machine is simple: swapping models (load/unload) is the most expensive part. Measuring tok/s continuously with Grafana (a tool that visualizes measured results) gives DGX 24.27 tok/s, M4 16.09 tok/s. Watching these numbers, I'm growing an allocation table that decides "which task flows to which lane" by win rate and cost.

Operation: 24 hours seen on the admin screen

Once the configuration takes shape, development tasks flow into the inbox (INBOX), get routed, are executed on each lane, become PRs (pull requests), pass multiple review gates, and wait for human approval. This flow starts running 24 hours a day.

One day's execution state: decisions-pending pile up, and queues for several products line up side by side. In the last 24 hours: 2.37 million tokens, 102 jobs, about $15.76 by reference conversion

In the last 24 hours, 2.37 million tokens, 102 jobs. The $15.76 shown on the admin screen is a reference-conversion amount for 24 hours computed from a registered rate table (token-rates.json), not the cloud's actual invoiced amount itself (when models with unknown rates are mixed in, it becomes a lower bound from the known portion only).

What matters is that this $15.76 carries none of what ran on the 72B lane (DGX). Recurring billing applies only to tasks sent to the cloud. The portion taken locally carries no per-token billing.

Of course, the DGX's hardware cost and electricity cost apply separately. It isn't running for zero yen. Still, I was able to replace the usage-based billing that grows every month in proportion to use with a one-time hardware cost and a fixed electricity cost. That the split became visible in numbers is the biggest gain from building this backbone.

It wasn't all smooth sailing

It's not only a clean story. In late July, the classification cluster stopped in a chain. Retries concentrated on a bloated classification prompt (4,251 tokens), and one node's four slots all fell into a 37-second serial queue and went silent. Before reaching the true cause, I misdiagnosed for 3 days from misreading the dashboard. The fixes were co-primary'ing (distributing traffic 50/50 across two nodes) and bounding the prompt (capping its size).

This was not an incident where the cloud's usage-based billing ran wild. It's an incident on hardware I bought once — one I caused myself and can fix myself. Being able to settle in and chase the true cause without worrying about billing is also a byproduct of shifting execution onto my own gear. I'll write the incident's details in a separate article.

Summary

Erase as much of the recurring billing as the design allows — take both the decisions and the hands-on work locally first, and send to the cloud only what's genuinely needed. The idea of replacing monthly usage-based billing with a one-time hardware cost
Lock the buy/no-buy criteria with an ADR before buying — rule out "buy it because I want it," and decide the purchase on measured speed alone
Choose models per role by measurement — 14B is enough for routing decisions (tied on accuracy with 72B, and 5× faster). There's also the trap where only 32B degrades. Run hands-on code generation limited to the zero-billing 72B lane
Run one model per host — avoid swap costs, and decide allocation by measuring tok/s continuously
Keep records — because the ADRs, benchmark reports, and postmortems are all in the repository, this article can be written

At the root of this replacement is a way of working where the hands-on work is turned over to AI and the human concentrates on the final approval = the decision. Right now it's Human-in-the-loop (HITL), with a person remaining at the approval gate; shift even that approval onto the mechanism, and it approaches HOOTL, where the human steps outside the loop. Placing the center of decisions (the orchestrator) on my own gear and cutting it off from external usage-based billing was one move toward running that foundation cheaply, for the long haul.

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.

Moving Half of Our AI Development to Local LLMs — by Splitting Work by Role, Not by Picking the Biggest Model

uehara — Mon, 10 Aug 2026 23:03:25 +0000

Uehara, EarthLink Network Co., Ltd. I build and run more than 20 products by myself, with Claude Code at the core of development. This is a field note from that work.

On July 30, 2026, on the control panel that runs our in-house AI development, the share of work handled by local LLMs reached 50.3%, with the cloud side at 48.2%. This does not mean our bill was cut in half. It means the split of execution volume — including transcripts — came out to roughly fifty-fifty.

How I thought about this changed a great deal from where I started.

At first, I thought "just put one big model in"

At the beginning my idea was simple: load a single 70B-class model onto the large memory of a DGX Spark, and it would make a good commander. Bigger must be smarter — a naive assumption.

Measuring on real hardware gave the opposite result

When I actually measured, the result was the reverse.

For the task of classifying work, I compared 14B, 32B, and 72B across 16 cases. The final classification agreement was the same for 14B and 72B; the 72B was simply five times slower. For code generation, the test pass rate was the same, and the 72B was six times slower. On top of that, most failures were not about how smart the model was, but about whether the generated diff could be applied at all.

"Bigger models are better" did not hold — at least not for this use.

I also stumbled once on memory estimation

Trying to co-host the 14B and the 72B on the same machine, I got the memory math wrong. On paper the model weights fit, but once you include context they physically do not. Classification stalled for four and a half minutes. After I shortened the classifier's context, both could stay resident and the stall dropped to about twelve seconds. Estimating without measuring turned straight into failure.

Not a ranking of models, but a division of roles

So I changed my approach. Instead of building a ranking of models, I split tasks by role. Light judgments go to a small local model; heavy work goes to the cloud. And dangerous operations — database, authentication, billing, production — always return to human approval, no matter what the model says. Code written by a local model gets no lighter a review.

What 50.3% means

That is where the 50.3% at the top comes from. It is not a story about local LLMs replacing the cloud. It is a record of splitting work by role, measuring it, and building something you can roll back when it fails — and of how that helped not only cost but also speed and safety.

What I will measure next is not the token ratio, but the total cost and time per successful task.

About the author

Uehara builds AI at EarthLink Network Co., Ltd. Since 2025 I have put Claude Code at the center of development, and I now build and run more than 20 products by myself. In this series I write about what actually happens on the ground — the wins and the failures alike — together with the numbers.

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.

DEV Community: uehara

Bringing shared foundations into one place — merging five shared repositories into one

The five shared repositories that had scattered

Options A / B / C, and the shape I chose

The design decision that mattered most: packages do not read process.env

What running the migration taught me

The conditions under which this transfers

Lessons you can reuse

Four Failures I Kept Repeating on AWS Amplify — Full env Replacement, Green Build but 500 at Runtime, Silent Outages, Domain Migration

Failure 1: update-app "replaces all" of your env

Failure 2: The build passes, but it fails at runtime

Failure 3: Failures progress silently

Failure 4: Domain migration, and CloudFront that won't release

Why build success and runtime success don't line up on Amplify

Transferable lessons

When anyone can build with a prompt, what counts as productivity? — We're moving our whole company to AI and keeping humans on judgment

Why "productivity" is being questioned again

Do we stop needing teams?

What we're actually doing: automating the entire company

The conclusion: humans become "machines that make decisions"

Next-generation productivity is measured by "the quality and count of decisions"

Summary

The 18 products EarthLink Network builds in-house

Building the foundation (shared components and infrastructure)

Building the site and audience

Getting work done (tasks → knowledge → orchestration → AI infrastructure)

Monitoring and managing

Supporting customers

Building verification and teaching material

How to read this list, and what's next

An AI's “Done” Can't Be Trusted — Quality Gates That Stop False Completion With Evidence

The classic false completion: passing off "the tests pass" as "it works"

Another face: "implemented" and "running in production" are also different things

Hallucinated reports: "I added one line" on an empty branch

The mechanism: why does an AI say "it's done"?

The fix: define completion by "evidence," not by "claim"

Distribution drift (deploy drift): a freeze invisible to push-based hooks

Summary — transferable lessons

I cut small AWS charges by picturing them at 100x scale

Front 1: Two NAT Gateways CDK stood up on its own, $64/month

Front 2: 87% of the Amplify bill was "build time"

Front 3: The cause of the $47/month was "a health check every 15 minutes"

Front 4: monitoring, backups, and the dev environment — the "non-product"

The mechanism: why the "non-product" becomes a future cost

Takeaways you can reuse

I looked at the GitHub Actions bill and built a CI + security base at home

Symptom: every run fails instantly and the log is empty

The turn: build a CI that doesn't depend on hosted, "in code"

Trap 1: the distributed security workflow stopped PRs across the whole org

Trap 2: going self-hosted broke the Docker build as a side cost

HA: caused a split-brain and fixed it the same day by dropping real hardware

Infra Portal: consolidate all products onto one screen

Takeaways you can reuse

When a Provider Retires Your LLM Model: Two Products, the Root Cause, and Preventing Recurrence

The symptoms: "does not exist," and a trap where the whole response format changes

The first hypothesis, and how it was wrong

The mechanism: why model retirement becomes a total outage so fast

The fix: stop pinning versions, make parsing tolerate broken responses, and detect retirements

Afterward, and the limits

Takeaways you can reuse

Cloud bills kept climbing from 24/7 AI development — I moved the decisions and the implementation to my own local LLMs and cut the cost

Before buying: to erase the recurring billing, I first decided "whether to buy my own hardware" by measurement

The day it arrived: the first thing I did after ssh

Selection: "bigger models are smarter" did not hold for classification

Configuration: the DGX is a 72B-only machine, the Macs are 14B classifiers

Operation: 24 hours seen on the admin screen

It wasn't all smooth sailing

Summary

Moving Half of Our AI Development to Local LLMs — by Splitting Work by Role, Not by Picking the Biggest Model

At first, I thought "just put one big model in"

Measuring on real hardware gave the opposite result

I also stumbled once on memory estimation

Not a ranking of models, but a division of roles

What 50.3% means

About the author

The design decision that mattered most: packages do not read `process.env`

Failure 1: `update-app` "replaces all" of your env