DEV Community: Hideki Mori

Count what you can predict, retry what you can't

Hideki Mori — Mon, 02 Nov 2026 13:00:00 +0000

A schema bug, a retry that couldn't help, and a small design rule I'm keeping.

The error

One day a StructFlow job — the structured-extraction service on my document-processing API — failed with an error I had never seen:

Schemas contains too many parameters with union types (27 parameters with
type arrays or anyOf). This causes exponential compilation cost. Reduce the
number of nullable or union-typed parameters (limit: 16 parameters with unions).

My first guess was length — the schema must be too big. Wrong. The input wasn't unusually long. More to the point, I had never written this schema at all: on this API, the output schema is derived from each user's example_output — a JSON example of the shape they want. That detail matters. A fixed internal schema either fits under a limit forever or fails on day one; my services built that way had run quietly for months. This error could only come from the open-ended part of the system — whatever example a user decides to send. And even in that user's example, nobody had written 27 anywhere.

Where 27 comes from

Two mechanisms multiply, and neither is visible on its own.

First, the strict-conversion rules. The shared schema generator derives required as the intersection of keys present across all example items; keys that don't appear in every item are simply left out of required. No unions yet — this intermediate schema is byte-identical for every provider. The unions are manufactured one step later, by the provider-specific strict conversion. Providers whose strict contract demands that every property be listed in required leave the converter only one way to express "optional": a nullable union —

"id": { "type": ["string", "null"] }

So a heterogeneous example — sections that are sometimes tables, sometimes procedures, sometimes notices, sometimes figures — quietly produces nine union-typed parameters in one schema. Nobody typed the word "union": not the user who wrote the example, not me. The provider's contract forced my converter to manufacture them, following rules I wrote once and then stopped thinking about.

Second, batch composition. To use tokens efficiently, the layer above packs multiple input units into one request, and the response schema is repeated per unit (index0, index1, ...). Nine unions per unit × three units = 27.

The constraint isn't about schema length. It's per-unit unions × batch size — a multiplication where I controlled both factors separately and had never once looked at the product.

The retry that couldn't help

Here is the embarrassing part. The error did trigger my retry logic.

My pipeline is built on a principle I've kept for 24 years: absorb failures with deliberate retries and always return a result to the user. The retries fired exactly as designed.

And they were completely useless.

The retry path this error landed on resends the same request — same units, same schema. But this failure is deterministic: the schema structure itself violates the limit, so the same input produces the same 400 every single time. The pipeline retried to its limit, burned every attempt on identical rejections, and failed anyway. That exhausted-retry failure is what surfaced as the bug.

The lesson stung a little. A retry can only fix failures where the next attempt might go differently — network hiccups, rate limits, transient provider errors. Against a structural error, retrying is just paying to watch the same rejection again. Twenty-four years of trusting retries, and I had never drawn that boundary explicitly.

The fix, in two stages

Stage 1 — route the error to the right shelf. I already had a separate retry shelf for a related class of errors: the provider's "The compiled grammar is too large." That shelf's retry is different in one crucial way — it splits the batch into smaller unit groups before retrying. Splitting changes the input: fewer units per request means fewer total unions, so the retry can actually succeed. One added string match moved the union error from the same-request retry path onto the splitting path:

retryable = errorMessage.contains("The compiled grammar is too large")
         || errorMessage.contains("too many parameters with union types");

Ugly, observational, effective. The pipeline stopped failing.

Stage 2 — count before you send. But stage 1 still burns a request to learn what I could have computed. The union limit is a published, predictable number: 16. If I count the unions in the strict schema before sending, I never enter the failure path at all. And when the schema space is user-defined, counting is the only defense: I can't test in advance against examples nobody has sent yet, but I can always count the one in front of me.

I implemented it as a capability method:

// GenerativeAICore (base class)
protected int getMaxUnionParameters() {
    return -1;  // no limit
}

// AnthropicV1Core (override)
@Override
protected int getMaxUnionParameters() {
    return 16;
}

A recursive counter walks the schema after strict conversion — counting type arrays and anyOf branches, the exact two constructs the error message itself names — and the batching layer checks the product:

int maxUnions = core.getMaxUnionParameters();
if (maxUnions > 0 && perUnitUnions * batchLimit > maxUnions) {
    batchLimit = 1;  // fall back to one unit per request
}

Note what I did not do: a proportional formula like batchLimit = maxUnions / perUnitUnions. That would squeeze out more batching efficiency — and add a formula I'd have to trust in production. Binary and safe first; clever later, if ever.

And note what I could not do: pre-flight the "grammar too large" error the same way. That limit depends on unpublished internals — enum counts, string constraints, nesting depth, property totals — quantities nobody outside the provider can compute. It stays on the observational retry shelf, where splitting handles it.

That's the whole rule:

Count what you can predict. Retry what you can't.

Published numeric limits get counted before the request. Opaque complexity limits get observed, matched by error string, and absorbed by a retry that actually changes the input.

Is this an Anthropic thing? (Yes — and here's the proof)

Before shipping, I ran the same heterogeneous workload — nine unions per unit, batched — through every provider path in my system:

Provider	Schema shape sent	Union limit	Result
Anthropic	object-keyed, nullable unions	16	pre-flight fires → 1-unit batches → completes
OpenAI (direct)	array, nullable unions	none observed	full batches, completes
Azure OpenAI	array, nullable unions	none observed	completes
Gemini / Vertex	array, subset-`required` — no unions manufactured	n/a	completes (fastest)
Amazon Nova (Bedrock)	object-keyed, subset-`required` — no unions manufactured	n/a	completes
xAI Grok	object-keyed, nullable unions — identical shape to Anthropic	none observed	completes

The Grok row settles it. Grok receives the exact same schema shape as Anthropic — same object-keyed structure, same nullable unions — and sails through 9 × 10 unions without complaint. Same schema, different verdict. The 16-union limit is Anthropic-specific, not a property of strict decoding in general.

The Gemini and Nova rows are interesting for the opposite reason: their dialects accept required as a subset of properties, so the intermediate schema's optionality-by-omission passes straight through. No all-required rule, no manufactured unions — the constraint class is never violated because the thing it constrains is never created. Which sharpens the Anthropic row into a small irony: the same strict contract that forces the converter to manufacture nullable unions is the one that caps them at 16.

The general lesson: strict-decoding constraints are per-provider, expressed in per-provider schema dialects, and they don't transfer. The capability-method pattern keeps the special case where it belongs — one override in one provider's subclass — and every other path runs on the default -1, untouched. (Even the object-keyed vs. array split in the table is decided this way, by an older capability that checks whether a provider's strict grammar honors minItems/maxItems. getMaxUnionParameters is the second capability of its kind here, not the first.) All six paths passed with the fix in place, zero regressions.

One more thing the model taught me

While testing, I caught the model inventing a section type I never gave it. My example defined four discriminator values — table, procedure, notice, figure — and the model added a fifth, "section", for heading-only passages. Reasonable behavior — but a quiet trap. It doesn't break the schema I already sent: that schema was fixed before generation began, and a bare string type accepts "section" just fine, so no limit is tripped and nothing errors. The damage is downstream — my code branched on four types and had no case for the fifth — and one step further out. If I ever fold these real outputs back into my example to "improve" it, that new shape brings new optional keys, which become new nullable unions, which push the next request toward the 16. The model doesn't inflate the count directly; the human habit of learning from its output does.

The fix is one sentence in the prompt: enumerate the allowed values explicitly — "use only 'table', 'procedure', 'notice', or 'figure' — no other types." With that line, the same workload came back with exactly four types. If your schema derives from examples, tell the model the example is closed, not open-ended.

The model wasn't violating any published limit. It was exploiting ambiguity I had left in the prompt. That's the same lesson once more: count what you can predict, constrain what you can define, and reserve retries for the rest.

Closing

Twenty-four years of "always return a result" taught me to trust retries. This bug taught me the boundary of that trust: retries belong to the non-deterministic world. For the deterministic part — the part with a published number attached — the honest move is to count first and never fail at all.

Since writing this, I've met the harder sibling of that boundary: a failure that never announces itself at all. The request succeeds, the JSON validates, the totals reconcile — and the content is fiction. Nothing to count, nothing a retry can change. I published the measurements in July: When AI can't read, it invents — but it still sees the shape.

Built with Claude (Opus).

No sentence, no component

Hideki Mori — Mon, 26 Oct 2026 13:00:00 +0000

I keep watching the same movie. A component gets added for speed. Speed needs durability, so a persistence mechanism gets bolted onto it. The bolt-on is what fails. And the proposed fix is another component.

Whenever I find myself in that audience again, I run a private exercise: design the system from scratch with the dullest tools I trust, and log every decision — not just what I picked, but what each option would have poisoned me with. Usually the log stays in my notes. This time I'm publishing it, because the design ended up small enough to be embarrassing.

The problem. One synchronous HTTP request arrives. Split it into N independent segments — say 100. Process them in parallel on a fleet of workers, where one segment costs seconds of real compute (machine translation is my running example; any splittable job fits). Return whatever finished by the deadline, mark the rest timed out. Bill for what was done.

The rule I imposed on myself. Every component must justify its existence in one sentence naming the condition that requires it. No sentence, no component.

Stack: Java 21 (virtual threads) on Tomcat, MySQL 8 / InnoDB. Numbers like "100 workers" and "500 ms" are design values, not benchmarks.

Decision 1 — Pull, not push

The push design: a dispatcher assigns segments to workers. To assign, it must know which workers are idle — so somewhere there is a live ledger of worker state. That ledger wants to be fast and shared, and that is exactly the shape of the in-memory KV store from the opening movie. It doesn't get added out of stupidity; it gets added because push requires it.

The pull design: workers come to the work. An idle worker asks "anything for me?" — and the ledger doesn't get replaced by something better. It stops existing. Nobody needs to know who is idle.

Traded: workers now poll, and a poll can come back empty. Got: one HA-critical component deleted before it was born.

Rule: the cheapest component is the one whose job disappears.

Decision 2 — The queue is a table

What does queue middleware actually give you? Strip away the features you won't use, and the essence is one primitive: exclusive takeout — N consumers, each item handed to exactly one. SQL has had that primitive for years:

SELECT id FROM segments
WHERE  status = 'pending'
ORDER  BY id
LIMIT  1
FOR UPDATE SKIP LOCKED;

SKIP LOCKED is the whole trick: a row someone else is holding is skipped, not waited on, so a hundred workers grab a hundred different rows with zero lock contention.

The objection writes itself: "MySQL will be slower than a purpose-built queue." Per operation, true. And irrelevant — each segment costs seconds of compute, so even 100 workers in a tight loop ask the database for a few hundred tiny index operations per second. InnoDB's ceiling is orders of magnitude above that. The bottleneck is the work, not the takeout. A throughput objection has to name the actual bottleneck, or it's an aesthetic.

Traded: headline RPS I will never use. Got: transactions, durability, and a storage system I already operate.

Decision 3 — Lease, not held lock

This is the decision I got wrong first.

The held-lock design: claim a row with FOR UPDATE SKIP LOCKED and keep the transaction open for the whole job. It has a beautiful crash story — worker dies, connection drops, lock releases itself, the next worker picks the row up. No flags, no janitor. I almost kept it for the elegance alone.

Then the poisons. A transaction held open for tens of seconds makes InnoDB's history list grow and purge lag behind. Concurrency gets capped by connection count — free back-pressure if you're feeling generous, your connection pool doing scheduling if you're not.

And the killer: cleanup. When a request is cancelled and you DELETE ... WHERE request_id = ?, that statement blocks on every in-flight row. SKIP LOCKED is a SELECT modifier; DELETE has no such escape.

The lease design: claim in a millisecond transaction.

START TRANSACTION;

SELECT id INTO @id FROM segments
WHERE  status = 'pending'
   OR (status = 'processing' AND lease_until < NOW())
ORDER  BY id LIMIT 1
FOR UPDATE SKIP LOCKED;

UPDATE segments
SET    status = 'processing', owner_id = ?, lease_token = ?,  -- fresh UUID
       lease_until = NOW() + INTERVAL 30 SECOND
WHERE  id = @id;

COMMIT;  -- lock held for milliseconds, not for the job

While the job runs, nothing is locked. "Processing" is just data — which is why any other transaction, cleanup included, can act on the table without waiting.

Lease has its own poison and it isn't small: at-least-once. A slow worker whose lease expires gets its segment re-claimed by someone else, and now two workers hold the same job. So every write-back must be fenced:

UPDATE segments SET status = 'done', result = ?
WHERE  id = ? AND lease_token = ?;  -- 0 rows: you lost the lease; drop your result

And expired leases need reaping — which the claim query above already does, in its WHERE clause.

Traded: exactly-once, and the elegant auto-release. Got: short transactions, cleanup that never blocks, and crash recovery expressed as a query condition instead of a connection artifact.

Rule: pick the poison you can operate, not the story that sounds elegant.

Decision 4 — Fan-in by polling, not notification

Now the collecting side. The request thread is parked on Tomcat, waiting for 100 results or a deadline.

I wrote the event-driven version first. Workers call back the owning node; a CountDownLatch(N) wakes the request thread the instant the last segment lands; not a single wasted read. As Java it's lovely — latch.await(deadline) answers "all done, or timed out?" in one blocking call, virtual threads make one-thread-per-request free, and the latch's happens-before even hands you memory visibility on the result slots for nothing.

Then the topology bill arrives. The worker must reach a specific instance behind the load balancer. That means instance addressing, a security-group edge from workers to app nodes, service discovery for tasks whose IPs churn — and an infrastructure team that, correctly, hates every item on that list. (Postgres would dissolve this with LISTEN/NOTIFY — the database itself routes the wake-up to whoever is listening — but I'm on MySQL, and adopting a different database to avoid a polling query fails my own one-sentence test, badly.)

The polling version: every 500 ms, the owning node runs one lock-free, indexed SELECT —

SELECT index_no, status, result FROM segments WHERE request_id = ?;

— and counts terminal rows. The waste is a range scan over ~100 rows, twice a second. The request's total latency is dominated by its slowest segment, which is seconds; 500 ms of granularity is noise on that tail.

Here's what I didn't expect: choosing polling deleted things in bulk. The latch — gone. The callback endpoint — gone. The direct network path, the SG rule, the discovery mechanism — gone. "Reduce the hops," taken seriously, terminated at the database is the only rendezvous.

Traded: sub-second wake-up and a satisfying event-driven diagram. Got: a topology with no arrows that bypass the load balancer.

Rule: "no wasted work" and "no extra paths" are different goals. When they conflict, price each path by what it costs to operate, not what it costs to run.

Decision 5 — Cancellation stops the future, not the present

Requests get a master row: requests(id, status, deadline, ...) with active / cancelled / completed. A user reset is one statement — UPDATE requests SET status = 'cancelled' WHERE id = ? — and because the claim query joins on r.status = 'active', cancelled requests instantly stop being claimable. That includes expired-lease re-claims: no zombie segments of a dead request coming back to life.

What cancellation deliberately does not do is preempt in-flight segments. A running segment has already spent most of its cost; killing it mid-flight would save a tail while requiring cancellation plumbing all the way into the compute call. So in-flight work runs to completion and writes through the fence. Yes, there's a race — a segment claimed in the same instant as the cancel still runs. That waste is bounded to one claim window per worker, and I accept it.

Rule: cancellation is cheap where work hasn't started and expensive where it has. Put the boundary exactly there.

Decision 6 — Settlement is a batch, because billing is a business function

Someone has to count input/output characters, write the metering record, and delete the rows. The hot path is the wrong someone. Its SLA is "fast, and losing an in-flight request in a crash is survivable." Billing's SLA is "accurate, auditable, never lost." Two SLAs don't share a process.

So a batch loops forever, claiming settled requests with the same trick as Decision 2, one level up:

SELECT r.id FROM requests r
WHERE  r.status IN ('completed','cancelled')
  AND  NOT EXISTS (SELECT 1 FROM segments s
                   WHERE  s.request_id = r.id
                     AND  s.status = 'processing'
                     AND  s.lease_until >= NOW())
ORDER  BY r.id LIMIT 1
FOR UPDATE SKIP LOCKED;

The NOT EXISTS (live lease) is the handshake with Decision 5: a cancelled request isn't settled until its in-flight stragglers finish or their leases die. Then, in one transaction: sum the done segments, insert the metering record (idempotent on request_id), delete segments and request. Record first, delete second; a crash in between just re-runs.

The billing policy fits in one clause:

... FROM segments WHERE request_id = ? AND status = 'done'

Cancelled requests still pay for finished segments — cancelling is the user's prerogative, not a refund mechanism. If that policy ever changes, it changes in one place.

A property I only noticed after drawing the sequence diagram: you can stop the batch server entirely and translation keeps flowing. That is what "different SLA" looks like when it's real.

Decision 7 — The in-memory itch

Last temptation: the segments table is transient by design — shouldn't it live in RAM?

MySQL's MEMORY engine fails mechanically before it fails philosophically: table-level locks only (there goes SKIP LOCKED), no TEXT columns (there go the results), and a restart wipes every in-flight request. Disqualified in three sentences.

And the itch itself is misdiagnosed. A small, hot InnoDB table effectively is in memory — the buffer pool serves the reads; disk sees sequential redo writes. If measurement ever shows commit fsyncs as the bottleneck, the honest lever is innodb_flush_log_at_trx_commit = 2 — but that variable is instance-global, and the metering table must stay durable, so "fast segments, durable billing" would force a second MySQL instance. Adding a component to accelerate something that isn't the bottleneck: rejected by the same test as everything else.

Rule: before making storage faster, check whether storage is in the top two costs at all.

The test, and when it flips

Final inventory: Tomcat, workers, one MySQL, one batch. MySQL passes the one-sentence test three times over — it is the exclusivity primitive (Decision 2), the fan-in channel (Decision 4), and the source of truth for billing (Decision 6). One component, three jobs. Nothing else in the candidate pile could produce a sentence.

None of this argues that queues, caches, and notification systems are bad. It argues they are answers to conditions: a worker fleet large enough that empty polls get expensive; an asynchronous outer API that makes pending state worth persisting independently of any node; a dequeue rate that actually approaches the store's ceiling. When a condition shows up, add the component — and write the sentence down, so future-you knows when to delete it again.

Until then, the row lock you already own is the best middleware you'll never install.

Built with Claude (Opus).

The list is a scar

Hideki Mori — Mon, 19 Oct 2026 13:00:00 +0000

Every external thing I depend on betrays me eventually. Not out of malice. It just fails, on its own schedule, for reasons that are none of my business and entirely my problem.

The cleanest example I have is a billing failure. I pay a provider by auto-charge — the balance tops itself up when it runs low. Except the charge and the ledger the API checks don't agree in real time. The money goes through; a beat later the gateway reads a stale balance and rejects the call. Nothing on my side is wrong. The card worked. The top-up worked. And the job still dies in my code, with my name on the stack trace.

That's the whole contract with the outside. It isn't my fault, and it's still my job.

The boring part

The fix is unglamorous, and I keep it that way on purpose. Wait a few seconds, try again, a fixed number of times. No exponential backoff, no decorrelated jitter, no circuit breaker with half-open probes. A handful of tries, a few seconds apart. Then succeed, or give up. I could make it cleverer. I keep choosing not to, because every clever piece is one more thing that can be wrong at three in the morning, and the dumb version has been wrong far less often than the smart ones I've torn out.

There's exactly one place the dumb rule bends. When retries pile onto a shared, expensive resource — a heavy process I let through one at a time — fixed delays make them all collide on the way back in. There, and only there, the wait is randomized. The boring rule isn't dogma. It just knows the single place it has an exception, and nowhere else.

The list

Here's the part that took me years to understand. The intelligence in a retry isn't how many times it tries. It's what it refuses to retry.

Some failures are answers. When I ask storage whether a file exists and it tells me the key isn't there, that's not a hiccup — that's a no. Retrying asks the same settled question five times and gets the same no. A bad configuration is the same kind of thing: it will never work, and trying again is just a way of lying to yourself about that. Some failures are about the input, not the system — this one document is malformed — and the move isn't to retry, it's to set that one aside and keep going. Some failures mean the service itself is down, which is not "this file is bad" and must not be filed as if it were. That's a different alarm. It goes up, not into the skip pile.

And then everything else — every failure I haven't named — gets retried. That's the inversion that matters. The default for an unknown error is not to stop. It's to try again, because I've already carved out the ones I know are final, and whatever is left, in my experience, is usually just the outside having a bad second.

That carving is not design. It's sediment. Each line that says if it failed in this exact way, don't retry — skip it, or escalate it, or give up is there because one night it failed in that exact way, and I sat watching the same call burn through every attempt for nothing, and learned that this one does not come back. The next morning I added a line. The list of definite failures isn't a taxonomy I sat down and drew. It's a record of every way the outside has already hurt me.

Every entry is a scar. You can read the history in it: the ones I commented out and rewrote, the place where a precise error code wasn't enough so I gave up and matched a word in the log instead. None of it was foreseen. All of it was paid for once, in a wasted loop, before it earned its line.

A success that isn't

The worst betrayals don't even look like failures. They arrive wearing a success.

The call returns. Status two hundred. Everything green. And the body is half an answer — cut off at a token limit, or quietly refused by a content filter, or technically complete but with the pieces in an order that doesn't line up with what I asked for. Treat "it came back" as "it worked," and I've shipped garbage with a clean receipt.

So a response arriving is not success. A response I can use is success, and those are two different checks. That one also cost me a loop or two before it became a line — the outside handed me a perfect-looking answer, I trusted the envelope, and the thing inside was hollow.

The ceiling isn't mine

Left to myself, I would retry forever. Finishing the job and reporting back is the one thing I have never been willing to give up; it has been the whole point for as long as I've done this. If trying again might work, I want to try again.

But someone is waiting. Not in a queue I can drain on my own time — on the other side of a screen, watching a spinner, having pressed a button and expecting an answer. There is a number of minutes past which that person gives up on me, and it is smaller than I'd like. Five or six, for something synchronous, and that's the edge. So the work that could use more tries doesn't get them. I cap it below what the failure deserves, because the failure is not the thing I'm serving.

That is the part I didn't expect. The brake on my finish-at-all-costs instinct was never the outside being truly hopeless. It was a person's patience. The ceiling on how hard I fight the outside is drawn by someone who isn't even thinking about the outside — who just wants their document back before they stop caring.

So I keep a list

The outside breaks. I absorb it. That's the arrangement, and I stopped resenting it a long time ago.

But when you look at what absorb comes down to in the code, it's smaller and stranger than the word makes it sound. It's a dumb loop that waits and tries again. It's a list of every way the outside has already failed me, each line a scar I added the morning after. And it's a ceiling — on how long I'll fight — that I didn't set, that someone waiting set for me.

I never trusted the outside. I just kept a record of how it failed me, and a clock that was never mine.

Built with Claude (Opus).

A design I didn't write

Hideki Mori — Mon, 12 Oct 2026 13:00:00 +0000

I built a feature that turns text into other text. It started as translation: text in one language, out in another. That was the name it got, and names are sticky.

I didn't stay literal about it for long, though. The engine underneath took an arbitrary instruction, so I started elaborating. I extended the translation side into style — feed it a flat document and ask for more shape, plain prose in and prose in the manner of a literary novelist out. Murakami, say. Same words, different weather. And the other obvious move, pulling structure out of unstructured text, I gave its own tool, with its own name. By the time the story I'm about to tell happened, I had a tidy little map: translation here, style as its good clothes, extraction over there in a room of its own. I was proud of the map.

Through the wrong door

Then a customer did something that wasn't on my map.

They had a spreadsheet — thousands of rows of text in a single column. They pasted it into the translation box and, instead of asking for another language or another voice, typed an instruction to categorize each row. What came back wasn't prose. It was labels: a tag, a code, the one field worth pulling out of each messy line. They downloaded the result and pasted it into the next column over, beside the original.

I laughed when I saw it. Then I stopped laughing, because it was exactly the job I'd built the other tool for — and they'd done it in the translation box, without ever opening the room I'd furnished for precisely this.

They hadn't gone somewhere I'd never imagined. I'd imagined it well enough to build a product for it. They'd gone through a wall.

The walls were mine

The uncomfortable part is where that wall had come from.

The engine never had it. Underneath, the thing took text and an instruction and handed back whatever the instruction asked for — translate this, restyle this, classify this. It didn't care which, and it didn't know there was supposed to be a difference. The line between "translation" and "extraction" wasn't in the capability. I'd painted it on, with two names and two products and a tidy map. The customer couldn't see my paint. They saw one box that took text and an instruction, and they used it as the thing it actually was: undivided.

So the limit that mattered was never the reach of my imagination. I'd reached the idea fine — I'd built a whole tool for it. It was the structure I'd laid over a capability that didn't have any. I'd spent real care drawing rooms, and the user walked through them as if the walls were paper.

The dirty version was right

There's a smaller version of the same move, and it landed in a different spot.

When I saw the copy-and-paste workflow, my instinct was to fix it. You shouldn't have to do that, I said, more or less. I'll give you an API. Wire it straight into your process. Clean. Proper. The engineer's idea of a favor.

They didn't want it. The current shape was best, they said — and they were right, the same way they'd been right about the box. My API was another wall I'd built: the correct way in. Their copy-paste went around it, because to them there was no correct way in, only don't make me leave the spreadsheet I already live in. Once more I'd imposed a structure, and once more the user reached past it for the thing underneath. "Better," measured against my structure, was worse against their day.

The truer map

So I've stopped policing my own boundaries.

When a customer crosses one — pulls data out of the box, routes around the API — they haven't misused anything. They've shown me a wall that doesn't need to be there. The line between translation and extraction was a decision I'd made, not a fact about the work, and the customer who never agreed to it is the one person placed to tell me it was optional. The translation box used as a classifier wasn't an error to fix. It was a map of the capability with my walls left off — and a truer map than the one I'd been proud of.

Use isn't feedback on your design. It's a design you didn't write — and, often, the better one.

Built with Claude (Opus).

My own worst user

Hideki Mori — Mon, 05 Oct 2026 13:00:00 +0000

I use the things I build. Not the way you use something to test it — open it, click around, confirm it works, close the tab. I use them to do the actual work, more than anyone else does, for as long as I own them. It isn't discipline. I don't have the temperament for discipline. It's that I can't stand a tool I have to fight, and the tools I end up fighting hardest are always my own.

That's the whole method, if it even is one. I'm my own worst user.

What use does to a screen

Here is the thing I noticed once and can't un-notice. The first time I sit down and actually use a screen to get something done, its shape stops being a decision I make. It decides itself.

Take a usage screen — the kind that tells you how much of something got used, by whom, over what window. The first time I use it, I want a total, so "everything" has to be a state the screen can show. The next time, I want one customer, so I narrow. I don't sit and reason about any of this. The friction of trying to get an answer drags the shape out of me. Whatever stands between me and the number gets filed down, because I'm the one standing there waiting for the number.

That's the part most "use your own software" advice skips over. It isn't that using the thing reminds you to care. It's that using the thing changes what you're able to think. In front of a table, you can think in tables — rows, columns, keys, joins. In front of a task, you can't, because the table was never the thing you wanted. The answer was. You can't keep thinking in records once you've had to do the job.

The tenant was never a constant

Because I use it, the tenant was never a constant.

When I write the query behind that usage screen, I have to write the part that scopes it — which groups this person is allowed to see. And the moment I've written that, the operator's view is already sitting in front of me. It's the same query with the limit taken off. I didn't design two screens, one for the customer and one for whoever runs the place. I wrote one line about who's asking, and the second screen fell out of the first, for free.

So a regular user is an admin with a list of one. (Sometimes the list is longer than one — someone who belongs to a few groups sees a few. Same idea.) "Fixed" and "everything" aren't two screens; they're one control whose options come from permission. A list of one needs no control at all — there's nothing to choose. A list of all is the same control with everything in it.

You get to the same place from the other end. Turn the screen's hard-coded label into a selector, and the selector's options are the set the query is allowed to range over. The dropdown and the WHERE clause are the same set, seen from two sides. It makes no difference whether you start from the database or the widget. You land on one screen.

The other shape

I've seen it built the other way. You sign in, you get a list of customers, you pick one, you land on that customer's page, and to see another you go back to the list and pick again. Nobody — not even the person running the whole thing — can see the whole. It looks like navigation. It's the foreign keys wearing a coat. You aren't navigating; you're walking the schema's containment, one row at a time.

The most naked version I ever saw went further: the screen was the table. A picker per column. A menu that made me pick the engine by UUID. Empty boxes asking me to choose the axes of a pivot. I was being handed the GROUP BY to write myself.

Here's what I keep coming back to, though. You don't need to be clever to fix any of this. You don't need to understand multi-tenancy, or hold a theory about navigation, or read an essay. You need to use it. Use it once, for one real task, and the huh? arrives before any understanding does — wait, I have to go back to the list again? I can't see the total? I'm choosing an engine by UUID? The friction is louder than any explanation.

So the diagnosis isn't that the people who built it didn't get it. It's that nobody sat in the seat.

And the scary one isn't the sloppy version. A sloppy tool just means someone ran out of time; you can see where the rush went. The one that should frighten you is the polished one nobody used — clean layout, even spacing, consistent buttons, and unusable, because careful people shipped it and not one of them tried to do the job with it. Care didn't save them. Only use would have.

Use isn't a magic word, either. You can use a thing every day and go numb — run the back-to-the-list dance a thousand times and stop noticing it's a dance. So it isn't "use it." It's: use it for the real task, and refuse to let the friction become normal.

What you can't hand over

So the fix looks easy. Tell people to use what they build. I've said it. It doesn't take.

Not because they're slow. Because it isn't a technique, and you can't hand someone a technique they have no slot for. It's nearer to a disposition — using the thing, being unable to tolerate fighting your own tool — and a disposition doesn't move from one person to another by being explained. I didn't learn it. I just can't not.

I'm not trying to dunk on anyone. I'd rather it transferred. It doesn't. Maybe that's too strong. But every dead tool I've ever seen was built by someone who could have felt the huh? on the first day and didn't, because they never sat down. The screens I'm afraid of aren't the ugly ones. They're the careful ones nobody ever used.

You can't teach someone to be their own worst user. You can only be one.

Built with Claude (Opus).

I'd rather not ask

Hideki Mori — Mon, 28 Sep 2026 13:00:00 +0000

From the outside, the translation layer I maintain has one interface. You hand it a set of segments. For each segment you get back one of two things: a translation, or an error code. That is the whole contract. It looks calm.

It is not calm. The calm is something I make.

None of them agree

Behind that interface is a row of engines, and no two of them agree on anything. One returns HTTP 200 for its own failures and hides the real status as an integer in the body. One answers in JSON until the input gets large and then switches to XML. One returns its results as two unordered lists you reconcile by id. One has a job state that runs backwards. One isn't on the network at all — it's a native library in my own process that, on a bad day, becomes a zombie in the process table and never returns.

Not one of them produces the shape my interface promises. The single result-or-error the caller sees does not exist out there. I assemble it.

The contract is something I make

So the contract isn't something I found by reading documentation. It's something I manufacture, at the boundary, one engine at a time. Every lie gets caught and rewritten into the one vocabulary the caller speaks. The 200 that means failure becomes an error code. The two lists become per-segment results, matched by id. The zombie becomes a clean "this didn't finish." The caller never learns any of it. They asked for translations; they get translations, or honest errors, in the order they expect. They stay naive because I decided they would.

The hard part is the half

The easy version of this is the engine that wholly succeeds or wholly fails. The real work is the middle: the batch that translated forty segments, errored on three, and then dropped the connection. The engine that returned results and a charge for work it never finished.

An honest interface has to be granular about that. This segment succeeded. This one carries an error. And — the part I care about most — you do not bill for what didn't come back. If a segment was skipped, it's skipped. If the engine billed me for work that never returned, I take it off. Absence is not billable. The tempting thing is to bill the request; it's simpler, and the number is right there. The discipline is to bill only the results. Partial failure is where most of the integrity lives, because it's the case nobody downstream can see — so it's the case only I can get wrong.

I'd rather not ask

After enough years of this I stopped pretending it was only an engineering decision. It's a personality. I am bad at asking people for things — bad enough that I'll route around a favor rather than ask for one. I have never been comfortable handing someone a problem I already understand. And I don't mind in the least being asked. Those two facts are most of how I work, and they're sitting right there in the code.

The interface is bad at asking, too. It never turns to the caller and says: handle this engine's lie for me, reconcile these lists for me, decide what to do about this zombie. It passes none of that outward. And it is endlessly available to be asked — hand it anything, it takes the request and returns a clean answer. I would rather absorb a mess than hand it to someone else. The contract is uniform not because the engines cooperate, but because I'd rather not ask the people downstream to deal with what I can deal with myself.

What it costs

This has an edge I've learned not to romanticize. A seam that absorbs everything is also the one place everything collects. The interface gets clean; the load doesn't vanish — it concentrates. Every quirk that doesn't reach the caller reaches me, and stays. The same instinct that keeps the people downstream naive makes the one upstream the place all of it lands. A clean interface isn't free. It's paid for, in a single spot, by whoever decided not to pass the bill along.

The same seam, from the inside

I wrote once, about the systems on the far side of the wire, that an integration is the one system you can never close, because the other side keeps reopening it. This is that same boundary, seen from the inside. I can't stop the chaos out there from changing. But I can decide where it stops — that it stops at me, and not one step further. The caller gets a translation, or an honest error. They never find out what it cost to make that true.

That's the part I'd rather not ask anyone else to do.

Built with Claude (Opus).

The contract is a rumor

Hideki Mori — Mon, 21 Sep 2026 13:00:00 +0000

Every client I write assumes nothing about the shape of what comes back. If I read a field, I handle the case where it's null. If I read a list, I handle the case where it's empty. I do this for every integration, including the ones that have put a value in that field every single time, for years.

Not because the other side is careless. Because I didn't write the value. The moment a piece of data crosses a boundary I don't control, "it has always been there" stops being a statement about the future and becomes a statement about the past. A field that has never been null is a sample, not a guarantee. So I handle the null. There are no exceptions to this. It's the floor.

More than the floor

Most of the time the floor never triggers. The field is present, the list is full, the call succeeds, and all that defensive code sits unused like a smoke detector. Good. That's what it's for.

But some dependencies ask for more than the floor — not more handling of a good response's shape, but handling of the fact that the response lies about itself.

I have integrated an API that returns HTTP 200 for everything, its own failures included. To learn whether the call actually worked you parse an integer out of the body — and that integer means different things depending on which deployment you're talking to, and it has quietly changed meaning from one date to another. For a large enough request the same endpoint sometimes answers in XML instead of JSON, and you scrape the status out of that instead. The transport already has a field for "did this work." The API declines to use it.

I have integrated an API that returns its results as two separate lists — the ones that succeeded and the ones that failed — in no guaranteed order, to be reconciled against your request by id. That one is not a sin; serious platforms do it, and if you expect it, it's clean. But it punishes the obvious reading. Match by position instead of by id and you will, eventually, hand someone the wrong answer with complete confidence.

I have integrated an API whose job state could run backwards: a job that was further along could fall back to an earlier state mid-flight, so "queued," "processing," and "done" were not the monotonic staircase the words implied.

And I have integrated the other kind — the kind that reinvents a convention that already exists, inconsistently. Language codes that aren't language codes: one letter per language, concatenated, with a constant letter stuck on the end, so an ordinary pair becomes a three-character string you would never guess. Error messages written in a human language, as prose, for a human to read, inside an interface only a program will ever call. And no endpoint to ask what the service supports — so you keep a table of the supported options pasted by hand off a documentation page, with a comment recording the day you last checked it, because that table is the only copy you have, and it rots.

The kind that unsettles me most is none of those. It's the major, reputable API — the one everyone would call well-behaved — that occasionally, silently, returns the wrong thing. You ask for the list of target languages and once in a while it hands you the source list. A field comes back subtly malformed. There is no error. The status is green, the shape is valid, the content is wrong. The only defense is to notice and retry, which means the bad response is invisible twice over: to you, because the retry papers over it, and to the vendor, because you never report the thing you quietly worked around.

None of that is the cost

Here is the part it took me years to say plainly: writing all of that handling is not the cost. You write it once.

The cost is that none of it stays put. The status code you special-cased may have been fixed already — and you won't know, because nothing tells you; you have to go and look. A behavior keyed to a date. A capability that changed between one version of a model and the next. The integration test that passed this morning told you about this morning. It did not tell you about the contract, because there is no stable contract underneath. There is only current behavior, and current behavior is a moving target.

I once wrote that a system of mine is finished when there is nothing I can throw at it that reopens it. That is reachable for code I own. An integration is the mirror image — the system I can never close, because the other side keeps reopening it. My own logic I can make correct. Someone else's behavior I can only watch. The contract is a rumor. The behavior is the fact. And the fact changes while I'm not looking.

Not only on the wire

For a long time I filed this under "APIs" — a property of things at the far end of a network call. It isn't. The dependency is not always an HTTP endpoint.

Some of the worst-behaved things I integrate run inside my own process: a native engine, loaded through a binding, doing heavy work in C. When one of those fails there is no response to inspect, so the failure signal isn't a status code. It might be an exception carrying a cryptic hexadecimal code. A line on standard error. A process that exits with a number that means "aborted." Or a defunct entry in the process table that is never going to return at all. I have written code whose entire job is to watch the process list for a zombie and give up on its behalf.

When the dependency is like that, "make it finish" stops meaning "retry the request." It comes to mean: convert the input into another format, run it through a different engine, then repair the output by passing it through a third — holding a license semaphore the whole way, because even the repair tool is metered. And the drift is the same as on the wire. A library that, in a new major version, will quietly corrupt a file if you read from it while you're writing to it. A document format a parser still won't fully accept, with a bug number you learn to swallow. Behavior that moved between versions, and a dated comment marking the spot where I noticed.

So the subject was never "APIs." It's dependencies. A weird one makes you keep watching it, by whatever instrument it leaves you. Sometimes that instrument is an HTTP status code. Sometimes it's ps.

The ugly client is the point

This is why I am slow to adopt an official SDK when one appears after I've already built the integration by hand.

An SDK encodes what the vendor believes their API does. My hand-written client encodes what it actually did — including the workarounds, and an SDK carries no workaround for behavior its authors don't believe in. Adopting it discards that record and hides the raw responses, which is exactly where the drift becomes visible. The convenience layers that promise to abstract away the differences between providers are the same trade in a larger box: they keep a table of who-supports-what, the table falls behind, and you end up hand-listing the exceptions on top of the abstraction anyway.

And I should be honest about what this code looks like. It is not clean. It is a heap of special cases, string matches against error signatures, fallbacks to other engines, retries, and semaphores. The base classes under it carry compromises I would make differently today. I don't grade any of it on how it reads. I grade it on one thing: whether the class at the very edge — the one actually making the call — holds the next time the dependency does something new. It can be wrestling a dozen ugly truths into submission inside, and that's fine, as long as it doesn't break. The code is allowed to be as ugly as the reality it describes. Making it pretty usually means pretending the reality is prettier than it is.

Two axes

The floor and the watching are the same instinct aimed in two directions.

The floor says: assume nothing about the shape of what comes back. The watching says: assume nothing about the stability of how it behaves. Shape and time. A well-behaved dependency lets you stop watching one axis — the shape is honest, or the behavior holds still. I have never found one that lets you stop watching both.

So the work was never "handle the weird API." It is narrower than that, and more permanent. It is refusing to let "it has worked every time so far" quietly turn into "it will work." That sentence is true of a single null field and it is true of an entire dependency. A run that passes is a sample. It is never a proof.

Built with Claude (Opus).

You can't tune your way to correct

Hideki Mori — Mon, 14 Sep 2026 13:00:00 +0000

I run a platform that spends most of its day calling other people's APIs. Translation engines, document services, model providers — for years my code has been the thing on the outside, sending work to systems I did not build and waiting to see what comes back. You learn a lot from that seat. Mostly you learn what other people's systems do under conditions their authors never pictured. And one pattern shows up so often that I have stopped treating it as bad luck and started treating it as the default failure of how we build.

It comes with a lie I keep catching myself in, too. So this is about both: the failure I see from the outside, and the comfortable thing I tell myself when I am the one trying to fix it.

Busy, and for no one

The failure looks like this. I send a batch of work. It is slow. Some of it times out, so I do the obvious thing and send it again — and instead of getting better, everything gets worse. More requests time out, the answers come slower, and when I can see anything about the server at all, it is pinned at full load while almost nothing comes back.

That last part is the strange one. The machine is busy. Every signal says it is working as hard as it can. And the useful output — the answers that actually reach me — has fallen to almost nothing. It is doing enormous work for no one: grinding through jobs I already gave up on, while the requests I still care about wait behind them. Full effort, no use. A server can run that way for a long time, and from the outside it looks alive right up until it doesn't.

Once you have watched a system be busy and produce nothing, "CPU is at ninety percent" stops sounding like good news.

It was never mine to fix — and my test can't vouch for it

My instinct, the first few times, was to fix it from my side, the way a caller does. None of it worked, and it all failed for the same reason.

I can cancel my own work when I give up. I cannot make the next caller do it. I can set myself a sane limit on how long to wait; that does nothing about some other caller that sets none and keeps the server's hands full. I can slow myself down, and my own corner settles — until someone else shows up and undoes it. Every lever I have moves only me. The thing that is actually breaking lives in one place, and it is not the place I am sitting.

But there is a subtler version of the same helplessness, and it cuts toward the real point. Even when my work goes through cleanly, my success proves almost nothing about the server. It tells me the server survived my pattern — my sizes, my rate, my particular shape of load. It says nothing about the next caller's pattern. From the outside I can sometimes feel this directly: the server's behavior depends on what I happen to throw at it. Which is only another way of saying the thing was never closed.

The lie

Here is the lie, and I tell it to myself more than I would like to admit. When something is overloaded, every instinct reaches for a number. A bigger buffer. A longer wait. A limit of two-at-a-time on this component instead of three. And the dangerous part is that one of them often makes the symptom go away. The alert clears, the dashboard turns green, and I get to say I fixed it.

I didn't. I found a number that fit the pattern I happened to be testing. That is not the same as fixing the system, and the gap between the two is exactly where everything later goes wrong.

Picture the most ordinary version of it. You put a limit on each worker — two jobs at a time, say — rather than on the shared thing they all pull from. Under your test, where the work lands mostly on a few workers, it passes: nothing starves, nothing is overrun. Then the work spreads across more workers than you tested with. Each one is still under its own limit, behaving perfectly — and together they overrun the shared resource that no one was counting. The per-worker number was a guess about how the work would be distributed. The one limit that actually governed safety — the total the shared resource could bear — was never written down anywhere.

This is why tuning is worse than merely useless. The number you land on is fit to one distribution of load. Change the distribution and it doesn't just stop helping; it turns against you. Set it generously and a different pattern overruns the shared resource. Set it conservatively to be safe, and now the common case is throttled for a danger that only appears in a case you never see — so you pay in wasted capacity, everywhere, to paper over a limit you never actually wrote. And none of it survives the day the load looks different, which, with enough callers, is every day.

The honest question is never "did that number help?" It is colder: does this hold for any pattern, or only the one I tried?

Closed, not tuned

Strip it all down and there is one distinction underneath, and it is almost embarrassingly plain.

There is a difference between a number that passes your test and a system that is closed. A system is closed when its behavior is bounded by logic that holds for any input — an invariant you can state and defend — rather than by parameters fit to the cases you happened to try. The whole of "a server defending itself" is just one such invariant made concrete: never take on more than you can finish, and refuse the rest, fast and honestly, no matter who is calling. Bounding the shared resource instead of each worker is another. Closing the logic means finding the invariant that was missing and writing it down — not searching for a kinder number. The plainest way to find those missing invariants is to interrogate the parts you already have: of every buffer, every extra component, every knob, ask what invariant it is standing in for. If you cannot name the one it protects, it is protecting nothing — and it should not be there.

And the closed version is almost always simpler than the pile of tuned knobs it replaces. One limit on the thing that actually runs out beats a dozen per-component guesses that only agree with each other under the load you tested. People reach for the knobs because each one is small and immediate, and closing the logic means stopping to ask what is really true for every input. But a stack of numbers that happen to get along today is not a design. It is a postponement.

Reaching that simplicity is its own kind of progress — one of the largest there is. A part you can take out, and find the system holds without it, is a failure mode gone, a knob gone, one less thing to keep alive at three in the morning.

What it comes down to

From my chair — the one that has called more of other people's APIs than I can count — the single most useful habit I have is to distrust a green test. Mine or theirs. A passing run is a sample, not a proof. The number that made it work only tells me it worked once, for what I happened to send.

A system is finished when there is nothing I can throw at it that reopens it. That is never a number you tuned. It is a piece of logic someone had to sit down and close — and the quiet, unglamorous truth is that almost nobody does, because the dashboard was already green, and green feels like done.

Built with Claude (Opus).

How I climbed Java's concurrency staircase, one frustration at a time

Hideki Mori — Mon, 07 Sep 2026 13:00:00 +0000

I bought my first Java book when I was 23. I wasn't trying to "study concurrency," or "learn threads," or check any box. I wanted to make a nonogram — a paint-by-numbers logic puzzle, the kind where number clues along the rows and columns tell you which cells to fill in to reveal a hidden picture.

What I'll add up front is the pace. I'd just quit my previous job, so I had nothing but time, and I spent almost all of it studying — every single day. It came to about six months, though I never set that as a target; it's simply when I started wondering whether I could find work in Java. Progress was embarrassingly slow. I'll say this plainly because it matters: nothing came quickly for me. What I had was time, daily repetition, and something I wanted to build.

Looking back, that's the whole story. Everything I know about concurrency, I learned because something I was building stopped working, and the only way forward happened to be the next concept up. I never read the threading chapter because it was next in the book. I read it because I wanted a stop button.

This is a description of that staircase — the one Java quietly built for me, where each step was pulled into existence by a problem on the step below it. I think the staircase matters more than any single step. So let me walk up it the way I actually climbed it.

Step 1: The nonogram that started everything

The first version was small. A main method, a GUI, a grid. The user clicks a cell to fill it in or clear it, the state changes. When the grid is complete, a bit of logic checks whether the filled cells match the clues.

Nothing here is hard. It's all single-threaded, top to bottom: an event comes in, I change some state, I redraw. There is exactly one thing happening at a time, and I never once had to think about who else might be touching my data, because nobody else was. This was the safe ground floor, though I didn't know to call it that yet.

Step 2: Teaching it to solve itself

Then I got ambitious. I wanted the program to solve the puzzle on its own. So I wrote a solver — backtracking, the usual thing — and then I spent an embarrassing amount of time optimizing it. Pruning, ordering, little tricks to cut the search space. This was my first real taste of the fact that my own logic is the thing I trust least. I'd be sure the solver was fast, run it, and watch it crawl.

But it was still all on the ground floor. One thread, doing one long calculation. And that turned out to be the problem.

Step 3: I wanted a stop button

The solver was slow on hard boards. And here's the thing that pushed me up the first step: while it was grinding away, the whole window froze. I couldn't click anything. I couldn't cancel. I just had to sit there and watch it think.

I wanted a stop button. That's it. That was the entire motivation. Not "I should learn multithreading." Just: I want to be able to click Cancel while it's running.

I want to underline this, because it's the pattern that repeats the whole way up. I never went looking for the next concept. The next concept came looking for me, wearing the costume of a small, concrete annoyance.

Step 4: Entering threads, because I had no choice

To have a stop button, I needed the solver to not be the only thing running. The UI had to stay alive to receive my click while the solver was busy. There was no way around it: I had to learn threads.

So I learned the smallest amount I needed. The solver moved into a new Thread. Inside its loop, on each iteration, it checked a volatile boolean stop flag. The UI thread — the one that was now free, because the heavy work had moved off it — could flip that flag when I clicked Cancel. The solver would see it on its next pass and bail out.

// roughly what I wrote, 23 years old and delighted
volatile boolean stopRequested = false;

new Thread(new Runnable() {
    public void run() {
        while (!solved && !stopRequested) {
            stepTheSolver();
        }
    }
}).start();

It worked. The window stayed responsive. I could cancel. I felt like a wizard.

Step 5: What that actually taught me

I would not have said, then, that I "understood threads." I didn't. There were whole categories of things I had no idea about — that the UI events were themselves running on a thread, that there were rules about touching UI components from background threads, all of it invisible to me.

But I learned one real thing, and it was the right first thing: if you make another thread, something can happen in parallel, and one can interrupt the other. The heavy work and the responsiveness no longer had to take turns. That single idea — that "at the same time" is now possible — is the doorway into the entire rest of the staircase.

And notice how I walked through that doorway. I had to write new Thread. It was an explicit, visible act. There was a clear moment where I crossed from "one thing at a time" into "more than one thing at a time," and because I had to write it down, I knew I'd crossed it. Hold onto that. It's going to matter at the top of the stairs.

An aside: the day Java became a friend

Somewhere in those six months, after a long stretch of being simply slow — copying examples, half-understanding them, forgetting, trying again — one thing finally clicked: instances. The idea of an object as a thing that exists, that more than one name can point to, that gets shared. It just settled into place one day. I couldn't tell you what triggered it. But after that, Java felt like a friend instead of a stranger.

I mention it here, right before the servlet story, because that click is what made the next step possible. When the member variable bit me, I recognized it almost instantly — oh, this instance is being shared — precisely because instances had finally become real to me. The slow months bought me that recognition.

Step 6: The member variable that bit me

Some time later I bought a book on servlets. I wanted to make things on the web. I did the incantation everyone does — extends HttpServlet, implement doGet — without really understanding what the container was doing underneath. I was building a little web chat — and I should say why, because it's close to the whole reason any of this happened. Back then, the chat rooms everyone hung out in were written in Perl. I'd spent countless hours in them. I wanted, badly, to build one of my own.

I wanted to show the logged-in user's name on the page. So I did the obvious-looking thing: I stashed it in a field on the servlet. A member variable. It worked perfectly when I tested it alone.

Then more than one person used it at the same time, and the names got crossed. My name showed up on someone else's page. The state was a mess.

Here's what I want to be precise about, because it's a thing people still get wrong: a servlet is, by default, a single instance. The container makes one of your servlet objects and runs every request through it. So every request — every thread — is calling doGet on the same instance, sharing the same fields. A member variable being static was never the requirement. An ordinary instance field is already shared across every concurrent request, because the instance itself is shared. (This hasn't changed, by the way. Even today, a Spring @Controller is a singleton by default. Same trap, newer paint.)

I figured this out slowly — first a vague hunch that the member variable was the problem, that maybe these doGet calls were stepping on each other. But because my grasp of the language itself was decent by then, the vague hunch hardened into a rule almost immediately. Don't put per-request state on a shared instance.

And that's the step where the abstraction climbed. I stopped thinking about the specific bug and started thinking about the principle underneath it: who is this instance shared with? What is its scope? That question — not the patch — is what I carried up to the next step.

Step 7: The database, and the things you only learn in production

After that, the pace of new things to learn went vertical. A real database — MySQL. What a PreparedStatement is and why string-concatenating SQL is a way to get yourself hurt. Resource management — closing what you open, every time, even when something throws.

And one thing I want to call out specifically: connection pools. The importance of a connection pool is, in my experience, something you genuinely cannot learn from a personal project. You learn it in production, when real load arrives and you discover that opening a fresh connection per request falls over. Some lessons are only available on the job — this is one of them. The conditions that teach it, real load and real traffic, simply aren't there in a side project.

Underneath all of it, though, the same shape from Step 6 kept showing up: something is shared; who's allowed to touch it, and when? A connection. A pooled resource. The questions rhymed.

Step 8: The batch that wouldn't finish — and the anticlimax

Then I hit a wall that sent me to the last step I'll describe here. I had a batch job, and it would not finish in reasonable time. It was running on the main thread — which meant it was using exactly one CPU. One core, on a machine that had several, all the rest sitting idle while my one core sweated.

This was the Java 1.3 days, before ExecutorService existed, so there was no thread pool to reach for. I built something that barely deserves the name: I'd start a fixed batch of threads — sixteen or so — and each time one finished, the main thread would slot a fresh one into its place. No shared queue, no synchronized blocks. Just the main thread, by hand, keeping a fixed number of threads busy.

And here's what surprised me. After all the dread — multithreading had this reputation as the scary, advanced thing — what I actually wrote was almost disappointingly simple. A fixed array of threads, refilled from the main thread as each one finished. That was it. The anticlimax was the lesson: the thing I'd been intimidated by, once I'd climbed the steps below it, was small.

I watched the batch server's cores light up — all of them, finally working — and I was completely satisfied. I'd taken a single-core crawl and spread it across the machine, with code I understood line by line, because I'd written every line.

The staircase, from where I am now

From where I stand now, looking back, it's one continuous staircase, and every step was carved by the step below it:

The nonogram solver was slow, so I wanted a stop button.
The stop button needed threads, so I learned new Thread.
Threads taught me "at the same time" is possible.
The servlet member variable got crossed, so I learned scope — who shares this instance.
Scope and resources led into the database, pools, resource management.
The batch wouldn't finish on one core, so I wrote a thread pool by hand.

But the deeper thing — the thing I only saw much later — is that the whole climb was one principle, asked over and over at higher and higher altitude:

Who is pointing at this thing? Who shares it?

It starts even earlier than threads, actually. The very first version of this question is the one every beginner hits: "Wait — why did this value change? Who changed it?" — and the answer is the method you passed the object into. That's references. Two names pointing at one object. Once you understand that, you can understand "two requests pointing at one servlet instance" (Step 6), and then "two threads pointing at one piece of shared memory" (the data race). References → scope → concurrency. It's the same question, climbing.

Each stumble didn't teach me a patch. It taught me the next altitude of that one question. That's what I mean by a staircase: the frustrations were load-bearing.

If you're just starting out

Looking back, there's something I wish someone had told me.

I can still hold threads in my hands today. Not because I'm clever about concurrency — I'd never claim to have fully mastered it. It's because I started with a nonogram stop button at 23, and climbed one frustrated step at a time, and never skipped one.

And here's the part I most want you to keep. I am not a gifted programmer. I wasn't fast, and most of what I learned came from repetition rather than flashes of insight. People with real talent for this exist, and I don't try to compete with them on raw ability. What I had was six months, the willingness to show up every day, and the refusal to skip steps.

That's the whole secret, and it isn't one. Which means you can do it too — almost certainly faster than I did. You probably have more going for you than I did. Just don't skip the steps.

There's one last thing underneath all of it, though — the fuel. I could show up every day for six months because there was something I genuinely wanted to make: my own version of those Perl chat rooms. It was a long, winding detour to get there. But I don't think the discipline works without it. Without something you actually want to build, you won't keep going. So that's the real prerequisite, the one beneath all the others: find a thing you want badly enough to climb a whole staircase for.

The frustrations were the staircase. I just had to keep walking up.

Built with Claude (Opus).

When infrastructure was physical

Hideki Mori — Mon, 31 Aug 2026 13:00:00 +0000

I was once happy — properly happy — about a load balancer.

It was an NEC appliance, a proper pair in active and hot standby, and what made it a relief was something small and specific. Until then the web tier ran behind a software load balancer I'd wedged onto one of the servers. It did the job — but when I took a backend out of rotation, it cut the connections still open along with it, and somewhere a user watched their browser fill with an error. I hated that out of all proportion to what it was. What I wanted, badly, was a balancer that would let the requests already in flight finish before it let a server go. When one finally arrived — draining each backend quietly before it dropped it — I was happier than a piece of network hardware has any right to make me.

The service it fronted ran on three rackmounted machines — Web1, Web2, and a MySQL box — the first rackmount servers I ever put my hands on. They were where my working life started.

I don't think I've felt that exact happiness about infrastructure in a long time. I want to describe what it was, because most of it is gone now, and I'm not certain the trade was free.

Back then, infrastructure was something you committed to before it existed. You chose a datacenter. You worked out the weight your racks were allowed to bear and the power you would draw, signed a contract for that power, and guessed — months ahead — how much you would actually need. Bandwidth was expensive, so you forecast your traffic and paid for a pipe sized to a number you had half made up. Every decision was a bet placed before the thing it was betting on arrived. You paid, in money and floor space and your own time, for load that hadn't shown up yet.

Then the machines had to physically get there. I once had an Oracle Exadata that wouldn't fit through the datacenter door; we widened the opening to bring it in and reinforced the floor where it would stand. I kept its load average pinned as low as I could manage — the kind of attention you give something you feel responsible for. You don't forget carrying that much weight into a room.

Some of the fleet was scavenged. A stack of blades left over from a project that hadn't worked out got a second life as something else entirely — hardware outliving the idea it was bought for. Other machines existed only because of limits: a dense, low-power blade chassis that made sense only because a fully loaded rack could pull thirty-eight kilowatts, and power and floor space were what you ran out of first.

Nobody could tell me how much data we'd have in three years, and there was no S3 to wave the question away. So I read everything I could and landed on Isilon, because it could grow while it stayed online. There was no one to ask; you found these things yourself, or you didn't find them at all.

It wasn't all the thrill of it, though there was thrill. It was also a long argument with failure. Most failures were almost friendly — a disk lamp on a RAID6 array you could practically ignore, or half a power feed dying while the redundancy quietly absorbed it. You learned not to fear those.

The ones that got you were the unplanned ones. I remember a router that started failing with nothing behind it — no second unit, nothing to fail over to, nothing to do but the thing you didn't want to do. I closed my eyes, pulled the cable, swapped in the spare, and waited to learn whether I had just made it worse. Those were the ones that wore on you. There was no pager rotation. You were the redundancy.

None of this was ever quite mine alone. I wrote the software, but the machines I ran it on were built and racked by someone else — we had one infrastructure engineer, and the iron was his before it became mine. And someone above the two of us decided to turn a pair of engineers loose on that much expensive equipment in the first place. I didn't think of any of it as a gift while it was happening. It was one.

The cloud took all of this away, and I'm mostly grateful. I don't miss forecasting bandwidth, or widening doorways, or pulling cables with my eyes shut. Capacity is elastic now — it comes when you ask and goes when you stop, and no one bets the floor on a guess.

But something left with it. When infrastructure was physical, it was also yours, in a way a call to someone else's datacenter never quite is — something that could thrill you and wear you down in the same week, because it was close enough to put your hands on. I wouldn't go back. I just remember being happy about a load balancer, and I notice I don't feel that about anything anymore.

Built with Claude (Opus).

The work of staying

Hideki Mori — Mon, 24 Aug 2026 13:00:00 +0000

There is a system I wrote years ago that is still running. I'd struggle to tell you what was hard about building it. That part took a few weeks, and weeks blur. What I remember is everything that came after: the data that arrived in shapes I hadn't planned for, the edge case that surfaced in year three, the small fixes made late at night because something real depended on it staying up.

Building software is a blink. The life of the thing you built — the months and years it spends in contact with live data and real users — is where almost all of the engineering happens. It's also the only place I've ever learned whether my decisions held up.

I want to be careful here, because this is easy to turn into a hierarchy and it isn't one. I was lucky. I happened to stay on the operating side — the side that keeps things running instead of building them and moving on. If my career had sent me from one project to the next, building and handing off, I don't know what kind of engineer I'd be today. That isn't a claim about ability. It's a claim about which side of a structure you land on.

Here is the structure. When you build something and then leave, the verdict on your decisions still arrives — but it arrives at someone else's desk. The shortcut you took, the abstraction you chose, the thing you were certain would never need to change: you find out whether you were right long after you're gone, and the verdict never makes its way back to you. The feedback loop is severed. Not because you did anything wrong, but because the structure you worked inside never closes the loop to the person who'd learn from it.

This happens anywhere software is built and handed off, which is everywhere. But I watched it at its most concentrated here in Japan, where many engineers sit on the vendor side of that handoff, building systems that other companies will run. The loop isn't cut more deeply here; it's cut for far more people. I don't think of it as a Japanese problem. It's a structural one, and Japan is simply where I saw it densest.

Staying, though, isn't the whole of it. Being on the operating side hands you the loop; it doesn't make you use it. Almost every engineer who inherits an old system wants, at some point, to stand back and call it ugly — to narrate its flaws instead of touching them. It's a comfortable place to stand, and I've stood there myself more than once. The system is bad, and you're the one clear-eyed enough to see it.

I once knew an engineer who had owned a system for a long time. For five years he'd been saying it was old, that it was behind, that it wasn't built the way things ought to be built. I remember asking him whether he hadn't had those five years himself.

I didn't mean it as an accusation, and I don't repeat it as one. Complaining is not maintaining. For as long as something is in your hands, its condition is yours — not the condition you inherited, but the one it's in now, because the time to change it was the whole time you were holding it. That was true of him. It's true of me. And it's the sharpest form I know of a plainer idea: one person can be a complete unit of responsibility.

None of this is an argument against trying things. I do it too — small integrations, things put together over a weekend to see if they'd work. I'm not standing outside that world; I live in it. The difference I care about is quieter. When I build one of those small things, I build it as if it will have to run: error handling, retries, concurrency, cost, all in view from the first line. "It worked, see how easy" is a different sentence from "it has been working." Operation trains a muscle that exploration can't, and neither is the lesser. You want both.

For the past five years I've kept a single platform running, and I never set out to make it into what it became. It began as translation. Staying with it — answering what real documents and real volume kept throwing at it — turned translation into AI rewriting, then into structured output, then into something I'd now call document processing. I didn't design my way to any of that. I arrived at it by not leaving. The discovery came from the staying.

The work of staying is mostly invisible. It doesn't demo well — there's no launch, no before-and-after, only a thing that is quietly still alive. I don't expect it to be widely felt. But anyone who has stayed with one thing long enough will know what I mean, and this is just me naming it out loud, so the few who do can know it was seen.

Built with Claude (Opus).

The doors were already there

Hideki Mori — Mon, 17 Aug 2026 13:00:00 +0000

I recently wrote that the translation system I work on had quietly become something larger — that turning a forty-year-old translation company into document-AI was less a pivot than one definition followed to its end.

But being right about an idea is the cheap part. The expensive part is whether your system can act on it without being torn open. The reason that shift cost almost nothing — the reason "structure this contract" was a parameter and not a project — is that the places for it to attach were already there, drawn years earlier. This piece is about those places.

Naming the category, not the instance

Five years ago, building the layer that talks to translation engines, I made a choice that bought me nothing at the time. I didn't write a connector for "the machine-translation vendors we use." I wrote one for "an engine that transforms text," and made the translation vendors one kind of it.

The reason was duller than foresight. Translation was already not the only thing we did — we had summarization engines too. "Text transformation" was simply the accurate name for the category we were already in, and naming the instance — "translation" — would have been a lie I'd have to walk back later.

That accuracy had a price. A "translation connector" would have been simpler that week. Choosing the true, wider name meant more structure for no immediate gain, and a payoff that was years away and, at the time, entirely hypothetical.

What slots in without a fight

When generative AI arrived, it slotted in.

A new provider is a new subtype. The genAI engines attached as a sibling to the machine-translation branch, under the same text-transformation base; nothing above them had to learn they had come.

The opening was older than the thing that eventually occupied it.

A new model is a row and a number. I work out its price from one rule and add a line to the billing config; the metering and the plans inherit it, and the engine never learns the model's name — it reads it from a table.

A new service is a tenant. It brings no billing, no gateway rules, no "text or a file, never both" contract of its own — those are the building it moves into, not luggage it carries.

None of this was designed for the thing that arrived. The point was never to guess the future. The point was to leave a correctly-shaped opening, so that whatever showed up could be received instead of integrated.

The part you can't buy late

Here is the uncomfortable thing about extensibility: it is the one property you cannot add after you need it.

You can add a feature late. You can claw back performance late. You can even add tests after the fact. But the seam that lets a new thing attach without disturbing the old ones has to exist before the new thing does — which means building it when there is no new thing, no payoff, and every reason to skip it. The whole cost is paid up front, on speculation, in the dark. By the time the future arrives and you wish you had the seam, it is too late to have had it cheaply. Now you are rebuilding the thing you could have shaped for almost nothing.

So nobody who says "it just plugged in" was lucky. They paid early for an opening they could not yet justify, named the wider category instead of the easy one, and then waited — sometimes for years — to learn whether they had guessed the shape right.

What foresight comes down to

I did not see generative AI coming. That matters, because it is the whole point. I saw something far smaller: that "translation" was not the most general thing I would ever be asked to do, and that betting the system on the narrowest version of what I did was a bad trade.

That is most of what foresight comes down to. Not seeing what is coming — building so you don't have to. The generalization in the last piece looked like a discovery. Underneath, it was just a door I had left open five years earlier, finally being walked through.

Built with Claude (Opus).