DEV Community: speed engineer

Your Average Latency Is Lying to You: The Tail at Scale

speed engineer — Mon, 06 Jul 2026 04:40:44 +0000

The problem

Here's a service that looks healthy. Median response time is 10ms. The dashboard shows an average around 20ms. Nobody's paging you.

Then you put it behind a request that fans out to 100 of these services in parallel — a search, a feed build, a page that queries 100 shards and needs every answer — and suddenly two-thirds of your users are waiting more than a second.

Nothing got slower. You just discovered that at scale, your p99 is the number that matters, and your average was hiding it.

Why it happens

Give that "healthy" service a fat tail: 99% of requests return in 10ms, but 1% take 1000ms (a GC pause, a cold cache, a lock, a TCP retransmit — pick your poison). The average is 0.99 × 10 + 0.01 × 1000 ≈ 20ms. Genuinely fine on one call.

Now fan out to N servers and wait for all of them, because you need every shard's answer before you can respond. The request is fast only if every server dodged its slow path. The probability that at least one server hits its 1% tail is:

P(slow request) = 1 − (0.99)^N

Watch it move:

N =   1  →   1.0%
N =  10  →   9.6%
N = 100  →  63.4%
N = 200  →  86.6%

At a fan-out of 100, the slow path that happens 1% of the time on a single server becomes the common case for the user. The tail didn't add up — it compounded. This is what Jeff Dean and Luiz Barroso named "the tail at scale": in fan-out systems, the 99th-percentile latency of a component is effectively the median latency of the whole request.

The trap is that every instinct points the wrong way. You optimize the average — shave p50 from 10ms to 8ms — and the user-facing number barely moves, because the user-facing number is built out of tails, not medians.

What to do about it

Attack the tail, not the average. Improve the per-server slow rate and the compounded number collapses. Take that 1% down to 0.1%:

N = 100, p = 1%    →  63.4%
N = 100, p = 0.1%  →   9.5%

A 10× improvement in the tail buys a ~6.6× improvement for the user. The same 10× on the median buys you nothing. So put your SLOs on p99/p99.9, not the average, and hunt the specific causes of tails: GC pauses, cold caches, head-of-line blocking, one slow disk in the fleet.

Send a backup request (hedging). You don't have to make every server fast — you have to stop waiting on the unlucky one. Fire the request to one replica; if it hasn't answered by, say, the 95th-percentile expected latency, fire a second copy to another replica and take whichever returns first. Because you only hedge the slow ~5%, you pay a few percent extra load and cut the high-percentile tail by roughly an order of magnitude. It's the highest-leverage trick in the tail-tolerance playbook.

Cancel the loser. Hedging without cancellation just doubles work under load and can push you into a spiral. "Tied requests" fix this: send to two replicas, tell each about the other, and whichever one starts executing cancels its twin. You get the latency win without paying double.

Reduce the fan-out. 1 − (1−p)^N is exponential in N. Fewer, fatter shards beat many thin ones for tail behavior. If you can answer from 10 servers instead of 100, you've moved from 63% to 10% for free — before any hedging.

Key takeaways

The average latency of a component is nearly useless once you fan out. In a request that waits on N servers, the user experiences the tail, not the median.
The math is 1 − (1−p)^N, and it compounds fast: a 1% per-server slow rate becomes 63% at a fan-out of 100.
Optimizing p50 is mostly wasted effort for user-facing latency. Move your SLOs and your attention to p99/p99.9.
Hedged requests (fire a backup after the 95th percentile, take the first answer) cut tail latency by ~10× for a few percent more load. Cancel the loser so you don't double the work.
Fewer shards = smaller N = smaller tail. Reducing fan-out is the cheapest tail fix there is.

Debugging Is a Search Problem: Bisect Everything

speed engineer — Sun, 05 Jul 2026 05:08:50 +0000

The problem

Two engineers get the same bug: a value that's correct when it enters the system and wrong when it leaves. One fixes it in fifteen minutes. The other burns the afternoon. Same tools, same access, same intelligence.

The difference isn't knowledge of the codebase. It's that one of them is running a binary search and the other is poking around.

Most engineers debug linearly: start where the error showed up, read the nearby code, change something, re-run, repeat. That's O(n) in the size of the problem — and worse, because most of those changes produce no information, so it's often not even that good.

The senior move is to treat every bug as a search over a space of possible causes, and to make each observation cut that space roughly in half. Ten halvings cover a thousand candidates. That's the whole trick, and it generalizes far past git bisect.

Why it happens

A bug lives somewhere in a space you can't see all at once. That space has several axes:

Time — which change introduced it.
Code — which module, function, or line.
Layers — at which stage of the pipeline the data went bad.
Input — which part of the payload triggers it.

Linear debugging searches one axis by hand — usually the code axis, starting at the symptom. But the symptom is where the bug surfaced, which is rarely where it lives. So you read the wrong neighborhood and mutate code hoping to get lucky. Each speculative edit that "might fix it" gives you almost nothing when it doesn't.

Binary search wins because information compounds. If you can find one observation that's true on one half of the space and false on the other, a single measurement deletes half your candidates — no matter which half.

What to do about it

Pick the axis with the cleanest before/after boundary and halve it.

Time — automate git bisect. If it worked last release and breaks now, you don't need to read the diff. Write a script that exits 0 on good and 1 on bad, and let git run the search:

git bisect start
git bisect bad                 # current commit is broken
git bisect good v2.3.0         # this release was fine
git bisect run ./repro.sh      # git binary-searches for you

Across 1,000 commits that's ~10 runs, not 1,000. The output is the exact commit that introduced the bug — often the entire investigation.

Layers — check the middle of the pipeline. Data flows A → B → C → D → E and comes out wrong at E. Don't start at A or E. Log the value at C. If it's already wrong at C, the bug is in A–C; if it's fine, it's in C–E. One print statement halved the system.

Input — shrink the repro. A 4,000-line request fails. Delete half. Still fails? Delete half of what's left. Passes now? Put it back and cut the other half. In a dozen steps a 4,000-line payload becomes the three fields that actually matter. This is delta debugging, and it works on config files, CSS, and feature flags — anything you can chop.

Code — disable half. Comment out half the handlers, half the middleware, half the plugins. Bug gone? It was in the half you removed. This feels crude; it is faster than reading.

The discipline underneath all four: each step's job is information, not a fix. Before you touch anything, ask "what's the one observation that splits this in half?" If a change can't tell you which half the bug is in, you're not searching — you're guessing.

Key takeaways

A bug is a point in a search space, not a line of code. Your job is to shrink the space, fast.
Every good observation halves what's left. log2(n), not n — ten steps cover a thousand suspects.
Bisect on whichever axis has the cleanest boundary: time (git bisect), layers (check the midpoint value), input (shrink the repro), or code (disable half).
Optimize each step for information, not for a fix. "Might this fix it?" is the wrong question. "Which half does this rule out?" is the right one.
The symptom is where the bug surfaced, not where it lives. Stop starting there.

Your Struct Layout Is Part of Your Wire Protocol

speed engineer — Sat, 04 Jul 2026 04:42:36 +0000

The problem

A message-processing system that comfortably handled 50,000 messages/second fell off a cliff one night: 8,000 msg/sec and collapsing. The confusing part was the dashboards. Network: green. Database: barely working. CPU utilization: 23%.

Nothing looked busy, and yet nothing was moving. Three weeks of profiling later, the culprit turned out not to live in any code path. It lived in a struct definition:

struct PaymentMessage {
    struct MessageHeader header;   // 28 bytes of mixed-size fields
    char     sender_id[16];
    char     recipient_id[16];
    uint64_t amount;
    char     currency[4];
    char     reference[32];
    uint8_t  flags;
} __attribute__((packed));         // 101 bytes. Practically always misaligned.

101 packed bytes means nearly every message straddled two cache lines — sometimes three.

Why it happens

CPUs don't read bytes. They read cache lines: 64-byte blocks. Ask for one byte and the CPU pulls in that byte's 63 neighbors whether you need them or not.

Most of us design binary protocols as if the struct exists in isolation — fields ordered by meaning, packed to save bytes on the wire. But in memory, that struct lands wherever the allocator put it, and the fast path (validate → route → count) reads fields scattered across the whole layout: magic at the front, checksum at the back, timestamp in the middle.

At tens of millions of field accesses per second, every extra cache line touched is an extra trip to memory. An L1 hit costs about a nanosecond; a main-memory fetch costs about a hundred. Multiply "2–3 lines per message instead of 1" by 50K messages/sec and the pipeline spends its life stalled on the memory bus while the cores look eerily idle.

What to do about it

1. Split hot from cold. Everything the fast path reads goes in the first 64 bytes, aligned to a line boundary:

struct CacheOptimizedMessage {
    // Hot: validation + routing — first 32 bytes
    uint32_t magic;
    uint32_t message_type;     // widened for alignment
    uint64_t timestamp;
    uint64_t amount;
    uint64_t sequence_id;

    // Warm: next 32 bytes
    char     sender_id[12];    // compacted
    char     recipient_id[12];
    uint32_t checksum;
    uint32_t payload_offset;   // variable data lives elsewhere
} __attribute__((aligned(64)));   // exactly one cache line

The fast path now costs exactly one cache fetch: validate, route, and update without ever reaching past the first line.

2. Move variable-length data behind an offset. Flexible fields get their own 64-byte-aligned section with a small offset table. One extra indirection when you need them — zero cost when you don't (which is most of the time).

3. Batch and prefetch. Once messages are line-aligned, batches become predictable, and the prefetcher becomes your friend:

for (size_t i = 0; i < count; i++) {
    if (i + 1 < count)
        __builtin_prefetch(&messages[i + 1], 0, 3);
    process_message_fast_path(&messages[i]);
}

The rework took the system from 14,700 to 50,100 msg/sec — 3.4× — on the same hardware, same algorithms. P50 latency went from 89ms to 12ms; P99 from 456ms to 34ms. Cache misses dropped 78%.

When to bother

Above ~10K messages/sec with CPU-bound processing: this is one of the highest-leverage changes available to you. Between 1K and 10K: measure first. Below 1K, or if you're IO-bound, or your messages vary wildly in size: skip it — you're trading real development velocity for latency you won't notice.

Key takeaways

CPUs fetch 64-byte cache lines, not fields. A struct that straddles lines multiplies memory traffic on every access.
Put every field your fast path reads into the first cache line; push cold metadata to the next.
aligned(64) guarantees your hot line starts at a line boundary — packed-and-misaligned undoes everything.
Variable-length data goes behind an offset, fetched only when needed.
Verify with perf stat (cache-misses, instructions-per-cycle) before and after. The numbers don't lie; intuition does.

This is a condensed version of a longer war story I published on Medium — the full piece walks through the L1/L2 layering, platform portability, and the batch buffer design: Binary Protocols: Designing Messages For Cache Lines

Your Manager Can't Remember What You Did in March — and That's a Systems Problem

speed engineer — Fri, 03 Jul 2026 03:42:05 +0000

The problem

Every consequential decision about your career — promotion, calibration rating, who lands on a layoff list — gets made in a room you're not in, usually in a few minutes per name.

Here's the uncomfortable part: the input to that decision is not your work. Your commits, your incident responses, your design docs — none of that is in the room. What's in the room is a lossy, human-memory summary of your work, reconstructed months after the fact by people juggling ten other names.

Most engineers treat this as politics. It's not. It's a data problem, and you can engineer around it.

Why it happens

Think of it as a caching problem.

Your actual output lives on disk: the repo history, the closed incidents, the migration that shipped without drama. Nobody reads disk during calibration. There's no time. They read cache — the handful of impressions that survived in your manager's memory.

And that cache has a brutal eviction policy:

Recency wins. Q1 work is mostly gone by review season. The last six weeks are over-weighted.
Loud events survive; quiet competence doesn't. The engineer who heroically fixed a burning outage generates a story. The engineer whose systems never caught fire generates... nothing. Prevention produces no events, so it's evicted first.
Manager change = cache flush. New manager, and your last two years of context drop to whatever made it into writing. In most orgs, that's a few one-line review summaries.

This is why the most load-bearing engineers so often score as "solid, meets expectations" while the firefighter — sometimes fighting fires of their own making — scores as "high impact." Nobody is being malicious. The work was simply never logged anywhere a decision-maker could read it.

What to do about it

Maintain an evidence log. Five minutes a week, three fields per entry:

What changed — the outcome, not the activity. "Migrated checkout to the new payment service" beats "worked on payments."
The number — before/after, with a link to the dashboard or PR. "p95 checkout latency 480ms → 210ms; conversion +1.9%." A claim with a number survives a calibration room. "Worked on performance" does not.
Who can vouch — the staff engineer who reviewed it, the PM who saw the metric move. Names make claims checkable, and checkable claims get repeated.

Then do three things with it:

Log prevention explicitly. "Added connection-pool alerting after incident #2141 — zero repeat incidents in 6 months" turns invisible work into a record.
Send your manager a short monthly summary. Five bullets, no prose. This isn't bragging — you're warming their cache with accurate data before eviction does its work. Most managers are quietly grateful: you're writing their calibration notes for them.
Write one-page decision docs for anything contested. Six months later, "why is the queue architecture like this?" has an answer with your name on it, instead of an archaeology project.

Before review season, distill the log into five promo-ready claims. What took your peers a panicked weekend of git-log archaeology takes you twenty minutes.

Key takeaways

Career decisions read from memory, not from history. Memory is a cache with recency and drama bias.
Prevention work is evicted first. If you don't log it with numbers, it functionally didn't happen.
Five minutes a week of evidence logging beats five hours of reconstruction the night before your review.
Legibility isn't politics. It's writing for the room you won't be in.

Prompt Rot: Why Your Best AI Prompt Quietly Stops Working

speed engineer — Thu, 02 Jul 2026 03:45:45 +0000

You saved a prompt two months ago that wrote the perfect customer reply. Today you run it and the output is... fine. Not great. You can't point to what changed.

Welcome to prompt rot.

What prompt rot actually is

A saved prompt isn't a fixed tool. It's a set of instructions balanced on top of three things that all quietly move: the AI model, your own facts, and the people using it. When any of them drift, the prompt keeps running — but the results slowly slide.

Three reasons a good prompt goes bad

1. The model changed under you. Providers update ChatGPT, Claude, and Gemini all the time. A prompt that fit last quarter's behavior can land differently today. Same words, different result.

2. Your facts moved, the prompt didn't. It still says "our starter plan is $9," uses the old product name, or leans on the pre-rebrand tone. The AI faithfully repeats yesterday.

3. The prompt got tweaked in private. Whoever wrote it keeps refining it in their own tab and never updates the shared copy. Everyone else is running a stale version and doesn't know it.

Why it's so sneaky

Prompt rot doesn't throw an error. The output is still plausible — just worse. A slightly off tone, a fact that's six months stale, a structure that used to be tighter. So nobody flags it. People just quietly stop using the prompt and go back to writing from scratch, which is the exact waste a shared prompt was supposed to kill.

How to keep prompts fresh (no technical skills required)

Date and label. Put a "last checked" date and a one-line "works for: [task]" note on every saved prompt. A prompt with no date is a prompt nobody trusts.
Re-test your top 5 monthly. Run your five most-used prompts on a real task and read the output like a first-time reader. If it slipped, fix the prompt — don't lower your bar.
Keep the old versions. When you "improve" a prompt, keep the previous one. Half the time the improvement is worse and you'll want to roll back.
Watch what quietly drops. A prompt whose usage falls off a cliff is often rotted, not unneeded. That drop is your signal to go re-test it.

Where PromptShip fits in

This is a big part of why we built PromptShip: a shared prompt library for non-technical teams. It keeps version history so you can roll back a change that made output worse, and shows usage so you can spot the prompt that quietly stopped getting used — usually the first sign of rot. Organize prompts by team (Marketing, Sales, Support), copy any of them into ChatGPT, Claude, or Gemini in one click, and everyone runs the current version instead of a private fork.

Key takeaways

Prompts decay silently — no error, just slowly worse output.
The usual culprits: the model updated, your facts changed, or the shared copy went stale.
Date them, re-test the top few monthly, keep version history, and watch for usage drop-offs.
A prompt library only pays off if the prompts inside it are still true.

When did you last re-check the prompts your team runs every day?

Why Your Project Quotes Are Always Wrong (And the Log That Fixes It)

speed engineer — Wed, 01 Jul 2026 04:47:57 +0000

Every freelancer I know has a number in their head for how long things take. "A landing page? Two days." "A logo revision round? An afternoon." We quote from that number. And that number is almost always wrong.

The problem: you quote from memory, not measurement

Here's the trap. You finish a project, feel good, and move on. Six weeks later a similar project comes in and you pull a quote out of the same fuzzy memory that has quietly forgotten the two evenings you spent untangling the client's brand assets, the three rounds of "small" revisions, and the hour you lost every morning just reloading context.

So you quote for the happy path — the version of the project where nothing goes sideways. Then reality shows up, the hours balloon, and you eat the difference. Not because you're slow. Because you estimated against a memory that was edited to make you feel competent.

Underquoting doesn't feel like a crisis. It feels like a slightly disappointing month, over and over.

The fix: keep an actuals log

You don't need a methodology. You need one habit: after every project, write down what you estimated and what it actually took. Two numbers and a one-line note.

A minimal version looks like this:

| Project type      | Quoted (hrs) | Actual (hrs) | Note                          |
|-------------------|--------------|--------------|-------------------------------|
| Landing page      | 16           | 27           | Client assets were a mess     |
| Logo, 3 concepts  | 8            | 14           | Revision rounds ran long      |
| Blog, 1500 words  | 4            | 5.5          | Research heavier than assumed |

Do this five times and something uncomfortable but useful happens: you get a personal multiplier. Mine hovered around 1.6x for design work — meaning my gut estimate needed to be multiplied by 1.6 to land near reality. That single number changed my quoting more than any pricing course.

Why the actuals are the hard part

The estimate is easy — you write it when you send the quote. The actual is where everyone falls off, because reconstructing real hours at the end of a project means fighting the same faulty memory that caused the problem in the first place.

The only reliable way to know your actual hours is to have captured them as they happened, in small pieces, without ceremony. That's the whole reason time tracking exists — not for surveillance, but so future-you can quote honestly.

This is where a low-friction tracker earns its place. I use FillTheTimesheet so the "actual hours" column fills itself from what I logged during the work, instead of me guessing on a Friday. The point isn't the tool — it's that your actuals need to be a byproduct of working, not a separate chore you'll skip.

Key takeaways

Your mental estimate is edited to make you look good; it's not data.
Underquoting rarely feels dramatic — it just quietly caps your income.
Log estimated vs. actual hours after every project. Two numbers, one note.
After ~5 projects you'll have a personal multiplier worth more than any template.
The actuals only exist if you captured time as you worked — that's the part to automate.

Quote from a spreadsheet of what actually happened, not from the story you tell yourself about how it went.

How to Build a Shared Prompt Library Your Team Will Actually Use

speed engineer — Tue, 30 Jun 2026 03:38:29 +0000

If your team uses ChatGPT, Claude, or Gemini every day, you already have a prompt library. It's just scattered across Slack DMs, sticky notes, a forgotten Notion page, and three people's heads.

The result is predictable: your best-performing prompt — the one that writes the perfect support reply or the cleanest sales follow-up — gets rewritten from scratch every week by someone who didn't know it already existed.

Here's a simple, repeatable system to fix that. No prompt-engineering background required.

Step 1: Capture before you organize

Don't start by designing the perfect taxonomy. Start by collecting. For one week, ask everyone to drop any prompt they reuse into a single place. You'll be surprised how many duplicates surface — and the duplicates are gold, because a prompt three people independently saved is a prompt that works.

Step 2: Name prompts by the job, not the tool

A prompt called "GPT thing" helps no one. Name it after the outcome: "Draft a refund-approval email," "Summarize a discovery call into 5 bullets," "Rewrite copy for a skeptical CFO." When the name describes the job to be done, the right person finds it in seconds.

Step 3: Group by team function, not by model

Marketing, Sales, HR, Support, Writing. People think in terms of their work, not in terms of which AI model they're pasting into. Organize around how your team is actually structured and adoption takes care of itself.

Step 4: Make reuse a one-click action

This is where most libraries die. If using a saved prompt means scrolling a doc, copying carefully around the formatting, and pasting into another tab, people stop bothering and go back to winging it. The library only works if grabbing a prompt is faster than rewriting one.

Step 5: Keep the best, retire the rest

Once a month, look at what's actually being used. Promote the winners, archive the dead weight. A library of 30 great prompts beats a graveyard of 300.

Where PromptShip fits in

We built PromptShip because we kept watching non-technical teams lose their best prompts. It's a shared prompt library — organize prompts by team function, copy any of them straight into ChatGPT, Claude, or Gemini with one click, and see usage analytics so you know what's actually working. There's a library of 50k+ community prompts to start from, and the free tier covers 200 prompts for a single user.

Think of it as Notion-for-prompts, built for Marketing, Sales, HR, and Support rather than engineers.

Key takeaways

Your team already has a prompt library — it's just disorganized
Capture first, organize second; duplicates reveal your best prompts
Name by outcome, group by team function
Reuse has to be effortless or the library dies
Prune monthly so quality stays high

What does your team's prompt workflow look like today — shared system, or scattered chaos?

How Japan Just Beat Claude Mythos

speed engineer — Fri, 26 Jun 2026 04:56:39 +0000

A founder I know forwarded me the Sakana launch tweet with one line on top: "should we switch?"
Look, the tweet worked on him. Fugu Ultra stands shoulder-to-shoulder with Fable 5 and Mythos preview, no export controls attached, and that was all it took. My honest first reaction was a small stomach drop, because if it were true it'd reset half my stack.
Then I read the actual post. Fugu isn't a model. It's an orchestration layer wearing a model's coat.
And the part that should bother you isn't the marketing. It's that a lot of senior people saw one benchmark beat Fable and stopped reading right there.
I've had three of these models in production this year, so I read launch posts like this one with a particular kind of suspicion. This one earned it.

Why this matters before you forward it to your CTO

The export-controls line isn't a benchmark claim. It's a procurement claim. Fable and Mythos access got suspended under a directive, so a vendor shows up saying you can have that frontier output anyway, through them, no paperwork.
And honestly, if legal just killed your Mythos access mid-project, that hurts. A wrapper that routes around it sounds like oxygen.
But you're not buying a model. You're buying a middleman who rents the same models you could rent yourself, then charges you to pick between them. Somebody should say that out loud before the contract gets signed.

The number everyone misread

The standard instinct is that a higher LiveCodeBench score means a smarter system, and usually that instinct is fair. Most of us live and die by SWE-bench and terminal-bench numbers when we pick a coding model.
That breaks the moment the thing on the leaderboard isn't a model.
Fugu Ultra scored 93.2 on LiveCodeBench. Fable scored 89.3. But Fable is inside Fugu Ultra. So is GPT. We don't actually know the full roster underneath, and that right there is part of the problem.
You didn't watch Japan beat Anthropic. You watched Anthropic's model, plus a few other models, plus a router, beat Anthropic's model running alone.
A collection of models beating one of its own members isn't a discovery. We've known that since boosting. Four of you post a faster total than Usain Bolt, and somehow that's "we beat Bolt."

What they actually built is a router, not a model

The framing is junk. The engineering underneath is actually interesting.
OpenRouter already shipped this shape with their Fusion idea, where one prompt fans out to several models at once and a judge model reads all the answers and stitches them into a single reply. Same path on every request, nothing learned about when to do what.
Sakana's twist is that the orchestrator is itself a small trained LLM. It doesn't fire every model on every request. It's trained to decide which models to call, when to delegate, and how to stitch the result. They ship two tiers. Fugu handles low-latency work; Fugu Ultra pulls in a deeper pool of agents for the hard multi-step stuff.
So the thing you call is a learned dispatcher, right? The raw intelligence still comes from the frontier models underneath: Opus, GPT, Fable, take your pick. Sakana owns the routing and the synthesis, and that's the whole product.
There's a name for this. It's the mixture-of-agents pattern, productized behind one endpoint. Useful, but nowhere near frontier.
Sakana FuguThat's why the leaderboard comparison doesn't survive a second look. The number proves the bundle is good. It says nothing about whether Sakana built the good part, and for a buying decision that's the whole game.
It's closer to an AI harness than a model, honestly. Claude Code is a harness too, and nobody puts Claude Code on a model leaderboard.

Same call, four invoices

The API is the seductive part. You hit one endpoint, you get one answer.

Looks like any other model call. That's the whole pitch. resp = client.messages.create( model="fugu-ultra", # not a model, a dispatcher messages=[{"role": "user", "content": prompt}], ) Under the hood this may have called Opus + GPT + Fable, run a judge pass, and billed you for every token of all of them. You can't see which. The selection logic is closed source.

That last comment is the real problem. You don't get to see the routing. You can't inspect which models ran, why, or how the final answer got assembled. It's a black box sitting on top of other black boxes.
Now the bill, in dollars. Somebody on Hacker News put it better than any analyst:
You already pay $200 each to Anthropic, OpenAI, Cursor, Google. It doesn't round up nicely, so you end up paying another $200 a month to Sakana just to coordinate it.
Call it another $200. The exact figure doesn't matter, the shape does. You're already paying every provider underneath, and now you're paying a margin on top to have something choose between them.
Do the math on a single hard request and it's worse than one tax.

`One hard request, Fugu Ultra fans out to ~3 models + a synthesis pass.

Solo call (what you do today):
~20K input + ~4K output, billed once, to one provider.

Fugu Ultra, same request:
~20K input x 3 models -> ~60K input across 3 providers
~4K output x 3 models -> ~12K output
synthesis pass reads it all -> ~32K input + ~4K output

~3x to 4x the tokens, in and out, for ONE answer,
then Sakana's margin on top.

At frontier prices, that's not a rounding error.
`

For a one-off hard problem where being right beats the bill, sure, maybe it's worth it. For your day-to-day coding loop, you're lighting money on fire to get an answer your existing frontier model would've handed you anyway.

The aftermath

To be fair, the team isn't a bunch of clowns. David Ha, co-founder and CEO, made managing director at Goldman running rates trading in Japan before he left for Google Brain, where he co-authored the World Models paper that a lot of us actually read. The beta ran with close to 500 early users building real things, and the demos aren't faked: small UIs, chess, 3D-cube solving, some ML work.
But the reaction split for a reason. The same crowd that respects the founder is also side-eyeing a lab that calls itself frontier while mostly selling B2B AI apps to Japanese businesses, with recruiting people describe as abrasive. He's clearly driven. But this thing just doesn't feel thought through.

There's no moat here

Plenty could still go wrong here, and I don't do happy endings.
First, defensibility. If the routing is a genuinely novel trick that reliably squeezes more out of the same models, then every frontier lab ships their own version inside a week. Anthropic and OpenAI already hold all the pieces. Why would they hand the coordination margin to a third party sitting on top of their own models? They wouldn't. They'd absorb it.
Then there's variance. A chunk of that "feels smarter" could be retry behavior and lucky sampling, not real lift. Run it across many benchmarks instead of one cherry-picked bench and the gap might shrink to noise. They showed one bench, which is the tell.
And the black box. When Fugu hands you a wrong answer, you can't tell which underlying model failed or why the router picked it. Good luck debugging that during an incident. You've outsourced the exact layer you need to see into, and that opacity bites you in production every single time.
So no, Japan didn't beat Mythos. A clever dispatcher rented Mythos by the token and stacked a few friends on top. If you're already on a frontier model, don't jump ship. If you're coming from something three or four months old, you'll feel a lift, but that lift is the frontier models underneath, and you can rent those directly without the extra invoice.
They're selling it as the last API key you'll ever need. What it actually is: four bills to coordinate them all.

Enjoyed the read? Let's stay connected!
🚀 Follow The Speed Engineer for more Rust, Go and high-performance engineering stories.
💡 Like this article? Follow for daily speed-engineering benchmarks and tactics.
⚡ Stay ahead in Rust and Go - follow for a fresh article every morning & night.

Your support means the world and helps me create more content you'll love. ❤️

Your users already built your product — look at their ugly workarounds

speed engineer — Fri, 26 Jun 2026 03:48:15 +0000

When you build software, users will happily tell you what they want. "Make it faster." "Add a dashboard." "Make it more like [competitor]." Most of it is noise. People are great at describing pain and terrible at prescribing solutions — and the solution they ask for is rarely the one that fixes the pain.

The real signal isn't in what users request. It's in what they've already cobbled together to survive without you.

Two products, one habit

I've built two SaaS products, and both came from staring at a workaround instead of a feature request.

FillTheTimesheet. Freelancers and agencies don't say "I need time-tracking software." They say tracking is annoying and they'll deal with it later. So I watched what "later" actually looked like: the last day of the month, a blank spreadsheet, and someone reconstructing three weeks of work from calendar invites, Slack scrollback, and memory. That monthly guessing ritual was the spec. It said: capture has to happen in the moment, with near-zero friction, or it won't happen at all. Nobody requested that. The workaround screamed it.

PromptShip. Teams using ChatGPT, Claude, or Gemini don't ask for a "prompt library." But watch them work and the same artifact shows up everywhere: a chaotic Google Doc, a pinned chat message, a personal notes file full of prompts that worked once. Someone writes a genuinely good prompt, shares it, and a month later everyone has lost it and rebuilt it from scratch. That messy doc is a feature request written in frustration. It said: these prompts are reusable assets the team keeps re-earning, so they need one shared home and one-click reuse — not better documentation discipline.

Why workarounds beat feedback

A workaround is a revealed preference. It already passed the only test that matters: someone was annoyed enough to do extra work to get around the gap. That's a person voting with effort, not opinion.

Feature requests are the opposite. They're cheap to say and easy to abandon. "I'd use that" predicts nothing. "I built a fragile spreadsheet to do that every Friday" predicts a real need.

So when I sit with a user now, I've stopped asking "what do you want?" I ask:

What do you do right before and after using the tool? (the manual glue)
Where's the spreadsheet, doc, or notes file you've built on the side?
What do you redo every week that you wish you didn't?

The answers map almost directly onto a roadmap.

The trap

The catch is that workarounds are a little embarrassing, so people hide them. Nobody volunteers "I reconstruct my hours from memory" or "our prompts live in a doc called final_FINAL_v3." You have to make it safe to admit the hacky thing. Once you do, they'll hand you the spec for free.

Takeaways

Treat feature requests as symptoms, not specs.
Hunt for the workaround: the spreadsheet, the doc, the pinned message, the manual ritual.
A workaround is effort someone already spent — the strongest signal you'll get.
Build the thing that makes the workaround unnecessary, then get out of the way.

The two products I'm proudest of weren't ideas. They were someone's ugly workaround, made a little less ugly.

FillTheTimesheet and PromptShip both started as someone's workaround. If you're running an ugly one right now, I'd genuinely love to hear it.

Your team's best AI prompts are dying in chat threads

speed engineer — Thu, 25 Jun 2026 03:45:37 +0000

The $0 asset your team keeps throwing away

Someone on your team just wrote a ChatGPT prompt that turns a messy bug report into a clean changelog entry. It works perfectly. They paste it in Slack. Three people react with a fire emoji.

Two weeks later, nobody can find it.

This happens on every team using AI right now. The prompts that actually work — the ones refined through ten frustrating iterations — live in DMs, sticky notes, and someone's chat history. They're treated as throwaway text instead of what they really are: reusable operational knowledge.

Why prompts are worth saving

A good prompt is basically a small program written in plain English. It encodes:

The exact context the model needs
The output format you actually want
The edge cases someone discovered the hard way

When you lose it, you don't just lose text. You lose the twenty minutes of iteration that produced it — multiplied by every teammate who later reinvents the same thing.

A lightweight system that works

You don't need anything fancy to start. Here's the minimum viable prompt library:

One shared location. Not five. One.
A title that describes the job, not the tool. "Turn meeting notes into action items" beats "ChatGPT prompt 3".
A category. Marketing, support, code, hiring — whatever maps to your team.
The prompt itself, copy-paste ready. No screenshots.
One line on when to use it.

That's it. The discipline is harder than the structure: every time a prompt earns a reaction in chat, it goes in the library before the thread scrolls away.

Where this breaks down

A shared doc or sheet gets you surprisingly far. It breaks when:

People stop copying prompts out because it's two clicks too many
There's no sense of which prompts are actually being used
Prompts drift, and nobody knows which version is current

At that point the bottleneck isn't structure — it's friction and visibility.

How PromptShip fits in

This is the itch we built PromptShip to scratch. It's a shared prompt library for teams: one-click copy straight into ChatGPT, Claude, or Gemini, categories, version history, and usage analytics so you can see which prompts your team actually relies on. Think Notion-for-prompts rather than a developer toolkit — it's aimed at marketing, sales, support, and HR folks as much as engineers. There's a free tier if you want to try the idea without committing.

But the tool is secondary. The mindset is the point: treat your best prompts as shared infrastructure, not disposable chat messages.

Key takeaways

The prompts that work are operational knowledge — capture them deliberately
Start with one shared location, job-based titles, and copy-paste-ready text
Watch for friction and version drift as your library grows
Whatever tool you use, the win is making good prompts findable and reusable

What's your team's system for this right now? Curious whether anyone's actually cracked it with plain docs.

Sunday Notes: Find Where Before You Theorize Why

speed engineer — Sun, 14 Jun 2026 03:47:56 +0000

This week I wrote about two things that have nothing to do with each other.

On Saturday it was a debugging story: a service was losing 30% of its UDP packets, I spent the better part of a day convinced a switch was dying, and the network turned out to be completely innocent — my own host was accepting the datagrams and then dropping them because a socket buffer kept filling during bursts. One kernel counter (netstat -su, "receive buffer errors") would have told me that in about ten seconds.

Earlier in the week it was product stuff: why teams keep losing their best work, why a freelancer's Friday disappears into "what was I even doing on Tuesday," why a marketing team rewrites the same prompt every quarter.

It wasn't until I sat down to write this recap that I noticed they're the same lesson.

The shared mistake

In the packet story, the expensive move was theorizing about why before establishing where. "The network is dropping packets" is a theory about cause. It sent me to the wrong team, the wrong dashboards, the wrong week. The cheap move — the one I skipped — was localizing first: are the packets dying on the wire, or after they reach my box? Two completely different buildings, identical symptom.

Product decisions have the exact same failure mode. "Users churn because we're missing feature X" is a theory about why. It's also, usually, the most expensive possible thing to act on first, because building X takes a quarter and the theory might be wrong. The cheap move is to localize: where in the week, the workflow, or the funnel does the value actually leak out?

When you force yourself to find where first, the answer is often boring and small — and boring and small is good news, because boring and small is cheap to fix.

Why I keep coming back to this

Both of the things I work on are, underneath the marketing, just instruments for making "where" visible.

FillTheTimesheet exists because "I undercharged this month" is a why-theory; the useful version is where the billable hours actually went, captured while they happened instead of reconstructed on Friday.

PromptShip is the same shape pointed at a different problem: "our team is bad at AI" is a why-theory; "the prompt that worked is sitting in one person's chat history and nobody else can find it" is a where. One is an identity crisis, the other is a Tuesday-afternoon fix.

Neither is glamorous. Both are counters you check before you theorize.

The Sunday takeaway

Find where before you theorize why. It works on packets, it works on churn, and it works on most arguments that have gone in circles for more than ten minutes. Locate the problem in space before you start explaining it — the explanation is usually cheaper, and more often correct, once you know where you're standing.

The full UDP debugging story, counters and buffer math included, is on Medium if that's your kind of weekend reading: I Lost 30% of My UDP Packets — The Debugging Story.

What's the last thing you assumed the cause of — before you actually measured where it was happening?

I Lost 30% of My UDP Packets — and the Network Was Innocent

speed engineer — Sat, 13 Jun 2026 03:54:53 +0000

A receiver pulling a UDP feed was missing roughly 30% of its messages. No errors, no exceptions, no stack traces — just gaps in the sequence numbers. The first suspect is always the network: a flaky switch, a saturated link, a tired NIC.

The network was innocent. The packets were being dropped on the receiving host, after they'd already arrived. Here's how to tell the difference, and why it matters.

Why UDP makes this sneaky

UDP has no retransmission and no backpressure. When a datagram is lost, nobody is notified — not the sender, not the receiver. The packet simply isn't there.

That means two completely different failures look identical from the application's point of view:

The network dropped the packet before it reached your machine.
Your own host accepted the packet and then threw it away after it arrived.

The application sees the same thing in both cases: a missing sequence number. But the fix is in a different building depending on which one it is.

Where the packets actually go

The receive path is: NIC → kernel socket receive buffer → your recv() call. The kernel parks incoming datagrams in a per-socket buffer until your code reads them. If your code doesn't drain that buffer fast enough, it fills, and the kernel drops the overflow. Crucially, the kernel counts those drops.

On Linux:

# Per-protocol summary — look for "receive buffer errors"
netstat -su

# Or straight from the kernel counters
cat /proc/net/snmp | grep -A1 Udp
#   InDatagrams  ... InErrors  RcvbufErrors ...

If RcvbufErrors is climbing, the network did its job and your host discarded the datagrams. That single counter collapses a week of "is it the switch?" into about ten seconds of certainty.

The actual cause

In this case the socket receive buffer was sitting at the default (~208 KB). The sender burst faster than a single receive thread could call recv(). Average throughput looked fine on every dashboard — but the bursts filled the buffer in milliseconds, and everything past the brim was dropped. The metric that mattered wasn't mean throughput; it was peak burst versus drain rate.

The fix, in order of leverage

Drain faster. The receive loop was parsing and doing a database write inline. Anything that isn't "copy bytes out of the socket" belongs off the hot path: recv() → hand the buffer to a queue → immediately loop back to recv().
Raise the buffer. Bump SO_RCVBUF, and raise net.core.rmem_max so the kernel actually honors the request. A bigger buffer doesn't fix a slow consumer — it absorbs bursts so a fast-enough consumer never falls behind. You usually need both this and #1.
Batch your syscalls. recvmmsg() pulls many datagrams per system call, which cuts per-packet overhead when volume is high.
Spread the load. If one core genuinely can't keep up, SO_REUSEPORT lets multiple threads share the same port with separate buffers.

Key takeaways

"Packet loss" is a location, not a cause. Find out where before you theorize about why.
With UDP, silent drops are the default — the protocol won't tell you, so the kernel counters have to.
RcvbufErrors is the first thing to check. It almost always points at a receive buffer that's too small or a consumer that's too slow.
A bigger buffer absorbs bursts; a faster drain prevents them. You usually want both.

The full debugging story — the live-feed before/after, the buffer math, and the exact counters I watched while tuning it — is on Medium:

Networking for Developers: I Lost 30% of UDP Packets — The Debugging Story

I write more like this on Medium as **The Speed Engineer* — performance engineering, debugging stories, and the lower-level systems work that doesn't fit in a tweet.*