I/O 2026 p99 Test: 1,847 Invoices, One Feature Timed Out

I/O 2026 p99 Test: 1,847 Invoices, One Feature Timed Out 4.2%

Google's I/O 2026 keynote demoed three AI platform updates and every single one ran under 300ms on stage. I wired all three into a live invoice pipeline, logged p50/p95/p99 for seven straight days, and 77 invoices never returned. Here's the tail latency the keynote didn't show you.

The setup: three features, one production pipeline, real customers

I run an invoice generation tool used by small business owners — the workflow is "click button, wait, get PDF, or get angry." That's the perfect test bed for I/O 2026's three headline updates because it's latency-sensitive, transactional, and the users are non-technical. If it breaks, they don't file a bug report. They generate a duplicate and email support.

The three features under test:

• WebMCP — browser-side agent calls, the Model Context Protocol wired into Chrome so a page can invoke agent tools without a server round-trip. Got the most stage time.
• Built-in AI (Gemini Nano on-device) — inference running locally in the browser. Boring segment. Barely any applause.
• Skills in Chrome — pages ship reusable agent capabilities that other pages can load and invoke.

I routed every incoming invoice request through a round-robin dispatcher so each feature got roughly a third of live traffic. Same input schema, same output requirement (a rendered PDF), same customers hammering it during business hours. Seven days. 1,847 requests. Paying users, not synthetic load.

The dispatcher looked like this — small, deliberately dumb, no retries so I could see raw failure:

FEATURES = ["webmcp", "builtin_ai", "skills"]

async def dispatch(request):
    feature = FEATURES[request.id % 3]
    t0 = time.perf_counter()
    try:
        result = await asyncio.wait_for(
            run_pipeline(feature, request), timeout=8.0
        )
        latency_ms = (time.perf_counter() - t0) * 1000
        log(feature, latency_ms, status="ok")
        return result
    except asyncio.TimeoutError:
        log(feature, 8000, status="timeout")
        raise

8-second hard timeout. That's already generous for a click-to-PDF flow. Anything past 3 seconds and users start clicking again.

WebMCP: beautiful p50, catastrophic tail

WebMCP was the star of the keynote. It's also the feature I would not ship into a customer-facing path today.

The p50 is genuinely good — 280ms, right in line with the stage demo. If I'd stopped measuring at the median I'd have written a glowing recap. But the distribution has a tail like a comet:

Metric	WebMCP
p50	280ms
p95	2,100ms
p99	6,400ms
Timeouts	77 / 1,847 (4.2%)

Under concurrent load — meaning three or four SMB users generating invoices in the same minute — p99 collapsed to 6.4 seconds and 4.2% of requests never returned at all. In a real SMB workflow that means: user clicks generate, sees a spinner, waits, clicks refresh, generates a duplicate invoice with a new number, then calls support to figure out which one to void.

The failure signature was consistent. Concurrent calls into the browser-side MCP transport queued behind each other and eventually the transport dropped the connection. This isn't a Chrome bug — it's the current shape of the WebMCP transport under contention. It'll improve. It's not there yet.

When WebMCP is safe to use

• Prototypes and internal tools where one user hits it at a time
• Async workflows where a 6-second occasional delay doesn't cascade
• Anything where you can queue and retry server-side without user visibility

Do not put it on the click path.

Built-in AI (Gemini Nano): the update nobody clapped for

Built-in AI got the shrug segment of the keynote. On stage it clocked 220ms and everyone moved on to the next slide. In production over seven days, it was the only feature that held up under real traffic.

Metric	Built-in AI (Gemini Nano)
p50	340ms
p95	780ms
p99	1,200ms
Timeouts	0 / 1,847

Zero timeouts. p99 sits at 3.5x p50, which is a normal, healthy distribution. The reason is architectural: on-device inference doesn't care about network jitter, edge congestion, regional outages, or Google's cloud load. Once the model is warm in memory, it's a local function call.

The tradeoff is model size — Nano is not GPT-4. For invoice generation the workload is narrow: parse a natural-language line item, normalize a currency string, classify an expense category. Nano handles all of that. If your workload is "write me a 2,000-word landing page," this is not your feature.

For SMB software specifically — where the AI touch is usually classification, extraction, or short generation — Built-in AI is the update you can ship this quarter without an asterisk.

Skills in Chrome: the cold-start trap

Skills in Chrome looked fine at first glance. p50 of 410ms, p95 of 1.4 seconds. Both acceptable. But the aggregate hid the real problem, which only showed up when I broke the data down by session position:

Session request #1: p50 = 2,510ms  (cold start)
Session request #2: p50 = 390ms
Session request #3+: p50 = 340ms

The first request in any new session paid ~2.1 seconds of cold-start overhead while the skill initialized in the page. For a power user hitting the tool fifty times a day, that cost amortizes to essentially zero. For a small business owner who opens the app twice a week to run payroll invoices, every single session feels broken.

This is the classic mismatch between how developers test features and how SMB users actually behave. Devs open the app, hit it a hundred times in a dev loop, and the cold start disappears into the noise. Real SMB traffic is bursty and infrequent — the majority of sessions are the first request. Cold start dominates.

Rules of thumb for cold-start sensitive features

• If p50 request-#1 is more than 4x p50 request-#5+, you have a cold-start problem
• Infrequent-user products (payroll, taxes, weekly reports) feel cold start harder than daily-use products
• A background pre-warm on page load can hide it, but costs battery/CPU on mobile

The two-rule filter I now run before shipping any new platform feature

Two heuristics from this test that would have kept me from shipping WebMCP into a client invoice flow. Both are cheap to check.

Rule 1: p99 must be less than 5x p50 under real concurrent load. If it's not, the feature has a tail-latency problem that will hit your users during exactly the moments they care most (peak business hours, month-end). WebMCP's ratio was 23x. Built-in AI's was 3.5x.

Rule 2: Cold-start on session request #1 must be under 1.5 seconds. If it isn't, your low-frequency users will experience the app as broken even though your dashboards look fine, because your dashboards are averaging warm requests.

Here's the shadow-test pattern I now use for any new platform feature — route 5-10% of live traffic through it in parallel with your existing path, log percentiles, don't user-facing the results for at least a week:

async def shadow_route(request):
    # Always serve from the known-good path
    primary = asyncio.sessions_spawn(current_pipeline(request))

    # Shadow 10% through the new feature, discard result
    if random.random() < 0.10:
        asyncio.sessions_spawn(measure_only(new_feature_pipeline, request))

    return await primary

The shadow task's result is thrown away. You're only recording latency and timeout rate. No user is exposed to the new feature's failures. After seven days you have a real distribution, not a keynote number.

Why the keynote ranking is inverted from production value

Google spent the most demo minutes on WebMCP, moderate time on Skills, and treated Built-in AI as a footnote. On real SMB workloads over a real week, that order is exactly backwards:

Feature	Keynote time	Production readiness (SMB)
WebMCP	High	Not yet — 4.2% timeout rate
Skills in Chrome	Medium	Conditional — fine for power users, breaks infrequent-use apps
Built-in AI	Low	Ship it

This isn't a knock on Google. Stage demos optimize for wow — a new agent capability with a compelling narrative will always beat "a smaller model runs faster locally." But your customers optimize for the request that doesn't time out at 4pm on a Thursday. The tail is where they live.

Every time a major platform ships a batch of features, the loudest one gets the recaps and the quiet one gets ignored. Nine times out of ten the quiet one is the durable pick, because it's usually the one that removes a network hop or a dependency, not the one that adds a new abstraction.

Why bizflowai.io helps with this

Most of the invoice, lead-follow-up, and CRM-sync automation I ship for SMBs at bizflowai.io runs on the boring pick — a small model, close to the data, with a hard timeout and a fallback path. Every new platform feature gets a seven-day shadow test against real client traffic before it touches a customer's click path, and the results get logged with p99 alongside p50 so we know what the tail actually looks like, not what the demo promised.

---

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.