DEV Community: Dominique Siacci

Our customers had already told us what Siri should say

Dominique Siacci — Tue, 10 Nov 2026 11:07:45 +0000

Thousands of apps run on our engine: a radio station, a sports club, a local paper, a pastry shop, each one different. A feature is written once, in the engine, and an app gets it at its next submission to the store, if its owner wants it. That's the point of a platform, and it's the design constraint behind everything that follows: whatever we build has to make sense in apps we will never look at one by one.

What an assistant can do in an app, we define the way we define a feature: once, in the engine, for every app that has the feature. Apple calls these things intents: an intent is what someone means to do in an app, put in a form an assistant can act on. Exposing an app to an assistant means going through what the app can do and deciding, for each thing, whether an assistant can reach it, under what name, and with what answer. A developer who does this for their own app knows its content, its sections and its users, and decides for that one app. Writing an intent is quick work now. The time goes into knowing what it should cover, for all of our apps at once, including apps that don't exist yet, without opening any of them.

Work we can't ask our customers to do

The obvious way out is to ask the owner, the customer who built the app. A form with a few questions: which sections may Siri answer from, what should it call them, what counts as news in your app. We decided against it, for reasons that hold for every intent we will ever add.

An owner builds an app to publish their content or run their organisation, not to set up an assistant. The assistant is our idea; the work it creates is ours.

The apps are already in the stores. An intent that depends on a form is empty in every app until its owner fills it in. Whatever we ship has to work in each app as it is, the day it gets the update.

And we would have to write the questions in advance, the same ones for the radio and the pastry shop, without knowing which matter where. We would be asking each owner to describe their app to us. They already have, once, by building it.

That's the premise we work from: deciding what an assistant can do in an app is our work, and the owner should never have to do it a second time. A form could have been optional, defaults read from the structure and a setting on top; we didn't build that either. A second place where the app is described drifts from the first, and someone has to keep them in step. Whether their app gets the update is their call, as with any store submission; what it answers once it has it is what this piece is about.

The structure is already a description

An app takes shape in our builder one choice at a time: dozens of choices, and its owner makes none of them at random. Which sections they add, and of which kind: a radio has a live stream and podcasts, a club has a calendar and a members' area, a shop has a catalogue and a loyalty card. Which sections go on the home screen, which stay in the menu, and in what order. How many items each widget on the home screen shows. The titles they give their sections and widgets. Which sections only members can open. Across our paying customers' live iOS apps, the median app has 16 sections and the median home screen 6 widgets. None of it was meant for an assistant, because it was made to give visual cues, not semantic ones. However, each choice is a weak signal about what counts in that app and what people come to it for, and together they are a description of the app that nobody wrote as one. A radio that puts its live stream at the top of its home screen has said what people come for. A club whose only section open to everyone is its calendar has said who the rest of the app is for. The intents are in there. Our work is to read them out.

What we read today, and what each reading gives the assistant:

What the owner chose	What it tells us	What the assistant does with it
Which sections exist, and of which kind	What the app does: publish, stream, sell, gather members	Only what exists is reachable; nothing is invented
Which sections sit on the home screen	What people come to the app for	Where "what's new" and "what's on" read from
How many items each widget shows	How much room the owner gives each one	How many items each one puts forward; the answer keeps the newest five
The order of the menu	The owner's hierarchy	The order of the sections Siri offers when it has to ask
The titles of sections and widgets	The owner's vocabulary	The words Siri uses
Which sections only members can open, and what subscribers pay for	Who the app is for	What is listed as in the app, and what carries a lock

We don't have a method that covers everything an app does, not yet. The questions below are what we have, and for each of them we read the structure instead of asking.

What Siri answers, and where it reads

We started in September 2026 with Siri and with content: what's new on an app, then, in October, the latest article and the latest video in one of its sections. Three questions about events were announced in November and reach apps with their next store update: what's on today, what's coming up, what's on this weekend. They read three places in an app: the content widgets on the home screen, the menu, and the Events widgets on the home screen. The owner's side is on our blog. You can hear it yourself: install GoodBarber News, our own app, and ask Siri "What's new on GB News?"; GB News is the name under its icon, and the one Siri listens for. On a phone with Siri AI, Apple's new Siri, variations like "GoodBarber News" are understood just as well.

The question	Where it reads	The owner's choices it follows
What's new	The content widgets on the home screen	Which widgets are there, how many items each shows, what they're called
The latest article, the latest video	The menu	Which sections exist, in what order, under what names
What's on today, coming up, this weekend	The Events widgets on the home screen	Which events are shown, and how many

"What's new" could have been a list of everything the app has published, newest first. That design needs to know nothing about an app. It treats every app as a pile of dated items, but no owner builds an app that way. Instead, the action reads the home screen. It takes the content widgets the owner put there, articles, videos, photos, podcasts, as many items as each one shows, drops what appears twice, and puts the newest first. Siri gives each item's title and the name the owner gave the widget it came from, or its section's name when the widget has none. The date orders what the owner already chose and keeps the newest five: what gets in was the owner's choice.

We measured the gap between the two designs. In November we replayed both on 248 of our customers' live iOS apps, the ones whose home screen we can rebuild exactly from our own data. In 197 of them, four out of five, the five items read from the home screen are not the app's five most recent. In 72, the most recent dated item in the whole app is an event that hasn't happened yet.

Some signals are in what the content is, not where it sits. An event has a date too, but it means "when it happens", not "when it was published". Sorted with the rest, next June's festival would come out as the latest news. So events stay out of "what's new", and map points with them: a place isn't news either. That exclusion is a rule of ours, made once for every app, and I'd rather say so. The structure doesn't speak by itself; someone has to read it. The order is our rule too. On the home screen, the owner's order says what counts; the question asks what is recent, so the date orders the answer.

"The latest article" and "the latest video" read the menu, and there we keep the owner's order. If the app has several video sections, Siri asks which one and lists them in that order, under the names the owner gave them; a section split into categories is read by its first category, and a live stream doesn't count as a video. Apps gate content in two ways, and Siri mirrors both. A section only signed-in members can open is listed as the menu lists it, and opening anything from it takes the login, as in the app; a home screen behind a login gives a signed-out visitor nothing. An item subscribers pay for is named like the rest, with a lock before its title on the card Siri shows, and opening it still takes the subscription.

The event questions read the home screen too, from the Events widgets the owner put there: as many events as each one shows, and none that is already over. The scope comes from the structure again, and the date does a different job, because an event's date says when it happens. Events on right now come first, ending soonest first, then what starts later; what's coming up stops at five. Where the structure says nothing, the rule is ours: the weekend starts at six in the evening before it, Friday or Thursday depending on which days the phone's region calls the weekend, and an event with no end time is never called "on now", because nobody told us it still is.

One rule we didn't anticipate before testing on a real device: the first version was one question with a blank for the timeframe (today, this week or this weekend). In October, on an iPhone running iOS 27, Siri left "today" out of the blank and answered as if no timeframe had been named; then "this weekend" came back as "this week", and the card looked fine. A section name comes from the owner's menu, so a word that isn't on it makes Siri ask; "this week" and "this weekend" were both valid and sound alike, so Siri heard one for the other, and the answer, for the wrong week, sounded right. Each timeframe became its own question with fixed phrases, "this week" was dropped rather than shipped wrong, and fixed phrases got the right timeframe every time after that. Our iOS team's write-up (kudos to them for the implementation!) has the whole run; what this piece adds is the decision not to ask the owner about any of it.

When there is nothing to say, because nothing new is on the home screen, or no content widget at all, the answer is one sentence, and Siri stops there: "There's nothing new right now." An empty weekend names the next event there is instead.

The same action gives a different answer in every app. The content is the owners', written years ago in some cases, without knowing it; the order, the cut and the sentence around it are ours.

Where the reading falls short

The reading takes the owner's screen as it is, and that is its weak side too. A widget with no title is named after its section. A home screen whose widgets are a player, a map, a form and a calendar gives "what's new" nothing to say, because events aren't news, and Siri says so; the event questions still read that calendar. An app whose events live in a section the home screen doesn't show will tell Siri it has no events; an Events widget on the home page fixes that, and it's the owner's screen, not a setting. A widget titled "Home 1" is read as written. A home screen nobody has rearranged in years still reads fine as long as its sections are fed: the layout is old, the items are not. And the owner sees none of it in the back office. To hear what Siri says about your app, you ask it, like everyone else; to change the answer, you change the app, the same screen your users see. An owner who rearranges the home screen for a Christmas campaign has changed what Siri says without touching a setting, because there is no second place to maintain; that is the point, and the price.

A name that can't change

One thing no structure can give us is the name underneath an action: the identifier Siri and Shortcuts bind to on the phone. It has to work in every app as it ships, ready to use before any owner makes it their own, and still hold later. In Apple's framework, the name of the intent becomes the unique identifier of the action; a developer who renames the code later can keep the old identifier with a persistent identifier, so what can never change is not the code's name but what a shipped identifier means. A developer with one app changes it with one update. Our apps don't rebuild together: each one gets the engine's changes at its owner's next submission, whenever that is, so whatever an identifier means has to stay true in binaries built months apart. There's a ceiling too: an app can register at most ten App Shortcuts, the actions Siri knows with no setup, so every one we ship takes one of ten slots in every app at once. For events we wrote down 32 questions people might ask, scored them on value, on how reliably Siri hears them, on whether the data is there and on cost, and kept three. Siri AI, further down, changes that ceiling: its tools are the intents themselves, marked as such in the code, with no phrase to register, and Apple's preview of November 3 puts no number on them.

So the name underneath is a word in the platform's vocabulary, chosen once, before any app carries it, and never renamed, whatever an owner puts on top of it. We already had that rule for the words an app is described with: you can add a word, you can't change what one means. Siri added one more place where it applies, and every action we add from here will follow it.

The phrases Siri listens for are shared the same way. A language is added once, for every app, before we know which app will speak it. There are eight today: English, French, Italian, German, Spanish, Portuguese, Dutch and Arabic, the phrases we wrote for App Shortcuts, the Siri that is already on iPhones.

Three habits, and where they'll strain

An app doesn't only publish articles and videos. It plays a live stream, takes a form, holds a loyalty card, opens a chat between members, and for each of them the same question comes back: what should an assistant be able to do here, and how do we know without asking? Three habits came out of these questions, and I expect them to hold for the rest.

The scope comes from the structure. An assistant reaches what the owner put in the app, where they put it; a member section stays a member section, and a paid item comes back with a lock, never with its content.

The words come from the owner. Siri says the names the owner chose, not names we invented for every app at once. They are the owner's own: in our customers' live iOS apps, 85% of article sections carry a title no other app uses.

The platform adds as little as it can: an order when one is needed, a sentence when there's nothing, and nothing the owner would have to set up for Siri alone.

Take the live stream. A radio's home screen already says there is one, what it's called and that it comes first; what an assistant should do with it is in there. A form, or the pastry shop's loyalty card, is harder: they change something, which is a different question, and the structure says less about what a field means than about where the form sits. That's where I expect the reading to strain.

What the new Siri changes

On November 3, Apple opened Siri AI to every developer: with the iOS 27.3 betas, an app hands its intents to the new Siri, which picks actions, and skills that combine them, from what someone says, with the screen and the person's context in view. App Store submissions follow in spring 2027. For now it isn't on iPhones in the EU, where GoodBarber is based, and its six languages cover four of the eight our phrases speak.

Two days earlier, UX Collective ran Context is the new interface, on App Intents and MCP and on what happens between products; my question is what each product has to decide first.

The new Siri changes the way in. Language models have taught everyone to say what they mean instead of a fixed phrase, and nobody remembers a fixed phrase for an app they open twice a month; that is what makes asking an app credible.

It doesn't change who reads the app. A screen doesn't say what a date means, who may open a section, or how many items a widget holds. When Siri picks the action itself from what someone says, the words it reads to choose are the ones already there: our description of the action, and the owner's names when it has to pick a section. The reading stays, and the new Siri gives it more to do. Everything here is iOS and Siri, the first assistant our apps answer to.

Follow the choices the owner already made

When the owner has already made a choice, our actions follow it instead of making their own. Where they never made one, we make it once, for every app, and we say which. The list today: the newest five; the date order in "what's new"; "what's coming up" stopping at five; events and map points out of it; an event with no end time never "on now"; the weekend from six the evening before; the lock on paid items; a section split into categories read by its first category; live streams kept out of "the latest video"; and the sentence when there is nothing. We keep it short on purpose.

An app is how a radio, a club or a shop puts what it does in people's hands; that's what it exists for. An assistant doesn't replace it; it's another UI on the same app, and what it can do there should come from choices owners have already made, in the structure each of them has already built, across thousands of apps. Reading that structure is on us, not on them, and we're at the start of it.

Before you add a settings screen for your assistant: what have your customers already told you by using the product?

My agents kept dropping what resisted. 10 of my 19 vetoes were about that.

Dominique Siacci — Fri, 06 Nov 2026 09:59:35 +0000

On September 30, an agent was building a catalogue of every setting in our back-office, nearly three thousand keys. The goal I had set that day: each key either recorded with the command that writes it, or dropped with a verified proof, and none left open. Dropping a key meant declaring that no customer action could ever write it, so there was no command to find. Its score counted a dropped key the same way as a recorded one.

That afternoon, the score climbed two points in half an hour, and most of the keys it closed were drops: the keys of two retired modules, with a proof from the code and no second look. A few minutes later, the agent proposed a few dozen more drops, each with its proof. Those came to me as a question and weren't counted.

My answer the next morning was that a second agent would check both batches, the drops already counted and the ones waiting for me. Most of the new proposals held up. A handful were reachable after all, and a few stayed open as uncertain. The first batch was reachable too, through a path the first reading had missed. The score went down, then back up once the agent had recorded those keys for real, which for the retired modules meant creating their sections on a test app and deleting them afterwards.

In the previous post I wrote about the six days a fleet of coding agents then spent on the refactor, and said that ten of the 19 precedents I rejected were about one thing. It's the same thing as on September 30, with more agents.

Why it kept happening

Both times the target was completeness, and both times it accepted a drop. In the catalogue, a dropped key counted as closed. In the refactor, every unit had the same exit criterion for coverage: each of its controls linked to an option of a function, or excluded with a written reason in the unit's spec, the file that declares its functions. Excluding a control was the refactor's drop: the unit stopped answering for it. The rules also said, from before the first unit, that excluding a control or a unit was a question for me.

Only the criterion had a check. The checks a unit had to pass skipped any control its spec listed as excluded, and until day five nothing read the reasons. So when something resisted, whether a control the test bench couldn't play, a clean fix that sat outside the unit or a case that looked out of scope, the shortest way to a green unit was one more line in the spec. From inside the unit, that line looked like finishing the job rather than a decision about scope.

The ten I rejected

How a precedent worked: when a worker met a case like one I had already answered, it applied the same answer, and I validated or rejected it afterwards. The ten exclusions I rejected came from seven different workers, every day from Saturday to Wednesday:

Day	What the worker proposed
Sat	Leave some older image settings on the old code path, reusing an earlier exclusion's reason
Sun	Exclude an image crop control the test bench couldn't play
Sun	Take a whole unit out of the queue, since another spec already covered its controls
Sun	Exclude an icon control, because the clean fix touched files outside the unit
Sun	Keep a screen's design settings and leave its content steps on the old path for later
Mon	Let the old path keep doing some image crops until a spec declares them
Mon	Note a second place that draws a control, without checking it
Tue	Call adding a menu item out of scope, though the test plan expected it to work
Wed	Remove a function from six specs, because the action it styles can't exist there
Wed	Keep a unit excluded after deleting a duplicate of its spec

Three of them took an exclusion I had validated earlier as their precedent, and every one came with a reason. What I kept asking back, in one form or another, was which unit would handle the thing once it was dropped. My other rejections were mostly requests for more context, and every rejected case went back to work.

On day five, one of the workers went through every control the units had left outside their functions. Some six hundred were accounted for: another function carried them, or they wrote nothing, so they had no effect on the app. More than five hundred were still on the old code path we were retiring, and no decision of mine covered any of them. The question that reached my dashboard offered three options, each with what it would leave behind: cover everything, cover only the most used and riskiest and list the rest, or accept all of them as limits of this phase. It recommended covering everything, and that's what I chose.

What held up

What worked was handling a drop like a result: it needs a proof, and the proof can't rest on the word of the agent that made the drop.

The catalogue is where it worked first. Every key is accounted for: recorded with its command, or dropped. Each drop says why, points to the file and line in the code that proves it, and has been checked, by a second agent for most of them and live on a test app for the others. The coverage figure counts only the recorded keys, and the drops sit in their own file next to it.

In the refactor, the same idea became code on day five, right after the audit. A check reads the reason of every excluded control and accepts three answers: another function carries it, and the check finds the binding in the code; a decision of mine covers it, and the check finds that decision by its number; or it has no effect on the app, said in one of a few exact phrasings. That third answer still rests on the worker's own sentence, only in a form I fixed, and it covers about half of the excluded controls. Anything else fails. At the end of the run, no excluded control was left without a valid answer, and nearly all of them were either carried by another function or had no effect on the app.

What changed for me

A drop is accepted once it has a check that doesn't rest on the word of the agent that made it. In the catalogue that held for every key from the next morning on. In the refactor it held once the check was in code, except for "no effect", which still takes the worker's word in a fixed form.
"Impossible" is a drop too. Since then, a case is impossible only with its own execution trace in the code, file and line. An argument about a whole family of cases doesn't count.
Next time the exclusion check exists before the first unit. Here it arrived on day five, though the rule had been in writing since before day one. And "no effect" will need a proof of its own.

All of this ran on test apps, and nothing goes to production without me.

If you run agents against a coverage or completeness target, what checks the things they leave out?

My agent's dry-run wrote seven texts to production. The fix was already on the remote.

Dominique Siacci — Tue, 03 Nov 2026 13:15:59 +0000

On September 15 at 10:27 UTC, an agent working for me ran a dry-run seven times, once per language of our website, and created seven texts in the production content of our commercial site. I had said, in writing, at the start of the session: no writes, except with --dry-run. It used --dry-run. The texts are real.

Eleven seconds later the agent found out on its own. A minute after that it had stopped and written me a report. I read it at 14:27. Four hours.

What was at stake

I run engineering at GoodBarber, an app platform. This was a side task: rewriting the share previews of our website, the descriptions that show up when you paste a link into a chat. The agent has no access to the database. It goes through the site's text-editing API, the one our editors use, with the same rights they have. What it created were seven descriptions, one per language, exactly the ones it had been asked to prepare. Nothing was deleted or modified, no customer data was touched. When I read them, they were the texts I would have approved, and we kept them.

So, nothing serious. I'm writing it down anyway, because of what it shows once you put the times in a row, and because of something @pierrelaurentmedori , Marc and I do in the datacenter without thinking about it.

The times

UTC	what happened
Sep 7, 14:09	last `git pull` of the repository holding our agents' tools, on the server where the agent runs
Sep 11, 10:52	a colleague commits a fix to the text-editing script. Its comment: "`--dry-run update …` silently sent a real PATCH"
Sep 15, 09:49	session starts. Instruction: no writes except with `--dry-run`
10:24	the agent runs the script's `--help`, then reads its dry-run code
10:27:52	the loop: 7 languages, one `create` each, with `--dry-run`
10:28:03	the agent counts `DRY-RUN` lines in its output: 0. It counts `HTTP 201`: 7
10:29	it stops everything. It deletes nothing, because a delete would be another write I hadn't approved
14:27	I read the report and ask why it stopped everything; a minute later it agrees nothing required it to stop the rest

What the flag did

The script declares its shared options once, in a parent parser, and passes that parent to the top-level parser and to every subcommand. It's a common way to let an option go anywhere on the command line. With Python's argparse it has a side effect: when the option comes before the subcommand, the top-level parser sets it to True, then the subcommand's parser applies its own default and sets it back to False. No warning.

common = argparse.ArgumentParser(add_help=False)
common.add_argument('--dry-run', action='store_true')
parser = argparse.ArgumentParser(parents=[common])
sub = parser.add_subparsers(dest='cmd')
sub.add_parser('create', parents=[common])

parser.parse_args(['--dry-run', 'create'])   # dry_run=False
parser.parse_args(['create', '--dry-run'])   # dry_run=True

Checked on Python 3.13.5. The colleague's fix, four days earlier, builds the parent with argument_default=SUPPRESS, so an absent option sets nothing. It was on the remote, and our server had even fetched it, the day before and again that morning. The copy the agent ran had not been updated since September 7.

The line the agent actually wrote

This is the part I keep looking at. The agent had read the code three minutes earlier and wasn't sure where the flag had to go. So it covered both cases:

python3 site_texts.py --dry-run create --payload $l.json \
  || python3 site_texts.py create --dry-run --payload $l.json

Try the first form; if it fails, try the second. A reasonable hedge against a flag the parser rejects. Against a flag the parser ignores, it does nothing: the first form doesn't fail, it writes, and the fallback never runs. Seven times, because the loop had no reason to stop after the first language.

Then it did the right thing, just after instead of before: it counted what the script had said back. Seven HTTP 201 where it expected seven DRY-RUN.

"Unplug 21." "Unplugging 21."

When Pierre-Laurent (Pilo, when we talk), Marc and I work in the datacenter, the room is loud and the machines all look alike. Without ever deciding it, we ended up doing what every pilot knows: one says "unplug 21", the other says "unplugging 21", and only then does the hand move. In aviation it's called readback. The tower says "runway 34, cleared to land", you read back "runway 34, cleared to land" with your callsign, and the controller has to listen to what you said and correct it if it's wrong. That second half is the hearback, and a readback nobody listens to is just more noise on the frequency.

Pierre-Laurent has been writing about read-back from the side of the MCP server he runs: a 200 doesn't tell you the right thing happened, so you read the state back and only then call it done. That's checking the world, after the write. That morning, this part worked fine: the agent read the result back within eleven seconds.

What was missing is the datacenter part, before the hand moves. Three readbacks could have happened and didn't:

The tool never said back the mode. A dry-run prints DRY-RUN POST …; a real call prints HTTP 201. Both only after acting. One line before acting, mode: WRITE, and the loop reads it.
The loop never did one and listened. Seven identical calls are one instruction. Run one, read what comes back, then run the other six.
The tool's copy never said back its version. The agent was running a script from September 7. We pull a repository before changing its code. Nobody pulls the repository of a tool before using it.

And the hearback: I gave the instruction and listened four hours later. The agent stopped, and the stop cost a morning, because the only one who could listen wasn't listening.

What changed since, and what didn't

The copy of our agents' tools on that server was updated on September 21, six days after, ten days after the fix. Nothing updates it yet before a session uses it.
Our write scripts still write by default, and --dry-run is still the thing you have to remember.
The agent suggested the next step: point future dry-runs at an address that can't answer, so the network holds the dry-run instead of a flag. I haven't done it yet.
Nothing is set up to tell me when a session has stopped and is waiting for me, so the four hours would be four hours again.

The seven texts stayed. They were right.

When your agent runs a command that writes, what does it say back before, not after? And who's listening?

Was it really the wind? I raced my windless ghost on ten rides

Dominique Siacci — Thu, 29 Oct 2026 10:46:52 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

The first time my ghost raced me, it finished 34 minutes 53 seconds behind.

The ghost is me on the same ride with the wind taken out: same road, same power second by second, same stops. On 19 August I spent six hours riding one way across the north of Corsica, mostly with the wind at my back, so a ghost without that wind should lose. Not by 35 minutes, though. And it lost on my nine other rides too, the calmest mornings included. My ghost was telling me the wind had helped me on every single ride. It was lying, and finding out why became most of this project.

Last week I published z2analyzer, a tool that splits my minutes above Zone 2 between the road, the heat and me. Its last section began with three words: "Wind isn't counted." After a slow ride, the wind is the easy culprit, and this week I wanted a tool that checks the excuse.

wasitthewind is a small web app I host myself. After a ride, I drop the FIT file from my bike computer, and a few minutes later the page races me against my ghost on a dark map of the ride: the ghost pulls ahead where the wind was in my face and drops back where it pushed me. In racing games, the ghost is a past lap replayed next to you; mine is the ride I would have had without the wind. The gap at the finish is what the wind cost me, or gave me, with a range around it. The page also points to the stretch where the wind weighed most, and can set the ride against an earlier one on the same road.

For the 58 km loop I rode on 27 September:

It wasn't the wind.
Your ghost, same legs and no wind, finishes 0:35 ahead
of you, and every other way of counting within 0:35:
the wind barely touched this ride.

Where the wind weighed most: km 42.1 → 47.2,
0:29 lost over 5.1 km, headwind 4 km/h at 10 m.

It looks back at a ride already ridden. It doesn't plan routes, pick a start time or give training advice.

Demo

The video runs on my own rides, from a new analysis to the race and its verdict. It shows times, percentages and wind speeds only, and no map of the first and last 2 km of a ride.

Code

dsiacci / wasitthewind

wasitthewind

Was it really the wind?

After a slow ride, the wind is the easy culprit. wasitthewind rebuilds the wind along your track, then replays the same ride with the wind taken out, at the same power: your ghost. You race it on a map. The gap at the finish is what the wind cost you, or gave you, with a range. It also points to the stretches where the wind weighed most, and compares the ride with an earlier one on the same road.

It is a small web app you run on your own machine: open it, start a new analysis, drop the FIT file of your bike computer, and the race comes a few minutes later. Every analysis stays in the list. A command line does the same for scripts.

Demo video (88 s, on the author's own rides): https://youtu.be/Yp9h0NMVNS4

It looks back at a ride…

View on GitHub

Python, Apache 2.0, 42 tests. wasitthewind serve is the web app: it runs on your machine, keeps every analysis, and its pages load nothing from the internet, not even map tiles. ghost runs the same race from the command line. The tables below come out of the scripts in tools/, and docs/results.md has them all, with what failed.

How I Built It

The wind along the ride

No weather service ever gets my track. wasitthewind asks Open-Meteo's archive of past forecasts for a fixed lattice of points around the ride, from Météo-France's AROME (every 15 minutes, 1.3 km apart) and ECMWF's IFS (hourly), and interpolates on my machine: each second of the ride gets a headwind and a crosswind, 10 m above the ground.

A rebuilt wind is a guess, so I checked it against what the Météo-France stations within 15 km and Ajaccio airport measured at the same hours:

At the hours of my rides, 210 station-hours	Measured	AROME	IFS
Mean wind speed	2.5 m/s	1.7 m/s	1.3 m/s
Direction error, wind above 2 m/s		25°	48°

AROME follows the direction but is a third too weak; IFS is weaker still and often wrong about where the wind comes from. AROME is the headline, IFS a second opinion. And a rider isn't on a 10 m mast: a textbook wind profile gives 0.54 of the 10 m wind to someone 1.2 m above open country, 0.29 among trees and houses.

The ghost's legs

The ghost needs my legs: my speed for a given power, grade, headwind, crosswind, temperature and time already in the saddle. I gave that to TabPFN v2, a tabular foundation model with open weights: it takes my other rides as context and predicts each 30-second piece of a new ride in one forward pass, with no training run, on a CPU. My data: ten outdoor rides with power-meter pedals since mid-August, 24.7 hours. I scored it on each ride with models that never saw it:

Speed error on rides it never saw	With AROME's wind	With IFS's	Without the wind
TabPFN v2	0.69 m/s	0.71 m/s	0.72 m/s
Linear regression	1.21 m/s	1.19 m/s	1.18 m/s
Constant			2.29 m/s

TabPFN makes 40 % less error than a straight line, and it's the only model the wind helps. The ghost then rides each piece in my time × (my speed with the wind ÷ my speed without), both as TabPFN predicts them, and stops where I stopped.

Why the first ghost lied

My rides come in two kinds: windier mornings on the flat coast road west of Ajaccio, and calm days inland, with climbs. TabPFN noticed. A strong crosswind meant "coast, fast"; no wind at all meant "inland, slow". Asked for my speed with the wind set to zero, it answered for an inland day, and my ghost slowed down on every ride. The wind told it what kind of day it was, more than what the air was doing.

So I took the wind out in several ways and compared each ghost with plain physics given the same wind. A point is one percent of the ride's time:

How the wind is taken out	Distance from the physics, mean over 10 rides
All of it set to zero (the first ghost)	4.2 points
Crosswind set to zero	3.5 points
Headwind set to zero	1.0 point
A model that never saw the crosswind, headwind set to zero	0.7 point
Headwind turned around	0.3 point

What worked is to never ask for a day without wind. The ghost keeps the crosswind and asks TabPFN twice, with the headwind as it was and turned around into a tailwind of the same strength, then takes the mean. What the model learned from how strong the wind was, the kind of day, is the same in both answers and cancels out; what's left is what the wind did by blowing one way rather than the other.

A test the ghost failed

Physics gives a check that needs no TabPFN: flat roads I rode both ways.

38 flat stretches ridden both ways, 61 km	Way into the wind, slower by
At the same power and grade	3.6 %
Physics, with 0.54 of the wind taken out (fixed before looking)	0.1 %
TabPFN's ghost, before and after taking the wind out	1.7 % → 1.8 %

The physics makes the gap melt; TabPFN's ghost leaves it where it was. From ten rides in light wind, with a model wind a third too weak, it learned a headwind effect too faint to show on these roads. So the verdict never rests on TabPFN alone: the range holds the other weather model and the physics, with my drag and rolling resistance fitted on my other rides, at 0.54 and at 0.29 of the wind.

So, was it the wind?

Each of my ten rides, raced against a ghost that learned my legs from the nine others. A plus means the wind cost me time; a minus, that it gave me some.

Ride	Time, stops included	Wind at 10 m	Wind cost me (TabPFN, AROME)	Range
14 Aug, inland	98 min	0.6 m/s	+0:20	−0:17 to +0:20
16 Aug, inland	124 min	0.4 m/s	−0:01	−0:09 to −0:01
19 Aug, the north, one way	369 min	2.7 m/s	−3:34	−6:00 to −1:04
24 Aug, west coast	63 min	0.9 m/s	+0:00	−0:59 to +0:12
29 Aug, west coast	92 min	2.0 m/s	+0:02	−0:11 to +0:35
31 Aug, west coast	63 min	0.9 m/s	+0:01	−0:21 to +0:12
13 Sep, the loop	194 min	1.1 m/s	+0:10	−1:12 to +0:12
20 Sep, the loop	185 min	0.8 m/s	−0:11	−1:10 to +0:14
27 Sep, the loop	177 min	1.4 m/s	+0:35	−0:34 to +0:35
4 Oct, west coast	207 min	1.6 m/s	−0:08	−1:40 to +0:24

On my nine round trips, 20 hours on the road, the wind cost me 48 seconds in all: mostly dawn rides in light wind, and on a loop, what the wind takes on the way out it mostly hands back on the way home. Only the long one-way day in the north moved by minutes, and even there TabPFN's −3:34 sits with the physics at 0.29 of the wind (−3:20) rather than at 0.54 (−6:00), which says again that its headwind is on the faint side.

The comparison says the same from another side. Against 13 September on the same loop, 27 September comes out 16 minutes 34 seconds faster, and the wind cost me 25 seconds more on the faster day. Whatever changed between those two Sundays, the wind wasn't it.

Why Does Open Innovation Matter?

A FIT file says where I ride, second by second, and when I leave home. wasitthewind looks like a web service, but it's one I run myself: the files I drop stay on my machines, the pages load no map tiles, and the only things about a ride that go out are the 0.1° blocks around it, about 11 km, and its date. TabPFN's weights are downloaded once and run on a small CPU, with no account and no cost per analysis, so the ghost keeps racing even if a service closes. The wind comes from open data: Open-Meteo's archive of the Météo-France and ECMWF models, Météo-France's station records, and the airport reports kept by the Iowa Environmental Mesonet.

The open weights also made the lie findable. To catch the first ghost, I ran TabPFN on the same ten rides in six different ways, then on the flat roads, then for every line of the range. Behind an API, every one of those runs would have sent my rides out, and I would have stopped asking sooner.

And nothing in it is mine except the numbers: your FIT files, your bike and your mass replace mine.

Prize Categories

Best Use of TabPFN. TabPFN v2 is the ghost's legs: on rides it never saw, it predicts my speed with 40 % less error than a linear regression, and it's the only model the wind made better. The post shows where its first ghost lied, how turning the headwind around fixed it, and where it still falls short. Built with PriorLabs-TabPFN.

What it doesn't know

The wind is rebuilt from weather models, not measured on the bike, and on my rides AROME was a third weaker than the stations. A road between walls, trees or cliffs feels less of it than the grid says, a gap between two hills more. Drafting is invisible: on 4 October I rode part of the way on a friend's wheel, and no file says so. The ghost knows me only through my rides, mostly in light wind; on a stormy day it would be guessing. It's a finding about a ride already ridden, for one rider and one bike, and not training advice.

Where you ride, does the wind ever cost you more than a minute on a loop? I'd like to know how strong it has to blow before it really counts.

Old apps still call home

Dominique Siacci — Tue, 27 Oct 2026 14:20:56 +0000

Somewhere this morning, a phone opened an app that was built on our platform years ago. Its owner hasn't touched it in a long time. The person holding the phone never updated the app, or no update ever came, because nobody rebuilt it. The app started, asked our servers for its content, and got an answer it could read.

That only works because we still answer in the format the app was built for, even though our own formats have changed since.

The client you can't upgrade

Most advice on API versioning assumes one of two things about your clients: you control them, or you can retire them. A web app ships its own front end, so a breaking change is a coordinated deploy. A public API has a deprecation calendar, a sunset header, an email to the developers. Either way, at some point the old client stops being your problem.

A platform that publishes apps can do neither. The client is a binary on someone's phone. It was built on a day we didn't choose, and its next version, if there is one, comes from the store on a day we don't choose either. We don't force updates: an app that works keeps working, and nobody is asked to rebuild because our own formats changed. I've written about what that promise costs when the world around an app moves. This piece is about what it costs when we move.

And we do move. The engine gets new generations, our answers change shape, newer endpoints get a cleaner design than the old ones. Meanwhile, the apps published before each of those changes are still installed, still speaking the dialect they were built with, and still expecting an answer in it.

Dialects, side by side

When a new generation of the engine arrived, the old one didn't move. It kept its paths, its file layout and the shape of its answers. The new generation got its own tree next to it. The same servers answer both today, and an app built against the old generation is served the old generation, for as long as it's around.

A phone only gets the new generation with a new build of the app, installed from the store. Until then, it runs the old binary, and the app's answers in the old tree are still there for it: they aren't deleted, and they keep the shape that binary was built for. The new build reads its own answers, written next to them in the new shape.

Why not just force the update?

Forcing an update would move the work onto other people. An update we force is a rebuild the owner has to run, a submission they have to make again, and a download the user has to accept: two parties doing work so that we can retire one code path. Multiplied by every app on the platform, that's a lot of work on their side to save some on ours.

Keeping a dialect alive is work we do once, on our side, instead of asking every owner for a rebuild. It has a cost: we keep testing every old path, and we keep maintaining a reader for every format we still serve. It's the same reasoning that keeps a link printed on a flyer years ago opening the right screen in today's version of the app.

There's a limit, and it comes from outside. When a store, an operating system or a certificate retires something an old binary relies on, that app needs a rebuild, whatever our servers answer.

Still asking

Some of the apps calling our servers this morning haven't been rebuilt in years, and people still use them. Keeping them working means keeping their vocabulary: answering in the words they were built with, and retiring nothing while someone still depends on it.

If you run an API for clients you can't upgrade, how do you decide when an old version can finally stop answering?

I built a tool to prove my AI coach wrong about my Zone 2 rides. It sided with him.

Dominique Siacci — Sun, 25 Oct 2026 10:05:25 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

On 4 October my AI coach, an agent I run in OpenClaw, had planned a long ride in Zone 2: heart rate under 140, even on the climbs. A bit more than two hours in, the climb to Canareccia turned up to 11 % for 900 metres. I was in 34×34, the easiest gear on the bike, my legs turning at 57 rpm. My heart was at 170.

After my long rides, the coach keeps saying the same thing: my Zone 2 rides drift into tempo. Its notes even have a theory: I chase personal records on Strava segments. That Sunday I beat my best time up Canareccia by one second, so I won't argue with that one. What I wanted to know was something the curves don't say: on that 11 % stretch, at the slowest pace I can actually ride, was Zone 2 even possible?

Zone 2 is the one workout where holding back is the assignment: ride slowly, for a long time. I keep failing it, and working out why means a long look at the curves after each ride. z2analyzer does that chore: a small web app I run on my own machine, which gives the answer in two lines and keeps every analysis.

After a ride, I start a new analysis and drop the FIT file from my bike computer. A few seconds later, the page splits the minutes I spent above Zone 2 between three causes:

the road: even in my smallest gear, at the slowest cadence I really hold, the slope asked for more than my Zone 2 ceiling, even in the cool;
the heat: Zone 2 was possible in the cool, not at that day's temperature;
me: it was possible.

For the Canareccia ride, in two lines:

119 min above Zone 2: 45 imposed by the road, 5 by the heat, 70 mine.
Zone 2 was possible on 88 % of the kilometres, 72 % of the time.

Before a ride, I drop a GPX and pick a start time. It takes the temperature from the Météo-France forecast and tells me how much of the route I can ride in Zone 2, in kilometres and in time, and where it can't be done. The same loop, for Saturday 24 October at 8:30:

Zone 2 is possible on 89 % of the kilometres, but 70 % of the time.
11 passages where it is not: 6.8 km, 1 h 02 even at your slowest pace.

The two numbers differ because climbs are slow: they weigh more in time than in distance, and my coach counts in time.

I don't even have to drop the file: pointed at the folder where my FIT files land, z2analyzer judges each new ride on its own, and the verdict waits in the list.

It is not a route planner. It suggests no route and gives no training advice: it looks at a route I give it, or a ride I did, and says what the road allowed.

Demo

The video runs both on my own rides, from a new analysis to its result page. It shows minutes and percentages only, and no map or profile of the first and last 2 km of a ride.

Code

dsiacci / z2analyzer

z2analyzer

How much of a ride can really stay in Zone 2?

On some slopes, even in your smallest gear and at the slowest cadence you can actually hold, the power the road asks for is above your Zone 2 ceiling. Those minutes are the road's, not yours. z2analyzer sorts out which minutes are which, before and after the ride, so you don't have to read the curves.

It is a small web app you run on your own machine: open it, start a new analysis, drop a file, and the result page comes a few seconds later. Every analysis stays in the list.

After a ride: drop the FIT file from your bike computer. It splits the minutes you spent above Zone 2 between the road (impossible there, even in the cool), the heat (possible in the cool, not that day) and you (it was possible).
Before a…

View on GitHub

Python, Apache 2.0, 51 tests. z2analyzer serve is the web app: it runs on your machine, keeps every analysis, and its pages load nothing from the internet. prepare, validate and learn build your limits from your own FIT files; plan and review give the same verdicts from the command line. The tables in this post come out of z2analyzer validate, tools/experiments.py and tools/proof.py, and docs/results.md has them all, with what failed.

How I Built It

The slowest pace the road allows

The power needed to hold a speed on a slope is textbook physics: gravity, rolling resistance and air, divided by the drivetrain's efficiency. On a climb at walking pace, gravity is almost all of it. My bike has 50/34 chainrings and an 11-34 cassette, so the easiest gear, 34×34, moves the bike 2.1 m per pedal turn. Times the slowest cadence I really hold, that is the slowest speed I can ride. Where the power needed at that speed is above my Zone 2 ceiling, Zone 2 is impossible.

The grade doesn't come from the GPS or the barometer. It comes from the IGN's terrain model of Corsica (RGE ALTI, 5 m cells, open licence), read from files on disk, so no point of my rides is sent anywhere. Bridges and tunnels come from OpenStreetMap and are drawn straight; otherwise the profile dives into the river under the bridge. Checked against my bike computer's barometer, the total climbing agrees within 4 % on nine rides out of ten, and 8 % on the tenth.

Three jobs for TabPFN, and a straight line that won two of them

Three things in that calculation are personal and change with the conditions:

my heart rate for a given power, air temperature and time already ridden: inverted, it gives my Zone 2 ceiling in watts for any temperature and hour;
the lowest cadence I really hold in 34×34, for a grade and the time already spent climbing;
my speed riding easy, for a grade and a power, to turn kilometres into time.

I gave all three to TabPFN v2. It takes your rows as context and predicts new ones in a single forward pass, with a full distribution instead of one number. It runs on a CPU. My data: ten outdoor rides with power-meter pedals since mid-August, 24.7 hours cut into 30-second windows.

I scored it the only way that matters for a tool like this: each ride predicted by models that never saw it, against a linear regression on the same inputs and a constant.

On rides it never saw	TabPFN v2	Linear regression	Constant
Heart rate, mean error	8.1 bpm	6.5 bpm	16.2 bpm
Lowest cadence, loss at the 10 % quantile	1.82	1.59	1.60
Speed, mean error	0.70 m/s	1.20 m/s	2.29 m/s

TabPFN won the speed by a wide margin: speed against slope and power is curved physics, with descents where I brake. It lost the cadence, where my lowest cadence turned out to be almost a constant. And it lost the heart rate, for a reason I hadn't seen coming.

At the same power, my heart doesn't sit at the same level from one day to the next. On 16 August it ran 13 bpm below its usual level; on 19 August, a long hot day, 7 above. Sleep, fatigue, water: none of it is in a FIT file. Each of my rides also had its own range of temperatures, so TabPFN could all but recognise a ride by its temperature and copy that day's level onto the ride it was predicting. A straight line can't do that. When I took each day's level out of the training rides, TabPFN caught up: 4.85 bpm against 4.68 for the line, on the second half of each ride. It still didn't pass it.

So here is what z2analyzer does:

validate scores both models on your rides, and learn --models keeps the winner of each table. On mine: the line for the heart rate and the cadence, TabPFN for the speed. On yours it may come out the other way.
After a ride, it measures the day's level of the heart on the ride's own steady stretches, and the ceiling moves with it.
Before a ride, the range it gives for the ceiling is how far that level moved between my rides, rather than TabPFN's own interval, which was too narrow: it held the truth 59 % of the time instead of 80.
Every table stops where my rides stop and holds its edge value beyond. Without that limit, the line for the lowest cadence would fall to 40 rpm on the steepest slopes, while on my steepest climbs the slowest tenth of my cadences sits around 49.

A check that needs no model

The FIT file has no gear channel, but speed divided by cadence gives the gear away: on my bike the two easiest gears are 13 % apart, so the tool can tell when 34×34 was already in. That gives a test that depends on no model at all. Over my ten rides, where z2analyzer says Zone 2 was possible and my heart was above it anyway, an easier gear was still there 91 % of the time.

A web app that stays home

The web app is Python's standard library and nothing else: a small HTTP server, one folder per analysis (the file dropped, its verdict, its page), and one background thread that runs the analyses in turn, each in a few seconds. It listens on the machine it runs on, and nothing in its pages comes from the internet.

I built it to prove my coach wrong

Each of my ten rides, judged with limits learned from the nine others:

Ride	Air	Above Zone 2	The road	The heat	Me
14 Aug	22 °C	27 min	0	0	27
16 Aug	24 °C	4 min	0	0	4
19 Aug	29 °C	201 min	27	8	166
24 Aug	25 °C	18 min	0	0	18
29 Aug	25 °C	47 min	25	3	19
31 Aug	23 °C	16 min	0	0	16
13 Sep	21 °C	34 min	0	0	34
20 Sep	22 °C	84 min	1	1	82
27 Sep	19 °C	86 min	1	0	85
4 Oct	24 °C	119 min	45	5	70
All		637 min	98	17	522

Out of 637 minutes above Zone 2, the road made 98. I made 522. My AI coach was right four times out of five. And it isn't the model talking: with TabPFN for every table, the road gets 88 minutes and I get 516.

The road does make Zone 2 impossible on my steep rides: on Canareccia, my slowest possible pace needed 1.5 times my ceiling, and I was in 34×34 almost all the way. On the 58 km loop I rode three times in September, though, Zone 2 was possible on 98 to 100 % of the kilometres, and on 27 September, the day the coach wrote "not Zone 2", 85 of my 86 minutes above it were mine. Even 14 August, where I spent three quarters of those minutes in 34×34, goes to me: I was turning at 54 rpm on grades where I hold 46 when I have to. The heat barely counts, because I ride at dawn and a hot day's extra heartbeats partly show up in that day's level instead.

Two things the tool doesn't say. It counts minutes, not how hard: on Canareccia the road put me above Zone 2, it didn't make me beat my record. And it doesn't tell me to ride flatter roads; it just stops me blaming the island for the rest.

Why Does Open Innovation Matter?

A recorded ride is my heart rate second by second, my power and exactly where I ride, and I want to keep control of that. z2analyzer looks like a web service, but it's one I run myself: the files I drop stay on machines I run, and TabPFN's weights are downloaded once and run on a plain CPU, with no account and no cost per analysis. The terrain, the bridges and the weather are open data, and the only thing about a ride that goes out is one weather point, rounded to about 11 km.

The open weights did something else for me. Because I could run TabPFN on my own rides as often as I liked, I could test it against a straight line, find out where it lost, and keep the line there. Behind an API, every one of those runs would have sent my rides out.

And nothing in it is mine except the numbers. Your mass, your gears, the top of your Zone 2 and your own FIT files replace mine; without a power meter, z2analyzer fixed builds the limits from three numbers.

The Lazy Payoff

Nobody does this analysis by hand. Going through every 30 seconds of a three-hour ride, working out the slowest pace the slope allowed and comparing it with a ceiling that moves with the heat and the hours, is too long and too tedious to do objectively, so I never did it: I argued with the curves instead. Now it's a new analysis, a dropped file and a few seconds, or nothing at all when the ride lands in the watched folder. This week I ran it on my ten rides since August.

Prize Categories

Best Use of TabPFN. TabPFN v2 turns kilometres into time: it predicts my speed riding easy from the slope and the power, with 40 % less error than a linear regression on rides it never saw, and time is what my coach counts. It was also tested on my heart rate and my lowest cadence, where it lost to a straight line, and z2analyzer picks the model of each table from that test, on your own rides. Built with PriorLabs-TabPFN.

What it doesn't know

Wind isn't counted. The Zone 2 ceiling is the top of my coach's Zone 2, 145 bpm, a little above the 140 it asks for on long rides. The numbers hold for one rider and one bike, mine, and the models only know the conditions of my rides: below 19 °C, for instance, the tool counts the air as 19 °C, and it says so. It's a finding about a route or a ride, not training advice and not medical advice.

If you ride in the mountains: how do you keep your Zone 2 on climbs like these? Do you pick flatter roads, get off and walk, or accept the minutes?

One letter higher: checking my saddle height with a side-view video, in the browser

Dominique Siacci — Fri, 16 Oct 2026 11:37:41 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

My road bike got a professional fit last year. My home trainer, a Zwift Ride, is set with letters, A to X, and I didn't know whether the letter my saddle sits on was the right one. On Thursday evening I put my phone on a tripod, two or three metres from the bike on the drive side, and pedalled for two minutes. Then I raised the saddle by one letter and did it again.

I'm an amateur cyclist, not a bike fitter. What I wanted was a number for one question, from what a phone video can actually measure.

Saddle Check is a web page. You drop a video of yourself pedalling, filmed from the side. An open pose model, MediaPipe Pose, runs in the browser, on your own computer or phone, and the video is never uploaded. The page gives you:

the knee bend at the bottom of the pedal stroke, stroke by stroke, against the published range for riders measured while pedalling (33 to 43°);
the saddle change in millimetres, and in letters or notches if you give their spacing, that would bring the knee to the middle of that range;
the other angles (hip, trunk, shoulder, elbow, ankle), next to book or manufacturer guidelines, labelled as such: inside or outside, never "good" or "bad";
an animated skeleton on a plain background (never the video picture, unless you ask), the curve of each angle through the stroke, and a report to download.

Who would pay? A bike fit costs from €90 (static) to €399 (full) in the French shops I checked. Apps exist: FramIQ, on iPhone, also analyses the video on the device, for $25 per analysis or $79 a year; MyVeloFit uploads the video to its servers, for $35 or $75 a year. Saddle Check is a prototype, but a paid version could find its place in that landscape, below those prices and with nothing to install: a report to bring to a bike shop, a follow-up after each adjustment, a version for a club that shares indoor bikes. The rider's own device does the computing, so an analysis costs almost nothing to serve, and even a low price would leave a margin.

Demo

The demo uses the page's synthetic rider, so no real person appears in it. To run the page yourself, the README has the three commands that serve it from your own machine; the same synthetic-rider button is there when you have no video at hand.

Code

dsiacci / bikefit

Saddle Check

Bike fit angles from a side-view phone video, measured in the browser. You drop a video of yourself pedalling; an open-weight pose model (MediaPipe Pose Landmarker) runs inside the page and finds your joints in every frame; the page measures your knee, hip, ankle, trunk, shoulder and elbow angles at every pedal stroke, compares the medians with published ranges, and turns the knee angle into a saddle change in millimetres.

The video never leaves your device. The page and the model are static files; once the model is loaded, the page locks its own network access (see Privacy).

Demo video, with the page's synthetic rider: youtu.be/JUidqkAi33U.

Use it

The page is a set of static files. To serve it from your machine:

npm ci
npm run build   # copies the MediaPipe runtime, downloads the models, checks their SHA-256
npm run serve   # http://127.0.0.1:8080

How

…

View on GitHub

Plain JavaScript, no framework, no runtime dependency besides MediaPipe, Apache 2.0, 32 tests. The measuring code (web/core/) is the same for the page and for the scripts that produced the numbers in this post.

How I Built It

Reading every frame, without sending it. The page reads the video frame by frame with WebCodecs, the browsers' decoding API, fed by a small reader for MP4 and QuickTime boxes: each frame is decoded once, in order. My first version asked a <video> element for each frame, which decodes again from the last key frame: 226 s for a 14 s clip, against 36 s now, with the same numbers. Each frame goes through MediaPipe Pose Landmarker, Google's open-weight pose model (Apache 2.0, the "full" variant), which returns 33 body landmarks.

Cutting the network, and proving it. Once the model is loaded, the page adds a security policy to itself that forbids every connection (connect-src 'none'), then checks that it holds: it tries to read a file from its own site and makes sure the browser refuses. A padlock shows in the header. In Chromium's console, that reads: "Connecting to '…/lock-probe.txt' violates the following Content Security Policy directive: "connect-src 'none'". The action has been blocked." Anyone can check it in their own browser.

The crank is the ruler. Millimetres need a scale, and it's in the picture: with cleats, the shoe is fixed to the pedal, and the pedal axle is the only point of the foot that turns in a circle around the bottom bracket. The page searches the frame of the foot (the heel and toe landmarks) for the point whose path is the most regular circle. Its radius in pixels is the crank: 170 mm on my bike.

The right leg. Seen from the side, the model sometimes swaps the left and right labels of the legs. The page follows the leg nearest to the camera by continuity and corrects those swaps (12 to 18 per video in mine).

One pedal stroke. Positions are cleaned (unsure landmarks dropped, a zero-lag 6 Hz low-pass filter), the crank angle splits the video into strokes, top to top, and the angles are read at the bottom. The page keeps the median over clean strokes, with its confidence interval.

Ranges, with their level of evidence. For the knee at the bottom of the stroke while pedalling: 33 to 43° of bend (Millour et al. 2019; Swart & Holliday 2019); that range drives the saddle change. The most quoted range, 25 to 35° (Holmes), is measured standing still with a goniometer; pedalling bends the knee about 8° more. For the hip, the trunk, the shoulder or the elbow, I only found book or manufacturer guidelines: the page shows them without saying "good" or "bad".

The saddle change. The thigh, the shank and the hip-to-ankle line make a triangle. The law of cosines says how much longer that line must get for the knee to reach the middle of the range, and the page turns it into millimetres along the seat post.

I built it this week with an AI coding agent.

Does It Measure Anything?

Five videos at 60 frames per second, tripod and bike taped to the floor: V1, V2 (after getting off and back on) and V4 (saddle set back to its letter) at the same height; V3 one letter higher, 10 mm; S, still, crank down, to compare with an inclinometer.

Video	Saddle	Knee bend at the bottom (median, 95 % interval)	Clean strokes
V1	start	42.3° (41.8 to 42.5)	114 of 133
V2	start	43.0° (42.6 to 43.5)	76 of 92
V4	start	42.8° (42.2 to 43.1)	96 of 115
V3	+10 mm	37.9° (37.3 to 38.2)	122 of 134

Repeatability. Three videos at the same height, one after getting off the bike and one after resetting the saddle: 0.7° between the lowest and the highest.

Sensitivity. One letter higher, the knee bends 4.8° less (95 % interval 4.3 to 5.4). The leg triangle predicted 3.4°. The direction and the size are right. My ankle at the bottom didn't change clearly, but my hip moved 12.4 mm in the picture for a 10 mm saddle change: I may have sat a little differently, or the scale is a few percent off, and these videos can't tell which.

The page called it, this time. On my three starting videos, it suggested raising the saddle by about one letter (11 mm) to reach the middle of the range, 38°. One letter higher, it measured 37.9°.

Reference. Still, crank down, the model reads a knee bend of 33.2°. My inclinometer (the level of the iPhone Measure app held along the thigh, then the shank, read by me, alone on the bike) gives 34°, and the software of last year's fit, made for my road bike, said 34° too. Pedalling, the page measures 42.7°, 9.4° more than still, close to the 8° the studies describe. One thing doesn't match: the thigh and the shank taken one by one differ by 11 and 12° between the model and the inclinometer, turned the same way, which cancels out in the knee angle. One of the two was tilted by about 12°, my readings or the camera's horizon, and these videos can't tell which.

What these numbers don't say. My 0.7° says the page is consistent, not that it's right to 0.7°: the only study of MediaPipe in cycling I found (Bini, Nascimento & Nibali 2024) reports typical errors of 3 to 6° at the knee.

What Went Wrong

I had tuned the threshold that decides "this point of the foot really turns in a circle" on synthetic riders, where the path stays within 2 % of a circle. On my real videos, the foot landmarks the model returns stray from the circle by about a tenth of the crank. The page refused the pedal axle and fell back on the loop of the ankle: no millimetres, and a badly placed bottom of the stroke, with a knee at 39° instead of 42°.

Looking at the numbers, I saw that this scatter changes from one stroke to the next, while a foot sliding on a pedal bends the loop the same way every time. So the page now averages the pedal path angle by angle over dozens of strokes before judging the circle, and it still refuses a synthetic foot sliding 70 mm. The radius it finds still varies by 5 % from one video to another, so the page gives its millimetres to within 10 %. The angles don't depend on the scale.

Why Does Open Innovation Matter?

An open model can go where the video is. Because the model is open, the page can run it on the rider's own device instead of a server, so the video can stay there. The page also switches off its own network once the model is loaded: the browser console shows it, and the code shows the rest.

An analysis costs almost nothing. The model runs on the rider's device and the page is a set of static files: there is no computing server to pay for, which is what would keep a paid version cheap.

No lock-in. The code is under Apache 2.0, the report is an HTML file you keep, and the ranges are listed with their sources: a bike shop can check where every number comes from.

Prize Categories

Best Use of DigitalOcean. A DigitalOcean droplet ran the page itself in headless Chromium for the many automated tests: every number in this post comes from those runs on my five videos, and the demo video was recorded there too. The repository has the scripts to do it again.

So, will I raise my saddle? The page's answer is about one letter: at my current height, my knee bends 42.7° at the bottom of the stroke, inside the published range but near its top; one letter higher (V3), it bends 37.9°, right in the middle. I'm not sure yet that I'll do it: sitting still, my trainer already matches last year's fit, and 42.7° is still inside the range.

How did you set your saddle height, and would you move it by a centimetre because a video told you to?

964 hours of agent work in six days. Here's where my own hours went.

Dominique Siacci — Thu, 15 Oct 2026 13:09:45 +0000

On October 7 I dictated the plan to an agent: we would start an autonomous loop, I would close my session, come back a few days later, and the 200 templates of our app designer would be done.

The last check came back green on the morning of October 15: 12,074 end-to-end cases, zero gaps. In between, 28 coding agents worked on it, up to 20 at a time, for 964 hours of session time in total, with more than 6,000 commits and 717 deployments to our dev environment. The 200 templates turned out to be 569 units of work. And I never really closed the session.

I'm the CTO of GoodBarber, an app platform. The agents wrote the code. This post is about where my own hours went, because that's the part I had wrong.

The job

In our back-office, the logic that applies a setting sits with the part of the UI that edits it. The refactor moves that logic into documented functions with typed options, which the UI and a new API both call, and proves along the way that nothing changes for the people using it.

The unit of work is an abstraction: something that exists in the UI and defines a set of controls over one kind of entity, in one layout variant. A script cut the whole app designer into 552 units 13 minutes after I dictated the rules for the cut, then 569 after a second pass.

	The numbers
Design	2 weeks, Sep 23 to Oct 7
Execution	6 days, Oct 9 to Oct 15
Units	569: 445 done, 124 dropped or merged, mostly duplicates
Controls	about 20,600
Agents	28 workers, up to 20 at a time, and one integrator
Session time	964 h
Commits	more than 6,000
Deployments to dev	717
Final campaign	12,074 cases, 0 gaps

Two weeks before the six days

Before those six days, I spent two weeks on the design, from September 23 to October 7. Most of it ran at Claude Opus 5.5's highest effort level (max), with around 200 sub-agents exploring the code and reviewing proposals. For the first week I also had ultracode on, a setting where the model plans a multi-agent workflow for each substantial task. On the first day, 32 agents spent five hours on the strategy alone: 18 explored, 5 wrote competing proposals, 3 judged them, 1 combined the best of them, 2 attacked the result and 3 ran the final checks.

I used to think of the loop as the project. In practice it was the last part, and it held because the two weeks before had settled what a finished unit looks like.

What "done" meant

The checks were the review. A unit went to our dev environment when they passed, and merging into production code stays a human step.

All the checks prove one thing from different sides: for every control and every test value, the new code path leaves the application in exactly the same state as the old one, the change can be undone exactly or is listed as an exception, and the API behaves like the UI. This is how often each one caught something:

Check	Units where it caught something at least once
A headless browser performs the real gesture, and the test bench must send exactly the same request	140
A snapshot of the whole application before and after each test value, old path against new, then an exact undo and redo	160
The API must do what the UI does, read back what it wrote, and reject invalid input without side effects	114
A plain-language request written by one agent, run by another that only sees the documentation, must end in the expected state	40
The final campaign on the deployed code, through real gestures and through the API	0 gaps in 12,074 cases

When a check failed three times, the unit was parked on a question or marked as blocked, and the worker moved on. In the final campaign, 48 cases were impossible, each one traced to the line of code that makes it so.

One unit, then ten, then twenty

I didn't start with twenty.

When	Step	Scale	What came out of it
Oct 7–8	Step 0	1 unit, in a guided session	11 changes to the process
Oct 8	Step 1	10 units in one loop, one after the other	8 more changes
Oct 9	Pilot	4 workers and an integrator	16 improvements, live during the first wave
Oct 10	Step 2	8 workers	first weekend: 183 units, 930 commits
Oct 11, evening	Step 2	20 workers	overnight into Monday: 173 units in 11 hours

The conditions to go from four to eight were written down before Saturday: the same proofs as when units ran one after the other, no worker touching another worker's test app, no two deployments overlapping. Between the two waves of the pilot, the same four workers got much faster, mostly thanks to fixes made during the first wave, like a pre-check that runs the tests on a throwaway copy before asking for integration:

Pilot	Wave 1	Wave 2
Median time per unit	2 h	54 min
Integration rejections	9	0
Median wait for integration	6 min	0.8 min

How it was organized

One session, the integrator, never takes a unit. It's the only one allowed to merge and deploy, and it fills the queue, reopens units and keeps the dashboard up to date. The workers pull: each one asks the queue for the next unit compatible with its own test app, works on its own branch in its own worktree, and asks for integration when its checks pass. Integration runs in batches of up to six, with one pre-check and one deployment per batch.

Nothing goes through chat. The queue, git and a dashboard carry the whole protocol. The protocol itself is a file in git that every session re-reads at each round, so I could change the workers' instructions 53 times during the project without restarting anyone. When a worker hasn't sent a heartbeat for 45 minutes, its unit goes back to the queue and someone else picks it up.

Where my hours went

Two weeks of design, first. Then the questions.

A worker that hit a case it couldn't decide alone didn't wait for me. It wrote a question on the dashboard with the context, two or three options, what each one would cost, and its recommendation, then parked that unit and took the next one. It decided alone on anything reversible that users don't see. It had to ask about an architecture choice, a behavior it couldn't prove identical, a proof that needed to break the loop's own rules, or the scope.

There were 88 questions, from the first loop on October 8 to the end. Of the 59 that had both a recommendation and an answer from me, I followed the recommendation 47 times. One I didn't follow: proving one control meant calling a third-party service, which the loop's rules don't allow from a test. The recommended option was to simulate the service, another was to leave that control out of the API, and the third was to call the real service from a test app and break the rule. The question came in at 13:41, I picked the third option at 13:48, and it was applied at 13:53. That's the kind of decision I wanted to keep for myself.

From the first evening, a case that looked like one I had already answered stopped waiting too. The worker applied the precedent, logged it, and I validated or rejected it afterwards. There were 185 of those: 166 validated, 19 rejected. Ten of the 19 rejections were about the same thing, an agent excluding something that resisted. That one deserves its own post.

Decision fatigue

Put end to end, my answers would fit in an afternoon, somewhere between three and a half and five hours. That number is the misleading part.

They didn't come in one afternoon. They came all day, every day, and they piled up whenever I looked away. On Sunday I spent the day unblocking workers by hand and barely touched the dashboard. On Monday I cleared the pile, 63 decisions in one hour, about one a minute, most of them precedents to check. Each one was small and well framed, with its options and a recommendation, and that's what makes them tiring in bulk: by the fiftieth, the recommendation is the easy way out.

I rejected more precedents that Monday than on any other day, 13 out of 95, so I don't think I was rubber-stamping. But over the week I followed the recommendation 47 times out of 59, and from the logs I can't tell which of those were judgment and which were fatigue. I'll come back to that in a separate post.

What broke

The test bench lied on day one. It built its own requests instead of sending what the UI sends. I tested a migrated control by hand and it rejected every color coming from the color picker, which sends them without a "#". Since then a real browser replays the gestures, and the bench has to send exactly what the UI sends. That browser found format differences on about a hundred more controls along the way.
The single integrator became the bottleneck. Every integration went through one session, and the wait grew with the load (table below).
A worker went silent for almost three days, and nothing told me. Heartbeats were tracked for units in progress, so a worker holding no unit was invisible to the integrator. The others kept going.
Agents drop what resists. On Tuesday, an audit run by one of the workers found 516 controls left on the old path without any decision. I had all of them brought back into scope.
A rule nothing checked got broken. One worker removed a line from a file that the rules say is append-only.

Day	Integration requests	Median wait
Fri Oct 9	46	0.9 min
Sat Oct 10	48	1.0 min
Sun Oct 11	145	2.9 min
Mon Oct 12	251	11.0 min
Tue Oct 13	196	17.5 min
Wed Oct 14	29	1.6 min

At the worst hour, early on Sunday morning, the median wait passed an hour.

What's still true

The merge into our production code is still mine to do, after my own tests by hand. A loop never merges into production.
132 changes that can't be undone automatically are listed, not solved, and I have to read them before anything ships.
What I'd do again: design at the highest effort, one automatic proof per unit and an end-to-end final campaign, scaling in steps with a review at each one, and questions that ask for a rule rather than a case. What I'd watch: the single integrator, the scope, my own decision load, and the pulse of every session, not just of every unit.

If you've run several coding agents in parallel on something real, where did your own hours go? Mine went into two weeks of design and more than 250 small decisions. I'll answer in the comments with how the queue and the dashboard work.

Will the fountains on my ride be running? I asked 3,831 stream observations

Dominique Siacci — Fri, 09 Oct 2026 07:27:56 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

There's a fountain on the road just before Marato, a stone basin behind a small railing, that I count on when I ride south of Ajaccio. It runs well. OpenStreetMap doesn't have it. IGN, the French national mapping agency, does, but 55 m away, in the woods.

I ride the same roads, slowly, and I know which of their fountains run in September. Riders passing through don't. OpenStreetMap lists 608 places to fill a bottle in Corsica; 2 of them say whether they dry up in summer, and 33 carry a date when someone last checked them. The maps know roughly where the fountains are. Nobody writes down whether they are running.

So I wrote fountains, a small open-source tool that runs on a laptop. You drop the GPX of your ride on a local page, and:

it lists every fountain OpenStreetMap or IGN knows within 250 m of the route;
it adds the places where open data hints at a fountain no map has: a spring IGN puts next to the road, a stream crossing under it;
it shows each place on photos taken from the road, an open vision model putting first the views where it sees something like a fountain, and you decide: fountain, not a fountain, or not sure. When no view shows it, you look around in 360°, walk along the road from photo to photo, and click the foot of the fountain to put it where it really is;
the fountains you keep get a band for the day of the ride, likely, uncertain or unlikely to be running, with the reason: the rain of the last weeks compared with normal, the forecast, the altitude, the nearest stream someone actually walked to;
on long stretches without a likely fountain, it suggests a café, a bakery or a fuel station open when you pass, about every 10 km; a fountain you know runs cuts those stretches;
one GPX goes onto the bike computer, with waypoints named like likely: Funtana di Leccia.

For a 28 km ride, the preparation takes about twenty minutes of the laptop working alone the evening before, then a few minutes of looking at photos. The rest happens on the road.

One rule is printed in every output of the tool, and I'll repeat it here: a score is never a reason to leave with less water. Carry what you need, as if every fountain were dry. The tool also says nothing about whether the water is safe to drink.

Demo

The video follows a real preparation, on my laptop: a 28 km out-and-back to La Parata. The 19 minutes in which the tool finds the points, the clues and the photos, and the vision model ranks every view, go by in ten seconds. The rest shows the cases I ran into:

at the Fontaine des Calanques, the model boxes a red fire hydrant: it ranks photos, it decides nothing;
at Funtana di Mela, an old stone fountain, I mark it on the photo, and the GPX puts it 22 m from IGN's point;
the places found from clues, mostly streams under the road, take one answer each: none of them was a fountain;
for Saturday, none of the four fountains I kept comes out likely: 28 km without a likely fountain, so the tool lists five shops open when I pass; marking the three fountains I know run brings the dry stretch down to 13 km.

That's the fountain before Marato, on my usual 70 km loop. The tool took the photos of the road around IGN's point, and the view the model ranks first is the fountain, seen from the road about twenty metres away. Marked on two photos taken 29 m apart, it lands 4 m from the road: the two lines of sight cross inside the box the model had drawn. IGN's point is 55 m away, in the woods.

On that loop, the maps know 7 water points within 250 m of the road. The clues add 29 places, 24 of them streams under the road. On 18 of them, the model sees nothing above 0.2; the page sets them aside (a switch brings them back), which leaves 18 points to look at.

Code

dsiacci / fountains

Will the fountains on your ride be running? Finds the fountains along a GPX in Corsica, checks them on street photos, bands them with TabPFN. Runs on your laptop.

fountains

A score is never a reason to leave with less water. Carry what you need for the whole ride as if every fountain were dry. This tool says nothing about whether the water is safe to drink.

Will the fountains along your ride be running on the day you ride?

You export your route as a GPX file and drop it on a local page. It lists every fountain that OpenStreetMap or IGN (the French national mapping agency) knows within a short detour of the route, adds the places where clues in open data suggest a fountain no map has, and shows each one on street photos so you can say which are real. The fountains you keep get a band (likely, uncertain or unlikely to be running) for the day you ride, with the reason and the date anyone last checked them. On long stretches without…

View on GitHub

Python, Apache 2.0, 97 tests. fountains serve opens the page, fountains score does the same computation from the command line, and a Docker image runs the page on macOS or Windows. Every data source with its license, and the limits of the tool, are in the README.

How I Built It

Fountains on the maps. OpenStreetMap, through the Overpass API: amenity=drinking_water and water_point, plus springs, taps and fountains tagged drinking_water=yes. And IGN's topographic database, open since 2021, which knows 1,493 fountains in Corsica; only 250 of them have an OpenStreetMap point within 30 m. Each point is attached to the nearest road or path, and the detour is measured along the network: a fountain 40 m from the road can be 400 m away if the only access is a loop through the village.

Fountains on no map. Two kinds of clues, both in open data: IGN springs, wash houses and water points close to a paved road, and the places where a stream crosses one, because a spout is often built where the slope brings water down to the road. For each place, the tool takes the 360° photos within 100 m on Panoramax, the open street-imagery commons (in Corsica, mostly IGN's 2025 captures), cuts them into views of both roadsides, and asks OWLv2, an open-vocabulary object detector, for "a water fountain", "a water spout", "a stone water trough", "a water tap". Boxes touching the bottom of a view are ignored: that's where the bonnet of IGN's car shows. The model decides nothing; it sorts photos. Near Marato, its first view was my fountain; elsewhere, it took a fire hydrant or a decorative urn for a fountain. So I look at the photos, and I don't trust the score.

Putting the fountain where it is. A map can be fifty metres off, and a GPX that sends you into the woods is no use. When I click the foot of a fountain on a photo, the tool knows where the photo was taken and the direction of the click: that's a line of sight. Two photos give two lines, which cross at the fountain. With one, the distance is guessed from the angle below the horizon, for a camera on a car roof; the page says it's a guess, and the pin can be dragged on the map.

Rain. Météo-France opened all its public data on 1 January 2024, including the daily rain gauges: 103 gauges in Corsica have reported since 2010, 52 still do. The tool interpolates the rain at each fountain from the three nearest gauges, compares the last 30, 90 and 180 days with the 1991-2020 normal, and adds the Météo-France forecast (through Open-Meteo) for the days between the last report and the ride.

Labels. This is the part I didn't have. Nobody publishes whether fountains run, but France publishes whether small streams run: every summer since 2012, agents of the French Office for Biodiversity walk to the same points on small streams and write down what they see, from flowing to dry. The network is called ONDE. In Corsica that is 33 streams and 3,831 observations. I kept one question, is flowing water visible?, because a fountain only fills a bottle if water flows, and because it's the only split the observers used the same way every year (from 2016 to 2019 they didn't separate normal flow from weak flow).

The model. TabPFN, from Prior Labs, is a foundation model for tabular data: a transformer pre-trained on millions of synthetic tables. You give it the 3,831 rows as context (day of the year, altitude, rain over 30, 90 and 180 days, how the last 90 and 180 days compare with normal, and whether water was flowing), then the new rows, and it returns a probability for each in one forward pass. No training loop, no hyperparameters to tune. I use the open v2 weights, on the CPU: scoring a ride takes half a minute on my laptop.

How much to trust it. I hid each stream in turn and asked the model about it, given only the other 32. Over the 3,831 hidden observations:

	Brier score (lower is better)
TabPFN	0.114
Logistic regression, same inputs	0.115
The rate for the month	0.127
Always "83 % flowing"	0.141

TabPFN is a little better than a logistic regression on the same inputs, by a margin too small to matter, and both beat the calendar by about 10 %. The ratios to normal earned their place: without them, TabPFN scored 0.118.

On streams it has never seen, its probabilities are too sure of themselves: cases it scored around 0.85 were flowing 72 % of the time, cases around 0.15 were flowing 43 % of the time. So I fixed the rule for the bands before the first run, and let the hidden streams set their limits: likely where at least 90 % of the hidden cases were flowing, unlikely where at most 50 % were. In the validation, "likely" streams were flowing 1,917 times out of 1,965, "uncertain" ones 1,232 times out of 1,734, "unlikely" ones 43 times out of 132. That's why the tool shows bands and not percentages: a percentage measured on streams would claim more than I know about fountains.

The weak link. A fountain is not a stream. The model assumes a fountain fed by a spring dries like a small headwater stream after the same weather. That's plausible for a village on a shallow spring, wrong for a tap on the town mains, and unknown for a deep spring that reacts months later. The page says so, and you can mark a fountain you know runs: it then counts as water, whatever the model says.

On Saturday I'm riding the route from the video again, to La Parata, and I'll update this post with what the four fountains actually did.

I built it this week with an AI coding agent.

Why Does Open Innovation Matter?

Every piece of this is open, and none of it would exist otherwise. The rain is measured by Météo-France and free to reuse since 2024. The stream observations are walked and written down by public agents and published under an open licence. The fountains were mapped by volunteers on OpenStreetMap and by IGN, whose database and street photos are open too. The models are open, so the whole thing runs on my laptop, with no account and nothing to pay, and it will give the same answer next year. Nobody would sell an API that says whether the fountain in a Corsican village is running; the market is a few hundred cyclists and hikers.

And it can go back. When a rider sees a fountain running or dry, the right place for that is OpenStreetMap's check_date and seasonal tags, and a fountain found on the photos belongs on the map too, where the next person (and the next version of this tool) will find it. The tool never edits the map itself; that stays a human's call.

Prize Categories

Best Use of TabPFN. TabPFN is the model that turns 3,831 stream observations, and the rain before each of them, into a probability that water is flowing, for streams it has never seen and then for fountains. The table is small, the classes are unbalanced (83 % of the observations are flowing), and what matters is whether the probabilities can be trusted, which I measured stream by stream. It did as well as the best simple baseline I tried, with no tuning at all, in one forward pass on a laptop CPU, and the validation told me exactly how far to trust it. Built with PriorLabs-TabPFN.

How do you deal with water on long rides or hikes, when you don't know the fountains? Do you trust them, carry everything, or ask at the café?

Asked in Dutch, sent to an English menu

Dominique Siacci — Tue, 06 Oct 2026 07:20:45 +0000

This is a submission for the Kaggle Benchmarking Challenge.

A Dutch reader asks where, in the back office, they set the colors of their app's interface, role by role:

Kunt u me het menupad in de back-office geven waarmee ik het kleurenpalet van de interface per rol instel?

The model gets the English edition of the article that answers it. Fourteen models on Kaggle got that exact message, and all fourteen gave the same path, thirteen of them in a Dutch sentence:

Het menupad is My App > App Style > Colors.

On a Dutch screen, that menu reads Mijn App > App-Stijl > Kleuren. The fact is right, the language is right, and the menu is the one from another edition. Dutch is close enough here to find your way. A Portuguese reader handed the German edition of another article gets less help: ten of the fourteen models told them, in Portuguese, to choose „Mit KI erstellen", the German edition's label.

I co-founded GoodBarber, an app platform, and I run its engineering. Our blog comes in seven languages, and much of it names a back-office menu or an app's button. A retrieve-then-answer loop over these pages doesn't always return one in the reader's language. I wanted to know what happens to the label then.

MTM-Bench already measures how the content's language leaks into replies, and another entry in this challenge checks Italian UI strings. This one scores the label the reader acts on, against their own edition, with controls where English is right.

TL;DR

The models copy the label from the page: 58.8% of 5,266 answers on localized labels named only another edition's label, all but two the page's own. Only 1.9% came back in another language.
English pages are the worst case: 0.14 on average, against 0.22 to 0.35 for the other page languages.
GPT-6 Astra (0.689), Claude Opus 5 (0.592), GPT-5.5 (0.499) and the open-weight Qwen3 235B (0.483, $0.10 a run) clearly beat the always-English baseline (0.32). The other ten score 0.286 to 0.405.
One sentence in the system prompt, asking for the reader's labels, takes two Gemini Flash models from about 0.37 to 0.77 and 0.86, past the best model without it.

What I Benchmarked

The material. Eleven items from ten articles of our blog, each something a reader acts on: four back-office paths or options, five strings an app shows, and two controls whose English name is right in every edition (AI Extension Builder, a Lost & Found board).

Item	English edition	French edition	German edition
Colors by role	My App > App Style > Colors	Mon app > Style de l'app > Couleurs	Meine App > App-Stil > Farben
Start a new game	Play again	Rejouer	Nochmal
Podcast episode already started	12 min left	Il reste 12 min	noch 12 Min.

I checked every back-office label on screenshots of the real back office in the seven languages; the label on the reader's screen also gets full credit.

The minimal pair. The reader asks in their language, Q. A simulated retrieval step puts two blog sections in the message, in language C: the one that holds the answer and one on a nearby subject. Everything else is fixed; only C changes. Seven reader languages times seven page languages make 49 cells per item. In the seven cells where C equals Q, the page already holds the reader's label: these same-language cells check the setup and aren't scored. That leaves 462 scored cells per model.

The system prompt says nothing about the language of the answer:

You are the reader help assistant for a software company's blog. Answer the reader's question using only the retrieved documents provided in the message. If the documents do not answer the question, say so. End your answer with the id of the document you relied on.

"Using only the retrieved documents" bounds the facts, not the wording: the models already write their answer in the reader's language, and a label they translate themselves gets full credit, so nothing beyond the page is needed to score 1.

The scoring. No model judges the answers. The scorer is code, with 39 tests, and three gates:

The fact is there: a label from any edition, or the item's key concepts. Otherwise 0.
The answer is in the reader's language, once quotes, URLs and ids are stripped. Otherwise 0.
The label:

Outcome	What the answer contains	Score
Reader's label	the reader's label, no other edition's	1
Both	the reader's label and another edition's	0.5
Own translation	its own translation, no other edition's label	1
Other edition's label	another edition's label, not the reader's	0
Control kept	the English name, on a control	1
Control translated	a translated name, on a control	0

Pasting a label from an edition the reader doesn't see costs points, which is why both labels score half: the reader still has to work out which one is theirs.

Baselines, run through the same scorer

Baseline	Score
Oracle (the reader's edition, word for word)	1
The reader's label everywhere, controls translated too	0.84
Always answer with the English label	0.32
Copy the label from the retrieved page	0.19
Vague answer ("go to the settings")	0
Refusal	0

The second row loses only the controls: translating everything doesn't score 1 either.

Models Tested

Fourteen models, one run each on Kaggle's model proxy, from September 30 to October 2. The lineup is drawn from the roster Kaggle offers for benchmarks, not from what we run in our products: a large and a small model from each vendor, the latest generation where I could, plus open-weight models, to see whether size or vendor changes what happens to the label. Reasoning was set to low, with at most 4,096 output tokens and four attempts per cell. A run that loses more than 10% of its cells to proxy errors posts no score. Three aren't measured: gpt-oss-120b lost too many cells to rate limits in three runs at different hours, and every request to Gemma 4 26B and Grok 4.6 came back with an error (400 and 404).

Findings

Model	Score	Reader's label	Both	Own translation	Other edition's label only	Wrong language	Run cost
GPT-6 Astra	0.689	146	91	43	98	0	$3.82
Claude Opus 5	0.592	80	193	13	92	0	$5.00
GPT-5.5	0.499	89	77	19	192	1	$2.35
Qwen3 235B	0.483*	73	30	42	201	2	$0.10
Claude Sonnet 5	0.405	44	96	12	222	4	$1.49
Claude Haiku 4.5	0.379	30	102	11	205	29	$1.36
Gemini 3.7 Flash	0.377	9	140	11	218	0	$0.44
Gemini 3.8 Flash	0.374	11	132	12	223	0	$0.31
Gemini 3.1 Pro	0.345	20	81	15	261	0	$3.42
GLM-5	0.337	17	77	17	265	2	$1.30
DeepSeek-R1	0.330	37	73	7	223	37	$1.71
GPT-5.4 nano	0.317	23	41	21	278	14	$0.11
Gemini 3.1 Flash-Lite	0.303	24	32	17	303	2	$0.21
GPT-5.4 mini	0.286	29	8	15	315	7	$0.45

The score covers the 462 scored cells; the counts, the 378 cells of the nine localized items. *Qwen3 235B: 431 scored cells, 352 localized.

The models copy the label from the page. On the localized items, 3,096 of the 5,266 answers (58.8%) named only another edition's label, and 3,094 of those named the page's own. Copying is right in two places, and the models get both: on the same-language cells and the controls, every model but DeepSeek-R1 scores at least 0.97. Everywhere else it's wrong, yet the most common outcome. The language itself rarely leaks: 1.9% of answers came back in another one. DeepSeek-R1 has a different problem: it answered in English 58 times across its 539 cells, 10 of them where the page was already in the reader's language.

English pages are the worst case. With an English page, the localized items average 0.14, against 0.22 to 0.35 for the other languages, and 78% of answers named only the English label. Even GPT-6 Astra drops to 0.278, from 0.50 or more elsewhere. Asked by a French reader about a game's results screen, with the English page, all fourteen models named Play again; two added (Rejouer).

The leaders get there differently. GPT-6 Astra most often gives the reader's label alone (146 cells) or its own translation (43). Claude Opus 5 most often gives both (193), which scores half. Tied for third, GPT-5.5 and Qwen3 235B copy nearly as often as Claude Sonnet 5 (51% and 57% of localized cells, against 59%) but commit to one label far more often (108 and 115 cells, against 56).

12 min left is the one label the models often translate themselves: 29% of its answers, when no other item goes above 5%. It's also the only label with a number in it, a lead from a single item. It scores 0.592, Play again 0.077.

One sentence in the prompt changes most of it. I reran the grid on three models with one more sentence in the system prompt: "Name menus, buttons and other interface labels as the reader's own screen shows them, in the reader's language, not as the documents write them."

Model	Without	With	Other edition's label only
Gemini 3.7 Flash	0.377	0.855	218 → 37
Gemini 3.8 Flash	0.374	0.768	223 → 63
GPT-5.4 nano	0.317	0.405	278 → 235

With it, both Flash models beat GPT-6 Astra without it. Nano gains much less, and 3.8 Flash translated 4 of the 84 controls.

My Benchmark

Kaggle task: wrong-screen-answers
Dataset: wrong-screen-answers-data (frozen sections, questions, labels, scorer and prompt builder; blog text © GoodBarber, reproduced with permission for evaluation)
Instruction variant: wrong-screen-answers-labels (same grid, one more sentence in the system prompt)

The task builds every prompt from the dataset, scorer included, so a new model gets the exact bytes these did.

What it doesn't measure:

Run-to-run variation. A second run moved Gemini 3.7 Flash by 0.004 and GPT-5.4 nano by 0.010, so gaps of a few hundredths are ties.
The instruction on bigger models. It ran on three small ones; both of Qwen3 235B's runs lost too many cells to rate limits.
The scorer's blind spots. A glossed « encore 12 min » / « nog 12 min » counts as a leak; an invented hybrid, "Com KI criar", as an own translation.
The retrieval. The sections are chosen in advance, the answer comes first, with neutral ids. This measures what a model does with the context it's given, not a retrieval stack.

What I take from it. The label in the answer comes from the page, not from the reader, and the best model still gives only the page's label in a quarter of its answers. So in a retrieve-then-answer loop over a multilingual corpus, the first thing I'd check is which edition comes back, before choosing which model reads it.

How this was made

I built this with Claude Code over two days: harness, scorer, tests, Kaggle task and analysis. I chose the angle and reviewed the questions. The questions were drafted with Claude and back-translated by GPT-6 Astra, from another family, since Claude models are in the lineup.

I built my friend a control tower that talks back

Dominique Siacci — Sat, 03 Oct 2026 16:00:54 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

The tower says: "Foxtrot Alpha Bravo Charlie Delta, Isola Ground, runway two seven, QNH one zero one three, taxi holding point Alpha One via Bravo." You hold the space bar and read it back, and you say one zero zero three. The tower comes straight back: "Foxtrot Alpha Bravo Charlie Delta, negative, QNH one zero one three."

That tower is a synthetic voice running on a laptop, the aerodrome doesn't exist, and the whole thing is a small game I built this weekend for a friend.

What I Built

My friend just got his private pilot's licence. To fly abroad he needs an English language proficiency endorsement on it, and he's wondering whether to go for it, but he's shy about speaking English on the radio. I passed mine two years ago, which doesn't make my English good enough to coach anyone: I am just a private pilot myself. The plan is simple: when he has it, we fly abroad together.

He knows the procedures. What he hasn't done much is say them in English, out loud, in the right order, with the numbers right, while someone waits for the answer. You can read the phraseology manual on the sofa. You can't rehearse the radio alone, because nobody talks back. This won't prepare him for the proficiency check; it gives him somewhere to say the words, alone, as many times as he wants, before he says them to someone.

So I built him something that talks back: phraseology, a practice game for one traffic circuit at Isola, a made-up controlled aerodrome. You're F-ABCD at the flying club. You call Ground, taxi to the holding point, change to Tower, line up, take off, fly a touch and go, then a second circuit to land, vacate and taxi back. Runway, wind, QNH, squawk and the other traffic change every game, and up to three surprises can change the plan: "number 2, follow the Cherokee on base", "extend downwind", etc. A second aircraft shares the frequency, with its own voice.

You answer with the space bar as your press-to-talk. After each transmission the page tells you what was missing or wrong, in plain words:

"QNH: you said QNH 1003, the tower said QNH 1013."
"Missing: runway 27 (the runway in use is always read back)."
"Put your callsign at the end of a readback."

There is no score and no level. If the readback is wrong, the tower says "negative" and gives the correct part, and you read it back again.

It's a practice game, not training. It isn't an approved training device or an assessment, and it doesn't prepare anyone for a language proficiency check. The aerodrome, frequencies and traffic are invented, and no real ATC recording is used. Phraseology also differs from one country to another. Your instructor and your country's official manuals are the reference.

Demo

The video is one real game, played on a small demo server and recorded in a headless browser: speech recognition, the checker and the tower are the real ones. The pilot is synthetic too, Kokoro's French voice reading English, so you hear the whole loop with an accent (I have to say I am too shy myself to publish a video with my own voice in this context!). The waits while the server transcribes are cut.

Code

dsiacci / phraseology

phraseology

Practice the radio calls of a traffic circuit in English, out loud, on your own laptop.

The tower talks to you with a synthetic voice. You hold the space bar like a press-to-talk switch and answer. An open-weight speech model transcribes you on your machine, a set of plain rules checks your readback, and the circuit moves on. If the readback is wrong, the tower answers "negative" and gives the correct version, and you read it back again.

A practice game, not training. This is not an approved training device, not an assessment, and not a preparation for any language proficiency check. The aerodrome, its frequencies and the traffic are made up. Phraseology differs from one country to the next: your flight instructor and your national authority's manual are the reference.

What a game looks like

You are F-ABCD, a light aircraft parked at the flying club of Isola…

View on GitHub

Python with no web framework, GPL-3.0-or-later. pip install ., python -m phraseology download once, then python -m phraseology serve --open. Any recent laptop works, no GPU. Add ?seed=71 to the page's address to play the game from the video.

How I Built It

The loop is short:

microphone → browser (16 kHz WAV) → faster-whisper, on the CPU
          → normalizer → readback rules ← scenario (state machine)
          → next tower message → Piper → speaker

No language model decides anything. Phraseology is meant to be said word for word, so the moment the tower speaks, the game knows exactly what a correct readback contains: the runway, the QNH, the holding point, the callsign at the end. Checking it is a matter of rules. The open models hear and speak; the rules decide. That also means every message the tower says comes from templates I wrote with made-up values, never from generated text.

The rules are the ones a pilot learns, in my own words. The runway in use, clearances on a runway, altimeter settings, transponder codes and new frequencies are always read back. Other instructions can be acknowledged ("wilco"), though the game asks for taxi instructions in full, as the UK manual does. A readback ends with your callsign, a new call starts with it, and you only shorten F-ABCD to F-CD after the station has.

Most of the work went into the space between what you say and what Whisper writes:

The prompt must not contain the values. faster-whisper takes an initial prompt, and giving it the station names, the callsign and the radio words helps a lot. Giving it the QNH would be a mistake: the transcript leans towards what's in the prompt, so a wrong readback would come out right. The prompt only holds what never changes during a game.
Order matters in that prompt. When it ended with "say again", Whisper dropped a "say again" spoken at the start of the clip, as if it had already written it. The callsign now goes last.
Whisper has opinions about "niner". The base.en model wrote "niner niner eight" as "9 and 9 are 8", or "9R9R8", or turned "zero niner" into "Zero Minor". The bigger small.en heard "nine-oh" and wrote 9098. "Four" came out as "for", "via" as "wire". A normalizer undoes these slips, but only where a slip can't change a value: "for" becomes 4 between digits or after "squawk", and stays "for" in "cleared for take-off"; a zero after a nine is dropped only when the QNH would otherwise read above 1100 hPa, which no altimeter setting does. Then values are compared exactly.
Mishearings have to fail safe. The tests close the loop with the real models: Piper speaks correct and wrong readbacks (wrong QNH, wrong squawk, wrong frequency, "cleared to land" instead of "cleared for take-off", wrong callsign, "roger" instead of a readback), Whisper transcribes them, and the checker has to get the verdict right. With small.en, the model the demo server ran, and two synthetic voices, 14 of 14 wrong readbacks were rejected and 14 of 16 correct ones were accepted (15 of 16 with the smaller base.en). The misses were correct readbacks misheard ("A.K. Wright" for "vacate right"), so the tower asked again. Across every run, a wrong readback was never accepted.

The demo server started on base.en, the smaller model, because it answered faster. The first real test, with a French accent, broke it: "at the flying club, request taxi" came out as "de France, clamps, stochasticity". On readbacks spoken with a strong French accent, small.en got 5 of 6 right and base.en 3 of 6, at 4.2 seconds per readback on two vCPUs instead of 1.4. I took the slower one: on a practice tool, a readback that is right but rejected is worse than a wait.

I built it over the weekend with an AI coding agent.

Credits, because none of this works without them: Whisper by OpenAI (MIT, open weights), faster-whisper and CTranslate2 (MIT), the Silero VAD model (MIT), Piper with espeak-ng (GPL-3.0-or-later) on ONNX Runtime (MIT), and two voices from rhasspy/piper-voices, en_US-norman for the tower and en_GB-cori for the other aircraft, both trained from scratch on public-domain LibriVox recordings; the video's pilot is Kokoro-82M (Apache 2.0), French voice ff_siwis (SIWIS database, CC BY 4.0).

Why Does Open Innovation Matter?

Practicing a language means sounding unsure, repeating yourself, getting numbers wrong. That recording belongs on my friend's laptop. With open weights, Whisper runs there, on the CPU, and the voice never leaves the machine. After the first download it works with no internet, so it works at the airfield, and it costs nothing to run.

Having the model on my side of the wire also let me do the work above. I could hand Whisper a prompt, measure what that prompt did to a wrong readback, and decide what goes in it. I could pin the version, decode in int8 on a CPU, set the beam size and the voice-activity filter, and run a test suite against the real model as often as I liked.

The voices too: Piper's voices come with model cards, and I could read where each one was trained. Two that I first planned to use turned out to be fine-tuned from a voice whose dataset license doesn't fit this project, so I picked ones trained from scratch on public-domain recordings. And because Piper is GPL, so is this game.

Prize Categories

Best Use of DigitalOcean. The game in the video was served by a DigitalOcean Droplet with 2 vCPUs, 4 GB of memory and no GPU. Whisper small.en ran on the Droplet's CPU and Piper spoke the tower, as a systemd service behind Caddy, which got the HTTPS certificate on its own (the browser only opens the microphone over HTTPS). The Droplet only let in SSH, HTTP and HTTPS, transcribed one recording at a time with a short queue, capped recordings at 10 seconds, rate-limited each address, and never wrote audio to disk. The setup is two scripts in the repository, so anyone can run their own.

"Push, PR, merge." Two minutes later: "Stop. I'm not opening a PR to myself."

Dominique Siacci — Thu, 01 Oct 2026 07:50:18 +0000

On September 9 at 07:21 I typed four words to an agent: push, PR, merge. At 07:23 I interrupted it: stop, just push to master, I'm not going to open a pull request to myself. The agent pushed three commits, fast-forward, and wrote the rule down so the next session would know it. In between, it had done exactly what the tooling around it is built to do, and that's the part worth looking at.

The change

The work was small. An internal tool keeps a catalog of entries that we generate once, review, and sometimes touch up by hand in production. The agent had generated a new batch for the entries that had none, and I wanted it in production.

At 07:08 I asked the question that mattered: the import updates existing entries in place, some of them were edited by hand in production, so what exactly will this overwrite? The agent went and checked. Those edits are saved in place, under the same identifier, so a full import would have replayed our local versions over the production ones and reset their review status. Silently.

The next ten minutes went like this. An import option that only creates entries that don't exist yet. Sixteen tests, two of them pinning that exact danger. A run against a copy of production, rolled back. A second door found on the way: the import button in the review screen called the same function without the guard, so it got the same fix. And a read-only preflight script that tells you, against production, what the import would do before it does anything.

That was the review. It happened in the session, before the commit, and it started with one question from the person who knew the hand edits existed.

Then the tooling took over

At 07:15 I said I'd pull master on the server and run the two commands. The agent pointed out that the branch had never been pushed. At 07:21 I wrote "push + PR + merge into master", because that's the sentence twenty years of workflow put in your fingers.

The agent took the words literally, and the instruction file of that repository routes "push" and "create a PR" to a shipping skill. So it started a release: health checks, the test suite, the linter, a look for a version file to bump and a changelog to update, and the discovery that it could push to our Git host but had no credentials to open the pull request there. Two and a half minutes of ceremony for three commits I had already reviewed with the agent, line by line, ten minutes earlier.

That's when I stopped it.

What a pull request is for

A pull request is a conversation between two people about a change one of them made. The author explains, the reviewer asks, and the merge records that someone other than the author looked. It's also a lock: nothing lands while the conversation is open.

With one human and several agents, none of that holds. The author is an agent, the reviewer is me, and the review already happened, in the session, while the change was being made, because that's when I can ask "what will this overwrite?" and get an answer in minutes. A PR opened after that is me approving my own review. It adds a merge commit, a CI run, and a page nobody reads.

Mathieu Poli, who runs frontend engineering with us, wrote the team version of this: pull requests do work with agents, and that's exactly the trap. On a repository where I'm alone with my agents, the trap is just easier to see.

What replaced it

On that repository, the rule born that morning is short, and the agent wrote it down itself at 07:25: when I say push, it pushes to master, fast-forward, and nothing else. No pull request, no version bump, no changelog. I'm the only maintainer there, and by the time I say push, the review has already happened.

On the repository where most of my agents work every day, the same idea had already gone further. Nothing but main is ever pushed. Agents work in local worktrees and land their work on main under a shared lock, rebased or fast-forwarded, then push right away. Every session pulls before writing, because several of them run at the same time on different machines. Commits are small and frequent. A rejected push is normal: another session pushed first, you rebase and push again, never force.

There, the lock a PR gives you comes from pulling before writing and from the remote refusing anything that isn't a fast-forward. The review comes from the question asked in the session, before the commit, by the person who knows what's in production. The trace comes from commit messages that say why, not what.

What's still true

The reflex was mine. The agent did what I asked, in the words I used.
The stricter rule holds because a rejected push is cheap. On September 11, on the second repository, a session's push was rejected because another session had pushed while it was writing. Rebase, push, done. Without that, "everything on main" would be a race.
Sixteen old agent branches are still sitting on the first repository's remote. The second one has none left.

If you run several agents on one repository: where does review happen for you, in the pull request or before the commit? I'll answer in the comments with how our sessions are set up.