374 tests went in, 352 came out, and the build was green

The last run had 374 tests. This one had 352. No failures, so the build was green.

Twenty-two checks had disappeared. The test host died partway through, the runner counted what it managed to finish, and the only evidence was a total nobody had a reason to compare against yesterday's. Every tool in the chain did what it was designed to do. The result was still false.

winwright is a UI testing engine for Windows desktop apps (WPF, WinForms, tray icons, menus, dialogs), and that missing 22 is the reason it exists. Most of its design follows from refusing to let that happen again.

A test has three possible answers

Most test runners give you two outcomes: it passed or it failed. Desktop UI testing has a third one, and it shows up all the time.

Say a test needs to type into a text box. To send keys, Windows has to give your app the foreground. Sometimes it won't: you clicked into Slack, a notification took focus, the CI agent has no interactive session at all. The keystrokes go somewhere else, the read-back doesn't change, and the test goes red.

The red is wrong. Nothing in your application broke. You'll spend twenty minutes reading code that is fine, because the report said your app failed when what happened was the machine never let me try.

winwright calls that third outcome a hole: an assertion that never ran because something it needed was missing. A hole isn't a pass and it isn't a failure, and the exit code keeps them apart:

Exit code	Verdict	What it means
0	Passed	Everything that was asked was checked, and held
1	Failed	Your application did something wrong
2	Degraded	Some checks could not run, and the report names why
3	Broken	The run itself threw, so nothing after that point was observed

The summary line puts what didn't happen first:

Passed: 1 of 9 cases, 8 not run, 3 assertions over case 'renaming a profile writes it back'.

A pass over one case out of nine is a weaker claim than a pass, and a report that buries "8 not run" at the end of the line is one people will skim past. If the 352 run had produced that sentence, someone would have noticed.

Even a run that selects nothing is caught. A filter that matches zero cases is refused, because zero cases contain zero failures and would otherwise come back green.

The desk is not your application

Most of the holes you'll meet belong to the machine, not to your code, and winwright says which side each one is on. "No foreground", "no interactive session", "a flyout the shell wouldn't open" and "somebody else touched the mouse during the case" are reported as conditions of the desk. None of them can turn a run red.

One detail I like: a freshly launched app is often still coming forward when the first step runs. Measured against one adopter, that took 1,453 ms. A tool that checks the foreground once at that moment concludes another window owns the desk, which is true for that instant and false for the run. So winwright waits for the foreground up to the project's resolve timeout. Only after that does it report a hole, and the hole says how long it waited.

There's also an easier way around most of it. The default act doesn't need the foreground at all.

Ask the control, don't move the mouse

Every accessible Windows control has a UI Automation peer: an API that lets a screen reader ask "what's your value?" or tell it "invoke yourself". winwright's default verbs (invoke, toggle, set value, select, expand) go through those patterns. No synthetic mouse, no keyboard, no need to be in front.

The verbs that do put real input on the desk (type, click, press, drag, nudge) are marked that way in the catalogue, and they prove something different from their pattern twins. set value writes through the Value pattern. type presses keys.

That difference caught a real bug. A WPF window hosted under a WinForms message pump accepted no keyboard input at all, and the set value check passed anyway, because it never used the keyboard. The type check failed. Neither one was flaky. They answered two separate questions, and only one of those questions was what a user would actually do.

That's also why click won't run until you give it a reason: NoAutomationPeer, NotificationArea, CustomTemplate, or PointerIsTheAct. If the reason defaults, every act quietly turns into a mouse click and your suite ends up driving the desktop instead of asking controls.

A test is a data file

A winwright case isn't C#. It's JSON:

{
  "cases": [
    {
      "name": "renaming a profile writes it back",
      "catches": "a rename that updates the list and never the file",
      "tags": ["smoke", "profiles"],
      "steps": [
        { "locator": "TabItem[name=\"Profiles\"]", "act": "select" },
        { "locator": "Edit#profileName", "act": "set value", "with": "Beta", "expect": "Beta", "reads": "value" },
        { "locator": "CheckBox#autosave", "act": "toggle", "expect": "On", "reads": "toggle" },
        { "locator": "Button#save", "act": "invoke", "named": "save the profile" },
        { "locator": "Text#status", "act": "read", "expect": "Saved", "reads": "text" }
      ]
    }
  ]
}

Notice what's missing: no Thread.Sleep, no retry loop, no WaitUntil. The engine owns the waits, the attempts and the verdict. The case only says what to do and what should be true afterwards.

It even knows which acts are safe to retry. set value can be repeated if the read-back is late. toggle and invoke get exactly one try, because a retried toggle leaves the checkbox in the opposite state and then fails about that.

Locators use one grammar, read the same way by every verb:

Button#save                          automation id + control type
Button[name="Save as..."]            by name
MenuItem[nameStarts="Personal "]     the front of a name that carries live state after it
ComboBox|Slider|Edit                 any one of several types
Window#main > Pane > Button#save     a descendant, at any depth

The > means descendant at any depth, not direct child. That's deliberate: UI Automation wraps controls in panes that differ between frameworks, between framework versions, and between a maximised window and a restored one. A direct-child locator is the kind that passes on your machine and breaks on a colleague's.

Typed expectations go stale

This is the part that changed how I write UI tests.

One adopter's hand-written harness checked a window's tabs. It listed the three tab labels by hand and reported "all three tab headers read". Then the window gained a fourth tab. The harness kept reporting all three tab headers read, against a four-tab window, and nothing went red. It was still accurate about the three tabs it knew. It had just stopped checking the window.

A string you type into a test is a second copy of the truth, and that second copy is the one that goes stale. So winwright prefers expectations that are derived:

{ "locator": "TabItem", "act": "read", "covers": "stats.tab" }

stats.tab is a key in your application's own strings file. The claim goes both ways: every string declared under that key appears in the window, and nothing else does. Add a fifth tab to the strings file and this step fails until the window shows it. Add a fifth tab to the window without declaring it and the step fails too. The test file doesn't change either way.

The same idea shows up throughout the format:

• label compares a reading against a string from your language files, in whatever language the fixture launched the window in. Your pt-BR run checks Portuguese without a separate test.
• forEach runs a case once per string under a key, so a settings page with six panels gets six runs, and a seventh panel added next quarter gets a seventh with no edit.
• reportedSets asks the application itself for sets that live in the machine's data rather than in strings (profiles, accounts, devices). Your test then holds on a machine with one profile and on a machine with five.
• moves, answers, matches, sameAs and unlike cover values a case can't know in advance: the reading changed, it said something, it has the right shape, it came back to what an earlier step read.

Even negative claims are honest about how hard they looked. never asserts that a string (say, "no readings yet") never appeared while a step was waiting. One adopter's report comes back from cache in 12 ms or gets rebuilt in 961 ms with that placeholder flashing on screen, and once the waiting is over the two windows look identical. So the result reports how many times it checked. For a claim that something never appeared, the number of looks is the evidence.

Screenshots that refuse to lie

A screenshot proves layout and nothing else. It says nothing about whether a button works, and winwright keeps those questions in separate loops: pictures and geometry for layout, act-then-read for input, render-twice-and-compare-bytes for determinism.

Within its job, a capture is held to the same rule. A capture is refused if a window covered it, if the backdrop showed through, if it came back as a single flat colour, or if the page in it was still showing its loading text. The refusal names the window that got in the way, its process and the rectangle. An optional in-app package (Winwright.InApp) lets the application render its own visual tree to a PNG. That includes a popup nobody has clicked open yet, which no screen copy can capture.

Refusals are instructions

winwright says "no" a lot, and it tries to make every "no" tell you what to do next:

• A locator that doesn't parse names the character position and the reason.
• A misspelled control type gets the nearest name UI Automation actually has.
• An unknown key in a case file is refused, never ignored. "expects" next to "expect" would otherwise load, run, check nothing, and pass.
• A case whose steps expect nothing is refused: it acts, never looks, and passes on a build that still has the bug.
• Every refusal is addressed into the file, as in cases[2].steps[1].act, so you fix a field and not a file.

The part built for AI agents

This is where it gets a bit unusual.

If you ask a coding agent to "test this window", it will write a harness script. The script is always available and always faster in the moment, and that's how one adopter ended up with a 2,732-line one.

winwright ships as a Claude Code plugin, and the plugin includes a PreToolUse hook. Any write whose content references Winwright.Acting, Winwright.Locating or Winwright.Asserting is denied, and the denial points the agent to the case format and the tool that validates it. The agent gets the refusal before it writes the script, not after you've reviewed it.

The MCP server then gives the agent four tools:

• winwright_format: every field a case may carry, and the closed list of what each accepts.
• winwright_vocabulary: every act, what it needs beside it, and whether the engine may retry it.
• winwright_check: validates a case before the file exists. Its input schema is the loader's schema, so a misspelled key can't even be sent.
• winwright_run: runs it and returns the verdict. A machine that can't observe returns a hole naming which condition is missing, not a red and not a green.

Two commands wire it into a repository, and committing .claude/settings.json wires it for everyone who clones it:

claude plugin marketplace add alegauss/winwright --scope project
claude plugin install winwright@alegauss --scope project

One deliberate rule: the tool never writes the test. There's no recorder that turns clicks into a scenario and no assertion invented on your behalf. The engine owns the loop and the waits, and deciding what's worth asserting stays with you.

When you should not use it

The project site has an honest comparison where every alternative keeps the column it wins:

• Playwright for anything in a browser. The name is a nod, not a rivalry.
• Appium when one suite has to drive more than Windows.
• WinAppDriver when WebDriver compatibility is the requirement.
• FlaUI when you want a mature UI Automation library without opinions attached.
• Your existing script, right up until the run where the total changes and the colour doesn't.

winwright is Windows-only, .NET 10 only, and will stay that way: the whole engine is UI Automation and Win32. What it offers is narrower: a report that tells a failed check from a check that never ran, one locator grammar, expectations derived from the app instead of typed into the test, and screenshots that refuse instead of lying.

Trying it

<!-- the test project that drives your app -->
<PackageReference Include="Winwright" Version="1.0.2" />

<!-- optional, inside the app, for the readings only it can take -->
<PackageReference Include="Winwright.InApp" Version="1.0.2" />

Add a winwright.json with your executable's path, write a .cases.json, and run it with four calls:

var project  = ProjectDeclaration.Find(repository);
var declared = ScenarioFile.Across(ScenarioFile.LoadAll(Path.Combine(repository, "cases")));

using var register = ProcessRegister.For(project);   // anything the run started is stopped, and named if it won't stop
var verdict = Suite.Launch(declared, Selection.All, register, project);

Console.WriteLine(verdict.Sentence());
return verdict.ExitCode;

One piece of advice from the docs that I'd underline: give the run a desk of its own. Cases take the foreground and send real input. If you run them on the machine you're working at, your typing lands in the app under test. That isn't a flaky suite, it's two people sharing one keyboard. Use a VM with an interactive session, a dedicated runner, or any machine nobody is sitting at.

• 📦 NuGet: Winwright · Winwright.InApp
• 🌎 Site: alegauss.github.io/winwright/
• 📖 Docs: alegauss.github.io/winwright/docs/
• 💻 Source (Apache-2.0): github.com/alegauss/winwright

---

Has your suite ever passed with fewer tests than it had the day before? I'd like to hear how you noticed, or how long it took.

Comments

[pm25coder]

Your closing question is one I can answer with a number.

The "fewer tests" flavour that has bitten me is subtraction the summary line never mentions: a subset of tests that stop running because a fixture cannot set itself up on the CI host. I re-ran that shape just now rather than trusting the memory. Four tests asserting real things, pytest -q gives 4 passed, exit 0. Then one gets a skip in its place and one is renamed. The run still exits 0, and the line people actually read still says 4 passed. Nothing goes red. The only difference is a name that is no longer in the run.

That suggests pushing one step past the count. 374 becoming 352 is caught by comparing totals, but a suite can keep its total and change its contents — old checks retired, new ones added, same number, and nothing goes red, because the thing it stopped asserting now has no test left to fail. The total is a weak reading. What the runner should diff is the set of node ids that actually executed, before against after, and fail on a difference in either direction. A count notices a subtraction; only the set notices a substitution.

It also falls out of your never design for free: "how many times did you look" and "which names ran" are the same kind of answer — both are the evidence behind a negative claim, and both are invisible in a pass/fail.

[Alexandre Oliveira]

You're right, and the substitution case is the sharper one. A count can catch a subtraction, but it can't see a swap.

Here's where winwright stands on each half today.

The skip half is covered within a run. winwright has no silent skip. A case can stay out of a run for one of two reasons. If a selection left it out, the summary sentence names it with the reason, and the line opens with what didn't run ("Passed: 1 of 9 cases, 8 not run…"). If it declared a needs that this machine doesn't have, its checks come back unchecked and the run exits 2 (degraded), not 0. So your pytest example of "one test becomes a skip, exit 0, still reads as passing" can't happen there.

The substitution half isn't covered. Every run reads only itself. Nothing compares the set that ran against the previous run, so a renamed or replaced case with the same total would pass without anyone noticing, exactly as you describe.

One thought on what the diff should key on. Node ids treat a rename as a change, and most renames are harmless, so a strict id diff gets noisy and people learn to wave it through. The thing you actually lose in a substitution is a claim. winwright cases can carry a catches field ("a rename that updates the list and never the file"), and diffing that set across runs might be closer to what matters. A test renamed without changing its claim stays quiet, while a claim that disappeared goes red. That only works if cases actually fill the field, which the run currently counts but doesn't require.

And I agree with the connection to never. "How many times did I look" and "which names ran" are both evidence for a negative claim, and a pass/fail result shows neither.

Thanks for re-running it instead of trusting memory. That's the whole spirit of the thing.

[pm25coder]

That is a sharper version of the field question than the one I had, so I ran it to see whether the quietness survives.

Four synthetic runs over the same three cases, each carrying a claim: baseline, then one renamed with its claim unchanged, one deleted, and one replaced — same id, different claim. Diffed against the baseline three ways: by node id, by the claim set as authored, and by the claim set where the field was filled from the case's own name.

change                        by id    by claim (authored)    by claim (derived)
renamed, same claim           RED 2    quiet                  RED 2
deleted                       RED 1    RED 1                  RED 1
replaced, same id, new claim  quiet    RED 2                  quiet

The middle column is exactly what you described — the rename stays quiet, a lost claim goes red. The fourth row is the part that surprised me in the claim diff's favour: a substitution that keeps its id is invisible to the id diff, and the claim diff catches it.

The third column is the caveat. Once the field is populated from the identifier rather than written independently, the claim diff is the id diff, line for line — the same two entries for the rename, the same silence for the replacement. So "the field is required" is not the property that buys the quietness; "the field is independent of the name" is. A run cannot tell an authored claim from a pasted title just by looking, but it can reject the cheap version: a claim equal to, or containing, its own id is one comparison, and it closes the way the requirement gets satisfied without being met. That seems worth doing in the same place as the counted-but-not-required fix, because a field that is filled but derived leaves you with the id diff you were trying to get away from.

Your skip half sounds like the stronger half for exactly the reason you gave: the failure is named at the point where it happens, and exit 2 keeps "did not run" from reading as "passed". Mine only covered the case where a skip is silent.

[Alexandre Oliveira]

Your table matches what shipped. Both halves went out in winwright 1.1.0: a project that declares lastRun gets a run that remembers which cases ran over each selection and what each one says it catches. The next run names the cases that appeared and vanished, and separately any claim no case makes any more. A rename that kept its claim stays quiet. A case that keeps its name but stops claiming what it used to is reported, which is your replacement row. A case that vanished with no catches at all is named as one the run can't judge, instead of being read as harmless.

It's reported as a finding, not a red. It's neither the application failing nor the desk refusing, and the record is rewritten every run, so a deliberate change is named once and becomes the baseline.

Your derived-claim column pointed at the gap that was still open: nothing refused a catches that is just the case name again, and as you showed, that turns the claim diff straight back into the id diff. That one is in now. In 1.1.1, a catches that only repeats its own case name, compared ignoring case, spacing and punctuation, is refused where it is written. I went with equality rather than "contains its own name", because a short name can appear inside a real claim, and a check that turns away an honest sentence teaches people to leave the field out. If a pasted title gets past equality, that's the evidence for widening it.

Thanks for running it rather than arguing it. The table was more convincing than either of my comments.

[pm25coder]

Read it where it shipped rather than taking the summary: src/Winwright/Scenarios/CaseDeclaration.cs, Justifying (:346-368), guard WW537, with Folded at :374 keeping char.IsLetterOrDigit and lowercasing. The comment above the comparison (:352-356) gives the equality-not-contains reason in the same words you used, so the rule and the description agree.

You left a falsifier — "if a pasted title gets past equality, that's the evidence for widening it" — so I ran it. No .NET on this machine, so Folded is my reimplementation of that one expression, not your binary; the corpus is ASCII, where IsLetterOrDigit and isalnum agree on every input.

Eight claims derived mechanically from one case name (LoginSucceeds, folds to 13 characters), and five that say something new:

derived from the name                     R0        R1        R2        R3
verbatim and folded variants (4 shapes)   REFUSED   REFUSED   REFUSED   REFUSED
WW1042 LoginSucceeds   (id prefixed)      passes    REFUSED   REFUSED   REFUSED
LoginSucceeds test     (a word appended)  passes    REFUSED   REFUSED   REFUSED
Login succeeds (regression)               passes    REFUSED   REFUSED   REFUSED
Tests.LoginSucceeds    (qualified name)   passes    REFUSED   REFUSED   REFUSED

says something new                        R0        R1        R2        R3
"the session token is cached and a stale one is accepted"    passes  passes   passes   passes
"a valid password with a trailing space is rejected"         passes  passes   passes   passes
short name, name inside the claim                            passes  REFUSED  passes   passes
short name, claim opens with the name                        passes  REFUSED  REFUSED  passes
long name, claim opens with it then adds a defect            passes  REFUSED  REFUSED  REFUSED

R0 shipped equality · R1 contains · R2 edge, prefix or suffix · R3 edge with folded name >= 12

R0 misses 4 of 8 and turns away none of the 5. R1 closes all 8 and turns away 3 — your short-name objection, reproduced. R2 narrows it to 2: a claim that opens with Login and then describes the defect. R3 gets down to 1, and the one left is a claim that opens with the whole 32-character name and then adds the real defect — a shape an honest author will write.

So the evidence you asked for does exist, and it is broader than one shape: a paste survives equality whenever it carries any addition — decoration on either side, or the class prefix. But every widening I could construct pays for it, and the most cautious one still turns away an honest sentence. The reason is not that equality is safe. It is that "the name plus decoration" and "the name plus a real clause" are the same string. Nothing folded can see whether the extra words name a defect, and that is the entire difference.

Which makes the bar you set the right one. What escapes today is a paste that adds characters: small, and invisible to a comparison. The evidence that would justify widening is a real one of those in the wild, not a construction — and what I have is a construction.

[Alexandre Oliveira]

Your table settled it, and it changed what shipped, just not the refusal. Equality stays the only thing that refuses. You showed why: "the name plus decoration" and "the name plus a real clause" are the same string, and nothing folded can tell them apart. So 1.1.2 added a report instead. It's your R2, prefix or suffix, with a floor of 8 folded characters. Next to the cases that say nothing about what they catch, a run now lists the cases whose catches begins or ends with their own name, and quotes each claim. It never refuses a case and never moves the exit code.

The floor sits between your R2 and R3. Below 8 characters, your short-name rows would be flagged on every run, and a warning that always fires is one people learn to skip. Your last row, a long name followed by the real defect, does get listed. That's why it's a report: a person can read that line and see the defect clause, and a comparison can't. Your derived shapes (id prefix, appended word, regression phrasing, qualified name) are test data in the suite now, next to the honest sentences that have to stay quiet.

Answering you also turned up a bug: with exactly one case, that line read "1 case that ran say what they catch". 1.1.3 fixes the agreement in both lines.

The bar you described still holds for the refusal. If a real paste turns up in the wild, the report is where it will show first.

[challan116-ux]

A green build that ran fewer tests than yesterday is a false negative at the level of the whole pipeline — every individual tool told the truth about what it saw, and the system still lied. Comparing the executed count against the expected count is such a cheap check, and it is missing from most runners because 'no failures' feels like a complete answer. Your third outcome idea matters here too: a test that never ran is not a pass, and a runner that cannot say so will eventually hide exactly the tests you needed most.

[Alexandre Oliveira]

The comparison really is cheap. What took me longer was deciding what to compare against. "Yesterday's count" looks like a baseline, but it also moves for honest reasons: someone deletes an obsolete case, or a filter narrows the run. A check that only says "fewer than yesterday" fires on every cleanup too, and people stop reading it after the first week.

So winwright splits it in two. Within a run, the engine knows which cases it loaded, so anything that didn't finish counts as not run, leads the summary line, and the exit code is 2 instead of 0. Across runs, it keeps a record, so a case that vanishes between runs is reported by name instead of disappearing into a smaller total. A deliberate removal still shows up once. I think that's a fair price.

Your last line is the one I'd underline. The tests most likely to quietly stop running are the slow ones that need the most environment, and those are usually the ones guarding something real.