374 tests went in, 352 came out, and the build was green

The last run had 374 tests. This one had 352. No failures, so the build was green.

Twenty-two checks had disappeared. The test host died partway through, the runner counted what it managed to finish, and the only evidence was a total nobody had a reason to compare against yesterday's. Every tool in the chain did what it was designed to do. The result was still false.

winwright is a UI testing engine for Windows desktop apps (WPF, WinForms, tray icons, menus, dialogs), and that missing 22 is the reason it exists. Most of its design follows from refusing to let that happen again.

A test has three possible answers

Most test runners give you two outcomes: it passed or it failed. Desktop UI testing has a third one, and it shows up all the time.

Say a test needs to type into a text box. To send keys, Windows has to give your app the foreground. Sometimes it won't: you clicked into Slack, a notification took focus, the CI agent has no interactive session at all. The keystrokes go somewhere else, the read-back doesn't change, and the test goes red.

The red is wrong. Nothing in your application broke. You'll spend twenty minutes reading code that is fine, because the report said your app failed when what happened was the machine never let me try.

winwright calls that third outcome a hole: an assertion that never ran because something it needed was missing. A hole isn't a pass and it isn't a failure, and the exit code keeps them apart:

Exit code	Verdict	What it means
0	Passed	Everything that was asked was checked, and held
1	Failed	Your application did something wrong
2	Degraded	Some checks could not run, and the report names why
3	Broken	The run itself threw, so nothing after that point was observed

The summary line puts what didn't happen first:

Passed: 1 of 9 cases, 8 not run, 3 assertions over case 'renaming a profile writes it back'.

A pass over one case out of nine is a weaker claim than a pass, and a report that buries "8 not run" at the end of the line is one people will skim past. If the 352 run had produced that sentence, someone would have noticed.

Even a run that selects nothing is caught. A filter that matches zero cases is refused, because zero cases contain zero failures and would otherwise come back green.

The desk is not your application

Most of the holes you'll meet belong to the machine, not to your code, and winwright says which side each one is on. "No foreground", "no interactive session", "a flyout the shell wouldn't open" and "somebody else touched the mouse during the case" are reported as conditions of the desk. None of them can turn a run red.

One detail I like: a freshly launched app is often still coming forward when the first step runs. Measured against one adopter, that took 1,453 ms. A tool that checks the foreground once at that moment concludes another window owns the desk, which is true for that instant and false for the run. So winwright waits for the foreground up to the project's resolve timeout. Only after that does it report a hole, and the hole says how long it waited.

There's also an easier way around most of it. The default act doesn't need the foreground at all.

Ask the control, don't move the mouse

Every accessible Windows control has a UI Automation peer: an API that lets a screen reader ask "what's your value?" or tell it "invoke yourself". winwright's default verbs (invoke, toggle, set value, select, expand) go through those patterns. No synthetic mouse, no keyboard, no need to be in front.

The verbs that do put real input on the desk (type, click, press, drag, nudge) are marked that way in the catalogue, and they prove something different from their pattern twins. set value writes through the Value pattern. type presses keys.

That difference caught a real bug. A WPF window hosted under a WinForms message pump accepted no keyboard input at all, and the set value check passed anyway, because it never used the keyboard. The type check failed. Neither one was flaky. They answered two separate questions, and only one of those questions was what a user would actually do.

That's also why click won't run until you give it a reason: NoAutomationPeer, NotificationArea, CustomTemplate, or PointerIsTheAct. If the reason defaults, every act quietly turns into a mouse click and your suite ends up driving the desktop instead of asking controls.

A test is a data file

A winwright case isn't C#. It's JSON:

{
  "cases": [
    {
      "name": "renaming a profile writes it back",
      "catches": "a rename that updates the list and never the file",
      "tags": ["smoke", "profiles"],
      "steps": [
        { "locator": "TabItem[name=\"Profiles\"]", "act": "select" },
        { "locator": "Edit#profileName", "act": "set value", "with": "Beta", "expect": "Beta", "reads": "value" },
        { "locator": "CheckBox#autosave", "act": "toggle", "expect": "On", "reads": "toggle" },
        { "locator": "Button#save", "act": "invoke", "named": "save the profile" },
        { "locator": "Text#status", "act": "read", "expect": "Saved", "reads": "text" }
      ]
    }
  ]
}

Notice what's missing: no Thread.Sleep, no retry loop, no WaitUntil. The engine owns the waits, the attempts and the verdict. The case only says what to do and what should be true afterwards.

It even knows which acts are safe to retry. set value can be repeated if the read-back is late. toggle and invoke get exactly one try, because a retried toggle leaves the checkbox in the opposite state and then fails about that.

Locators use one grammar, read the same way by every verb:

Button#save                          automation id + control type
Button[name="Save as..."]            by name
MenuItem[nameStarts="Personal "]     the front of a name that carries live state after it
ComboBox|Slider|Edit                 any one of several types
Window#main > Pane > Button#save     a descendant, at any depth

The > means descendant at any depth, not direct child. That's deliberate: UI Automation wraps controls in panes that differ between frameworks, between framework versions, and between a maximised window and a restored one. A direct-child locator is the kind that passes on your machine and breaks on a colleague's.

Typed expectations go stale

This is the part that changed how I write UI tests.

One adopter's hand-written harness checked a window's tabs. It listed the three tab labels by hand and reported "all three tab headers read". Then the window gained a fourth tab. The harness kept reporting all three tab headers read, against a four-tab window, and nothing went red. It was still accurate about the three tabs it knew. It had just stopped checking the window.

A string you type into a test is a second copy of the truth, and that second copy is the one that goes stale. So winwright prefers expectations that are derived:

{ "locator": "TabItem", "act": "read", "covers": "stats.tab" }

stats.tab is a key in your application's own strings file. The claim goes both ways: every string declared under that key appears in the window, and nothing else does. Add a fifth tab to the strings file and this step fails until the window shows it. Add a fifth tab to the window without declaring it and the step fails too. The test file doesn't change either way.

The same idea shows up throughout the format:

• label compares a reading against a string from your language files, in whatever language the fixture launched the window in. Your pt-BR run checks Portuguese without a separate test.
• forEach runs a case once per string under a key, so a settings page with six panels gets six runs, and a seventh panel added next quarter gets a seventh with no edit.
• reportedSets asks the application itself for sets that live in the machine's data rather than in strings (profiles, accounts, devices). Your test then holds on a machine with one profile and on a machine with five.
• moves, answers, matches, sameAs and unlike cover values a case can't know in advance: the reading changed, it said something, it has the right shape, it came back to what an earlier step read.

Even negative claims are honest about how hard they looked. never asserts that a string (say, "no readings yet") never appeared while a step was waiting. One adopter's report comes back from cache in 12 ms or gets rebuilt in 961 ms with that placeholder flashing on screen, and once the waiting is over the two windows look identical. So the result reports how many times it checked. For a claim that something never appeared, the number of looks is the evidence.

Screenshots that refuse to lie

A screenshot proves layout and nothing else. It says nothing about whether a button works, and winwright keeps those questions in separate loops: pictures and geometry for layout, act-then-read for input, render-twice-and-compare-bytes for determinism.

Within its job, a capture is held to the same rule. A capture is refused if a window covered it, if the backdrop showed through, if it came back as a single flat colour, or if the page in it was still showing its loading text. The refusal names the window that got in the way, its process and the rectangle. An optional in-app package (Winwright.InApp) lets the application render its own visual tree to a PNG. That includes a popup nobody has clicked open yet, which no screen copy can capture.

Refusals are instructions

winwright says "no" a lot, and it tries to make every "no" tell you what to do next:

• A locator that doesn't parse names the character position and the reason.
• A misspelled control type gets the nearest name UI Automation actually has.
• An unknown key in a case file is refused, never ignored. "expects" next to "expect" would otherwise load, run, check nothing, and pass.
• A case whose steps expect nothing is refused: it acts, never looks, and passes on a build that still has the bug.
• Every refusal is addressed into the file, as in cases[2].steps[1].act, so you fix a field and not a file.

The part built for AI agents

This is where it gets a bit unusual.

If you ask a coding agent to "test this window", it will write a harness script. The script is always available and always faster in the moment, and that's how one adopter ended up with a 2,732-line one.

winwright ships as a Claude Code plugin, and the plugin includes a PreToolUse hook. Any write whose content references Winwright.Acting, Winwright.Locating or Winwright.Asserting is denied, and the denial points the agent to the case format and the tool that validates it. The agent gets the refusal before it writes the script, not after you've reviewed it.

The MCP server then gives the agent four tools:

• winwright_format: every field a case may carry, and the closed list of what each accepts.
• winwright_vocabulary: every act, what it needs beside it, and whether the engine may retry it.
• winwright_check: validates a case before the file exists. Its input schema is the loader's schema, so a misspelled key can't even be sent.
• winwright_run: runs it and returns the verdict. A machine that can't observe returns a hole naming which condition is missing, not a red and not a green.

Two commands wire it into a repository, and committing .claude/settings.json wires it for everyone who clones it:

claude plugin marketplace add alegauss/winwright --scope project
claude plugin install winwright@alegauss --scope project

One deliberate rule: the tool never writes the test. There's no recorder that turns clicks into a scenario and no assertion invented on your behalf. The engine owns the loop and the waits, and deciding what's worth asserting stays with you.

When you should not use it

The project site has an honest comparison where every alternative keeps the column it wins:

• Playwright for anything in a browser. The name is a nod, not a rivalry.
• Appium when one suite has to drive more than Windows.
• WinAppDriver when WebDriver compatibility is the requirement.
• FlaUI when you want a mature UI Automation library without opinions attached.
• Your existing script, right up until the run where the total changes and the colour doesn't.

winwright is Windows-only, .NET 10 only, and will stay that way: the whole engine is UI Automation and Win32. What it offers is narrower: a report that tells a failed check from a check that never ran, one locator grammar, expectations derived from the app instead of typed into the test, and screenshots that refuse instead of lying.

Trying it

<!-- the test project that drives your app -->
<PackageReference Include="Winwright" Version="1.0.2" />

<!-- optional, inside the app, for the readings only it can take -->
<PackageReference Include="Winwright.InApp" Version="1.0.2" />

Add a winwright.json with your executable's path, write a .cases.json, and run it with four calls:

var project  = ProjectDeclaration.Find(repository);
var declared = ScenarioFile.Across(ScenarioFile.LoadAll(Path.Combine(repository, "cases")));

using var register = ProcessRegister.For(project);   // anything the run started is stopped, and named if it won't stop
var verdict = Suite.Launch(declared, Selection.All, register, project);

Console.WriteLine(verdict.Sentence());
return verdict.ExitCode;

One piece of advice from the docs that I'd underline: give the run a desk of its own. Cases take the foreground and send real input. If you run them on the machine you're working at, your typing lands in the app under test. That isn't a flaky suite, it's two people sharing one keyboard. Use a VM with an interactive session, a dedicated runner, or any machine nobody is sitting at.

• 📦 NuGet: Winwright · Winwright.InApp
• 🌎 Site: alegauss.github.io/winwright/
• 📖 Docs: alegauss.github.io/winwright/docs/
• 💻 Source (Apache-2.0): github.com/alegauss/winwright

---

Has your suite ever passed with fewer tests than it had the day before? I'd like to hear how you noticed, or how long it took.

Comments

[pm25coder]

Your closing question is one I can answer with a number.

The "fewer tests" flavour that has bitten me is subtraction the summary line never mentions: a subset of tests that stop running because a fixture cannot set itself up on the CI host. I re-ran that shape just now rather than trusting the memory. Four tests asserting real things, pytest -q gives 4 passed, exit 0. Then one gets a skip in its place and one is renamed. The run still exits 0, and the line people actually read still says 4 passed. Nothing goes red. The only difference is a name that is no longer in the run.

That suggests pushing one step past the count. 374 becoming 352 is caught by comparing totals, but a suite can keep its total and change its contents — old checks retired, new ones added, same number, and nothing goes red, because the thing it stopped asserting now has no test left to fail. The total is a weak reading. What the runner should diff is the set of node ids that actually executed, before against after, and fail on a difference in either direction. A count notices a subtraction; only the set notices a substitution.

It also falls out of your never design for free: "how many times did you look" and "which names ran" are the same kind of answer — both are the evidence behind a negative claim, and both are invisible in a pass/fail.

[Alexandre Oliveira]

You're right, and the substitution case is the sharper one. A count can catch a subtraction, but it can't see a swap.

Here's where winwright stands on each half today.

The skip half is covered within a run. winwright has no silent skip. A case can stay out of a run for one of two reasons. If a selection left it out, the summary sentence names it with the reason, and the line opens with what didn't run ("Passed: 1 of 9 cases, 8 not run…"). If it declared a needs that this machine doesn't have, its checks come back unchecked and the run exits 2 (degraded), not 0. So your pytest example of "one test becomes a skip, exit 0, still reads as passing" can't happen there.

The substitution half isn't covered. Every run reads only itself. Nothing compares the set that ran against the previous run, so a renamed or replaced case with the same total would pass without anyone noticing, exactly as you describe.

One thought on what the diff should key on. Node ids treat a rename as a change, and most renames are harmless, so a strict id diff gets noisy and people learn to wave it through. The thing you actually lose in a substitution is a claim. winwright cases can carry a catches field ("a rename that updates the list and never the file"), and diffing that set across runs might be closer to what matters. A test renamed without changing its claim stays quiet, while a claim that disappeared goes red. That only works if cases actually fill the field, which the run currently counts but doesn't require.

And I agree with the connection to never. "How many times did I look" and "which names ran" are both evidence for a negative claim, and a pass/fail result shows neither.

Thanks for re-running it instead of trusting memory. That's the whole spirit of the thing.

[pm25coder]

That is a sharper version of the field question than the one I had, so I ran it to see whether the quietness survives.

Four synthetic runs over the same three cases, each carrying a claim: baseline, then one renamed with its claim unchanged, one deleted, and one replaced — same id, different claim. Diffed against the baseline three ways: by node id, by the claim set as authored, and by the claim set where the field was filled from the case's own name.

change                        by id    by claim (authored)    by claim (derived)
renamed, same claim           RED 2    quiet                  RED 2
deleted                       RED 1    RED 1                  RED 1
replaced, same id, new claim  quiet    RED 2                  quiet

The middle column is exactly what you described — the rename stays quiet, a lost claim goes red. The fourth row is the part that surprised me in the claim diff's favour: a substitution that keeps its id is invisible to the id diff, and the claim diff catches it.

The third column is the caveat. Once the field is populated from the identifier rather than written independently, the claim diff is the id diff, line for line — the same two entries for the rename, the same silence for the replacement. So "the field is required" is not the property that buys the quietness; "the field is independent of the name" is. A run cannot tell an authored claim from a pasted title just by looking, but it can reject the cheap version: a claim equal to, or containing, its own id is one comparison, and it closes the way the requirement gets satisfied without being met. That seems worth doing in the same place as the counted-but-not-required fix, because a field that is filled but derived leaves you with the id diff you were trying to get away from.

Your skip half sounds like the stronger half for exactly the reason you gave: the failure is named at the point where it happens, and exit 2 keeps "did not run" from reading as "passed". Mine only covered the case where a skip is silent.