A skill nobody reaches for is dead weight

Muhammad Usman — Fri, 24 Jul 2026 20:30:59 +0000

Same task. Two ways of asking. Two very different results. That was the moment it clicked for me, and it has bugged me ever since.

I gave my AI coding agent something to do, got back work that was fine but not good, rephrased the request, and got back exactly what I wanted. Nothing else changed. Same model, same repo, same afternoon. The only variable was me, and how I happened to frame it.

The model was fine. I was the variable.

That is an uncomfortable thing to admit, because it means the quality of my output was riding on whether I remembered the right way to ask on any given day. On a good day I pointed the agent at the failure log first, told it to reproduce before fixing, reminded it of the pattern we already use. On a distracted day I forgot half of that and got half the result.

The knowledge of how to do the work well existed. It just lived in my head, and only when I was paying attention. Nowhere I could reuse it. Nowhere anyone else could either.

Now picture a whole team doing that

For one person this is just friction you learn to live with. Scale it to a team and it turns into something worse. Everyone drives the agent a little differently. One person knows the trick that gets clean results. Another doesn't. One remembers the generator exists and another hand-writes the thing it would have scaffolded correctly.

The output stops being a function of the tool and starts being a function of who is holding the keyboard. That is the real cost, and it hides well, because on any given day the work looks fine. It is the variance that gets you, not the average.

So I wrote the good version down

I started capturing the ways of working that actually produced good results, as small reusable skills the agent could pick up on its own. Not a wiki page nobody opens. Not one giant instructions file that scrolls past the point of being read. Just focused playbooks, each describing how one kind of task should go. How to chase down a hard bug. How to build something test-first. How to turn a rough idea into a proper spec before a line of code gets written.

The part that mattered most was making them easy to reach for. I added a single front door, a router, so nobody has to memorize a catalog. You describe what you are about to do and it points you at the right playbook. All of it is opt-in and made of plain text, which sounds like a small detail but isn't. When a playbook is wrong, fixing it is a one-line change, not a debate.

Matt Pocock's thinking on skills, and on where automation is actually worth adding, shaped a lot of how I approached this (@mattpocockuk). The idea that stuck: a skill only earns its keep if the agent reaches for it at the right moment. Writing it is the easy half. Making it get used is the real work.

What changed wasn't speed

Before, every session started cold. The method lived in whoever was at the keyboard that day. Now the method is the same no matter who asks. The agent runs the same disciplined loop on a bug whether I kick it off or a teammate does.

There is some speed in that, but speed is not the point. Predictability is. I stopped spending attention on remembering how to ask, and I stopped quietly worrying that a good result today would be an average one next week.

Two things surprised me. A skill nobody reaches for is dead weight, so the router ended up mattering as much as the skills. And the ones that earn their place are almost always the boring repeated tasks, not the clever edge cases I thought I was building for.

If you're the one deciding this for a team

You do not standardize how a team uses AI by publishing a policy document. Nobody reads it, and it is stale a week later. You standardize it by making the good path the default path, so the right way to do something is also the easy way.

That is cheap to try. Low risk, easy to change, easy to throw away whatever doesn't prove useful. For something that quietly sets the quality of a growing share of your team's work, that is a trade worth making.

AI-Driven Playwright E2E Tests From One Reference Doc

Muhammad Usman — Mon, 20 Jul 2026 11:11:17 +0000

Writing E2E Tests With One Doc Instead of New Tooling

We had a growing monorepo with two frontend apps and zero end-to-end coverage. The blocker was never Playwright itself. It was that every new feature meant babysitting an AI agent through the tests, click by click, and the output still needed rewriting by hand.

Here is what actually fixed it, and the part I got wrong on the first try.

Why a dedicated package, not a shared test util

The first instinct was to drop Playwright into the existing shared testing package. That does not work, for boring but real reasons:

The unit-test runner and Playwright's runner cannot cohabit in one package.
E2E runs must not be build-cached, while unit tests are cacheable. One package can't be both.
Browser binaries and report output have no business leaking into a shared utility package.

So the e2e suite is its own package, built from scratch. That separation is what let the rest of the decisions stay clean.

The structure that made tests cheap to write

Three rules, enforced by the setup rather than by review comments:

Selectors live only in Page Object Models. Spec files never contain a raw locator.
Page objects are injected as fixtures. No manual instantiation in a spec, ever.
Auth runs once. A setup project logs in, saves the session state to disk, and every test loads it. No login flow inside a test.

The result is that a spec reads like the user story and nothing else:

test('user can open the create form', async ({ createItemPage }) => {
  await createItemPage.goto();
  await expect(createItemPage.heading).toBeVisible();
});

Already authenticated, page object already wired in, no selector in sight.

The part I got wrong first

With the structure in place, I expected the AI-assisted authoring to be the easy win. It was the opposite. Pointing Codegen and the agent at a page produced tests that looked plausible and were quietly wrong. The agent guessed expected values it had no way to know, and every generated spec came back to my desk for a manual rewrite. It was slower than just writing them myself.

The missing piece was context. The agent knew how to click a button. It did not know what the feature was supposed to do — the state machine, the validation limits, which behaviors were correct and which were known bugs.

So I wrote a skill for one feature that spelled all of that out: the flows, the rules, the routes, the confirmed bugs. The next tests it wrote were good. Not "good for AI" good. Better than what I was writing by hand, because it now had the right direction and never lost the thread halfway through a complex flow.

Then I generalized it.

One doc per feature, not a skill per feature

A skill per feature does not scale. The insight was that the skill was generic; only the facts were feature-specific. So I split them.

The authoring engine became one reusable skill. It reads a per-feature reference doc, discovers the routes, reuses or extends the page objects, drives the real app, captures the actual outcomes as assertions, then runs and self-heals the spec. On top of that sits a thin layer: the official Playwright browser-driving skill for the low-level interaction, and one dedicated skill for the single most complex domain that earned extra shortcuts.

What a new feature needs now is one document: entities, state machine, validation rules, and a numbered list of known bugs. That is it. The reference doc is the contract. If it does not exist, the agent is told to stop and ask for it instead of inventing rules from the UI, which is exactly the failure mode that cost me the first week.

Before: walk the agent through every flow, then rewrite the output.
After: write one ground-truth doc, point the generic skill at it, review real tests.

The reusable idea

The lesson had nothing to do with Playwright. Agents do not fail e2e work because they can't drive a browser. They fail because they guess at behavior no one wrote down. Encode the business truth in a document the agent is forced to read, keep the authoring engine generic, and adding coverage for the next feature costs one doc instead of a new pile of tooling.

DEV Community: Muhammad Usman