DEV Community: Jesse Houwing

Fixing complex bugs with AI - A deepdive into the PowerShellScriptAnalyzer

Jesse Houwing — Thu, 17 Sep 2026 22:01:50 +0000

Down the Rabbit Hole - Susan Dussaman - CC BY-SA 2.0

Some build failures are annoying. Others are simply impossible. Mine was the second kind.

On a plain ubuntu-latest runner, a recursive Invoke-ScriptAnalyzer run would stop dead with an error saying that Get-Command was not recognised as the name of a cmdlet. Not some obscure module. Not a typo in a script. Get-Command. The single most fundamental discovery cmdlet in PowerShell, missing from PowerShell.

The same workflow had been perfectly happy on PSScriptAnalyzer 1.24. The only thing that changed was the analyser version.

What follows is the story of PR #2206. It started as a bug fix, turned into a performance project, and nearly became a pile of AI-generated slop. Along the way, AI tooling helped me, led me in the wrong direction, and showed that getting the code ready is not the same as getting it released.

It starts because it is open source

The broken workflow was mine. It was actions-semver-checker, one of the small projects I maintain, and the only change in the failing pull request was the pinned analyser version. So I filed issue #2205 with links to both runs.

This is easy to take for granted.

In most of my working life, that would have been the end of my involvement. You raise a ticket. You attach logs. You wait. Somebody you will never meet triages it against priorities you cannot see, and your only remaining move is to pin to the last version that worked and hope the next release quietly fixes it. You are a spectator to a problem that is costing you time.

Here I could carry on. The repository is public, so I cloned it, diffed 1.24 against 1.25, and had a source-level diagnosis in the issue within twenty minutes: AvoidAlias asks the engine about every non-alias command it sees; that question travels through CommandInfoCache, which used a pool of child runspaces and ran Get-Command inside them via PowerShell.Create(); PR #2125 had recently taught that path to parse Module\Command and pass -Module along, and the build reference for System.Management.Automation had moved from 7.4.7 to 7.4.14.

So the error had a fairly ordinary cause. We were not losing Get-Command from PowerShell. We were losing it from our own child runspace, under concurrency, on a machine with a particular set of modules on disk.

In that first write-up I was careful about what I could not prove. My repository does not use module-qualified invocations, so I could not show that #2125 alone caused it. I could only show that it was the most likely place to look. It is important to keep measurements and suspicions separate.

No support ticket or NDA. I could read the code, form a theory, write a test and open a pull request. That ability to take ownership is one of the main reasons open source matters.

The first fix was the obvious one, and it was not enough

My first instinct was containment. Catch the transient failure, return null, and let the rules follow their existing "I could not resolve that command" path. A failed lookup should not take down an entire analysis run.

One detail mattered: the cache stored Lazy<CommandInfo> values, and Lazy<T> caches exceptions as well as results. One unlucky lookup would poison that entry for the rest of the process. So the fix evicted only the faulted entry rather than clearing everything.

Then I went a step further and serialised all command lookups onto a single dedicated runspace behind a lock. If the engine is not thread-safe here, stop being concurrent here.

That felt like the answer. It compiled, the tests went green, and the original failure disappeared.

It was not the answer.

The review that changed the question

Twenty-six days later, one of the maintainers replied. And in a few short paragraphs he handed me a pile of context I had no way of acquiring on my own.

He told me this was not the first time CommandInfoCache had produced concurrency issues — there had been several, over a period, from different reporters. He told me the root cause as the maintainers understood it was PowerShell engine internals not being thread-safe, which turned my careful hedge ("I cannot prove #2125 alone triggers this") into a settled question. He pointed me at an attempt to fix it on the PowerShell side. And he mentioned, almost in passing, that he had previously seen lock-based optimisations in this area degrade performance badly by removing concurrency.

I want to be fair about what I could have found on my own. Working through the 1.24-to-1.25 diff revealed links to the PowerShell runtime repository, where the underlying concurrency problems are documented.

What he added was the knowledge that had not been written down. This was not a one-off. The maintainers already agreed on the likely cause. Most importantly, a previous lock-based fix had made this code much slower. It was tried and abandoned, so there was no merged change for me to find.

He did not replace a day's research with five minutes of magic. He gave that research context that only a long-serving maintainer had. He told me which questions were already settled, which ideas had caused trouble before, and where the real risk was.

Then he asked for one specific thing.

Analyse PowerShell's own build.psm1 in a freshly started shell, then immediately again in that same shell. Do it for my branch and for the implementation in 1.25. Post the numbers.

Those two measurements matter because PSScriptAnalyzer gets used in two very different ways, and the rest of this story hangs on the distinction.

In a pipeline, it runs once. A GitHub Actions job starts a runner, installs the module, runs Invoke-ScriptAnalyzer -Recurse, and exits. Nothing is cached or warmed up. If it takes thirteen seconds instead of three, every contributor pays those ten seconds on every push and pull request. If it fails, the check is red and somebody has to find out why.

In an editor, it keeps running. The PowerShell extension in VS Code keeps the analyser alive in a background process and runs it again as you type. The first analysis after opening a workspace is a one-off start-up cost. After that, the caches are hot and .NET has optimised the busy code paths. Those later runs need to be fast and consistent, without warnings appearing and disappearing between keystrokes.

So the benchmark measures both situations:

The first run in a fresh process stands in for the CI job. It is the number most users experience.
The immediate repeat run in the same process stands in for the editor. It shows what happens once everything is populated and optimised.

The two numbers can move in opposite directions. Aggressive caching makes the second look good at the expense of the first. Serialising everything behind a lock — my fix — protected correctness but made the first run much slower. Non-determinism is also easy to miss in timings: in an editor, it becomes a warning that appears and disappears while you work.

Keep that trap in mind, because I walked straight into it later on and very nearly shipped the result.

(In the pull request I call these the cold and warm runs.)

He had seen this type of fix go wrong before and asked for the cheapest experiment that would settle the question.

At that point the pull request stopped being "can we make it stop crashing" and became "can we make it stop crashing at a cost anyone would accept". The information that changed the work was not in the repository. It came from a person.

The review was the most valuable input I received in the entire project. A maintainer is not an obstacle between you and a merge.

Building a benchmark

So I built a benchmark. Not a stopwatch in a script — an isolated GitHub Actions matrix across Ubuntu and Windows, analysing two real projects (the PowerShell repository itself, and actions-semver-checker, the very repository that had started all this), timing both the fresh-process run and the immediate repeat, and taking a SHA256 fingerprint of the normalised findings so I could prove the analyser's output had not changed while its timings did.

Every comparison from here on is my branch against the implementation in 1.25, built from source in the same job, on the same runner, analysing the same files.

Then the benchmark told me my solution was bad.

Analysing the PowerShell repository in a fresh process, 1.25 took 7.45 to 8.12 seconds. My branch took 25.90 to 27.54. On the semver project, 9.49–12.76 seconds became 33.55–36.80. Repeat runs in the same process were fine, even slightly better. But the fresh-process runs — the ones that represent almost every real use — were roughly three times slower.

The maintainer's response was pragmatic: perhaps make it an opt-in switch, so people who need determinism can pay for it and people who need throughput can keep their concurrency.

I did not want to ship that compromise. A correctness fix that you have to opt into is a correctness fix most people will never get.

There was one encouraging result in that matrix. The 1.25 samples kept failing and needing reruns. Mine never did. The benchmark had also become a determinism test.

How it nearly became slop

The middle of the project is not the part people usually publish.

Over a few hours on 14 September, the branch gained retry loops, telemetry counters, two build variants, per-rule exception handling and a summary job for the growing matrix. I then had to fix the matrix because a later change overwrote the results and made the retry variants indistinguishable.

Every change was reasonable on its own. Copilot produced several of them quickly from a clear brief. I ran eleven separate agent tasks on my fork; eight were merged and three were closed as superseded. I would not have explored half as many directions without it.

But I had built an increasingly complex system for recovering from the problem, followed by another system for measuring the recovery. The diff and test suite kept growing, while none of it addressed why we were doing something dangerous in the first place.

This is what I mean by a vibe-coded mess. The individual changes were not bad code and could have passed review on their own. The problem was the direction: an agent will keep adding layers unless you stop and ask a different question.

The numbers did not improve enough to justify the complexity.

So I deleted it. Retries, telemetry, build variants, rule-level catches — all of it came out in one commit. The repository-wide regression test, which the automated reviewer had flagged as depending on whatever happened to be in the repo, was replaced by a small, self-contained test that hammers the rules in parallel. The branch got smaller for the first time.

Asking a different question

With the scaffolding gone, I went back to instrumenting, but this time to answer one question: why are we calling into PowerShell so much?

The answer was stark.

A failed Get-Command lookup cost around 59 ms. A hit cost around 0.8 ms. Seventy-four times more expensive, because a miss sends PowerShell off to autodiscover modules across the entire PSModulePath.
On a real 111-file repository, 170 of 223 lookups — 76% — resolved to nothing. Those misses accounted for roughly 77% of total runtime.
Priming the cache did not help. Get-Command * took 78 seconds to enumerate around 116,000 commands, and later misses were no faster because PowerShell does not cache negative lookups.
The cache was already shared across Invoke-ScriptAnalyzer calls within a process. A second analysis in the same session performed zero lookups. The editor case was already in good shape. The pipeline case — one analysis, from nothing — was the whole problem.

So the expensive operation was not "look up a command". It was "look up a command that does not exist". And once you frame it that way, the question becomes: which of these 170 non-existent commands should we never have asked about?

That turned out to be a much better question, and the answers were not performance bugs. They were correctness bugs that happened to be slow.

Four lookups we should never have made

Probing Get- in front of a hyphenated name. AvoidAlias has a sensible fallback: if it sees process, it tries Get-Process. But for an unrecognised Test-ActionInput it produced Get-Test-ActionInput, which matched none of the 116,289 commands on my machine and doubled the cost of every miss.

Ignoring functions the script itself defines. If my script defines its own Get-ChildItem, the analyser was resolving that name against the cmdlet installed on my machine. UseCorrectCasing could then advise me to rename a function based on a command my script deliberately shadows. Collecting FunctionDefinitionAst names once per analysis fixed the advice and removed many 59 ms misses at the same time.

Treating member names as commands. UseShouldProcessCorrectly built a call graph with a vertex for every invocation member, so $x.Substring(1) triggered Get-Command Substring. Worse, $dir.mkdir() could be credited with SupportsShouldProcess, because mkdir happens to exist as a PowerShell function on Windows. A method call was being scored against an unrelated cmdlet. Tagging vertices reached through genuine command invocation fixed the semantics and the cost.

Forgetting that modules dot-source themselves. A function in one library file routinely calls a function defined in a sibling file, with no import visible in either. I grouped files by directed dot-source relationship and gave each file the functions reachable through its own dot-source closure — deliberately not merging entire connected components, so a test script that dot-sources one library does not inherit functions from its siblings.

That last one took a 111-file module from 13.5 seconds to 2.9 in a fresh process. Repeat runs unchanged. Findings identical.

Then I removed the instrumentation counters. They had done their job.

The complete list of fixes

Here is the full set, sorted by the kind of problem it is.

Threading and lifetime bugs

Concurrent use of a non-thread-safe engine surface. CommandInfoCache resolved commands through a pool of child runspaces, each running Get-Command via PowerShell.Create(). PowerShell's command-discovery internals are not safe to drive this way from multiple threads. The symptoms included CommandNotFoundException for Get-Command itself, null reference exceptions, pipeline creation failures, even process aborts and hangs in PowerShell 5.1. The pool is replaced with one dedicated runspace behind a re-entrant lock.

Lazy<T> exception poisoning. The cache stored Lazy<CommandInfo> values, and Lazy<T> caches the exception just as faithfully as it caches a result. One transient failure permanently poisoned that command for the rest of the process, so a momentary race became a persistent wrong answer. Now only the faulted entry is evicted — and crucially, genuine negative results are still cached, so "this command really does not exist" does not become a repeated 59 ms lookup.

Unsynchronised disposal. The disposal paths did not take the runspace lock, so teardown could race an in-flight lookup. Both paths now lock.

Metadata access escaping the lock. Even with lookups serialised, rules were reaching back into the engine afterwards — resolving parameters on a CommandInfo re-enters PowerShell on the caller's thread, outside the protection. Metadata reads are now serialised alongside the lookups through a Func<CommandInfo, T> callback, and static export metadata is used in preference to dynamic parameter resolution where it is available.

Get-Command resolved by unqualified name. The internal call used the bare name, which is itself subject to shadowing and discovery failure. It is now fully qualified as Microsoft.PowerShell.Core\Get-Command.

Non-deterministic output as a consequence. All of the above combined into the symptom that bothers me most: the same input produced 154, 155 or 157 findings depending on which lookups happened to lose a race. In a pipeline that means a build that fails on a rerun of an unchanged commit. In an editor it means warnings that appear and vanish while you type. Static analysis that is not reproducible is not much use in either.

The analyser could also crash. This is what users saw.

Your check goes red with an error saying Get-Command does not exist. Nothing in that message points at the analyser's internals. You re-read your own diff and wonder whether the runner image changed, a module failed to install, or PowerShell itself is broken. Eventually you rerun the job. A fresh run succeeded between 55 and 59 times out of 100, so it may turn green without explaining anything.

When that happens often enough, people pin to an older version, add continue-on-error, exclude the rule, or remove the analyser from the pipeline. That is what I did first: I reverted to 1.24 and continued with other work. People rarely file an issue to say they have stopped using your tool. An unreliable check can leave you with silence and a linter nobody trusts.

Logic and semantic bugs

Script-defined functions did not shadow installed commands. If a script defined its own Get-ChildItem, the analyser resolved that name against whatever was installed on the machine running the analysis. UseCorrectCasing would then recommend casing taken from a cmdlet the script deliberately overrides. The analyser was modelling the host instead of the code.

Dot-sourced siblings were invisible. The same bug at module scale. A library file calling a function defined in a sibling file — the normal way a PowerShell module composes itself — resolved against the host rather than the module. Fixed by giving each file the functions reachable through its directed dot-source closure, deliberately without merging whole connected components, so a test script that dot-sources one library does not inherit unrelated siblings' functions.

Member names were resolved as commands. UseShouldProcessCorrectly created a call-graph vertex for every invocation member, so $x.Substring(1) produced a Get-Command Substring. The real damage was the false negative: $dir.mkdir() was credited with SupportsShouldProcess, because mkdir exists as a PowerShell function on Windows. A method call was silently scored against an unrelated cmdlet, and a rule violation went unreported. Only vertices reached through genuine command invocation are resolved now.

The Get- prefix probe applied to hyphenated names. The fallback exists to turn process into Get-Process. Applied to an unresolved Test-ActionInput it asked about Get-Test-ActionInput, which matches none of the 116,289 discoverable commands on my machine.

A rule's own test fixture was passing for the wrong reason. A fixture declared SupportsShouldProcess and never called ShouldProcess, which should be a violation. But it defined a local Set-Service that was resolved as the system cmdlet, masking the result. The shadowing fix exposed that false negative in the test suite.

Performance problems

Failed lookups dominated the fresh-process run. 59 ms for a miss against 0.8 ms for a hit, because a miss sends PowerShell autodiscovering modules across the whole PSModulePath. On a real 111-file repository, 170 of 223 lookups resolved to nothing — 76% of the calls, roughly 77% of the runtime.

Every semantic fix above is also a performance fix. Script-local and dot-source resolution answer from the syntax tree instead of the engine. Dropping the Get- probe halves the cost of the misses that remain. Not resolving member names removes an entire category of lookup that was never meaningful. The speedup comes from asking fewer questions, not from answering them faster.

Files are parsed once per run. The dot-source pre-pass retains the syntax trees it builds rather than discarding them, so a recursive analysis no longer reparses. -Fix opts out, because it rewrites each file before analysing it.

Serialisation stopped mattering. The lock cost that made fresh-process runs three times slower in September is still there — I did not remove it, it is what makes the analysis deterministic. It simply stopped being on the hot path once lookup volume collapsed.

One thing is deliberately not changed: parameter and metadata re-evaluation. I assumed it was a bottleneck, but instrumentation showed around 97 lock acquisitions per pass. It was not worth optimising.

The trade I nearly accepted

By this point I was pleased with the progress.

The pipeline case had gone from thirteen and a half seconds to under three. That was the number I had been chasing since the maintainer's review, it was the number in the issue that started all this, and it was the number almost every user would feel. I had been staring at it for days.

The editor case, meanwhile, was a couple of seconds slower on repeat runs. I started telling myself that was fine: CI pays for the first run on every push, while a second or two in a long-lived editor session is barely noticeable. A four-times improvement in CI for a small editor regression seemed acceptable.

But it was the trade I had told myself not to make when I decided to measure both cases. It is easy to ignore a principle when one of the numbers looks so good.

I would have had to explain that trade-off to a maintainer who had already warned me about regressions in this area. Instead of arguing it away, I investigated it.

The benchmark was lying to me, twice

Two things then made the harness itself the suspect.

The repeat-run numbers looked like a regression, but the first run wrote its findings to disk before the second measurement began. Garbage collection for those allocations landed on the second run's clock. Running a collection before each timed run removed this effect.

The second problem was less obvious. Once the fresh-process run became three times faster, the second run was no longer a steady state. .NET starts with quick, unoptimised code and recompiles methods after enough use. That work used to fit inside a 13-second first pass. Now it continued into passes two and three, so my "fully warmed up" run was still warming up. A few discarded analyses before the measurement also removed this effect. Disabling tiered compilation confirmed the cause: in that case the second pass was already stable.

The lesson applies beyond this project: a benchmark is software, with its own bugs. Improving the product had invalidated assumptions in the measurement.

Chasing the repeat-run numbers also revealed work the analyser repeated after everything was cached: files were reparsed and lookups repeated. Almost all of that work also happened on the first run, so improving the editor case improved the pipeline case too.

The regression was not the price of a faster first run. It showed that I had more work to remove. I did not foresee that; I found it because I was unwilling to accept the regression.

Evidence reviewers can check

My concern with AI-assisted open source contributions is not the code but the claims. A pull request saying "fixes a race condition, improves performance" gives a maintainer little they can verify.

So I set out to produce evidence that a reviewer could check without trusting me.

The final cross-platform CI results. "First run" is a freshly started process analysing the project once; "repeat run" is a second analysis immediately afterwards in that same process.

Operating system	Project analysed	First run, 1.25	First run, this PR	Repeat run, 1.25	Repeat run, this PR
Ubuntu	PowerShell	6.62 s	2.80 s	0.10 s	0.11 s
Ubuntu	semver checker	12.92 s	5.21 s	1.96 s	1.82 s
Windows	PowerShell	8.49 s	2.60 s	0.10 s	0.07 s
Windows	semver checker	12.96 s	6.52 s	2.29 s	2.23 s

The pipeline case is two to three times faster, while the editor case is unchanged or slightly better. What looked like a trade-off was work I had not removed yet.

On the 111-file project across two CI attempts, 1.25 crashed 27 times. The runs that survived reported 154, 155, or 157 findings for the same input. The proposed changes completed all 40 measurements and always produced the same fingerprint and 154 findings. One caveat: the benchmark retries 1.25's failures which also means the timings exclude its worst runs and make it look faster than it actually is.

I proceeded with local stress testing, 100 freshly started processes per configuration: 1.25 succeeded 59 times out of 100 on PowerShell 7.6.6, and 55 out of 100 on Windows PowerShell 5.1. With the proposed changes applied: 100 out of 100 in both. Across 400 analyses in total, 86 runs of 1.25 died outright — 58 command-not-found errors, 22 null references, several pipeline creation errors, two process aborts and two timeouts. All of those signatures went into the PR description, because "it crashes sometimes" is an anecdote and a list of distinct failure modes is evidence.

One experiment also justified deleting the retry code. In 100 freshly started PowerShell 7 sessions, 35 hit a caught failure on their first attempt. Twelve recovered later in the same session. Twenty-three kept failing until a 120-second cut-off. Retrying is not recovery when the failure may have corrupted the state you are retrying in. Twelve recoveries out of thirty-five failures is not a reliable strategy. I only know that because I measured it.

Build and test problems along the way

This project also took many hours on things that had nothing to do with the bug but were necessary to work on it.

The build pins .NET SDK 8.0.4xx, and having SDK 10 installed does not help. Building with -PSVersion 7 produces only the net8 output; the module root DLL is the net462 build that Windows PowerShell 5.1 loads, and it stays stale until you build -All. I spent some time investigating why fixed code still failed before I understood that.

My editor's built-in test runner was worse than useless here, because it runs Pester without importing the repository build — so Invoke-ScriptAnalyzer silently resolved to the globally installed PSScriptAnalyzer. The symptom is beautifully misleading: your new behaviour tests fail while every control test passes. The fix is to replicate the harness and prepend out to PSModulePath explicitly.

PSUseCorrectCasing is a configurable rule and is off by default, so -IncludeRule alone does not switch it on. For A/B comparisons, build the baseline in a git worktree rather than stashing; otherwise it is easy to lose track of which build is running. Test fixtures that use Add-Type must also be C# 5 compatible, because the compiler Windows PowerShell 5.1 uses rejects null-conditional operators and expression-bodied members.

These detours are often called yak shaving. Copilot handled much of it: it read build errors, found the SDK mismatch and noticed the stale module. Work that might have taken my evening happened while I did something else. But each loop used tokens, and the agent focused on the immediate problem rather than the state of the machine. It installed .NET 8 into a temporary folder at least five times before I noticed and installed it system-wide in about a minute. It had solved the local problem each time without asking why it kept returning.

It also changed how I worked. I could start a task, do other work, check the result and correct the direction. I steered part of this project from my phone in a school car park while waiting to collect my kids.

Those detours still had to be correct. The test runner using the globally installed module could have sent me changing code that was already right. This work is mostly invisible in the finished pull request, but it is a normal part of contributing to an established project. AI can handle much of it, but somebody still has to notice when it solves the same problem five times.

Where AI helped

Copilot was very good at bounded, well-specified work. Build an isolated benchmark matrix. Make this fixture cross-platform. Trace where these 106 unresolved lookups come from. Port this test to C# 5 syntax. With a clear brief and a way to verify the result, the agents produced work I merged with minor edits. They were also useful for the tedious middle of the investigation: another stress-test variation, readable result formatting, or broken links in a pull request description. The instrumentation that produced the 59 ms figure took minutes rather than an afternoon. That made the difference between running the experiment and deciding it was too much work.

What it did not do was tell me when to stop. It did not question the four layers of retry machinery, notice that a local Set-Service made a test pass for the wrong reason, or ask whether the repeat-run slowdown came from the product or the benchmark. I changed direction because the numbers looked wrong.

Could an agent have got there by itself? Possibly. An agent reviewing a diff can identify these problems, and repeated review and revision can improve the result. I have not found a reliable way to know when that loop is finished. It may stop at a plausible answer or continue until it loses direction.

That is why I kept steering it by hand. A few minutes deciding where to look is cheaper than asking the machine to discover the direction. "Trace where these unresolved lookups come from" is targeted and verifiable. "Find out why this is slow and fix it" is open-ended and can use many tokens exploring ideas I could reject quickly. Often the human contribution is simply narrowing the search before the machine starts.

This is the difference between vibe-coded code and code written with the same tools but properly reviewed. It is not the amount of code the model produced. It is whether the person directing it can ask a precise question. "Is the dot-source closure transitive, and should it be?" requires enough knowledge to understand why transitivity could be dangerous. Without that knowledge, you are left with "make it faster" and "fix the failing test", and may not be able to spot a plausible but wrong answer.

Expertise can also narrow the questions you ask. I spent this project asking "why is this lookup happening?" and much less time asking "should command resolution work this way at all?" or "how would I design this today?". People familiar with a codebase tend to accept its existing shape. An agent does not, which can be annoying but also useful. Next time I want to spend some time asking the basic questions before I choose a direction.

There is another limit: what an agent can know. An assistant can read an issue tracker and follow links into another repository. That is how the PowerShell runtime issues behind this bug came to light.

It cannot see what was never committed: an approach tried two years ago and abandoned because it halved throughput, or a conclusion a maintainer never wrote down. A repository records what survived, not everything that was attempted. A model can compare an approach with the code that exists. A maintainer can also tell you which ideas have failed before.

AI made me much faster at producing possible solutions and evidence. It did not decide which agent branches should survive, what to remove, or when the numbers were misleading. Without these tools, however, I would almost certainly never have done the work.

This is not my job. It is a side project on somebody else's repository, fitted between client work, family and everything else. I was never going to spend three uninterrupted evenings fixing the build, writing a benchmark harness and running four hundred stress analyses by hand. I would have pinned to 1.24 and moved on. Instead I could work in twenty-minute pieces, with an agent continuing while I did something else.

So the wasted tokens and repeated SDK installs need that context. The alternative was not a cleaner version of this pull request. It was no pull request, an unfixed bug, and a linter that a few more people might stop trusting.

The hard part comes next

The technical work is nearly done. The crashes are gone, a first analysis in a fresh process is two to three times faster, and there is a benchmark anyone can rerun. Now it needs to be released.

That carries risk. Three of the four main improvements are behavioural changes to command resolution. Functions defined by a script now shadow installed cmdlets. Member invocations are no longer resolved as commands. Dot-sourced files now share their function definitions. These are fixes, but they also mean that some users will see different findings.

Some of those differences are the whole point — the false SupportsShouldProcess credit from $dir.mkdir() should never have been there. But "we removed a false negative" reads very differently when it lands in your pipeline as a new build failure on a Friday afternoon.

Then there are custom rules that I cannot see or test. I have checked the projects I can reach: internal ones, actions-semver-checker, PSScriptAnalyzer's own build and test suite, and a handful of open source repositories. The results are consistent, but that is not proof.

So please go and break it

This is where I need your help.

If you use PSScriptAnalyzer in CI, in your editor, or with custom rules, please run the proposed build against your code and tell me what you find. Ten minutes of your time is worth more than another day of mine because I cannot test your code myself.

Build it from the pull request:

# Install the .NET 8 SDK if you do not have it (on Linux/macOS: https://dot.net/download)
winget install Microsoft.DotNet.SDK.8

# Requires the GitHub CLI
git clone https://github.com/PowerShell/PSScriptAnalyzer.git
Set-Location PSScriptAnalyzer
gh pr checkout 2206

# -All also builds the net462 output that Windows PowerShell 5.1 loads
./build.ps1 -Configuration Release -All

$proposed = (Resolve-Path ./out/PSScriptAnalyzer/1.25.0/PSScriptAnalyzer.psd1).Path
$project = 'C:\src\your-project'

Then analyse your project twice, once with the version you use today and once with the proposed one, and compare the results. Use a fresh process for each so neither run can contaminate the other:

$analysis = {
    param($ModulePath, $ProjectPath, $OutputPath)

    $ErrorActionPreference = 'Stop'
    $temporaryPath = "$OutputPath.tmp"
    try {
        Import-Module $ModulePath -Force -ErrorAction Stop
        Write-Host "Analysing with PSScriptAnalyzer $((Get-Module PSScriptAnalyzer).Version)"

        # Collect everything before writing anything, so a failed run cannot leave a partial CSV.
        $findings = @(Invoke-ScriptAnalyzer -Path $ProjectPath -Recurse -ErrorAction Stop)
        try {
            New-Item $temporaryPath -ItemType File -Force | Out-Null
            $findings |
                Select-Object RuleName, Severity, ScriptName, Line, Column, Message |
                Sort-Object RuleName, ScriptName, Line, Column |
                Export-Csv $temporaryPath -NoTypeInformation -ErrorAction Stop
            Move-Item $temporaryPath $OutputPath -Force
        }
        finally {
            if (Test-Path $temporaryPath) {
                Remove-Item $temporaryPath -Force -ErrorAction SilentlyContinue
            }
        }
    }
    catch {
        $details = $_ | Out-String
        [Console]::Error.WriteLine($details)
        exit 1
    }
    exit 0
}

function New-AnalyzerSnapshot {
    param($ModulePath, $OutputPath)

    Remove-Item $OutputPath, "$OutputPath.tmp" -Force -ErrorAction SilentlyContinue
    & pwsh -NoProfile -Command $analysis -args $ModulePath, $project, $OutputPath
    if ($LASTEXITCODE -ne 0) {
        Remove-Item $OutputPath, "$OutputPath.tmp" -Force -ErrorAction SilentlyContinue
        throw "PSScriptAnalyzer failed with exit code $LASTEXITCODE. No comparison was made."
    }
}

# What you get today, followed by what you get with this pull request
New-AnalyzerSnapshot PSScriptAnalyzer "$PWD/before.csv"
New-AnalyzerSnapshot $proposed "$PWD/after.csv"

# Anything listed here is a behaviour change worth knowing about
Compare-Object (Import-Csv ./before.csv) (Import-Csv ./after.csv) `
    -Property RuleName, ScriptName, Line, Column, Message

Add -CustomRulePath and -Settings to the Invoke-ScriptAnalyzer call if you use them. Those are the cases I can predict least and would most like to hear about.

The longer script is deliberate. When I tested a simple pipeline against actions-semver-checker using my installed 1.25 version, the baseline printed The term 'Get-Command' is not recognized, exited with code 1, and still wrote 48 findings to before.csv. The proposed build wrote 154. The apparent 106 additions were not behaviour changes; the failed run had dropped those findings. This script collects the complete result before replacing the CSV, deletes stale output, and stops if either analysis fails.

Some differences are expected: a finding may disappear because a member call is no longer resolved as a command, casing advice may stop firing on a function defined by the script, or a new PSUseShouldProcess warning may reveal a previous false negative. If something else changes — a rule stops working, a custom rule throws, a finding moves without a clear reason, or the analyser crashes — please report it on PR #2206. The rule name and a small example are enough; you do not need to diagnose it.

An identical result is useful too. "No change on a 300-file module with four custom rules" is real evidence, and I need more of it.

Being able to suggest a fix to an open source project also means taking responsibility for what the fix does to its users. I can give the maintainers evidence, explain the behaviour changes, and respond when something breaks. I cannot test code I do not have.

Generating code has become cheap. Earning a maintainer's trust has not. I would not want it to.

PR #2206 is open at the time of writing. The benchmark workflow is on the branch, so you can run it yourself — and I would rather you did than take my word for any of the above.

A trust checklist for GitHub Actions

Jesse Houwing — Fri, 04 Sep 2026 17:09:14 +0000

Every action you reference in a workflow runs on your runner, with access to your workspace and whatever token you hand it. uses: some-org/some-action@v3 is an instruction to execute someone else's code inside your CI environment. It deserves at least as much scrutiny as a production dependency, and usually receives considerably less.

What follows is the checklist I use when assessing whether an action can be trusted. A caveat before the list: this is partly a wish list. Some of these attributes are present on very few actions today. Not all of them can be detected automatically or queried through an API, and several require manual review or AI assistance to establish. Treat it as a set of weighted signals rather than a pass/fail gate.

The list is split into two tiers: signals that carry real weight, and signals that are worth a glance but should not be decisive.

Tier 1: signals that carry weight

Tooling and maintenance

Signal	How to verify
The repository uses GitHub Advanced Security for Actions, ActionLint, and/or zizmor	PR checks and workflow run logs
The repository uses Dependabot or Renovate — for both its action references and its underlying implementation dependencies — and is current with updates	Pull requests and workflow logs
The repository configures a Renovate/Dependabot cooldown (minimum release age)	Configuration file on the default branch
The action has unit and/or integration tests	Repository contents and workflow runs
The repository uses StepSecurity Harden-Runner or GitHub's new egress firewall in its own workflows	Workflow files in the repository

On that last point: check that the firewall is not running in audit mode only. Audit mode records egress without blocking it, which is a useful first step but provides no actual containment.

Runtime and dependency hygiene

Signal	How to verify
Uses a currently supported Node handler (JavaScript actions)	`action.yml`
Uses an up-to-date base image (Docker actions)	`action.yml`
Does not depend on unpinned actions, npm packages or Python packages, or on versions that cannot be controlled through inputs — no bare `npx eslint`	Not always straightforward. AI-assisted review, such as Copilot code review, frequently finds these
Does not use floating Docker image tags such as `latest`	`action.yml` for Docker actions. Harder for actions that invoke the Docker CLI internally; AI review often catches these
For composite actions: references pinned, locked or immutable actions	Largely verifiable with GHAS for Actions, plus manual review
Validates remotely downloaded artefacts	Sometimes detectable with AI. Otherwise manual review

The unpinned-dependency category is the one I would prioritise. An action that pins its own action references but then runs npx some-tool at execution time has simply moved the unpinned dependency one level down, where it is harder to see.

Release integrity

Signal	How to verify
Versioning is implemented correctly — semantic versioning, correctly maintained floating tags	Difficult to inspect. I wrote a GitHub Action and PowerShell library for it. A surprising number of actions have a questionable history where version numbers are concerned
Uses immutable releases	Release metadata. A recently introduced feature; many actions still need to adopt it
Has release attestation, and the attestation validates	Should be attached to the GitHub release, and validation should be performed against the original source
Passes an anti-virus scan	You must do this yourself — GitHub does not virus-scan action releases

One detail worth knowing about attestations: they are bound to the source repository and cannot be transferred. If your organisation mirrors actions into an internal repository, the attestation does not travel with the mirror. Validation has to happen against the original source.

Governance

Signal	How to verify
Has licence, security and contributing policies	Policy files on the main branch
Uses a licence your organisation accepts	Licence file. In practice it is often sufficient that a licence is defined at all

Tier 2: weaker signals

These are worth looking at. None of them should decide the matter on its own, and each comes with a meaningful caveat.

Signal	Why it is weak
Published on the Marketplace	Several widely used actions are not, including GitHub's own CodeQL actions
No dependencies with known security issues	Sometimes you are simply waiting for a fix to propagate through the dependency chain. Sometimes a vulnerability does not apply to the Actions hosting model at all and can safely be ignored
No open security advisories for recent versions	Useful, but absence of advisories is not evidence of absence of problems
A reasonable OpenSSF Scorecard score	Genuinely hard to achieve. Few organisations or individuals make the effort. I attempted it for my own actions and abandoned it — a great deal of work for very little recognition
More than one maintainer	Wishful thinking for many niche actions
Hosted by an organisation rather than a personal account	Same caveat
The repository has required PR checks	Only visible in recent pull requests and workflow logs
Recent commits went through pull requests	Wishful thinking for many niche actions
Recently updated	Often unnecessary for PowerShell or Bash actions. Considerably more important for Docker and JavaScript actions
Verified creator badge	The badge adds very little and represents no additional security guarantee
Many stars	Stars can be bought in bulk, and cheaply
Few open issues	Some actions accumulate many issues precisely because the community is active — Renovate, for example. Others run aggressive stale-bot configurations to keep the count low
Few open pull requests	Same reasoning. Pull request cycle time would be a better indicator, but it is not easy to read out
Used by organisations you trust	The dependents view shows which organisations use the action directly on public GitHub. A reasonable indication of who else has extended trust
Has a clear feedback channel (issues, community, email, support link)	Indicates responsiveness rather than security
From a well-known organisation (AWS, Microsoft, Azure, GitHub)	No guarantee. Large vendors have been caught by recent supply-chain worms such as Shai-Hulud
The release is a signed commit	Commit signing remains awkward in public open source, and its meaning is limited — anyone can sign a commit
The repository has Copilot code review, CodeRabbit or a similar integration enabled	AI offers no guarantee and is sometimes wrong, but it does often catch things that would otherwise slip past

The recurring theme in this tier is that most of these signals measure popularity or project health , not security. They are easy to query, which is precisely why they get over-weighted. Ease of measurement is not the same as relevance.

Working through it in practice

Not every signal costs the same to check. It helps to separate them by effort:

Run automated checks on the candidate action. ✅
Is there a clear disqualifier? ❓
- Yes: reject the action or request changes. ❌
- No: continue with AI-assisted review. 🤖
Are there findings that need confirmation? ❓
- Yes: perform a manual review. 🔎
- No: manually review high-risk areas only. ⚠️
Make a decision. ✅
Lock the resolved commit regardless of the decision. 🔒

Automatable or queryable: presence of policy files, licence type, Node handler version, action.yml contents, immutable release status, marketplace presence, advisory status, maintainer count, update recency.

Requires AI assistance to find reliably: unpinned transitive dependencies, floating Docker tags in shell invocations, unvalidated remote downloads. These are the categories where reading every line manually is impractical and pattern matching is unreliable.

Requires manual review: versioning history, artefact validation logic, whether an egress firewall is actually enforcing, and the overall question of whether the action does what it claims.

The control that matters most

The checklist tells you whether an action is worth trusting at the moment you look at it. Something else has to make that assessment hold, because a floating tag can be force pushed at any time — by the maintainer, or by whoever compromises the account. The version you assessed is not necessarily the version that executes tomorrow.

Three mechanisms address this, and they achieve broadly the same effect:

Mechanism	Where it applies
Commit SHA pinning	Consumer side, per reference
Immutable releases	Producer side, per release
Action lock files	Consumer side, per repository

All three ensure the code you reviewed is the code that runs. My clear preference is for immutable releases and lock files over raw SHA pinning.

Immutable releases

When a repository has immutable releases enabled, a release published with a release-specific tag such as v1.0.0 becomes unchangeable. The tag cannot be moved afterwards.

Note that this applies to tags tied to a GitHub release. Floating convenience tags such as v1 or v1.1 — which maintainers deliberately move to the latest compatible patch — are not covered, and are not intended to be. That is precisely why consumers still need a locking mechanism of their own.

For maintainers, this is one of the cheapest items in Tier 1 to adopt, and it improves the situation for every consumer at once.

Action lock files

gh-actions-lock is a gh CLI extension, part of GitHub's Workflow Dependency Pinning effort. It produces a lockfile at .github/workflows/actions.lock that pins every workflow dependency to a verified commit.

gh extension install github/gh-actions-lock
gh actions-lock # scan workflows, pin, write the lockfile
gh actions-lock --relock # re-resolve refs that have legitimately moved

The important point is that this is not merely SHA pinning with extra steps. The lockfile carries verification that a bare SHA in a uses: line cannot:

Onboarded workflows enforce that every dependency is present in the lockfile, and that the locked commit is what executes on the runner.
Lockfiles are verified against forgery: the SHA must exist in the refs it claims to exist in.
Repository identity is recorded, and redirects or mismatches are blocked at runtime.
A locked action must have a branch containing the locked commit, which makes impostor-commit attacks considerably harder.

That last point deserves emphasis. A raw SHA in a workflow file offers no protection against an impostor commit — a commit pushed to a fork, which remains reachable through the upstream repository's object store and therefore resolves successfully. It looks entirely legitimate in a uses: line. The lockfile's reachability check is what closes that gap.

Do note that the extension is a technical preview and pre-1.0; the lockfile format and flags may change.

Why not raw SHA pinning

SHA pinning works, and it is better than a floating tag. But as a practice it has real drawbacks:

It destroys readability. uses: some-org/some-action@a1b2c3d… conveys nothing. The version has to be carried in a trailing comment, which nothing validates and which drifts from reality over time.
It is per-reference. Every workflow file must be maintained individually, rather than governed by one lockfile.
It offers no verification. As above, a SHA alone does not prove the commit belongs to the repository's history.

A lock file gives you the same guarantee with better ergonomics and meaningfully stronger verification. Immutable releases give you upstream guarantees that need no consumer-side maintenance at all.

Whichever you choose, combine it with dependency automation: let Renovate or Dependabot propose updates, configured with a cooldown period, so that a compromised release has time to be discovered before it reaches your pipelines.

Using the list sensibly

Very few actions will satisfy the whole of Tier 1. That is not a reason to discard the list; it is a reason to use it comparatively. When two actions solve the same problem, the checklist gives you a defensible basis for choosing one. When only one action solves the problem, it tells you which risks you are accepting, and where compensating controls are needed.

It also gives maintainers a roadmap. Much of Tier 1 is achievable by a single maintainer with focused effort — policy files, dependency automation with a cooldown, locked references, immutable releases, an attestation on the release. That is a materially different security posture from where most actions currently sit, and it does not require an organisation behind it.

Tooling that covers part of this

StepSecurity's GitHub Action Advisor implements a portion of the automatable side of this list — maintenance and release signals, dependency and pinning hygiene, policy files, advisories — and presents them as a per-action risk assessment. It also adds signals this checklist cannot easily produce on its own, such as observed runtime behaviour drawn from Harden-Runner telemetry across the organisations that use it.

Organisations that have granted the StepSecurity GitHub App access to their repositories get more from it: the advisor can then evaluate the actions actually referenced across the estate, rather than only the one you happen to be looking up.

It does not replace Tier 1 entirely — the AI-assisted and manual categories above remain yours to cover — but it is a reasonable way to get the queryable signals in one place instead of assembling them by hand.

Can one agent start another? Queueing sub-tasks in GitHub Agentic Workflows

Jesse Houwing — Fri, 04 Sep 2026 16:56:32 +0000

Consider a "dark factory" setup: a software factory with almost no people on the production line. A small number of people orchestrate the system, but agents do most of the work: planning, coding, reviewing, securing, deploying and reporting back. The goal is to define the factory itself as code.

For that to work, agents need to trigger on each other's work with minimal human interaction. One agent may create an issue, another may turn it into a plan, another may implement the code, another may review or secure it, and another may prepare the deployment. The hand-off can happen through an IDE, issues, pull requests, comments, workflow dispatches or other integration points, but the operating model assumes that most work is performed by agents.

That ambition quickly runs into GitHub's accountability model. GitHub has designed Copilot coding agent and Actions interactions so that certain work cannot silently cascade from one agent to another without a human or accountable identity in the loop. The visible symptom may look like an API problem:

POST /agents/repos/{owner}/{repo}/tasks
→ 404

The 404 is the product of deliberate interaction boundaries around identity, billing, legal accountability and human oversight. Understanding those boundaries determines which orchestration patterns are viable today and which remain unavailable.

The accountability rule that explains everything

From GitHub's docs on the agent tasks API:

The agent tasks API only supports user-to-server tokens. You can authenticate using a personal access token, an OAuth app token or a GitHub App user-to-server token.

Server-to-server tokens, such as GitHub App installation access tokens, are not supported.

The GITHUB_TOKEN your workflow receives automatically is an installation access token. It is server-to-server, and there is no user behind it — which is precisely its purpose.

So when a workflow attempts to start a cloud agent session with GITHUB_TOKEN, GitHub is not confused about permissions. It rejects the request because there is no accountable party.

That accountability matters for three concrete reasons:

Licensing and cost. A cloud agent session consumes AI Credits (AIC), and those are charged to the token owner. With no user behind the token, there is nobody to charge.
Abuse prevention. If a workflow token could spawn agents, any repository with a workflow could provision unlimited agent capacity without a single Copilot licence.
Responsibility. When an autonomous agent produces questionable changes, there must be an identity on the audit trail.

gh-aw states this plainly in its own source (pkg/workflow/github_token.go):

// Note: The default GITHUB_TOKEN is NOT included as a fallback because it does not have
// permission to create agent sessions, assign issues to bots, or add bots as reviewers.

The distinction that matters: sub-agent vs. new agent session

This is the misconception at the root of the problem, and it is worth stating explicitly.

The two options may sound similar, but they run in different places, use different identities and carry different accountability boundaries:

	Sub-agent	Cloud agent session
Where it runs	Same runner, same job	GitHub-hosted, separate lifecycle
Trigger	In-process, same run	`POST /agents/.../tasks`
Auth	Inherits the run's context	Needs a user token
Who's accountable	Whoever/whatever triggered the workflow	The token owner
Cost model	Part of the run's inference	AI Credits charged to the token owner

A sub-agent is comparable to a function call. A cloud agent session is comparable to delegating work to an external party.

Which raises the question that resolves the whole problem: why is a separate agent session needed at all? Why can the workflow's own sub-agent not implement the code?

It can. The agent already running in your workflow has a checked-out repository, a filesystem, and a shell. It can write the code, run the tests, and open the pull request. Spawning a second agent to perform work the first agent is already positioned to do adds indirection — and it is precisely that indirection which requires the token you do not have.

Pattern 1: Do the work in-run (start here)

This is the pattern that works today, requires no PAT for the coding path, and is most likely what you actually need.

---
on:
  issues:
    types: [labeled]
permissions:
  contents: read
  issues: read
  copilot-requests: write # org-billed inference, no PAT
engine: copilot
safe-outputs:
  create-pull-request:
    draft: true
  add-comment:
---

# Implement the labelled issue

Only proceed if the triggering issue carries the label `agentic:code-needed`.

1. Read the issue body and any linked design notes.
2. Implement the change directly in the working tree.
3. Run the test suite. If tests fail, fix and re-run — do not open a PR with red tests.
4. Open a **draft** pull request describing what you changed and why.
5. Comment on the originating issue linking the PR.

The agent edits files on the runner. gh-aw's safe-output layer takes the resulting diff, creates the branch, and opens the pull request through a controlled, permission-scoped code path — the agent itself never holds a write token.

One constraint to be aware of: a pull request opened with GITHUB_TOKEN produces a restricted event. From the Actions events reference:

When a pull request is created or updated by a workflow using GITHUB_TOKEN, pull_request events [...] create workflow runs that require approval. [...] With the exception of workflow_dispatch and repository_dispatch, other GITHUB_TOKEN-triggered events do not create workflow runs at all.

So your CI may wait for manual approval. This is the same accountability principle in a different form. If you need CI to run automatically, use repository_dispatch, or configure gh-aw's CI-trigger token with a PAT.

Pattern 2: Two-phase with a gate

If you genuinely want plan-then-implement as separate runs, split them across an artifact GitHub can observe — an issue, a label, a comment.

One practical shape is:

A schedule, issue or other trigger starts a planner workflow.
The planner workflow creates or updates an issue and applies a label such as agentic:code-needed.
A gate is applied, either by a human or by a separate approval signal such as agentic:approved.
The builder workflow starts on the approved label or dispatch event.
The builder workflow's own agent or sub-agent implements the change and runs tests.
The workflow opens a pull request using create-pull-request.

# planner.md
safe-outputs:
  create-issue:
    labels: [agentic:code-needed]
    max: 3


# builder.md
on:
  issues:
    types: [labeled]

Same caveat: an issue created by GITHUB_TOKEN won't trigger the issues.labeled run. Options:

Have a human apply the approval label — often exactly the gate you want anyway.
Use repository_dispatch from the planner.
Give the planner's safe output a PAT so the created issue carries a user identity.

Note that the third option reintroduces the human implicitly: the PAT owner is the accountable party, having pre-authorised the work rather than approving each instance.

Pattern 3: Spawning a cloud agent session (supported, with a PAT)

If you require a genuinely independent agent session, gh-aw supports it. The requirement is an explicit user identity.

safe-outputs:
  create-agent-session:
    base: main
    max: 1 # default 1, maximum 10
    github-token: ${{ secrets.GH_AW_AGENT_TOKEN }}

Or, for existing issues:

safe-outputs:
  assign-to-agent:
    target: triggering
    github-token: ${{ secrets.GH_AW_AGENT_TOKEN }}

The token requirements, per gh-aw's Copilot Cloud Agent reference:

Fine-grained PAT — repository permissions:

Read: metadata
Read & write: actions , contents , issues , pull requests

Classic PAT — the repo scope.

gh aw secrets set GH_AW_AGENT_TOKEN --value "<your-pat>"

GH_AW_AGENT_TOKEN is a magic secret — gh-aw looks for it by name, so you don't reference it in frontmatter. The resolution chain the compiler emits is GH_AW_AGENT_TOKEN || GH_AW_GITHUB_TOKEN || GITHUB_TOKEN, and that last fallback is the one that 404s.

A GitHub App is not an alternative here. gh-aw's documentation is explicit: "The Copilot assignment API requires a Personal Access Token [...] GitHub App installation tokens are rejected regardless of permissions." An App is server-to-server, so the same constraint applies.

This pattern works, but consider what it implies. A PAT is a standing delegation of a specific person's identity, Copilot entitlement, and AI Credit balance. The "no human in the loop" claim becomes "one person pre-authorised everything, indefinitely, until the token expires." That is a legitimate architecture, but it should not be mistaken for having removed the human.

Pattern 4: Fan-out inside a single run

For parallelism without spawning sessions, keep the work inside the run. gh-aw caps every safe output:

safe-outputs:
  create-issue:
    max: 5
  create-agent-session:
    max: 3 # each still needs the PAT

Your workflow agent can analyse, decide, and emit up to max outputs from a single run. The bound is deliberate: an agent able to spawn agents without limit is a fork bomb with a cost attached.

Summary

Supported today


Agent writes code and opens a PR in one run	✅ No PAT needed for the code path
Sub-agents / custom agents inside a run	✅ Same runner, same job
Multi-workflow chains via issues/labels/comments	✅ Subject to the `GITHUB_TOKEN` event rule
Spawning cloud agent sessions	✅ With a user PAT
Assigning Copilot to issues/PRs	✅ With a user PAT
Cross-repo agent sessions	✅ Explicit `target-repo` / `allowed-repos`
Org-billed inference, zero PATs	✅ `permissions: copilot-requests: write`

Not supported


Cloud agent session with plain `GITHUB_TOKEN`	❌ Server-to-server rejected by design
GitHub App installation token for agent tasks or assignment	❌ Same reason
Org billing as a substitute for user identity	❌ Covers inference , not spawning
Unbounded self-replicating agents	❌ `max` caps, hard ceiling of 10
Silent workflow chain reactions on `GITHUB_TOKEN` events	❌ Only `workflow_dispatch` / `repository_dispatch`
`target-repo: "*"` on `create-agent-session`	❌ Explicit repos only

Inference vs. spawning: a distinction worth being precise about

gh-aw does offer a genuine "no PAT required" path, and it is easily over-interpreted.

permissions: copilot-requests: write lets the Actions token pay for Copilot inference, with the AI Credits charged against your organisation's Copilot tenant rather than any individual. No COPILOT_GITHUB_TOKEN, no PAT. It requires the organisation policy Settings → Copilot → Policies → Copilot CLI → "Allow use of Copilot CLI billed to the organization".

This is a significant convenience. It also applies only to inference — the model calls made by the agent running inside your workflow.

copilot-requests: write → the agent in your workflow can THINK, AIC billed to the org ✅
copilot-requests: write → the agent in your workflow can SPAWN another agent ❌

Spawning still requires a user-to-server token, and the resulting session's AI Credits are charged to that token's owner , not to the organisation tenant. I expect this to relax eventually in an enterprise context, where cost can roll up to an enterprise-level identity that is itself an accountable principal — but I am not aware of any preview flag for it today, and the API documentation gives no indication of one.

What to tell your agent

Copilot performs poorly on this topic at present, largely for reasons outside its control. The surface changed recently (gh-aw renamed create-agent-task to create-agent-session in 0.4.0), there is very little public example code, and models fill the gaps with confident invention — such as recommending a support ticket for a feature flag that does not exist.

Two measures help considerably:

Clone github/gh-aw and add it to your workspace. Then instruct the agent: "When in doubt about how agentic workflows function, read the actual implementation instead of making assumptions." (This warrants a separate article — it is the most effective technique I know for fast-moving dependencies.)
Put the constraint in your instructions file. Something like:

<!-- .github/copilot-instructions.md -->
## Agentic workflow constraints

- `GITHUB_TOKEN` CANNOT create Copilot cloud agent sessions. The agent tasks API
  requires a user-to-server token. Do not suggest support tickets or feature flags.
- Prefer implementing code in-run with a sub-agent + `create-pull-request` over
  spawning a new agent session.
- Only use `create-agent-session` / `assign-to-agent` when `GH_AW_AGENT_TOKEN`
  is configured, and say so explicitly.
- Run `gh aw compile` before committing any workflow change.

Conclusion

The instinct to "queue a sub-task" by spawning a new agent is inherited from human organisations, where delegation is how parallelism is achieved. In an agentic workflow it is usually an anti-pattern: you already have an agent running on a runner, with the repository checked out and a shell available. Use it.

Use a cloud agent session when you genuinely need what it uniquely provides — an independent lifecycle, work that outlives your run, and its own review surface. Then treat the PAT as a requirement rather than an obstacle. It is GitHub's answer to the question of who authorised the work, and that question deserves an answer.

References

Ground your GitHub Copilot agent in a local clone instead of letting it guess

Jesse Houwing — Fri, 04 Sep 2026 15:34:13 +0000

A long debugging session lost to GitHub Agentic Workflows. Not because the tool was broken, but because the agent kept inventing how it worked. The model was not being unreasonable. It was being asked an impossible question.

github/gh-aw renamed a core safe output from create-agent-task to create-agent-session in version 0.4.0. There are perhaps a few dozen real-world examples of the tool in public. A significant portion of what exists online describes the previous shape. An LLM asked about a topic with thin, contradictory, recently changed training data will not say "I don't know" — it will interpolate, and it will do so fluently.

The remedy is straightforward: stop asking it to recall, and give it something to read.

The technique

cd ~/src
git clone https://github.com/github/gh-aw.git

Then in VS Code: File → Add Folder to Workspace… → pick the clone.

Then, in your Copilot session:

I've added the github/gh-aw repo to this workspace at ~/src/gh-aw. That's the actual implementation and documentation for GitHub Agentic Workflows. When you're unsure how something works, read the source there instead of assuming.

That is the entire technique. The agent now has grep access to ground truth.

I use this regularly — for debugging VS Code extensions, for comparing a framework's actual behaviour against its documented behaviour, and for any dependency that moves faster than model training cycles. It consumes a considerable number of tokens, and it is highly effective.

Why this is preferable to the alternatives

Source	Strength	Limitation
Model memory	Fast and free	Often stale on fast-moving projects, and can be confidently wrong
Generic web search	Reasonably current	Finds blog posts, old documentation and partial examples; usually not the actual implementation
GitHub MCP code search	Searches the source repository directly, so it is much better grounded than generic web search	Slower than local grep and may consume more tokens
Local clone	Exact code you depend on; greppable, cite-able and testable	Requires setup and can be token-intensive if the agent reads broadly

Documentation describes intent. Source code describes behaviour. When the two disagree — and in a fast-moving repository they will — you want your agent reading the artefact that actually runs. GitHub MCP code search is a useful alternative when you do not have a local clone: it searches the repository directly, so it is better grounded than generic web search, but it is slower than local grep and may consume more tokens.

A worked example

While researching the agent-session problem, I compared gh-aw's documentation against its code. Two discrepancies emerged immediately.

Discrepancy one — cross-repository support.

The safe-outputs specification says:

Type: create_agent_session — Cross-Repository Support: No (same repository only)

The reference doc, and the actual Go source, say otherwise:

// pkg/workflow/create_agent_session.go
type CreateAgentSessionConfig struct {
    BaseSafeOutputConfig `yaml:",inline"`
    Base string `yaml:"base,omitempty"`
    TargetRepoSlug string `yaml:"target-repo,omitempty"` // ← cross-repo
    AllowedRepos []string `yaml:"allowed-repos,omitempty"` // ← cross-repo
}

An agent relying on the specification document would have reported that cross-repository use was impossible. The struct definition settles the question immediately.

Discrepancy two — the API version header.

GitHub's public docs show:

-H "X-GitHub-Api-Version: 2022-11-28"

gh-aw's implementation sends:

// actions/setup/js/create_agent_session.cjs
const response = await githubClient.request("POST /agents/repos/{owner}/{repo}/tasks", {
  owner: repoParts.owner,
  repo: repoParts.repo,
  prompt: taskDescription,
  base_ref: baseBranch,
  headers: { "X-GitHub-Api-Version": "2026-03-10" },
});

If you were debugging a raw curl request against a preview API, that difference is decisive — and no amount of careful prompting would have surfaced it.

The discrepancy that mattered most. The token resolution chain is not documented in prose anywhere, but the source comment sets it out in full:

/**
 * Token precedence:
 * 1. config["github-token"] — per-handler PAT configured in the workflow frontmatter
 * 2. GH_AW_AGENT_SESSION_TOKEN — agent token injected by the compiler
 * (evaluates to: GH_AW_AGENT_TOKEN || GH_AW_GITHUB_TOKEN || GITHUB_TOKEN)
 * 3. global github — step-level token (fallback)
 */

That is the answer: the silent fallback to GITHUB_TOKEN, and the precise reason a correctly written workflow returns 404 with no useful error message. A targeted grep resolved what a long prompting session could not.

Bake it into your instructions

The clone only helps if the agent actually reaches for it. Make that automatic:

<!-- .github/copilot-instructions.md -->

## Grounding sources

The `github/gh-aw` repository is available in this workspace at `../gh-aw`.
It contains the implementation and documentation for GitHub Agentic Workflows.

- When you hit a snag, or are unsure how agentic workflows behave, **read the
  implementation** rather than making assumptions.
- Prefer, in order: `pkg/workflow/**` (Go compiler) →
  `actions/setup/js/**` (runtime handlers) → `docs/src/content/docs/**` →
  `CHANGELOG.md`.
- When docs and code disagree, **the code wins** — and say so in your answer.
- Cite file paths and line numbers for any claim about behaviour.
- Always run `gh aw compile` before committing workflow changes.

The citation requirement is more valuable than it first appears. It serves two purposes: it gives you a quick way to verify a claim, and it makes fabrication considerably harder for the model. An answer citing pkg/workflow/create_agent_session.go:12 is much easier to check. An answer without a citation cannot be checked directly.

The complementary half: GitHub MCP for what a clone cannot provide

A clone gives you code. It does not give you discussion. Issues, pull requests, discussions, and release notes — the layer that tells you whether something is a known problem — live on GitHub.

Enable the GitHub MCP server and your agent can query that too:

"Search gh-aw issues for create-agent-session 404"
"What changed in gh-aw between the version I have and main?"
"Are there open PRs touching the agent session token chain?"

This is where the undocumented reality surfaces: a maintainer explaining in an issue comment why something behaves as it does, a pull request that changed semantics recently, or a discussion thread describing the workaround most people use.

Choosing between a clone and MCP

Question	Preferred source
"How does X actually work?"	Clone — read the source. If unavailable, GitHub MCP code search
"What are the valid config fields?"	Clone — read the struct/schema. If unavailable, GitHub MCP code search
"Why does this fail?"	Clone first, then MCP for known issues
"Is this a known bug?"	MCP — search issues
"What changed recently?"	Clone `CHANGELOG.md` + `git log`, then MCP for PRs
"How do others use this?"	MCP — code search across GitHub
"Is a fix coming?"	MCP — open PRs and milestones

You can skip the clone and point custom instructions at the repository URL, letting the agent search the repository through GitHub MCP. This is still much better grounded than generic web search, because the agent is searching the actual source repository. It is slower than a local clone, requires more remote calls, and may consume more tokens. I clone by default and use MCP for the discussion layer and for repository code search when cloning is not practical.

Practical notes

Match the version you actually run. A clone of main describes the future. If you're on a release, check out the matching tag:

git -C ~/src/gh-aw fetch --tags
git -C ~/src/gh-aw checkout v0.4.0

Otherwise you risk implementing a feature that is not yet present in your version, which is a particularly frustrating class of bug.

Refresh it deliberately. Run git pull before a debugging session, and note the SHA you grounded against so that it remains clear later what "the code says" referred to.

Scope the search. Left unbounded, an agent will search across a repository of ten thousand files. Direct it:

Look only in pkg/workflow/ and actions/setup/js/ for how the token is resolved.

Budget for the cost. This is not inexpensive; you are paying tokens to read someone else's codebase. It remains considerably cheaper than a long debugging session based on an invented diagnosis.

Treat fetched content as untrusted input. Issue bodies, pull request descriptions, and README files can be influenced by third parties in public repositories. If your agent reads content that begins issuing it instructions, that is prompt injection rather than documentation. Worth being aware of even if you never encounter it.

When it's worth it

Grounding is worthwhile when:

The dependency changed recently
Public examples are scarce
Behaviour is subtle — auth chains, token precedence, compilation order
Docs and reality have visibly diverged
You're debugging something that "should work"

Skip it for stable, well-documented libraries with many public examples. There is no need to clone React to establish what useEffect does.

The heuristic I apply: if the model sounds confident but the result does not work, it is guessing. That is the point at which to clone.

Conclusion

A great deal of effort goes into prompt engineering — better phrasing, better structure, better framing. That effort is worthwhile, but a larger improvement is available before any of it: give the model the actual source of truth and instruct it to read.

Most agent failures I investigate are not reasoning failures. They are retrieval failures. The model reasoned correctly over information that was wrong.

You would not ask a new team member to implement against a library from memory. You would point them at the repository. Apply the same approach to your agent.

git clone <the-thing-you-depend-on>
# File → Add Folder to Workspace

Then instruct it to read.

Quick-start checklist

✅ Clone the dependency repo locally
✅ Check out the tag matching your installed version
✅ Add it to your VS Code workspace
✅ Add a "grounding sources" section to .github/copilot-instructions.md
✅ Include "when in doubt, read the implementation" as an explicit instruction
✅ Require file:line citations in answers
✅ Enable the GitHub MCP server for issues, PRs, and discussions
✅ Refresh the clone before serious debugging sessions

Configure Azure Pipelines app in ghe.com

Jesse Houwing — Fri, 19 Jun 2026 13:50:47 +0000

I recently helped a client migrate from Azure DevOps to GitHub Enterprise Managed Users with Data Residency (ghe.com for short). And as part of the migration we rewired the exiting Azure Pipelines from the old repository located on Azure Repos to the new one on GitHub.

With GitHub Enterprise this is quite easy, create a new service connection in Azure DevOps and click the magic authorize button to install the app into GitHub automatically. Unfortunately no such luck in ghe.com, even though there is a special service connection type:

Select the GitHub Enterprise Cloud with data residency service connection type to link Azure Pipelines with ghe.com

The screen mentions: "GitHub Organization should have azure pipelines app already installed", but there is no documentation anywhere on how to actually do that.

Unfortunately, this already expects the Azure Pipelines app to be installed in the target GitHub organization

And unfortunately, it turned out we weren't the first to hit this snag. And the error message is not very helpful either:

The error message you receive is "Could not complete OAuth Authorization; verify you have enough permissions to perform this operation"

I reached out to my contacts in GitHub and Azure DevOps to see if they knew what to do, but unfortunately, even they did not know.

I decided to play around some more and stumbled upon this well hidden url based on the url of the Azure Boards app by replacing boards with pipeline

https://{enterprise-slug}.ghe.com/apps/external-app/azure-boards
https://{enterprise-slug}.ghe.com/apps/external-app/azure-pipelines

And lo and behold, from this location you can install the Azure Pipelines app:

By navigating directly to the hidden Azure Pipelines GitHub App installation page you can install the app

Click Install to and select your GitHub organization, after which the configuration of the service connection will work without a hitch:

Select the GitHub Organization you want to link to your Azure DevOps Project.

Select the repos you want to enable for Azure Pipelines:

Select which repos in GitHub should be available to Azure Pipelines

After which you are redirected to Azure DevOps to complete the installation:

Select the correct Azure DevOps organization and project

If you now go back into the settings of your Azure DevOps Project, you should be able to create the new service connection:

Create the new service connection, which should work without any issues.

Click Authorize after which you'll be able to Save the new service connection.

To rewire a pipeline, navigate to the pipeline you want to rewire and click Edit :

Edit your existing Azure Pipelines Definitions to rewire them to GitHub

Click the ⁞ button and then select Triggers:

Use the triggers editor to navigate to the classic Pipelines editor

This brings you to the classic pipelines editor on which you can navigate to the YAML tab:

In the classic Pipelines editor, navigate to the YAML tab to change the source location

Select the Get Souces element in the pipeline

Select the Get Sources step to rewire the pipeline to GitHub

And rewire the pipeline to GitHub by selecting the GitHub tile and picking the Service Connection we previously created:

After selecting GitHub, change the service connection, repository and default branch

Auto-assign GitHub Copilot AI Credit budget to users

Jesse Houwing — Fri, 29 May 2026 13:26:23 +0000

The deadline for GitHub Copilot Usage Based Billing is approaching rapidly. And today I've been busy setting up automation to assign each user their own bucket of Copilot AIC budget.

On June 1st, all organizations should get access to the User Based Budgets and Universal User Budgets features in GitHub, allowing you to setup a a single universal budget for all your users as well as individual overrides which can either be higher or lower.

Setting up universal budget

To setup a universal budget, go to the enterprise's Billing & Licensing panel, select Budgets and Alerts in the left menu an create a new budget of type "Bundled AI Credits budget":

Note: the UI shows "premium request" until June 1st. Budgets should automatically transfer to "AI Credits".

Select the Bundled Premium Requests (before June 1st)/AI Credits budget (after June 1st)

Then set a user level budget, but do not select any specific user:

Select the User budget scope, but do not select a user

Finally select the default budget for all user without an individual budget override:

Set the Budget amount for all users without an individual budget

Setting an individual user budget

Setting an individual user budget follows the same flow, but in step 2 you need to select a specific user to which the budget applies:

To set an individual user's budget, select a specific user in step 2

Unfortunately you can't select a team or multiple users in this step, so each user needs their own individual budget. As you can imagine, this is a lot of work if you want to have specific control over different sets of users.

Automating the problem away

Of course my employer is one of those places that wants control over specific groups of users. And because I'm a lazy administrator by nature, I'd like to be able to leverage our existing approval infrastructure as much as possible. Most approvals in our organization are handled by EntraID group memberships and there are existing approval workflows in Entra to allow people to request and approve access to specific groups.

GitHub, unfortunately, doesn't allow linking a user budget to a group or a team, so there needs to be a mapping from group membership to budgets somewhere. And I ended up putting that in a GitHub Actions workflow of course.

The workflows is quite simple:

It pulls all copilot users from the assigned seats API.
It pulls all users from EntraID
It pulls all group memberships for the copilot users from EntraId
It pulls all user level budgets from GitHub
It compares the membership to the budget setup and either creates, updates or deletes the budget in GitHub.

And it sets the universal budget based on a preconfigured value.

All of this runs in a single GitHub Actions workflow an executes a single PowerShell script.

I won't share the full script here in the blog post, but I will share a few of the basic building blocks so you can implement your own logic based on your own criteria:

# Get existing Copilot budgets from the GitHub enterprise billing API.
function Get-ExistingBudgets {
    param(
        [string]$Enterprise,
        [string]$ProductSku
    )

    $allBudgets = @()
    $page = 1
    do 
        # Does not follow standard github paging, so need to manually page through the results
        $response = gh api "/enterprises/$Enterprise/settings/billing/budgets?page=$page" | ConvertFrom-Json
        $allBudgets += $response.budgets
        $page++
    } while ($response.has_next_page)

    return $allBudgets
}

# Create a new individual per-user budget.
function New-IndividualBudget {
    param(
        [string]$Enterprise,
        [string]$GitHubLogin,
        [decimal]$Amount,
        [string]$ProductSku
    )

    $body = @{
        budget_amount = $Amount
        budget_scope = "user"
        user = $GitHubLogin
        budget_type = "BundlePricing"
        budget_product_sku = $ProductSku
        prevent_further_usage = $true
        budget_alerting = @{
            will_alert = $false
            alert_recipients = @()
        }
    }

    Write-Output "Creating individual budget of `$$Amount for user $GitHubLogin"
    $_ = ($body | ConvertTo-Json -Depth 5) | gh api --method POST /enterprises/$Enterprise/settings/billing/budgets --input -
}

# Update an existing budget amount.
function Update-Budget {
    param(
        [string]$Enterprise,
        [string]$BudgetId,
        [decimal]$Amount,
        [string]$ProductSku
    )

    $body = @{
        budget_amount = $Amount
        budget_product_sku = $ProductSku
        prevent_further_usage = $true
    }

    Write-Output "Updating budget $BudgetId to `$$Amount"
    $_ = ($body | ConvertTo-Json -Depth 5) | gh api --method PATCH /enterprises/$Enterprise/settings/billing/budgets/$BudgetId --input -
}

# Delete a budget.
function Remove-Budget {
    param(
        [string]$Enterprise,
        [string]$BudgetId
    )

    Write-Output "Deleting budget $BudgetId"
    $_ = gh api --method DELETE /enterprises/$Enterprise/settings/billing/budgets/$BudgetId
}

# Ensure the universal budget exists and has the correct amount.
function Update-UniversalBudget {
    param(
        [string]$Enterprise,
        [string]$ProductSku,
        [decimal]$TargetAmount,
        $ExistingBudgets
    )

    $universalBudget = $ExistingBudgets | Where-Object { $_.budget_scope -eq "multi_user_customer" -and $_.budget_product_sku -eq $ProductSku }

    if ($universalBudget) {
        if ($universalBudget.budget_amount -ne $TargetAmount) {
            Write-Output "Updating universal budget from `$$($universalBudget.budget_amount) to `$$TargetAmount"
            Update-Budget -Enterprise $Enterprise -BudgetId $universalBudget.id -Amount $TargetAmount -ProductSku $ProductSku
        }
        else {
            Write-Output "Universal budget already set to `$$TargetAmount"
        }
    }
    else {
        Write-Output "Creating universal budget of `$$TargetAmount"
        $body = @{
            budget_amount = $TargetAmount
            budget_scope = "multi_user_customer"
            budget_type = "BundlePricing"
            budget_product_sku = $ProductSku
            prevent_further_usage = $true
            budget_alerting = @{
                will_alert = $false
                alert_recipients = @()
            }
        }
        $_ = ($body | ConvertTo-Json -Depth 5) | gh api --method POST /enterprises/$Enterprise/settings/billing/budgets --input -
    }
}

Because of the change from per-user premium request budget to AI credits and the additionally included credits during the promotional period from June 1st to August 31st I've added a few helpers:

# Determine the correct budget_product_sku based on the current date.
# Before June 1 2026: premium_requests
# From June 1 2026 onwards: ai_credits
function Get-BudgetProductSku {
    param(
        [datetime]$AsOf = [datetime]::UtcNow
    )
    $cutoverDate = [datetime]::new(2026, 6, 1, 0, 0, 0, [System.DateTimeKind]::Utc)
    if ($AsOf -lt $cutoverDate) {
        return "premium_requests"
    }
    return "ai_credits"
}

# Determine the universal budget amount based on the current date.
# Until May 31 2026: $0 (no included credits)
# June 1 - August 31 2026: $70
# September 1 2026 onwards: $39
function Get-UniversalBudgetAmount {
    param(
        [datetime]$AsOf = [datetime]::UtcNow
    )
    $june1 = [datetime]::new(2026, 6, 1, 0, 0, 0, [System.DateTimeKind]::Utc)
    $sept1 = [datetime]::new(2026, 9, 1, 0, 0, 0, [System.DateTimeKind]::Utc)

    if ($AsOf -lt $june1) {
        return [decimal]0.00
    }
    elseif ($AsOf -lt $sept1) {
        return [decimal]70.00
    }
    else {
        return [decimal]39.00
    }
}

Add your own logic to link a budget to a specific user, then call this from a simple workflow:

name: Assign Copilot Budgets

on:
  workflow_dispatch:
  schedule:
    - cron: "15 4 * * *"

jobs:
  update:
    permissions: 
      contents: read
      id-token: write

    runs-on: ubuntu-latest

    steps:      
    - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v4
      with:
        fetch-depth: 0

    - name: Azure CLI Login
      uses: azure/login@532459ea530d8321f2fb9bb10d1e0bcf23869a43 #v3.0.0
      with:
        client-id: ${{ secrets.AZURE_CLIENT_ID }}
        tenant-id: ${{ secrets.AZURE_TENANT_ID }}
        allow-no-subscriptions: true

    - name: Assign Copilot Budgets
      run: |
         .\assign-copilot-budget.ps1
      env:
        GH_TOKEN: ${{ secrets.GH_TOKEN }}
      shell: pwsh

You'll need to use a Classic PAT token with at least the following scopes:

manage_billing:enterprise
manage_billing:copilot
read:enterprise
read:user
user:email

Measuring the Value of AI

Jesse Houwing — Tue, 26 May 2026 13:16:04 +0000

We've deployed it everywhere. Now what?

The honeymoon is over. After months of aggressive rollouts, enterprises across the globe have given their employees access to AI coding assistants, copilots, and chat interfaces. The credit card bills are arriving. And now the inevitable question lands on someone's desk: What are we getting for this?

Organizations are scrambling to measure the return on their AI investment. They're building dashboards tracking token consumption, counting AI-assisted pull requests, and monitoring how many lines of code were generated versus written by hand. Some are going further — tracking tokens that led to PRs that led to production deployments, as if the pipeline from keystrokes to customer value were a simple conveyor belt.

It isn't. And the uncomfortable truth is that this measurement problem isn't new at all.

This isn't a new problem

Long before AI assistants existed, businesses struggled to quantify the value of knowledge work. Agile coaches spent decades watching organizations try — and fail — to measure the output of Scrum teams. Lines of code, velocity points, features shipped: none of these correlated reliably with business outcomes.

The problem is structural. As Barber (2008) demonstrated in research on value chain measurement, the further you move from tangible, physical processes toward intangible value creation, the harder measurement becomes. Software development, creative work, and strategic thinking exist almost entirely in that intangible space. Baruch Lev's seminal work Intangibles: Management, Measurement, and Reporting (2000) — cited over 6,000 times — established that organizations systematically fail to account for intangible value creation, precisely because it resists the neat quantification that accounting systems demand.

AI hasn't created a new measurement problem. It has amplified an existing one and made it impossible to ignore.

The Traps We Keep Falling Into

Goodhart's Law: When the Metric Becomes the Target

"When a measure becomes a target, it ceases to be a good measure." Charles Goodhart formulated this principle in 1975 about monetary policy, but Fritz (2016) demonstrated its devastating applicability to software development. The moment lines of code became a productivity metric, developers produced more lines of code — without producing more value.

We're walking into the same trap with AI. The moment "tokens consumed per PR" becomes a metric, teams will optimize for it. They'll either starve their AI of context to appear efficient, or feed it bloated prompts to appear productive, depending on which direction management rewards. Neither behavior correlates with value produced.

The McNamara Fallacy: Counting What's Countable

During the Vietnam War, Secretary of Defense Robert McNamara insisted on measuring success through body counts and territory captured — metrics that were quantifiable but ultimately meaningless for determining who was winning. The McNamara Fallacy, as van Nieuwenhuizen describes it, is "prioritizing quantitative metrics while ignoring equally vital qualitative factors that resist easy measurement."

Token usage is today's proxy. It tells you something is happening. It tells you resources are being consumed. It tells you absolutely nothing about whether the right things are being built, whether customers are happier, or whether your product is improving.

The fallacy is seductive because the alternative — measuring qualitative outcomes — is genuinely hard. So organizations default to measuring what their billing systems already capture.

Self-Serving Attribution Bias: We Believe Our Own Hype

There's another problem lurking beneath the measurement challenge. Research by Shepperd, Malone, and others (2008) on self-serving attribution bias shows that people systematically over-attribute positive outcomes to their own efforts and under-attribute them to external factors. Libby and Rennekamp (2012) demonstrated this in forecasting contexts: after a success, people attribute it internally, leading to overconfident predictions about future performance.

When a developer says "AI made me 10x more productive," they genuinely believe it. When a manager says "Our AI investment led to a 30% increase in shipping velocity," they genuinely believe it too. But these are uncontrolled observations colored by confirmation bias and the sunk-cost pressure of having championed the investment in the first place.

Even when organizations attempt to estimate value beforehand, those estimates are rarely tracked after the fact. The hypothesis that a feature would produce X value is tested at go/no-go time, then forgotten. Nobody goes back six months later to verify whether the value actually materialized. Estimates don't correlate with reality — they correlate with what was politically convenient to claim at funding time.

The Shifted Filter: From Pre-Build to Post-Build

Here's what AI has fundamentally changed about value creation in organizations: the capacity constraint used to be a natural quality filter.

In the pre-AI world, building anything required convincing a team or a product owner to spend their limited time on it. Plenty of good ideas died in this filter, but the most important ones usually survived. Scarcity imposed discipline.

Now that filter is evaporating. With AI-assisted development, the cost of building something has dropped dramatically. When capacity is infinite (or feels that way), everything gets built. Business stakeholders who used to have to fight for engineering time can now point agents at problems themselves.

The result? The filter moves from pre-build to post-build. Instead of asking "Should we build this?" teams must now ask "Now that we built this, should we ship it?" And crucially: most organizations have never developed the muscle for this.

A/B testing, eye tracking, funnel optimization — these post-build validation techniques exist in marketing departments and growth teams. But they're almost entirely absent from internal business software development. And you can tell just by using most enterprise software.

Imagine a product like SAP with infinite capacity to add even more schemas, tables, and configuration options. The complexity crisis of enterprise software isn't a resource problem — it's a judgment problem. AI removes the resource constraint without addressing the judgment gap.

The Polishing Problem

There's a related dysfunction: AI makes it trivially easy to generate an MVP-quality implementation of anything, but finishing things — truly finishing them — still requires human judgment, taste, and iteration.

I recently rewrote a piece of software I'd been maintaining for ten years. The AI-assisted rewrite took roughly a week. Making myself proud of the result took three more months of tweaks and refinements. The initial rewrite was the easy part. The value was in the polish.

Organizations optimizing for velocity — for things shipped, story points earned, PRs merged, tokens consumed, features delivered — are implicitly de-prioritizing this polish. Every half-baked, AI-generated feature that ships without refinement represents a small quality-of-life tax on every user who has to interact with it.

The Non-Technical Blind Spot

If measuring AI value for developers is hard, measuring it for non-technical departments can sometimes be even harder. At least with engineers, we have many crude proxies, like commits and deployments. With HR, finance, and operations teams paying per-seat for AI assistants, organizations typically have no visibility whatsoever into usage or impact or the ability to directly correlate AI usage to measurable productivity improvements.

Yet sometimes the value story for non-technical work is actually simpler. The use of AI can enable teams to quickly automate tasks they occasionally do manually. Of which the results can multiply if the "occasionally" turns into "frequently" suddenly. I recently supported a team who applies content updates on a website once in a while, and was face with a request to update 75 pages all at once. From word templates. Automating that away saves 2 weeks of manual labor, and also removes the pain of smaller batches in the future.

It's the ambient, non-targeted AI features that defy measurement. The auto-complete suggestions, the email drafts, the meeting summaries. Each one might save 30 seconds. Multiply by thousands of employees and the theoretical value is enormous. But theoretical value isn't the same as realized value, and nobody is checking whether those 30-second savings actually compound into better outcomes rather than simply expanding the time available for other low-value work.

Plus, sometimes the devil is in the details, not the summarized conversation of a 2 hour teams conversation. The quirked eyebrow of a participant as something is said. The person dropping from the call after a comment was made...

How to Measure Better

If pure output metrics are misleading and self-reported value is biased, what alternatives exist? Here are approaches grounded in measurement science:

1. Measure Outcomes, Not Outputs

Douglas Hubbard's framework from How to Measure Anything (2014) argues that most "immeasurable" things can be measured — but only if you define what you actually mean by value. Stop counting tokens, PRs, and features shipped. Instead, identify the business outcomes you care about (customer retention, time-to-resolution, revenue per feature, support ticket volume) and measure those over time. The AI investment is a confounding variable, not the thing you're measuring.

2. Use Controlled Comparisons

Rather than measuring before/after (which conflates AI impact with everything else that changed), run controlled experiments. Give some teams AI access for specific tasks and others not. Rotate. Compare outcomes, not effort. This is operationally inconvenient, which is exactly why almost nobody does it — and why everyone's "measurement" is actually just observational storytelling.

3. Track Value Hypothesis Closure Rates

For every feature or project that gets built, require a falsifiable value hypothesis before build starts. Then measure whether that hypothesis proved true after deployment. Track the ratio of hypotheses confirmed versus falsified. If AI is helping you ship more things, but the hypothesis hit rate is falling, you're producing more waste faster. That's negative ROI regardless of what the token dashboard says.

4. Measure What Didn't Get Built

One of the most important potential values of AI is compression — not building more things, but building the same things with fewer resources, freeing capacity for quality, exploration, or rest. If your team is shipping the same amount as before but with less overtime, less burnout, and fewer shortcuts, that's value. It just doesn't show up in velocity charts.

5. Adopt the Post-Build Filter

If AI is shifting your constraint from capacity to judgment, invest in judgment infrastructure. Feature flags, gradual rollouts, user research, usage analytics. The organizations that will get the most value from AI aren't the ones building the most — they're the ones with the tightest feedback loop between shipping and learning whether they should have shipped.

6. Accept Irreducible Uncertainty

Some value is genuinely unmeasurable in the moment. How much value was produced by allowing Microsoft Word to italicize text? Probably immense — if it never had basic formatting, we'd all be using something else. But no ROI calculation could have predicted that in 1985. Some infrastructure investments only reveal their value in retrospect, through the things they enabled that nobody foresaw. Not everything needs a dashboard. Some things need trust and taste.

The Honest Answer

Companies want a clean number. "Our AI investment produced €X in value." That number doesn't exist and it won't exist, for the same reason that organizations never successfully quantified the value of email, spreadsheets, or the internet itself. These are enabling technologies whose value is diffused across every process they touch, modified by how skillfully they're used, and contingent on decisions that haven't been made yet.

What you can measure is cost. What you can observe is whether your customers are happier, your employees are less burned out, your products are more refined, and your judgment about what to build is getting sharper.

The hardest pill to swallow: maybe that feature your AI-powered team just shipped in record time should never have been built in the first place. And no token dashboard in the world will tell you that.

References:

The why for Usage Based Billing for GitHub Copilot

Jesse Houwing — Mon, 11 May 2026 12:58:31 +0000

Last month GitHub announced Usage Based Billing for GitHub Copilot. This completely changes the way its costs are calculated. A lot has already been written about the impact and a lot is still unknown.

In this post I want to explore the reasons why this switch was inevitable and how it enables new scenarios to make Copilot perform better than it did before.

Quick recap of copilots current pricing model

GitHub Copilot is currently available in 5 different plans:

GitHub Copilot Free - severely rate limited free version with no option to buy additional requests.
GitHub Copilot Pro - Individual license with limited Premium Request Units.
GitHub Copilot Pro+ - Individual license with more extensive Premium Request Units.
GitHub Copilot for Business - License for organizations with centralized policies and limited Premium Request Units.
GitHub Copilot Enterprise - License for organizations with centralized policies and more extensive Premium Request Units.

Each of these licenses has a fixed monthly price. GitHub Copilot includes unlimited suggestions and a limited set of free models.

And about a year ago, GitHub introduced the concept of Premium Request Units to pay for more powerful models and and PRUs are the used to charge for specific features like the Cloud Coding Agent and Copilot Code Review.

When the Cloud Coding Agent was introduced, it could use more than 1 PRU per request. This made the cost of the agent unpredictable and GitHub soon changed the price of the Cloud Coding Agent to a single PRU per request.

When a user has exhausted their included PRU budget, additional PRUs are charged. Additional PRUs have a fixed cost.

Usages of PRUs

The primary use of PRUs are to charge the costs of requests to the users of GitHub Copilot. Each model is assigned a multiplier of 0, 1, up to 50 (highest so far) PRUs.

The model multiplier wasn't just used to charge the cost, it's an open secret that it was also used to divert users to other models to load balance them. Most people are more hesitant to use expensive models carelessly. The introduction of the "Auto" mode rests on this principle. Users get a 10% discount on PRUs if they let GitHub select their model. While most users seem to assume this will route them to the "best model for the task", this generally sends the request to the "model with the best availability".

Premium Request Units were also reduced for promotional purposes. When Claude Opus 4.7 was introduced, it's multiplier was temporarily set to 3x, later increased to 15x. This promotional use isn't just to get us "hooked" on the more powerful model features, it's also to allow GitHub to get crucial statistics on how these new models perform and to help them iron out bugs.

How GitHub pays for model usage

The model providers, Anthropic, Google, OpenAI and X, don't charge the cost of their models in Premium Request Units. They charge the cost in terms of Tokens. Input Tokens, Output tokens and Cached tokens at the highest level.

Given the amount GitHub uses, they have probably got a really good deal.

But GitHub needs to balance the cost for the Tokens it spends against what it can charge its users in Premium Request Units.

GitHub also balances the costs between Tokens and PRUs by limiting the maximum token window for models. For example, if you're using Claude Opus 4.6 through GitHub Copilot, the maximum token window is 200k tokens. If you use the same model directly through Anthropic you have a maximum of 1m tokens.

As Agent Mode and Cloud Coding Agent got more powerful, more tools appeared, the models are using more and more tokens for every request, reducing the available token window for your requests.

Other ways GitHub control costs

GitHub has a few other options to control the maximum number of tokens you can use for each Premium Request Unit they charge you.

Agent Mode - Agent Mode spends tokens. GitHub can control the token spend by forcing the agent to come to an early conclusion or to postpone work to a future session.

Cloud Coding Agent - The cloud coding agent spends tokens and action minutes. GitHub can control the token spend by forcing the agent to come to an early conclusion or to postpone work to a future session. The workflows running the Cloud Coding Agent runs on is limited to 2 hours.

Code Review - Similar to the Cloud Coding Agent, when using GitHub's Code Review feature, GitHub can control the token spend by limiting the total time and the total of tokens the feature can spend.

When using these features, GitHub can also control which model is used to perform these tasks.

Some features, like tool optimization in Visual Studio Code also limit the total number of tokens used for each session by default.

Recent creative uses

New features, like subagent and fleets have multiplied the number of tokens you can use in a single Agent request. Visual Studio Code extensions have allowed users to steer the agent without completing the Agent request.

These creative usages, some people might call them abuse, have allowed some people to use significantly more tokens costs than the PRU is worth. In some cases thousands time more.

And instead of figuring out how to charge its users in Premium Request Units in this new model, GitHub will charge us for the actual tokens instead.

So what changes

On June 1st 2026 GitHub will change the way we all pay for our AI usage.

Instead of the abstract thing, called Premium Request Units, acting as a proxy for the real cost of AI usage, we're going to be charged in tokens, the same unit in which GitHub is charged.

Unfortunately, the different model providers, each charge different prices for different kinds of tokens. Your usage of each model is translated from the price per token into AI Credits, the credits are deducted from the budget included in your account.

GitHub explains how the new pricing model works on the GitHub Blog.

The benefits

The introduction of tokens isn't meant to simply increase the cost of AI, but it is also expected to more equally spread the load across the available models as users will be more likely to try to find the most cost effective model to use for the work they're trying to accomplish, instead of using most powerful model available to them for all their tasks. This should result in better model availability and higher performance.

A key benefit of GitHub Copilot is that you get access to the models of Anthropic, Google and OpenAI through a single subscription. And now that we are charged for the actual usage of tokens, GitHub can start raising the tokens available for each session, making the models more comparable to subscribing directly through each model provider.

Additional changes for Business accounts

In addition to the new pricing, GitHub is introducing a number of welcome changes for business accounts (Copilot for Business and Copilot Enterprise).

Pooled usage - Under the current model, when a user has spent all of its budget, additional requests are automatically charged. Under the new model, the budget of all users is pooled together. So the unused budget from less frequent users can be consumed by more frequent users in the same organization. Only when the full pooled budget is used up, will additional usage be charged.

Individual additional budgets - Under the current model, additional charges can be controlled through budgets, these are pooled together for all users in the same cost center. In the new model it's possible to set a per-user budget as well as a universal budget for all users in addition to the already existing budget control options.

In addition to these changes, existing policy options to control costs are still available:

Enterprise, Organization and Cost Center Budgets - Existing budget options to limit the total spend at the Enterprise, Organization and Cost Center can still be configured to limit the total spend on AI costs.

Limit model availability - Enterprise and Organization level policies allow admins to control the available models. This can remove less effective and more expensive model options.

Limit feature availability - Enterprise and Organization level policies allow admins to control the available features. This can remove more expensive features like Code Review and Cloud Coding Agent for specific users.

Don't hamstring your users, educate them

But, ultimately, limiting the cost isn't the path forward. Optimizing the value delivered through AI is. Use the available limits to bridge the time it takes to educate your users.

Keep your examples in sync with your Action

Jesse Houwing — Mon, 02 Mar 2026 09:53:49 +0000

Last month I added a few more GitHub Actions to the Marketplace and while working on them found that it was easy to forget to update my examples as the implementation of the Action evolved.

So, what started as a small script to automatically validate the examples against the action.yaml quickly grew into its own action.

Introducing the Actions Example Checker

The Actions Example Checker is a simple action you can add to your repository. It will scan for all action.yaml files and extract the metadata. It will then search through all markdown files in your repo and extract all YAML script blocks:

```yaml
- uses: your/action@version
```

Any examples that reference your action will be checked against the schema.

Additionally, you can add a action.schema.yaml with extended metadata about each input, like: allowed values, datatype, regex. I've got a few more plans with this, but for now it's purely used to check the assigned values in your examples.

Sample workflow

In order to add the Actions example checker to your repo, add this workflow:

name: Validate Documentation

on:
  pull_request:
  push:
    branches: [main]
  workflow_dispatch:

permissions: {}

jobs:
  validate-docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
        with:
          sparse-checkout: |
            **/action.yaml
            **/action.yml
            **/action.schema.yaml
            **/action.schema.yml
            **/*.md
          sparse-checkout-cone-mode: false
      - uses: jessehouwing/actions-example-checker@ea0821955c7ecf2c98e5b184054ba31486914a7b # v0.0.4
        token: ${{ secrets.GITHUB_TOKEN }}

Example schemas

I've already added schema validation to most of my own actions, they may be able to help you setup your own:

Let me know if you find this useful!

Automatic versioning for GitHub Actions authors

Jesse Houwing — Sat, 07 Feb 2026 10:34:01 +0000

As a maintainer of 3 GitHub Actions, I've always struggled a bit with the versioning of them. GitHub's documentation provides guidance, but no easy way to verify you did everything right. Based on a quick scan of the actions on the marketplace I'm not alone, even some of GitHub's own actions have some inconsistencies when it comes to it.

This is why I cobbled together v1 of my Actions Semver Checker action a couple of years ago and it has served me well. But as I threw the action together in a short amount of time way back when, it contained a few bugs, was very hard to maintain and I always dreaded making changes to it, since I had fixed my own repos, so it was hard to test whether I broke anything or accidentally introduced unwanted side effects.

This is why I had Copilot Agent use the existing v1 implementation as a blueprint to build a new version. But with a lot of improvements. I gave Copilot the order to refactor the action with the following constraints:

Make sure the old implementation and the new one can be called side-by-side so it's easy to compare the differences.
Extract the validation rules into separate scripts so they're easier to maintain and test.
Provide manual remediation options wherever possible.
Implement an autofix functionality which tries to execute the manual remediation steps.
Add unit tests for everything.
Add integration tests for everything.
Try to remain backwards compatible with v1 as much as possible when it comes to the inputs.
I gave it links to all the relevant GitHub docs and blogs for:
- GitHub CLI release commands
- Releases REST API
- Tags REST API
- GitHub's GraphQL API
- Blogs discussing immutable releases
- GitHub Actions versioning

I used Claude Opus to generate a plan for this, since it's a pretty big effort. I went over the plan and made a few tweaks and then had the Copilot Coding Agent spend a good 45 minutes to do the work. And this resulted in my first vibe coded GitHub Action. And the result? It did mostly what it needed to do. It surfaced a couple of gaps in the initial plan and the request to keep both old and new code side-by-side caused some interesting interweaving between the 2 implementations.

But now that I had a test suite of 90+ tests, I felt much better changing the actual behavior. And after many round trips with the Copilot Coding Agent as well as a few hands-on sessions inside Visual Studio Code with Copilot, I am proud to release v2 of the GitHub Actions Semver Checker Action.

What does it do exactly?

Good question, the action is meant to be installed into the repository which hosts a GitHub action, like actions/checkout or jessehouwing/actions-semver-checker. It analyzes all branches, tags and releases and validates them based on a number of rules:

There should be a vX tag or branch (configurable) which points to the latest vX.y.z tag.
There should be a vX.Y tag or branch (configurable) which points to the latest vX.Y.z tag.
There should be a vx.y.z tag
There should be no vx.y.z branch
new - There should be a release for every vx.y.z tag
- new - That release should be immutable
- new - The release for the highest vx.y.z should be set to "latest"
new - There should be no release for vX and vX.Y tags
new - The action should be published to the marketplace
- new - The action.yml should have the correct metadata elements
- new - The latest version of the action should be published to the marketplace
new - Immutable releases should be turned on

And for almost all of these validations, I managed to build an autofix option which automatically corrects any issues found.

What's new

As you can see in the list above, v2 adds a lot of new validations and it also adds the autofix feature to automatically resolve most of the issues found. If the action is not able to fix the issue, manual remediation steps are logged to the workflows summary pages.

Autofixes available:

delete/update/create branches
delete/update/create tags
convert branch to tag
convert tag to branch
delete/update/create/publish releases
automatically set the latest version
republish releases to make them immutable

There are a few other improvements worth noting:

Fully configurable which suites of rules you want to run
You no longer need to check out the whole repo to run the action
Retry logic and handling of rate limits
Available as module on the PowerShell Gallery

There is one unfortunate change to v1 I could not work around

You should pass the ${{ secrets.GUTHUB_TOKEN }} to the token: input of the action
For some validations the token must have contents: write permissions (to read draft releases)
For autofix functionality the token must have contents: write permissions
To fix certain issues, the token needs permissions which cannot be granted to GitHub Actions, such as workflows: write. Use a GitHub App or Fine-grained Personal Access Token instead.

Recommended usage

I recommend you install the action into your action repository with its default settings, this will analyze your repo and will detect all rule violations.

Change the settings of the action to match your desired behavior:

floating-versions-use can be set to tags or branches depending on how you want to manage the vx and vx.y versions.
check-minor-version can be set to none, warning or error to validate the existence of vx.y versions.
check-releases can be set to none, warning or error to validate the existence of releases for vx.y.z tags.
check-release-immutability can be set to none, warning or error to validate whether releases are immutable.
ignore-preview-releases can be set to true or false to exclude preview releases when checking floating versions (vx an vx.y).
ignore-versions can exclude versions from validation completely. This can be used to ignore really old versions or versions that were accidentally made immutable and can no longer be fixed.

name: Check SemVer

on:
  push:
    branches: ['main']
    tags:
      - '*'
  workflow_dispatch:

permissions: {}

jobs:
  check-semver:
    permissions:
      contents: write

    concurrency:
      group: '${{ github.workflow }}'
      cancel-in-progress: true

    runs-on: ubuntu-latest

    steps:
      - uses: jessehouwing/actions-semver-checker@v2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          check-release-immutability: none

⚠️ be extra careful with the check-release-immutability setting prior to turning on autofix. Most automatic fixes can be easily undone using the git commandline, but once a release is made immutable, there is no way back.

Once your existing tags have been cleaned up and you understand the implications, turn on release immutability checks:

    steps:
      - uses: jessehouwing/actions-semver-checker@v2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          check-release-immutability: error

Use the action in read-only mode for a while, manually performing the suggested remediation steps.

When you're confident the action does what you want it to do, consider turning on autofix: true.

    steps:
      - uses: jessehouwing/actions-semver-checker@v2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          autofix: true

With autofix turned on, the only thing you need to do to release a new version and update all tags and releases, is to push a new tag to to create a new release with the correct name: vx.y.z. This will trigger autofix to do the rest on your behalf.

Improved Dependency Submission for GitHub Actions

Jesse Houwing — Sat, 27 Dec 2025 18:47:09 +0000

If you've read some of my earlier blogs about GitHub Actions Security, you may have followed my advice to pin your actions and configure Dependabot. Recently someone brought to my attention that you then Security Advisories for GitHub actions do not show up in the Dependency Graph and Dependabot won't create security updates.

I decided to solve that problem.

Background

When a security vulnerability is found and a security advisory is created, it will have the action and its affected versions as metadata. Versions , not SHAs.

When Automatic Dependency Submission submits the versions of actions used by a repository it records:

@v1.2.3 as 1.2.3
@v1 as 1.*.*
8e8c483db84b4bee98b60c0593521ed34d9990e8 as 8e8c483db84b4bee98b60c0593521ed34d9990e8

The Security Advisory Database can't match 8e8c483db84b4bee98b60c0593521ed34d9990e8 against the vulnerable version range and won't show the vulnerability in the dependency graph.

Another issue exists when actions are forked to an organization (a common technique to create a private actions marketplace). The dependency is recorded using the organization name that holds the fork, not the upstream.

The Security Advisory Database can't match myorg/actions_checkout against the vulnerable action actions/checkout and won't show the vulnerably in the dependency graph.

Impacted features

Without recording the exact version of each action used, a number of features won't work as you might expect:

Dependency Graph won't show which actions you use are vulnerable.
Dependabot Security Updates won't be created for workflows using vulnerable actions.
Dependency Review Action won't block pull requests introducing vulnerable actions into your workflows.

Solution

To solve the issue I created a new manual submission action which mimics the automatic submission, but adds a few extra features:

It resolves the SHA to the highest, most specific version and records that as an additional dependency.
It resolves a wildcard version to the current specific version and records that as an additional dependency.
It can be configured to look up the upstream of a forked action and records that as an additional dependency.

With these additional dependencies recorded, the Dependency Graph now contains all the metadata to determine whether the GitHub actions you depend on are vulnerable and all features should now work as expected.

How to

Changing how Actions' dependencies are reported, you simply add my jessehouwing/actions-dependency-submission action to your workflows (and optionally add actions/dependency-review-action:

name: Submit and validate dependencies of GitHub Actions

on:
  push:
    branches:
      - main
  pull_request:
  schedule:
    - cron: '33 4 * * *'

permissions: {}

jobs:
  submit-actions-dependencies:
    runs-on: ubuntu-latest
    permissions:
      # submitting dependencies requires 'contents: write'
      contents: write

    steps:
      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
      - uses: jessehouwing/actions-dependency-submission@e848a29fd84b874cce3e45ceb00619bc72dbeca3 # 1.0.2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
      - uses: actions/dependency-review-action@3c4e3dcb1aa7874d2c16be7d79418e9b7efd6261 # 4.8.2
        if: github.event_name == 'pull_request'
        with:
          retry-on-snapshot-warnings: true

You can test whether the action is working correctly by adding a vulnerable action to your repository:

name: Submit Dependencies
on:
  workflow_dispatch:

jobs:
  submit-dependencies:
    if: false
    runs-on: ubuntu-latest
    permissions: {}
    steps:
      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe # v4.1.2

This should show as a high vulnerability in the Dependency graph of the repository:

Dependency Graph showing actions/download-artifact@v4.1.2 as vulnerable.

And you'll also see the original report using the SHA:

Dependency Graph showing actions/download-artifact referenced by SHA.

You can find the action and additional documentation and examples here.

GitHub Billing - Let Organization and Repository admins set the Cost Center in GitHub

Jesse Houwing — Wed, 24 Sep 2025 08:48:31 +0000

We rely on GitHub Cost Centers to split the GitHub bill across the different entities that make up our company. What resource is tied to which cost center is managed in the GitHub Enterprise Settings and for now this can only be done be a user with Enterprise Owner or Billing Administrator permissions.

I'd been searching for a way to allow Organization Owners and Repository Administrators to set the Cost Center on a repository themselves without having to involve one of these highly privileged accounts. We found a solution using GitHub Actions and Repository Custom Properties.

Since my last blog post on Cost Centers, a few small improvements were released by GitHub:

There is now an API to associate a Repository with a Cost Center.
It's now possible to associate a Repository to a different Cost Center than the Organization that owns it.

These two improvements allow us to solve this problem.

Creating the Custom Property

GitHub has a feature called Custom Properties for repositories which allows you to link arbitrary data to a repo. You can define these properties at the Enterprise, Organization and Repository level.

Since our cost centers are defined at the Enterprise level, I chose to create this custom property at the same level. To add a new custom property navigate to your enterprise, select the Policies tab and then drill down into Repository and then Custom Properties. Choose New Property to create your new property:

Navigate to the Custom Properties section and click new property

Give the property a clear Name and Description. Since our list of cost centers is clearly defined, I've chosen the Type single-select to allow our users to pick a cost center from a predefined list.

Give the property a unique name and set the type to single select

Next, add the list of cost centers you want people to be able to select from as options. I've also included an option inherit which I'll explain later:

Add the required cost-center options.

And finally make the field required and set the default value to inherit.

Set the permission to allow users with properties permission to edit this field, make it required and set the default value.

Now when you create a new repository or navigate to the settings of an existing repository, you can set the cost-center custom field:

Users are required to select a cost center when creating a new repository.

And existing repositories will now show the custom property in the repository settings:

The reason for Inherit

In the above example we added a special value inherit , when this value is set, we don't actually set any cost center on the repository and this causes all costs generated by this repository to automatically flow to the cost center associated to the Organization. If the Organization isn't associated to a cost center, the costs will flow to the Enterprise.

Making it all work using GitHub Actions

Creating the Custom Property itself doesn't do anything by itself. In order to actually associate the Repository to the Cost Center, we still need to update the configuration in the GitHub Enterprise Billing & Licensing section:

Billing Administrators can associate Organizations, Repositories and Users to Cost Centers

Without additional automation each repository needs to be manually added to a cost center by editing the Cost Center and adding the Repository. This is a tedious manual process.

Instead, to update the Cost Center configuration I've created a GitHub Actions workflow that runs once a day and compares the Cost Center configuration against the values of the Custom Properties and performs updates to the Cost Center configuration if needed.

The basic workflow is quite simple:

Query the current Cost Center configuration
Query all Organizations in our GitHub Enterprise
Query all Repositories in each GitHub Organization
Query the Custom Properties for each Repository
Update the Cost Center configuration in case it doesn't match the Custom Property

The workflow relies on the GitHub CLI t0 query the data.

$costCenters = (invoke-gh -fromJson -- api /enterprises/$enterprise/settings/billing/cost-centers).costCenters

$orgs = get-allorganizations $enterprise | % { $_.login }

update-repocostcenters

The update-repocostcenters function checks which updates are needed:

function update-repocostcenters {
    Write-Output "Updating Costcenters for repositories in $enterprise." 

    foreach ($org in $orgs) {

        Write-Output "Processing organization $org"
        $repos = invoke-gh -slurp api /orgs/$org/repos --paginate --jq '.[] | { name: .name, full_name: .full_name }' | ConvertFrom-Json

        foreach ($repo in $repos) { 

            $customProperties = invoke-gh api /repos/$($repo.full_name)/properties/values --jq '.' | ConvertFrom-Json
            $repoCostCenterProperty = $customProperties | Where-Object { $_.property_name -eq "cost-center" } | Select-Object -ExpandProperty value -ErrorAction Ignore
            $repo | Add-Member -NotePropertyName cost_center -NotePropertyValue $repoCostCenterProperty -Force

            $currentCostCenter = $costCenters | ? { $_.resources | ? { $_.type -eq "Repo" -and $_.name -eq $repo.full_name } }

            if ( $repo.cost_center -eq "inherit" ) {
                $repo.cost_center = $null
            }

            $targetCostCenter = $null

            if ($repo.cost_center) {
                $targetCostCenter = $costCenters | ? { $_.name -eq $repo.cost_center }
                if (-not $targetCostCenter) {
                    Write-Warning "Costcenter for repository $($repo.full_name) not found."
                }
            }

            if ($null -eq $currentCostCenter) {
                if ($null -ne $targetCostCenter) {
                    Write-Output "Updating costcenter for repository $($repo.full_name) from unassigned to $($targetCostCenter.name)."
                    Update-CostCenterResources -Handles $repo.full_name -ResourceType "Repo" -Action "Add" -CostCenter $targetCostCenter -Enterprise $enterprise
                }
                else {
                    Write-Verbose "Repository $($repo.full_name) does not have a cost center assigned."
                }
            }
            else {
                if ($null -eq $targetCostCenter) {
                    Write-Verbose "Repository $($repo.full_name) does not have a cost center assigned."
                    Write-Output "Updating costcenter for repository $($repo.full_name) from $($currentCostCenter.name) to unassigned."
                    Update-CostCenterResources -Handles $repo.full_name -ResourceType "Repo" -Action "Delete" -CostCenter $currentCostCenter -Enterprise $enterprise
                }
                elseif ($currentCostCenter.id -ne $targetCostCenter.id) {
                    Write-Output "Updating costcenter for repository $($repo.full_name) from $($currentCostCenter.name) to $($targetCostCenter.name)."
                    Update-CostCenterResources -Handles $repo.full_name -ResourceType "Repo" -Action "Delete" -CostCenter $currentCostCenter -Enterprise $enterprise
                    Update-CostCenterResources -Handles $repo.full_name -ResourceType "Repo" -Action "Add" -CostCenter $targetCostCenter -Enterprise $enterprise
                }
            }
        }
    }
}

Unfortunately, you can't simply update the associated Cost Center in a single call, when changing the current value, an add plus delete is required.

I shared the Update-CostCenterResources function in a previous blogpost. GitHub added the ability to set Organizations and Repositories recently, so here is the updated version:

function Update-CostCenterResources {
    param(
        [Parameter(Mandatory = $true)]
        [string[]]$Handles,

        [ValidateSet('User', 'Repo', 'Org')]
        [string]$ResourceType = "User",

        [Parameter(Mandatory = $true)]
        [ValidateSet('Add', 'Delete')]
        [string]$Action,

        [Parameter(Mandatory = $true)]
        $CostCenter,

        [Parameter(Mandatory = $true)]
        [string]$Enterprise
    )

    switch ($Action) {
        'Add' {
            $method = 'POST'
            $Handles = $Handles | Where-Object { 
                $handle = $_
                return (($costCenter.resources | ? { $_.type -eq $ResourceType } | ? { $_.name -eq $handle }).Count -eq 0)
            }
        }
        'Delete' {
            $method = 'DELETE'
            $Handles = $Handles | Where-Object { 
                $handle = $_
                return (($costCenter.resources | ? { $_.type -eq $ResourceType } | ? { $_.name -eq $handle }).Count -gt 0)
            }
        }
    }

    # Call fails when processing too many users at once. Thus batching the calls...
    $count = 0
    do {
        $batch = $Handles | Select-Object -Skip $count -First 30
        $count += $batch.Count

        if ($batch.Count -gt 0) {
            switch ($ResourceType) {
                'User' { 
                    $body = @{
                        users = [string[]]$batch
                    }
                }
                'Org' { 
                    $body = @{
                        organizations = [string[]]$batch
                    }
                }
                'Repo' { 
                    $body = @{
                        repositories = [string[]]$batch
                    }
                }
            }


            $_ = ($body | ConvertTo-Json) | gh api --method $method /enterprises/$Enterprise/settings/billing/cost-centers/$($CostCenter.id)/resource --input -
        }
    } while ($batch.Count -gt 0)
}

All of this runs from a simple GitHub workflow which has access to a Personal Access token with the required permissions to access the Cost Center information as well as sufficient permissions to query all Repositories in all Organizations in the enterprise.

name: Set Costcenters

on:
  schedule:
    - cron: "5 4 * * *"

jobs:
  update:
    permissions: 
      contents: read

    runs-on: ubuntu-latest

    steps:      
    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
      with:
        fetch-depth: 0

    - name: Assign Costcenters
      run: |
         .\assign-costcenters.ps1
      env:
        GH_TOKEN: ${{ secrets.GH_TOKEN }}
      shell: pwsh

You can find the whole script here: