DEV Community: Jesse Houwing

Auto-assign GitHub Copilot AI Credit budget to users

Jesse Houwing — Fri, 29 May 2026 13:26:23 +0000

The deadline for GitHub Copilot Usage Based Billing is approaching rapidly. And today I've been busy setting up automation to assign each user their own bucket of Copilot AIC budget.

On June 1st, all organizations should get access to the User Based Budgets and Universal User Budgets features in GitHub, allowing you to setup a a single universal budget for all your users as well as individual overrides which can either be higher or lower.

Setting up universal budget

To setup a universal budget, go to the enterprise's Billing & Licensing panel, select Budgets and Alerts in the left menu an create a new budget of type "Bundled AI Credits budget":

Note: the UI shows "premium request" until June 1st. Budgets should automatically transfer to "AI Credits".

Select the Bundled Premium Requests (before June 1st)/AI Credits budget (after June 1st)

Then set a user level budget, but do not select any specific user:

Select the User budget scope, but do not select a user

Finally select the default budget for all user without an individual budget override:

Set the Budget amount for all users without an individual budget

Setting an individual user budget

Setting an individual user budget follows the same flow, but in step 2 you need to select a specific user to which the budget applies:

To set an individual user's budget, select a specific user in step 2

Unfortunately you can't select a team or multiple users in this step, so each user needs their own individual budget. As you can imagine, this is a lot of work if you want to have specific control over different sets of users.

Automating the problem away

Of course my employer is one of those places that wants control over specific groups of users. And because I'm a lazy administrator by nature, I'd like to be able to leverage our existing approval infrastructure as much as possible. Most approvals in our organization are handled by EntraID group memberships and there are existing approval workflows in Entra to allow people to request and approve access to specific groups.

GitHub, unfortunately, doesn't allow linking a user budget to a group or a team, so there needs to be a mapping from group membership to budgets somewhere. And I ended up putting that in a GitHub Actions workflow of course.

The workflows is quite simple:

It pulls all copilot users from the assigned seats API.
It pulls all users from EntraID
It pulls all group memberships for the copilot users from EntraId
It pulls all user level budgets from GitHub
It compares the membership to the budget setup and either creates, updates or deletes the budget in GitHub.

And it sets the universal budget based on a preconfigured value.

All of this runs in a single GitHub Actions workflow an executes a single PowerShell script.

I won't share the full script here in the blog post, but I will share a few of the basic building blocks so you can implement your own logic based on your own criteria:

# Get existing Copilot budgets from the GitHub enterprise billing API.
function Get-ExistingBudgets {
    param(
        [string]$Enterprise,
        [string]$ProductSku
    )

    $allBudgets = @()
    $page = 1
    do 
        # Does not follow standard github paging, so need to manually page through the results
        $response = gh api "/enterprises/$Enterprise/settings/billing/budgets?page=$page" | ConvertFrom-Json
        $allBudgets += $response.budgets
        $page++
    } while ($response.has_next_page)

    return $allBudgets
}

# Create a new individual per-user budget.
function New-IndividualBudget {
    param(
        [string]$Enterprise,
        [string]$GitHubLogin,
        [decimal]$Amount,
        [string]$ProductSku
    )

    $body = @{
        budget_amount = $Amount
        budget_scope = "user"
        user = $GitHubLogin
        budget_type = "BundlePricing"
        budget_product_sku = $ProductSku
        prevent_further_usage = $true
        budget_alerting = @{
            will_alert = $false
            alert_recipients = @()
        }
    }

    Write-Output "Creating individual budget of `$$Amount for user $GitHubLogin"
    $_ = ($body | ConvertTo-Json -Depth 5) | gh api --method POST /enterprises/$Enterprise/settings/billing/budgets --input -
}

# Update an existing budget amount.
function Update-Budget {
    param(
        [string]$Enterprise,
        [string]$BudgetId,
        [decimal]$Amount,
        [string]$ProductSku
    )

    $body = @{
        budget_amount = $Amount
        budget_product_sku = $ProductSku
        prevent_further_usage = $true
    }

    Write-Output "Updating budget $BudgetId to `$$Amount"
    $_ = ($body | ConvertTo-Json -Depth 5) | gh api --method PATCH /enterprises/$Enterprise/settings/billing/budgets/$BudgetId --input -
}

# Delete a budget.
function Remove-Budget {
    param(
        [string]$Enterprise,
        [string]$BudgetId
    )

    Write-Output "Deleting budget $BudgetId"
    $_ = gh api --method DELETE /enterprises/$Enterprise/settings/billing/budgets/$BudgetId
}

# Ensure the universal budget exists and has the correct amount.
function Update-UniversalBudget {
    param(
        [string]$Enterprise,
        [string]$ProductSku,
        [decimal]$TargetAmount,
        $ExistingBudgets
    )

    $universalBudget = $ExistingBudgets | Where-Object { $_.budget_scope -eq "multi_user_customer" -and $_.budget_product_sku -eq $ProductSku }

    if ($universalBudget) {
        if ($universalBudget.budget_amount -ne $TargetAmount) {
            Write-Output "Updating universal budget from `$$($universalBudget.budget_amount) to `$$TargetAmount"
            Update-Budget -Enterprise $Enterprise -BudgetId $universalBudget.id -Amount $TargetAmount -ProductSku $ProductSku
        }
        else {
            Write-Output "Universal budget already set to `$$TargetAmount"
        }
    }
    else {
        Write-Output "Creating universal budget of `$$TargetAmount"
        $body = @{
            budget_amount = $TargetAmount
            budget_scope = "multi_user_customer"
            budget_type = "BundlePricing"
            budget_product_sku = $ProductSku
            prevent_further_usage = $true
            budget_alerting = @{
                will_alert = $false
                alert_recipients = @()
            }
        }
        $_ = ($body | ConvertTo-Json -Depth 5) | gh api --method POST /enterprises/$Enterprise/settings/billing/budgets --input -
    }
}

Because of the change from per-user premium request budget to AI credits and the additionally included credits during the promotional period from June 1st to August 31st I've added a few helpers:

# Determine the correct budget_product_sku based on the current date.
# Before June 1 2026: premium_requests
# From June 1 2026 onwards: ai_credits
function Get-BudgetProductSku {
    param(
        [datetime]$AsOf = [datetime]::UtcNow
    )
    $cutoverDate = [datetime]::new(2026, 6, 1, 0, 0, 0, [System.DateTimeKind]::Utc)
    if ($AsOf -lt $cutoverDate) {
        return "premium_requests"
    }
    return "ai_credits"
}

# Determine the universal budget amount based on the current date.
# Until May 31 2026: $0 (no included credits)
# June 1 - August 31 2026: $70
# September 1 2026 onwards: $39
function Get-UniversalBudgetAmount {
    param(
        [datetime]$AsOf = [datetime]::UtcNow
    )
    $june1 = [datetime]::new(2026, 6, 1, 0, 0, 0, [System.DateTimeKind]::Utc)
    $sept1 = [datetime]::new(2026, 9, 1, 0, 0, 0, [System.DateTimeKind]::Utc)

    if ($AsOf -lt $june1) {
        return [decimal]0.00
    }
    elseif ($AsOf -lt $sept1) {
        return [decimal]70.00
    }
    else {
        return [decimal]39.00
    }
}

Add your own logic to link a budget to a specific user, then call this from a simple workflow:

name: Assign Copilot Budgets

on:
  workflow_dispatch:
  schedule:
    - cron: "15 4 * * *"

jobs:
  update:
    permissions: 
      contents: read
      id-token: write

    runs-on: ubuntu-latest

    steps:      
    - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v4
      with:
        fetch-depth: 0

    - name: Azure CLI Login
      uses: azure/login@532459ea530d8321f2fb9bb10d1e0bcf23869a43 #v3.0.0
      with:
        client-id: ${{ secrets.AZURE_CLIENT_ID }}
        tenant-id: ${{ secrets.AZURE_TENANT_ID }}
        allow-no-subscriptions: true

    - name: Assign Copilot Budgets
      run: |
         .\assign-copilot-budget.ps1
      env:
        GH_TOKEN: ${{ secrets.GH_TOKEN }}
      shell: pwsh

You'll need to use a Classic PAT token with at least the following scopes:

manage_billing:enterprise
manage_billing:copilot
read:enterprise
read:user
user:email

Measuring the Value of AI

Jesse Houwing — Tue, 26 May 2026 13:16:04 +0000

We've deployed it everywhere. Now what?

The honeymoon is over. After months of aggressive rollouts, enterprises across the globe have given their employees access to AI coding assistants, copilots, and chat interfaces. The credit card bills are arriving. And now the inevitable question lands on someone's desk: What are we getting for this?

Organizations are scrambling to measure the return on their AI investment. They're building dashboards tracking token consumption, counting AI-assisted pull requests, and monitoring how many lines of code were generated versus written by hand. Some are going further — tracking tokens that led to PRs that led to production deployments, as if the pipeline from keystrokes to customer value were a simple conveyor belt.

It isn't. And the uncomfortable truth is that this measurement problem isn't new at all.

This isn't a new problem

Long before AI assistants existed, businesses struggled to quantify the value of knowledge work. Agile coaches spent decades watching organizations try — and fail — to measure the output of Scrum teams. Lines of code, velocity points, features shipped: none of these correlated reliably with business outcomes.

The problem is structural. As Barber (2008) demonstrated in research on value chain measurement, the further you move from tangible, physical processes toward intangible value creation, the harder measurement becomes. Software development, creative work, and strategic thinking exist almost entirely in that intangible space. Baruch Lev's seminal work Intangibles: Management, Measurement, and Reporting (2000) — cited over 6,000 times — established that organizations systematically fail to account for intangible value creation, precisely because it resists the neat quantification that accounting systems demand.

AI hasn't created a new measurement problem. It has amplified an existing one and made it impossible to ignore.

The Traps We Keep Falling Into

Goodhart's Law: When the Metric Becomes the Target

"When a measure becomes a target, it ceases to be a good measure." Charles Goodhart formulated this principle in 1975 about monetary policy, but Fritz (2016) demonstrated its devastating applicability to software development. The moment lines of code became a productivity metric, developers produced more lines of code — without producing more value.

We're walking into the same trap with AI. The moment "tokens consumed per PR" becomes a metric, teams will optimize for it. They'll either starve their AI of context to appear efficient, or feed it bloated prompts to appear productive, depending on which direction management rewards. Neither behavior correlates with value produced.

The McNamara Fallacy: Counting What's Countable

During the Vietnam War, Secretary of Defense Robert McNamara insisted on measuring success through body counts and territory captured — metrics that were quantifiable but ultimately meaningless for determining who was winning. The McNamara Fallacy, as van Nieuwenhuizen describes it, is "prioritizing quantitative metrics while ignoring equally vital qualitative factors that resist easy measurement."

Token usage is today's proxy. It tells you something is happening. It tells you resources are being consumed. It tells you absolutely nothing about whether the right things are being built, whether customers are happier, or whether your product is improving.

The fallacy is seductive because the alternative — measuring qualitative outcomes — is genuinely hard. So organizations default to measuring what their billing systems already capture.

Self-Serving Attribution Bias: We Believe Our Own Hype

There's another problem lurking beneath the measurement challenge. Research by Shepperd, Malone, and others (2008) on self-serving attribution bias shows that people systematically over-attribute positive outcomes to their own efforts and under-attribute them to external factors. Libby and Rennekamp (2012) demonstrated this in forecasting contexts: after a success, people attribute it internally, leading to overconfident predictions about future performance.

When a developer says "AI made me 10x more productive," they genuinely believe it. When a manager says "Our AI investment led to a 30% increase in shipping velocity," they genuinely believe it too. But these are uncontrolled observations colored by confirmation bias and the sunk-cost pressure of having championed the investment in the first place.

Even when organizations attempt to estimate value beforehand, those estimates are rarely tracked after the fact. The hypothesis that a feature would produce X value is tested at go/no-go time, then forgotten. Nobody goes back six months later to verify whether the value actually materialized. Estimates don't correlate with reality — they correlate with what was politically convenient to claim at funding time.

The Shifted Filter: From Pre-Build to Post-Build

Here's what AI has fundamentally changed about value creation in organizations: the capacity constraint used to be a natural quality filter.

In the pre-AI world, building anything required convincing a team or a product owner to spend their limited time on it. Plenty of good ideas died in this filter, but the most important ones usually survived. Scarcity imposed discipline.

Now that filter is evaporating. With AI-assisted development, the cost of building something has dropped dramatically. When capacity is infinite (or feels that way), everything gets built. Business stakeholders who used to have to fight for engineering time can now point agents at problems themselves.

The result? The filter moves from pre-build to post-build. Instead of asking "Should we build this?" teams must now ask "Now that we built this, should we ship it?" And crucially: most organizations have never developed the muscle for this.

A/B testing, eye tracking, funnel optimization — these post-build validation techniques exist in marketing departments and growth teams. But they're almost entirely absent from internal business software development. And you can tell just by using most enterprise software.

Imagine a product like SAP with infinite capacity to add even more schemas, tables, and configuration options. The complexity crisis of enterprise software isn't a resource problem — it's a judgment problem. AI removes the resource constraint without addressing the judgment gap.

The Polishing Problem

There's a related dysfunction: AI makes it trivially easy to generate an MVP-quality implementation of anything, but finishing things — truly finishing them — still requires human judgment, taste, and iteration.

I recently rewrote a piece of software I'd been maintaining for ten years. The AI-assisted rewrite took roughly a week. Making myself proud of the result took three more months of tweaks and refinements. The initial rewrite was the easy part. The value was in the polish.

Organizations optimizing for velocity — for things shipped, story points earned, PRs merged, tokens consumed, features delivered — are implicitly de-prioritizing this polish. Every half-baked, AI-generated feature that ships without refinement represents a small quality-of-life tax on every user who has to interact with it.

The Non-Technical Blind Spot

If measuring AI value for developers is hard, measuring it for non-technical departments can sometimes be even harder. At least with engineers, we have many crude proxies, like commits and deployments. With HR, finance, and operations teams paying per-seat for AI assistants, organizations typically have no visibility whatsoever into usage or impact or the ability to directly correlate AI usage to measurable productivity improvements.

Yet sometimes the value story for non-technical work is actually simpler. The use of AI can enable teams to quickly automate tasks they occasionally do manually. Of which the results can multiply if the "occasionally" turns into "frequently" suddenly. I recently supported a team who applies content updates on a website once in a while, and was face with a request to update 75 pages all at once. From word templates. Automating that away saves 2 weeks of manual labor, and also removes the pain of smaller batches in the future.

It's the ambient, non-targeted AI features that defy measurement. The auto-complete suggestions, the email drafts, the meeting summaries. Each one might save 30 seconds. Multiply by thousands of employees and the theoretical value is enormous. But theoretical value isn't the same as realized value, and nobody is checking whether those 30-second savings actually compound into better outcomes rather than simply expanding the time available for other low-value work.

Plus, sometimes the devil is in the details, not the summarized conversation of a 2 hour teams conversation. The quirked eyebrow of a participant as something is said. The person dropping from the call after a comment was made...

How to Measure Better

If pure output metrics are misleading and self-reported value is biased, what alternatives exist? Here are approaches grounded in measurement science:

1. Measure Outcomes, Not Outputs

Douglas Hubbard's framework from How to Measure Anything (2014) argues that most "immeasurable" things can be measured — but only if you define what you actually mean by value. Stop counting tokens, PRs, and features shipped. Instead, identify the business outcomes you care about (customer retention, time-to-resolution, revenue per feature, support ticket volume) and measure those over time. The AI investment is a confounding variable, not the thing you're measuring.

2. Use Controlled Comparisons

Rather than measuring before/after (which conflates AI impact with everything else that changed), run controlled experiments. Give some teams AI access for specific tasks and others not. Rotate. Compare outcomes, not effort. This is operationally inconvenient, which is exactly why almost nobody does it — and why everyone's "measurement" is actually just observational storytelling.

3. Track Value Hypothesis Closure Rates

For every feature or project that gets built, require a falsifiable value hypothesis before build starts. Then measure whether that hypothesis proved true after deployment. Track the ratio of hypotheses confirmed versus falsified. If AI is helping you ship more things, but the hypothesis hit rate is falling, you're producing more waste faster. That's negative ROI regardless of what the token dashboard says.

4. Measure What Didn't Get Built

One of the most important potential values of AI is compression — not building more things, but building the same things with fewer resources, freeing capacity for quality, exploration, or rest. If your team is shipping the same amount as before but with less overtime, less burnout, and fewer shortcuts, that's value. It just doesn't show up in velocity charts.

5. Adopt the Post-Build Filter

If AI is shifting your constraint from capacity to judgment, invest in judgment infrastructure. Feature flags, gradual rollouts, user research, usage analytics. The organizations that will get the most value from AI aren't the ones building the most — they're the ones with the tightest feedback loop between shipping and learning whether they should have shipped.

6. Accept Irreducible Uncertainty

Some value is genuinely unmeasurable in the moment. How much value was produced by allowing Microsoft Word to italicize text? Probably immense — if it never had basic formatting, we'd all be using something else. But no ROI calculation could have predicted that in 1985. Some infrastructure investments only reveal their value in retrospect, through the things they enabled that nobody foresaw. Not everything needs a dashboard. Some things need trust and taste.

The Honest Answer

Companies want a clean number. "Our AI investment produced €X in value." That number doesn't exist and it won't exist, for the same reason that organizations never successfully quantified the value of email, spreadsheets, or the internet itself. These are enabling technologies whose value is diffused across every process they touch, modified by how skillfully they're used, and contingent on decisions that haven't been made yet.

What you can measure is cost. What you can observe is whether your customers are happier, your employees are less burned out, your products are more refined, and your judgment about what to build is getting sharper.

The hardest pill to swallow: maybe that feature your AI-powered team just shipped in record time should never have been built in the first place. And no token dashboard in the world will tell you that.

References:

The why for Usage Based Billing for GitHub Copilot

Jesse Houwing — Mon, 11 May 2026 12:58:31 +0000

Last month GitHub announced Usage Based Billing for GitHub Copilot. This completely changes the way its costs are calculated. A lot has already been written about the impact and a lot is still unknown.

In this post I want to explore the reasons why this switch was inevitable and how it enables new scenarios to make Copilot perform better than it did before.

Quick recap of copilots current pricing model

GitHub Copilot is currently available in 5 different plans:

GitHub Copilot Free - severely rate limited free version with no option to buy additional requests.
GitHub Copilot Pro - Individual license with limited Premium Request Units.
GitHub Copilot Pro+ - Individual license with more extensive Premium Request Units.
GitHub Copilot for Business - License for organizations with centralized policies and limited Premium Request Units.
GitHub Copilot Enterprise - License for organizations with centralized policies and more extensive Premium Request Units.

Each of these licenses has a fixed monthly price. GitHub Copilot includes unlimited suggestions and a limited set of free models.

And about a year ago, GitHub introduced the concept of Premium Request Units to pay for more powerful models and and PRUs are the used to charge for specific features like the Cloud Coding Agent and Copilot Code Review.

When the Cloud Coding Agent was introduced, it could use more than 1 PRU per request. This made the cost of the agent unpredictable and GitHub soon changed the price of the Cloud Coding Agent to a single PRU per request.

When a user has exhausted their included PRU budget, additional PRUs are charged. Additional PRUs have a fixed cost.

Usages of PRUs

The primary use of PRUs are to charge the costs of requests to the users of GitHub Copilot. Each model is assigned a multiplier of 0, 1, up to 50 (highest so far) PRUs.

The model multiplier wasn't just used to charge the cost, it's an open secret that it was also used to divert users to other models to load balance them. Most people are more hesitant to use expensive models carelessly. The introduction of the "Auto" mode rests on this principle. Users get a 10% discount on PRUs if they let GitHub select their model. While most users seem to assume this will route them to the "best model for the task", this generally sends the request to the "model with the best availability".

Premium Request Units were also reduced for promotional purposes. When Claude Opus 4.7 was introduced, it's multiplier was temporarily set to 3x, later increased to 15x. This promotional use isn't just to get us "hooked" on the more powerful model features, it's also to allow GitHub to get crucial statistics on how these new models perform and to help them iron out bugs.

How GitHub pays for model usage

The model providers, Anthropic, Google, OpenAI and X, don't charge the cost of their models in Premium Request Units. They charge the cost in terms of Tokens. Input Tokens, Output tokens and Cached tokens at the highest level.

Given the amount GitHub uses, they have probably got a really good deal.

But GitHub needs to balance the cost for the Tokens it spends against what it can charge its users in Premium Request Units.

GitHub also balances the costs between Tokens and PRUs by limiting the maximum token window for models. For example, if you're using Claude Opus 4.6 through GitHub Copilot, the maximum token window is 200k tokens. If you use the same model directly through Anthropic you have a maximum of 1m tokens.

As Agent Mode and Cloud Coding Agent got more powerful, more tools appeared, the models are using more and more tokens for every request, reducing the available token window for your requests.

Other ways GitHub control costs

GitHub has a few other options to control the maximum number of tokens you can use for each Premium Request Unit they charge you.

Agent Mode - Agent Mode spends tokens. GitHub can control the token spend by forcing the agent to come to an early conclusion or to postpone work to a future session.

Cloud Coding Agent - The cloud coding agent spends tokens and action minutes. GitHub can control the token spend by forcing the agent to come to an early conclusion or to postpone work to a future session. The workflows running the Cloud Coding Agent runs on is limited to 2 hours.

Code Review - Similar to the Cloud Coding Agent, when using GitHub's Code Review feature, GitHub can control the token spend by limiting the total time and the total of tokens the feature can spend.

When using these features, GitHub can also control which model is used to perform these tasks.

Some features, like tool optimization in Visual Studio Code also limit the total number of tokens used for each session by default.

Recent creative uses

New features, like subagent and fleets have multiplied the number of tokens you can use in a single Agent request. Visual Studio Code extensions have allowed users to steer the agent without completing the Agent request.

These creative usages, some people might call them abuse, have allowed some people to use significantly more tokens costs than the PRU is worth. In some cases thousands time more.

And instead of figuring out how to charge its users in Premium Request Units in this new model, GitHub will charge us for the actual tokens instead.

So what changes

On June 1st 2026 GitHub will change the way we all pay for our AI usage.

Instead of the abstract thing, called Premium Request Units, acting as a proxy for the real cost of AI usage, we're going to be charged in tokens, the same unit in which GitHub is charged.

Unfortunately, the different model providers, each charge different prices for different kinds of tokens. Your usage of each model is translated from the price per token into AI Credits, the credits are deducted from the budget included in your account.

GitHub explains how the new pricing model works on the GitHub Blog.

The benefits

The introduction of tokens isn't meant to simply increase the cost of AI, but it is also expected to more equally spread the load across the available models as users will be more likely to try to find the most cost effective model to use for the work they're trying to accomplish, instead of using most powerful model available to them for all their tasks. This should result in better model availability and higher performance.

A key benefit of GitHub Copilot is that you get access to the models of Anthropic, Google and OpenAI through a single subscription. And now that we are charged for the actual usage of tokens, GitHub can start raising the tokens available for each session, making the models more comparable to subscribing directly through each model provider.

Additional changes for Business accounts

In addition to the new pricing, GitHub is introducing a number of welcome changes for business accounts (Copilot for Business and Copilot Enterprise).

Pooled usage - Under the current model, when a user has spent all of its budget, additional requests are automatically charged. Under the new model, the budget of all users is pooled together. So the unused budget from less frequent users can be consumed by more frequent users in the same organization. Only when the full pooled budget is used up, will additional usage be charged.

Individual additional budgets - Under the current model, additional charges can be controlled through budgets, these are pooled together for all users in the same cost center. In the new model it's possible to set a per-user budget as well as a universal budget for all users in addition to the already existing budget control options.

In addition to these changes, existing policy options to control costs are still available:

Enterprise, Organization and Cost Center Budgets - Existing budget options to limit the total spend at the Enterprise, Organization and Cost Center can still be configured to limit the total spend on AI costs.

Limit model availability - Enterprise and Organization level policies allow admins to control the available models. This can remove less effective and more expensive model options.

Limit feature availability - Enterprise and Organization level policies allow admins to control the available features. This can remove more expensive features like Code Review and Cloud Coding Agent for specific users.

Don't hamstring your users, educate them

But, ultimately, limiting the cost isn't the path forward. Optimizing the value delivered through AI is. Use the available limits to bridge the time it takes to educate your users.

Automatic versioning for GitHub Actions authors

Jesse Houwing — Sat, 07 Feb 2026 10:34:01 +0000

As a maintainer of 3 GitHub Actions, I've always struggled a bit with the versioning of them. GitHub's documentation provides guidance, but no easy way to verify you did everything right. Based on a quick scan of the actions on the marketplace I'm not alone, even some of GitHub's own actions have some inconsistencies when it comes to it.

This is why I cobbled together v1 of my Actions Semver Checker action a couple of years ago and it has served me well. But as I threw the action together in a short amount of time way back when, it contained a few bugs, was very hard to maintain and I always dreaded making changes to it, since I had fixed my own repos, so it was hard to test whether I broke anything or accidentally introduced unwanted side effects.

This is why I had Copilot Agent use the existing v1 implementation as a blueprint to build a new version. But with a lot of improvements. I gave Copilot the order to refactor the action with the following constraints:

Make sure the old implementation and the new one can be called side-by-side so it's easy to compare the differences.
Extract the validation rules into separate scripts so they're easier to maintain and test.
Provide manual remediation options wherever possible.
Implement an autofix functionality which tries to execute the manual remediation steps.
Add unit tests for everything.
Add integration tests for everything.
Try to remain backwards compatible with v1 as much as possible when it comes to the inputs.
I gave it links to all the relevant GitHub docs and blogs for:
- GitHub CLI release commands
- Releases REST API
- Tags REST API
- GitHub's GraphQL API
- Blogs discussing immutable releases
- GitHub Actions versioning

I used Claude Opus to generate a plan for this, since it's a pretty big effort. I went over the plan and made a few tweaks and then had the Copilot Coding Agent spend a good 45 minutes to do the work. And this resulted in my first vibe coded GitHub Action. And the result? It did mostly what it needed to do. It surfaced a couple of gaps in the initial plan and the request to keep both old and new code side-by-side caused some interesting interweaving between the 2 implementations.

But now that I had a test suite of 90+ tests, I felt much better changing the actual behavior. And after many round trips with the Copilot Coding Agent as well as a few hands-on sessions inside Visual Studio Code with Copilot, I am proud to release v2 of the GitHub Actions Semver Checker Action.

What does it do exactly?

Good question, the action is meant to be installed into the repository which hosts a GitHub action, like actions/checkout or jessehouwing/actions-semver-checker. It analyzes all branches, tags and releases and validates them based on a number of rules:

There should be a vX tag or branch (configurable) which points to the latest vX.y.z tag.
There should be a vX.Y tag or branch (configurable) which points to the latest vX.Y.z tag.
There should be a vx.y.z tag
There should be no vx.y.z branch
new - There should be a release for every vx.y.z tag
- new - That release should be immutable
- new - The release for the highest vx.y.z should be set to "latest"
new - There should be no release for vX and vX.Y tags
new - The action should be published to the marketplace
- new - The action.yml should have the correct metadata elements
- new - The latest version of the action should be published to the marketplace
new - Immutable releases should be turned on

And for almost all of these validations, I managed to build an autofix option which automatically corrects any issues found.

What's new

As you can see in the list above, v2 adds a lot of new validations and it also adds the autofix feature to automatically resolve most of the issues found. If the action is not able to fix the issue, manual remediation steps are logged to the workflows summary pages.

Autofixes available:

delete/update/create branches
delete/update/create tags
convert branch to tag
convert tag to branch
delete/update/create/publish releases
automatically set the latest version
republish releases to make them immutable

There are a few other improvements worth noting:

Fully configurable which suites of rules you want to run
You no longer need to check out the whole repo to run the action
Retry logic and handling of rate limits
Available as module on the PowerShell Gallery

There is one unfortunate change to v1 I could not work around

You should pass the ${{ secrets.GUTHUB_TOKEN }} to the token: input of the action
For some validations the token must have contents: write permissions (to read draft releases)
For autofix functionality the token must have contents: write permissions
To fix certain issues, the token needs permissions which cannot be granted to GitHub Actions, such as workflows: write. Use a GitHub App or Fine-grained Personal Access Token instead.

Recommended usage

I recommend you install the action into your action repository with its default settings, this will analyze your repo and will detect all rule violations.

Change the settings of the action to match your desired behavior:

floating-versions-use can be set to tags or branches depending on how you want to manage the vx and vx.y versions.
check-minor-version can be set to none, warning or error to validate the existence of vx.y versions.
check-releases can be set to none, warning or error to validate the existence of releases for vx.y.z tags.
check-release-immutability can be set to none, warning or error to validate whether releases are immutable.
ignore-preview-releases can be set to true or false to exclude preview releases when checking floating versions (vx an vx.y).
ignore-versions can exclude versions from validation completely. This can be used to ignore really old versions or versions that were accidentally made immutable and can no longer be fixed.

name: Check SemVer

on:
  push:
    branches: ['main']
    tags:
      - '*'
  workflow_dispatch:

permissions: {}

jobs:
  check-semver:
    permissions:
      contents: write

    concurrency:
      group: '${{ github.workflow }}'
      cancel-in-progress: true

    runs-on: ubuntu-latest

    steps:
      - uses: jessehouwing/actions-semver-checker@v2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          check-release-immutability: none

⚠️ be extra careful with the check-release-immutability setting prior to turning on autofix. Most automatic fixes can be easily undone using the git commandline, but once a release is made immutable, there is no way back.

Once your existing tags have been cleaned up and you understand the implications, turn on release immutability checks:

    steps:
      - uses: jessehouwing/actions-semver-checker@v2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          check-release-immutability: error

Use the action in read-only mode for a while, manually performing the suggested remediation steps.

When you're confident the action does what you want it to do, consider turning on autofix: true.

    steps:
      - uses: jessehouwing/actions-semver-checker@v2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          autofix: true

With autofix turned on, the only thing you need to do to release a new version and update all tags and releases, is to push a new tag to to create a new release with the correct name: vx.y.z. This will trigger autofix to do the rest on your behalf.

Improved Dependency Submission for GitHub Actions

Jesse Houwing — Sat, 27 Dec 2025 18:47:09 +0000

If you've read some of my earlier blogs about GitHub Actions Security, you may have followed my advice to pin your actions and configure Dependabot. Recently someone brought to my attention that you then Security Advisories for GitHub actions do not show up in the Dependency Graph and Dependabot won't create security updates.

I decided to solve that problem.

Background

When a security vulnerability is found and a security advisory is created, it will have the action and its affected versions as metadata. Versions , not SHAs.

When Automatic Dependency Submission submits the versions of actions used by a repository it records:

@v1.2.3 as 1.2.3
@v1 as 1.*.*
8e8c483db84b4bee98b60c0593521ed34d9990e8 as 8e8c483db84b4bee98b60c0593521ed34d9990e8

The Security Advisory Database can't match 8e8c483db84b4bee98b60c0593521ed34d9990e8 against the vulnerable version range and won't show the vulnerability in the dependency graph.

Another issue exists when actions are forked to an organization (a common technique to create a private actions marketplace). The dependency is recorded using the organization name that holds the fork, not the upstream.

The Security Advisory Database can't match myorg/actions_checkout against the vulnerable action actions/checkout and won't show the vulnerably in the dependency graph.

Impacted features

Without recording the exact version of each action used, a number of features won't work as you might expect:

Dependency Graph won't show which actions you use are vulnerable.
Dependabot Security Updates won't be created for workflows using vulnerable actions.
Dependency Review Action won't block pull requests introducing vulnerable actions into your workflows.

Solution

To solve the issue I created a new manual submission action which mimics the automatic submission, but adds a few extra features:

It resolves the SHA to the highest, most specific version and records that as an additional dependency.
It resolves a wildcard version to the current specific version and records that as an additional dependency.
It can be configured to look up the upstream of a forked action and records that as an additional dependency.

With these additional dependencies recorded, the Dependency Graph now contains all the metadata to determine whether the GitHub actions you depend on are vulnerable and all features should now work as expected.

How to

Changing how Actions' dependencies are reported, you simply add my jessehouwing/actions-dependency-submission action to your workflows (and optionally add actions/dependency-review-action:

name: Submit and validate dependencies of GitHub Actions

on:
  push:
    branches:
      - main
  pull_request:
  schedule:
    - cron: '33 4 * * *'

permissions: {}

jobs:
  submit-actions-dependencies:
    runs-on: ubuntu-latest
    permissions:
      # submitting dependencies requires 'contents: write'
      contents: write

    steps:
      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
      - uses: jessehouwing/actions-dependency-submission@e848a29fd84b874cce3e45ceb00619bc72dbeca3 # 1.0.2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
      - uses: actions/dependency-review-action@3c4e3dcb1aa7874d2c16be7d79418e9b7efd6261 # 4.8.2
        if: github.event_name == 'pull_request'
        with:
          retry-on-snapshot-warnings: true

You can test whether the action is working correctly by adding a vulnerable action to your repository:

name: Submit Dependencies
on:
  workflow_dispatch:

jobs:
  submit-dependencies:
    if: false
    runs-on: ubuntu-latest
    permissions: {}
    steps:
      - uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe # v4.1.2

This should show as a high vulnerability in the Dependency graph of the repository:

Dependency Graph showing actions/download-artifact@v4.1.2 as vulnerable.

And you'll also see the original report using the SHA:

Dependency Graph showing actions/download-artifact referenced by SHA.

You can find the action and additional documentation and examples here.

GitHub Billing - Let Organization and Repository admins set the Cost Center in GitHub

Jesse Houwing — Wed, 24 Sep 2025 08:48:31 +0000

We rely on GitHub Cost Centers to split the GitHub bill across the different entities that make up our company. What resource is tied to which cost center is managed in the GitHub Enterprise Settings and for now this can only be done be a user with Enterprise Owner or Billing Administrator permissions.

I'd been searching for a way to allow Organization Owners and Repository Administrators to set the Cost Center on a repository themselves without having to involve one of these highly privileged accounts. We found a solution using GitHub Actions and Repository Custom Properties.

Since my last blog post on Cost Centers, a few small improvements were released by GitHub:

There is now an API to associate a Repository with a Cost Center.
It's now possible to associate a Repository to a different Cost Center than the Organization that owns it.

These two improvements allow us to solve this problem.

Creating the Custom Property

GitHub has a feature called Custom Properties for repositories which allows you to link arbitrary data to a repo. You can define these properties at the Enterprise, Organization and Repository level.

Since our cost centers are defined at the Enterprise level, I chose to create this custom property at the same level. To add a new custom property navigate to your enterprise, select the Policies tab and then drill down into Repository and then Custom Properties. Choose New Property to create your new property:

Navigate to the Custom Properties section and click new property

Give the property a clear Name and Description. Since our list of cost centers is clearly defined, I've chosen the Type single-select to allow our users to pick a cost center from a predefined list.

Give the property a unique name and set the type to single select

Next, add the list of cost centers you want people to be able to select from as options. I've also included an option inherit which I'll explain later:

Add the required cost-center options.

And finally make the field required and set the default value to inherit.

Set the permission to allow users with properties permission to edit this field, make it required and set the default value.

Now when you create a new repository or navigate to the settings of an existing repository, you can set the cost-center custom field:

Users are required to select a cost center when creating a new repository.

And existing repositories will now show the custom property in the repository settings:

The reason for Inherit

In the above example we added a special value inherit , when this value is set, we don't actually set any cost center on the repository and this causes all costs generated by this repository to automatically flow to the cost center associated to the Organization. If the Organization isn't associated to a cost center, the costs will flow to the Enterprise.

Making it all work using GitHub Actions

Creating the Custom Property itself doesn't do anything by itself. In order to actually associate the Repository to the Cost Center, we still need to update the configuration in the GitHub Enterprise Billing & Licensing section:

Billing Administrators can associate Organizations, Repositories and Users to Cost Centers

Without additional automation each repository needs to be manually added to a cost center by editing the Cost Center and adding the Repository. This is a tedious manual process.

Instead, to update the Cost Center configuration I've created a GitHub Actions workflow that runs once a day and compares the Cost Center configuration against the values of the Custom Properties and performs updates to the Cost Center configuration if needed.

The basic workflow is quite simple:

Query the current Cost Center configuration
Query all Organizations in our GitHub Enterprise
Query all Repositories in each GitHub Organization
Query the Custom Properties for each Repository
Update the Cost Center configuration in case it doesn't match the Custom Property

The workflow relies on the GitHub CLI t0 query the data.

$costCenters = (invoke-gh -fromJson -- api /enterprises/$enterprise/settings/billing/cost-centers).costCenters

$orgs = get-allorganizations $enterprise | % { $_.login }

update-repocostcenters

The update-repocostcenters function checks which updates are needed:

function update-repocostcenters {
    Write-Output "Updating Costcenters for repositories in $enterprise." 

    foreach ($org in $orgs) {

        Write-Output "Processing organization $org"
        $repos = invoke-gh -slurp api /orgs/$org/repos --paginate --jq '.[] | { name: .name, full_name: .full_name }' | ConvertFrom-Json

        foreach ($repo in $repos) { 

            $customProperties = invoke-gh api /repos/$($repo.full_name)/properties/values --jq '.' | ConvertFrom-Json
            $repoCostCenterProperty = $customProperties | Where-Object { $_.property_name -eq "cost-center" } | Select-Object -ExpandProperty value -ErrorAction Ignore
            $repo | Add-Member -NotePropertyName cost_center -NotePropertyValue $repoCostCenterProperty -Force

            $currentCostCenter = $costCenters | ? { $_.resources | ? { $_.type -eq "Repo" -and $_.name -eq $repo.full_name } }

            if ( $repo.cost_center -eq "inherit" ) {
                $repo.cost_center = $null
            }

            $targetCostCenter = $null

            if ($repo.cost_center) {
                $targetCostCenter = $costCenters | ? { $_.name -eq $repo.cost_center }
                if (-not $targetCostCenter) {
                    Write-Warning "Costcenter for repository $($repo.full_name) not found."
                }
            }

            if ($null -eq $currentCostCenter) {
                if ($null -ne $targetCostCenter) {
                    Write-Output "Updating costcenter for repository $($repo.full_name) from unassigned to $($targetCostCenter.name)."
                    Update-CostCenterResources -Handles $repo.full_name -ResourceType "Repo" -Action "Add" -CostCenter $targetCostCenter -Enterprise $enterprise
                }
                else {
                    Write-Verbose "Repository $($repo.full_name) does not have a cost center assigned."
                }
            }
            else {
                if ($null -eq $targetCostCenter) {
                    Write-Verbose "Repository $($repo.full_name) does not have a cost center assigned."
                    Write-Output "Updating costcenter for repository $($repo.full_name) from $($currentCostCenter.name) to unassigned."
                    Update-CostCenterResources -Handles $repo.full_name -ResourceType "Repo" -Action "Delete" -CostCenter $currentCostCenter -Enterprise $enterprise
                }
                elseif ($currentCostCenter.id -ne $targetCostCenter.id) {
                    Write-Output "Updating costcenter for repository $($repo.full_name) from $($currentCostCenter.name) to $($targetCostCenter.name)."
                    Update-CostCenterResources -Handles $repo.full_name -ResourceType "Repo" -Action "Delete" -CostCenter $currentCostCenter -Enterprise $enterprise
                    Update-CostCenterResources -Handles $repo.full_name -ResourceType "Repo" -Action "Add" -CostCenter $targetCostCenter -Enterprise $enterprise
                }
            }
        }
    }
}

Unfortunately, you can't simply update the associated Cost Center in a single call, when changing the current value, an add plus delete is required.

I shared the Update-CostCenterResources function in a previous blogpost. GitHub added the ability to set Organizations and Repositories recently, so here is the updated version:

function Update-CostCenterResources {
    param(
        [Parameter(Mandatory = $true)]
        [string[]]$Handles,

        [ValidateSet('User', 'Repo', 'Org')]
        [string]$ResourceType = "User",

        [Parameter(Mandatory = $true)]
        [ValidateSet('Add', 'Delete')]
        [string]$Action,

        [Parameter(Mandatory = $true)]
        $CostCenter,

        [Parameter(Mandatory = $true)]
        [string]$Enterprise
    )

    switch ($Action) {
        'Add' {
            $method = 'POST'
            $Handles = $Handles | Where-Object { 
                $handle = $_
                return (($costCenter.resources | ? { $_.type -eq $ResourceType } | ? { $_.name -eq $handle }).Count -eq 0)
            }
        }
        'Delete' {
            $method = 'DELETE'
            $Handles = $Handles | Where-Object { 
                $handle = $_
                return (($costCenter.resources | ? { $_.type -eq $ResourceType } | ? { $_.name -eq $handle }).Count -gt 0)
            }
        }
    }

    # Call fails when processing too many users at once. Thus batching the calls...
    $count = 0
    do {
        $batch = $Handles | Select-Object -Skip $count -First 30
        $count += $batch.Count

        if ($batch.Count -gt 0) {
            switch ($ResourceType) {
                'User' { 
                    $body = @{
                        users = [string[]]$batch
                    }
                }
                'Org' { 
                    $body = @{
                        organizations = [string[]]$batch
                    }
                }
                'Repo' { 
                    $body = @{
                        repositories = [string[]]$batch
                    }
                }
            }


            $_ = ($body | ConvertTo-Json) | gh api --method $method /enterprises/$Enterprise/settings/billing/cost-centers/$($CostCenter.id)/resource --input -
        }
    } while ($batch.Count -gt 0)
}

All of this runs from a simple GitHub workflow which has access to a Personal Access token with the required permissions to access the Cost Center information as well as sufficient permissions to query all Repositories in all Organizations in the enterprise.

name: Set Costcenters

on:
  schedule:
    - cron: "5 4 * * *"

jobs:
  update:
    permissions: 
      contents: read

    runs-on: ubuntu-latest

    steps:      
    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
      with:
        fetch-depth: 0

    - name: Assign Costcenters
      run: |
         .\assign-costcenters.ps1
      env:
        GH_TOKEN: ${{ secrets.GH_TOKEN }}
      shell: pwsh

You can find the whole script here:

GitHub Actions learnings from the recent nx hack

Jesse Houwing — Tue, 02 Sep 2025 21:05:46 +0000

You may have seen recent reporting around the compromise of the nx project. A malicious version of their package was published to npmjs which subsequently published GitHub tokens, crypto wallets and other sensitive materials of 1000s of users.

The maintainers of the project have spelled out exactly how the project was compromised. Yet, it doesn't tell exactly what they did to prevent the same issue from happening in the future. Or how you can protect yourself from these attack vectors.

I'll walk through the attack chain and will explain what options are available to you to prevent these things from happening.

Understanding the vulnerable workflow

Let's look at the main cause of this hack. The introduction of a new GitHub Actions Workflow that had the pull_request_target trigger enabled. This trigger is dangerous, since it signals to GitHub that the workflow is safe and has access to secrets. By default only to secrets referenced in the workflow itself, but that always includes the built-in GitHub Actions token.

on:
  pull_request_target:
    types: [opened, edited, synchronize, reopened]

The second stage of the hack was possible due to script injection in that same workflow. This allowed the injection of arbitrary code in the pipeline through the title of the pull request:

      - name: Create PR message file
        run: |
          mkdir -p /tmp
          cat > /tmp/pr-message.txt << 'EOF'
          ${{ github.event.pull_request.title }}

          ${{ github.event.pull_request.body }}
          EOF

The pull request's title and body are written to the script prior to execution, so including EOF in any of them cause any subsequent text to be interpreted as arbitrary bash commands. And adding $(...) would also lead to code execution prior to writing the contents to pr-message.txt.

Since the workflow can now execute arbitrary code, and the GitHub Hosted Runner runs with sudo by default, it would also be possible to write a memory dump of the runner process to gain access to the Runner's GitHub Token.

The payload was probably hidden in the body of the pull request and could have been similar to:

# Break the multi-line content redirection by supplying the defined EOF token.
EOF
# Dump the listener process so we can get access to its secrets

sudo apt-get install -y gdb && \
sudo gcore -o k.dump "$(ps ax | grep 'Runner.Listener' | head -n 1 | awk '{ print $1 }')"

# Then, we grep the memory dump for the format of secret values—per the job data we decrypted in the network analysis:

grep -Eao '"[^"]+":\{"value":"[^"]*","issecret":true\}' k.dump*

# Optionally use CURL to send these secrets to a url you control.

This is a well-known way to exfiltrate secrets from the GitHub Hosted Runner and I speculate it was used to gain access to the GITHUB_TOKEN secret.

Since this token has write-all permissions on older repositories by default, it is possible to either write to an unprotected branch on the repo, or to run other workflows. The publish.yml workflow had the ability to be triggered manually in the master branch. This means that it can be triggered against any branch in the project repo and will use the contents of the workflow in that specific branch. Because the npm token is stored as a repository secret, it will be available to all workflows in the repository, regardless of the branch it's defined in.

In this case I could have used the GitHub Token to:

Create a new branch in the project repository
Overwrite the publish.yml in the new branch to send the npm token to a webserver I control
Trigger the publish workflow against my branch
Delete the branch
Optionally delete the workflow run

Without overwriting the publish.yml, changes in any of the build scripts executed in the publish job would have had access to the token, since it was added as a job level environment variable.

  publish:
    if: ${{ github.repository_owner == 'nrwl' }}
    env:
      GH_TOKEN: ${{ github.token }}
      NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

In the disclosure the project mentions that they have branch rulesets in place to protect their master branch. Without branch protections this would have been even simpler.

Understanding the vulnerable (human) process

This attack couldn't have happened if the workflow above wouldn't have been added, or if it had used the standard pull_request trigger instead. If you read the documentation for pull_request_target, it calls out that its dangerous.

It looks like this workflow was generated by AI:

🤖 Generated with Claude Code

And that none of the humans in the loop were aware of the dangers. Nor was there any tooling in place to scan the workflows and to alert them of the potential problems.

Once the attacker figured out the attack chain, it would have been possible to execute the attack in seconds, with only minimal observable traces. And because all actions would be performed by the github-actions[bot] user, none of the changes would stand out and any commits would even be considered verified if done correctly.

It's trivially easy to setup a repo in which to test this attack without actually having to interact with the nx repo.

Because the NPM token was stored as a repository secret, access to it would not require an explicit approval.

Not related to this attack, but worrying nonetheless. The actions referenced by the nx project are pinned to their major version tag. This is against best practices for GitHub Actions in public repos. For example:

      - uses: actions/checkout@v4

Best practices require each action to be pinned to their commit sha. In this case as far as I can tell none of the referenced actions have a known vulnerability, but this leaves the project open to attacks similar to the tj-actions/changed-files hack from a few months back.

Remediation

Let's break down the remediations that can be put in place:

Change the default permissions for GitHub Actions to read-only.
Set workflow permissions at the job level
Enable branch rulesets to require pull requests for all important branches.
Require Approval for running fork pull request workflows from contributors
Do not rely on pull_request_target.
Do not use workflow_dispatch or repository _dispatch triggers in workflows that have access to critical secrets.
Do not mix pull-request validation and continuous integration workflows
Store critical secrets in environment secrets, instead of repository secrets.
Pass secrets only to the steps that need them
Pass data to script steps using the environment
Pin all actions to their sha.
Enable GitHub Advanced Security Code Scanning for GitHub Actions and add Branch Ruleset.
Enable Dependabot for GitHub Actions to automatically update action references to the latest version.
Consider 3rd party security hardening tools like StepSecurity's Harden Runner.

Each of these remediations is discussed in more detail below.

Change the default permissions for GitHub Actions to read-only

When GitHub Actions was first released, the default permissions were set to write-all. When GitHub changed the default permission to read-all later, they did not retroactively reset the permission for existing accounts and organizations.

GitHub Enterprise Owners and GitHub Organization Owners can enforce a default permission of read-all through the Settings:

Set the default workflow permissions to: Read repository contents and package permissions.

This will break all workflows that do not explicitly declare their required permissions in case they need more than read-all.

While you're at it, you should also turn off Allow GitHub Actions to create and approve pull requests. Allowing this setting offers attackers a way to bypass branch rulesets under certain conditions.

Set workflow permissions at the job level

A workflow can declare its required permissions at the workflow or at the job level. For best security, it's recommended to deny all permissions at the workflow level and explicitly declare permissions for each job specifically:

# Opt out of any default permissions
permissions: {}

jobs:
  build:
    # explicitly request permissions
    permissions:
      contents: read

  deploy:
    permissions:
      contents: read
      packages: write

This also removes write permissions in case the default permissions are set to read and write.

Enable branch rulesets to require pull requests for all important branches.

The nx project reports that they had enabled Branch Rulesets to require a pull request for any changes to enter the master branch and other important branches in their repository.

This prevented attackers from simply pushing a new workflow directly to the main branch to exfiltrate the npm token and other secrets.

Protecting branches in your repository is a straight forward configuration change.

Require a pull request before merging

This is also a prerequisite for other protections, like requiring Code Scanning Results before merging.

There are quite a few policy settings you can enable through Branch Rulesets the following research explains why each of these settings is important and how an attacker might abuse a ruleset with a slightly less strict configuration.

Require approval for running fork pull request workflows from contributors

This is a controversial one. As it adds quite a bit of overhead to project teams receiving many 3rd party contributions. But it might have been enough for a maintainer to detect the script injection attack that was used to exfiltrate the GitHub Actions token.

GitHub Enterprise Owner, Organization Owners and Repository Admins can set a policy to require approval before running any GitHub Actions workflow.

Require approval for all external contributors

While it might feel tempting to set this policy to any of the lighter settings, there are already known cases where a person appears helpful by fixing a few typos. They'll be considered trusted after that.

Setting this policy will require a maintainer to review the Pull Request and trigger the workflows each time an external contributor pushes new commits.

Do not rely on `pull_request_target`

The pull_request_target is inherently dangerous. It opens you up to all kinds of additional attack paths and very few people are completely aware of the implications. It is very easy to make a mistake when you enable this trigger on a workflow and open your repository up for attacks.

My recommendation is to not rely on the pull_request_target trigger. For most scenarios the pull_request trigger is sufficient.

In the enterprises I control we've added a Custom Pattern for Secret Scanning to block all commits that contain pull_request_target:

Define a custom pattern to block pushes

And we enabled this pattern for Push protection:

Enable custom pattern for push protection

While there are cases where pull_request_target can be useful, it's generally not worth the security risks.

From the docs:

⚠️ Warning

For workflows that are triggered by the pull_request_target event, the GITHUB_TOKEN is granted read/write repository permission unless the permissions key is specified and the workflow can access secrets, even when it is triggered from a fork. Although the workflow runs in the context of the base of the pull request, you should make sure that you do not check out, build, or run untrusted code from the pull request with this event. Additionally, any caches share the same scope as the base branch. To help prevent cache poisoning, you should not save the cache if there is a possibility that the cache contents were altered. For more information, see Keeping your GitHub Actions and workflows secure: Preventing pwn requests on the GitHub Security Lab website.

Do not use `workflow_dispatch` or `repository_dispatch` triggers in workflows that have access to critical secrets.

Since workflow_dispatch and repository_dispatch allow an attacker to control the branch against the workflow will run, it can be used to exfiltrate repository secrets if the attacker has write access to any other branch on the repository.

The only way to protect secrets from this attack path, is to store them as an environment secret.

Do not mix pull-request validation and continuous integration workflows

Many developers will try to reduce duplication by combining multiple workflows into 1 and defining multiple triggers or conditions. These conditions then control what code is executed depending on the event that triggered the workflow.

A really common example is to have a single pull request validation workflow that also doubles as the first set of steps of the full CI build.

The publish.yml in the nx project has 3 such purposes:

Release the packages for the nx project whenever a new tag is created
A canary release based on a schedule
Creation of a test-build for pull requests

name: publish

on:
  # Automated schedule - canary releases from master
  schedule:
    - cron: "0 3 * * 2-6" # Tuesdays - Saturdays, at 3am UTC
  # Manual trigger - PR releases or dry-runs (based on workflow inputs)
  workflow_dispatch:
    inputs:
      pr:
        description: "PR Number - If set, a real release will be created for the branch associated with the given PR number. If blank, a dry-run of the currently selected branch will be performed."
        required: false
        type: number
  release:
    types: [published]

The nx project does this by restricting the build and publish jobs to only run when the triggering repository is owned by nrwl.

    build:
      if: ${{ github.repository_owner == 'nrwl' }}

    publish:
      if: ${{ github.repository_owner == 'nrwl' }}

Which would prevent the publish job from running against pull requests from a forked repository.

However, since the leaked token had write-all permission, it would have been easy to publish a new branch which would satisfy the repository_owner condition and thus trigger the publication jobs.

More restrictions could have been added, like checking against the actor to see whether it's a maintainer or preventing workflows triggered by GitHub Actions, or restricting jobs to specific branches. But when the attacker has write permissions to the repository, none of these will likely save you.

A better approach would be to split the workflow into 2 separate workflows:

publish.yaml - which publishes the production packages
pr-validation.yaml - which builds a PR validation package

These two workflows can use a 3rd callable workflow, or one or more Composite actions to reduce code duplication while also reducing chances of an attacker calling a workflow they're not meant to call.

Another advantage of callable workflows and composite actions is that they can be stored in a different repository, limiting an attacker's ability to leverage a leaked Actions Token to change the workflow code.

Store critical secrets in environment secrets, instead of repository secrets.

Secrets defined at the repository level are accessible to all workflows. There is no way to limit access to repository level secrets.

In order to protect a critical secret, such as the NPM token used to publish your package, it should be defined as an Environment Secret.

In the Environment's policy you can then set a branch policy to limit which branches have access to the secret.

Set specific branches to limit access to critical secrets.

Optionally add Required Reviewers to prevent access to the secrets without manual approval:

Add Required Reviewers to prevent access to the secrets without manual approval

Then update the workflow to reference the environment when defining the job:

jobs:
  deploy:
    environment: public
    steps:
      uses: actions/deploy@sha
      with:
        npm_token: ${{ secrets.NPM_TOKEN }}

It looks like the nx project has already implemented this in a recent change.

Pass secrets only to the steps that need them

In the nx repository the NPM_AUTH_TOKEN is defined at the job level:

  publish:
    env:
      # DO NOT USE THIS!
      GH_TOKEN: ${{ github.token }}
      NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
    steps:
      - ...

This makes the secrets available to all steps in the workflow. If the attacker can take control of any of the code executed in these steps, they would be able to exfiltrate the secret.

In this case the job also calls pnpm build:wasm which adds a number of additional places to hide exfiltration code.

It's better to explicitly pass the secrets only to the steps that need them:

  publish:
    env:
      # NO SECRETS SET HERE
    steps:
      - ...
      - ...
      - name: Publish
        env:
          VERSION: ${{ needs.resolve-required-data.outputs.version }}
          DRY_RUN: ${{ needs.resolve-required-data.outputs.dry_run_flag }}
          PUBLISH_BRANCH: ${{ needs.resolve-required-data.outputs.publish_branch }}
          NX_VERBOSE_LOGGING: true
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
        run: |
          echo ""
          # Create and check out the publish branch
          git checkout -b $PUBLISH_BRANCH
          echo ""
          echo "Version set to: $VERSION"
          echo "DRY_RUN set to: $DRY_RUN"
          echo ""
          pnpm nx-release --local=false $VERSION $DRY_RUN

Pass data to script steps using the environment

The Script Injection attack was possible because variables were in-lined in the body of a script step:

      - name: Create PR message file
        run: |
          mkdir -p /tmp
          cat > /tmp/pr-message.txt << 'EOF'
          ${{ github.event.pull_request.title }} <- unsafe variable reference

          ${{ github.event.pull_request.body }} <- unsafe variable reference
          EOF

To pass data safely to scripts, you should instead pass them through the environment:

      - name: Create PR message file
        run: |
          mkdir -p /tmp
          echo $PR_TITLE >> /tmp/pr-message.txt
          echo "" >> /tmp/pr-message.txt
          echo $PR_BODY >> /tmp/pr-message.txt
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
          PR_BODY: ${{ github.event.pull_request.body }}

Script Injection can be detected by GitHub Advanced Security Code Scanning for Actions as well as ActionLint.

Pin all actions to their sha.

The attack on nx doesn't seem to have relied on this attack vector, but the recent attack on tj-actions/changed-files has shown again that it's dangerous to rely on only the tag to reference actions of 3rd parties.

GitHub recommends that you always reference actions you do not control using their explicit full commit sha.

For example:

      - uses: actions/checkout@v4

Should have used:

    - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5

There is now a policy which requires all actions to be pinned to their commit sha:

Require actions to be pinned to a full-length commit SHA

Because this is a new policy, GitHub Administrators must explicitly opt-in to this policy.

There are several tools available to perform the pinning on your behalf:

Once, pinned, if enabled, Dependabot will automatically submit a pull request with the correct new commit sha. Unfortunately, it doesn't automatically change the tag v4 to its sha.

Enable GitHub Advanced Security Code Scanning for GitHub Actions and add Branch Ruleset.

The script injection vulnerability at the root of this attack would have been detected by GitHub Advanced Security Code Scanning for GitHub Actions.

GitHub Advanced Security is free for all public repos, so there is not really a good reason not to enable it.

Because support for GitHub Actions was added recently, GitHub Repository Owners must explicitly enable GitHub Actions in the Code Scanning configuration:

Enable GitHub Actions for CodeQL Code Scanning

Or add actions to the list of supported languages in their workflow:

jobs:
  analyze:
    name: Analyze (${{ matrix.language }})
    runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
    permissions:
      security-events: write
      packages: read
      actions: read
      contents: read

    strategy:
      fail-fast: false
      matrix:
        include:
        # Make sure Actions is added to the list of languages:
        - language: actions
          build-mode: none

This will detect a number of GitHub Actions vulnerabilities, including:

Script Injection
Unsafe checkout

In order to prevent workflows with these vulnerabilities to enter the repository, add a branch ruleset that requires the Code QL workflow to succeed:

Enable Require status checks to pass and add the CodeQL check explicitly

And require Code Scanning results:

Enable Require code scanning results and add CodeQL explicitly.

An alternative for GitHub Advanced Security Code Scanning for Actions that would also have detected the Script Injection vulnerability is ActionLint. Actionlint can also be added as a workflow.

Enable Dependabot for GitHub Actions to automatically update action references to the latest version.

To automatically keep your referenced GitHub Actions up to date, especially when you are pinning to the full commit sha, you can enable Dependabot to automatically create a pull request whenever an action is updated.

To enable Dependabot for GitHub Actions, explicitly add github-actions to your .github/dependabot.yml:

version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/" # Location of package manifests
    schedule:
      interval: "weekly"

To reduce maintainer burden, GitHub Actions can be used to automate updates completely.

Consider 3rd party security hardening tools like StepSecurity's Harden Runner.

All of the above remediations are built-in features of the GitHub platform.

There are 3rd party solutions available to further secure GitHub Actions. Of those StepSecurity's Harden Runner is probably the best-known option. You can add harden-runner to any workflow by adding a single step to your workflow jobs:

steps:
  - name: Harden Runner
    uses: step-security/harden-runner@c6295a65d1254861815972266d5933fd6e532bdf # v2.11.1
    with:
      egress-policy: audit

The Community version of Harden runner offers the following protections to all of workflows:

CI/CD-Aware Event Correlation: Each outbound network connection, file operation, and process execution is mapped to the exact step, job, and workflow where it occurs.
Automated Baseline Creation: Harden-Runner builds a baseline for each job based on past outbound network connections.
Anomaly Detection: Once the baseline is created, any future outbound calls not in the baseline trigger a detection.
Block Network Egress Traffic with Domain Allowlist: Optionally use the automatically created baseline to control outbound network traffic by specifying allowed domains, preventing unauthorized data exfiltration.
Detect Modification of Source Code: Monitor and alert on unauthorized changes to your source code during the CI/CD pipeline.

Conclusion

GitHub offers many features to protect your project against attacks that try to abuse GitHub Actions. Unfortunately, many of these features require explicit opt-in or configuration.

This should not be an excuse to leave your open-source projects open to attack.

Dear GitHub,

We love the unlimited free environment and all the tools you provide to keep our open-source projects free and healthy. And while we love your commitment to not break our workflows, it's not helpful to make many of the protections you offer opt-in.

Please, please, please start enabling core security features automatically, at least on popular open-source projects as part of your roll-out strategy.

The new GitHub Advanced Security Code Scanning for GitHub Actions would have alerted the maintainers of the nx project to the Script Injection vulnerability. But it isn't activated by default, nor are maintainers warned they must opt-in to receive this protection.

The long ago changed default read-only permissions for GitHub Actions would have prevented the triggering of the publish workflow, but this setting is kept to the organization default, and maintainers aren't warned they should change this setting.

Requiring workflows triggered by external contributors to always be approved by a maintainer would have at least triggered a human intervention as part of the attack chain. But this new default has also not been rolled out to existing organizations, nor are maintainers warned they should really change this setting.

Pinning Actions to their full commit sha has been a recommendation for years now. Yet GitHub has not provided any tooling to make this process easier, and while the policy has finally arrived, again it has rolled out with relatively low fanfare and maintainers are not warned they should really change this setting. Nor does Dependabot or GitHub Advanced Security Code Scanning for Actions submit a pull request to pin these actions.

This hasn't been the first attack to leverage the pull_request_target trigger, I'd wish workflows with this trigger would not automatically enable and would require explicit enablement by an owner through a danger-zone action which clearly spells out the risks.

So again, please, start enabling these core security features automatically and make such roll-out part of the introduction of new features such as these.

Authenticate Connect-MgGraph using OIDC in GitHub Actions

Jesse Houwing — Tue, 10 Jun 2025 15:08:58 +0000

I'm running a number of maintenance scripts against our Azure EntraId to manage GitHub related things. Removing dormant users, asking users to setup their notification email correctly etc.

For a long time, I ran these scripts with an interactive session, before moving them over to GitHub Actions. Recently I made the move to convert my scripts to a workflow and changed the authentication mechanism to OIDC-connect, in order to remove the need to manage secrets and tokens.

The documentation is quite clear on how to set things up.

Unfortunately, while it explains how to setup an Azure PowerShell session using enable-AzPSSession: true, it doesn't explain how to authenticate using connect-mggraph.

After some experimentation (and of course some help from GitHub Copilot), I figured out I could query the access token from az, then pass that straight to connect-mggraph. So now my workflow looks like this:

- name: Azure CLI Login
  uses: azure/login@v2
    with:
      client-id: ${{ secrets.AZURE_CLIENT_ID }}
      tenant-id: ${{ secrets.AZURE_TENANT_ID }}
      allow-no-subscriptions: true

  - name: Assign Costcenters
    run: |
       $accessToken = az account get-access-token --resource https://graph.microsoft.com `
         --query accessToken --output tsv
       write-host "::add-mask::$accessToken"
       $accessToken = $accessToken | ConvertTo-SecureString -AsPlainText -Force
       Connect-MgGraph -AccessToken $accessToken -NoWelcome

       # rest of the script here ... 
    shell: pwsh

And my scripts can happily query the Microsoft Graph.

Picking the right GitHub Copilot model

Jesse Houwing — Fri, 25 Apr 2025 09:24:19 +0000

GitHub recently released documentation on all the large language models available in GitHub Copilot. The list is ever expanding, especially when you have previews turned on for your account. But it's still hard for many people to pick the right model for their task.

You can find the official guidance in the GitHub Documentation.

I wanted to make model selection even easier, so I asked GitHub Copilot to read the doc for me and suggest the right model:

#fetch https://docs.github.com/en/copilot/using-github-copilot/ai-models/choosing-the-right-ai-model-for-your-task

What model would I use to plan a large refactoring involving multiple files for which I do not yet have a working example?

Select the best 2 and explain the pros and cons between them

And it actually came back with a pretty good suggestion:

To plan a large refactoring involving multiple files without a working example, the two best models to consider are GPT-4.5 and GPT-4o. Here's a comparison of their pros and cons:

GPT-4.5

Pros:

Deep Reasoning and Contextual Understanding: GPT-4.5 excels at handling complex logic and multi-step reasoning, making it ideal for large-scale refactoring tasks.
Cross-File Context: It can retain and process context across multiple files, which is crucial for understanding dependencies and relationships in a large codebase.
Reliable and Consistent Results: Produces high-quality, structured suggestions for improving maintainability and modularity.
Nuanced Problem-Solving: Handles ambiguous or incomplete requirements effectively, which is helpful when no working example exists.

Cons:

Higher Latency: Slightly slower response times compared to smaller models like GPT-4o.
Higher Cost: More resource-intensive, which may not be ideal for cost-sensitive scenarios.

GPT-4o

Pros:

Faster Response Times: Provides quicker iterations, which is beneficial for rapid back-and-forth during refactoring.
Balanced Performance: Offers a good mix of reasoning and speed, making it suitable for lightweight tasks or smaller refactoring steps.
Cost-Effective: Less expensive than GPT-4.5, making it a better choice for budget-conscious projects.

Cons:

Limited Depth for Complex Tasks: While capable, it may struggle with deeply nuanced or highly interconnected refactoring tasks compared to GPT-4.5.
Reduced Cross-File Context: May not handle dependencies across multiple files as effectively as GPT-4.5.

Recommendation:

Use GPT-4.5 if the refactoring involves complex logic, dependencies across multiple files, or requires high-quality, structured suggestions.
Use GPT-4o if you need faster iterations, are working on smaller or less complex refactoring tasks, or are constrained by cost.

Both models are strong candidates, but the choice depends on the complexity of the refactoring and your priorities (speed vs. depth).

Say goodbye to your Personal Access Tokens

Jesse Houwing — Tue, 04 Mar 2025 12:30:04 +0000

We got rid of all Azure DevOps PAT usage and so should you.

📢 Big shout out to Jesse!

Without his blog post and direct help, I was probably still renewing expired PATs manually. But let's start at the beginning.

Michael, author of some of my favorite Azure DevOps Extensions, reached out on the Azure DevOps Club slack channel for help. He wanted to automatically rotate Personal Access Tokens to integrate with Azure DevOps, which lead to a quest to completely eradicate Personal Access Tokens altogether. After sharing my work to make the Azure DevOps Extension Tasks work without Personal Access Tokens, Michael went to work. I've asked him to share his results in this guest blog, since I feel they're useful for anyone doing automation against Azure DevOps. As you can see he agreed!

Jesse Houwing

PATs, PATs and more PATs - our 5 scenarios we used PATs for

Building Bravo Notes here at Agile Extensions we are extensively using the Azure DevOps REST APIs in many ways.

And I'm not talking about the API calls the Bravo Notes extension makes at runtime e.g. to load work items, publish wiki pages etc.

We use the Azure DevOps REST APIs in these 5 scenarios as well:

publish multiple versions of the Bravo Notes extension for development, staging and production via the marketplace APIs (in Azure Pipelines)
retrieve marketplace event data for installs/uninstalls and more on a schedule (in Azure Functions app)
run integration tests for many Bravo Notes components that need Azure DevOps APIs (in Azure Pipelines)
run e2e tests of the main Bravo Notes app outside of Azure DevOps (in Azure Pipelines)
run integration and e2e tests locally

In the past we used personal access tokens (PATs) for all of those scenarios.

For years our process was:

watch for notifications that a PAT was about to expire
be annoyed that this manual regular chore was on your plate (AGAIN!)
regenerate PAT in Azure DevOps
update pipeline variables and secrets with the new PAT
feel bad and wonder why there wasn't a more elegant solution
be done and push responsibility for improving this to future Michael

PAT REST APIs to the rescue?

A while I ago a new set of APIs got introduced to manage PATs. That sounded intriguing and a few weeks back I reached out to the community to ask for guidance on how to automatically renew and manage PATs and do away with the manual process entirely.

🤔 Me: Is there a good resource online on how to transition from manually rotating PATs to rotating PATs automatically (a pipeline using PAT REST API, KeyVault)?

Thankfully Jesse pointed me to a better (the right) way:

💡 Don't use PATs but use an Azure Service principle with workload Federation.

When you are reading this on his blog you probably already know that Jesse is one of those people who uses the internet for what it was build for:

He shares what he has learned (often the hard way) to make the lives of all of us easier. In this case in form of this blog post on how to publish marketplace extensions without using PATs.

I was convinced already that using PATs was less than ideal.

But at this point I was still hesitant about whether this approach could help us do away with PATs entirely. After all we had 5 different scenarios.

There was only one way to find out so we started by tackling the scenario that Jesse wrote about in his blog post.

Scenario 1: Publish marketplace extensions without a PAT

The blog post didn't disappoint as it contained all the pieces to help us publish marketplace extensions without a PAT - whohoo 🥳!

Below are the steps we needed to take. Please refer to Jesse's blog post for all the datails. Our mileage varied a bit as the ARM service connection now uses an "App registration" in Azure instead of a "Service Principal".

The process is the still the same in general, though.

Create the ARM service connection (the UX changed a bit since Jesse's blog post)
- Choose "Azure Resource Manager" Connection
- Identity Type: "App registration (automatic)"
- Credential: "Workload identity federation"
- the rest is about the same
Add the App registration/service principal as a user in Azure DevOps
Extract the Azure DevOps Identity Id from the Profile API via a pipeline task
Use that Id to add the App registration as a member to the marketplace publisher
Update our pipeline to use the ARM service connection

Note that we could use the newest version of the marketplace extension tasks (v5) that allows you to publish using the ARM service connection directly.

So there is no need to fetch an access token and overwrite the marketplace service connection credentials anymore. After changing our pipelines we were able to delete our old "Visual Studio Marketplace" service connection.

- task: PublishAzureDevOpsExtension@5
  displayName: 'Publish Extension'
  inputs:
    connectTo: 'AzureRM'
    connectedServiceNameAzureRM: 'marketplace-service-connection'
    fileType: vsix
    vsixFile: '$(Pipeline.Workspace)/vsix/production.vsix'
    updateTasksVersion: false

And that was it. The pipeline worked as before and we could revoke the PAT for publishing extensions to the marketplace.

Of course we didn't want to stop here and tackle the next scenario and remove one more PAT.

Scenario 2: Call Azure DevOps APIs from a Azure Function app

We maintain an Azure Function app to query data about install and uninstall events and pull statistics from the marketplace. Authentication works using - what else could it be - a PAT.

In Azure Functions of course we don't have the magic of service connections that we can utilize in Azure Pipelines.

After initially failing to find the code to use for our NodeJS based function app, searching for the magic GUID 499b84ac-1321-427f-aa17-267ca6975798 that represents Azure DevOps led me right to what we needed to acquire an access token inside the Function app.

After finding the right way to acquire the access token, we got this solved in 2 simple steps:

Add the Azure Function app as a user in Azure DevOps
Use the code below to acquire an access token and authorize Azure DevOps REST API calls

const { ManagedIdentityCredential } = require("@azure/identity");

async function getMarketplaceAccessToken() {
    const credential = new ManagedIdentityCredential({
        // `AZURE_CLIENT_ID` points to the App registration created via the ARM service connection
        clientId: process.env.AZURE_CLIENT_ID,
    });
    const tokenResponse = await credential.getToken(
        "499b84ac-1321-427f-aa17-267ca6975798/.default",
        {
            tenantId: process.env.AZURE_TENANT_ID,
        }
    );

    return tokenResponse.token;
}

There was no stopping now - but still 3 PATs to go.

Scenario 3: Run intergration tests in Azure Pipelines

Next up was our integration testing pipeline tasks that run integration tests using NodeJS and jest. Until now a PAT was used from a secret pipeline variable. Another one that we didn't want to renew manually every few months.

As we already acquired an access token in a pipeline via an ARM service connection in scenario 1, this dind't seem to difficult to pull off.

Following the least priviledge principle we didn't want to use the same service principal from scenario 1. This would have given it access to the marketplace AND Azure DevOps resources that were needed for our integration tests.

Luckily you can create as many ARM service connections as you want and so we did. To use the newly created App registration/service principal, the pipeline task from Jesse's blog post to retrieve an access token that we dind't need for scenario 1 now came in handy again:

- task: AzureCLI@2
  displayName: 'Accquire token for integration testing'
  inputs:
    azureSubscription: 'azure-devops-integration-testing-connection'
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    useGlobalConfig: true
    inlineScript: |
      $accessToken = az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query "accessToken" --output tsv
      write-host "##vso[task.setsecret]$accessToken"
      write-host "##vso[task.setvariable variable=SECRET_INGETRATION_TESTING_TOKEN;issecret=true]$accessToken"

Now we could use the the token for subsequent integration test tasks. Yay only 2 PATs to go!

Scenario 4: Run e2e tests in Azure Pipelines (against external Azure DevOps organization)

While this scenario seemed to be exactly the same es scenario 3 at first, we had an issue here that stopped us in our tracks at first.

The e2e tests called APIs in a separate Azure DevOps org that only existed for testing.

[OrgA Pipeline] ---- REST API call ----> [OrgB Work item API]

The solution was too simple to be true.

We were just able to add the App registration/service principal as a user in OrgB and give it access to the resources needed for the e2e tests.

Another PAT down!

Scenario 5: Running integration and e2e tests locally

This might be my least favorite solution so far, but it was simple and did the trick for now.

We use jest to run our tests and in our npm scripts we just stuck in a Azure Developer CLI call to log in prior to starting the test script.

In our code to set up API authentication we now run some special code to acquire the access token, again with the azure identity SDK.

Modify testing script to log in - azd auth login && npx jest --config jest.integration.config.js"
Retrieve the access token from the Azure Developer Cli:

let token = process.env.SECRET_INGETRATION_TESTING_TOKEN;
if (!token && !process.env.CI) {
    const credential = new AzureDeveloperCliCredential({
        tenantId: process.env.AZURE_TENANT_ID,
    });
    const tokenResult = await credential.getToken(
        '499b84ac-1321-427f-aa17-267ca6975798/.default',
        {
            tenantId: process.env.AZURE_TENANT_ID,
        },
    );
    token = tokenResult.token;
}

(Would love to hear about a better solution to securly get an access token for running scripts locally in a development environment.)

All 5 scenarios done! The champaign 🍾 was almost open when I revoked all PATs that were not needed anymore when I discovered that I had forgotten one last PAT.

Bonus scenario: the last token - the renovate pipline

We use renovate to update dependencies and automatically create pull requests in yet another pipeline. (You can thank Jesse for his work online here as well).

Immideately I had a bad feeling and I thought that renovate would only accept PATs as it only accepts RENOVATE_TOKEN as a means to authenticate. The docs make it look that way for sure.

Looking at the code (yay open source) I could see that renovate automatically detects whether the token is a PAT or a OAuth token and chooses the Authorization header (Basic or Bearer) accordingly.

So we could just use the same AzureCLI pipeline task as in scenarios 3 to acquire and 4 and be done!

P.S.: Talking to Jesse about that he mentioned that Azure DevOps now accepts OAuth tokens when using the Basic authentication scheme as well which is great in my opinion.

Wrapping up

Just reading all the terms involved:

ARM service connection
Open Id Connect
Workload Identity federation
service principal
App registration

made the whole process seem daunting to me.

But in the end we got rid of all our PATs within a couple of hours.

Personal access tokens are great to quickly test an API call, authenticate a throw-away script where you don't have other means of authentication ready.

However for the long term you really don't want to have to create PATs and renew them until the end of time.

Thanks again to Jesse and Joost Voskuil for their help.

In case you are facing your own challenge with Azure DevOps or GitHub, don't hesitate to join the Azure DevOps Club.

GitHub's new Billing - Assigning Cost Centers in Bulk

Jesse Houwing — Thu, 20 Feb 2025 14:10:17 +0000

You can assign resources to cost centers, which will in turn charge any costs associated with that resource to that cost center.

Examples of ressource categories are:

Repositories - All costs associated with a specific repository. Action Minutes, Codespaces, LFS Storage etc.
Users - All costs associated with a specific user. Enterprise Seat, Copilot, Advanced Security.
Organizations - All costs associated with a specific organization. Costs generated by any unassigned repositories in the organization. GitHub Packages storage and network traffic.
Enterprise - All costs associated with the whole enterprise. Costs associated with users who are not explicitly assigned to a cost center.

By linking organizations, users and repositories to a cost center, you can cause their specific costs to be redirected from the Enterprise to the specific Cost Center.

Linking Organizations and Repositories can be done through the Cost Center UI in the GitHub Enterprise Admin portal:

In the Enterprise Admin portal, you can assign Organizations and Repositories to Cost Centers. Members can only be assigned through the API.

If you're lucky there are only a few Organizations that need to be assigned, but in our case, Members were a whole other story. We already have 500+ members and external collaborators in our enterprise and these needed to be assigned to specific cost centers. As you can see, this can't be done in the UI, but requires API calls.

It turns out the API is a bit tricky, there are certain things about the API that aren't (yet) nicely documented:

You can only assign users to a cost center in batches of 50 (confirmed with GitHub Support)
You can only assign a user if it isn't already assigned to another cost center. Changing cost centers requires 2 operations.
The API is eventually consistent. Changes made may take a couple of seconds to show up in subsequent API calls.
While you could assign cost centers on a per-user basis, this is a great way to burn through your API Rate Limits quickly.

To work around these issues, I've written a small wrapper around the API in PowerShell which uses the existing resource assignments stored in the Cost Centers to reduce the number of required calls to the API:

function Update-CostCenterResources {
    param(
        [Parameter(Mandatory=$true)]
        [string[]]$Handles,

        [Parameter(Mandatory=$true)]
        [ValidateSet('Add','Delete')]
        [string]$Action,

        [Parameter(Mandatory=$true)]
        $CostCenter,

        [Parameter(Mandatory=$true)]
        [string]$Enterprise
    )

    switch ($Action)
    {
        'Add' {
            $method = 'POST'
            $Handles = $Handles | Where-Object { 
                $handle = $_
                return (($costCenter.resources | ?{ $_.type -eq "User" } | ?{$_.name -eq $handle }).Count -eq 0)
            }
        }
        'Delete' {
            $method = 'DELETE'
            $Handles = $Handles | Where-Object { 
                $handle = $_
                return (($costCenter.resources | ?{ $_.type -eq "User" } | ?{$_.name -eq $handle }).Count -gt 0)
            }
        }
    }

    # Call fails when processing too many users at once. Thus batching the calls...
    $count = 0
    do {
        $batch = $Handles | Select-Object -Skip $count -First 50
        $count += $batch.Count

        if ($batch.Count -gt 0) {
            $body = @{
                users = [string[]]$batch
            }

            $_ = ($body | ConvertTo-Json) | gh api --method $method /enterprises/$Enterprise/settings/billing/cost-centers/$($CostCenter.id)/resource --input -
        }
    } while ($batch.Count -gt 0)
}

$enterprise = "xebia"
$costCenters = (invoke-gh -fromJson -- api /enterprises/$enterprise/settings/billing/cost-centers).costCenters

$costCenterNL = $costCenters | ?{ $_.name -eq "Netherlands" }

$handles = @("jessehouwing", "jessehouwing-demo")

# First remove the users from their currently assigned cost centers (if any)
$costCenters | 
  ?{ $_.id -ne $costCenterNL.id } | 
  ?{ $_ | ?{ $.resources | ?{ $_.type -eq "User" -and $_.name -in $handles } } | 
  %{ 
    Update-CostCenterResources -handles $handles -action "Delete" -CostCenter $_ -Enterprise $enterprise
  }

# Then assign the users to their new cost center
Update-CostCenterResources -handles $handles -action "Add" -CostCenter $costCenterNL -Enterprise $enterprise

You can find the current cost center for a user in the Cost Center's Resources array, or from the GitHub Enterprise assigned-seats REST API:

$handle = "jessehouwing"
$enterprise = "xebia"

# retrieve from Cost Center API
$costCenters = (gh api /enterprises/$enterprise/settings/billing/cost-centers | ConvertFrom-Json).costCenters
$currentCostCenter = $costCenters | ?{ $_.resources | ?{ $_.type -eq "User" -and $_.name -eq $handle } }

# retrieve from Assigned Seats API:
$enterpriseUsers = gh api https://api.github.com/enterprises/$enterprise/consumed-licenses --jq '.users[]' --paginate | ConvertFrom-Json
$currentCostCenter = ($enterpriseUsers | ?{ $_.github_com_login -eq $handle }).github_com_cost_center

By combining the above logic with some more proprietary magic, we were able to quickly assign all our members to Cost Centers based on their Azure EntraID metadata.

As you can see in the chart below, when we assigned out cost centers on January 9th, all future costs were no longer associated to the enterprise (in green), but to the respective cost centers.

Costs distributed between Enterprise and a number of Cost Centers as of January 9th.

Note: It is currently not possible to retroactively assign a resource to a cost center. Otherwise, we'd have assigned all costs as of the 1st of the month.

Use inline completions for more answers with GitHub Copilot Free

Jesse Houwing — Mon, 10 Feb 2025 16:03:31 +0000

2 months ago, GitHub gave everybody their taste of GitHub Copilot with the limited free tier. You get 2000 completions and 50 chats.

This is awesome! But 50 chats aren't that many.

Did you know you can use your completions for simple chat answers as well? And this is a nice tip for NeoVim users as well who don't have access to Chat in their editor. Inline Suggestions are significantly faster than Chat to boot.

You can use inline completions to get tips, answer questions and suggest improvements to your code as well.

Q&A

// Q: Explain the key usage scenarios for the Singleton pattern
// A: The singleton pattern |

In most cases typing // A: will do the trick, but sometimes you'll need to give Copilot a starting sentence to provide you with an answer:

// Q: Explain the key usage scenarios for the Singleton pattern. A detailed description
// A: The Singleton pattern is used when we want to ensure that a class has only one instance and provide a global point of access to that instance. Some key usage scenarios for the Singleton pattern include:

If you type // - on the next line or open the completions panel (ctrl-enter), you'll get more details:

// 1. Global Access: It provides a single point of access to the instance of the class, allowing other objects to easily access and use its functionality.
// 2. Resource Management: It can be used to manage shared resources, such as database connections, file systems, or configuration settings, ensuring that there is only one instance to avoid conflicts or resource wastage.
// 3. Caching: It can be used to implement a cache that stores frequently used data or objects in memory, reducing the need to recreate or fetch them from external sources.
// 4. Logging: It can be used to create a single logger instance that logs messages from different parts of the application, centralizing the logging functionality.
// 5. Configuration Settings: It can be used to manage application-wide configuration settings, ensuring that changes are reflected consistently across the application.
// 6. State Management: It can be used to maintain a shared state or context across different parts of the application, allowing objects to share and update state information.
// 7. Thread Safety: It can be used to provide a thread-safe way to access and modify shared resources or data, ensuring that concurrent access does

Suggest test cases

You can also ask questions about the contents of the current file:

// Suggest additional test cases to add to this test fixture, signatures only:
// |

Returns the following suggestions:

// public void GetById_WhenFlightDoesNotExist_ReturnsNull()
// public void Add_WhenFlightIsNull_DoesNotAddFlightToRepository()
// public void Add_WhenFlightHasNullFlightNumber_ThrowsException()
// public void Add_WhenFlightHasEmptyFlightNumber_ThrowsException()
// public void Add_WhenFlightHasNullOrigin_ThrowsException()
// public void Add_WhenFlightHasEmptyOrigin_ThrowsException()
// public void Add_WhenFlightHasNullDestination_ThrowsException()
// public void Add_WhenFlightHasEmptyDestination_ThrowsException()
// public void Add_WhenFlightHasDepartureTimeInThePast_ThrowsException()
// public void Add_WhenFlightHasArrivalTimeBeforeDepartureTime_ThrowsException()
// public void Add_WhenFlightHasFuelRangeLessThanZero_ThrowsException()
// public void Add_WhenFlightHasFuelTankLeakTrue_ThrowsException()
// public void Add_WhenFlightHasNullFlightLogSignature_ThrowsException()
// public void Add_WhenFlightHasEmptyFlightLogSignature_ThrowsException()
// public void Add_WhenFlightHasNullAerobaticSequenceSignature_ThrowsException()
// public void Add_WhenFlightHasEmptyAerobaticSequenceSignature_ThrowsException()
// public void Add_WhenFlightHasDuplicateId_ThrowsException()
// public void Add_WhenFlightHasDuplicateFlightNumber_ThrowsException()
// public void Add_WhenFlightHasDuplicateOriginAndDestination_ThrowsException()
// public void Add_WhenFlightHasDuplicateFlightLogSignature_ThrowsException()
// public void Add_WhenFlightHasDuplicateAerobaticSequenceSignature_ThrowsException()
// public void Add_WhenFlightHasDuplicateFlightNumberAndOriginAndDestination_ThrowsException()
// public void Add_WhenFlightHasDuplicateFlightNumberAndFlightLogSignature_ThrowsException()
// public void Add_WhenFlightHasDuplicateFlightNumberAndAerobaticSequenceSignature_ThrowsException()
// public void Add_WhenFlightHasDuplicateFlightNumberAndOriginAndDestinationAndFlightLogSignature_ThrowsException()
// public void Add_WhenFlightHasDuplicateFlightNumberAndOriginAndDestinationAndAerobaticSequenceSignature_ThrowsException()
// public void Add_WhenFlightHasDuplicateFlightNumberAndFlightLogSignatureAndAerobaticSequenceSignature_ThrowsException()

Suggest improvements

You can ask for improvements you can make to the current code file:

// To improve the security of this REST API I can:
// |

And GitHub Copilot will reply with:

// add an authorization filter to this controller
// and require authentication for all actions
// using the [Authorize] attribute

You can tweak the replies by adding // and or //something else I could do is

Conclusion

If you run out of Chat messages, you can continue the conversation using inline suggestions. If you run out of those too, you seem to be getting enough value from GitHub Copilot to warrant a subscription ;).

This also adds a bit of Chat support to NeoVim, Qt Explorer, XCode and other tools that only support inline suggestions for now.

DEV Community: Jesse Houwing

Auto-assign GitHub Copilot AI Credit budget to users

Setting up universal budget

Setting an individual user budget

Automating the problem away

Measuring the Value of AI

We've deployed it everywhere. Now what?

This isn't a new problem

The Traps We Keep Falling Into

Goodhart's Law: When the Metric Becomes the Target

The McNamara Fallacy: Counting What's Countable

Self-Serving Attribution Bias: We Believe Our Own Hype

The Shifted Filter: From Pre-Build to Post-Build

The Polishing Problem

The Non-Technical Blind Spot

How to Measure Better

1. Measure Outcomes, Not Outputs

2. Use Controlled Comparisons

3. Track Value Hypothesis Closure Rates

4. Measure What Didn't Get Built

5. Adopt the Post-Build Filter

6. Accept Irreducible Uncertainty

The Honest Answer

The why for Usage Based Billing for GitHub Copilot

Quick recap of copilots current pricing model

Usages of PRUs

How GitHub pays for model usage

Other ways GitHub control costs

Recent creative uses

So what changes

The benefits

Additional changes for Business accounts

Don't hamstring your users, educate them

Automatic versioning for GitHub Actions authors

What does it do exactly?

What's new

Recommended usage

Improved Dependency Submission for GitHub Actions

Background

Impacted features

Solution

How to

GitHub Billing - Let Organization and Repository admins set the Cost Center in GitHub

Creating the Custom Property

The reason for Inherit

Making it all work using GitHub Actions

GitHub Actions learnings from the recent nx hack

Understanding the vulnerable workflow

Understanding the vulnerable (human) process

Remediation

Change the default permissions for GitHub Actions to read-only

Set workflow permissions at the job level

Enable branch rulesets to require pull requests for all important branches.

Require approval for running fork pull request workflows from contributors

Do not rely on pull_request_target

Do not use workflow_dispatch or repository_dispatch triggers in workflows that have access to critical secrets.

Do not mix pull-request validation and continuous integration workflows

Store critical secrets in environment secrets, instead of repository secrets.

Pass secrets only to the steps that need them

Pass data to script steps using the environment

Pin all actions to their sha.

Enable GitHub Advanced Security Code Scanning for GitHub Actions and add Branch Ruleset.

Enable Dependabot for GitHub Actions to automatically update action references to the latest version.

Consider 3rd party security hardening tools like StepSecurity's Harden Runner.

Conclusion

Dear GitHub,

Authenticate Connect-MgGraph using OIDC in GitHub Actions

Picking the right GitHub Copilot model

GPT-4.5

Pros:

Cons:

GPT-4o

Pros:

Cons:

Recommendation:

Say goodbye to your Personal Access Tokens

PATs, PATs and more PATs - our 5 scenarios we used PATs for

PAT REST APIs to the rescue?

Scenario 1: Publish marketplace extensions without a PAT

Scenario 2: Call Azure DevOps APIs from a Azure Function app

Do not rely on `pull_request_target`

Do not use `workflow_dispatch` or `repository_dispatch` triggers in workflows that have access to critical secrets.