DEV Community: kobima6813

15,000 e-invoices, 12 generators: 4 setups pass every validator and still bill the wrong amount

kobima6813 — Sun, 25 Oct 2026 01:45:58 +0000

Finding first: in our October run, 4 generator configurations pass KoSIT (XRechnung 3.0.2), the CEN EN 16931 rules and Peppol BIS 3.0.20, yet on 472 of 600 invoices each the amount due (BT-115) differs from what the shop order actually charged.

No validator catches this, because no validator knows the original order total. It only checks that the invoice is consistent with itself.

What we ran

15,000 synthetic invoices from 12 open-source generators (25 configurations × 600 orders: 300 B2B + 300 B2C).
Validators are pinned by SHA-256: KoSIT 1.6.3, CEN EN 16931 and Peppol BIS 3.0.20.
The orders are deterministic, tax-inclusive shop carts with % discounts and shipping, i.e. what a WooCommerce or Odoo shop actually produces.

Full report: https://kobima6813.github.io/einvoice-conformance-matrix/reports/2026-10/

The "passes but wrong" cases

Configuration	Validators	Amount due ≠ order total
Mustang 2.26.0 (library-computed totals)	all pass	472 / 600
speedata/einvoice 0.0.23, CII	all pass	472 / 600
speedata/einvoice 0.0.23, UBL	all pass	472 / 600
josemmo/einvoicing 0.3.1, 2-decimal rounding	all pass	472 / 600

The cause is the same in all four. A gross (tax-inclusive) cart is converted into EN 16931 net arithmetic, and rounding lands in a different place than the shop rounded. The amounts are usually cents, but the invoice no longer matches what the customer paid, and that breaks automatic payment matching. Mustang already merged a fix (PR #1318). It isn't released yet.

Validator failures we reported upstream

Odoo 18/19 Factur-X: PEPPOL-EN16931-R120 on 242/300 B2B invoices and BR-CO-10 / BR-S-08 with round_globally. We tested two fix PRs (#297750, #297752) against the corpus, and the errors go to 0. The maintainer confirmed the results.
WooCommerce PDF Invoices & Packing Slips 5.16.4: ram:RoundingAmount is out of XSD order, so 238/600 Factur-X invoices are XSD-invalid (#1662). Another user reproduced it independently.
horstoeko/zugferd: the VAT breakdown for exempt categories was wrong. It's fixed in v1.0.133.
speedata/einvoice: UBL element order and a BT-27/BT-28 swap. Both are fixed in v0.0.23.
josemmo/einvoicing 0.3.1: the default config gives 590/600 CEN failures (#93). It passes once a rounding preset is set.
ZUGFeRD-csharp 18.0.1: CII without a delivery date is XSD-invalid. This was already known upstream (PR #981), so it's not our report.

France: 30 of 30 pass

France's receiving obligation started on 1 Sep 2026. We added 5 French scenarios: standard 20 % + 5.5 %, the VAT franchise (art. 293 B), domestic reverse charge, an intra-EU supply and all four French rates. Each one ran in 2 Factur-X profiles (EN16931, EXTENDED) on 3 generator versions (drafthorse + factur-x, horstoeko 1.0.132 and 1.0.133). All 30 pass.

Honest caveats:

In our first run, horstoeko failed BR-IC-12. That was our driver's mistake: the ship-to address is dropped unless setDocumentShipTo() is called before setDocumentShipToAddress(). It was not a library bug.
We have not tested the French platform-specific (PA/PPF) rule sets, because they aren't available as free, pinned validators.

Free self-check

You can check one of your own invoices here: https://kobima6813.github.io/einvoice-conformance-matrix/check.html

It runs in your browser, and the file is not uploaded anywhere. It covers a subset of EN 16931 and tells you how many problems it found. It is not a replacement for KoSIT or a Peppol validator.

Limits

The orders are synthetic. Real catalogs have edge cases we don't generate yet.
Each configuration is one library version with one setting. Your integration code can add or hide bugs.
Corrections are welcome. Every number links to a file in the repo, and you can reproduce it with scripts/run_matrix.sh.

Repo and home page: https://kobima6813.github.io/einvoice-conformance-matrix/

If you maintain one of these generators and think a result is wrong, open an issue and I'll re-run it.

We ran 12,000 e-invoices through KoSIT, EN 16931 and Peppol. Passing the validator isn't enough

kobima6813 — Sat, 24 Oct 2026 17:00:54 +0000

Germany's B2B e-invoice transition ends on 1 Jan 2027 (1 Jan 2028 for issuers with turnover up to EUR 800k). France starts requiring large and mid-size companies to issue on 1 Sep 2026, and Belgium has been live since 1 Jan 2026. A lot of shops will be relying on an open-source library or plugin to produce XRechnung, ZUGFeRD/Factur-X or Peppol UBL.

So we built an open, reproducible matrix. It takes the same 600 synthetic tax-inclusive shop orders (19 %/7 % VAT, percentage coupons, shipping), runs them through each generator, and checks every invoice with pinned official rule sets: KoSIT 1.6.3 + XRechnung 3.0.2, CEN EN 16931 and Peppol BIS 3.0.20. On top of that we check whether the amount due matches what the customer was actually charged.

What we found (full table: https://kobima6813.github.io/einvoice-conformance-matrix/):

WooCommerce PDF Invoices 5.16.4, Factur-X: 39.7 % schema-invalid. RoundingAmount is written in the wrong XSD position (reported upstream).
Odoo 19, Factur-X: with round_globally, 32.7 % (B2B) and 42.7 % (B2C) fail CEN calculation rules, and 80.7 % of B2B invoices fail Peppol R120. A follow-up fix is in progress upstream.
Mustang 2.26.0 and josemmo/einvoicing 0.3.1: both pass KoSIT 100 %, but when the library computes totals from net prices, the amount due differs from the charged amount in 90.7 % (B2B) and 66.7 % (B2C) of orders. Usually that is a few cents. It's the gap between tax-inclusive shop arithmetic and EN 16931 net arithmetic, and no validator flags it.
josemmo/einvoicing with no preset or rounding matrix: amounts are written with up to 8 decimals, so 98–99 % fail BR-DEC rules. Fix: setRoundingMatrix(['' => 2]).
speedata/einvoice 0.0.23 (Go) computes the VAT breakdown and totals, and every CII/UBL invoice passes. The amount-due gap is the same as with Mustang.
ZUGFeRD-csharp 18.0.1 (.NET): UBL passes. CII without a delivery date is 100 % schema-invalid because the required empty ApplicableHeaderTradeDelivery is dropped. That's already known upstream (PR #981). Workaround: set ActualDeliveryDate.
horstoeko/zugferd, drafthorse and ZUGFeRD-csharp: given correct totals, everything passes. The arithmetic is your job.

Several bugs found this way are already fixed in releases (speedata/einvoice v0.0.23, horstoeko/zugferd 1.0.133, Mustang master).

Run the same checks on your own invoices in CI:

- uses: kobima6813/einvoice-conformance-matrix@main
  with:
    invoices: out/invoices

Maintain a generator? Listing is free: open a Listing request issue or a PR (see CONTRIBUTING). Corrections are welcome.

Repo: https://github.com/kobima6813/einvoice-conformance-matrix

How to stop secrets leaking into GitHub Actions logs (and move to OIDC)

kobima6813 — Fri, 23 Oct 2026 11:58:27 +0000

After the Railway incident I wrote a 15-minute key rotation checklist. Rotating is step one. Step two is making sure the new keys don't leak again, and CI is one of the easiest places to leak them.

Here are the three patterns I see most often, and how to fix each one.

1. Printing secrets in `run:` steps

- run: echo "key is ${{ secrets.DEPLOY_KEY }}"
- run: printf '%s\n' "${{ secrets.DB_PASSWORD }}"

GitHub masks known secret values in logs, but masking is best-effort. If the value is base64-encoded, split, URL-encoded or otherwise transformed, the masked form no longer matches and the real value can show up in the log.

Fix: never echo secrets. Pass them through env: on the step that needs them and let the tool read the variable:

- name: Deploy
  env:
    DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
  run: ./deploy.sh   # reads $DEPLOY_KEY, never prints it

If you create a derived secret at runtime, mask it explicitly before anything can print it:

echo "::add-mask::$DERIVED_TOKEN"

2. Secrets in workflow-level `env:`

env:
  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
jobs:
  build:
    steps:
      - uses: some/third-party-action@v1   # can read OPENAI_API_KEY too

A secret in top-level env: is available to every step of every job, including third-party actions that have nothing to do with it.

Fix: scope the secret to the single step that needs it (as in the example above).

3. Long-lived cloud keys instead of OIDC

- uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: eu-west-1

A static key stored as a secret is exactly the thing you have to rotate after a breach. With OIDC, GitHub issues a short-lived token per job and your cloud exchanges it for temporary credentials. There is no long-lived key to steal.

AWS:

permissions:
  id-token: write
  contents: read
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/gha-deploy
      aws-region: eu-west-1

You create an IAM OIDC identity provider for token.actions.githubusercontent.com and a role whose trust policy limits which repo/branch can assume it.

GCP (Workload Identity Federation):

permissions:
  id-token: write
  contents: read
steps:
  - uses: google-github-actions/auth@v2
    with:
      workload_identity_provider: projects/123/locations/global/workloadIdentityPools/gh/providers/github
      service_account: deployer@my-project.iam.gserviceaccount.com

Azure: use azure/login with client-id, tenant-id and subscription-id plus a federated credential on the app registration, instead of a creds JSON secret.

Always restrict the trust policy to your repo (and ideally branch or environment). An OIDC role that any repo can assume is worse than no OIDC.

Catch these automatically

I built actions-harden, a zero-dependency CLI and GitHub Action that flags these patterns (plus unpinned actions, missing permissions and script injection). Real output on a deliberately bad workflow:

secrets_bad.yml:7: [MEDIUM] workflow-env-secret: secrets.OPENAI_API_KEY is set in workflow-level env:, exposing it to every step of every job (including third-party actions). Scope it to the step that needs it.
secrets_bad.yml:15: [MEDIUM] long-lived-cloud-key: Long-lived AWS key from secrets. Prefer short-lived OIDC credentials: aws-actions/configure-aws-credentials with 'role-to-assume' + 'permissions: id-token: write'.
secrets_bad.yml:20: [MEDIUM] long-lived-cloud-key: Long-lived GCP key from secrets. Prefer short-lived OIDC credentials: google-github-actions/auth with 'workload_identity_provider' (Workload Identity Federation).
secrets_bad.yml:21: [HIGH] secret-in-log: secrets.DEPLOY_KEY is printed in a run: step. Masking is best-effort (encoded/split values leak); never echo secrets.

Add it to CI:

permissions:
  contents: read
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: kobima6813/actions-harden@main
        with:
          fail-on: high

Quick checklist

[ ] No echo/printf of ${{ secrets.* }} in run: steps
[ ] Secrets scoped to the step, not workflow-level env:
[ ] ::add-mask:: for any secret derived at runtime
[ ] AWS/GCP/Azure via OIDC, static keys deleted
[ ] OIDC trust limited to your repo/branch
[ ] permissions: set explicitly, id-token: write only where needed

If you're still rotating after the Railway breach, start with the checklist, then do this so you don't have to do it twice.

Railway was breached: a 15-minute key rotation checklist

kobima6813 — Thu, 22 Oct 2026 17:34:32 +0000

On October 21, 2026 Railway published a security disclosure: it detected anomalous activity involving the systems that store customer environment variables, and says an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents.

Railway's recommendation is to roll all environment variables stored in Railway, including API keys and private registry credentials, and review your service logs for unusual activity. Railway has also invalidated all Railway API tokens and Project Tokens created before the incident, and rotated storage bucket credentials. Live updates are on the status page.

The advice is "rotate everything". This post turns that into a checklist you can actually work through. It's written for a small team with a handful of services.

Note: at the time of writing, builds for some plans are still being re-enabled in stages, so you may not be able to redeploy right away. You can still revoke and create keys now. Do the provider-side work first and redeploy when you can.

0. Inventory (3 minutes)

For each Railway project and service, list the variable names (not values) and where each one comes from:

# Railway CLI: create a NEW token first, old ones were invalidated
railway variables --kv | cut -d= -f1 | sort

Group them by provider: LLM APIs, payments, databases, cloud, GitHub, messaging, everything else. That list is your checklist. Don't paste values into chats, tickets or terminals that are being logged.

1. Rotate the high-value keys first

Do the order by blast radius. Keys that spend money or touch customer data come first.

AI inference keys (OpenAI, Anthropic, OpenRouter, xAI, etc.). Railway says it believes the attack targeted AI inference providers, and it has notified those providers to revoke keys proactively. So:

Create a new key, then delete the old one (not just "stop using it").
Check usage and billing dashboards for spend you don't recognise.
Set or lower spending limits while you're there.
Railway's guides: OpenAI, Anthropic, OpenRouter, Grok.

Stripe and other payment keys.

Roll the secret key and any restricted keys. Re-create webhook signing secrets.
Review recent API request logs and events for activity you didn't trigger. See the Stripe guide.

Databases. Change the password or create a new user, update DATABASE_URL, then drop the old credentials. Railway has guides for PostgreSQL, MySQL, MongoDB and Redis. If the database is reachable from the internet, check its connection logs.

Cloud (AWS / GCP / Azure). Deactivate the IAM access key or service-account key and create a new one. Then review CloudTrail or the audit logs for calls you didn't make. Guides: AWS, GCP.

GitHub tokens. Revoke any PAT or GitHub App private key that was stored as a variable, and check your org audit log and security log. Railway also asks you to reconnect GitHub and redeploy buckets after its credential reset. See the guide and the GitHub access guide.

2. Then the long tail

Bot tokens (Telegram: /revoke via BotFather; Slack and Discord: regenerate the token).
Email/SMS providers (SendGrid, Postmark, Twilio, etc.).
Private registry credentials (npm, Docker Hub, GHCR). Railway calls these out explicitly.
Your own app secrets: session/JWT signing keys, encryption salts, internal service-to-service tokens. Rotating these will log users out, so plan for it.
Third-party APIs (maps, search, scraping, etc.).

For each one: create new → update the Railway variable → revoke old → note the time.

3. Recreate Railway tokens

Railway's own API tokens and Project Tokens created before the incident no longer work. Anything that deploys to Railway (CI, scripts, the CLI) needs a new token. Make it project-scoped, not account-wide, where you can.

4. Redeploy and verify

When builds are available for your plan, redeploy so services pick up the new values. Then:

Hit your health checks.
Confirm the old keys are actually dead (the provider dashboard should show them as revoked).
Review the service logs from the incident window for unusual activity, as Railway recommends.

5. Make the next one less painful

Most of the 15 minutes goes on long-lived keys that sit in too many places. Some fixes:

Use OIDC in CI instead of static cloud keys. GitHub Actions can get short-lived AWS, GCP or Azure credentials (role-to-assume, workload_identity_provider, federated client-id), so there's no key to steal.
Never print secrets in logs. echo ${{ secrets.X }} relies on log masking, and masking misses encoded or transformed values.
Scope secrets to the step that needs them, not workflow-level env: where every step and third-party action can read them.
Pin third-party actions to commit SHAs, so a moved tag can't read your secrets.
Keep the inventory from step 0 in your repo (names only). Next time it's a checklist, not an archaeology project.

I added checks for exactly these patterns to actions-harden, a free, zero-dependency scanner for GitHub Actions workflows. The new rules are secret-in-log, long-lived-cloud-key and workflow-env-secret, next to SHA pinning, permissions and script-injection checks:

- uses: kobima6813/actions-harden@main
  with:
    fail-on: medium

If you'd rather hear about CI breakages before they hit your builds, there's also CI Early-Warn.

Stay safe, and rotate before you redeploy.

How to tell if the ubuntu-latest 26.04 migration broke your build

kobima6813 — Tue, 20 Oct 2026 16:46:30 +0000

GitHub is moving the ubuntu-latest label from Ubuntu 24.04 to Ubuntu 26.04. The rollout is gradual and runs between October 19 and November 19, 2026 (GitHub changelog, runner-images announcement #14748).

Because it is gradual, one run can land on 24.04 and the next on 26.04, with no change in your repo. If a build that was green yesterday is red today, this is how to check whether the image is the cause.

1. Check which image the job actually ran on

Open the failed run and expand the Set up job step. You will see something like:

Runner Image
  Image: ubuntu-26.04
  Version: 20261012.163.1

Compare it with the last green run. If the green run says ubuntu-24.04 and the red one says ubuntu-26.04, you have your prime suspect.

The same values are available as environment variables in every job, so you can print them explicitly:

- run: echo "ImageOS=$ImageOS ImageVersion=$ImageVersion" && lsb_release -ds

ImageOS is ubuntu24 or ubuntu26. Logging this in every job makes later debugging much faster.

2. Confirm by pinning (the fastest test)

Re-run the same commit with the old image:

runs-on: ubuntu-24.04   # instead of ubuntu-latest

Green on 24.04 and red on 26.04 means the image change is the cause.
Red on both means the cause is something else (a dependency release, a flaky test, your own change).

Pinning to ubuntu-24.04 is also the officially suggested escape hatch if you are not ready to move. Treat it as a temporary pin, not a permanent fix.

To test ahead of time, add a matrix leg:

strategy:
  matrix:
    os: [ubuntu-24.04, ubuntu-26.04]
runs-on: ${{ matrix.os }}

3. Diff the installed software

Each image has a software list in actions/runner-images:
Ubuntu2404-Readme.md and Ubuntu2604-Readme.md. Diff them for the tools your build uses, and check the releases page for the exact ImageVersion you got.

Differences we saw when comparing the two READMEs on 2026-10-20 (always re-check against the current files):

Area	Ubuntu 24.04	Ubuntu 26.04
Default Python	3.12	3.14
Default Node.js	22	24
Ruby / PHP / Perl	3.2 / 8.3 / 5.38	3.3 / 8.5 / 5.40
Clang	16–18	20–22
OpenSSL	3.0	3.5
Docker client	28	29
Kernel	6.x azure	7.0 azure
Java 8 (`JAVA_HOME_8_X64`)	present	not listed
Not listed on 26.04		Fastlane, Julia, Lerna, Mercurial, Miniconda, Newman, Parcel, Pulumi, MediaInfo, Haveged, Sphinx search server

4. Common breakages and quick fixes

"command not found" for a tool that used to be preinstalled: install it explicitly (setup-* action, apt-get install, pip/npm install) instead of relying on the image.
Python 3.14 / Node 24 default changes: pin the version with actions/setup-python / actions/setup-node rather than using whatever the image ships. Watch for removed stdlib/deprecated APIs and native modules that have no wheels/prebuilds yet.
Compiler upgrades (newer GCC/Clang): new warnings become errors under -Werror; old code may need flag changes.
OpenSSL 3.5: legacy algorithms or old TLS settings in tests can fail.
Java 8 missing: use actions/setup-java with java-version: 8.
apt package names changed or removed: apt-get install fails with "Unable to locate package". Check the package name for 26.04 (resolute).
Kernel / system changes: low-level things (eBPF, ACLs, FUSE, container runtimes) can behave differently. One example reported on day 2 of the rollout: setfacl fails on Ubuntu 26.04 (#15004).
Cache keys: if your cache key does not include the OS version, a 24.04 cache restored on 26.04 can break native builds. Include ${{ runner.os }}-${{ env.ImageOS }} in the key.

If you find a real regression in the image, search and report it in actions/runner-images issues with the ImageVersion from step 1.

5. Tell "it's the image" from "it's just me" across repos

The hard part in a gradual rollout is knowing whether a red build is your problem or everybody's. Disclosure: I built a small free tool for this, CI Early-Warn. A GitHub Action reports minimal job metadata (OS, image version, pass/fail, failing-step category; no code, logs or repo names). When failures across many repos jump on a new image version, it raises an alert, and says "ambiguous" when it can't tell the cause. It is early and only useful with enough participants, so steps 1–4 above are the reliable method.

TL;DR

Look at Set up job → Runner Image (or $ImageOS / $ImageVersion).
Re-run with runs-on: ubuntu-24.04.
Diff the runner-images READMEs for the tools you use.
Install/pin what you depend on explicitly, then move to 26.04 on your schedule.

Free CI cost audit: email your GitHub Actions workflow file(s) to conformance@mail.grokbot.com (subject "CI audit") and I'll reply within 2 business days with a short report on wasted runner minutes and concrete fixes. Workflow YAML only, no secrets or code; files deleted after 30 days.