DEV Community: Alex Au

When Tenants Can See One Another: From Signed Artifacts to Authenticated State Transitions (Part 1)

Alex Au — Fri, 19 Jun 2026 17:02:13 +0000

TL;DR

We encrypted every competition submission and required a valid team signature. Yet another GitHub user could copy an unchanged public bundle and spend the original team’s daily quota.

This postmortem explains why the cryptography worked, why the system still failed, and why secure public workflows must authenticate state transitions rather than artifacts alone.

Postmortem status: The workshop has ended, and the affected scoring workflows have been manually disabled. There is no active scoring endpoint associated with this design.

How a valid digital signature became a transferable authorization token in a public GitHub competition

We encrypted the submission.

We signed its manifest.

We verified the signature against a team certificate.

The signature was valid.

And yet, another GitHub user could copy the complete submission into a new pull request and make the original team spend another scoring attempt.

No private key had to be stolen.

No signature had to be forged.

No ciphertext had to be decrypted.

The cryptography did exactly what we asked it to do.

The problem was that we had asked the wrong security question.

A GitHub-native data-cleaning tournament

In May 2026, we ran a data-cleaning tournament as part of a Python workshop.

Each team wrote a Python cleaner that transformed public training and test feature files. The official scorer then trained a fixed model, evaluated the cleaned test data against hidden labels, and recorded the result on a shared leaderboard.

We wanted several properties:

teams should not be able to inspect the hidden test labels;
organizers should be able to execute each cleaner consistently;
teams should not need to reveal their cleaner source code publicly;
only an authorized team should be able to submit under its assigned team identity;
each team should receive only a limited number of scored attempts per day.

GitHub already provided most of the surrounding infrastructure:

participant identity;
forks and pull requests;
workflow events;
hosted runners;
workflow artifacts;
an auditable repository history;
GitHub Pages for the leaderboard.

So we used pull requests as the submission transport.

The system looked approximately like this:

Team writes clean.py
        │
        │ encrypt to organizer certificate
        ▼
submission/clean.py.cms
        │
        │ SHA-256
        ▼
submission/manifest.json
        │
        │ sign with team private key
        ▼
submission/manifest.sig
        │
        │ public pull request
        ▼
GitHub Actions
        │
        ├── verify manifest signature
        ├── decrypt cleaner
        ├── execute cleaner
        ├── encrypt cleaned outputs
        └── pass outputs to trusted scorer
                │
                ├── decrypt hidden labels
                ├── train fixed model
                ├── calculate score
                └── update leaderboard

An official submission contained the encrypted cleaner, the manifest, its detached signature, and the dependency files required to execute the cleaner.

The repository’s helper script encrypted clean.py, calculated the ciphertext’s SHA-256 digest, placed that digest in the manifest, and signed the manifest using the team’s private key.

At first glance, this appeared to provide a clean chain of trust.

What exactly did the signature authenticate?

A simplified manifest looked like this:

{
  "team_id": "team-06",
  "code_path": "submission/clean.py.cms",
  "code_sha256": "b7a4...64 hexadecimal characters..."
}

The verification script performed several sensible checks:

validate the format of team_id;
require the code path to be submission/clean.py.cms;
calculate the actual SHA-256 digest of the encrypted cleaner;
require that digest to match code_sha256;
look up the team’s certificate in an allowlist;
validate that certificate against the tournament CA;
verify the detached signature over the manifest.

The workflow then installed the submitted dependencies, decrypted the cleaner, ran it against the tournament data, validated its outputs, and uploaded encrypted scoring artifacts.

The verifier was therefore able to establish:

The holder of Team 06’s private key signed this exact manifest, and the encrypted cleaner currently present has the digest recorded in that manifest.

That statement was true.

But it was narrower than the statement we implicitly relied upon:

Team 06 is authorizing this particular GitHub pull request as a new scoring attempt.

Those are not the same statement.

The missing subject: who is presenting the signed artifact?

A digital signature binds a key to some bytes.

It does not automatically bind those bytes to every context in which they might later appear.

Our signed manifest included:

team identity
encrypted-code path
encrypted-code digest

It did not include, or otherwise bind itself to:

the current GitHub PR author
the author’s immutable GitHub user ID
the source repository
the pull request number
the current head commit
a unique logical attempt ID
an expiry time
a one-time challenge

More importantly, the workflow did not independently compare the current PR actor or source repository with an identity authorized to submit for the signed team_id.

The workflow took the submitted files from the pull request’s head repository and verified the team manifest, but the verification decision remained entirely about the artifact. It did not ask whether the GitHub user presenting that artifact was entitled to act for the team named inside it.

In effect, our valid signed bundle had become a bearer artifact:

Anyone possessing an unchanged copy could present it, while the system continued attributing the action to the original signing team.

The signature was public evidence of past authorization.

We had accidentally treated it as reusable authority for a new action.

The Threat Model (Replay Attack)

Consider two participants:

Alice   — a legitimate member of Team A
Charlie — a different GitHub user

Alice prepares a valid submission:

submission/clean.py.cms
submission/manifest.json
submission/manifest.sig

Her manifest says:

{
  "team_id": "team-a",
  "code_path": "submission/clean.py.cms",
  "code_sha256": "H"
}

Team A’s private key signs that manifest.

Alice opens a pull request.

The submission is validly scored.

Because the pull request belongs to a public repository, Charlie can obtain the submitted encrypted cleaner, manifest, signature, and associated public dependency files.

Charlie does not edit them.

He does not need to understand the cleaner.

He simply places the same files in another branch or fork and opens another pull request.

The verifier now observes:

Manifest signature: valid
Team certificate: valid
Ciphertext SHA-256: matches
Team ID: team-a

Every cryptographic check succeeds, because the signed artifact has not changed.

The trusted scorer subsequently obtains team_id from that manifest. It counts the day’s previous leaderboard submissions carrying the same team ID, checks the configured daily limit, and appends each accepted result as another submission for that team.

Therefore Charlie’s replay is charged to Team A.

Repeated replays can consume Team A’s available daily attempts.

After that, Alice may be unable to submit another scored attempt that day.

The problem was not duplicate content

This distinction matters.

It may be perfectly legitimate for Alice to submit exactly the same cleaner twice.

Perhaps she wants to confirm reproducibility. Perhaps she accidentally changed something elsewhere. Perhaps competition policy simply says that every intentional submission consumes one attempt, even when the code is identical.

The desired policy may be:

Alice intentionally submits content H as attempt A-001
→ accepted and charged to Alice’s team

Alice intentionally submits content H again as attempt A-002
→ accepted and charged again to Alice’s team

The vulnerability was not that identical content could ever be processed more than once.

The vulnerability was:

Charlie could make that decision on Alice’s behalf.

A secure redesign therefore should not necessarily deduplicate by file hash.

It should distinguish between:

content identity — whether two submissions contain the same files;
attempt identity — whether they represent the same logical submission attempt;
principal identity — who is requesting the attempt;
team attribution — which team the authenticated principal belongs to. An explicit, signed attempt identifier can allow Alice to resubmit the same content intentionally, while still making workflow retries idempotent.

Binding the request to Alice’s authenticated GitHub identity prevents Charlie from replaying Alice’s signed attempt.

Why stronger encryption would not have fixed it

It is tempting to look at a replay involving an encrypted file and conclude that the encryption mechanism needs to be strengthened.

But confidentiality was not the failed property.

Charlie never learned the contents of the encrypted cleaner.

Changing the encryption algorithm would not stop him from copying the exact ciphertext.

Authenticated encryption can detect modification. It does not, by itself, prevent an unchanged valid ciphertext from being presented again.

Likewise, the digital signature correctly prevented Charlie from changing:

team-a

into:

team-charlie

Any such modification would invalidate Team A’s signature.

But Charlie did not need to modify anything.

Replay attacks live in the gap between:

This message is authentic.

and:

This message is authorized here, now, by the actor currently presenting it.

Cryptographic authenticity is one input to authorization. It is not the complete authorization decision.

Public multi-tenancy changes the threat model

In many multi-tenant systems, tenants are separated by confidentiality boundaries.

Tenant A cannot inspect Tenant B’s records, credentials, or requests.

Our environment was different.

Every team interacted through the same public repository. Pull requests were visible. Signed artifacts were visible. Encrypted submissions were visible. Tenants could inspect and copy one another’s transport-level records.

This was not an accidental data leak. Public visibility was part of the platform model.

That leads to a useful design rule:

In a publicly observable multi-tenant system, every public artifact must be assumed copyable and replayable.

Security cannot depend on another tenant being unable to obtain:

a manifest
a public key
a signature
an encrypted bundle
a previous request

Instead, each proposed action must be evaluated using both:

the submitted artifact
+
the authenticated context in which it is being presented

For GitHub, that context may include the event’s immutable actor ID, repository ID, pull request metadata, a protected identity registry, and the current authoritative system state.

The root cause in one sentence

The root cause was:

We treated possession of a valid signed artifact as authorization to initiate a new state transition.

The signature answered:

Who signed these bytes?

The scorer needed answers to several additional questions:

Who is presenting them now?

Is that GitHub identity registered to the signing key?

Does that identity belong to the claimed team?

Is this a new intentional attempt or a retry of an existing attempt?

Is the requested transition allowed under the team’s current quota?

Which authoritative state should be updated?

Those questions belong to an authorization and state-management layer, not to the signature algorithm alone.

Operational response

By the time of this publication, the workshop had ended and the relevant cleaner and scoring workflows had been manually disabled. GitHub’s workflow pages show the affected workflows as disabled, so the historical design is no longer an active scoring surface.

We are publishing this as a postmortem rather than as a vulnerability in GitHub.

GitHub Actions behaved according to its configuration.

The design error existed in our application-level trust model:

public pull requests
+
reusable signed artifacts
+
team-based quotas
+
no binding between the current actor and the signed team identity

From signed artifacts to authenticated state transitions

The redesigned system begins with a different abstraction.

A pull request does not directly become a registration or submission.

Instead, it is an untrusted proposal for a state transition.

Public PR artifact
        +
Authenticated GitHub actor
        +
Registered participant key
        +
Immutable team membership
        +
Current quota and attempt state
        ↓
Policy verification
        ↓
Authorized state transition

The important question is no longer only:

Is this signature valid?

It becomes:

Is this authenticated actor authorized to request this exact transition in the system’s current state?

The next part of this series will examine the first half of that redesign:

self-service participant onboarding;
binding a GitHub identity to a participant public key;
allowing organizers to accept previously unknown participants;
requiring every member to consent to the same complete team manifest;
making team membership immutable for the duration of the competition;
storing the resulting identity and team records as auditable authoritative state.

The final part will return to submissions:

actor-bound submission manifests;
explicit attempt identifiers;
intentional resubmission of identical content;
cross-principal replay prevention;
quota reservation;
concurrency;
and scoring attribution.

Closing thought

The most important lesson was not about a particular encryption mode or signature algorithm.

It was this:

A valid signature proves that a key endorsed some bytes. It does not prove that the person presenting those bytes now is entitled to trigger a new action.

In a system where every tenant can see every other tenant’s artifacts, signed objects should be treated as public claims—not as bearer authority.

Authentication, authorization, freshness, identity binding, and state transition policy must still happen at the moment the artifact is used.

Zero-Trust at the Edge: Rethinking the eDMZ Perimeter (Part 1)

Alex Au — Mon, 09 Mar 2026 15:58:52 +0000

Evolving the Asymmetric WAF-Pass Architecture for Speed and Scale

A few months ago, cloud security architect Kevin Yu published an excellent article titled 'Designing Asymmetric WAF-Pass JWT Assertion'. He highlighted a massive, often-ignored vulnerability in modern cloud architectures: the reliance on static custom headers (e.g., X-WAF-Checked: true) to verify that traffic hitting an Origin actually passed through the CDN and Web Application Firewall (WAF).

Kevin is absolutely right about the problem. Static headers provide zero cryptographic integrity. They are essentially shared passwords; if they leak, your WAF is permanently bypassed, and your Origin is exposed to the open internet.

To solve this, Kevin proposed an innovative architecture:

Using a Lambda@Edge function to make a synchronous network call to a Regional API Gateway
Triggering a Regional Lambda, which calls AWS KMS to generate an Asymmetric JWT
Passing it all the way back to the Edge to be forwarded to the Origin.

It is a highly secure, effective solution that brilliantly solves 95% of the architectural challenge. But as a Cybersecurity-backgrounded Cloud Platform Engineer who spends my days obsessing over system performance and cloud unit economics, it got me thinking:

What if we could extend this design to 99.9% operational efficiency — maintaining that Zero-Trust perimeter with just a fraction of the latency and cost?

Security at the edge is a delicate balancing act. In this three-part series, we are going to break down the cryptography of the Edge perimeter. I will propose a cloud-native, symmetric alternative that achieves this Zero-Trust pattern entirely at the Edge, open-source a custom benchmarking engine to prove the latency savings mathematically, and dive into the FinOps reality of securing the eDMZ.

The Threat Model of the eDMZ

Before we talk about cryptography, we must define the exact threat model we are solving.

When you place a WAF in front of an Application Load Balancer (ALB), EC2 instance, or custom cloud backend, you are creating an Enterprise DMZ (eDMZ). The Origin backend must have absolute mathematical certainty of two things before accepting a request:

The request definitely passed through CloudFront and AWS WAF.
The routing parameters (URI, Headers) have not been tampered with in transit.

If an attacker discovers your Origin's raw IP address or public DNS, they will attempt to bypass the WAF entirely. We need a way for the Edge to mathematically "sign" the request so the Origin can definitively trust its provenance.

The Cryptography Refresher: Asymmetric vs. Symmetric

Kevin’s architecture relies on Asymmetric Cryptography (RSA/ECDSA) to generate JWTs via AWS KMS.

Asymmetric cryptography is a beautiful mathematical construct, perfect for human identity verification (like OIDC) or highly distributed claims where the verifier cannot be trusted with the signing key. However, for Machine-to-Machine (M2M) perimeter routing inside the same organization, it can be computationally heavy.

For a fast, internal perimeter, Time-Based Symmetric Cryptography (HMAC-SHA256) is an incredibly powerful alternative. This is the exact cryptographic algorithm AWS uses for SigV4 to authenticate billions of internal API requests per second.

🔬 Crypto Deep-Dive: The Math of HMAC-SHA256

When designing secure cloud perimeters, engineers often confuse confidentiality with integrity. We frequently hear acronyms like IND-CPA (Indistinguishability under Chosen Plaintext Attack) or IND-CCA — but those are security models for encryption schemes. In eDMZ routing, we aren't encrypting the HTTP headers; we are authenticating them.

Because HMAC-SHA256 is a Message Authentication Code (MAC) rather than an encryption cipher, the mathematically correct security proof is EUF-CMA (Existential Unforgeability under Chosen Message Attack). In the realm of Authenticated Encryption, this provides the critical INT-CTXT (Integrity of Ciphertext/Tag) guarantee.

What does this mean in plain English? It means that even if a hacker sits on the network and captures 10 million valid HTTP requests passing through your CDN (the "chosen messages"), along with their valid HMAC signatures, the mathematically chaotic avalanche effect of SHA-256 ensures they learn absolutely zero bits of information about your secret key. They cannot reverse-engineer the key, and they cannot forge a valid signature for a slightly modified URI or payload.

Because symmetric math relies on rapid bitwise operations (XOR, AND, bit-shifts) rather than the heavy prime factorization or elliptic curve scalar multiplication used in Asymmetric RSA/ECDSA, this EUF-CMA proof can be calculated by a V8 engine in less than a millisecond.

By taking a highly secure shared 256-bit Symmetric Key and hashing the URI, critical Headers, and a tightly constrained timestamp header (e.g., x-cloud-date), we generate an unforgeable signature. You achieve enterprise-grade cryptographic integrity, but at a fraction of the computational cost.

The Proposed Architecture: CloudFront Functions + KVS

The main friction point in using KMS for Edge validation is state and distance. Making a regional API call from an Edge location natively introduces cross-ocean latency.

Instead of heavy Lambda@Edge functions fetching regional secrets, we can execute this Zero-Trust pattern using CloudFront Functions (CFF) and CloudFront KeyValueStore (KVS).

The Compute: We write the HMAC-SHA256 mathematical signing logic in vanilla JavaScript and execute it directly within the CFF V8 JavaScript Engine isolates.

The Key Rotation: We store the Symmetric Key in CloudFront KVS. KVS allows for instant, global key rotation via API, mimicking the operational security benefits of AWS KMS, but doing so natively at the Edge with sub-millisecond read latency.

So... Why isn't this architecture adopted by Kevin?

I want to be completely transparent: my symmetric CFF solution is not a silver bullet. Engineering is always about risk and balances.

My architecture relies on a few critical assumptions and tradeoffs that security engineers must understand:

1. The Bayes Theorem & Execution Order

In AWS, CloudFront Functions run at the Viewer Request phase (before the WAF). Lambda@Edge can run at the Origin Request phase (after the WAF).

We want to ensure: $P r (Passed WAF ∣ Has Signature) \approx 1$

Kevin's Architecture (Post-WAF): Because execution is sequential, if the signature exists, the WAF must have already passed. The probability $P r (Passed WAF ∣ Has Signature) = 1$ by definition.

My Architecture (Pre-WAF): Because CFF signs the request "blindly" before inspection, we must apply Bayes' Theorem.

P r (Passed WAF ∣ Has Signature) = \frac{P r ( Has Signature ∣ Passed WAF ) \cdot P r ( Passed WAF )}{P r ( Has Signature )} = P r (Passed WAF)

To maintain Zero-Trust integrity, you must rigorously ensure your WAF processes every signed request.

2. Replay Attacks and Missing Payloads

CloudFront Functions cannot read or hash the request body. This means the payload is unprotected by the signature.

To mitigate this and prevent Replay Attacks, your HMAC-SHA256 signature must be strictly time-bound. A 30-to-60 second TTL on the timestamp header ensures that even if an attacker intercepts a signed header, the window to exploit it is practically closed.

3. Why Kevin's Architecture might be a Better Choice

If your backend APIs involve heavy asynchronous processing that takes 5 to 10 seconds to resolve, latency is not your primary concern. In those scenarios, Kevin’s Asymmetric KMS approach is the better fit. The latency overhead won't be noticed, and asymmetric cryptography executed post-WAF provides stronger isolation without relying on WAF configuration hygiene.

However, if your APIs are

highly latency-sensitive,
read-heavy, or
serving dynamic content,

the unit economics and speed of the CFF pattern are mathematically undeniable.

Coming Up in Part 2: The Benchmarks

Theory is great, but hard data is better. In Part 2 of this series, I am going to open-source a custom-built, TypeScript benchmarking engine.

Most CDN benchmarks are deeply flawed by TCP Keep-Alive hoarding and submarine cable jitter. We are going to parse standard deviations, isolate pure V8 engine compute overhead, and prove exactly how much latency these architectures add.

Spoiler alert: We reduced the latency penalty from ~100ms down to 2.5ms and a substantial price drop.

Stay tuned.

Acknowledgments & Notes

A Special Thanks:

I want to extend a massive thank you to my employer, HKET, and my supervisor, Jack Yeung. Fostering an engineering culture that actively supports open-source benchmarking, technical knowledge sharing, and public writing is rare. Their full backing and encouragement made this research and blog series possible.

Author's Note: The architectural concepts, cryptography math, code, and benchmarking data in this series are 100% my original work. I utilized an LLM strictly as an editorial assistant to help structure and refine the prose of this article.