DEV Community: Çalgan Aygün

Trust No Pipeline: Why Attackers Stopped Hacking Your Cluster and Started Riding Your CI/CD (And How to Stop Them on Google Cloud)

Çalgan Aygün — Sat, 07 Nov 2026 10:00:52 +0000

Sourced from: my talk "Trust No Pipeline: Securing Software Delivery & Workload Identity in the Cloud" (slides), the companion live demo repo, and the public 2025–2026 incident write-ups linked at the bottom.

So here is the thing. We spent the last decade turning our production clusters into fortresses. Private GKE nodes. VPCs. Cloud Armor. IAM policies reviewed by three different teams. Firewalls on top of firewalls.

And attackers looked at all of that and said: "Cool. We will not touch it."

Why would they? Why fight the fortress when you can hijack the delivery truck?

Your CI/CD pipeline is that truck. It drives through the front gate every ten minutes. Nobody checks it at the gate, because it is your truck. It carries roles/owner. It runs preinstall hooks from packages you have never read. It pulls GitHub Actions by mutable @v1 tags. And somewhere in its glovebox there is probably a Service Account JSON key that expires in the year 9999.

That is the whole plot twist of modern supply chain attacks. Nobody is breaking in. They are riding in.

And here is the kicker: this is not a theoretical "what if" talk. Every pattern in this post comes from a real incident in 2025 or 2026. Tens of thousands of repositories. Hundreds of npm packages. Billions of monthly downloads. All of it exploiting the same thing: we trust our pipelines way more than we should.

So let us stop doing that.

The Bait and Switch That Hit 23,000 Repos Overnight

Let me start with the one that made everyone pay attention.

March 2025. CVE-2025-30066. tj-actions/changed-files, a GitHub Action used by more than 23,000 repositories.

The attackers did not hack Google Cloud. They did not hack AWS. They did not even hack the victims. They stole a bot token, and used it to repoint every single version tag of the action -- @v1 all the way through @v45 -- to a malicious commit.

That commit ran a little Python script called memdump.py. It read /proc/$pid/mem of the Runner.Worker process and dumped everything it found straight into the build logs. Cleartext. GCP Service Account JSON keys, AWS keys, npm tokens, GitHub tokens. In public logs, for a lot of open-source projects.

If your workflow said uses: tj-actions/changed-files@v45, you were not running the code you reviewed. You were running whatever the tag pointed to that morning.

How they actually got there: six steps from SpotBugs to Coinbase

What makes this one fascinating is that it was not a single hack. It was a cascade:

SpotBugs PR trap. A maintainer PAT was added to spotbugs/sonar-findbugs, then stolen through a malicious pull_request_target PR.
Rogue invite and leak. On March 11, that PAT invited a rogue user (jurkaofavak) to spotbugs/spotbugs. Two minutes later a workflow leaked the Reviewdog maintainer's PAT, AES + RSA encrypted, so nobody would notice it in the logs.
reviewdog/action-setup@v1 poisoned. The leaked PAT overwrote the mutable v1 tag.
Pivot to tj-actions. tj-actions/eslint-changed-files used reviewdog/action-setup@v1 as a transitive dependency, which exposed the tj-actions privileges.
The actual target: coinbase/agentkit. Five days after Coinbase added tj-actions/changed-files, the attacker forked the relevant repos to stage the attack.
All @v1–@v45 repointed. And 23,000+ repositories came along for the ride.

Read that list again. Every single hop is a pipeline trusting another pipeline. Nobody exploited a buffer overflow. Nobody needed a zero-day. Mutable tags and over-privileged tokens did all the work.

Then the Worms Showed Up

If tj-actions was a targeted heist, 2025 and 2026 gave us something worse: self-propagating supply chain worms.

Shai-Hulud 2.0 (Sept–Nov 2025) ran inside preinstall hooks, fired up an embedded TruffleHog scanner to harvest cloud credentials (GOOGLE_CREDENTIALS, AWS keys, Vault tokens), then automatically republished trojanized versions of every package the victim owned. As a bonus, it registered persistent rogue self-hosted GitHub Actions runners named SHA1HULUD.

Mini Shai-Hulud (May 2026) was the streamlined sequel. Triggered via pull_request_target ("pwn requests"), it poisoned pnpm caches and dumped OIDC tokens directly from CI runner memory, hitting ecosystems including TanStack, Mistral AI, and UiPath.

ChainDrop (August 2026) compromised core JavaScript maintainer accounts -- most notably keyv (think flat-cache, cacheable-request) -- and spread to 400+ npm packages and 1,300+ versions. It used a preinstall hook (setup.mjs) to drop an obfuscated Bun payload on Linux, macOS, and Windows, and routed its command-and-control through Ethereum smart contracts so corporate DNS blocklists had nothing to block. It then committed malicious updates under the dummy author claude with the very innocent message chore: update config.

And the most uncomfortable lesson from ChainDrop? SLSA Level 2 did not save anyone. The maintainer's credentials were compromised before the build, so the provenance looked perfectly valid. The build system faithfully attested that it built the malware.

What do all of these have in common? They eat keys. Static keys, long-lived tokens, OIDC tokens sitting in runner memory. Starve them of keys and they stop being worms and start being very sad scripts.

The Four (Okay, Five) Pillars

The rest of this post is the defense. I structured the talk around four pillars, plus a "pillar zero" that I think gets ignored way too often. For each one I will show the idea, the actual config, and the attack from the demo repo failing against it.

Pillar 0: A Stolen Token Should Not Equal a Valid Commit

Here is something that does not get talked about enough. Anyone can create a commit with someone else's name and email:

git commit --author="Trusted Maintainer <maintainer@example.com>" -m "chore: update config"

Git will happily accept it. GitHub will happily display it. That is exactly how ChainDrop's claude commits blended in.

What we actually care about is authenticity (who really made this commit?) and integrity (has it been tampered with since?). Commit signing gives you both. If a signed commit is modified, the signature breaks immediately.

But here is where it gets good. If you enforce that only signatures from maintainers' physical YubiKey / FIDO keys are accepted, then a stolen GitHub PAT is no longer enough to land code. The attacker has your token. They do not have the piece of hardware on your keychain.

# 1. Lock Git to the maintainer's YubiKey GPG key
git config --global user.signingkey 0x9F8B1234
git config --global commit.gpgsign true
git config --global tag.gpgSign true

# 2. Every commit now requires a physical touch on the key
git commit -S -m "feat: release v1.0.0"

Then enforce it on both ends: turn on Settings → Rules → Require signed commits in GitHub, and verify against a known keyring in CI:

- name: Verify Maintainer Hardware GPG
  run: |
    gpg --import .github/maintainer-yubikeys.asc
    git verify-commit HEAD

Stolen PAT without the YubiKey? Rejected.

Pillar 1: Kill Static Secrets (Vampire Keys vs. Melting Ice Cubes)

Why did Shai-Hulud and ChainDrop do so much damage? Because teams still export GCP Service Account JSON keys.

A JSON key is a vampire. It lives until December 31, 9999 unless somebody drives a stake through its heart. It sits in a GitHub secret, gets loaded into the runner's environment, and waits for the first preinstall hook with a TruffleHog binary to come along and grab it.

An ephemeral token is an ice cube. It is issued on demand and melts in about an hour. Even if it ends up dumped in a public log, by the time someone finds it, it is a puddle.

On Google Cloud, you can kill the vampires for your entire project or organization with one Organization Policy. Five lines of YAML:

# disable-sa-keys.yaml
name: projects/${PROJECT_ID}/policies/iam.disableServiceAccountKeyCreation
spec:
  rules:
  - enforce: true

# 1. Audit what is already out there
gcloud iam service-accounts keys list \
  --iam-account="ci-deployer@${PROJECT_ID}.iam.gserviceaccount.com" \
  --managed-by=user \
  --format="table(name.basename(), validAfterTime, validBeforeTime, keyAlgorithm)"

# 2. Block key creation going forward
gcloud org-policies set-policy disable-sa-keys.yaml

While you are there, also look at iam.disableServiceAccountKeyUpload, and check IAM & Admin → Service Accounts → Keys for anything with that lovely 9999 expiry date. The Recommender insights (google.iam.serviceAccount.Insight) will tell you which keys and accounts have not been used in ages.

The demo: trying to mint a backdoor key

Attack vector 2 in the demo plays the attacker who already compromised the CI identity and wants persistence. The obvious move is to mint a long-lived key:

gcloud iam service-accounts keys create /tmp/backdoor-key.json \
  --iam-account=ci-prod-deployer@${PROJECT_ID}.iam.gserviceaccount.com

With the Org Policy enforced, GCP says no. No key, no persistence, no year-9999 souvenir.

Pillar 2: Workload Identity Federation (a.k.a. Passport Control)

"Okay, no JSON keys. Then how does my GitHub Actions workflow talk to Google Cloud?"

Workload Identity Federation (WIF). And the easiest way I have found to explain it is airport passport control:

The CI runner gets a signed OIDC JWT from GitHub for that specific job. That is the passport.
Google Cloud STS is customs. It checks the signature and the claims inside the passport against your rules.
If everything checks out, STS hands back a visitor badge -- a short-lived token valid for roughly 15–60 minutes.

There is nothing stored in GitHub. Nothing to steal from a secrets store. Nothing that lives until 9999.

So everybody migrated to WIF and lived happily ever after, right?

The #1 OIDC mistake: who is on the VIP list?

Not quite. This is where Nx s1ngularity (August 2025) and Mini Shai-Hulud come back in.

A lot of teams moved to WIF and configured their provider to check exactly one thing:

assertion.repository == 'acme/app'

Some checked only repository_owner, which is even worse.

The problem? A pull_request_target workflow, an unmerged feature branch, or a job on a rogue self-hosted runner still carries "repository": "acme/app" in its OIDC token. So your customs officer checks the passport, sees the right country, and waves through an untrusted PR to mint a production GCP token.

You moved away from static keys and then handed out VIP badges to anyone who can open a pull request.

The fix is a strict CEL attribute condition on the provider. Check the immutable numeric owner ID (it survives org renames), the exact repo, the protected branch, the event type, and -- because of SHA1HULUD -- the runner environment:

gcloud iam workload-identity-pools create "github-prod-pool" \
  --project="${PROJECT_ID}" \
  --location="global" \
  --display-name="GitHub Actions Prod Pool"

gcloud iam workload-identity-pools providers create-oidc "github-oidc-provider" \
  --project="${PROJECT_ID}" \
  --location="global" \
  --workload-identity-pool="github-prod-pool" \
  --display-name="GitHub Actions Hardened OIDC" \
  --issuer-uri="https://token.actions.githubusercontent.com" \
  --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository,attribute.ref=assertion.ref,attribute.workflow_ref=assertion.job_workflow_ref" \
  --attribute-condition="assertion.repository_owner_id == '${GITHUB_ORG_ID}' && assertion.repository == '${GITHUB_REPO}' && assertion.ref == 'refs/heads/main' && assertion.event_name == 'push' && assertion.runner_environment == 'github-hosted'"

You can tighten it further with assertion.job_workflow_ref to pin the exact workflow file allowed to deploy.

And you do not even need an intermediate service account anymore. Grant the role directly to the federated principal:

gcloud artifacts repositories add-iam-policy-binding "prod-containers" \
  --location="${REGION}" \
  --role="roles/artifactregistry.writer" \
  --member="principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/github-prod-pool/attribute.repository/${GITHUB_REPO}"

A pattern I like a lot on top of this: split identities by environment. A read-only terraform plan identity that PRs are allowed to use, and a separate apply / deploy identity restricted to the production GitHub Environment with required human reviewers.

The hardened workflow

On the GitHub side, this is what the deploy workflow looks like. Notice three things: least-privilege permissions, id-token: write only on the job that actually needs it, and every action pinned to a full 40-character commit SHA (this alone would have neutralized tj-actions):

name: Build and Deploy to GKE
on:
  push:
    branches: [ "main" ]

# Global default: least privilege
permissions:
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    permissions:
      contents: read
      id-token: write # ONLY on the job that exchanges OIDC tokens
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

      - run: npm ci --ignore-scripts

      - uses: google-github-actions/auth@6fc4af4b145ae7821d527454aa9bd537d1f2dc5f # v2.1.7
        with:
          workload_identity_provider: "projects/123456789/locations/global/workloadIdentityPools/github-prod-pool/providers/github-oidc-provider"

That --ignore-scripts is not decoration, by the way. More on that in a second.

The demo: forging a PR token at the STS border

Attack vector 3 builds an OIDC token that looks exactly like what Mini Shai-Hulud would bring to the border -- right repository, right owner ID, but:

{
  "iss": "https://token.actions.githubusercontent.com",
  "repository": "acme-corp/payments-api",
  "ref": "refs/pull/99/merge",
  "event_name": "pull_request_target",
  "runner_environment": "self-hosted"
}

It then sends it straight to https://sts.googleapis.com/v1/token asking for a cloud-platform access token. STS rejects the exchange. Customs did its job.

The repo also has a real GitHub workflow for this (.github/workflows/attack-sim-oidc-hijack.yml) that triggers on pull_request and tries to authenticate against the hardened provider. Open a PR, watch it fail at the GCP boundary. It is honestly one of the most satisfying red X's you will ever see in GitHub Actions.

The demo: the worm that never gets to run

Attack vector 1 is the ChainDrop / Shai-Hulud pattern in miniature. The demo package is about as small as it gets:

{
  "name": "chaindrop-shai-hulud-simulator",
  "private": true,
  "scripts": {
    "preinstall": "node setup.mjs"
  }
}

And setup.mjs does what the real worms did first -- it scans process.env for anything that smells like a credential (safely, values are redacted in the simulation):

const sensitiveVars = Object.keys(process.env).filter((k) =>
  /GOOGLE|GCP|AWS|TOKEN|SECRET|KEY|GITHUB|KUBE/i.test(k)
);

Run a plain npm install and the hook fires before a single dependency is even fetched. Run npm install --ignore-scripts (or drop ignore-scripts=true into your .npmrc) and it simply never executes. One flag. That is the difference between "CI runner leaked its environment" and "nothing happened".

Pillar 3: Stop Sharing the Master Key on GKE

So the pipeline is locked down. What about inside the cluster?

The legacy setup on GKE is that every Pod on a node inherits the node's service account. And very often that is the Compute Engine default service account, ...-compute@developer.gserviceaccount.com, sometimes with roles/editor and the full cloud-platform scope.

That is one giant master key on a lanyard, shared by every Pod on the node. One SSRF in one Pod, one call to the metadata server, and the attacker owns the whole project.

Workload Identity Federation for GKE flips this. One Pod, one scoped identity. Each Kubernetes ServiceAccount gets its own tiny, least-privilege IAM badge, and the GKE Metadata Server hands out short-lived tokens for that identity only.

And just like on the CI side, you no longer need an intermediate Google service account. Bind IAM directly to the Kubernetes ServiceAccount:

kubectl create namespace payments-prod
kubectl create serviceaccount payments-api-ksa --namespace payments-prod

gcloud secrets add-iam-policy-binding "stripe-webhook-secret" \
  --role="roles/secretmanager.secretAccessor" \
  --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/payments-prod/sa/payments-api-ksa"

Read that --member string once slowly. It says: exactly this ServiceAccount, in exactly this namespace, in exactly this cluster's identity pool, can read exactly this one secret. That is what least privilege looks like when you stop pretending.

The workload itself is hardened too. From the demo repo:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-vault-api
  namespace: payments-prod
spec:
  template:
    spec:
      serviceAccountName: payments-api-ksa
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          # Immutable digest, verified by Binary Authorization
          image: us-central1-docker.pkg.dev/.../payments-api@sha256:e882491c...
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

The namespace enforces the restricted Pod Security Standard, and the ServiceAccount sets automountServiceAccountToken: false. Non-root, read-only root filesystem, no capabilities, image by digest.

The demo app (app/main.py) is a tiny Python service that asks the GKE Metadata Server who it is and how long its token lives, and reports whether GOOGLE_APPLICATION_CREDENTIALS exists in its environment. On a properly configured cluster, it tells you its scoped principal, a short TTL, and "static_json_keys_present": false. Which is exactly the answer you want.

To verify on your own cluster, check Kubernetes Engine → Clusters → Security for Workload Identity: Enabled (<PROJECT_ID>.svc.id.goog), and make sure your node pools have the GKE Metadata Server enabled and are not running on the default Compute SA.

Pillar 4: No Provenance? No Entry to GKE

Here is the last gap. Even if your pipeline is perfect, what stops someone from pushing an image from their laptop -- with a stolen maintainer token, just like ChainDrop -- and deploying it?

Three things, stacked:

1. Immutable tags in Artifact Registry

Remember tj-actions? Mutable tags were the entire attack. Container tags have exactly the same problem: :v1.0.0 today might not be :v1.0.0 tomorrow.

gcloud artifacts repositories create prod-containers \
  --repository-format=docker \
  --location="${REGION}" \
  --immutable-tags \
  --description="Production container registry with immutable tags"

Once a tag is pushed, it is pinned. Attack vector 4 in the demo tries to repoint an existing v1.0.0 release to a poisoned image, and Artifact Registry refuses. And in your manifests, deploy by digest (@sha256:...) anyway.

2. SLSA Level 3 provenance from Cloud Build

ChainDrop proved SLSA L2 is not enough when the attacker is upstream of the build. You want builds running on isolated, hermetic builders that you control -- not on a maintainer's laptop. Regional Cloud Build generates verifiable build provenance for you:

gcloud builds submit \
  --region="${REGION}" \
  --tag="${REGION}-docker.pkg.dev/${PROJECT_ID}/prod-containers/payments-api:v1.0.0" .

gcloud artifacts docker images describe \
  "${REGION}-docker.pkg.dev/${PROJECT_ID}/prod-containers/payments-api:v1.0.0" \
  --show-provenance

In the console, open the image digest in Artifact Registry and look at the Build Provenance tab: builder identity, source commit, the whole chain. The Vulnerabilities tab gives you CVE scanning for free while you are there.

3. Binary Authorization as the GKE admission gate

Provenance is useless if nothing checks it. Binary Authorization is the bouncer at the cluster door:

defaultAdmissionRule:
  evaluationMode: REQUIRE_ATTESTATION
  enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG

Now try to sneak in a rogue Pod. The demo has one ready, appropriately named:

apiVersion: v1
kind: Pod
metadata:
  name: chaindrop-rogue-miner
  namespace: payments-prod
spec:
  serviceAccountName: payments-api-ksa
  containers:
    - name: rogue-container
      image: docker.io/library/alpine:latest
      command: ["sh", "-c", "echo 'Attempting unauthorized workload execution...' && sleep 3600"]

kubectl apply -f k8s/rogue-pod-attack.yaml

Error from server (Forbidden): admission webhook "imagepolicywebhook.image-policy.k8s.io" denied the request:
Image docker.io/library/alpine:latest denied by Binary Authorization default admission rule:
No valid attestation found for attestor projects/your-gcp-project-id/attestors/prod-ci-attestor

No stamp, no entry. It does not matter whose token was used to push it.

The Cheat Sheet: Threat → Control

Here is the whole talk in one table:

Threat / Incident	Root Vulnerability	CI/CD Fix	Google Cloud / GKE Fix
`tj-actions` / `reviewdog` (CVE-2025-30066)	Mutable action tags + runner memory secret dump	Pin actions to 40-char commit SHAs	Kill JSON keys (`iam.disableServiceAccountKeyCreation`) + short-lived WIF tokens + VPC-SC
Nx `s1ngularity` & loose OIDC trust	`pull_request_target` + WIF checking only `repository`	Never check out untrusted PR code in privileged workflows	Strict WIF CEL condition (`repository_owner_id`, `ref`, `job_workflow_ref`)
Shai-Hulud 2.0 & Mini Shai-Hulud	Rogue `SHA1HULUD` runners + `pnpm` cache / PR OIDC theft	Audit runners; `permissions: contents: read`	Require `runner_environment == 'github-hosted'` + `ref == 'refs/heads/main'` in CEL
ChainDrop & GhostAction	Compromised maintainers + `preinstall` droppers	`--ignore-scripts`; OIDC Trusted Publishing for npm	Immutable tags + Cloud Build SLSA L3 + Binary Authorization + VPC-SC
GKE Pod compromise / SSRF	Default Compute SA exposed via metadata server	Drop capabilities; network policies	Workload Identity Federation for GKE with direct per-KSA IAM bindings

A Reality Check (Because Nothing Is Perfect)

Strict CEL conditions will break things. The first time someone tries a legitimate workflow_dispatch from a release branch, it will fail. That is the point, but you will need to model your real deployment paths in the condition (or in separate providers) instead of opening it back up to "just the repository".

SHA pinning is a maintenance tax. Forty-character hashes do not update themselves. Use Dependabot or Renovate to bump pinned SHAs, otherwise you trade a supply chain risk for an "everything is two years old" risk.

--ignore-scripts is not free. Some packages genuinely need install scripts (native modules, binary downloads). You will need an allowlist or a separate, isolated step for those. It is still a much better default than "run arbitrary code from 1,500 transitive dependencies with my cloud credentials in the environment".

Hardware keys are a people problem. YubiKeys get lost, left at home, or forgotten in a laptop bag. Plan for backup keys and a revocation process before you enforce signed commits on the main branch.

Short-lived tokens are short, not zero. A one-hour token dumped from runner memory is still a one-hour window. This is exactly why VPC Service Controls matter: a stolen token should not work from an attacker's workstation IP in the first place.

Binary Authorization needs a break-glass plan. When prod is on fire at 3 AM and your attestor pipeline is down, you need a documented, audited bypass. Otherwise somebody will "temporarily" turn enforcement off and forget about it.

Getting Started

Everything above is in the demo repo, and you can run it from Cloud Shell with one command:

git clone https://github.com/calganaygun/trust-no-pipeline-gcp-demo.git && \
cd trust-no-pipeline-gcp-demo && \
./scripts/01-setup-gcp-guardrails.sh

The bootstrap script detects your active project and sets up:

The required APIs (iam, sts, artifactregistry, cloudbuild, binaryauthorization, containerscanning)
A CI service account (ci-prod-deployer) and the iam.disableServiceAccountKeyCreation policy
A Workload Identity Pool (github-prod-pool) with a CEL-locked OIDC provider requiring repository_owner_id, refs/heads/main, and github-hosted runners
An Artifact Registry repo (prod-containers) with immutable tags, bound directly to the federated principal
A sample payments-api:v1.0.0 build through regional Cloud Build, with provenance

Then launch the attack simulator and watch all five attacks fail:

./scripts/02-run-attack-vectors.sh

#	Attack	Blocked by
1	ChainDrop / Shai-Hulud `preinstall` worm	`npm install --ignore-scripts`
2	Minting a backdoor Service Account JSON key	`iam.disableServiceAccountKeyCreation`
3	PR / rogue runner OIDC token exchange	WIF CEL attribute condition at GCP STS
4	`tj-actions`-style tag overwrite of `:v1.0.0`	Artifact Registry immutable tags
5	Unattested rogue Pod on GKE	Binary Authorization admission

If you want to click through the slides too, they are a single index.html -- run python3 -m http.server 8080 in Cloud Shell and use Web Preview, or just grab them at calgan.dev/krk-iam.

The Bottom Line

If you take nothing else from this post, take the four stickers from the last slide back to your desk on Monday morning:

Sign and pin. Hardware-backed GPG commits, and 40-character SHAs for every action.
Burn your JSON keys. iam.disableServiceAccountKeyCreation, today.
Check the branch and the runner. WIF CEL: main + github-hosted, not just the repo name.
Stamp your artifacts. SLSA L3 provenance in, Binary Authorization at the door.

Attackers have figured out that your pipeline is the softest, most privileged thing you own. It has production credentials, it runs untrusted code all day long, and nobody audits it like they audit prod.

The era of "it is our CI, it is fine" is over. The era of treating the delivery truck with the same suspicion as anything else coming through the gate is here.

Trust no pipeline. Not even your own.

References

All the sourcing for this post:

Trust No Pipeline -- slides -- the interactive deck from the talk
trust-no-pipeline-gcp-demo -- the live attack/defense demo repo used throughout this post
CISA alert on tj-actions/changed-files (CVE-2025-30066) -- the official advisory
NVD: CVE-2025-30066 -- the CVE record
Nx s1ngularity GitHub security advisory -- the pull_request_target compromise of nx
Wiz Research: Shai-Hulud supply chain worm -- breakdown of the npm worm waves
Panther Labs: Shai-Hulud analysis -- detection-focused write-up
Aikido: credential-stealing npm worm -- package-level analysis
2026 npm worm defense analysis -- Mini Shai-Hulud and defenses
Elastic Security Labs: Shai-Hulud & ChainDrop -- ChainDrop deep dive
Palo Alto Unit 42: ChainDrop npm worm -- Bun dropper and Ethereum C2 analysis
StepSecurity: ChainDrop incident breakdown -- CI/CD-centric timeline
GitGuardian: GhostAction, 3,000+ secrets stolen -- rogue workflow secret exfiltration
Google Cloud: Workload Identity Federation with deployment pipelines -- the official WIF + GitHub Actions guide
Google Cloud: Workload Identity Federation for GKE -- per-KSA identities on GKE
Google Cloud: Binary Authorization -- admission-time attestation enforcement

Good thing to re-focus on, regarding todays "consciousness-bullshit" discussions :)

Çalgan Aygün — Tue, 06 Oct 2026 08:58:05 +0000

Çalgan Aygün

Feb 24

Resisting the Eye of the Machine: A Reflection on AI and Data Ownership

#discuss #ai #data #privacy

4 min read

Good thing to re-focus on, regarding todays "consciousness-bullshit" discussions :)

Çalgan Aygün — Tue, 06 Oct 2026 08:58:02 +0000

Çalgan Aygün

Feb 24

Resisting the Eye of the Machine: A Reflection on AI and Data Ownership

#discuss #ai #data #privacy

4 min read

Building your own Git remote in under an hour

Çalgan Aygün — Tue, 18 Aug 2026 19:59:27 +0000

git·vodka's idea had been sitting in the back of my mind for a while.

not as some big product idea. I just wanted a tiny git remote I could run for myself, friends, and family. something a little nicer than throwing bare repos on a server, but without turning it into another giant developer platform.

today, with everyone suddenly talking about GitHub alternatives again after the recent downtime, I checked git.vodka and saw the domain was available.

so I bought it.

from there it was mostly impulse. pointed the domain at Cloudflare, opened Codex, and started building.

the whole thing runs on Cloudflare: workers for the app, R2 for git objects and blobs, and D1 for the database. it also contains its own source code, which felt like a fun requirement for something hosting git repos.

there wasn't really a plan beyond "let's see if this works."

it did.

from buying the domain to having a working version online, the whole thing took less than an hour. Codex wrote most of the boring parts while I kept changing my mind about how I wanted it to work.

it's currently running entirely on Cloudflare's free tier, which is kind of ridiculous in the best way.

anyway, that's git·vodka.

you can poke around at git.vodka.

and yes, the main advantage of owning your own git hosting service is that nobody can stop you from taking the username c.

[Boost]

Çalgan Aygün — Wed, 12 Aug 2026 06:31:58 +0000

Çalgan Aygün

Jul 31

I hacked every dumb screen in my house into one dashboard

#gadget #programming #opensource #automation

4 min read

Agent Sandboxes: Giving AI Agents Their Own Little Linux Box (And Why You Should Care)

Çalgan Aygün — Fri, 07 Aug 2026 09:38:40 +0000

Sourced from: GKE Agent Sandbox docs, kubernetes-sigs/agent-sandbox

So here is the thing. We are asking AI agents to do increasingly wild stuff. Write code. Browse the web. Run shell commands. Spin up browsers. Click around. Deploy websites. Use a computer like a human would. And for the most part, they are doing it.

But there is a lurking problem that keeps platform engineers up at night.

Where the hell is this code running?

When you ask an agent to "run this script" or "install this package" or "check if that endpoint responds", you are effectively handing over a loaded gun to a very enthusiastic intern who has read every programming book but has absolutely zero survival instincts. That code could be malicious. Or buggy. Or it could just go nuclear on your host system because the agent decided to rm -rf / thinking it was cleaning up temp files.

That is where Agent Sandboxes come in.

Think of them as giving each agent its own airtight, disposable Linux container. A little VM crib. A playpen with rubber walls where they can run around and break things without ever touching your real infrastructure.

And here is the kicker: this is not some vaporware future concept. Google Cloud ships Agent Sandbox as a managed GKE feature, and the open-source project behind the Kubernetes APIs lives under kubernetes-sigs/agent-sandbox. You can run the open-source controller on your own Kubernetes cluster or use the managed GKE integration.

What GKE Agent Sandbox Actually Is

Let me quote the docs directly so we are on the same page:

"GKE Agent Sandbox helps you manage isolated, stateful, and single-replica workloads on GKE. It is optimized for use cases like AI agent runtimes, where untrusted, LLM-generated code must be executed in a secure and performant environment."

What that means in plain English is:

You get a Kubernetes-native way to spin up a single, stateful container that behaves like a lightweight VM. It has a stable hostname, persistent storage that survives restarts, kernel-level isolation, and network policies that default to "deny all". Your agent can connect to it, do its thing, disconnect, come back later, and find everything exactly as it left it. Or it can get garbage collected after a TTL and disappear without a trace.

With a SandboxWarmPool, already-started sandboxes can be assigned in milliseconds instead of waiting for a cold Pod to schedule, pull an image, and start. Cold-start latency still depends on your cluster, scheduler, image, and runtime.

The Architecture: CRDs All the Way Down

The whole thing is built on Kubernetes custom resource definitions. There are four main ones that matter:

1. Sandbox (the core)

This is the primitive. A single, stateful Pod with a stable hostname, optional persistent storage, and full lifecycle management. You create this, and the controller handles the rest -- Pod creation, network identity, volume provisioning, scheduled deletion, pausing, and resuming.

It is the answer to the question: "I want a Linux box that stays alive for my agent. How do I get one?"

2. SandboxTemplate (the blueprint)

Tired of writing the same YAML over and over? Templates let you codify runtime configuration once -- image, resource limits, runtime class (gVisor, Kata, standard), network policies, environment variables -- and reuse them everywhere. Platform teams define the templates. Developers just pick one and go.

3. SandboxClaim (the request)

This is the user-facing abstraction. In the current v1beta1 API, a developer or agent SDK creates a SandboxClaim that references a SandboxWarmPool. The warm pool references the SandboxTemplate. The controller adopts an available pre-warmed sandbox from that pool, or creates capacity according to the pool configuration, and hands back a ready-to-use environment.

This is the Claim Model in action. You separate the what (I need a sandbox) from the how (here is where it runs on the cluster). And honestly? This is the pattern Kubernetes has needed for stateful workloads for a long time.

4. SandboxWarmPool (the performance hack)

Here is where it gets fast. A SandboxWarmPool maintains a set of pre-warmed Pod instances in a ready state. When a claim comes in, the controller instantly assigns a Pod from the pool instead of waiting for image pulls and container startup.

Warm pools reduce provisioning latency by keeping Sandbox instances pre-created and ready to be claimed. On GKE, Pod snapshots are a separate feature that can save and restore sandbox state for pause/resume and faster recovery; they are not required for a warm pool to work.

Security: Default Deny Is the Only Way

Here is something that does not get talked about enough. When you give an agent a Linux box, you are trusting it with network access. A lot of agent sandbox solutions just throw a container at you and say "good luck".

GKE Agent Sandbox implements a Default Deny network security posture. Out of the box, code inside a sandbox cannot access your internal networks, your GKE control plane, or anything it should not. You explicitly define allowed egress and ingress rules in your SandboxTemplate.

Why does this matter? Because the most realistic threat from LLM-generated code is not "the AI becomes self-aware and takes over the world". It is "the AI wrote a Python script with a typo that accidentally hit the production database". Default deny prevents the accident. Fine-grained rules let you open specific doors when you need to (e.g., pip needs to reach PyPI, your agent needs to call an API).

Isolation Runtimes: Pick Your Poison

The Sandbox CRD is runtime-agnostic. You can pair it with:

Standard containers -- fast, familiar, but minimal isolation between workloads
gVisor -- Google's user-space kernel. Gives you a second security boundary between your container and the host kernel. This is the recommended default for untrusted code execution.
Kata Containers -- hardware-level VM isolation. Each sandbox gets its own lightweight VM with its own kernel. The heaviest isolation, but also the strongest.

The beauty of the API is that switching between these is a field change in your SandboxTemplate. You do not redesign your architecture. You just change runtimeClassName.

Programmatic Access SDKs

You do not need to write Kubernetes YAML to use this. There are first-class client libraries:

Python SDK -- high-level client for LangChain, Vertex AI Agentic SDK, or any Python agent framework. Create, query, manage, and destroy sandboxes from your agent code.
Go SDK -- for platform engineers building controllers and services on top of Agent Sandbox.

Both SDKs abstract away the claim lifecycle. With the current Python SDK, your agent creates a sandbox with client.create_sandbox(warmpool="python-sandbox-pool", namespace="default"). Behind the scenes, the SDK creates a SandboxClaim, the controller assigns a Sandbox from the warm pool, and the client returns a handle for command execution and file operations.

The Comparison Table

Feature	GKE Agent Sandbox (managed)	kubernetes-sigs/agent-sandbox (open source)
Type	Managed GKE add-on	Self-installed Kubernetes controller
CRDs	Sandbox, SandboxTemplate, SandboxClaim, SandboxWarmPool	Sandbox, SandboxTemplate, SandboxClaim, SandboxWarmPool
Provisioning speed	Warm pools reduce claim latency to millisecond-scale assignment	Varies by cluster; warm pools avoid cold-start work
Isolation runtimes	gVisor is required for managed Agent Sandbox workloads	Runtime-agnostic through Kubernetes `runtimeClassName`
Network security	Default deny, fine-grained egress/ingress rules	Configurable via standard K8s NetworkPolicies
Snapshots	GKE Pod snapshots integrate with pause/resume	Platform/runtime dependent; not required for warm pools
SDKs	Python + Go	Python + Go
Lifecycle mgmt	Managed by Google (upgrades, patches)	Self-managed via controller
Limitations	Requires GKE 1.35.2-gke.1269000+ for full feature support	Compatibility is release-specific; pin and test a release
Licensing	Google Cloud SLA	Apache 2.0
Best for	Google Cloud native teams who want "it just works"	Multi-cloud, self-managed, air-gapped environments

The Practical Takeaway

Agent Sandbox solves a real problem that has been getting more urgent by the month. As LLMs get better at generating code, the volume of untrusted code being executed in agent loops is going to explode. Every one of those executions is an opportunity for something to go wrong.

The old approach was: "just run it in a Docker container, it will be fine." A container can absolutely have a hostname and persistent volumes, but a one-off Docker workflow does not give you the Kubernetes-native Sandbox lifecycle, claim/warm-pool allocation model, controller-managed identity, or stronger isolation runtime by itself. Those are the abstractions Agent Sandbox is adding.

Agent Sandbox gives you those lifecycle and allocation primitives on top of Kubernetes, with SDKs in Python and Go. Isolation strength still depends on the runtime you configure; the managed GKE feature enforces gVisor for sandbox workloads.

The open-source project under kubernetes-sigs/agent-sandbox is the bedrock. Google's managed GKE add-on is the "press this button and it works" version. Both are worth knowing about.

A Reality Check (Because Nothing Is Perfect)

Latency is low but not zero. Even with warm pools and Pod snapshots, you are looking at hundreds of milliseconds to a second for sandbox assignment. For most agent use cases that is fine. For real-time loops that need sub-100ms responses, you will feel it.

State management is your problem. A sandbox is stateful by design. But what happens when an agent session crashes mid-execution? Do you keep the sandbox around? For how long? The TTL feature helps, but designing your garbage collection strategy is on you.

Network policy complexity is real. Default deny is great for security. But the first time your agent cannot pip install because egress to PyPI is blocked, you will spend time debugging. And the first time a client's script needs to reach an internal API, you will be writing policy rules. It is manageable. But it is not zero-effort.

Costs add up in warm pools. Each pre-warmed Pod is a running container burning CPU and memory while it waits. If you keep a pool of 20 sandboxes hot, you are paying for 20 idle containers. The pause/resume via Pod snapshots helps offset this, but it is something to budget for.

Monitoring is a blank canvas. Your existing Kubernetes monitoring? Probably set up for Deployments and StatefulSets. Ephemeral sandboxes that live for seconds to minutes are a different monitoring challenge. Log aggregation, metrics collection, cost attribution -- you are building some of this yourself.

Getting Started

If you want to reproduce the open-source setup on a fresh Kubernetes cluster, the sequence below is complete: install the controller and extensions, create an executable Python runtime template, create a warm pool, give an in-cluster client the required RBAC, and run the SDK from a normal Python Pod.

The commands below intentionally use kubectl apply -f - <<'EOF' so you do not need to write YAML files locally.

1. Install Agent Sandbox with the extension CRDs

As of this writing, the current upstream release is v0.5.3. Pinning the version makes the example reproducible instead of silently changing when a new release lands.

export VERSION="v0.5.3"

kubectl apply -f \
  https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml

Wait for the controller components to come up:

kubectl get pods -n agent-sandbox-system

Verify that both the core and extension APIs are installed:

kubectl api-resources | grep -i sandbox

You should see resources including sandboxes, sandboxclaims, sandboxtemplates, and sandboxwarmpools.

2. Create a Python runtime template and warm pool

A plain python:3.12-slim container with sleep infinity is not enough for sandbox.commands.run(). The SDK sends HTTP requests to the runtime's /execute endpoint on port 8888, so the sandbox image must implement that API.

The upstream Python runtime image does exactly that:

kubectl apply -f - <<'EOF'
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate
metadata:
  name: python-sandbox-template
  namespace: default
spec:
  podTemplate:
    spec:
      containers:
      - name: python-sandbox
        image: registry.k8s.io/agent-sandbox/python-runtime-sandbox:v0.1.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8888
---
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
  name: python-sandbox-pool
  namespace: default
spec:
  replicas: 1
  sandboxTemplateRef:
    name: python-sandbox-template
EOF

Verify that the template, pool, Sandbox, and Pod exist:

kubectl get sandboxtemplates,sandboxwarmpools,sandboxes,pods -n default

Wait until the warm-pool sandbox Pod is Running.

3. Create a dedicated ServiceAccount and RBAC for the SDK client

Do not run the SDK with the namespace's default ServiceAccount in a real deployment. Give the client its own ServiceAccount with the permissions it needs to create/delete claims and resolve the underlying Sandbox.

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sandbox-client
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sandbox-client
  namespace: default
rules:
- apiGroups:
  - agents.x-k8s.io
  resources:
  - sandboxes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions.agents.x-k8s.io
  resources:
  - sandboxclaims
  verbs:
  - get
  - list
  - watch
  - create
  - delete
- apiGroups:
  - extensions.agents.x-k8s.io
  resources:
  - sandboxwarmpools
  - sandboxtemplates
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  - services
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sandbox-client
  namespace: default
subjects:
- kind: ServiceAccount
  name: sandbox-client
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: sandbox-client
EOF

Verify the critical permissions using the fully qualified resource name syntax supported by kubectl auth can-i:

kubectl auth can-i get sandboxes.agents.x-k8s.io \
  --as=system:serviceaccount:default:sandbox-client \
  -n default

kubectl auth can-i create sandboxclaims.extensions.agents.x-k8s.io \
  --as=system:serviceaccount:default:sandbox-client \
  -n default

kubectl auth can-i delete sandboxclaims.extensions.agents.x-k8s.io \
  --as=system:serviceaccount:default:sandbox-client \
  -n default

All three should return yes.

4. Start a Python client Pod

The SDK can run locally using kubectl port-forward, but that mode requires kubectl in the environment running the Python process. For this reproducible example, run the client inside the cluster and use SandboxInClusterConnectionConfig, which connects directly to the sandbox runtime and does not spawn kubectl.

kubectl run python-pod \
  --image=python:3.12-slim \
  --restart=Never \
  --overrides='{"spec":{"serviceAccountName":"sandbox-client"}}' \
  --command -- sleep infinity

Wait for it:

kubectl wait --for=condition=Ready pod/python-pod -n default --timeout=120s

Install the Python SDK in that Pod, pinned to the controller release used above:

kubectl exec -n default python-pod -- \
  pip install k8s-agent-sandbox==0.5.3

5. Create a sandbox and execute Python inside it

Run the client directly through stdin; no Python file is required:

kubectl exec -i -n default python-pod -- python - <<'PY'
from k8s_agent_sandbox import SandboxClient
from k8s_agent_sandbox.models import SandboxInClusterConnectionConfig

client = SandboxClient(
    connection_config=SandboxInClusterConnectionConfig()
)

sandbox = client.create_sandbox(
    warmpool="python-sandbox-pool",
    namespace="default",
)

try:
    result = sandbox.commands.run(
        'python3 -c \'print("hello from my sandbox")\''
    )
    print(result.stdout)
finally:
    sandbox.terminate()
PY

Expected output:

hello from my sandbox

At this point the complete path is working:

python-pod
    |
    | SandboxInClusterConnectionConfig
    v
SandboxClaim -> SandboxWarmPool -> Sandbox -> runtime Pod:8888
                                      |
                                      +-> POST /execute

6. Troubleshooting the three failures you are most likely to hit

If you get:

SandboxWarmPoolNotFoundError: SandboxWarmPool "python-sandbox-pool" not found

check that the warm pool exists in the same namespace:

kubectl get sandboxwarmpools -n default

If you get a 403 Forbidden mentioning sandboxclaims or sandboxes, check the ServiceAccount RBAC:

kubectl auth can-i get sandboxes.agents.x-k8s.io \
  --as=system:serviceaccount:default:sandbox-client \
  -n default

If you get:

HTTPConnection(... port=8888): Failed to establish a new connection: Connection refused

check the image used by the sandbox Pod:

kubectl get pods -n default \
  -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[*].image,IP:.status.podIP

The sandbox must run a runtime server that implements /execute on port 8888; a normal Python image that only sleeps will not work with sandbox.commands.run().

7. Clean up

kubectl delete pod python-pod -n default --ignore-not-found
kubectl delete sandboxwarmpool python-sandbox-pool -n default --ignore-not-found
kubectl delete sandboxtemplate python-sandbox-template -n default --ignore-not-found
kubectl delete rolebinding sandbox-client -n default --ignore-not-found
kubectl delete role sandbox-client -n default --ignore-not-found
kubectl delete serviceaccount sandbox-client -n default --ignore-not-found

On GKE instead

For the managed GKE feature, Agent Sandbox requires GKE 1.35.2-gke.1269000 or later. On an existing Standard cluster, Google also requires a gVisor-enabled node pool before enabling Agent Sandbox. The current command uses the beta cluster surface and includes the cluster location:

gcloud beta container clusters update "${CLUSTER_NAME}" \
  --location="${LOCATION}" \
  --enable-agent-sandbox

GKE additionally enforces sandbox workload requirements such as runtimeClassName: gvisor, disabling automatic ServiceAccount-token mounting, running as non-root, dropping Linux capabilities, setting CPU/memory limits, and scheduling onto the gVisor sandbox node pool. Use the managed GKE deployment manifest from the official GKE guide rather than the minimal self-managed template above.

The Bottom Line

GKE Agent Sandbox -- and the open-source kubernetes-sigs/agent-sandbox project it is built on -- is what happens when someone finally builds the right abstraction for agent runtimes. It is Kubernetes-native, which means it fits into existing infrastructure without fighting it. It is fast enough for interactive use. It is secure by default. And it has SDKs that let agents manage their own environments programmatically.

The era of "just run it in Docker and pray" is ending. The era of "give the agent its own Linux box, properly isolated, properly managed, properly fast" is here.

And honestly? It is about damn time.

References

All the sourcing for this post:

GKE Agent Sandbox documentation -- the primary source on the managed GKE add-on
kubernetes-sigs/agent-sandbox (~3k stars) -- the open-source CNCF SIG project under SIG Apps
Agent Sandbox documentation site -- full docs, getting started guides, Python/Go SDK references
Agent Sandbox Python SDK -- programmatic sandbox management from Python
Agent Sandbox Go SDK -- same, but for Go

I hacked every dumb screen in my house into one dashboard

Çalgan Aygün — Fri, 31 Jul 2026 19:46:50 +0000

This Kindle was supposed to be obsolete.

The tiny LCD beside it was designed to do little more than rotate a slide show.

Now both show live transit departures, weather, tasks, and anything else I can pull from an API—and I control them from the same self-hosted dashboard.

No custom firmware on the LCD. No soldering. No cloud subscription.

It started with a cheap Chinese display:

The GeekMagic Ultra, or "SmallTV" as people call it, a 240×240 IPS panel with a stock firmware that just rotates through a gallery slideshow or weather UI. Static images only, no dynamic data unles you use weather screen, no API for live content. But it has a /doUpload endpoint and a /set call. Punch in its IP, overwrite the same JPEG every time, and suddenly that dumb gallery display is showing live transit departures and Todoist tasks.

No firmware mods. No soldering. Just one script.

I built geekmagic-ultra-scripts to prove it works. Transit boards, task lists, weather — all running on stock hardware that cost pocket change. That project was fun. I thought it was the end of the story.

Then I found a used, EOL'd Kindle 4 NT for 100 PLN (like 25 USD) on the second-hand market. Amazon abondaned it, but I haven't :) It runs on Linux. It has an e-ink display that uses almost no power. A command called eips can render images. It can toggle Wi-Fi, report battery, last weeks on a charge, and wake from deep sleep via its RTC.

One jailbreak way to an SSH session and I knew: this thing is a perfect pull-based display. I highly inspired from melihkarakelle's project called kindle4-weather-display.

Now I had two completely different hacked screens: one push (GeekMagic), one pull (Kindle), and no unified way to manage them. Every device needed its own script, its own config, its own workflow.

So I sat down and built the thing that ties them all together.

hackreen is a self-hosted control plane and design studio for every hacked display in your house. One web UI, one declarative screen language, deploy to any device.

What it actually does

Boots in two minutes (docker compose up -d):

GeekMagic → push mode. Configure the endpoint. Hackreen renders a 240×240 JPEG, overwrites the same gallery file, calls /set, and the display updates instantly. Zero hardware hacks.
Kindle 4 → pull mode. Copy-paste one wget | sh command into the Kindle. It installs a client that wakes Wi-Fi every 30 minutes, pulls a fresh image, renders it with eips, reports battery, then goes back to RTC deep sleep for days of battery life. SSH stays reachable. There's even a debug mode: hold the center 5-way key during power-on and you get a shell.
Hackreen Screen Language (HSL). Declarative JSON format. Define data sources (clock, weather, Todoist, transit APIs), compose layers with text, images, icons, shapes. Every live source ships with sample data so the preview works offline.
Design Studio. Visual editor with instant preview. Connect to live APIs, iterate in real time, push when it looks right — without touching a config file.
Secrets Vault. AES-256-GCM encrypted. Tokens stay safe at rest, never exposed by the API.
2,000+ Lucide icons. Built-in. icon:lucide:cloud-rain, done.

The Kindles pull. The GeekMagic gets pushed. You design once, deploy everywhere.

Why self-hosted?

This is a home lab tool. You run it on your LAN. Access through a VPN. No third party, no subscription, no "we changed our API, sorry." You own your data and your screens.

The lineage

geekmagic-ultra-scripts — the first proof of concept. Turned a stock gallery slideshow into a live information display by repeatedly overwriting one file. Transit, Todoist, weather templates.
Kindle 4 NT discovery — second-hand e-ink device that could pull and display images with almost zero power consumption. Proved the pull model worked.
Hackreen — the orchestrator. Brought push and pull under one roof with a proper design pipeline, HSL language, and a unified API.

What's next

Every hacked screen in this house goes into Hackreen. The SmallTV in the kitchen. The Kindle on the desk. Whatever cheap display I find next — it's getting a pull URL or a push button. The question isn't "can I show this data on that screen?" anymore. The answer is always yes, it's just another screen in the dashboard.

Check it out: github.com/calganaygun/hackreen

This project is highly inspired by Melih Karakelle's tweets and projects, so special thanks to him.

Resisting the Eye of the Machine: A Reflection on AI and Data Ownership

Çalgan Aygün — Tue, 24 Feb 2026 21:08:49 +0000

AI both creates and consumes. For someone like me, who’s invested deeply in both the benefits and risks of these evolving systems, the duality is impossible to ignore. On one hand, AI supplements creativity, fast-tracks productivity, and offers insights unprecedented in human history. On the other, AI is a voracious consumer that treats every public thought, image, and pixel as a potential resource for improvement—its improvement, not ours.

What happens to ideas when they’re not just shared, but consumed, repurposed, and disjointed from their original intention? This is not a fight against inevitable progress but an invitation to consider where the boundaries should lie.

The Hunger of the Machine

Today’s AI systems are participants in a digital ecosystem. To feed their model-building appetite, they consume everything: tweets, screenshots, vague status updates, unfinished sketches, throwaway jokes—anything to refine prediction and replication capabilities. These systems don’t ask for permission; most of the time, they don’t even acknowledge the humans who originally created the content.

As someone interested in sparking friction within this process, I’ve explored ways to actively undermine AI’s consumption. One approach stems from visual obfuscation principles to disrupt AI readers at the OCR level, reducing their ability to reconstruct coherent text segments.

Dynamic Segmentation and My Implementation

The core idea relies on radical dynamic segmentation. Imagine text that never fully "settles"—it pulses, shifts, even disassembles itself momentarily. While the human eye has an incredible ability to "fill in the blanks" and interpret movement, machines struggle with this flux.

h43z's initial concept focused on kinetic text—characters in constant motion to disrupt machine vision. This was the spark that got me thinking, but I took a different route: breaking characters into partial segments that flash independently. The chunking method was something I figured out through experimentation.

The implementation is JavaScript-based, using Canvas to analyze each character's pixels and fragment them in various ways—radial slices, concentric rings, diagonal strips, random distributions. Each chunk flashes at randomized intervals. The result: text that humans can read but machines can't easily parse.

I tested against OCR libraries like Tesseract and even LLM-adjacent pipelines. The simpler systems failed completely. Modern Live/Video APIs struggled but occasionally recovered—revealing both the method's potential and its limits.

Why it works: It exploits the gap between human and machine perception. We thrive on approximation and motion. Machines expect clean, static data. Every fragmented flash introduces chaos that challenges their assumptions about how information should look.

Limitations and Lessons

Technology can’t be a self-contained solution to problems of technological overreach. One critical limitation is that dynamic segmentation, while effective on smaller scales, is computationally expensive to produce and impractical for systems requiring long text readability.

Moreover, it raises a practical question: How many people would opt to obfuscate their content actively? To scale such efforts and normalize resistance, the tools need to be seamless, almost invisible to the writer. A browser extension that dynamically applies segmentation while retaining human readability—a future goal—might bridge this gap.

The Philosophical Trade-Offs

This technical exploration also emphasizes deeper philosophical questions. By participating in this cat-and-mouse game against AI, do we solve the problem or play into its competitive escalation? As much as the act of resistance feels gratifying, it’s worth asking: What happens when the effort to defend against machines becomes indistinguishable from innovations driving them forward?

AI reveals how much has been surrendered—privacy, ownership, sometimes even the joy of creating for humans rather than systems. But agency remains deeply human. Perhaps that’s the key takeaway: less about domination and more about choosing what remains ours.

In Closing: Toward Balancing Agency

Dynamic segmentation obviously is not a silver bullet. It won’t end AI overconsumption, but it does add another tool to the resistance. My experiments aim to create what friction I can—disrupting the expectation that AI must own everything it sees. At the heart of this is the hope that small, deliberate acts can start larger conversations about human ownership in an AI-driven culture.

As experiments continue, the tension between control and participation stays alive. To stabilize both technology and thought, we’ll need boundaries that don’t eliminate creativity but celebrate and protect its imperfection.

Acknowledgments: To h43z, whose kinetic text concept was the initial inspiration that sparked my thinking on this problem. While their approach focused on character movement, it pushed me to explore the chunking and partial segmentation methods documented here.

Additional References:

My implementation of dynamic segmentation tests: Live Demo
The foundational thought by h43z: Original Tweet

Redefining Event-Driven Architecture on Google Cloud

Çalgan Aygün — Mon, 16 Feb 2026 19:10:10 +0000

Building event-driven systems with Cloud Run, Pub/Sub, and Eventarc is a powerful pattern, but the gap between "Hello World" tutorials and production stability is wide. These are the hard-won lessons from the trenches of Google Cloud serverless architecture.

1. Choosing Your Transport: Eventarc vs. Pub/Sub

While Eventarc often uses Pub/Sub under the hood, the choice of which to interface with directly depends on your source and required control.

Feature	Eventarc	Direct Pub/Sub
Best For	GCP-native events (GCS, Firestore, Audit Logs)	Custom inter-service messaging
Setup	Managed wrapper, high convenience	Manual configuration, high control
Filtering	Built-in attribute filtering	Fine-grained policies & custom attributes
IAM	Uses `eventarc.eventReceiver`	Uses `pubsub.subscriber`

Pro Tip: Use Eventarc for "plumbing" Google Cloud events. Use Direct Pub/Sub when you need a custom event bus or specific retry/ordering logic.

2. The Infrastructure Reality Check

The Idempotency Requirement

Pub/Sub guarantees at-least-once delivery by default. While exactly-once delivery exists as an opt-in feature, your consumers must assume duplicates will happen. If your handler isn't idempotent, you risk double-charging users or corrupting state.

The Non-Negotiable Flow:

Extract a unique Idempotency Key (UUID/Hash) from the event.
Check a fast-access cache (Memorystore or Firestore) for the key.
If present, discard the message as a duplicate.
If absent, process the event and store the key with a TTL.

Managing Cold Starts

Cloud Run’s "scale to zero" is a budget-saver but a latency-killer. A container startup typically adds 2–8 seconds to the first request.

The Fix: Set min-instances for latency-critical paths (this incurs cost).
The Alternative: Use Cloud Run Jobs for asynchronous batch processing where start-time overhead is less relevant.

Eventarc Propagation Delays

When deploying Eventarc triggers via Terraform or CLI, filtering rules can take 60–120 seconds to propagate across the Google network.

Production Fix: Build a "warm-up" delay into your CI/CD pipeline. Do not trigger integration tests immediately after a successful deployment, or you will see intermittent, false-positive failures.

3. Calculating Realistic Costs

Tutorials often suggest serverless is "pennies," but at scale, the math changes. Pub/Sub is billed at $40 per TiB, with a 1 KB minimum per message.

Example: 10M Events/Day (300M/Month)

Data Volume: 300M messages × 1 KB (min size) ≈ 293 GiB (0.286 TiB).
Ingest & Delivery: (0.286 TiB × $40) + (0.286 TiB × $40) = $23/month.
Compute (Cloud Run): 300M invocations (assuming 1s duration, 512MB RAM) ≈ $150–$200/month.

Total: ~$175–$225/month. Significant, but highly predictable if you monitor message size and invocation duration.

4. Advanced Production Patterns

Regional Co-location

Eventarc triggers for multi-region services (like Firestore nam5) are often pinned to specific regions (e.g., us-central1). If your Cloud Run service resides elsewhere, you will incur cross-region egress charges and increased latency. Always verify the region mapping table in the GCP documentation.

Payload Sanitization at the Edge

Using Eventarc Advanced, you can use Common Expression Language (CEL) to transform or redact payloads before they reach the consumer. This is vital for PII compliance.

// Example: Redacting an email address in-flight
message.setField("data.email", re.extract(message.data.email, "(^.).*@(.*)", "\\1***@\\2"))

Observability and Dead-Letter Topics (DLTs)

In a distributed system, an event can "disappear" if a filter drops it or a consumer fails silently.

DLTs: Every subscription must have a Dead-Letter Topic.
Alerting: Monitor the subscription/dead_letter_message_count metric. A rising count is your first sign of a logic bug or schema mismatch.
Tracing: Use OpenTelemetry to inject Trace IDs into event attributes, allowing you to follow a single request from the producer through the bus to the consumer logs.

5. When to Avoid Event-Driven

Don't "cargo-cult" this architecture if it doesn't fit your needs. Skip it if:

You require strong consistency and immediate transactions.
Your end-to-end latency requirements are < 100ms.
The overhead of debugging distributed traces outweighs the scaling benefits.

In these cases, stick to Synchronous gRPC/HTTP or Cloud Workflows for structured orchestration.

Final Takeaway

The combination of Cloud Run and Eventarc is one of the most robust patterns in 2026. By respecting the boundaries of the platform—accounting for propagation delays, ensuring idempotency, and co-locating regions—you can build a system that scales effortlessly from zero to millions of events.

A CAPTCHA Bypass Technique: Audio Files

Çalgan Aygün — Fri, 30 Jan 2026 12:38:29 +0000

This is a draft write-up from 2019, documenting a CAPTCHA bypass technique I discovered back then. All code and images shown are examples for educational purposes.

One day back in 2019, I got tired of repeatedly logging into my school's student system just to enroll in a full class. Then a lightbulb went off in my head: I could automate these attempts and refocus on my actual work.

When it comes to automating website processes, I love using user scripts.

1. Automation Plan

Our school system expired user sessions after a while, even if you were actively working. So first, I needed to automate the login process. Once successfully logged in, navigating and enrolling in a class would be straightforward (just some click event magic).

2. Login UI

The school system had an internally managed CAPTCHA service. It displayed two numbers and asked for their sum. The images were slightly corrupted—but not enough to prevent OCR. However, I didn't want to rely on any API or image processing system to solve this CAPTCHA.

3. Inspecting the Audio Service for CAPTCHA

After deciding not to use any image-related service, I started inspecting the network traffic of the login system. I noticed that the answer to the generated CAPTCHA was stored server-side in a session associated with me.

When I clicked the audio playback button, I realized it was reading out the answer directly. Two different endpoints returned two different audio files, split into tens and ones digits.

The audio system seemed like a perfect route for me. I had no intention of feeding these audio files into a speech-to-text service—I was looking for the fastest, hackiest solution possible.

Since the same audio files should have the same file size, I decided to test this hypothesis first.

And bingo! By mapping each tens and ones digit to their file sizes beforehand, I could automatically determine the answer.

4. The Solution

Here's an example of how the code worked:

// Example mapping of file sizes to digit values
const tensMap = {
  1234: 0,    // 0 tens
  1456: 10,   // 1 ten
  1567: 20,   // 2 tens
  // ... etc
};

const onesMap = {
  987: 0,     // 0 ones
  1023: 1,    // 1 one
  1145: 2,    // 2 ones
  // ... etc
};

// Example: Fetch audio files and determine answer by file size
async function solveCaptcha() {
  const tensResponse = await fetch('/captcha/audio/tens');
  const onesResponse = await fetch('/captcha/audio/ones');

  const tensSize = parseInt(tensResponse.headers.get('content-length'));
  const onesSize = parseInt(onesResponse.headers.get('content-length'));

  const tensValue = tensMap[tensSize] || 0;
  const onesValue = onesMap[onesSize] || 0;

  return tensValue + onesValue;
}

// Use it in the login flow
const answer = await solveCaptcha();
document.querySelector('#captcha-input').value = answer;
document.querySelector('#login-button').click();

When I tested this approach, it worked perfectly!

5. Lessons Learned

This experience taught me a few things:

Look for alternative attack vectors: When the obvious solution (OCR) seems complex, there might be simpler paths (audio files, metadata, etc.)
Security through obscurity fails: Just because a CAPTCHA uses audio doesn't mean it's secure—especially if the files are deterministic
File size is metadata: Even without processing the content, file properties can leak information

This technique worked because the audio files were pre-generated and static. A more secure implementation would generate unique audio files or add random noise to prevent size-based fingerprinting.

The Uncomfortable Truth About AI-Assisted Development

Çalgan Aygün — Wed, 28 Jan 2026 11:59:53 +0000

Or: How We Learned to Stop Worrying and Start Debugging at 2am

I need to tell you about a report that made me uncomfortable. Not because it revealed anything shocking, but because it quantified what I've been watching happen in real-time across engineering teams for the past year.

CodeRabbit analyzed 470 pull requests — 320 co-authored by AI, 150 written by humans. The headline number: AI-generated code contains 1.7x more issues than human-written code.

But that's not the uncomfortable part. The uncomfortable part is that we already knew this. We just chose not to measure it.

The Velocity Trap

Here's what's happening on the ground:

Teams adopted AI coding assistants in 2024. By 2025, they're shipping 20% more PRs per developer. Product managers are thrilled. Engineers feel productive. The metrics look great.

Then production incidents spike by 23.5%.

At first, you blame other factors. Infrastructure changes. New team members. Bad luck. But when you dig into the post-mortems, a pattern emerges:

A null pointer dereference that should have been caught by basic error handling
Hardcoded credentials that somehow made it through review
A database query executing 500 times in a loop instead of being batched
Exception handling that swallows errors and returns success anyway

These aren't exotic bugs. They're boring bugs. The kind a junior developer makes in their first month. The kind that code review is supposed to catch.

Except we're not reviewing like we used to. Because the code looks fine. It compiles. It follows naming conventions. The structure makes sense. So we skim it, assume the AI got it right, and hit approve.

What the Numbers Actually Mean

Let's break down what CodeRabbit found, because the devil is in the details:

Logic errors: 2x higher

Algorithm/business logic: 2.25x
Concurrency control: 2.29x
Null handling: 2.27x
Exception handling: 1.97x

This tells you something fundamental: AI doesn't understand execution flow. It pattern matches syntax. It knows what error handling looks like, but not when you actually need it. It generates code that passes the happy path and explodes on edge cases.

Security vulnerabilities: 1.57x higher

XSS injection: 2.74x
Insecure object references: 1.91x
Password handling: 1.88x
Insecure deserialization: 1.82x

This is the scary one. Because security bugs don't just crash your app—they compromise your users. And AI is literally trained on public code repositories, including all the insecure code that's been written over the past two decades. It's regurgitating attack vectors from 2015 Stack Overflow answers.

Performance issues: 7.9x more I/O problems

This one made me laugh, then cry. AI optimizes for "code that works" not "code that works efficiently". Why batch database queries when you can just loop? Why cache when you can fetch? It's technically correct. It's also a production disaster waiting to happen.

Code quality: 3.15x worse readability

The irony here is brutal. One of the selling points of AI assistants is that they write "clean code". Except what they actually write is verbose, repetitive code with inconsistent naming and unnecessary abstraction. It's formatted nicely. That's not the same thing as readable.

The 90th Percentile Problem

Here's the number that keeps me up at night: at the 90th percentile, AI PRs contain 26 issues versus 12 for humans.

This means AI doesn't just create more bugs on average — it occasionally creates absolute disasters. Code that's so broken it shouldn't have made it past the IDE, let alone into production.

And because we're shipping faster, we're hitting these edge cases more often. It's not a theoretical risk. It's a ticking time bomb in your codebase.

Why This Is Hard to Fix

The obvious answer is "just review AI code more carefully". But that misses the psychological trap we've fallen into.

AI-generated code feels trustworthy because it's consistently formatted, confidently written, and superficially correct. It doesn't have the telltale signs of junior code like hesitant variable names, inconsistent style, obvious copy-paste errors.

So your brain does a pattern match: "this looks like senior-level code" → "probably fine" → approve.

Except it's not senior-level code. It's senior-looking code generated by a system that has no mental model of what it's building.

The traditional markers we use for code quality — structure, naming, formatting — have been decoupled from actual correctness. We need new heuristics, and we haven't built them yet.

What Actually Works

I'm not going to tell you to stop using AI assistants. I use them daily. They're legitimately useful for scaffolding, refactoring, and handling boilerplate.

But here's what I've learned:

1. Ground the model in your context

Don't just paste code into ChatGPT and expect it to understand your domain. Give it your architecture docs. Your API contracts. Your error handling conventions. The more context you provide, the less it has to guess.

2. Treat AI output as untrusted by default

Would you merge code from a contractor you've never worked with before without thorough review? No? Then don't do it for AI.

Any code touching auth, payments, PII, or critical business logic gets manual review. No exceptions.

3. Automate what AI gets wrong

AI is terrible at formatting, naming consistency, and basic security checks. So automate those with linters, formatters, and SAST tools in your CI pipeline.

Don't waste human review time catching things machines can catch. Use humans for semantic review; does this actually solve the problem correctly?

4. Use independent review tools

This is CodeRabbit's obvious pitch, but it's also correct: don't use the same AI that generated code to review it. That's like asking someone to grade their own homework.

Independent static analysis, security scanners, and code review tools catch different classes of bugs than generative models do.

5. Accept the maintenance tax

AI-assisted development is a trade-off: velocity now, maintenance cost later. If you're not willing to pay down the technical debt, don't take on the loan.

Budget time for refactoring. Expect to fix edge cases in production. Plan for the eventual rewrite when the generated code becomes unmaintainable.

The Bigger Question

The uncomfortable truth isn't that AI generates bugs. It's that we're willing to accept more bugs in exchange for speed.

That's a legitimate trade-off in some contexts. Early-stage startups trying to find product-market fit? Ship fast, fix later. Mature companies with millions of users and regulatory compliance? Maybe slow down.

But let's be honest about what we're doing. We're not "augmenting developer productivity" in a risk-free way. We're shifting the risk/reward curve.

More features, faster iteration, shorter cycle times; at the cost of more incidents, more maintenance burden, and more time spent debugging.

Is that worth it? Depends on your context. But you can't answer that question if you're not measuring the cost.

What Happens Next

AI coding assistants aren't going away. They're getting better. Context windows are expanding. Models are learning from feedback. The tooling is improving.

But the fundamental problem remains: LLMs are pattern-matching engines, not reasoning systems. They don't understand your invariants. They don't trace execution paths. They don't think about failure modes.

They surface-fit syntax without semantic understanding.

Until that changes, AI-generated code will always carry a quality tax. The question is whether we're willing to pay it; and whether we're honest about the price.

Key Takeaways

AI code creates 1.7x more issues than human code across logic, security, performance, and maintainability
The worst AI PRs (90th percentile) contain 26 issues vs 12 for humans; catastrophic failure modes are real
Teams are shipping 20% more PRs but seeing 23.5% more production incidents
AI optimizes for "looks correct" not "is correct"; it passes shallow tests but fails on edge cases
Effective mitigation requires grounding models in context, automating quality checks, and treating AI output as untrusted
The velocity gain is real, but so is the maintenance cost; choose consciously

Full CodeRabbit report: https://www.coderabbit.ai/whitepapers/state-of-AI-vs-human-code-generation-report

AI Productivity Tips 1 - SQL Companion Bot with Gemini 1.5 Pro

Çalgan Aygün — Sun, 25 Aug 2024 10:27:36 +0000

In my current role, I often need to analyze database records and evaluate user performance metrics to identify anomalies. However, manually writing these one-off queries across multiple tables isn't efficient, especially when dealing with complex joins. To streamline this process, I turned to AI.

Here’s a step-by-step guide on how to boost your productivity using AI for SQL queries:

Start by navigating to Google AI Studio and create a new chat prompt.

Next, set up a SQL Helper system prompt. Provide your database schema in the system prompt, also specify the RDBMS you’re using so that the AI can generate queries with the correct syntax. I tested this with The DB Book's schema.

Here’s an example system prompt:

Act as a DB Admin/SQL Developer and generate the requested SQL queries based on the following DDL/Schema. Provide explanations and ensure that the queries are safe to run in a production environment. Add "Do not forget to review the SQL query before running." to all SQL responses.

RDBMS/SQL Syntax: <Your RDBMS>

My database DDL: <Your database and table DDLs>

Now, ask your query! For example, I asked: I need to identify the students who are enrolled in a course taught by Teacher 1 and list all their other courses in a comma-separated format in a column.

And just like that, the AI (thanks to Gemini 1.5 Pro) generated the query I needed.