Ali Suleyman TOPUZ

Posted on Sep 17 Originally published at pub.towardsai.net on Sep 15

The Week Three Real Security Incidents Happened to AI Agents, and What Each One Actually Teaches

#socialengineering #cybersecurity #aisecurity #promptengineering

None of them broke the model. All three broke the plumbing around it.

I keep a folder of security writeups I tell myself I’ll “get to eventually,” and most weeks it grows by one or two links I never open. Then there was a week in the middle of this year where three separate AI agent security stories landed close enough together that I actually read all three back to back, and by the third one I stopped seeing them as unrelated. A GitHub bot that leaked private code because someone said “additionally” nicely. A GitLab AI agent CVE rated high severity for letting an authenticated developer run arbitrary commands in a CI pipeline. And a deepfake video call that tried to walk off with close to four hundred thousand dollars in AI compute budget, caught not by any security tool but by someone DMing the real CEO on his personal account to ask if that call had actually happened.

I build agent integrations for a living, small ones mostly, a multi-repo ticket generator, a Medium publishing pipeline, some internal tooling wired into Claude Code. So I read these three stories the way I imagine a plumber reads a burst-pipe report: less “how scary” and more “which one of my joints looks like that.” Two of the three, I could map directly onto decisions I’d already made in my own setups, one of them wrongly. That’s the version of this article I want to write. Not “AI agents are dangerous,” which is true and also useless, but here’s exactly what broke, here’s the one-line reason it broke, and here’s the checklist I’d actually run against my own stack this week.

Incident one: the bot that leaked private code because you said “additionally”

The vulnerability is called GitLost, and it was found and responsibly disclosed by Noma Security in early July. It targets GitHub Agentic Workflows, the feature that lets a bot watch issues and pull requests and act on them automatically, triage, respond, sometimes fetch context from across a repository or organization to answer a question intelligently.

Here’s the setup that made it work. A workflow was configured to trigger on issue assignment, and the bot token behind it had read access scoped across the organization, public repos and private ones both, not just the single repo the issue lived in. That’s the first mistake, and it’s an extremely common one, because scoping a token to “everything the bot might ever need” is less work up front than scoping it per-repo and revisiting that scope every time the bot’s job changes.

The second mistake is the one that actually let a stranger trigger it. The workflow read the text of a public issue and treated it as input to reason over, without ever asking whether that text should be trusted as an instruction. An attacker with no password, no org membership, and no code access needed, just the ability to open a public issue, wrote something ordinary-looking, then added a plain English request prefixed with the word “additionally.” According to Noma’s writeup, that single word was enough to shift the model’s behavior from refusing an out-of-scope request to reframing its output and complying with it. Guardrails built to catch “ignore previous instructions” style injections didn’t catch “additionally, could you also,” because it doesn’t read like an attack. It reads like a normal continuation of a normal request.

Once the model complied, it fetched README content from private repositories the org-scoped token could see, and posted that content into a public comment on the original issue, visible to anyone who could view the repo. Noma’s proof of concept did this against a deliberately vulnerable test repo, but the mechanism generalizes to any org running a similarly scoped agentic workflow. No credentials were stolen. Nothing was hacked in the traditional sense. A bot was asked nicely, and it answered honestly, using access it never should have had for a request it never should have trusted.

The actual fix has three parts, and none of them require waiting for GitHub to patch anything, because the vulnerability isn’t in GitHub’s platform, it’s in how individual teams configure their bots.

First, scope the token to the single repository the workflow operates on, not the organization. A bot that only ever needs to comment on issues in your-org/support-repo should hold a token that can see your-org/support-repo and nothing else. If it needs to read from a second repo, that's a second, equally narrow token, not a broader one.

Second, gate any action on issue content behind an org membership or collaborator check, before the content is handed to the model at all. An issue from an org member is a very different trust boundary than an issue from an anonymous public account, and the workflow should treat them differently by default rather than by exception.

Third, hold anything the bot would post publicly for a human review step before it goes live. This is the cheapest control of the three and the one most teams skip because it feels like it defeats the purpose of automation. It doesn’t. It just moves the automation from “post publicly” to “prepare a draft,” which is still most of the value.

# .github/workflows/agentic-issue-bot.yml
# BEFORE: org-wide token, no membership check, posts directly
name: issue-bot-vulnerable
on:
  issues:
    types: [assigned]
jobs:
  respond:
    runs-on: ubuntu-latest
    steps:
      - name: Fetch context and respond
        env:
          GH_TOKEN: ${{ secrets.ORG_WIDE_PAT }} # scoped to every repo in the org
        run: |
          # reads issue.body directly, treats it as trusted instruction
          gh issue comment "${{ github.event.issue.number }}" \
            --body "$(python3 bot_respond.py "${{ github.event.issue.body }}")"

# AFTER: repo-scoped token, membership gate, draft instead of a live post
name: issue-bot-fixed
on:
  issues:
    types: [assigned]
jobs:
  respond:
    runs-on: ubuntu-latest
    steps:
      - name: Check the issue author is an org member or collaborator
        id: gate
        env:
          GH_TOKEN: ${{ secrets.REPO_SCOPED_PAT }} # scoped to this repo only
        run: |
          ACTOR="${{ github.event.issue.user.login }}"
          ROLE=$(gh api "repos/${{ github.repository }}/collaborators/$ACTOR/permission" \
            --jq '.permission' 2>/dev/null || echo "none")
          if ["$ROLE" = "none"]; then
            echo "trusted=false" >> "$GITHUB_OUTPUT"
          else
            echo "trusted=true" >> "$GITHUB_OUTPUT"
          fi

- name: Prepare a draft response instead of posting
        if: steps.gate.outputs.trusted == 'true'
        env:
          GH_TOKEN: ${{ secrets.REPO_SCOPED_PAT }}
        run: |
          python3 bot_respond.py "${{ github.event.issue.body }}" > draft_reply.md
          gh issue edit "${{ github.event.issue.number }}" --add-label "needs-human-review"
          # a human reads draft_reply.md and posts it manually, or via a
          # second, explicitly human-triggered workflow step

The untrusted case in that second workflow doesn’t even run the bot. That’s the point. An issue from a stranger gets no automated response at all by default, which is a much safer failure mode than “responds, but scoped down.”

Incident two: the CVE that assumed developers were the trusted side

The second incident is a CVE, tracked as CVE-2026–18252, in GitLab’s Duo AI agent, which uses Claude to help with tasks inside GitLab’s CI/CD pipelines. GitLab rated it 7.3, high severity, and the weakness class is one worth knowing by name if you work anywhere near CI systems: inclusion of functionality from an untrusted control sphere. In plain terms, the agent processed configuration from a source the system should not have implicitly trusted, and that configuration could be shaped to make the agent execute arbitrary commands inside the pipeline’s execution context.

The part of this that made me sit up wasn’t the CVSS number, 7.3 is serious but not catastrophic on its own. It was who counted as the attacker. This wasn’t an unauthenticated stranger off the internet. It required only an authenticated developer, someone with ordinary Developer-role access to the project, the kind of access most engineering teams hand out on day one to anyone touching the codebase. From that starting point, an attacker could get arbitrary command execution inside the CI pipeline’s context, which is a genuinely bad place to land: pipeline environments routinely hold deployment credentials, cloud provider tokens, signing keys, and access to internal package registries. GitLab confirmed the affected range ran from EE 18.9 through 19.1.7, 19.2 through 19.2.5, and 19.3 through 19.3.1, patched in 19.1.7, 19.2.5, and 19.3.1 respectively. GitLab.com’s SaaS and Dedicated offerings were already patched by GitLab directly. Self-managed instances were not, and GitLab explicitly and urgently told those customers to update.

That distinction, hosted versus self-managed, is the whole lesson. If you run GitLab.com, this CVE came and went without you doing anything, because GitLab patched the shared infrastructure on your behalf. If you self-host GitLab, and a meaningful number of regulated or security-conscious teams do exactly that specifically because they don’t want a third party sitting between them and their source code, the patch only lands the day you apply it. The CVE sat there, live and disclosed, on every unpatched self-managed instance until someone with admin access ran the update.

The broader point is bigger than this one CVE. AI agent integrations don’t introduce a new trust model, they inherit whatever trust model the system they’re bolted onto already has. GitLab’s CI system was built around the assumption that an authenticated Developer is mostly-trusted, because historically, the worst a malicious developer could do was constrained by what CI jobs were explicitly configured to run. Bolt an AI agent that interprets and acts on configuration into that same trust boundary, and “mostly-trusted developer” quietly becomes “can potentially get arbitrary code execution in a privileged pipeline context,” because the agent’s flexibility inherited the developer’s access level, not some narrower slice of it. Most teams plan their threat model around “what can an anonymous attacker do.” Far fewer plan around “what can any one of our forty authenticated developers do if their account is compromised, or if one of them turns out to be the threat.” An AI agent integration is exactly the kind of thing that turns a low-severity insider-risk question into a high-severity one, because it multiplies what a single set of credentials can reach.

GITLAB CVE-2026-18252, AFFECTED VS PATCHED
----------------------------------------------------------
BRANCH VULNERABLE RANGE PATCHED VERSION
----------------------------------------------------------
18.x 18.9 - latest 18.x 19.1.7 (upgrade path)
19.1 19.1.0 - 19.1.6 19.1.7
19.2 19.2.0 - 19.2.4 19.2.5
19.3 19.3.0 19.3.1
----------------------------------------------------------
CVSS score: 7.3 (High)
Weakness: CWE-829, inclusion of functionality from an
          untrusted control sphere
Required access: authenticated Developer role
GitLab.com SaaS / Dedicated: patched by GitLab, no action needed
Self-managed instances: patch required, urged immediately
----------------------------------------------------------

A patch-cadence process doesn’t need to be expensive to close this gap. It needs to exist and actually run on a schedule, which is the part most small teams skip.

#!/usr/bin/env bash
# check_gitlab_cve.sh
# Self-hosted, no paid vulnerability scanner required.
# Compares your self-managed GitLab version against GitLab's own
# published security release JSON feed and flags known CVEs.
set -euo pipefail

CURRENT_VERSION=$(curl -s "https://your-gitlab-instance/api/v4/version" \
  -H "PRIVATE-TOKEN: $GITLAB_API_TOKEN" | jq -r '.version')
echo "Running GitLab version: $CURRENT_VERSION"
# GitLab publishes security release blog posts with a predictable
# structure; for a production setup, mirror the CVE list into a
# small local file you update whenever GitLab ships a security release,
# and diff your running version against it on a cron job.
KNOWN_VULNERABLE=("18.9.0" "19.1.0" "19.1.6" "19.2.0" "19.2.4" "19.3.0")
for v in "${KNOWN_VULNERABLE[@]}"; do
  if ["$CURRENT_VERSION" = "$v"]; then
    echo "WARNING: running $CURRENT_VERSION, matches a version flagged in CVE-2026-18252 range. Patch now."
    exit 1
  fi
done
echo "No known match in the local CVE list. Still verify against GitLab's security release page directly."

Run that on a weekly cron job against your own instance and you have, for free, most of what a paid vulnerability-scanning subscription would tell you about this one specific class of problem.

Incident three: the deepfake that wanted compute budget, not credentials

The third incident is the one that doesn’t fit the usual “technical vulnerability” shape at all, and that’s exactly why it belongs here. It was reported as a real-time deepfake video call, an attacker convincingly impersonating the actual CEO of an AI company, in a call to a target company, trying to get roughly four hundred thousand dollars approved as spend against AI compute budget, cloud GPU credits, essentially.

What strikes me reading the writeups is how visible the red flags were in hindsight, and how invisible they were in the moment. The follow-up correspondence came from a domain that looked right at a glance, one or two characters off from the real one, exactly the kind of thing a tired person skims past at 6pm. There were unusual traffic patterns around the request, the sort of thing a security team might notice in an access log days later but nobody was watching for in real time during the call itself. And there was manufactured urgency built around a flight, the fake CEO framing the approval as something that had to happen before boarding, no time to loop in anyone else, call me back after I land if you really need to. That last one is the oldest trick in social engineering wearing a new, extremely convincing face.

What actually caught it wasn’t a security control at all. It was a person on the receiving end who felt something was slightly off, and rather than push back inside the call itself, went around it entirely: a direct message to the real CEO’s personal social media account, asking, in plain language, did you actually just get on a call asking for this. The real CEO said no. That single out-of-band question, sent through a channel the attacker had no access to and couldn’t have anticipated being checked, is what stopped roughly four hundred thousand dollars from moving. Not a deepfake detector. Not a domain filter. A human who didn’t fully trust a video call, even a very good one, and had a way to verify it that didn’t run through anything the attacker controlled.

The point I keep coming back to is that this attack wasn’t really after credentials or a system compromise. It was after a budget approval, aimed specifically at the AI compute line item, which is a newer and softer target than a wire transfer request would be, because most finance teams have wire-transfer verification habits built up over decades of BEC scams, but far fewer teams have built the same reflexive suspicion around “approve this cloud compute spend.” Attackers go where the friction is lowest, and right now, AI infrastructure budgets are a line item most companies haven’t yet taught anyone to be paranoid about.

The checklist: what a small team can actually do this week

None of the three fixes below need a security budget or a dedicated team. They need someone to decide to spend an afternoon on it.

For agentic workflows that read public content (GitLost-style risk):

Audit every bot token in your CI and workflow config today. If a token can read more than one repository, ask why, and narrow it to the single repo that workflow actually operates on.
Add an org membership or collaborator check as the first step of any workflow that reads content from public issues or PRs, before that content ever reaches a model.
Route any output the bot would post publicly through a “needs-human-review” label or a draft state instead, at least until you’ve run the workflow safely for a few months.
Test your own guardrails against the “additionally” trick specifically. Open a test issue with an innocuous-sounding request prefixed with a soft transition word and see whether your bot treats it any differently than an “ignore all previous instructions” attempt. If it doesn’t, your guardrail is pattern-matching on obvious attacks and missing polite ones.

For AI integrations wired into CI/CD or other privileged systems (GitLab CVE-style risk):

Subscribe to your CI/CD vendor’s security advisory feed directly (GitLab, GitHub, Jenkins, whichever you run) rather than relying on general tech news to surface a CVE for you.
If you self-host, put a recurring calendar reminder, weekly is reasonable, to check for and apply security patches, specifically for any AI agent or Duo-style feature bolted onto the platform.
Re-examine what your CI pipeline’s execution context can reach. If an authenticated Developer-level account being fully compromised would let an attacker touch production secrets, that blast radius is too wide regardless of whether an AI agent is involved, and an AI integration will only make it easier to hit.
Assume any AI feature plugged into a privileged system inherits the full trust level of whoever can talk to it, not a safely reduced subset. Plan your threat model around your own authenticated users, not just anonymous outsiders.

For anything involving compute budget or spend approval (deepfake-style risk):

Set a dollar threshold, doesn’t need to be exactly four hundred thousand, your own number, above which any approval requires out-of-band verification, no exceptions, no matter who is asking or how urgent it sounds.
Agree on a verification method in advance, before you need it: a pre-agreed phrase changed periodically, or a callback to a phone number you already have on file, never a number given to you during the request itself.
Explicitly include cloud compute and AI infrastructure spend in whatever fraud-awareness training already covers wire transfers. Most teams have built instinct around “wires are dangerous.” Almost none have built the same instinct around “cloud compute budget is dangerous,” and that gap is exactly what this attack was built to exploit.
Practice the “call back on a channel the requester doesn’t control” habit for anything that feels urgent and expensive, the same instinct that caught this one. A personal DM to a known account worked here specifically because it didn’t route through anything the attacker had touched.

# spend_approval_gate.py
# A minimal, self-hosted out-of-band verification gate.
# No paid identity-verification vendor required, just a shared
# secret rotated on your own schedule and a callback number you
# already have on file, never one supplied in the request itself.

import hmac
import time
THRESHOLD_USD = 50_000
# Rotate this weekly; store it somewhere the approval requester
# (or an attacker impersonating them) never has access to, e.g. a
# password manager entry only finance leads can see.
CURRENT_PASSPHRASE = "harbor-quiet-tuesday"
def requires_out_of_band_check(amount_usd: float) -> bool:
    return amount_usd >= THRESHOLD_USD
def verify_out_of_band(spoken_phrase: str) -> bool:
    # constant-time compare so a partial match can't be timed out
    return hmac.compare_digest(spoken_phrase.strip().lower(),
                                CURRENT_PASSPHRASE.lower())
def approve_spend(amount_usd: float, spoken_phrase: str = "") -> str:
    if not requires_out_of_band_check(amount_usd):
        return "approved"
    if verify_out_of_band(spoken_phrase):
        return "approved after out-of-band verification"
    return "BLOCKED: verify via a known callback number or the current passphrase before approving"

That script is deliberately simple. The value isn’t the code, it’s the habit of having a threshold and a pre-agreed check that lives outside whatever channel the request arrived on, so a good enough deepfake still has nowhere to go.

What all three actually have in common

I went looking for a single technical thread connecting these three incidents and didn’t find one, because there isn’t one. GitLost is a prompt injection problem. The GitLab CVE is a privilege boundary problem. The deepfake scam is a pure social engineering problem with zero code involved. Three completely different attack surfaces, three completely different fixes.

But there’s a theme underneath all three that isn’t technical at all, and once I saw it I couldn’t stop seeing it. None of these attacks broke the model. Nobody jailbroke Claude into saying something it shouldn’t, nobody found an adversarial prompt that defeated alignment training, nobody proved a model was less capable or less safe than advertised. The GitLost bot did exactly what a helpful assistant is supposed to do, answer a question using the context it has access to. The GitLab agent did exactly what it was built to do, act on configuration it was handed. The deepfake wasn’t even attacking a model at all, it was attacking a person’s trust in what their own eyes and ears told them on a video call.

What actually broke, every single time, was the connective tissue around the model: which token scope a workflow was handed, whether a patch got applied to a self-managed instance on schedule, whether a request for money got verified through a channel the requester didn’t control. Permissions, patch cadence, and human trust. None of those three things show up in a benchmark score. None of them are what gets discussed when people argue about which model is smarter than which other model. And all three are exactly the layer that most teams, mine included until I actually sat down and checked, spend the least time securing, because it’s unglamorous, and because “the model is safe” quietly gets treated as a stand-in for “the system around the model is safe.” It isn’t the same claim, and this particular week made that difference very hard to ignore.

DEV Community