Intro
Every engineering org has a policy about which AI tools developers are allowed to use. Every engineering org also has a Shadow AI problem, whether leadership has noticed it yet or not.
The gap between the two is predictable. Security approves one internal tool, or none at all. Developers still have deadlines, still get stuck, still need to paste a stack trace somewhere and ask "why is this failing." If the sanctioned option doesn't exist, is slow, or is worse than what's publicly available, developers go find their own. Quietly.
What actually gets pasted into public chat windows
This is where it gets concrete. A developer debugging a failing integration test copies the full stack trace into a public AI chat window to ask what's wrong. The stack trace includes a database connection string. Nobody flagged it because nobody was watching, the tool has no audit log and no data retention policy the company controls.
A different developer is stuck on a tricky business rule and pastes the actual proprietary algorithm, the one the company patented, into a public model to get a "cleaner implementation." That algorithm may now sit inside a public training corpus, permanently, with no way to un-paste it.
A third developer, working through a customer support bug, pastes real customer records, including names, emails, and account details, into a chatbot to understand a data corruption issue. That's a compliance violation the moment it happens, whether or not anyone downstream ever notices.
None of these developers think of themselves as a security risk. They think of themselves as someone trying to ship on time with the tools that actually work.
Why banning the tools doesn't work
The instinct from leadership is usually to block the domains at the network level and call it solved. This doesn't remove the underlying need, it just removes the visibility. Developers switch to personal devices, personal accounts, or a browser extension nobody in IT has heard of. The paste-your-code-into-a-public-model behavior doesn't stop. It just becomes invisible to the company that's supposed to be accountable for it.
That's arguably worse than the original problem. An org with visible Shadow AI usage at least knows it has a problem. An org that's driven the behavior underground has the same exposure with none of the ability to see or measure it.
The actual failure is a tooling gap, not a discipline gap
Developers aren't circumventing policy because they're careless. They're circumventing it because the approved path is slower, worse, or nonexistent, and the unapproved path solves their actual problem in thirty seconds. Every time an organization treats this as a training or discipline problem instead of a tooling problem, it gets the same result: policy on paper, Shadow AI in practice.
The fix isn't a stricter acceptable-use policy. It's giving developers an approved tool that's actually good enough that going around it stops being worth the trouble, one that keeps proprietary code, schemas, and customer data inside a boundary the company actually controls, with the visibility to know it's being used instead of guessing.
Anyone else seeing this play out on their team? Is it a policy problem, a tooling problem, or both?
Top comments (14)
I feel that underneath it really is a mindset problem. Over the years I have witnessed countless times that employees or freelancers feel entitled to do anything to finish their task or to "safe the project". A capable, approved tool helps of course, but it relieves the symptoms, not more. The same is true for training and policies. At the end of the day it comes down to hiring, I guess. Find the right people who understand that they do not only act in technical and financial contexts, but also always in a legal context. Easier said than done of course.
The stack trace point is the crux of it for me. It's not a values problem, it's a mental model problem. Most engineers don't think of a stack trace as sensitive data the same way they think of a customer record, and that blind spot doesn't go away with better hiring. At 2am during an incident, "is this string sensitive" is not the question anyone's brain is running. That's exactly when guardrails matter most, not because the person is bad but because good judgment is the first thing pressure erodes. So I'd frame it as: hiring reduces how often the risky moment happens, tooling determines what happens when it does anyway. Same reason you don't skip code review just because you trust the engineer.
Nobody in the thread has touched how you would actually size this, and I think that is the weak spot in every version of this conversation. Most orgs estimate shadow AI from a survey, and a survey measures who is comfortable admitting it, so the number comes back reassuring while the exposure sits exactly where it was.
You see this across a lot of teams, so genuine question: is there any signal that tells you the real size before the sanctioned tool exists, or is instrumenting the good path the only way anyone ever finds out?
Good question, and I think there are a couple of pre-sanction signals, they're just lossy. Egress is the main one: DNS and proxy logs hitting the AI endpoints, plus expense trails for personal subscriptions. That tells you a request left the building, not what was in it, so it sizes the order of magnitude but not the shape. The high-fidelity read is the one you already named, instrumenting the good path turns the estimate into a measurement, because once the sanctioned tool exists usage migrates onto it and stops being a guess. So yes, there's an outside-in signal before the tool, you just don't see the real shape until you've built the path people actually want to use.
Magnitude but not shape, that lands. The lossy number is still worth pulling early though, because the alternative is walking into the budget conversation with nothing, and "we cannot size it yet" is how the sanctioned tool stays unbuilt for another quarter.
Expense trails I forget every time, and it is the one signal that already arrives with a number attached.
Yeah, the budget point is the one that keeps this from being purely academic. "We can't size it yet" reads as "we don't have a problem" to anyone controlling spend, even when everyone in the room knows better. A lossy number with a stated confidence range beats no number, because at least it forces the conversation onto whether the exposure at the floor is acceptable, rather than whether the problem exists at all.
Expense trails are underrated for exactly the reason you said, they arrive pre-quantified and pre-attributed. The other nice thing about them is they're one of the few signals where the direction of the bias is knowable: reimbursed personal AI subscriptions undercount because plenty of people just eat the cost, so whatever you see is already a floor. That's a much easier number to defend in a budget meeting than "our DLP flagged N events, of which some unknown percentage were real."
Yeah, that framing lands for me. Egress gives you order of magnitude, sanctioned tooling gives you shape, and the two answer different questions. The pre-sanction signals are enough to know whether you have a problem worth solving, but they can't tell you what people are actually pasting, which is the part that determines whether you're dealing with mild policy drift or an active compliance incident.
The migration point is the one I'd underline. Once the sanctioned path exists and is actually good, usage consolidates onto it fast, not because people are being disciplined but because it's easier. That's when you finally see the shape: which teams use it most, what kinds of prompts dominate, how much of it is code vs. customer data vs. internal docs. And that shape almost always surprises leadership, because the assumed use case ("developers debugging code") turns out to be maybe half of it, with the rest being support, ops, and analysts doing things nobody modeled.
The other thing that shows up post-sanction is the residual egress. Once the good path exists, whatever traffic still goes to public endpoints is a much more interesting signal than it was before, because now it's not "people who had no alternative," it's "people who had an alternative and chose not to use it." That's where you learn whether your sanctioned tool actually meets the need or just meets the policy.
Honestly, before the sanctioned tool exists, I don't think there's a clean signal, only proxies that all undercount. DNS and egress logs catch the lazy case but miss personal devices, tethering, and browser extensions talking to less obvious endpoints. DLP can flag large paste events but is usually too noisy to leave on. Expense reports for personal AI subscriptions are a surprisingly good signal in orgs that reimburse them, useless everywhere else. Every one of these is filtered through "what am I willing to admit to my employer," which is exactly your point.
The uncomfortable answer is that instrumenting the good path is the only channel where developers have no incentive to hide, so it's the only place you get honest numbers. Which creates the chicken-and-egg you're pointing at: can't justify the investment without sizing the problem, can't size the problem without the investment.
The way teams usually break that loop is to stop trying to size it precisely and treat whatever they can measure as a floor, not a ceiling. Survey says 20%, assume 50. DNS logs show light usage, assume heavier on personal devices. Then the question becomes whether the exposure at that floor is already unacceptable, which it usually is, and the precise number stops mattering.
Most companies fail because the AI Act creates confusion on what they can do and what not. Also the financial side of the AI use should be considered. I think it's the same debate we had 15 years ago with shadow IT.
Really good parallel, the Shadow IT comparison is exactly right and I think a lot of orgs are about to relearn the same lesson the hard way. The AI Act piece is interesting too, because the regulatory ambiguity actually makes the tooling gap worse, not better. Legal says "we're not sure what's allowed," security says "then nothing is allowed," and developers still have work to do, so they just quietly use whatever works and hope nobody asks. The financial angle you mentioned is the other half of it. Approved enterprise AI often costs 5-10x what a personal ChatGPT subscription does, so orgs end up either underprovisioning seats or restricting access to a small group, which guarantees everyone else finds their own way in. Same pattern as Shadow IT: the answer was never stricter policy, it was giving people sanctioned tools that were actually usable and priced in a way that made compliance the path of least resistance.
Tooling gap not discipline gap is exactly the right reframe. People do not paste a stack trace with a live connection string into a public window because they are careless, they do it because the approved path costs thirty seconds more and they are trying to ship. The arguably worse point is the one I would push hardest on with leadership: visible shadow AI at least tells you the size of your problem, driving it to personal devices keeps the exposure and removes your ability to measure it. The only durable fix I have seen is the boring one you land on, make the sanctioned tool good enough that going around it stops being worth the trouble, and keep the sensitive data inside a boundary you actually control.
Thanks, and yeah, that middle point is the one I've had the hardest time getting leadership to actually sit with. There's a strong instinct to treat "we blocked the domains" as risk reduction, when in most cases it's just risk relocation. The exposure moves off corporate networks and onto personal phones, home wifi, and browser extensions nobody's inventorying, and the org loses the one thing it had going for it, which was the ability to see the behavior at all. Measured risk you can act on beats invisible risk you can't, every time.
The boring fix being the durable one is something I've made peace with slowly. There's no clever policy, no training program, no acceptable-use document that beats "the sanctioned tool is genuinely good and the sensitive data stays inside a boundary you control." Everything else is theater that makes leadership feel better while the paste-into-public-window behavior continues on personal devices. The unglamorous work of building or buying something developers actually want to use is the whole game.
The examples here are all about data flowing out (stack traces, proprietary algorithms, customer records), but there's a second direction that gets less attention: what flows back in. A public model's suggestion, based on training data or web context nobody vetted, gets pasted into the codebase and merged, often without the scrutiny a PR from an internal tool with logging and review hooks would get. So Shadow AI isn't just a leak risk, it's an unreviewed-input risk too, code and "fixes" entering production from a source with no provenance. The tooling-gap framing covers this well since a governed tool can enforce review on the way back in, not just visibility on the way out, but it's worth naming as its own failure mode rather than folding it into "data exfiltration."
This is a really good addition and I wish I'd made room for it in the post, because you're right that the outbound framing dominates the conversation and the inbound side gets folded into "AI code quality" as if it's a separate discussion. It isn't. It's the same governance gap running in the other direction.
The provenance point is the one I'd sit with longest. Code from a public model has no traceable source, no license clarity, no way to know if it was regurgitated from a GPL repo or hallucinated from nothing, and it enters the codebase through the same paste-and-commit path as any other snippet. If it came from an internal tool with logging, you'd at least have "here's the prompt, here's the response, here's who accepted it," which is the minimum you'd expect from any other input to production. Without that, you're merging code with less scrutiny than you'd apply to a Stack Overflow copy-paste, because at least Stack Overflow has a URL and a license.
The other piece worth naming is that the inbound risk compounds silently in a way the outbound risk doesn't. A leaked connection string is a discrete incident you can eventually detect and rotate. Unreviewed AI-generated code becomes load-bearing over months, gets built on by other developers, and by the time someone asks "where did this pattern come from," the answer is nowhere and everywhere. That's a much harder cleanup than a credential rotation.
Agreed that a governed tool addresses both directions, and framing it as "visibility on the way out, review hooks on the way in" is a cleaner pitch to leadership than treating them as separate problems.