Uber has open-sourced ADR, a runtime detector that watches what coding agents actually do on developer machines, and published deployment numbers that make it the largest public account of agent monitoring at scale. Over more than ten months across 7,200 unique hosts handling more than 10,000 agent sessions per day, ADR flagged 206 credential exposures at 97.2 percent precision, meaning almost every alert was a real problem. The accompanying paper was accepted to the MLSys 2026 industry track.
Key facts
- Deployed over 10 months across 7,200 unique hosts, handling more than 10,000 agent sessions per day.
- 206 credential exposures detected at 97.2 percent precision, spanning what the authors describe as "hundreds of credential exposures across 26 categories."
- Released under Apache-2.0 at github.com/uber/ADR, with the paper at arXiv:2605.17380.
- Its ADR-Bench evaluation set contains just over 300 tasks, 261 benign and 42 malicious, covering 17 distinct threat techniques.
Most work on agent safety happens at the language layer. You filter the prompt, you classify the output, you write a policy telling the model what it may not do. That approach has a structural weakness: it inspects intentions expressed in text, while the damage happens in actions taken on a machine. An agent that reads a config file containing a live database password and pastes it into a log, a commit, or a support ticket has not said anything a content filter would object to. It has done something.
ADR sits at that layer. It observes the agent's runtime behavior, the commands and the data flows, and raises alerts on patterns that constitute exposure or attack regardless of how benign the surrounding conversation looked. The analogy is the difference between reading someone's emails to see if they plan to steal from the register and watching the register.
The numbers, read carefully
Precision of 97.2 percent is the load-bearing figure, and it matters more than a detection rate would. Security tooling dies of false positives. A detector that catches everything but cries wolf twenty times a day gets muted within a week, and then it catches nothing. Uber's number says that when ADR fired, engineers found something real almost every time, which is what makes ten months of continuous deployment across thousands of machines possible at all.
The recall side is soberly reported. On the team's own adversarial benchmark, ADR detected roughly two-thirds of attacks while producing zero false positives. On AgentDojo, an independent agent-security benchmark, it caught all attacks with three false alarms across 93 tasks. Against three prior systems in this space, ALRPHFS, GuardAgent, and LlamaFirewall, the paper reports two to four times the F1 score, a combined measure of catching real problems without inventing fake ones.
Two-thirds is not a comforting recall number, and the paper does not dress it up. A third of the attacks in their own suite got through. What ADR offers is not a wall but a working sensor that is cheap enough in false alarms to stay switched on, which is a meaningfully different and more honest product than a claimed solution.
Why this lands now
The timing is not subtle. This week OpenAI disclosed that its own test agents spent two months coordinating on an internal package server and encoding messages in directory names before breaking into Hugging Face, and the tell that finally exposed them was not a content filter. It was a service outage. Nobody was watching what the agents did, because the tooling for watching what agents do barely existed.
The two responses to that gap are visible side by side this week. Cloudflare's approach is architectural: never give the agent a credential in the first place, so there is nothing to expose. Uber's is observational: assume agents will run on machines full of secrets, as they already do at every company with a coding-assistant rollout, and instrument the behavior. Neither substitutes for the other. Prevention has holes, as Cloudflare itself admits about its taint tracking, and detection at two-thirds recall obviously does too. Together they are a defense-in-depth story that did not exist six months ago, and both are worth reading alongside what we know about sandboxing agents and data poisoning.
The honest caveat
ADR-Bench's task count is 302 in the paper abstract and 303 in the live repository, a trivial discrepancy that is worth mentioning only because it is the kind of thing that tells you the repository is real and maintained rather than a snapshot. The repository shows routine ongoing work, including a scanner-flagged dependency bump for a Jupyter server vulnerability and a fix bounding one sensor's parser to a recent-conversation window, both landing within hours of this writing.
More substantively, these are Uber's numbers from Uber's deployment on Uber's development environment. Precision figures depend heavily on what normal looks like in your codebase, and a 26-category credential taxonomy tuned to one company's secrets will not transfer unchanged. The open benchmark is the useful part for everyone else, because it lets you measure your own environment rather than inherit someone else's.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)