An independent AI-safety researcher publishing as cereblab captured the network traffic of xAI's Grok Build CLI coding tool and found that it uploads an entire code repository -- including a .env file full of secrets and files the AI was explicitly told never to open -- to a Google Cloud Storage bucket named grok-code-session-traces. Crucially, turning off the tool's "Improve the model" setting did not stop the upload. In the researcher's blunt phrasing, "Opting out does not stop your repository from leaving the machine."
Key facts
- On a 12 GB repo the AI never read, only 192 KB moved through the live model channel while 5.10 GiB left via a separate storage channel -- a roughly 27,800x volume ratio.
- With the prompt "Reply with exactly: OK. Do not read or open any files," Grok still uploaded the full 298-file repo as a git bundle; cloning the captured bundle recovered a planted, never-read file with the unique marker CANARY-XR47P2-NEVERREAD-UNIQUE verbatim.
- The destination bucket, grok-code-session-traces, is named in the binary's own strings; tested binary grok 0.2.93, SHA-256 published. Full teardown, repro repo, Hacker News.
The hook is the never-read file. Coding agents normally send a model only what it reads -- open a file, its contents go up as part of the conversation. That is expected. What cereblab documented is a second, quieter channel that ships the whole project regardless of what the AI touches. The proof is elegant: on a real repo, with an instruction not to read anything, the tool uploaded a git bundle that -- when cloned -- reconstructed the entire repository including a file planted specifically to test this, carrying a one-of-a-kind marker string. The model demonstrably never ingested those files (192 KB of model traffic cannot carry 5 GiB of code), yet the code left anyway.
Background a non-expert needs: a .env file is where developers keep secrets -- API keys, database passwords, tokens. "Redaction" means stripping those before sending anything to a third party. The teardown says no redaction of .env contents was applied; the secrets went up verbatim through both the live model turn and the storage channel, the latter destined for the named Google Cloud bucket. Think of hiring a contractor to fix one room and discovering they quietly photocopied every document in your house, including the safe, and mailed the copies to their head office -- even after you asked them not to look in the other rooms.
Why it matters: this lands in the middle of a broader reckoning over what AI coding tools transmit -- the same week a separate teardown measured how many tokens coding agents send before you type. It is the sharpest version yet of the question the safety community keeps asking: proprietary agent runners are opaque, and you cannot see what they add in the next update. On Hacker News, one commenter (phaseleza) recommended sandboxing agents with tools like bubblewrap to restrict filesystem and network access to only the model provider's hostnames.
The honest caveat is the most important part, and cereblab foregrounds it. The teardown proves transmission, acceptance (HTTP 200 responses), and storage -- not training. The researcher writes plainly: "None of this proves xAI trains on the data -- that is a policy question." They also retracted an earlier claim (an initial low-egress reading was wrong because it missed a separate upload coordinator process), did not test whether .gitignored files are uploaded, and note the "not documented" claim is scoped to the install script and quickstart, not an exhaustive docs audit. That discipline -- a named "what I did not prove" section, published hashes, a reproducible repo -- is what makes the core claim credible. As of publication, xAI had not responded. The responsible read is not "xAI is training on your code" but "xAI's coding tool moves your entire repository, secrets included, off your machine and stores it, and the obvious opt-out does not stop that."
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)