It is easy to roll out an AI coding tool with a license and a Slack announcement and leave the training for later. Rizel Scarlett, a developer advocate at Block in our January 2025 episode, wants it the other way round. Companies already run new hires through security and harassment training, and she argues they might also need a "how to code with AI" training. Her baseline is short:
Like don't just check-in code, review it just as you would review a co worker's code and stuff like that.
Rizel Scarlett, on Chain of Thought ep 10
Here is how I'd turn that into a training plan and a merge checklist.
What does the agent have access to?
Scarlett works on Goose, Block's open-source, on-machine developer agent. It has a Rust back end and an Electron front end, and you can drive it from the terminal or the app. It connects to your file system and to other tools through Model Context Protocol servers, and Block collaborated with Anthropic on building out MCPs. She describes it as useful for almost any engineering task because it is so extensible.
That extensibility is the first thing to teach. Before anyone reviews code, they should know what the agent could touch while writing it: which directories, which MCP servers, which credentials sit in the environment. My concern is that a reviewer who only sees the diff may not see what else the agent read or ran. So the first session of the training is an inventory, and the first rule is to start a new agent in a scratch directory with no production data until the team has seen how it behaves.
How do you keep juniors confident without making them dependent?
Scarlett's favorite argument cuts against the "AI replaces juniors" line. When you are a junior developer, she says, you are scared to ask questions, and sometimes you do not understand the answer when you do. For her, starting out young and as a Black woman engineer added another layer of that fear.
But I'm like, if I had this when I was a junior developer, this would have made me so much more confident.
Rizel Scarlett, on Chain of Thought ep 10
Her point is that a junior can ask the tool why it chose something and why a method exists, as many times as they need. It works like a peer programmer. I would build the training around that use: ask why, check the answer against the docs, then write the test yourself. The agent explains; the developer still owns the proof.
This is also where prompting technique belongs. Show developers few-shot prompting, the prompt-level form of few-shot learning: put two or three examples of the code style you want into the prompt. Then show them that a well-prompted answer still needs review.
What should the training cover?
Scarlett's list is plain: keep writing tests, keep testing manually, and learn how hallucinations work and how to reduce them. I would teach it in four short sessions:
- Access: what the agent can read, run, and change on your machine, and how to scope it.
- Hallucinations: how a model produces plausible code that calls a function that does not exist, and how to catch it.
- Review: read agent output the way you would read a coworker's pull request, line by line.
- Tests: write or extend the tests yourself, then run them and test the change by hand.
AI safety in a team setting starts with these habits. The guardrails go in before the agent runs, and the review happens before the merge.
A sketch of the pre-merge procedure you could print on the training's last slide:
# Sketch only: a pre-merge procedure for agent-written changes, not a tool.
PRE_MERGE = [
"List the files, servers, and credentials the agent could reach.",
"Diff the change against the original file and read every line.",
"Check for new dependencies or network calls.",
"Run the test suite, then test the behavior by hand.",
"Explain the change to a teammate without rereading the prompt.",
]
Why does open source change the review job?
Scarlett liked working with GitHub Copilot but did not know what was happening under the hood, and could not have as much influence as she wanted. At Block she can see feedback and push her own code to improve Goose, and so can community members. She also calls open source "authentic marketing": contributors talk about the project at conferences because they feel part of it.
For a team adopting an open-source agent, that has a practical side. You can read the code and propose fixes, and community contributions keep changing it. Treat each update like any other dependency bump: read the release notes, rerun your checks, and do not let an agent upgrade itself into production. The open source AI topic page has more conversations on how these projects grow.
Checklist
- Run a "how to code with AI" session for every new hire, next to security onboarding.
- Inventory what an agent can reach before anyone reviews what it wrote.
- Review agent output like a coworker's pull request, line by line.
- Keep writing tests and testing by hand; do not take the agent's word for it.
- Treat open-source agent updates as dependency upgrades with their own review.
The full conversation, with the transcript, is on Chain of Thought.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (0)