Block put Goose in front of 12,000 employees in about eight weeks, at a company that handles Square and Cash App. The worry at that scale is something sensitive leaking. Angie Jones, VP of Engineering for AI Tools and Enablement at Block, described how that rollout grew.
Yeah. Um, there's no playbook for this, Conor.
Angie Jones, VP of Engineering for AI Tools and Enablement at Block, on Chain of Thought episode 48
I'd still stage an internal agent in the order that conversation shows. A company-wide week is a late stage.
Who should get the agent first?
A machine learning engineer at Block started tinkering with a tool to automate developer tasks, before the group was using the word agents. He hit problems with models and tool calling. Anthropic showed Block a draft of the Model Context Protocol (MCP) before launch, and that draft was what Goose needed. The team rebuilt Goose as an MCP client, then launched it as an open source client.
Goose began as a developer tool. The early MCP servers were for engineers, including GitHub. Finance, marketing, sales, design, and executive assistants then asked why engineering got to move faster. That demand is how you notice a second cohort. It is a weak reason to turn the agent on for the whole company the same day.
I'd keep the first cohort in engineering until the agent can reach their systems and you can refuse every other connector. Then the requests came in for Airtable, Workday, Snowflake and Figma, and Block had to build a whole bunch more MCP servers for its employees. Add a job function only after the connectors for that job exist.
What should the pilot measure?
Keep the pilot to a few users and one task you can time. Decide before you start whether the result expands the cohort or stops it.
Block's workshops asked what a team wanted to solve, then showed a custom setup with Goose and MCP servers. They also stood up education and places to get help. I'd time the task that workshop is built around. Record it without the agent and with the agent, on work the cohort already does.
Angie said the common thread across jobs was velocity: people saved time and shipped faster. In one engineering case, Goose sat in Slack, read a bug conversation, used the GitHub MCP server, suggested fixes, and opened a pull request. She said that took five minutes. On sprint work, the team assigned tickets to Goose in Jira and in Linear. A three-week sprint was done a little after a week, at what she called the midway point, and they pulled in more tickets twice.
Yeah. Ran out of work, Conor, when the whole sprint was done.
Angie Jones, VP of Engineering for AI Tools and Enablement at Block, on Chain of Thought episode 48
Also log every connector people ask to install. Thousands of community MCP servers already existed, and Block would not put a random server from GitHub on systems that see financial data. If the named task is not faster, or the asks are for servers you will not review, do not add the next cohort.
What do you switch off when a cohort goes wrong?
Three controls already in Block's setup are what I would use as the kill switch. I would want each one working before the headcount grows.
If a connector is not on the allow list, the agent will not run it. Angie said the install will not work. Security reviews every addition, including servers Block built, because a miss is not always intentional. She said the majority of allowed servers are their own, plus a handful of official corporate MCP servers after the same review.
Tools that can change, edit, or delete are annotated destructive. Non-destructive actions can run. A destructive action waits until a person sees what the agent wants to do and why, and answers yes or no.
Block tied authentication on its own MCP service to its identity provider. The login matches Workday: a browser window, a biometric check, then access. She said API keys, tokens, and scopes were a little tedious for engineers and complicated for everyone else. If the next group would still paste tokens, do not add them.
Prompt injection is why the stop has to cover data the agent reads, not only a bad user message. The security team builds guardrails against prompt injection, and Goose can flag a malicious MCP server when someone connects one. The red team found invisible characters in recipes. A recipe bundles system instructions, a prompt, and MCP servers so someone can share a workflow. Hidden characters in that bundle can push the agent toward harmful actions on the machine. Review shared recipes, and keep the destructive-action check on.
What has to be true before the next cohort?
Demand from other teams arrived after Goose was already a developer tool. That is the moment to apply a gate. This sketch is a review aid I would take into that meeting. Block does not run this code.
# Sketch only. A gate between cohorts for an internal agent.
def allow_next_cohort(state: dict) -> bool:
return all([
state["new_servers_passed_security_review"],
state["internal_servers_were_reviewed_too"],
state["auth_is_identity_provider_oauth"],
state["destructive_tools_wait_for_a_person"],
state["named_task_is_faster_than_baseline"],
state["workshop_targets_that_teams_pain"],
])
Hold the new team to the same workshop standard: start from their pain and end with a custom setup. Model choice is a bad reason to stall them, and a bad reason to skip review. Angie said no one has won the model race, and that locking the company to one model would be foolish because strengths differ. Engineers at Block tend to use Anthropic's Claude models for software development. Other roles that depend more on reasoning may want a GPT model. Databricks hosts the mix they use, including Gemini from Google, GPT from OpenAI, and Anthropic models. Local open source models may require a stronger machine, and she called that group a very small percentage. An approved model can vary by person. An unreviewed server stays off the list.
If the connector boundary is still fuzzy, the episodes on the MCP topic page go further on the protocol. A server security has not reviewed stays off, including one an employee already built.
Checklist
- Start with the group that already needs tool calling, and give them connectors for their systems only.
- Time one real task in the pilot, and log every connector people ask to add.
- Block unlisted servers, and make destructive tools wait for a yes or no.
- Route login through the identity provider before a non-engineering team joins.
- Hold the next cohort until the gate above is true, including review of servers you built.
The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (0)