DEV Community

Cover image for Built for Humans, Flooded by Agents: Why Git Infrastructure Is Being Rebuilt
James Anderson
James Anderson

Posted on AI-assisted

Built for Humans, Flooded by Agents: Why Git Infrastructure Is Being Rebuilt

#git #ai #distributedsystems #devops

In May 2026, Anthropic merged around 2,500 code changes a day into its monorepo. By September, it was merging more than 30,000.

That's not a typo, and it's not because they hired twelve times more engineers. Agents were writing the code. And the thing that broke wasn't the agents. It was Git.

Within roughly two weeks in October, three very different organizations published how they're rebuilding source control for this new kind of traffic: Anthropic, GitHub, and Enroute. Andrew Nesbitt noticed the pattern in his weekly source control roundup, and once you read all three side by side, it's hard to unsee.

They took three completely different approaches. But underneath, they arrived at surprisingly similar ideas.

Git was designed for a human at a laptop

Git is decentralized by design. Every clone is a complete repository. You can read history, run blame, diff, and check out any revision while fully offline. Pushes are occasional, because humans are slow.

For the vast majority of projects, that's still the right design. Anthropic's own post says so directly.

Agents break almost every one of those assumptions.

GitHub's numbers show how fast things changed. In September 2026 alone, developers and agents made 7.38 billion commits on GitHub, more than five times as many as a year earlier. Pushes grew 4.9x year over year, from 0.69 billion to 3.35 billion a month. GitHub Actions ran 3.26 billion times in the same month. Total Git activity on the platform more than doubled in a year, from 218 billion events a month to 473 billion.

GitHub's engineering post breaks down what that load actually does to a Git server:

  • Commit latency becomes the agent's speed limit. An agent in a tight loop commits after nearly every action. Latency no human would notice becomes the bottleneck.
  • Writes converge on one point. Thousands of agents on their own branches in one repository produce a sustained write rate that hits a single part of the architecture.
  • Merges fight over one ref. Trunk-based development and merge queues funnel everything into main.
  • Every push becomes thousands of reads. CI and code scanning fetch the same branch tip thousands of times per minute.
  • Maintenance never stops. Every write adds compaction and garbage collection work, and the cost compounds.

Anthropic's scale makes this concrete. Its main branch history is almost 100 GB, and a compressed working copy is over 25 GB. CI performs about 25 million checkouts a day. Fetched naively, Anthropic estimates that would be over 600 PB of Git traffic per day.

They tried everything Git offers first: shallow clones, sparse checkouts, promisor remotes, repository snapshots kept in blob storage. At their peak, they ran several hundred internal Git mirrors just to absorb the load. It bought time, but it didn't scale.

So what did each team do?

Approach 1: GitHub fixes the plumbing, without changing Git

GitHub can't replace Git. A billion repositories depend on it working exactly as it does. So GitHub is rebuilding the infrastructure underneath while keeping Git itself untouched.

To understand the problem, you need to know how GitHub stores repositories today. Its storage system, Spokes, keeps a full copy of every repository on the local disks of several file servers, five by default. When a push updates a ref, a three-phase commit protocol uses a quorum so everyone sees a consistent state.

Here's the catch GitHub identified: the mechanism for durability is the same one used for scale. The copies on disk are the source of truth, so adding read capacity means adding another durable replica. And every replica participates in every write. A push is only as fast as the slowest replica.

So adding replicas to handle read load makes writes slower. For most repositories, that trade-off is fine. For agent-scale repositories, it's a ceiling.

GitHub's redesign rests on two ideas:

Coordinate only what needs agreement. The only part of a push that truly needs consensus is the ref update itself. Storing objects, validating connectivity, and secret scanning are expensive, but they can happen in parallel. That shrinks the critical path of a push to one small step.

Separate storage from compute. The authoritative copy of each repository moves into Azure Blob Storage. Read capacity comes from lightweight compute workers that cache data and can be added or removed as traffic changes. Losing a worker becomes closer to a cache miss than a disaster. Maintenance like compaction and garbage collection moves to separate workers, off the hosts serving live requests.

A follow-up post describes one concrete bottleneck along the way. Moving the busiest repositories to Azure was blocked by a repository-wide lock that every write held across a 25 ms cross-region round trip. GitHub is switching to per-reference serialization, so only writes to the same ref wait for each other. That alone gave 3–4x write throughput in internal tests. Across the full new architecture, GitHub reports up to 35x higher write throughput in internal benchmarks.

Approach 2: Enroute keeps Git, but rebuilds storage like a database

Enroute, an open source Git server, takes a middle path. Clients still speak normal Git. But the storage engine underneath was rebuilt from scratch.

Enroute's starting observation is that many existing "Git on object storage" designs still send every push through one primary node, and every replica still keeps a full copy of the repository on disk. That's the same scaling ceiling GitHub hit, just moved somewhere else.

Instead, Enroute stores each repository as a Raft-replicated log of refs plus a list of immutable object layers in object storage like S3. New pushes add layers, which get merged in the background, so there's no git gc to schedule. Packfiles are rewritten on ingestion so that reading from object storage needs only a few requests.

The result: more than 7,500 pushes per second into a single repository, with every push written to a majority of ledger nodes before it's acknowledged.

Enroute is also refreshingly honest about the trade-off. Against stock Git serving a repacked linux.git, full clones are about 17% slower, while shallow clones are more than twice as fast. That's a sensible bet when most of your traffic is CI and agents doing shallow fetches, not humans cloning entire histories.

Approach 3: Anthropic leaves Git behind

Anthropic went furthest. A team of seven built Scion, an in-house version control system, and migrated its entire monorepo off Git in five months.

The key realization was this: like most companies, Anthropic was already using Git as a centralized system with one authoritative copy. And nearly all of its clients were agents and CI jobs, which are online by definition. Git's best feature, working offline with a full copy, was the very thing preventing it from scaling.

So Scion flips Git's model:

The server does the thinking. In Git, every client can act as a full server. In Scion, the server answers history and blame queries, creates commits, and performs merges and rebases. The client keeps only a bounded local cache. It can compute very little on its own, but it needs far less data.

Commits are changesets in a log. A Git commit points to a full tree. A Scion changeset is an entry in a write-ahead log on Spanner that lists only the files that changed, with content-addressed pointers to blobs in storage. There are no packfiles, no locking, and refs are discouraged except for main and release branches, because global state every server must agree on is "a looming scaling problem." Moving a ref is just a compare-and-swap on a single database row.

Expensive queries use derived data. Since a changeset only records what changed, Scion builds extra structures like blame indexes, path history, and skip lists over the commit graph. Each is defined recursively from a changeset's parents, and can be computed eagerly or on demand.

The fetch protocol is inverted. Git's fetch negotiation tells the server what the client already has, and the server computes what's missing. Scion's client instead asks only for specific objects it wants, walking down from the root tree. Every request is for an immutable object, so any cache layer can answer it. Sparse and virtual checkouts need no special support: the client just doesn't ask for what it doesn't need.

That last point enables a tiered cache: a daemon per machine, regional caches, and authoritative servers behind them. Each tier maintains a cache hit rate above 99%. Anthropic reports the new infrastructure is 200x smaller in CPU and memory than its old Git mirror fleet, and CI checkouts now take seconds instead of minutes.

Not entirely new, and worth saying so

When Scion's post first went up, engineer Rain (sunshowers) pointed out that its "derived data" design closely matches Mononoke, the server side of Meta's Sapling, which the original post credited only for "hard-won lessons." Anthropic updated the post two days later to say Mononoke pioneered many of the techniques Scion uses, especially its Bonsai data model, and to name Google's Piper as prior art too.

It's an important footnote. Companies with enormous monorepos solved many of these problems years ago. What's new is that agents are pushing far smaller organizations into that territory, and much faster.

Anthropic is also clear that Scion isn't a Git replacement for general use. It only makes sense for truly enormous monorepos, it requires its own code review, search, and CI, and it isn't being released.

The pattern underneath

Put the three approaches side by side and the same ideas keep appearing:

Idea GitHub Enroute Anthropic
Separate durable storage from serving Azure Blob + compute workers Object storage layers Spanner + blob storage + caches
Shrink coordination to the ref update Per-ref quorums Raft log of refs Ref move is one compare-and-swap
Immutable, content-addressed data Git objects Immutable layers Content-addressed blobs and changesets
Move maintenance off the hot path Separate workers Background layer merging No packfiles, no maintenance
Keep the Git client Yes Yes No

If that table looks familiar, it's because these are database ideas. Write-ahead logs, consensus on small pieces of state, immutable data that caches freely, compaction in the background. Git hosting is going through the same evolution databases went through when they had to scale beyond a single machine.

The real difference between the three is just one row: how much of Git's original model each is willing to give up.

What this means for the rest of us

Most teams won't hit these limits next month. But Anthropic expects this class of problem to become common within six to 12 months, and the ripple effects are already visible.

GitHub now says that during periods of unusual traffic, unauthenticated HTTPS clones of public repositories may temporarily require authentication. It's also previewing push attribution, which records who actually pushed a commit alongside the author and committer, something that matters more when many pushes come from agents.

Anthropic's post also suggests Git itself could absorb some of these ideas without abandoning its model: richer server-side APIs across forges, a different negotiation protocol, and better composition between Git's shallow and sparse features.

In the meantime, if your CI does full clones on every run, that's the easiest win available today. Git already supports most of what these teams are pushing toward:

# Shallow clone of just the latest commit
git clone --depth 1 https://github.com/org/repo.git

# Partial clone: fetch file contents only when needed
git clone --filter=blob:none https://github.com/org/repo.git

# Combine partial clone with sparse checkout for big monorepos
git clone --filter=blob:none --sparse https://github.com/org/repo.git
cd repo
git sparse-checkout set services/api
Enter fullscreen mode Exit fullscreen mode

It won't get you to 30,000 merges a day. But it moves your pipeline in the same direction the biggest teams are going: fetch only what you need, when you need it.

The bigger question

Git won because it was built for a world of distributed humans working offline. That world hasn't disappeared, and for most projects Git is still exactly right.

But a growing share of commits now come from machines that are always online, never sleep, and push thousands of times a day. For that traffic, all three teams independently concluded the same thing: the server has to do more, the client has to do less, and coordination has to shrink to the smallest possible piece.

So here's what I keep wondering: if most commits in a few years come from agents, does Git's distributed design remain the default, or does it become the special case?

I'd love to hear from anyone running CI or agent workloads at scale. Have you started feeling this yet?

Sources

Top comments (0)