Agent-Scale Git Infrastructure: GitHub's Per-Reference Quorums and Object Storage Split

GitHub just published the architecture behind moving nearly all repositories to Azure. The interesting part is not the migration itself but the distributed systems plumbing they built to handle what they call "agent-scale" workloads. AI coding agents create access patterns that break traditional Git infrastructure: thousands of concurrent commits, independent read queries for repository state, and consistency requirements that span distributed replicas.

The solution splits into two pieces. Per-reference quorums maintain consistency when agents generate commits across replicas. Object storage separation lets reads scale independently from writes when agents query repository state without blocking on write operations.

The Agent-Scale Problem

Human developers push code in bursts. They clone a repository, work locally for hours or days, then push a batch of commits. AI agents behave differently. They generate commits continuously, query repository state for context retrieval, and run parallel operations across branches. A single agent workflow might:

• Clone a repository to analyze dependencies
• Query commit history for context windows
• Generate and push commits to feature branches
• Read object data for RAG pipelines
• Repeat across hundreds of repositories simultaneously

Traditional Git infrastructure assumes reads and writes share the same storage layer. When agents scale up, read queries block on write locks. Replication lag causes consistency issues when agents read from one replica and write to another. The system needs to handle concurrent writes without coordination overhead and scale reads without waiting for write operations to complete.

Per-Reference Quorums for Consistency

GitHub's per-reference quorum system maintains consistency at the Git reference level (branches, tags) rather than the repository level. Each reference tracks its own quorum state across replicas.

When an agent pushes a commit to a branch:

1. The write goes to the primary replica
2. The system calculates a quorum for that specific reference
3. The write succeeds when a majority of replicas acknowledge
4. Other references in the same repository continue operating independently

This granularity matters because agents often work on separate branches simultaneously. A quorum failure on one branch does not block operations on other branches. The system can accept writes to feature-a even if replicas for feature-b are temporarily unavailable.

The quorum check happens per reference, not per repository. A repository with 100 branches tracks 100 independent quorum states. This adds coordination overhead but eliminates the bottleneck of repository-wide locks.

Object Storage Separation

Git stores data in two layers: references (pointers to commits) and objects (the actual commit, tree, and blob data). GitHub separated these layers into different storage systems.

References live in a strongly consistent datastore that supports quorum operations. Objects live in eventually consistent object storage (Azure Blob Storage). This split enables independent scaling:

• Reference reads query the consistent datastore for current branch pointers
• Object reads fetch commit and file data from object storage without coordination
• Writes update both layers but only block on reference quorum, not object replication

When an agent queries repository state for a RAG pipeline, it reads the reference to get the current commit SHA, then fetches objects directly from blob storage. The object fetch does not wait for quorum or replication. If the agent needs strong consistency, it reads from the reference layer. If it tolerates eventual consistency (common for context retrieval), it reads objects directly.

Architecture Components

Component	Purpose	Consistency Model	Scaling Constraint
Reference Store	Branch/tag pointers	Strong (quorum-based)	Write coordination overhead
Object Storage	Commits, trees, blobs	Eventual	Network bandwidth to blob storage
Quorum Coordinator	Tracks replica state per reference	Strong	Metadata storage for quorum state
Read Proxy	Routes object reads to nearest replica	Eventual	Geographic distribution of replicas

The reference store uses a distributed consensus protocol (likely Paxos or Raft variant) to maintain quorum state. Each reference tracks which replicas have acknowledged the latest write. The quorum coordinator maintains this metadata separately from the Git data itself.

Object storage replication happens asynchronously. When an agent writes a commit, the objects replicate in the background while the reference update blocks on quorum. This means an agent can push a commit, get acknowledgment, and immediately read the reference from any replica. Reading the objects might require fetching from the primary replica until background replication completes.

Implementation Pattern

The migration moved 99.99999% of repositories, leaving a small fraction that hit edge cases. The remaining repositories likely have:

• Extremely large object counts that exceed blob storage limits
• Custom replication requirements (compliance, data residency)
• Active writes during migration windows that caused quorum failures

For the repositories that migrated successfully, the pattern looks like:

class AgentScaleGitBackend:
    def __init__(self, reference_store, object_storage, quorum_config):
        self.refs = reference_store
        self.objects = object_storage
        self.quorum = quorum_config

    def push_commit(self, repo_id, ref_name, commit_sha, objects):
        # Write objects to blob storage (async, no blocking)
        self.objects.write_async(repo_id, objects)

        # Update reference with quorum
        replicas = self.refs.get_replicas(repo_id, ref_name)
        acks = []

        for replica in replicas:
            ack = replica.update_ref(ref_name, commit_sha)
            acks.append(ack)

        # Block until quorum reached
        if len(acks) >= self.quorum.majority(replicas):
            return {"status": "success", "sha": commit_sha}
        else:
            raise QuorumFailure(f"Only {len(acks)} of {len(replicas)} replicas acknowledged")

    def read_commit(self, repo_id, ref_name, consistency="eventual"):
        if consistency == "strong":
            # Read from reference store (quorum read)
            commit_sha = self.refs.read_with_quorum(repo_id, ref_name)
        else:
            # Read from any replica (eventual)
            commit_sha = self.refs.read_any(repo_id, ref_name)

        # Fetch objects from blob storage (always eventual)
        commit_data = self.objects.read(repo_id, commit_sha)
        return commit_data

The key insight is that agents rarely need strong consistency for object reads. They need the correct reference (which branch points to which commit) but can tolerate eventual consistency for the commit contents. This matches RAG patterns where agents retrieve context from a recent commit without requiring the absolute latest version.

Failure Modes

Per-reference quorums introduce new failure scenarios:

Partial quorum failure: Some references in a repository reach quorum while others do not. Agents can continue working on successful branches but cannot push to failed branches. The system needs branch-level retry logic rather than repository-level retries.

Object replication lag: An agent reads a reference that points to a commit, but the commit objects have not replicated to the nearest blob storage region. The read falls back to the primary region, adding latency. Agents need timeout handling for cross-region object fetches.

Quorum state divergence: Network partitions can cause replicas to disagree on quorum state for a reference. The coordinator needs conflict resolution logic to reconcile divergent states when partitions heal.

Metadata overhead: Tracking quorum state per reference scales linearly with reference count. Repositories with thousands of branches accumulate significant metadata. The system needs garbage collection for deleted references and compaction for quorum state history.

Observability Requirements

Agent-scale infrastructure needs different metrics than human-scale systems:

• Per-reference quorum latency: Track how long each reference takes to reach quorum, not just repository-level averages
• Object fetch fallback rate: Measure how often reads fall back to primary region due to replication lag
• Concurrent operation count: Track simultaneous operations per repository to identify agent workload patterns
• Quorum failure correlation: Detect when multiple references in the same repository fail quorum (indicates replica issues)

These metrics expose the distributed systems behavior that agents depend on. A spike in object fetch fallback rate indicates replication lag is affecting agent read performance. Correlated quorum failures suggest replica health issues rather than individual reference problems.

Technical Verdict

Use this architecture when:

• AI agents are primary Git consumers generating continuous commits
• Read scaling needs to exceed write scaling by orders of magnitude
• You can tolerate eventual consistency for object reads but need strong consistency for references
• Your repositories have many branches with independent write patterns

Avoid this architecture when:

• Human developers are the primary users with bursty commit patterns
• Strong consistency is required for all reads (compliance, audit trails)
• Repository count is low enough that vertical scaling handles load
• Operational complexity of distributed quorum systems exceeds your team's capacity

The per-reference quorum pattern trades coordination overhead for independent scaling. It makes sense when agents create workload patterns that break monolithic Git servers but adds complexity that only pays off at scale.

Source Links

• The bridge to agent-scale Git infrastructure (GitHub Blog)