Welcome to another story in the "Lessons Learned" series, where we discuss real-world vulnerabilities from the perspective of an application security engineer, focusing on the underlying root causes and the measures we can take to prevent similar issues in our applications.
In today's story, we discuss a write-up showing how a missing input sanitization check on a git push option was enough to achieve remote code execution on GitHub's backend infrastructure and access millions of private repositories belonging to other customers. You can find the full write-up here.
Impact of the Vulnerability
A critical RCE (Remote Code Execution) vulnerability assigned CVSS 8.7, allowing any authenticated user to execute arbitrary commands on GitHub's backend storage nodes using nothing but a standard git push command. This allowed access to millions of private repositories.
How the git push Pipeline Works
To understand the vulnerability, we first need to understand what happens when you run git push against GitHub:
- babeld — A git proxy that acts as the entry point. It receives the SSH connection along with all push options and credentials.
-
gitauth — An authentication and authorization service. It verifies the user's credentials and checks whether they have write access to the target repository. Once this is done, it builds an internal header called
X-Statcontaining all security metadata — who you are, your push options, file size limits, branch naming rules, and so on — and passes it downstream. -
gitrpcd — An internal RPC server running on the storage nodes. It receives the request from
babeld, parses theX-Statheader, and sets up the environment for downstream processes. Critically, it performs no authentication of its own — it fully trusts everything in theX-Statheader. - The pre-receive hook — A compiled Go binary that enforces security policies before a push is accepted.
The key thing to note: once past gitauth, nothing authenticates again. All services on the storage nodes trust the X-Stat header completely.
The Vulnerability: X-Stat Option Injection
The X-Stat header carries its fields as semicolon-delimited key=value pairs, and uses last-write-wins semantics — if a key appears twice, the later value silently overrides the earlier one.
Git supports a feature called push options (git push -o), which are arbitrary strings a user can pass to the server. babeld takes these push option values and embeds them directly into the X-Stat header — without sanitizing semicolons.
Since ; is the X-Stat field delimiter, a push option value containing a semicolon breaks out of its designated field and creates new, attacker-controlled fields. For example:
git push -o 'anything;large_blob_rejection_enabled=bool:false'
This results in the X-Stat header being built this way
X-Stat: ...; large_blob_rejection_enabled=bool:true; ...;
push_option_0=x;large_blob_rejection_enabled=bool:false;
push_option_count=1; ...
And when the service on the storage nodes splits on ; to process, this header parses as:
push_option_0 = x
large_blob_rejection_enabled = bool:false ← INJECTED (overrides earlier bool:true)
push_option_count = 1
This produces a header where large_blob_rejection_enabled appears twice — first set to true by the legitimate flow, then overridden to false by the injected value. The attacker's value wins because it appears later.
This is a form of option injection — very similar to command injection, but instead of injecting a command, we are injecting a configuration option that changes how the service behaves.
Escalation to RCE
Bypassing individual flags like large_blob_rejection_enabled is interesting, but the real question is: can this be turned into code execution?
The pre-receive hook binary supports custom pre-receive hooks — admin-defined scripts that run before a push is accepted. By reverse engineering the binary, the security researchers discovered it has two execution paths controlled by the rails_env field from the X-Stat header: a production path that runs hooks inside a sandbox, and any other value that runs hooks directly — no sandbox, no isolation — as the git service user with full filesystem access.
The escalation to RCE chains three injections together:
-
Bypass the sandbox: Inject a non-production
rails_envvalue to switch to the unsandboxed execution path. -
Redirect the hook directory: Inject
custom_hooks_dirto control where the binary looks for hook scripts. -
Path traversal to arbitrary execution: Inject
repo_pre_receive_hookswith a crafted hook entry whose script path resolves — via path traversal — to any binary on the filesystem.
The result: a single git push -o command with carefully crafted options executes arbitrary code as the git service user.
Extending the Attack to GitHub.com
The exploit worked primarily on GitHub Enterprise Server. However, the researchers found that GitHub.com had an additional boolean flag in the X-Stat header controlling whether the server operated in enterprise mode. On GHES this defaults to true; on GitHub.com it defaults to false. Since this flag was also carried in the X-Stat header, injecting it as well was enough to make the full exploitation chain work on GitHub.com.
With code execution on a shared GitHub.com storage node running as the git user — a user with broad filesystem access to every repository on that node by design — the researchers confirmed that millions of repository entries belonging to other users and organizations were accessible.
The Fix
GitHub mitigated this by properly sanitizing git push option values before embedding them in the X-Stat header, ensuring semicolons and other delimiter characters are not allowed. As noted in GitHub's own post, the fix was straightforward: since the server already knows the list of push options it accepts, a strict allowlist of expected values or characters is sufficient to make this class of injection impossible.
GitHub also applied additional defense-in-depth measures, removing software from the storage nodes that didn't need to be there.
Lessons Learned
1. Input Validation — Simple but Powerful
The root cause of this vulnerability was a single missing sanitization check: push option values were embedded into an internal header without stripping the delimiter character. A tight input validation rule — for example, not allowing semicolons in push option values, or even better, only allowing a list of expected values — would have prevented this entirely.
This is a good reminder that input validation is one of the most powerful and cost-effective security controls available. It protects against a wide range of injection attacks — SQL injection, command injection, XSS, and as we see here, option injection — because most injection attacks rely on user input containing special characters that allow breaking out of the intended context. If those characters are never allowed in, the attack surface shrinks dramatically.
One nuance worth highlighting from this specific case: push options are a nested input — an option within an option. It is easy to apply input validation to the top-level parameters and miss that one of those parameters itself contains another list of values. Make sure your input validation applies to all levels of nesting, not just the outer layer.
2. Tools Are Useful, But Not Sufficient
A natural question when reviewing this vulnerability is: would a SAST tool have caught this? The answer is most likely no — at least not with default, out-of-the-box rules.
Security scanners typically look for known dangerous patterns and known dangerous sinks. In this case, the injection happened through an internal protocol header that is constructed across service boundaries — babeld builds the header, gitrpcd parses it. No single component looks obviously vulnerable in isolation; the issue only emerges when you understand how data flows across the full pipeline.
This is why tools alone are not enough. You need:
- Threat modeling during the design phase to reason about how user-controlled input flows through your multi-service architecture and where it could end up in a dangerous context. For more on threat modeling, you can check my Threat Modeling Handbook series.
- Security code review to catch missing input validation and unsafe data handling that automated tools may miss, especially across service boundaries.
3. Apply Least Privilege to Every Component
As part of GitHub's fix, they noted that some software running on the storage nodes had access to things it didn't need. This is a reminder of the least privilege principle: every component of your application — whether a worker node, a storage node, or a microservice — should only hold the credentials and permissions it actually needs to do its job.
The practical implication: when you provision credentials for a component, scope them down. If a service only needs to read from one table, don't give it access to the whole database. If it only needs to write to one S3 prefix, don't give it access to the full bucket. This doesn't prevent RCE from happening, but it significantly limits the blast radius when it does.
4. Tenant Isolation — Application Level Is Not Always Enough
The most significant impact of this vulnerability was broken tenant isolation: one GitHub customer could access the private repositories of other customers. For any SaaS or multi-tenant application, this is one of the most serious failure modes.
In this case, gitauth did enforce tenant isolation at the application level — it checked whether the user had access to the target repository. But the injection vulnerability allowed bypassing that entirely by achieving code execution on the storage node directly, after authentication had already happened.
This points to a broader principle: application-level tenant isolation can be bypassed if an attacker can execute code on the underlying infrastructure. Where possible, it is worth adding a platform-level layer on top — for example, ensuring that the node or process handling a request only has access to the data belonging to that specific customer, not all customers.
A practical example: if you have worker nodes that process jobs per customer — say, generating PDFs — you can scope the database role assigned to each worker to only have access to rows belonging to that customer's ID, and scope the S3 permissions to only allow writes to that customer's prefix. Even if an attacker achieves full code execution on that worker node, they cannot reach other customers' data.
A note on trade-offs: It is important to acknowledge that this is not always feasible. In GitHub's case, the storage architecture is optimized for storage efficiency (e.g. deduplication across forks of public repositories) and resilience (e.g. three replicas per repository with consistency and partition tolerance prioritized). Fully isolating storage nodes per customer would break both of these properties. As a security engineer, your job is not to always push for the most secure option, but to understand these trade-offs, put the risk in context, and help the team take an informed decision — documenting the accepted risk and the justification for it clearly.
Conclusion
Three services, each making a perfectly reasonable assumption about the data passing through them. None of those assumptions was wrong in isolation — and together they created a critical RCE on one of the world's largest platforms. The patterns that made this possible — unsanitized input in an internal protocol, implicit trust across service boundaries, no defense-in-depth at the platform level — appear across many codebases and architectures.
The takeaway is not that multi-service architectures are inherently insecure. It is that as systems grow more complex, the assumptions each component makes about the data it receives need to be made explicit, validated, and tested — especially at the boundaries where user-controlled input enters your internal infrastructure.
Stay tuned for another story and another set of Lessons Learned!
NOTE: This analysis is also available on my YouTube channel in video form on https://youtu.be/XPcTuNEPLu0
Top comments (0)