DEV Community

Cover image for I Turned an S3 Bucket Into a Git Remote—No Server Required
Robert Pitt
Robert Pitt

Posted on

I Turned an S3 Bucket Into a Git Remote—No Server Required

#git #aws #opensource #devtools

Most Git remotes eventually involve running or depending on a server.

That means another service to deploy, secure, monitor, upgrade, and keep available—even when all you really need is somewhere to push and fetch a repository.

I wanted something simpler:

  • Normal Git commands
  • Storage I already control
  • Authentication through AWS credentials and IAM
  • No server, database, mounted filesystem, or custom account system

So I built git3: a Git remote helper that stores complete Git repositories in Amazon S3 or S3-compatible object storage.

How git3 connects Git to S3

Install it in one line

git3 supports Linux and macOS on amd64 and arm64, and requires Git 2.38 or newer.

curl -fsSL https://github.com/robertpitt/git3/releases/latest/download/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Once installed, an S3 URL behaves like a Git remote:

git clone s3://my-bucket/repos/example
Enter fullscreen mode Exit fullscreen mode

That is the basic idea: your S3 bucket becomes the remote.

Authentication uses the standard AWS credential chain

There is no separate git3 account system.

If your AWS credentials are already configured, clone normally:

git clone s3://my-bucket/repos/example
Enter fullscreen mode Exit fullscreen mode

To use a named AWS profile:

AWS_PROFILE=development git clone s3://my-bucket/repos/example
Enter fullscreen mode Exit fullscreen mode

You can also provide credentials through environment variables:

AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_DEFAULT_REGION=us-east-1 git clone s3://my-bucket/repos/example
Enter fullscreen mode Exit fullscreen mode

This also means existing AWS authentication methods—including shared profiles, IAM Identity Center, web identity, and compute roles—remain responsible for resolving credentials.

Credentials, profiles, regions, endpoints, and encryption settings never need to appear in the Git remote URL.

Creating a repository

The S3 bucket must already exist, but you do not need to initialize anything inside it.

Add the remote and push a branch:

git remote add origin s3://my-bucket/repos/example
git push -u origin HEAD:refs/heads/main
Enter fullscreen mode Exit fullscreen mode

The first valid branch push creates the repository under that bucket prefix.

A single bucket can contain multiple repositories:

s3://my-bucket/repos/api
s3://my-bucket/repos/frontend
s3://my-bucket/repos/infrastructure
Enter fullscreen mode Exit fullscreen mode

Which Git operations work?

git3 supports the everyday full-repository workflows I wanted it to feel boringly normal to use with.

Command Support
git clone Full repository clones
git fetch Incremental synchronization
git pull Fetch and integrate normally
git push Create or update branches and tags
git push --delete Delete branches and tags
git push --force Forced ref updates
git push --force-with-lease Lease-checked forced updates
git push --atomic Atomic multi-ref pushes
git submodule Supported with s3:// protocol permission

Signed commits and signed tags are preserved as ordinary Git objects.

Both SHA-1 and SHA-256 repositories are supported without translating between object formats.

How can Git use an S3 URL?

Git supports external remote helpers.

When Git encounters this:

git clone s3://my-bucket/repos/example
Enter fullscreen mode Exit fullscreen mode

it invokes a helper named git-remote-s3.

git3 implements that helper protocol and translates Git’s fetch and push operations into S3 reads and conditional writes.

The flow looks roughly like this:

  1. Git invokes git-remote-s3.
  2. Native Git creates and verifies packfiles.
  3. git3 uploads immutable packs and transaction records.
  4. A small HEAD document publishes the new ref state.
  5. Fetch installs native packs directly into .git/objects/pack.

There is no Git server between the developer and the bucket. Git computation happens on the workstation or CI runner.

Native Git repositories, not a mounted filesystem

git3 does not pretend that an S3 bucket is a POSIX filesystem.

It also does not replace Git’s local object database with a custom implementation.

After a successful clone or fetch, the repository contains normal Git packfiles. Standard local commands continue working:

git log
git checkout
git fsck
git repack
Enter fullscreen mode Exit fullscreen mode

You could even remove git3 after fetching and the local repository would remain an ordinary Git repository.

That property was important to me: S3 is the remote storage layer, but Git still owns the local repository.

Publishing ref updates safely

S3 objects are not updated using filesystem locks, so git3 uses conditional requests as a compare-and-swap operation.

Most repository data is immutable. One small mutable HEAD object identifies the current repository state.

A push:

  1. Reads and validates the current state.
  2. Creates and uploads immutable object data.
  3. Builds a transaction describing the ref updates.
  4. Replaces HEAD only if it has not changed since it was read.

If another writer publishes first, the conditional write fails instead of silently overwriting their update.

Accepted multi-ref changes are published through one HEAD replacement, which makes the visible ref update atomic.

Fetching without downloading everything again

Each client keeps a verified logical cursor.

An active client fetches only the transactions and Git data published after that cursor. Physical pack identities are not treated as repository history, so remote or local repacking does not force clients to start again.

When nothing has changed, a fetch requires:

  • One conditional S3 read
  • Zero Git-object bytes
  • No material local writes

Cold clones download the current compacted packset and the bounded transaction tail.

No S3 listing during normal Git operations

Normal clone, fetch, push, and maintenance operations do not need s3:ListBucket or s3:DeleteObject.

A reader can be limited to:

{
  "Effect": "Allow",
  "Action": ["s3:GetObject"],
  "Resource": "arn:aws:s3:::BUCKET/PREFIX/.git/git3/*"
}
Enter fullscreen mode Exit fullscreen mode

A writer adds s3:PutObject and multipart-upload permissions.

Listing and deletion are reserved for explicit garbage collection. GC is dry-run by default and uses a published, resumable barrier before deleting anything.

git s3 gc origin
git s3 gc origin --execute --older-than 30d
Enter fullscreen mode Exit fullscreen mode

Maintenance and diagnostics

git3 includes several administrative commands:

git s3 doctor origin
git s3 fsck origin --full
git s3 maintenance origin
git s3 gc origin
git s3 set-head origin refs/heads/main
Enter fullscreen mode Exit fullscreen mode

Maintenance compacts immutable packs for efficient cold clones without changing the logical repository history.

S3-compatible services

Alternative S3-compatible endpoints can be configured without changing the remote URL:

export GIT3_ENDPOINT=https://objects.example.com
export GIT3_PATH_STYLE=true
Enter fullscreen mode Exit fullscreen mode

An endpoint needs the S3 behavior git3 relies on, including conditional writes, range requests, multipart uploads, and compatible error semantics.

What git3 intentionally does not provide

git3 is a storage and synchronization tool, not a GitHub or GitLab replacement.

It does not currently provide:

  • Shallow or partial clones
  • Integrated Git LFS storage
  • Server-side hooks
  • Protected branches
  • Pull requests or code review
  • Merge queues
  • A web interface
  • git archive --remote
  • Git wire-protocol server endpoints
  • Windows release binaries

If you need an independent LFS service, it can still be configured separately.

There is also an important security boundary: any AWS principal that can overwrite the reserved S3 prefix is effectively a repository administrator. git3 cannot enforce branch protection against someone who already has direct write access to the underlying repository metadata.

Where might this be useful?

git3 may be a good fit when:

  • Your organization already uses S3 and IAM extensively
  • You want a private Git remote without operating a service
  • CI workers already receive temporary AWS credentials
  • You want repository storage inside your own cloud boundary
  • You need Git synchronization but not forge features
  • You want a remote that can work with compatible object-storage platforms

It is probably not the right choice for a high-traffic public forge, a workflow centered on pull requests, or a very large monorepo that depends on partial cloning.

Try it

git3 is open source under the Apache-2.0 license:

github.com/robertpitt/git3

I would especially appreciate feedback on:

  • The S3 storage and concurrency model
  • Which missing Git operation matters most to you
  • S3-compatible services you would like to see tested
  • Workflows where a serverless Git remote would be useful

Would you use an S3 bucket as a Git remote? Let me know in the comments.

Top comments (0)