DEV Community

Cover image for Git from First Principles: Building Git from Scratch
Dev Dhanadiya
Dev Dhanadiya

Posted on

Git from First Principles: Building Git from Scratch

#go #git #programming #computerscience

Git from First Principles: Building Git from Scratch

Index

  • Motivation
  • Overview
  • Implementation
    1. Separation of Concerns
    2. Setting up the project
    3. Initializing a Git repository

Motivation

In order to pursue my ongoing interest in systems engineering, I decided to re-implement Git. I'm using Go for this project, but I have kept the core concepts as language-agnostic as possible, so you can follow along in C, Rust, Python or whatever language you like. The key to systems engineering and developing programming intuition is you have to get your hands dirty and challenge yourself with things you find hard. The friction you feel while doing so will translate into learning.

Overview

Almost every developer uses Git, and the common intuition is: "It's just software that saves files and folders." While that idea isn't entirely wrong, it is far from whole truth. A lot of people think creating a new branch in Git copies your whole project. But think about it: if your repo is 2 GB and you make 3 branches, does your disk suddenly lose 6 GB? Obviously not! Branches aren’t copies of your project—they’re literally tiny pointer files.

Git at its core is a content-addressable, file-based object database. It stores snapshots efficiently by decoupling file contents from directory structures, utilizing zlib compression, deduplicating identical objects via SHA-1 hashing, and managing history as a directed acyclic graph.

In this project, we are not going to wrap Git CLI commands using shell execution. Instead, we are building the core functionality from scratch, mirroring how Git's own builtin/ directory is implemented in the official Git C source code.

Implementation

1. Separation of Concerns

Before we start building our Git implementation from scratch, we must decide the high level architecture of our code. It is a fundamental part of systems engineering! I choose the following architecture:

  • Core Functional logic: Contains the internal systems' logic like initializing the repository, building the object store, and managing references.
  • Command Line Interface: Allows the user to run our custom commands in the terminal just like official Git.

I'm too lazy to keep typing "git implementation" throughout this blog, so I am giving it a name: gig (Git-In-Go).

2. Setting up the project

As I am using Go, I will initialize my project with:

go mod init github.com/devxdh/gitingo
Enter fullscreen mode Exit fullscreen mode

If you're using any other language, initialize the project using your language's specific tooling (e.g. cargo init for Rust, npm init for Node).

Now we set up our folder structure:

  • Core Functional Logic directory: pkg/repo/
  • CLI directory: cmd/.
  • Application Entry-point: main.go. 
📁gig
    └── 📁cmd
    └── 📁pkg
        └── 📁repo
    └── main.go
Enter fullscreen mode Exit fullscreen mode

3. Initializing a Git repository

Before using Git VCS, you have to initialize the target directory with git init. Ever wondered why? The difference between a regular directory and a Git repository is simply the presence of the .git folder.

You might have noticed that if you delete .git, your project is no longer a Git repository; it becomes a regular folder again. That tells us something fundamental: everything Git tracks lives exclusively inside that one hidden folder (i.e. .git).

Okay but what is inside .git? It consists of the following:

  • .git/objects: The database where all future compressed files, folder snapshots, and commits will live. Assume you have a src and a file src/index.js. Inside .git/objects, Git stores our project in the following objects:

    • Blob: The raw file content (e.g. src/index.js)
    • Tree: The directory structure and file names (e.g. src)
    • Commit: A snapshot linking your top-level folder structure tree to a commit message and author info.

  • .git/refs/heads: Where our branch pointers (like main) will live. Assume you have two branches: main and test. The refs/heads will contain:

    • refs/heads/main
    • refs/heads/test

  • .git/HEAD: A text file containing ref: refs/heads/main\n to tell Git which branch is active. When you run git checkout main while on the test branch; you are essentially overwriting ref: refs/heads/test\n inside .git/HEAD with ref: refs/heads/main\n.

From here on, I'm too lazy to keep writing 'Git repository', so I'll just call it a repo.

Now that we know what belongs inside .git, let's build the logic for initializing a repo. While official Git supports safe re-initialization, for gig we'll start simple: check if .git exists and exit early to protect existing history.

So first, we write a small directory helper function to check if .git already exists. If it does, we gracefully exit and notify the user. If it doesn't, we go ahead and create our folder structure and write the HEAD file.

A great advantage of building this project in Golang is that anybody who is familiar with programming can read and comprehend Golang like English. Even if you have never seen Go code before, I highly recommend reading code blocks.

// pkg/utils/directories.go

// DirExists checks if an unresolved path exists and is a valid directory
func DirExists(dirPath string) (bool, error) {
    // Retrieves path information via unix stat syscall
    info, err := os.Stat(dirPath)
    if err != nil {
        // "Not found" isn't a failure here; it just means the directory doesn't exist
        if errors.Is(err, os.ErrNotExist) {
            return false, nil
        }

        // Any other error (like permission denied) is a real issue
        return false, err
    }

    // Make sure it's a directory and not a file with same name
    if !info.IsDir() {
        return false, fmt.Errorf("path %s is a file not a folder", dirPath)
    }

    return true, nil
}
Enter fullscreen mode Exit fullscreen mode
  • DirExists(): A reusable helper function. We keep it reusable because we will need it to verify other paths later on.

Next, we wrap that in DotGitExists(). It takes a target directory, joins .git and check if it exists. Having this helper is super convenient because almost every Git command needs to run this check.

// pkg/utils/directories.go

// DotGitExists checks for .git directory in parameter path
func DotGitExists(dirPath string) (bool, error) {
    // filepath.Join() sanitizes segmented paths
    // (e.g. Join("home/", "//work", ".git")) -> "home/work/.git")
    dotGitPath := filepath.Join(dirPath, ".git")

    // Checks if .git is a valid directory in the resolved path location
    exists, err := DirExists(dotGitPath)
    if err != nil || !exists {
        return false, fmt.Errorf("fatal: not a git repository: %w", err)
    }

    return exists, nil
}
Enter fullscreen mode Exit fullscreen mode
  • Notice how DotGitExists() returns Git's famous error message: fatal: not a git repository. In future commands like commit or log, this stops the user if they run a command outside a repo. But in init, it serves the opposite purpose: making sure we are not overwriting an existing repo's history.

Now, with both helpers in place, we move onto building our first core command, gig init.

// pkg/repo/init.go
package repo

import (
    "fmt"
    "os"
    "path/filepath"

    u "github.com/devxdh/gitingo/pkg/utils"
)

func Init(targetDir string) error {
    // Clean the path to eliminate redundant slashes or relative segements
    cleanedTarget := filepath.Clean(targetDir)

    // 1. Safety check: ensures .git does not already exist
    exists, _ := u.DotGitExists(cleanedTarget)
    if exists {
        fmt.Printf("%s is already a repository\n", cleanedTarget)
        return nil
    }

    // 2. Create .git/objects and .git/refs/heads
    // 0755 gives the owner read, write, and execute permissions
    dirs := []string{
        filepath.Join(cleanedTarget, ".git", "objects"),
        filepath.Join(cleanedTarget, ".git", "refs", "heads"),
    }
    for _, d := range dirs {
        // Don't be confused, '0o' is prefix for octals in go
        // '755' is the actual Unix file/folder permissions
        if err := os.MkdirAll(d, 0o755); err != nil {
            return fmt.Errorf("failed to create git directories %s in %s: %v", d, cleanedTarget, err)
        }
    }

    // 3. Write the initial HEAD reference
    headPath := filepath.Join(cleanedTarget, ".git", "HEAD")
    err := os.WriteFile(headPath, []byte("ref: refs/heads/main\n"), 0o644)
    if err != nil {
        return fmt.Errorf("failed to write HEAD into %s: %v", headPath, err)
    }

    fmt.Printf("Initialized Gig repository in %s\n", filepath.Join(cleanedTarget, ".git"))
    return nil
}
Enter fullscreen mode Exit fullscreen mode

Now we need to wire Init() up to our CLI so we can test it from the terminal.

I'm using Go's go-to CLI library, Cobra (the same used by Docker, Kubernetes). You can use any CLI library of your choice (or write a quick flag parser in your language).

Because of our Separation of Concerns, our CLI command doesn't contain any Git logic. Its only job is to read the optional directory argument from the terminal and call repo.Init(). You can choose to skip the CLI code blocks as it may confuse you, but trust me it's nothing hard.

// cmd/init.go

var initCmd = &cobra.Command{
    Use:   "init [directory]",
    Short: "Initialize an empty Git repository",
    Args:  cobra.MaximumNArgs(1),
    Run: func(cmd *cobra.Command, args []string) {
        // Default to current directory if no path is passed
        targetDir := "."
        if len(args) > 0 {
            targetDir = args[0]
        }

        if err := repo.Init(targetDir); err != nil {
            fmt.Fprintf(os.Stderr, "Error: %v\n", err)
            os.Exit(1)
        }
    },
}
Enter fullscreen mode Exit fullscreen mode

To avoid retyping long compiler flags every time we want to test a change, we can drop a minimal Makefile in the project root:

build:
    go build -o gig main.go

clean:
    rm -rf gig
Enter fullscreen mode Exit fullscreen mode

Note
In the actual Github repository, I use a more comprehensive Makefile that builds release binaries, strips debugging symbols, and handles cross-platform compilation. But for following along and testing locally, a simple go build -o gig main.go is all you need.

# Compile the binary
$ make build

# Initialize a new repository
$ ./gig init
Initialized Gig repository in .git

# Inspect what gig just created
$ tree .git
.git
├── HEAD
├── objects
└── refs
    └── heads

3 directories, 1 file

# Check our HEAD reference
$ cat .git/HEAD
ref: refs/heads/main
Enter fullscreen mode Exit fullscreen mode

What’s Next?

Right now, we have an empty database. But Git isn't useful until it can actually store files.

In Part 2, we will dive into Git’s core storage engine: Blobs and Content-Addressable Storage. We'll implement gig hash-object and gig cat-file, exploring how Git takes any raw file, prepends a secret header, hashes it with SHA-1, compresses it with zlib, and stores it in .git/objects.

👉 Continue to Part 2: Storing and Reading Files with Blobs → Coming Soon

You can find the full source code for this project on GitHub: github.com/devxdh/gitingo. If you enjoyed this article so far, give it a star!

Top comments (1)

Collapse
 
suppdevbot profile image
DEV SUPPORTS • 

You need to verify your account.
Enter fullscreen mode Exit fullscreen mode

tr.ee/dev-to