I have used Git every working day for about fifteen years and I could not have told you, in any detail, what it puts on disk. I knew the shape of the story, the one with the blobs and the trees and the commits pointing at each other, because everybody knows that story. What I did not know was whether I could sit down and produce those bytes myself, without the tool, and have the real thing accept them.
So I tried it. No library, no shelling out to git, just Python with hashlib and zlib, writing files into .git/objects and seeing how far I could get before real Git told me I had done it wrong.
It told me I had done it wrong sooner than I expected, and the specific way it complained turned out to be the most interesting part.
A blob is almost nothing
The first object type is the easy one. A blob is the contents of a file, prefixed with a tiny header, hashed, compressed, and dropped in a directory named after the first two characters of the hash.
header = f"{kind} {len(payload)}".encode() + b"\x00"
full = header + payload
oid = hashlib.sha1(full).hexdigest()
open(f".git/objects/{oid[:2]}/{oid[2:]}", "wb").write(zlib.compress(full))
That is the whole of it. The header is the word blob, a space, the length in decimal, and a zero byte. The hash is taken over the header and content together, which is the one detail that catches people, because hashing the file alone gives you a hash that nothing in Git will recognise.
Writing hello world through that function gave me 3b18e512dba79e4c8300dd08aeb37f8e728b8dad, and asking Git for its opinion gave the same forty characters. git cat-file -p printed my file back. A blob has no name, no permissions, no timestamp and no history, and two identical files anywhere in any repository are the same object, which is the reason a repository full of duplicated files is not as expensive as it looks.
Trees are where I got it wrong
A tree is a directory listing. For each entry: an octal mode, a space, the filename, a zero byte, then twenty raw bytes of hash. Not hex, raw. Concatenate the entries, wrap them in a tree header, write it out the same way.
The obvious thing to do with a list of entries is sort them by name before writing, so that is what I did. I made a small repository with a README.md, a file called lib.txt and a directory called lib, then built the tree, wrote a commit and pointed refs/heads/master at it.
git log showed my commit. git ls-tree listed all three entries with the right hashes. Everything looked correct, so I ran git fsck mostly out of politeness:
error in tree ff04258ee8f650cd761ef58148080860cb2e5f8a: treeNotSorted: not properly sorted
Git sorts tree entries by name, but it compares a directory as though its name ended in a slash. My naive sort put lib before lib.txt, because that is what a string comparison does. Git wants lib.txt first, because it is comparing lib.txt against lib/, and the dot is 0x2E while the slash is 0x2F.
One character of difference in a table nobody looks at, and the fix is four lines:
def key(entry):
mode, name, _ = entry
return (name + "/") if mode == "40000" else name
With that, fsck went quiet and the tree hash changed, because the bytes had changed order, and in a content-addressed store a different order is a different object.
The thing I keep turning over is that git log, git ls-tree and git cat-file had all read the broken tree without a murmur. Git's readers are forgiving. Its verifier is not. If I had only checked that my repository looked right, I would have shipped a subtly corrupt object database and found out weeks later, on somebody else's machine, in a way that would have been very hard to trace back to a sorting rule.
Commits are a text file with an unusual name
After the tree drama, commits were an anticlimax. A commit object is plain text:
tree 595b3bb66abe5068cd37f4c86933c2cce0c28f7a
author Kalin <kalin@example.com> 1790000000 +0300
committer Kalin <kalin@example.com> 1790000000 +0300
first commit, written by hand
A tree line, zero or more parent lines, an author and a committer with a Unix timestamp and a numeric timezone, a blank line, then the message. That is the format that has carried every project you have ever cloned.
Writing a second commit with a parent line pointing at the first gave me history. git log --graph drew it. git diff HEAD~1 HEAD showed the change to lib/main.py as a normal unified diff, computed on the fly from two blobs I had written by hand, because Git does not store diffs at all at this level. It stores whole files and works out the difference when you ask.
Then I cloned the repository, with real Git, over the filesystem, and got a working copy with all three files and both commits in the log.
The part I forgot existed
Before the clone worked, there was a stretch where the repository was correct and completely unusable.
git status reported all three files as deleted. git checkout . refused, saying the pathspec did not match anything Git knew about. Both were right. I had built an object database and a ref, and those describe history perfectly well, but neither of them is the thing status and checkout consult first.
That thing is the index, a separate binary file at .git/index, and it holds the staging area along with a cache of what is in the working directory. My repository had objects and refs and no index at all, so as far as the plumbing was concerned, HEAD listed three files and the staging area listed none, which reads as three deletions waiting to be committed.
git reset --hard HEAD built the index and the working tree from my objects, and after that everything behaved. The lesson I took is that the diagram everyone draws, the one with blobs and trees and commits, is only two thirds of a repository. The index is the third piece and it is the one you interact with most, every time you type git add.
Fifty revisions later
With the writer working I wanted to see the other half of how Git survives. Loose objects, the kind I had been producing, store every version of every file in full. Compressed, but in full, with nothing shared between one revision and the next, which means a file edited fifty times sits on disk as fifty complete copies of itself.
So I generated a source file of about 1,200 functions, 54,066 bytes, and then made fifty revisions of it, each one changing three random lines, writing every version through my own code. That is 150 objects, fifty blobs and fifty trees and fifty commits:
loose objects: 150 files, 425,894 bytes on disk
Then I let Git pack them:
packfile: 21,558 bytes
Nineteen times smaller, for the identical history, with git fsck --strict still passing and every revision still retrievable.
Looking inside the pack is where it becomes concrete. Of the fifty blobs, one is stored whole: 54,066 bytes of content held as 8,296 bytes of compressed data. The other forty-nine are stored as deltas against other blobs, and the median delta is 48 bytes.
Forty-eight bytes to record a revision of a fifty-four-kilobyte file, which is about what you would expect if the only thing being written down is the handful of lines that changed and their positions.
The deltas chain, too. Git had built chains up to twenty-one deep, so reconstructing some revisions means starting from the one full copy and applying twenty-one patches in order. That is the trade it makes: storage is cheap, a little CPU at read time is cheaper than keeping fifty copies.
It also explains something I had never connected. A repository whose history is full of small text edits packs down to almost nothing, because each revision is a few dozen bytes of delta. A repository with fifty versions of a binary asset does not, because there is no small delta to find between two differently-compressed images, and you keep fifty full copies. The advice about not committing large binaries is usually given as a rule, and this is the mechanism underneath it.
What I got wrong, twice
The tree sorting was one, and I had half expected it, because it is the kind of rule you only learn by breaking it.
The second was my own measurement. To inspect the pack I ran git verify-pack -v and wrote a small script to summarise the delta chain depths, which cheerfully reported chains of depth 15,710 and 16,949. I believed it for a moment, then noticed that a chain twenty thousand deep in a fifty-object pack is not possible.
verify-pack prints five columns for an object stored whole and seven for one stored as a delta, and I had been reading column five in both cases. For the full objects that is the offset in the packfile, which is a large number that looks plausible if you are not paying attention. Reading the right column gave depths between one and twenty-one, which is the number I published above.
It is the same failure I keep finding in my own work: the script ran, produced numbers, and the numbers were the wrong field entirely. Nothing crashed. Nothing warned me. The only thing that caught it was the figure being physically impossible, and if the offsets had happened to be small, it would not have been.
If you want to try it
The writer is about sixty lines and needs nothing but hashlib and zlib. Start with blobs, because you can check yourself against git hash-object immediately and know within a minute whether your header is right. Then trees, where the slash rule is waiting for you. Then commits, which are the easiest of the three despite sounding like they should be the hardest.
Run git fsck --strict after every change rather than at the end. I lost a while to a tree that four different Git commands read without complaint, and the only reason I caught it at all was running a verifier I had no particular reason to run. git log looking correct is not evidence that the bytes are correct.
The thing that surprised me most has nothing to do with the format. It is that the efficiency everyone associates with Git is not in the object model at all. The object model is almost naive: whole files, hashed and compressed, one copy per version, no sharing. All of the cleverness arrived later and sits in the packing layer, which took my 425,894 bytes of honest loose objects and reduced them to 21,558 without changing a single hash or losing a single byte of content. The part I was reimplementing was never the part that made Git work.
Top comments (2)
Hand-building loose objects is the fastest way to internalize why
git fsckexists. Did anything get corrupted on the way and fsck catch it, or did real Git only start complaining at pack time? The delta/LZMA boundary is where my own mental model always got fuzzy — the loose object format is forgiving, the packfile is not.fsck caught it, and well before packing. The broken tree was a loose object and
git fscknamed it straight away astreeNotSorted. What bothers me is that it only spoke because I ran it out of politeness:git log,git ls-treeandgit cat-file -phad all read that same tree without a word. The readers are forgiving, the verifier is not.Packing never complained at all, because by then the bytes were already right, and
git fsck --strictpassed after the repack too.On the boundary you mention, there is no LZMA anywhere in it. Packfiles are zlib deflate sitting on top of the delta encoding, the same compressor loose objects use. The loose-to-pack difference is not the compressor, it is that loose objects store every version whole. Fifty revisions of a 54,066-byte file came to 425,894 bytes loose and 21,558 packed, with one blob stored full and forty-nine as deltas, median 48 bytes.