Why Package Managers Keep Trying to Use Git as a Database (And Why It Always Fails)

#architecture #database #git #software

Why Package Managers Keep Using Git as a Database (And Why It Always Fails)

Package managers need a place to store package metadata, version histories, and integrity hashes.

Many of them have chosen Git as that store. The pattern looks convenient—Git already handles versioning, is distributed, and is battle‑tested. In practice it creates more problems than it solves.

The anti‑pattern in practice

Problem	What happens when Git is used as a DB	Why a relational store is better
Concurrent writes	Git uses a single‑writer lock (`git lock`). When two processes push at the same time one fails with “could not lock ref”. The manager must retry or abort.	Relational databases provide row‑level locking and transaction isolation. Multiple writers can succeed without manual retries.
Missing indexes	Package metadata lives in plain text files (e.g., `package.json`). Searching for a package version requires scanning the entire repository or loading many objects into memory.	A relational table can index `name`, `version`, and `checksum`. A simple `SELECT` uses the index and returns in milliseconds, regardless of repo size.
Atomicity	A Git commit groups changes, but the commit is atomic only at the repository level. If a manager writes two related files and crashes after the first, the repository can end up in an inconsistent state.	A DB transaction can roll back all changes if any step fails, guaranteeing consistency.
Scalability	Large registries (hundreds of thousands of packages) blow up the `.git` directory. Object packing becomes costly and fetches slow down.	Relational engines store rows efficiently and can shard or replicate as needed.

A concrete example

Below is a minimal illustration of the same operation—adding a new package entry—implemented with Git vs. SQLite.

Using Git (pseudo‑code)

import subprocess
import json
import os

def add_package(name, version, sha):
    # Write metadata file
    pkg_path = f"packages/{name}/{version}.json"
    os.makedirs(os.path.dirname(pkg_path), exist_ok=True)
    with open(pkg_path, "w") as f:
        json.dump({"name": name, "version": version, "sha": sha}, f)

    # Commit to Git
    subprocess.run(["git", "add", pkg_path], check=True)
    subprocess.run(
        ["git", "commit", "-m", f"Add {name}@{version}"],
        check=True,
    )

Problems:

No lock handling. If two processes call add_package simultaneously, the second git commit fails with a lock error.
To find a version you must walk the packages/ tree or run git grep, both O(N) operations.

Using SQLite (real code)

import sqlite3

def init_db():
    conn = sqlite3.connect("registry.db")
    conn.execute(
        """CREATE TABLE IF NOT EXISTS packages (
               name TEXT NOT NULL,
               version TEXT NOT NULL,
               sha TEXT NOT NULL,
               PRIMARY KEY (name, version)
           )"""
    )
    conn.commit()
    return conn

def add_package(conn, name, version, sha):
    try:
        conn.execute(
            "INSERT INTO packages (name, version, sha) VALUES (?, ?, ?)",
            (name, version, sha),
        )
        conn.commit()
    except sqlite3.IntegrityError:
        raise ValueError(f"{name}@{version} already exists")

Advantages:

SQLite handles concurrency with file locks internally; multiple writers are serialized safely.
A lookup SELECT * FROM packages WHERE name=? AND version=? uses the primary‑key index and returns instantly.
The schema enforces uniqueness, preventing duplicate entries.

Why the pattern persists

Historical inertia – early npm and similar tools were built before robust, embedded DBs were common in JavaScript runtimes.
Perceived simplicity – a single Git repo feels like “just files”.
Distributed nature – Git’s push/pull model matches the idea of a decentralized registry.

These reasons are understandable, but they ignore the cost of maintaining correctness at scale.

Takeaway

Git excels at source code versioning, not at random‑access data queries or high‑concurrency writes. For a package registry:

Use a relational database (SQLite, PostgreSQL, MySQL) for the core metadata store.
Keep Git only for the immutable artifact blobs if you need content‑addressable storage.
Treat the DB as the source of truth for lookups, version resolution, and access control.

Switching away from Git as a database removes lock contention, enables indexed queries, and provides transactional safety—exactly the guarantees a package manager needs.

Source: Package managers keep using Git as a database

DEV Community