DEV Community: Oliver Seifert

pgit: What If Your Git History Was a SQL Database?

Oliver Seifert — Tue, 17 Mar 2026 06:09:58 +0000

Originally published on oseifert.ch

TL;DR: Built a Git-like CLI backed by PostgreSQL with automatic delta compression. Import any git repo, query its entire history with SQL. Benchmarked on 20 real repositories (273,703 commits): pgit outcompresses git gc --aggressive on 12 out of 20 repositories, while giving you full SQL access to every commit, file version, and change pattern. Then I gave an AI agent a single prompt and it produced a full codebase health report on Neon's own repo in under 10 minutes.

What is pgit?

pgit is a Git-like version control CLI where everything lives in PostgreSQL instead of the filesystem. You get the familiar workflow (init, add, commit, push, pull, diff, blame), but your repository is a database. And that means your entire commit history is queryable.

pgit init
pgit import /path/to/your/repo --branch main
pgit analyze coupling

file_a                    file_b                    commits_together
────────────────────────  ────────────────────────  ────────────────
src/parser.rs             src/lexer.rs              127
src/db/schema.go          src/db/migrations.go      84
README.md                 CHANGELOG.md              63

No scripts. No parsing git log output. No piping things through awk. Just answers.

The most common analyses are built in as single commands: churn, coupling, hotspots, authors, activity, and bus-factor. All support --json for programmatic consumption, --raw for piping, and display results in an interactive table with search and clipboard copy.

But everything is PostgreSQL underneath. When the built-in analyses aren't enough, drop down to raw SQL:

The coupling analysis above, as raw SQL

SELECT pa.path, pb.path, COUNT(*) as times_together
FROM pgit_file_refs a
JOIN pgit_paths pa ON pa.path_id = a.path_id
JOIN pgit_file_refs b ON a.commit_id = b.commit_id
  AND a.path_id < b.path_id
JOIN pgit_paths pb ON pb.path_id = b.path_id
GROUP BY pa.path, pb.path
ORDER BY times_together DESC;

This finds every pair of files changed in the same commit, counts co-occurrences, and ranks by frequency. The a.path_id < b.path_id condition avoids counting the same pair twice. pgit analyze coupling optimizes this further: it computes pairs in memory and filters out bulk reformats (commits touching 100+ files) that produce noise, not signal.

Want to know your maintenance hotspots? That's pgit analyze churn. Or as SQL:

Churn analysis as raw SQL

SELECT p.path, COUNT(*) as versions
FROM pgit_file_refs r
JOIN pgit_paths p ON p.path_id = r.path_id
GROUP BY p.path
ORDER BY versions DESC;

Counts how many versions (commits) exist per file and returns the most-modified files. This is exactly what pgit analyze churn runs under the hood.

Under the hood, pgit uses pg-xpatch, a PostgreSQL Table Access Method (basically a custom storage engine) that I built on top of my xpatch delta compression library (I wrote about building xpatch here). When you insert file versions, pg-xpatch automatically stores only the deltas between consecutive versions. When you SELECT, it reconstructs the full content transparently. You just write normal SQL.

Why Did I Build It?

After building xpatch, a delta compression library that hits 2-byte medians on real code repositories, I kept asking myself: "Where could delta compression be useful where it isn't used yet?"

Databases were the obvious answer. Every application that stores versioned data (document editors, audit logs, config history) is keeping full copies of content that's 99% identical to the previous version. Delta compression could save massive amounts of storage, but nobody builds it into the database layer itself.

So I started building pg-xpatch: a proper PostgreSQL Table Access Method that does delta compression transparently. I tried SQLite first, but its extension API is limited and write performance with custom storage was painfully slow. PostgreSQL was a completely different story: the extension API is powerful, and the results were immediately promising.

But I needed to benchmark it. And from my xpatch work, I already knew that git history is the perfect test corpus: millions of incremental text changes across thousands of files, easy to obtain, representative of real-world editing patterns. So I started importing git repositories into PostgreSQL to stress-test the compression.

And at some point the benchmark tool became the actual project. That became pgit. It turned out to be the best decision I could have made, not just as a product, but as dogfood. Running pgit against real repositories surfaced bugs, edge cases, and performance problems in pg-xpatch that no synthetic benchmark would have caught. Things like: what happens when a single file has 79,000 versions in one delta chain? What about repositories with 30,000+ files per commit? pg-xpatch now has 570+ tests and handles all of it without issues.

Benchmarks: git vs pgit

Here's where it gets interesting. I benchmarked pgit against git on 20 real repositories across 6 languages (Rust, Go, Python, JavaScript, TypeScript, C), totaling 273,703 commits. The comparison is pgit's actual compressed data size versus git gc --aggressive packfile size, the best git can do.

The scorecard: pgit 12 wins, git 8 wins.

Repository	Commits	Raw Size	git --aggressive	pgit	Winner
serde	4,353	203.5 MB	5.6 MB	3.9 MB	pgit (30%)
ripgrep	2,208	111.8 MB	3.0 MB	2.7 MB	pgit (10%)
tokio	4,403	195.7 MB	8.3 MB	7.7 MB	pgit (7%)
cargo	21,850	1.2 GB	29.9 MB	29.8 MB	pgit (0%)
fzf	3,499	213.3 MB	3.5 MB	3.0 MB	pgit (14%)
gin	1,967	51.7 MB	1.9 MB	1.7 MB	pgit (11%)
cli (GitHub)	10,820	288.6 MB	41.8 MB	41.3 MB	pgit (1%)
flask	5,516	167.3 MB	6.1 MB	5.5 MB	pgit (10%)
requests	6,405	112.4 MB	9.3 MB	9.1 MB	pgit (2%)
express	6,128	150.0 MB	5.8 MB	5.7 MB	pgit (2%)
core (Vue)	6,930	598.9 MB	11.6 MB	11.2 MB	pgit (3%)
svelte	10,982	782.7 MB	96.5 MB	96.0 MB	pgit (1%)
ruff	14,206	2.8 GB	51.3 MB	51.4 MB	git (0%)
hugo	9,538	570.6 MB	108.8 MB	111.0 MB	git (2%)
prettier	11,084	2.0 GB	66.1 MB	91.1 MB	git (38%)
react	21,378	2.2 GB	105.0 MB	112.3 MB	git (7%)
jq	1,871	121.2 MB	3.9 MB	4.2 MB	git (8%)
redis	12,940	2.0 GB	71.6 MB	76.9 MB	git (7%)
curl	37,860	3.3 GB	48.4 MB	49.3 MB	git (2%)
git	79,765	7.3 GB	90.6 MB	111.3 MB	git (23%)

Let me put this in perspective. git gc --aggressive is git's best compression mode. It's significantly slower than normal git gc and is designed to squeeze out every byte. pgit outcompresses it on the majority of repositories while making the entire history SQL-queryable. Against normal git gc (the numbers are in the full benchmark), pgit wins on nearly every repository.

The results split along a clear line. pgit wins on repositories where most changes are incremental edits to source code, which is the majority of the benchmark suite. Delta compression within each file's version chain captures most of the redundancy, and pgit's path-to-group mapping deduplicates content from renames, copies, and reverts automatically by grouping related files into shared delta chains.

Git pulls ahead on repositories with heavy cross-file similarity between unrelated files: prettier (38%), git/git (23%). Git's packfile format can delta-compress any object against any other object in the entire repository, regardless of file path. pgit deduplicates renamed and copied files, but unrelated files with similar content are compressed independently.

What surprised me is how far targeted deduplication gets you. Grouping related files into shared delta chains, without git's arbitrary cross-object matching, beats git's best mode on 12 out of 20 repositories. And you get SQL queryability on top.

It's Not Just About Storage

You might expect that storing everything in delta-compressed PostgreSQL tables would kill query performance. It doesn't. Here are real numbers on the git/git repository (79,765 commits, 7,291 files):

Command	Time
show	0.23s
diff	0.18s
blame	0.7s (warm cache), 7.3s (cold)
log	1.5s
stats	0.13s

That's sub-second for most operations on a repository with 79k commits. The trick is working with pg-xpatch's storage model: use normal heap tables for metadata lookups (paths, refs, file hashes) and only touch delta-compressed tables when you need actual file content. Primary key lookups and front-to-back sequential scans are fast; JOINs onto compressed tables and COUNT(*) on delta chains are not.

This is documented in the xpatch query patterns guide, which is worth reading if you work with any kind of columnar or compressed storage, since the principles apply broadly.

Use Cases

pgit isn't trying to replace git for your daily development workflow. Git's ecosystem (GitHub, CI/CD, IDE integrations, merge tooling) is unmatched, and pgit doesn't compete with any of that.

What pgit does well is let you understand a codebase's history programmatically. Things like:

Coupling analysis: which files always change together? (reveals hidden dependencies)
Churn detection: which files have the most versions? (identifies maintenance hotspots)
Size trends: how has the codebase grown over time? (tracks architectural health)
Bus factor: which files have only one contributor? (knowledge silos)
Full-text search across history: pgit search "TODO" --path "*.rs" --all searches every version of every file
Custom analytics: any question you can express in SQL, you can answer

The most common analyses are built in, no SQL needed:

# most frequently modified files
pgit analyze churn
# files always changed together
pgit analyze coupling
# churn aggregated by directory
pgit analyze hotspots --depth 2
# files with fewest authors
pgit analyze bus-factor
# commit velocity over time
pgit analyze activity --period month

All of these support --json for programmatic consumption, --path for glob filtering, and display results in an interactive table. For anything beyond the built-ins, drop down to raw SQL with pgit sql.

These are the kinds of analyses that engineering teams either build custom tooling for, pay for expensive third-party services, or (most commonly) just don't do at all because the barrier is too high. With pgit, the barrier is a single command, or a SQL query if you need something custom.

pgit for Agents

Here's what I think is the most interesting use case, and the one I'm most excited about.

AI coding agents are getting good. Really good. They can read code, write code, run tests, fix bugs. But there's one thing they're still bad at: understanding the history of a codebase. When an agent modifies a file, it doesn't know that this file has been reverted 5 times in the last month. It doesn't know that every time someone touches tenant.rs, they also need to update timeline.rs. It doesn't know that the function it's about to refactor has been growing by 20 lines per quarter for two years.

Agents already speak SQL, or at least the models powering them can write it trivially. What they're missing is a SQL-queryable interface to git history.

To test this, I gave Claude Opus 4.6 a short prompt:

Analyze the Neon database repository (https://github.com/neondatabase/neon) using pgit, a git-like CLI backed by PostgreSQL that makes git history SQL-queryable. It is globally installed. Import the repo and produce a short codebase health report covering: most frequently modified files, file pairs most often changed together, codebase size trends over time, and the largest current files. Refer to https://github.com/ImGajeed76/pgit/blob/main/docs/xpatch-query-patterns.md for SQL performance guidelines.

No step-by-step instructions. No hand-holding. Just a description of what I wanted.

In 9 minutes and 36 seconds, it produced a full codebase health report. It figured out pgit --help on its own, imported the repository (8,471 commits), wrote optimized SQL queries following the performance guidelines, and delivered actionable findings:

Most frequently modified files:

File	Versions
Cargo.lock	743
pageserver/src/tenant/timeline.rs	676
test_runner/fixtures/neon_fixtures.py	579
pageserver/src/tenant.rs	562
pageserver/src/http/routes.rs	434

Strongest file coupling:

File A	File B	Co-changes
tenant.rs	timeline.rs	289
Cargo.lock	Cargo.toml	257
tenant.rs	http/routes.rs	174
image_layer.rs	delta_layer.rs	104

Largest files at HEAD:

File	Size
pageserver/src/tenant.rs	476 KB
storage_controller/src/service.rs	434 KB
pageserver/src/tenant/timeline.rs	329 KB

The agent's summary was genuinely insightful: "tenant.rs at 476 KB with 562 versions is the top candidate for decomposition." It spotted that the pageserver subsystem dominates every metric (churn, coupling, file size) and that development velocity has been accelerating, with Q1 2025 as the peak quarter (746 commits).

This isn't a hypothetical use case. This is a real agent, analyzing a real repository, producing real insights, with a 4-sentence prompt. And with pgit analyze, an agent doesn't even need to write SQL for the common cases. pgit analyze churn --json and pgit analyze coupling --json give it structured data directly. SQL is there when the agent needs to go deeper, but the built-in analyses lower the floor even further.

The combination of pgit's command-line interface, SQL escape hatch, and an agent's ability to reason over structured data makes codebase analysis something you can just ask for.

What's Next

I'm happy with where pgit is. The compression holds up against git, the SQL interface works, and it's useful for real analysis, from manual queries to fully autonomous agent workflows. It does what I set out to make it do.

If you run into bugs or have a compelling feature idea, issues and PRs are welcome. The underlying pg-xpatch extension is the piece I'm most excited about long-term. It works for any versioned data (document editors, audit logs, config snapshots, CMS content history), and pgit is just one application of what a delta-compressed storage engine can do.

If you want to try pgit:

go install github.com/imgajeed76/pgit/v4/cmd/pgit@latest

pgit: github.com/ImGajeed76/pgit
pg-xpatch: github.com/ImGajeed76/pg-xpatch
xpatch: github.com/ImGajeed76/xpatch
Full benchmark results: BENCHMARK.md

I Got Obsessed With 2-Byte Deltas and Built a Delta Compressor That Actually Hits Them

Oliver Seifert — Thu, 11 Dec 2025 22:43:38 +0000

TL;DR: Needed tiny deltas for an encrypted document format. xdelta3 uses 40+ bytes for simple edits. Built xpatch with 7 specialized algorithms and a tag system that references old versions. Result: median delta of 2 bytes on real code repositories. Yes, 2 bytes. Tested on 1.2 million real-world changes. Available for Rust, Python, Node.js, C/C++, and WebAssembly. It's fast as hell and actually works.

The median delta is 2 bytes.

Not a typo. Two bytes. That's a uint16. That's the header plus literally nothing else because the files being compared are identical and the algorithm just says "hey, go look at version 5, it's the same."

Let me show you how I ended up here, because it started with a much simpler problem and turned into a three-week rabbit hole that I'm genuinely excited about.

The Problem That Broke Me

I'm building a document format. The requirements list reads like a wish list from hell: E2EE, realtime collaboration, full version history, offline-first sync, and it all needs to actually work without requiring a dedicated server farm.

Here's the catch that makes everything complicated: you can't compress after encryption. Once you encrypt a delta, that's it. No more compression opportunities. Which means every single delta needs to be tiny, or storage costs will murder you before you even launch.

I started with Myers diff. You know, the classic. It's elegant, it's well-tested, it works. But for the kinds of micro-edits you get when capturing changes every 500ms? Way too verbose. Storing "user added 5 characters" was taking dozens of bytes.

I moved to xdelta3. Better. Actually pretty good for larger changes. But still burning 40+ bytes just to encode "hello" → "hello world". That's 8x overhead on a 6-byte change. Not going to work.

Then I found gdelta. There's this research paper from 2021 called "Exploring the Potential of Fast Delta Encoding" and it looked promising. Fast, good compression on small files, exactly what I needed. Problem: the only implementation was three-year-old research code with Python bindings that barely compiled.

So I did what any reasonable developer would do at 2 AM: said "fuck it, I'll rewrite it in Rust." Two weeks later, https://github.com/ImGajeed76/gdelta exists and actually works. It's fast, it's efficient, the algorithm is solid.

But I kept staring at the output. Simple edits were still around 20 bytes. And I couldn't stop thinking: "Why? Why do I need 20 bytes to store the fact that someone typed 'hello'?"

That question wouldn't leave me alone.

The Realization That Changed Everything

Most document edits aren't random. They follow patterns. Really predictable patterns:

Adding text: "hello" becomes "hello world"
Deleting text: "hello world" becomes "hello"
Reverting changes: "hello" → "hello world" → "hello" (classic "actually no" moment)
Repeating patterns: "test test test", indentation, decorators
Everything else: the actual complex structural changes

General-purpose algorithms like xdelta3 or gdelta try to handle all of these with one approach. They're built for the worst case - arbitrary binary changes, no assumptions about the data.

But what if you don't have to make one algorithm that handles everything? What if you could have specialized algorithms for each pattern, try them all, and just pick whichever produces the smallest result?

That's when I started building xpatch.

Building It: The Simple Stuff First

Algorithm 1: Just Store The Damn Characters

The most common case is adding text. So I just... store it.

fn encode_add(position: usize, data: &[u8]) -> Vec<u8> {
    let mut encoded = encode_varint(position);
    encoded.extend_from_slice(data);
    encoded
}

That's it. Position where the insertion happens (variable-length integer so small positions use less space), then the actual characters. Adding "world" at position 6? About 6 bytes total.

Compare to gdelta's 20+ bytes for the same change. Already winning.

Algorithm 2: Deletion Is Even Simpler

fn encode_remove(start: usize, end: usize) -> Vec<u8> {
    let mut encoded = encode_varint(start);
    encoded.extend(encode_varint(end - start));
    encoded
}

Two numbers: where it starts, how much to remove. Fixed overhead of about 3 bytes for typical positions. You literally cannot make deletion more compact than this without getting into bitpacking nonsense.

Algorithm 3: Wait, What If We Tokenize This?

Here's where I went a little crazy.

Text isn't random characters. It's words. Common words appear all the time. What if instead of storing "hello world" as 11 bytes, we could store it as two token references?

So I built a tokenizer. The wordlist is exactly 2^14 tokens (16,384 words) because that's the sweet spot for varint encoding - small token IDs fit in one byte, larger ones in two. Sorted by frequency so common words get the smallest IDs.

The data structure is a trie with array-indexed children:

struct TrieNode {
    token_id: Option<usize>,
    children: [Option<Box<TrieNode>>; 256],
}

fn find_longest_match(&self, text: &[u8], start: usize) -> Option<(usize, usize)> {
    let mut node = &self.root;
    let mut last_match = None;
    let mut pos = start;

    while pos < text.len() {
        // Direct array indexing - O(1) lookup per byte
        if let Some(next_node) = &node.children[text[pos] as usize] {
            node = next_node;
            pos += 1;

            if let Some(token_id) = node.token_id {
                last_match = Some((token_id, pos - start));
            }
        } else {
            break;
        }
    }

    last_match
}

No HashMap overhead, just direct array access. O(1) per byte. Greedy longest-match means it always picks the biggest token it can find.

The best part? If tokenization produces a larger result than just storing the characters raw, it automatically falls back to the Chars algorithm. No tradeoff. You always get the smaller encoding.

Algorithm 4 & 5: Pattern Detection

Spotted "aaaaaaa" in the text? Don't store 7 bytes. So I added pattern detection that stores this instead:

// [position][repeat_count][pattern]
// "aaaaaaa" = [alog + tag][7]["a"] = 3 bytes

Works for single characters and for token patterns. If someone writes "test test test test", you don't need to store "test" four times. Store it once and say "repeat this 4 times."

I optimized pattern detection too. It checks powers of 2 first (1, 2, 4, 8, 16, 32, 64) because they're fastest to verify with bitwise operations and they're super common in code.

Algorithm 6 & 7: The "Fuck It, Use The Real Algorithm" Option

For complex changes - scattered edits, structural rewrites, cases where nothing else works - I fall back to gdelta. It's a proper general-purpose delta compressor, and it's good at its job.

There's also a GDeltaZstd variant that adds zstd compression on top of gdelta. Slower to encode, but sometimes you get significant size wins. Worth trying.

The Selection Logic

So how does it actually pick? Here's the code:

let (best_algo, best_data) = match analyze_change(base_data, new_data) {
    ChangeType::ContinuousAdd { position, data } => {
        // Start with simple character encoding
        let mut best_algo = Algorithm::Chars;
        let mut best_data = encode_add(position, &data[..]);

        // Try tokenization - might be smaller
        if let Ok(token_data) = encode_tokens(position, &data[..])
            && token_data.len() < best_data.len()
        {
            best_algo = Algorithm::Tokens;
            best_data = token_data;
        }

        // Check for repeating patterns
        if let Some((pattern, repeat_count)) = detect_repeating_pattern(&data[..])
            && repeat_count >= 2
        {
            // Try both character and token-based repetition
            // Pick whichever is smaller
        }

        (best_algo, best_data)
    }
    ChangeType::ContinuousRemove { start, end } => {
        (Algorithm::Remove, encode_remove(start, end))
    }
    ChangeType::Complex => {
        // Try gdelta with and without zstd, pick the winner
    }
};

Try all the algorithms that make sense for the change type, pick whichever produces the smallest output. Simple strategy, but it works beautifully.

The SIMD Detour (Because Why Not)

To figure out which algorithm to use, you first need to analyze what actually changed. Did someone add text in the middle? Delete a section? Completely rewrite it?

The key operation is finding the common prefix - how much of the beginning stayed the same.

fn analyze_change(old: &[u8], new: &[u8]) -> ChangeType {
    if new.len() > old.len() {
        // Might be an addition - find where they diverge
        let position = find_common_prefix(old, new);

        // Check if everything after the addition matches
        // If it does, we have a continuous insertion
        let added_len = new.len() - old.len();
        let expected_end = position + added_len;

        if expected_end <= new.len() {
            let old_suffix = &old[position..];
            let new_suffix = &new[expected_end..];

            if old_suffix == new_suffix {
                return ChangeType::ContinuousAdd {
                    position,
                    data: new[position..expected_end].to_vec(),
                };
            }
        }
    }

    // Similar logic for deletions...

    // Otherwise its complex
    ChangeType::Complex
}

So obviously I added AVX2 SIMD support:

#[cfg(target_arch = "x86_64")]
#[target_feature(enable = "avx2")]
unsafe fn find_common_prefix_avx2(a: &[u8], b: &[u8]) -> usize {
    use std::arch::x86_64::*;

    let len = a.len().min(b.len());
    let mut i = 0;

    // Process 32 bytes at a time with AVX2
    while i + 32 <= len {
        let a_vec = _mm256_loadu_si256(a.as_ptr().add(i) as *const __m256i);
        let b_vec = _mm256_loadu_si256(b.as_ptr().add(i) as *const __m256i);
        let cmp = _mm256_cmpeq_epi8(a_vec, b_vec);
        let mask = _mm256_movemask_epi8(cmp);

        if mask != -1 {
            return i + mask.trailing_ones() as usize;
        }
        i += 32;
    }

    // Handle remaining bytes
    while i < len && a[i] == b[i] {
        i += 1;
    }

    i
}

Does it make a difference? Yeah, actually. Comparing 32 bytes at once instead of one by one adds up when you're processing millions of deltas.

Is it overkill for most use cases? Probably. Did I do it anyway? Absolutely.

The Tag System (This Is Where It Gets Wild)

Okay, so at this point xpatch is already pretty good. Specialized algorithms, automatic selection, SIMD optimization. It beats xdelta3 and gdelta on small edits.

But then I had another realization.

Watch this editing pattern:

Version 1: "hello"
Version 2: "hello world" (added text)
Version 3: "hello" (wait no, changed my mind, delete it)

Sequential diffing goes:

v1 → v2: 9 bytes (add " world")
v2 → v3: 3 bytes (remove 6 characters)
Total: 12 bytes

But here's the thing: v3 is identical to v1. Like, exactly the same. Why are we encoding a diff at all? Why not just say "hey, this is the same as version 1"?

That's what the tag system does.

How Tags Work

Every delta starts with a compact header:

fn encode_header(algo_type: Algorithm, tag: usize) -> Vec<u8> {
    let algo_type = algo_type as u8;

    if tag < 16 {
        // Fits in 4 bits alongside the algorithm - single byte total
        vec![(algo_type << 5) | (tag as u8)]
    } else {
        // Need more space - use variable-length encoding
        let first_bits = (tag & 0x0F) as u8;
        let mut bytes = Vec::new();
        bytes.push((algo_type << 5) | 0x10 | first_bits);

        // Encode the rest as a varint
        let mut remaining = tag >> 4;
        loop {
            let mut byte = (remaining & 0x7F) as u8;
            remaining >>= 7;
            if remaining != 0 {
                byte |= 0x80;  // Continuation bit
            }
            bytes.push(byte);
            if remaining == 0 { break; }
        }

        bytes
    }
}

The header is [3 bits: algorithm][1 bit: flag][4 bits: tag or varint].

Tags 0-15 are free - they fit in the first byte alongside the algorithm identifier. The tag tells you which previous version was used as the base for this delta.

The Magic Moment

Back to our editing example. Encoding v3 using v1 as the base (tag = 1):

Comparing v3 to v1:
- They're identical
- ChangeType: ContinuousAdd with position 0 and zero bytes
- Header: [Chars algorithm + tag 1] = 1 byte
- Data: [position varint: 0] = 1 byte
- Total: 2 bytes

That's it. Two bytes to represent "this version is the same as version 1." You see it uses tag 1, you grab version 1 as the base, apply the (empty) delta, and get the correct result.

This is why the median delta hits 2 bytes. When files revert to previous states - which happens way more often than you'd think - the delta is basically free.

The Implementation Is Beautifully Dumb

When implementing this, you don't need to be smart about which version to reference. You just brute-forces it:

let mut best_delta = encode(0, previous_version, current_version, enable_zstd);

for tag in 1..MAX_TAG_DEPTH {
    if tag >= version_history.len() {
        break;
    }

    let base_version = version_history[current - tag];
    let candidate_delta = encode(tag, base_version, current_version, enable_zstd);

    if candidate_delta.len() < best_delta.len() {
        best_delta = candidate_delta;
    }
}

Try encoding against the last N versions (default: 16), pick whichever produces the smallest delta. That's the whole algorithm.

Checking 16 versions sounds expensive, but remember - encoding is microseconds. Even checking 16 times is still sub-millisecond. For document editing where you're capturing changes every 500ms, you could check the last 1000 versions and users wouldn't even notice.

The Benchmarks (Where I Got Really Excited)

Alright, time for the numbers. And these aren't synthetic benchmarks or cherry-picked examples - this is real-world data from actual git repositories with all their messy, real-world chaos.

Test corpus:

tokio (Rust async runtime): 1,805 files, 133,728 deltas
mdn/content (MDN Web Docs): 28,914 files, 1,225,740 deltas
Total: 30,719 files, 1.2 million deltas

Every single commit, every single file change. If it happened in the git history, it's in the benchmark.

My hardware: AMD Ryzen 7 7800X3D (16 threads), 64GB RAM, Fedora Linux. Nothing fancy, just a normal development machine.

tokio Results (Code Repository)

Algorithm	Median Delta	Compression Ratio	Space Saved	Median Encode	Median Decode
xpatch_tags	2 bytes	0.0019	99.8%	208 µs	0 µs
xpatch_sequential	68 bytes	0.0165	98.4%	14 µs	0 µs
vcdiff (xdelta3)	97 bytes	0.0276	97.2%	15 µs	3 µs
gdelta	69 bytes	0.0180	98.2%	1 µs	0 µs

Tag optimization impact: 88.7% smaller deltas (median) compared to sequential mode.

Two. Fucking. Bytes. On a real code repository.

mdn/content Results (Documentation Repository)

Algorithm	Median Delta	Compression Ratio	Space Saved	Median Encode	Median Decode
xpatch_tags	23 bytes	0.0063	99.4%	104 µs	0 µs
xpatch_sequential	25 bytes	0.0069	99.3%	10 µs	0 µs
vcdiff (xdelta3)	50 bytes	0.0169	98.3%	9 µs	2 µs
gdelta	26 bytes	0.0077	99.2%	0 µs	0 µs

Tag optimization impact: 8.8% smaller deltas (median) compared to sequential mode.

Still less than half the size of vcdiff, but the tag advantage is way smaller here.

The Surprise That Made Me Think

I fully expected documentation to benefit more from tags. Documentation is iterative, right? You draft something, revise it, hate it, revert it, try again. Code is supposedly more linear - you write it, it works (or doesn't), you move on.

Turns out I was completely wrong.

Code benefits 10x more from tags than documentation does.

Looking deeper at the tag statistics:

tokio (code): average tag 1.9, median 2 - jumping back ~2 commits on average
mdn/content (docs): average tag 1.1, median 1 - mostly sequential

Code has way more rollbacks. Maybe it's test-driven development where you write tests, break things, revert. Maybe it's experimental features that get backed out. Maybe it's refactoring attempts that get abandoned. Whatever it is, code repositories have tons of "actually let's go back to how it was" moments.

I sure as hell didn't expect that.

The maximum beneficial jump I saw was around 10 commits back, though I only tested with a max depth of 16. For document editing where you're saving every 500ms, you could probably search back hundreds or thousands of versions and find even better matches.

The Speed Numbers

For the mdn/content benchmark specifically, here's something wild:

Extracting versions from git: 5 hours 26 minutes
Running all benchmarks: 2 minutes 42 seconds (4 algorithms, serial processing)

Think about that. xpatch benched 1.2 million deltas in less time than it takes git to extract the files to disk.

The median encode times are 104-208 µs when trying 16 different encodings (tag mode). Sequential mode with just one encoding is 10-14 µs median. Decode is sub-microsecond regardless of algorithm.

These aren't "fast enough" numbers. This is "you'll never notice" performance.

Where xpatch Loses

I'm not going to pretend this is perfect for everything. There are cases where xpatch doesn't win:

Worst case: Large structural rewrites, complete file replacements, binary data with no patterns.

For those scenarios, xpatch falls back to gdelta mode and performs roughly the same as vcdiff. Sometimes slightly better, sometimes slightly worse, generally competitive.

But that's fine. xpatch wasn't built for those cases. It was built for micro-edits and high-frequency version tracking.

For that use case, it dominates.

Try It Yourself

As a library:

cargo add xpatch

use xpatch::delta;

let base = b"Hello, world!";
let new = b"Hello, beautiful world!";

// Encode
let delta = delta::encode(0, base, new, false);

// Decode
let restored = delta::decode(base, &delta).unwrap();
assert_eq!(restored, new);

Or use the CLI:

# Install
cargo install xpatch --features cli

# Create and apply deltas
xpatch encode old.txt new.txt -o patch.xdelta
xpatch decode old.txt patch.xdelta -o restored.txt

Language Bindings:

xpatch is also available for Python (pip install xpatch-rs), Node.js (npm install xpatch-rs), C/C++, and WebAssembly. Check the repository for installation and usage examples.

Links:

Repository: https://github.com/ImGajeed76/xpatch
Demo Editor: https://github.com/imgajeed76/xpatch_demo_editor
Documentation: https://docs.rs/xpatch

Was The Rabbit Hole Worth It?

Definitely.

I started three weeks ago with a simple frustration: "20 bytes is too much for adding 5 characters."

I built 7 specialized algorithms. Implemented SIMD optimizations. Created a custom tokenizer with 16,384 words. Designed a tag system that makes version history smaller instead of larger.

I tested it on 1.2 million real-world changes across 30,000 files.

Result: median delta of 2 bytes.

It works. It's fast. It's been tested to hell and back on real data.

And honestly? I'm kind of proud of this one.

The thing about going down rabbit holes is you never know what you'll find. Sometimes you waste a week on something that doesn't work. Sometimes you build something that solves a problem you didn't even know existed.

This was the second kind.

Star the repo if you think this problem was worth solving. Or if you just like seeing "median: 2 bytes" in benchmark tables.

I know I do.

Oh wow, you really read through all of this! Thanks! Here's a cookie 🍪

Holy Shit, This Escalated Quickly - Axogen v0.5.0 Is Here And I Can't Even

Oliver Seifert — Wed, 30 Jul 2025 14:19:27 +0000

Originally I wanted to write this article at the end of the week, but it just happened too much in the meantime.

Since my last article about 1.5 weeks went past. I was on holidays for 5 days so not much progress then, but in the other time we got this shit that made me realize: maybe I actually built something people want?

TL;DR: My .env rant got featured in DEV's Top 7, I got so hyped I locked
myself in my house for 4 days straight and shipped secret detection, 10+ file
formats, nested commands, backup systems, and proper type safety. We're at v0.5.0 now and honestly I can't believe this is working.

Wait, Dev.to Featured Me in Their Top 7?!

So first things first - I GOT PROMOTED ON THE DEV.TO WEEKLY TOP 7. Like how freaking crazy is that!!!!!

dev.to staff for The DEV Team

Jul 28 '25

The Type Safety Revolution (Finally!)

Remember how the old API was kinda... loose? Yeah, I fixed that.

The new recommended way uses dedicated functions for each target type, and TypeScript now actually knows what you're doing:

import {defineConfig, env, json} from "@axonotes/axogen";

export default defineConfig({
    targets: {
        someService: json({
            path: "output/someService.json",
            variables: {
                name: "MyService",
                version: "1.3.0",
            },
        }),
        myEnvironment: env({
            path: "output/.env",
            variables: {
                PRODUCTION: "true",
                API_KEY: "12345",
                DATABASE_URL: "postgres://user:pass@localhost:5432/mydb",
            },
        }),
    },
});

You still can use the old structure, but now you get proper IntelliSense for each target type.

Secret Detection That Prevents Disasters

Here's something that kept me up at night: what if someone accidentally pushes their production API keys because Axogen generated them into a non-gitignored file?

So I built secret detection. Before generation of each target, all variables are scanned. If they look like secrets (API keys, passwords, tokens, etc.) and the target file is NOT gitignored, Axogen refuses to generate.

Also, did you notice 😏, this time I used a screenshot to show off the colored
output? I know, I know, I'm a rebel.

But sometimes you actually want to generate "secrets" - like development database URLs or test API keys. Just wrap them with unsafe():

import {defineConfig, env, unsafe} from "@axonotes/axogen";

export default defineConfig({
    targets: {
        myEnvironment: env({
            path: "output/.env",
            variables: {
                PRODUCTION: false,
                API_KEY: unsafe("your-dev-api-key-here", "Development mode"),
            },
        }),
    },
});

The second parameter is required - you have to explicitly say WHY this is safe. No more "oops, I pushed the prod keys" moments.

Zod Schema Validation (Because Type Safety Everywhere)

Remember how I said this was powered by Zod but you couldn't really use Zod's full power? Fixed that too.

Now you can validate your target configurations before generating, and TypeScript will complain if your variables don't match your schema:

import {defineConfig, json} from "@axonotes/axogen";
import * as z from "zod";

export default defineConfig({
    targets: {
        someService: json({
            path: "output/someService.json",
            schema: z.object({
                name: z.string().describe("The name of the service"),
                version: z.string().describe("The version of the service"),
            }),
            variables: {
                name: "MyService",
                version: 2, // TSC will error: Type 'number' is not assignable to type 'string'
            },
        }),
    },
});

Your editor lights up with red squiggles before you even try to generate. And if you somehow ignore TypeScript (why would you do that?), Zod catches it at runtime with beautiful error messages.

Backup System (I have trust-issue okay?)

You know that moment when you run a generation command and accidentally overwrite something important? Yeah, me too.

Axogen can now create backups of your targets before overwriting them:

import {defineConfig, json} from "@axonotes/axogen";

export default defineConfig({
    targets: {
        someService: json({
            path: "output/someService.json",
            variables: {
                name: "MyService",
                version: "1.3.0",
            },
            backup: true,
            // backupPath: "your/own/backup/path.json", // Optional
        }),
    },
});

Default backup location is .axogen/backup/{{path}}. Currently keeps one backup - though I'm thinking about adding more sophisticated backup options with retention policies.

Conditional Generation

Sometimes you only want to generate certain configs under certain conditions. Now you can:

export default defineConfig({
    targets: {
        productionSecrets: env({
            path: "secrets/.env.prod",
            variables: {
                REAL_API_KEY: "super-secret-key",
                PROD_DATABASE_URL: "postgres://prod-server/db",
            },
            condition: process.env.NODE_ENV === "production",
        }),
    },
});

The condition is just a boolean - use it for whatever logic you want. Environment checks, feature flags, time-based generation, whatever makes sense for your use case.

Command System Overhaul (Nested Commands Are Here)

The old command system was... functional. The new one is beautiful.

You can now create as many nested command groups as you want, and they're all auto-registered with full type safety. There are actually 5 different ways to define commands:

import {cmd, defineConfig, group, liveExec} from "@axonotes/axogen";
import * as z from "zod";

export default defineConfig({
    commands: {
        hello: "echo 'Hello, world!'", // Simple string
        build: async (context) => {
            // Direct function
            console.log("Building...");
            await liveExec("bun run build");
        },
        dev: cmd({
            help: "Give a string command some help text",
            command: "echo 'This is a dev command'",
        }),
        echo: cmd({
            help: "A command that echoes the input",
            args: {
                input: z.string().describe("The input to echo"),
            },
            exec: (context) => {
                console.log(context.args.input);
            },
        }),
        database: group({
            help: "Database management commands",
            commands: {
                migrate: cmd({
                    help: "Run database migrations",
                    exec: (context) => console.log("Running migrations..."),
                }),
                seed: cmd({
                    help: "Seed the database with test data",
                    exec: (context) => console.log("Seeding database..."),
                }),
                backup: group({
                    help: "Database backup operations",
                    commands: {
                        create: cmd({
                            help: "Create a database backup",
                            options: {
                                name: z.string().describe("Backup name"),
                            },
                            exec: (ctx) => {
                                console.log(
                                    `Creating backup: ${ctx.options.name}`
                                );
                            },
                        }),
                    },
                }),
            },
        }),
    },
});

Now axogen run database backup create --name "before-migration" just works. The help system is automatically generated. IntelliSense knows about all your args and options. It's like having a professional CLI framework built into your config tool.

Environment Loading Revolution

The loadEnv function got a complete makeover. You can now configure the loading process and select different files:

import {loadEnv} from "@axonotes/axogen";
import * as z from "zod";

const env = loadEnv(
    z.object({
        NODE_ENV: z.enum(["development", "production"]).default("development"),
        PORT: z.coerce.number().default(3000),
        DATABASE_URL: z.url().describe("The URL of the database"),
    }),
    {
        path: ".env.custom",
        // Any dotenvx options work here
        verbose: true,
        override: true,
    }
);

Full dotenvx compatibility means you get all the advanced environment loading features, but with Zod validation on top.

Universal File Loading

But wait, there's more! Ever wanted to load TOML, YAML, or other config formats with the same type safety?

import {loadFile} from "@axonotes/axogen";
import * as z from "zod";

const config = loadFile(
    "myFile.toml",
    "toml",
    z.object({
        key1: z.string().describe("Description for key1"),
        key2: z.number().describe("Description for key2"),
        key3: z.boolean().describe("Description for key3"),
    })
);

Works with basically any config format you throw at it. Same Zod validation, same type safety, same beautiful error messages.

File Format Explosion

Speaking of formats, I went a bit crazy with the supported file types:

For loading:

json, json5, jsonc, hjson
yaml
toml
ini, properties
env
xml, csv, cson

For generating:

All the loading formats, plus...
template (nunjucks, handlebars, mustache)

Performance & Code Quality (The Boring But Important Stuff)

Version 0.5.0 isn't just about features. I spent a lot of time on:

Better error handling: Clearer messages, better stack traces
Performance improvements: Mainly TSC cleanup - wasn't slow before, just a tiny bit faster now
Code structure: Much cleaner internals, easier to contribute to
Safety improvements: More validation, fewer edge cases

The kind of stuff you don't notice until you don't have to think about it.

Join the Discord (Why Not?)

Oh, and I haven't mentioned this before, but we already had a Discord server for Axonotes:

Join the Axonotes Discord

Come ask questions, complain about bugs, or just chat about config management. People can join and ask Axogen questions there too if they want.

What's Next?

We're at v0.5.0 now, and honestly? I'm just getting started. Some big things on the roadmap:

Project initialization: npx @axonotes/axogen init command
Secrets management: Doppler, Vault, AWS Secrets Manager integration
Runtime loading: Import your generated configs directly into your app

But right now, I'm just excited that this thing I built in frustration is helping some people. The few GitHub stars (about 5 now), the Discord conversations - it's all reminding me why I love building tools.

With that in mind, Axogen is still not production ready. But I'm working on it! I still have 1.5 weeks of spare time to spend lol.

Try The New Hotness

If you tried the old version and bounced off, give v0.5.0 a shot. If you haven't tried it yet, now's a great time:

npm install @axonotes/axogen@latest

# Check out the updated docs for examples
# (not updated yet XD, maybe just use this article as a guide):
# https://axonotes.github.io/axogen/

The API is much more stable now, the type safety is useful, and the features solve real problems instead of just being neat ideas.

Configuration management doesn't have to suck. Your environment variables can have types. Your scripts can be intelligent. Your deployments can be consistent.

Still feels surreal, honestly.

Don't forget to leave a star on GitHub if you like it! ⭐

Built with ❤️, excessive caffeine, and the validation of internet strangers by Oliver Seifert. Now at v0.5.0 - still proceed with reasonable caution, but maybe a little less caution than before.

I Built a TypeScript-Native Config System Because .env Files Drive Me Crazy

Oliver Seifert — Mon, 21 Jul 2025 19:20:36 +0000

You know that moment when you change a port number and suddenly half your services can't talk to each other? You spend 20 minutes debugging only to realize you forgot to update the WebSocket URL in three different places. Again.

Yeah, I was tired of that too.

I spent three days building something about it. Now I'm at v0.3.0 and already seeing so much potential that I had to share.

The Problem That Actually Broke Me

I was working on AxonotesCore when I hit my breaking point. Here's what my actual project looks like:

AxonotesCore/
├── dashboard/               # SvelteKit frontend (port 5173)
│   └── .env                 # Most config here
├── server/                  # Rust SpacetimeDB module  
├── SpacetimeDB/             # SpacetimeDB CLI (port 3000)
├── scripts/                 # Key generation utilities
└── package.json             # Many hardcoded scripts

My dashboard/.env file is a beautiful disaster of hardcoded URLs and ports:

WORKOS_REDIRECT_URI="http://localhost:5173/auth/callback"
PUBLIC_SPACETIME_WS="ws://localhost:3000"
# Plus a dozen other auth and JWT variables that look like keyboard mashing...

But the real nightmare lives in package.json. Look at this beauty:

{
  "scripts": {
    "presrv:dev": "bun run sdb:build:cli",
    "srv:publish": "./bin/spacetimedb-cli publish --project-path server -c axonotes",
    "comment:server:dev": "ADD --auth-required to the dev command in PRODUCTION!",
    "dash:dev": "cd dashboard && bun run dev",
    "scr:setup:keys": "cd scripts && bun run generate-keys.ts"
  }
}

See that hardcoded http://localhost:5173 buried in the middle of that command? Me neither, because it's actually way further down:

{
  "name": "axonotes-core",
  "version": "0.0.1",
  "private": true,
  "type": "module",
  "scripts": {
    "prepare": "husky",
    "setup": "bun run scr:setup:keys && bun run dash:install",
    "format": "prettier --write . && bun run srv:format && bun run sdb:format",
    "format:check": "prettier --check .",
    "lint": "eslint .",
    "lint:fix": "eslint . --fix",
    "sdb:comment": "--- SpacetimeDB Subtree Management ---",
    "sdb:pull-upstream": "git subtree pull --prefix=SpacetimeDB https://github.com/clockworklabs/SpacetimeDB.git master --squash",
    "sdb:push-fork": "git subtree push --prefix=SpacetimeDB https://github.com/axonotes/SpacetimeDB.git master",
    "sdb:pull-fork": "git subtree pull --prefix=SpacetimeDB https://github.com/axonotes/SpacetimeDB.git master --squash",
    "sdb:build:cli": "mkdir -p bin && cd SpacetimeDB && cargo build --release -p spacetimedb-cli -p spacetimedb-standalone && cp target/release/spacetimedb-cli ../bin/ && cp target/release/spacetimedb-standalone ../bin/",
    "presdb:login": "bun run sdb:build:cli",
    "sdb:login": "./bin/spacetimedb-cli login",
    "presdb:logout": "bun run sdb:build:cli",
    "sdb:logout": "./bin/spacetimedb-cli logout",
    "sdb:format": "cd SpacetimeDB && cargo fmt --all",
    "comment:server": "--- SpacetimeDB Server ---",
    "presrv:dev": "bun run sdb:build:cli",
    "srv:publish": "./bin/spacetimedb-cli publish --project-path server -c axonotes && ./bin/spacetimedb-cli generate --lang typescript --out-dir dashboard/src/lib/module_bindings --project-path server",
    "comment:server:dev": "ADD --auth-required to the dev command in PRODUCTION!",
    "srv:dev": "./bin/spacetimedb-cli start --in-memory --allowed-oidc-issuer http://localhost:5173 --allowed-oidc-issuer https://auth.spacetimedb.com",
    "srv:format": "cd server && cargo fmt --all",
    "presrv:clear": "bun run sdb:build:cli",
    "srv:clear": "./bin/spacetimedb-cli server clear",
    "srv:logs": "./bin/spacetimedb-cli logs axonotes",
    "comment:dashboard": "--- SvelteKit Dashboard ---",
    "dash:install": "cd dashboard && bun install",
    "dash:dev": "cd dashboard && bun run dev",
    "dash:build": "cd dashboard && bun run build",
    "dash:preview": "cd dashboard && bun run preview",
    "comment:scripts": "--- Utility Scripts ---",
    "prescr:setup:keys": "cd scripts && bun install",
    "scr:setup:keys": "cd scripts && bun run generate-keys.ts"
  }
}

There it is! Hiding in the srv:dev command like a bug in production code. Every time I change the dashboard port, I have to hunt through scripts, update the WORKOS callback URL, and remember which comments are actual warnings versus just... despair.

We needed a separate markdown file explaining how to run anything. New developers would look at this and just give up.

I thought "there's gotta be a tool for this, right?"

Turns Out, Not Really

Here's the thing: we're stuck in this weird middle ground. dotenv (58 million weekly downloads) gives you basically nothing - no validation, everything's a string, zero sync capabilities. Enterprise stuff like HashiCorp Vault requires a dedicated DevOps team and costs more than my coffee budget.

Everything else makes you learn new languages (looking at you, Jsonnet) or locks you into cloud providers.

I needed something that felt like TypeScript, worked with normal web development, and actually solved the URL synchronization nightmare. So I built it.

Meet Axogen: Starting Simple, Getting Powerful

The core idea is dead simple: define your configuration once in TypeScript, generate multiple formats automatically. But here's the kicker - it's powered by Zod under the hood, which means you get real schema validation, not just "hope this string is actually a number."

Let me show you:

// axogen.config.ts
import {z, defineConfig, createTypedEnv} from "@axonotes/axogen";

const env = createTypedEnv({
    SPACETIME_PORT: z.coerce.number().default(3000),
    DASHBOARD_PORT: z.coerce.number().default(5173),
    WORKOS_CLIENT_ID: z.string().min(1, "WorkOS Client ID is required"),
    DATABASE_URL: z.url("Must be a valid database URL"),
    JWT_PRIVATE_KEY_BASE64: z.string().base64("Invalid base64 encoding"),
    NODE_ENV: z.enum(["development", "staging", "production"]).default("development"),
});

export default defineConfig({
    targets: {
        dashboard: {
            path: "dashboard/.env",
            type: "env",
            variables: {
                WORKOS_CLIENT_ID: env.WORKOS_CLIENT_ID,
                WORKOS_REDIRECT_URI: `http://localhost:${env.DASHBOARD_PORT}/auth/callback`,
                PUBLIC_SPACETIME_WS: `ws://localhost:${env.SPACETIME_PORT}`,
                DATABASE_URL: env.DATABASE_URL,
                JWT_PRIVATE_KEY_BASE64: env.JWT_PRIVATE_KEY_BASE64,
            },
        },
    },
});

Put your actual values in .env.axogen:

PACETIME_PORT=3000
DASHBOARD_PORT=5173
WORKOS_CLIENT_ID=your_client_id
DATABASE_URL=postgresql://localhost:5432/axonotes
JWT_PRIVATE_KEY_BASE64=eW91cl9wcml2YXRlX2tleQ==
NODE_ENV=development

Run axogen generate and it creates dashboard/.env with all the URLs automatically calculated. Change DASHBOARD_PORT to 3000, regenerate, and the auth callback becomes http://localhost:3000/auth/callback automatically.

One source of truth. Everything else just follows.

And Yeah, It's Actually Fast

I haven't even optimized this yet, but somehow it's just... fast. Like, really fast:

✅ Stress test completed in 2191ms
📊 Generated 10000 configuration files
📈 Average time per target: 0.22ms

That's 10,000 config files in 2.2 seconds. For comparison, most tools take longer than that just to parse your package.json.

It turns out when you're basically just validating once and converting JSON to different formats, things can be pretty snappy. Who knew? This means you can run axogen generate constantly during development without thinking about it - change a port, regenerate, keep coding.

Zod-Powered Type Safety

This isn't just string replacement. Remember, we're using Zod schemas for validation. Try this:

# In .env.axogen
DATABASE_URL=not-a-url
JWT_PRIVATE_KEY_BASE64=definitely_not_base64!
SPACETIME_PORT=not_a_number

Run axogen generate and watch it fail fast with beautiful, colored output (well, here its just black/white):

❌ Environment variable validation failed

  Missing Required:
    • SPACETIME_PORT

  Validation Errors:
    • DATABASE_URL: Must be a valid database URL
    • JWT_PRIVATE_KEY_BASE64: Invalid base64 encoding

ℹ️  Check your .env.axogen file and ensure all required variables are set with correct values.

No more silent failures. No more wondering if that PORT should be 3000 or "3000". No more invalid URLs making it to production. The config gets validated using Zod's full power before anything gets generated.

Want custom validation? Just use any Zod method:

const env = createTypedEnv({
    API_RATE_LIMIT: z.coerce.number().min(1).max(1000),
    WEBHOOK_SECRET: z.string().min(32, "Webhook secret must be at least 32 characters"),
    ALLOWED_ORIGINS: z.string().transform(s => s.split(',')).pipe(z.array(z.url())),
});

It's like having TypeScript for your environment variables. Well, it literally is TypeScript for your environment variables.

Fixing the Scripts Nightmare (Now With Style)

Here's where Axogen v0.3.0 gets really powerful. Remember that package.json disaster? You can start simple:

export default defineConfig({
    // ... targets
    commands: {
        "dev:spacetime": `./bin/spacetimedb-cli start --in-memory --allowed-oidc-issuer http://localhost:${env.DASHBOARD_PORT}`,
        "dev:dashboard": `cd dashboard && bun run dev --port ${env.DASHBOARD_PORT}`,
    },
});

That's it. Just strings that reference your config variables. But then you realize - wait, I want help messages! And maybe some custom logic. So you can upgrade to the full power:

export default defineConfig({
    commands: {
        "dev:spacetime": command.define({
            help: "Start SpacetimeDB server in development mode",
            exec: async () => {
                console.log("🚀 Starting SpacetimeDB server...");
                const result = await liveExec(
                    `./bin/spacetimedb-cli start --in-memory --allowed-oidc-issuer http://localhost:${env.DASHBOARD_PORT}`,
                    { outputPrefix: "SDB" }
                );
            },
        }),

        "dev:all": command.string(
            "concurrently 'axogen run dev:spacetime' 'axogen run dev:dashboard'",
            "Start all development servers"
        ),
    },
});

Now when you run axogen run --help, you get a proper command list with descriptions. And axogen run dev:spacetime --help shows you exactly what it does. The commands automatically use the right ports, and the output actually shows you what's happening:

❯ axogen run dev:spacetime
✅ Environment variables validated successfully
🚀 Starting SpacetimeDB server...
🚀 Running: ./bin/spacetimedb-cli start --in-memory --allowed-oidc-issuer http://localhost:5173

[SDB] spacetimedb-standalone version: 1.2.0
[SDB] spacetimedb-standalone path: /home/user/AxonotesCore/bin/spacetimedb-standalone
[SDB] database running in data directory /home/imgajeed/.local/share/spacetime/data
[SDB] warning: some trace filter directives would enable traces that are disabled statically
[SDB]  | `spacetimedb=debug` would enable the DEBUG level for the `spacetimedb` target
[SDB]  | `spacetimedb_client_api=debug` would enable the DEBUG level for the `spacetimedb_client_api` target
[SDB]  | `spacetimedb_lib=debug` would enable the DEBUG level for the `spacetimedb_lib` target
[SDB]  | `spacetimedb_standalone=debug` would enable the DEBUG level for the `spacetimedb_standalone` target
[SDB]  | `spacetimedb_commitlog=info` would enable the INFO level for the `spacetimedb_commitlog` target
[SDB]  | `spacetimedb_durability=info` would enable the INFO level for the `spacetimedb_durability` target
[SDB]  | `axum::rejection=trace` would enable the TRACE level for the `axum::rejection` target
[SDB]  = note: the static max level is `off`
[SDB]  = help: to enable logging, remove the `max_level_off` feature from the `tracing` crate
[SDB] 2025-07-21T18:42:45.127087Z DEBUG /home/user/AxonotesCore/SpacetimeDB/crates/standalone/src/subcommands/start.rs:186: Starting SpacetimeDB listening on 0.0.0.0:3000

Commands That Are Actually Intelligent

Because this is TypeScript, your commands can be smart. They can read your config, validate options, make decisions, and even handle complex workflows:

commands: {
    setup: command.define({
        help: "Initialize the project with key generation",
        exec: async (ctx) => {
            if (!env.JWT_PRIVATE_KEY_BASE64) {
                console.log("🔑 Generating JWT keys...");
                // Generate and save keys automatically
                console.log("✅ Keys generated! Update your .env.axogen");
            } else {
                console.log("✅ Keys already exist");
            }
        },
    }),

    deploy: command.define({
        help: "Deploy the application",
        options: {
            environment: z.enum(["staging", "production"]).default("staging"),
            skipTests: z.boolean().default(false),
        },
        exec: async (ctx) => {
            if (ctx.options.environment === "production" && !ctx.options.skipTests) {
                console.log("🧪 Running tests before production deploy...");
                // Run tests with live output
            }

            console.log(`🚀 Deploying to ${ctx.options.environment}...`);
            // Smart deployment logic based on environment
        },
    }),
}

Run with typed, validated options:

axogen run deploy --environment production --skipTests

Your commands get full IntelliSense, runtime validation, and proper error messages if you pass invalid options. It's programmable infrastructure, not just string replacement.

The Console Themes That Make Everything Beautiful

Here's something I'm particularly proud of in v0.3.0 - console themes. Because why should your configuration tool look boring?

# Try different themes
axogen run dev:all --theme aura          # Vibrant neon colors
axogen run dev:all --theme catppuccin    # Soothing pastels  
axogen run dev:all --theme doom-one      # Classic sophisticated
axogen run dev:all --theme astrodark     # Default balanced theme

Each theme is carefully crafted with semantic colors - success is always green, errors are red, but the exact shades and the way they work together creates completely different vibes. It's like having VS Code themes for your terminal output.

The themes aren't just pretty - they're functional. Different command types get different colors, so you can instantly tell if something is a file operation (cyan), command execution (orange), or debug info (blue).

When I Add Docker, Kubernetes, Whatever (It Just Works)

Right now, AxonotesCore has three services and it's fully migrated to Axogen. But here's where it gets really cool - Axogen can generate any format using templates. When I add Docker Compose:

export default defineConfig({
    targets: {
        docker: {
            path: "docker-compose.yml", 
            type: "template",
            template: "docker-compose.yml.njk",
            variables: env,
        },
    },
});

My docker-compose.yml.njk template:

version: '3.8'
services:
  spacetimedb:
    build: ./SpacetimeDB
    ports:
      - "{{ SPACETIME_PORT }}:3000"
    environment:
      ALLOWED_ORIGINS: "http://dashboard:{{ DASHBOARD_PORT }}"

  dashboard:
    build: ./dashboard
    ports:
      - "{{ DASHBOARD_PORT }}:5173"
    environment:
      PUBLIC_SPACETIME_WS: "ws://spacetimedb:3000"
      {% if NODE_ENV == "production" %}
      WORKOS_REDIRECT_URI: "https://app.axonotes.com/auth/callback"
      {% else %}
      WORKOS_REDIRECT_URI: "http://localhost:{{ DASHBOARD_PORT }}/auth/callback"
      {% endif %}

This generates different configs for different environments. Want Kubernetes manifests? Terraform files? Nginx configs? Just add more templates. One TypeScript config, infinite outputs.

The Full Power: Real Production Config

Want to see what this looks like in a real production application? Here's a glimpse of AxonotesCore's actual config (simplified for readability):

const env = createTypedEnv({
    AXONOTES_ENV: z.enum(["local", "staging", "production"]).default("local"),
    WORKOS_CLIENT_ID: z.string(),
    WORKOS_API_KEY: z.string(),
    JWT_PRIVATE_KEY_BASE64: z.string().base64(),
    JWT_PUBLIC_KEY_BASE64: z.string().base64(),
});

const getConfig = () => {
    switch (env.AXONOTES_ENV) {
        case "local":
            return {
                dashboardOrigin: "http://localhost:5173",
                spacetimeEndpoint: "ws://localhost:3000",
            };
        case "staging": 
            return {
                dashboardOrigin: "https://staging.axonotes.com",
                spacetimeEndpoint: "wss://staging-api.axonotes.com",
            };
        case "production":
            return {
                dashboardOrigin: "https://axonotes.com", 
                spacetimeEndpoint: "wss://api.axonotes.com",
            };
    }
};

export default defineConfig({
    targets: {
        dashboard: {
            path: "dashboard/.env",
            type: "env",
            variables: {
                WORKOS_REDIRECT_URI: `${config.dashboardOrigin}/auth/callback`,
                PUBLIC_SPACETIME_WS: config.spacetimeEndpoint,
                JWT_ISSUER: config.dashboardOrigin,
                // 20+ more variables, all automatically calculated
            },
        },
    },
    commands: {
        "srv:dev": command.define({
            help: "Start SpacetimeDB server with environment-specific settings",
            exec: async () => {
                const authFlag = env.AXONOTES_ENV === "production" ? "--auth-required" : "";
                await liveExec(
                    `./bin/spacetimedb-cli start --in-memory ${authFlag} --allowed-oidc-issuer ${config.dashboardOrigin}`,
                    { outputPrefix: "SDB" }
                );
            },
        }),
        // 15+ more commands, all environment-aware
    },
});

One environment variable change (AXONOTES_ENV=production) and the entire configuration shifts - different URLs, different auth settings, different security flags. Everything updates automatically.

Why I'm Building This (Despite Being Super Early)

Look, I built this in three days total. We're at v0.3.0. The API will probably change. But I had to share because the problem is everywhere.

We're stuck between dotenv (does basically nothing) and enterprise tools requiring dedicated DevOps teams. Research actually shows that 28% of outages are caused by "someone making a change to the environment" https://newrelic.com/resources/report/observability-forecast/2024/state-of-observability/outages-downtime and the Uptime Institute found that 66-80% of outages involve human error https://journal.uptimeinstitute.com/how-to-avoid-outages-try-harder/ (including configuration issues).

TypeScript developers love type safety but give it up for configuration. Teams onboard new developers with .env.example files that drift out of sync. Full-stack projects manually coordinate URLs across services. DevOps gets called for every environment variable change.

There's this huge gap where most development teams live. Axogen sits right there: familiar TypeScript, real type safety, automatic synchronization, and commands that actually look good when they run - but simple enough that you don't need a platform team.

It already works. I've migrated AxonotesCore completely to Axogen.

Try It Yourself

# Install axogen (npm, bun, yarn, etc.)
npm install @axonotes/axogen@0.3.1

# Create a minimal config using echo
echo 'import {defineConfig, createTypedEnv, z} from "@axonotes/axogen";
const env = createTypedEnv({
    NODE_ENV: z.string().default("development"),
    PORT: z.coerce.number().default(3000),
    DATABASE_URL: z.url(),
});
export default defineConfig({
    targets: {
        app: {
            path: ".env",
            type: "env", 
            variables: {
                NODE_ENV: env.NODE_ENV,
                PORT: env.PORT,
                DATABASE_URL: env.DATABASE_URL,
            },
        },
    },
});' > axogen.config.ts

# Create a matching .env.axogen file using echo
echo 'NODE_ENV=development
PORT=3000
DATABASE_URL=https://your.database.url' > .env.axogen

# Generate your config with validation
axogen generate

Check out the docs for more examples, or the GitHub repo to see how it works.

The Bigger Picture

Configuration should be boring. It should just work. You shouldn't choose between too simple (dotenv) or too complex (enterprise config management). You shouldn't give up type safety the moment you touch environment variables.

You should define your config once, in a language you already know, with the full power of TypeScript and Zod. Your commands should run with proper output and themes that make you actually enjoy working with your infrastructure. That's what Axogen is trying to be.

Give it a try, let me know what breaks, and tell me what you'd actually use this for. I'm @imgajeed76 on GitHub if you want to complain about bugs or suggest features.

Or just star the repo if you think this is a problem worth solving. Building dev tools is way more fun when people might actually use them.

Already done reading? Why not read the follow-up article?

Oliver Seifert

Jul 30 '25

Holy Shit, This Escalated Quickly - Axogen v0.5.0 Is Here And I Can't Even

#typescript #tooling #webdev #programming

Comments

7 min read

Built with ❤️ and mild frustration by Oliver Seifert. Currently v0.3.0, proceed with reasonable caution.

DEV Community: Oliver Seifert

pgit: What If Your Git History Was a SQL Database?

What is pgit?

Why Did I Build It?

Benchmarks: git vs pgit

It's Not Just About Storage

Use Cases

pgit for Agents

What's Next

I Got Obsessed With 2-Byte Deltas and Built a Delta Compressor That Actually Hits Them

The Problem That Broke Me

The Realization That Changed Everything

Building It: The Simple Stuff First

Algorithm 1: Just Store The Damn Characters

Algorithm 2: Deletion Is Even Simpler

Algorithm 3: Wait, What If We Tokenize This?

Algorithm 4 & 5: Pattern Detection

Algorithm 6 & 7: The "Fuck It, Use The Real Algorithm" Option

The Selection Logic

The SIMD Detour (Because Why Not)

The Tag System (This Is Where It Gets Wild)

How Tags Work

The Magic Moment

The Implementation Is Beautifully Dumb

The Benchmarks (Where I Got Really Excited)

tokio Results (Code Repository)

mdn/content Results (Documentation Repository)

The Surprise That Made Me Think

The Speed Numbers

Where xpatch Loses

Try It Yourself

Was The Rabbit Hole Worth It?

Holy Shit, This Escalated Quickly - Axogen v0.5.0 Is Here And I Can't Even

Wait, Dev.to Featured Me in Their Top 7?!

Top 7 Featured DEV Posts of the Week

The Type Safety Revolution (Finally!)

Secret Detection That Prevents Disasters

Zod Schema Validation (Because Type Safety Everywhere)

Backup System (I have trust-issue okay?)

Conditional Generation

Command System Overhaul (Nested Commands Are Here)

Environment Loading Revolution

Universal File Loading

File Format Explosion

Performance & Code Quality (The Boring But Important Stuff)

Join the Discord (Why Not?)

What's Next?

Try The New Hotness

I Built a TypeScript-Native Config System Because .env Files Drive Me Crazy

The Problem That Actually Broke Me

Turns Out, Not Really

Meet Axogen: Starting Simple, Getting Powerful

And Yeah, It's Actually Fast

Zod-Powered Type Safety

Fixing the Scripts Nightmare (Now With Style)

Commands That Are Actually Intelligent

The Console Themes That Make Everything Beautiful

When I Add Docker, Kubernetes, Whatever (It Just Works)

The Full Power: Real Production Config

Why I'm Building This (Despite Being Super Early)

Try It Yourself

The Bigger Picture

Holy Shit, This Escalated Quickly - Axogen v0.5.0 Is Here And I Can't Even