DEV Community

Cover image for I lost my best AI prompt after 40 tweaks. So I built a tiny git for prompts.
lululuhu
lululuhu

Posted on

I lost my best AI prompt after 40 tweaks. So I built a tiny git for prompts.

#ai #rust #opensource #productivity

🤯 We've all been there.

You're iterating on a prompt for an LLM. You tweak one word, the output gets better. You tweak another, it gets worse. You paste it into ChatGPT, it kind of works. You change "concisely" to "in 3 bullets", and suddenly the summaries are perfect.

Then tomorrow comes. And you have no idea which version was the good one.

Your folder looks like this:

prompts/
├── summarize.txt
├── summarize_old.txt
├── summarize_v2.txt
├── summarize_v2_final.txt
├── summarize_v2_final_FINAL.txt
├── summarize_REALLY_final.txt
└── summarize_use_this_one.txt
Enter fullscreen mode Exit fullscreen mode

And the worst part? The "good" one might be any of them. 😭

You can't diff them. You can't tell what changed between v2_final and v2_final_FINAL. You can't roll back to the version that actually worked.

🤔 Why not just use git?

Good question. I asked myself the same thing.

You can. And technically, PromptVault borrows git's entire model — content-addressed objects, trees, commits. But in practice, prompts live next to code. They're mixed into repos, notebooks, chat exports, and random .txt files on your desktop.

Prompts deserve their own version control that:

  • 📂 Tracks .txt/.md/.prompt/.j2/.yaml files specifically
  • 📁 Lives in a .pv/ folder that doesn't collide with your code repo's .git/
  • 🔄 Is built for the iterate-and-compare workflow that prompt engineering actually is
  • 🚫 Doesn't require a GitHub repo, doesn't sync to the cloud, doesn't ask for an account

So I built it. 🛠️

🚀 Enter PromptVault

PromptVault is git, but purpose-built for prompts. It's written in Rust, ships as a single ~2MB static binary, and runs entirely on your machine. No account, no cloud, no telemetry.

$ pv init
Initialized empty prompt vault in ./.pv

$ pv add prompts/summarize.md
added: prompts/summarize.md

$ pv commit -m "refine: summarize now reports tone + title"
[main 791151d] refine: summarize now reports tone + title
 1 prompt
Enter fullscreen mode Exit fullscreen mode

Now iterate. See exactly what changed:

$ pv diff prompts/summarize.md
diff -- prompts/summarize.md
 You are a precise summarizer.

-Summarize the text below in 3 concise bullets, then propose a title.
+Summarize the text below in 3 concise bullets, propose a title, and note the tone.

 {{text}}
Enter fullscreen mode Exit fullscreen mode

Walk back through every iteration:

$ pv log
commit 791151d…
parent 4100199…
Date:   Mon Aug 10 10:04:33 2026 +0000

    refine: summarize now reports tone + title

commit 4100199…
Date:   Mon Aug 10 10:04:33 2026 +0000

    feat: initial prompt set
Enter fullscreen mode Exit fullscreen mode

Restore any past version by hash, tag, or branch:

$ pv show 4100199   # prefix works too 🔍
$ pv revert v1.0    # restore working tree to a tagged version ⏪
Enter fullscreen mode Exit fullscreen mode

🌿 Branches for A/B testing

This is the killer feature for prompt engineering. You want to test two variants of the same prompt? Branch it.

$ pv branch experiment
$ pv checkout experiment
# ... tweak the prompt, commit it ...
$ pv checkout main       # working tree restores to main's version
$ pv checkout experiment # ...and back to experiment's version
Enter fullscreen mode Exit fullscreen mode

Switch branches and the working tree restores instantly. No copying files, no renaming, no "wait, which folder was the experiment in?" 🎯

When you're done, compare them against a dataset:

$ pv ab main:summarize.md experiment:summarize.md -d cases.jsonl --show
A/B:  A=main:summarize.md  B=experiment:summarize.md  (3 cases)

[1/3] DIFF  differs
--- A vs B ---
-A You are a precise summarizer.
+B You are a concise summarizer.

  hello world
--- end ---

Summary: 0 identical, 3 differing (of 3)
Enter fullscreen mode Exit fullscreen mode

Pure local. No model calls. No API keys. 🔒

💸 Evals without the API bill

One thing that annoyed me about existing prompt tooling: everything wants to call a model. Every eval run costs money. Every test sends data to OpenAI.

PromptVault takes a different stance: it never calls a model. It only renders templates and checks assertions.

$ pv eval summarize.md --dataset cases.jsonl --show
Eval: summarize.md  (3 cases)

[1/3] PASS  contains "3 concise bullets"
--- rendered prompt ---
You are a precise summarizer.
...
--- end ---

Summary: 2/2 passed (100%)
Enter fullscreen mode Exit fullscreen mode

You write a JSON Lines dataset, each line fills the prompt's {{variables}}, and PromptVault renders + asserts. ✅

If you want to actually run the prompt through a model, pipe the rendered output to whatever runner you trust (curl, the OpenAI CLI, ollama, your own script).

This separates two concerns that existing tools conflate:

  1. Did my template render correctly? (cheap, local, deterministic) ⚡
  2. Does the model produce good output? (expensive, non-deterministic, model-dependent) 💰

PromptVault does (1). You choose how to do (2).

📋 The full feature list

For a "tiny git for prompts", it ended up with more than I planned:

  • 📸 Snapshots with messages (like git commits)
  • 📊 Line-level diff so you can see exactly which instruction you changed. pv diff --stat for a one-line summary per file.
  • 🌿 Branches & merging — pv branch experiment, pv merge experiment (fast-forward or three-way, with conflict markers)
  • 🏷️ Tags & rollback — pv tag v1.0, pv revert v1.0
  • 🚫 .pvignore — gitignore-style, so drafts stay out of the vault
  • 🔀 Ref-to-ref diff — pv diff v1 v2 compares any two commits/tags/branches
  • 📂 Ref:path access — pv show HEAD:summarize.md reads any file at any ref
  • 🧰 Stash / reset / clean / grep / export / stats / blame — everyday git-class utilities
  • 🖥️ TUI — pv tui launches an interactive commit browser
  • 🐚 Shell completions — bash / zsh / fish / elvish / powershell
  • ☁️ Remote sync — pv push / pv pull syncs the vault to any git host as a backing store
  • 🤖 Model runner (opt-in, build with --features run) — pv run against OpenAI/Anthropic/Ollama. API keys live only in env vars; nothing is ever logged or stored.
  • ⚖️ A/B testing — pv ab main:x experiment:x -d dataset.jsonl renders two versions against the same dataset and diffs them

🔧 How it works under the hood

PromptVault is a tiny git. On pv init it creates:

.pv/
├── HEAD              → "ref: refs/heads/main"
├── index.json        → staging area (path → blob hash)
├── objects/          → content-addressed store (SHA-256)
│   └── ab/cdef…      → "<type>\0<data>"  (blob / tree / commit)
└── refs/heads/main   → latest commit hash
Enter fullscreen mode Exit fullscreen mode

Every prompt version is a blob addressed by the SHA-256 of blob\0<content>. A tree maps paths to blobs. A commit points to a tree + parent + message. Identical content is stored once. Nothing ever leaves your machine unless you explicitly pv push. 📦

It uses Myers diff for line-level changes (same algorithm as git), and the glob matcher for .pvignore is a dynamic-programming implementation to avoid the exponential backtracking that naive recursive matchers hit on patterns like *a*a*a*. 🧠

📦 Install

Three options:

# 1. Prebuilt binary (no Rust toolchain needed) 📥
#    Download from https://github.com/lululuhu/PromptVault/releases
#    Available for: Linux/macOS/Windows, x86_64 and aarch64

# 2. cargo 🦀
cargo install promptvault

# 3. From source 🔧
git clone https://github.com/lululuhu/PromptVault
cd PromptVault
cargo build --release
# binary: target/release/pv  (put it on your PATH)
Enter fullscreen mode Exit fullscreen mode

Then, in any folder where you keep prompts:

pv init
Enter fullscreen mode Exit fullscreen mode

That's it. You're versioning prompts. 🎉

🔒 The trust model

A quick word on this, because I think it matters.

PromptVault is local-first. Everything lives in .pv/ on your machine. No account, no cloud, no telemetry, no analytics, no "phone home". The only network calls are the ones you explicitly make:

  • ☁️ pv push / pv pull to sync to a git remote you configured
  • 🤖 pv run (opt-in feature) to send a rendered prompt to a model API

The pv run command reads API keys only from environment variables. Nothing is logged. Nothing is stored beyond the rendered prompt itself. 🔐

Important caveat: if your prompts contain secrets, PII, or confidential information, committing them to a vault stores that data on disk in plaintext (content-addressed, but unencrypted). Pushing to a remote sends it to that git host. Same rule as git — don't commit secrets you wouldn't commit to git. ⚠️

🗺️ What's next

This is v0.2.0. The roadmap ahead:

  • 🤝 Better conflict resolution UX (currently three-way merge produces conflict markers; a pv mergetool is planned)
  • 📜 Prompt templates with conditionals/loops (right now it's just {{var}} substitution)
  • 🌐 A web UI for browsing history (the TUI is nice but a browser is nicer for sharing)
  • 🪝 Hooks (pre-commit, post-checkout) for automation

If you have opinions, the issue tracker is open. 🚪

🎯 Try it

If you've ever lost a good prompt to "v2_final_FINAL.txt" syndrome, give it a spin:

⭐ Star it if it's useful.
🐛 Open issues if it's not.
I'm building this in the open and good ideas ship fast.

PromptVault is MIT-licensed, written in Rust 🦀, and runs on Linux/macOS/Windows. The author has no affiliation with any AI lab.

Top comments (0)