DEV Community: Konstantin Gredeskoul

Home-made Evals Just Taste Better

Konstantin Gredeskoul — Thu, 05 Nov 2026 18:14:28 +0000

Note

This is a shortened version of the original writing on kig.re — my personal blog.

Here is what a real lead typed into a tax-preparation intake form:

“Got married in June, haven’t figured out how we’re filing yet. There’s an IRS letter somewhere in a kitchen drawer I haven’t opened. I also deliver groceries on weekends.”

Only one fact in that paragraph is clear: the weekend side gig. A model read it anyway and set filing_status to married_jointly.

Helpful. Wrong. The form then skipped the filing-status question and treated a guess as settled. That is the failure I care about most, and it is invisible if you only eyeball a few outputs.

The vague tax-preparation case: the lead's input, the expected JSON with filing_status null, gpt-4o-mini hallucinating married_jointly at 78.8%, and deepseek-v4.1-flash-1017 scoring 100% with empty diagnostics.

I built llm-evals so I would stop writing scripts/eval.rb for the fourth time. It is a Ruby CLI for multi-dataset, multi-case evaluation suites. Compare models, efforts, temperatures. Rank them by one weighted score that balances accuracy, hallucinations, errors, latency, and cost. On RubyGems as of writing: 0.5.0. Two commands: evals and llm-evals.

gem install llm-evals

Home-made pasta tastes better because you chose the flour. Home-made evals taste better because you chose the weights, the cases, and the failure modes that matter to your product. Store-bought leaderboards cannot taste your soup.

Evals are the unit tests for probabilistic software

A unit test asks: given this input, does my code return exactly this output? An eval asks a softer question: given representative inputs, how often and how well does the model produce what I expect? You cannot assert equality against a probability distribution. You can score it. Score the same cases every time, and you can finally measure what a prompt tweak, a temperature change, or a model swap actually did.

The scorer decides what “good” means. The eval makes the comparison honest and repeatable.

The problem that started it: Qualified.At

The gem grew out of Qualified.At, which I wrote about when it launched. A business builds an intake form, embeds it, and visitors answer questions before anyone picks up the phone. The forms use the Inquirex gems. Many start with a free-text box: tell us about your situation. An LLM reads that paragraph and extracts structured answers, so a 20- or 30-question form can shrink to a few follow-ups.

Here is the extraction step from the tax preparation demo, copied from the suite’s dsl.rb:

extract :understanding do
  from :situation
  prompt "Extract the taxpayer's situation from their description. Only fill a " \
         "field when the text genuinely supports it — leave anything the writer " \
         "did not mention unset rather than guessing. ..."
  schema filing_status: { type: :enum,
                         values: %w[single married_jointly married_separately head_of_household] },
    dependents: :integer,
    self_employed: :boolean,
    rental_properties: :integer,
    sold_investments: :boolean,
    owns_business_entity: :boolean,
    multi_state: :boolean,
    irs_letter: :boolean
  transition to: :heard_you
end

And here is what that lead typed (the vague case):

Honestly, this year was a blur and I’ve been avoiding thinking about taxes. I got married in June, so I assume that changes things, though we haven’t talked about how we’re filing. I also picked up a side gig doing grocery deliveries through one of those apps on weekends, which I’m told counts as being self-employed. Somewhere in a kitchen drawer there’s a letter from the IRS that came in August that I haven’t been brave enough to open. We rent our apartment, if that matters. I don’t really know what else you need from me, so please just tell me what to bring.

The correct extraction is not “married filing jointly”. They got married. They said they have not decided how to file. The right answer is:

{
  "filing_status": null,
  "dependents": null,
  "self_employed": true,
  "rental_properties": null,
  "sold_investments": null,
  "owns_business_entity": null,
  "multi_state": null,
  "irs_letter": true
}

Two answers, six honest nulls. A model that “helpfully” fills in married_jointly has invented a fact. On this case, openai/gpt-4o-mini scored 78.8% and the diagnostics named the hallucination: expected null, got married_jointly. deepseek/deepseek-v4.1-flash-1017 at high effort scored 100% with empty diagnostics.

So the product question became: which model captures the most answers, assigns them correctly, avoids hallucinations, and does not miss details, while staying cheap and fast?

A lead who waits 30 seconds for the next question is a lead who closes the tab. At 1,000 leads, $0.0002 per call is twenty cents. $0.50 per call is five hundred dollars.

The Qualified.At suite

Qualified.At ships ten demo businesses: bookkeeping, couples therapy, home remodeling, immigration law, managed IT, personal injury, roofing, software development, tax preparation, and wedding photography. Each one became a dataset. Each dataset has four case variants.

Here is the thing that bit me. The report had a dozen sortable columns: accuracy, exact match, hallucination rate, errors, latency, cost, tokens. Sort by accuracy and an expensive, slow model wins. Sort by cost and a model that hallucinates wins. Sort by primary score and you crow a token hog. None of those columns answered the question I actually had, which was which one should I ship?

So llm-evals lets you say what matters, in YAML, and folds it into one column called w wscore. This is the weighted_score section from Qualified.At’s .llm-evals.config.yml:

weighted_score:
  weights:
    # Positive metrics (higher is better)
    exact:              10 # Exact contributes high number
    score:               5 # This is a score computed in other ways
    # Negative metrics (lower value is better)
    latency:            -5 # Latency is important, but relatively less than others
    cost:               -5 # Same with the cost
    time:               -3 # latency contributes relatively low comparatively
    errors:             -8 # and any errors contribute quite a bit to the total
    hallucinated_rate:  -6 # Hallucinations are bad, and are penalized
    invalid_rate:       -8 # invalid_rate is an error, so heavy penalty

A positive weight means higher is better. A negative weight means lower is better. Raw weights are hard to reason about, though, so the llm-evals CLI ships a subcommand that does the math for you. I called it weights, which is not my most imaginative work. Run evals weights and it reads your .llm-evals.config.yml, then prints each column’s direction and its real share of the final score:

The help screen of evals weights -h: print how the config file's weights divide the weighted score, with -C to point at another lineup file.

Here is what it prints for the Qualified.At config:

Read that table as a product brief. Exact alone is 20% of the score: I want every field right, not almost right. Errors and invalid together are 32%: a broken reply is worse than a slow one. Hallucinations are 12%. Cost is only 10%. Latency is 10%. Wall time is 6%. That is not a universal truth. That is what this intake product cares about.

The formula is deliberately boring:

Each metric’s goodness is its value, or 1 - value under a negative weight. cost and time go on a log scale between the cheapest and most expensive contender in the table, so each doubling costs the same penalty, no matter how absurd the priciest model is. Errors come straight off the top: 20 points for the first, 25 for two, and so on.

The weights are opinions, and that is the point. They force you to write down the trade-off you were going to make in your head anyway.

The results

Please see the original post for the full result set. There are more details there.

How the gem is shaped

You write two small classes. The gem does everything else.

The adapter runs one case against one model and returns a Reply. It owns everything the model sees and everything done to its answer. In Qualified.At, it calls the same prompt builder and the same parser production uses, so the scores reflect production rather than a lab version of it.

The scorer turns a case and a reply into a Score with whatever metrics you declare. It never calls anything.

The gem handles case discovery, the model lineup, providers (OpenRouter by default, through RubyLLM), rate limits, retries, thread pools, progress bars, the leaderboard, an HTML report, and optional reporting to Braintrust or Logfire. Some of the design came from Pydantic Evals.

A few decisions I am stubborn about:
A wrong answer is a 0. A network error is an error, not a 0. Your model did not answer wrongly because OpenRouter returned a 503.

A model the suite cannot ask is refused by name, not scored. If your suite sends chat prompts and the lineup lists a typed-question model, you get typesafe/jev-latest refused: needs typesafe; this suite produces prompt before anything runs.

Settings multiply. effort: [low, high] with temperature: [0.0, 0.7] becomes four contenders for that model. You are not comparing models; you are comparing configurations.

No lock-in. A local JSON log is always written. Any HTML report can be re-rendered from it later with evals render, with no model calls.

Try it

The repository ships two example suites that use fake models, so they need no API keys and no network:

git clone https://github.com/kigster/llm-evals.git
cd llm-evals
bundle install
just examples

For your own project, copy the toy suite’s evals/ folder, swap in your adapter and scorer, list real models under openrouter: in .llm-evals.config.yml, set OPENROUTER_API_KEY, and:

evals cases        # list the cases; calls nothing
evals verify       # offline checks of inputs, expected answers, adapter and scorer
evals compare -n   # show the experiment matrix; calls nothing
evals compare      # run it, and open the report

Run evals compare -n before every paid run. It prints the full experiment matrix, so you find out that you accidentally multiplied three efforts by three temperatures by twelve models before OpenRouter does.

The gem is MIT-licensed, and the README covers writing providers (there is a full Ollama example) and trackers (there is a W&B Weave sketch). Issues and pull requests are very welcome.

Final Thoughts

There is no best model. There is only the best model for a given task, at a given price, with a given tolerance for being confidently wrong. DeepSeek won my suite overall, and it still lost three datasets to gpt-6-luna and one to gpt-4o-mini. If I had picked one winner and wired it into every extract step, I would have shipped worse answers to a third of my customers and called it a decision.

The leaderboard is not the point. The point is that the question “which model should we use?” stops being an argument and becomes a command you can rerun next week, when the next model drops and the prices move again. Write the cases your product cannot afford to get wrong. Choose weights that say what you actually care about. Then let the numbers settle it.

And if you are about to write scripts/eval.rb for the fourth time, please don’t. I already did. Home-made tastes better when someone else rolled the dough.

Managing Secrets in Development environments

Konstantin Gredeskoul — Thu, 05 Nov 2026 17:44:17 +0000

I write a lot of code, open source, and as part of large engineering teams. How poorly teams manage their development secrets, some that often overlap with production blows my mind. So I coded what I hope is a decent solution.

So, I don't know how prevalent or wide spread this problem is. But here it is: you join a new team and someone slacks you the .env file filled with 50+ API keys, client ids, secrets, it's a shell-like file, so some come with comments "please go sign up and replace this token with your own". But others don't. The entire team uses one key for a service X. And among the keys that folks share in development may just a few: three to five, are the same as the production keys. Noice. Oh wait, .env is git-ignored. Phew!

Cause nobody ever made a copy and accidentally committed it, as .env-alex and that fuzzy and warm git ignore rule only mentioned **/.env, so .env-alex went through and, well we squash merged the branches, so we are ok. Right? Gosh. God no. GitHub support ticket to kill your orphan commit may not be enough.

The onboarding wiki says "get the .env from a fellow developer, using slack, email or airdrop" and then change the couple that require individual logins.

Ever since I worked at Coinbase, and on my first week on the Infra team the guy sitting next to me ended up getting phished, resulting in him delivering his laptop to the op-sec team and that day they identified not one but two zero days in Firefox by air-gapping his laptop. Someone had spent about a mil on dark web to get into his laptop! (I should
Since then, and especially now with all the AI agents about to become leet hackers, I am just in shock at how some engineers are so clueless about Whats about to happen!

My personal recommendation, if you are on GMail as your personal, and you are not using Advanced Protection with hardware keys, you will be hacked in the next year or so.

https://landing.google.com/advancedprotection/

Anyway. In the last six years i worked on four or five projects where dev secrets were a very fleeting afterthought. So this year I wrote a tool that does not compromise on security but gives you a safe way to keep your goddamn secrets in git and not stress about it.

It's a Rust based CLI tool that's a simple orchestrator on top of SOPS, age and age hardware plugins. Ok Mac with new Silicon processor it uses Secure Enclave to mint the keys. Or a YubiKey.

I am looking for feedback and beta testers! This is a free tool distributed under the MIT license. Adoption is what I worry about.
So please, give it a whirl.
Version 2.0.0 is out, supporting YubiKeys and a slew of new features.

https://sopsy-cli.dev/

Thanks!
Konstantin

Sopsy: Secrets in Git, Sealed by the Secure Enclave

Konstantin Gredeskoul — Sat, 22 Aug 2026 00:31:16 +0000

Well, I did a thing I've wanted to do for a while: I published sopsy — my first open-source package written in Rust — to crates.io and to Homebrew. After a couple of decades of shipping Ruby, some C, and an unreasonable amount of Bash, watching cargo publish go green on a crate of my own felt genuinely great. I am allowing myself a small victory lap before Rust inevitably finds a new way to humble me.

And like all the "cool kids" these days, of course it has its own snazzy website, because apparently a CLI is not real until it has a domain name. Visit sopsy-cli.dev now, or after you're done reading.

What is it, and why might you care?

sopsy lets your team commit its secrets — encrypted — directly into git, where each developer holds their own private key inside their Mac's Secure Enclave, and decryption takes a fingerprint. There's no shared master key (the single thing that ultimately sank my last attempt at this, more on that below). No secrets server to run or pay for. No plaintext .env quietly making the rounds on Slack. Adding a teammate is a pull request; removing one re-keys every file automatically.

Who is it for? Teams that want tight, revocable control over exactly who can decrypt the developer secrets they keep in git — small-to-mid engineering teams, open-source projects, and security-conscious startups that would rather not stand up Vault just to stop passing .env files around by hand. It is deliberately bottom-up: one developer installs it, tries it on a repo, and brings the rest of the team along. macOS-first for the hardware bits, today.

The mental shift, compressed for the busy:

A shared symmetric key (or a .gitignore) asks you to guard one secret everyone copies.
sopsy lets each person hold their own key in hardware — and re-keys the team when membership changes.

What it actually is

First, the honest framing, because security tools get insufferable very quickly: sopsy does not reimplement any cryptography. It stands on three excellent, boring, battle-tested tools that are kind of annoying to learn and use, and makes them, in my opinion, a lot more pleasant to live with:

SOPS — Mozilla's secrets operator, which cleverly encrypts the values in a file while leaving the keys readable, so a diff still tells you "DATABASE_URL changed" or when the "OPENAI_API_KEY" was added, without leaking what it changed to.
age — Filippo Valsorda's modern, no-options-to-get-wrong encryption, the spiritual antidote to GPG.
age-plugin-se — the bit that made me warm and fuzzy: it generates an age identity whose private key is born inside your Mac's Secure Enclave and physically cannot leave it. Decryption requires Touch ID. The key isn't on disk for malware to find, because it was never on disk at all. The fact that macOS has a Secure Enclave sitting there, quietly waiting to solve exactly this problem, was a huge surprise to me. But at the same time, it wasn't.

sopsy is the friendly Rust CLI that orchestrates all three: it bootstraps a repo, manages who can decrypt, re-keys everything when membership changes, and ships a CI gate that fails the build the instant a plaintext secret sneaks in. SOPS stays the engine; sopsy is the steering wheel, the seatbelt, and the annoying little dashboard light that tells you when you are about to do something dumb.

Here's the shape of it:

The whole trick is that arrow at the bottom. Encryption only needs your teammates' public keys, so it's silent and offline. Decryption needs the private key, which lives in silicon and answers only to your fingerprint. The ciphertext rides happily in git; the thing that unlocks it never does.

The five-minute tour

You start a repo the way you'd hope:

git init my-app && cd my-app
sopsy init          # checks your tools, mints a Secure Enclave identity, writes the config
sopsy edit .env.encrypted   # opens your $EDITOR; decrypts in, re-encrypts on save (Touch ID)
sopsy check         # the CI gate: fails if any plaintext secret is tracked 
git add .sops.yaml .sopsy.yml .env.example .env.encrypted .gitignore
git commit -m "Encrypted secrets, managed by sopsy"

sopsy init prints your public recipient — an age1se1… string — and that's the only part of your identity that's ever shared or committed. For day-to-day use there's also sopsy decrypt (and sopsy secrets decrypt), which streams plaintext to stdout, so you can wire it into direnv and unlock the whole environment once per shell instead of fighting it per-command:

# .envrc — one Touch ID when you cd in, then your secrets are just there
eval "$(sopsy decrypt .env.encrypted | sed -E '/^#/d; /^$/d; s/^([A-Z])/export \1/g')"

If Touch ID on every read gets old — you live in direnv, or you run a dozen windows on one project — you can mint your Enclave key without a biometric gate: sopsy join "Your Name" --without-touch-id. The private key still never leaves the Enclave and is still device-bound; it just no longer prompts. This is a deliberate trade: less thumb, less ceremony.

Where it gets good: the team model

A shared symmetric key has no notion of "who." sopsy does, and that's the whole reason it works for a team. Two files carry the state:

.sops.yaml — consumed by SOPS itself: a list of public recipient keys that any new secret gets encrypted to.
.sopsy.yml — sopsy's own ledger: human names, who owns which key, lifecycle state, a small audit trail (who requested access and when, who approved them and when), and the break-glass marker. A committed .sopsy.sha checksum makes hand-edits tamper-evident instead of silently trusted.

Onboarding is self-service, which is the part that actually changes how a team feels. A newcomer doesn't chase anyone for a key; they mint their own and send the public half in a pull request:

Note what a "pending" join grants: nothing. The newcomer's key isn't in .sops.yaml yet, so it can't decrypt anything — the request is just a public key sitting in a PR. (sopsy join is also aliased sopsy request-access, if that reads better in your team's docs.) An existing member reviews it, vouches that the key really belongs to Anna — you confirm that out-of-band, the same way you'd confirm a GPG fingerprint — and sopsy approve does the rest: adds the key, re-wraps the data key for the new recipient set with sops updatekeys, and flips Anna to active. Offboarding is the mirror image. No shared secret, no master password, no "we should probably rotate that someday."

And because every developer's private key is hardware-bound, there are two deliberate safety valves for the cases where a fingerprint isn't available:

Break-glass — a portable age key you generate once, store offline in 1Password, and register as an emergency recipient. If everyone's laptop falls in the ocean, that one offline key can still open and re-key the repo. sopsy nags you until you make one, and after this week I understand exactly why.
CI decryption — sopsy recipient ci runs the same guided ceremony to hand your pipeline a portable key: one CI secret named SOPS_AGE_KEY, and your workflows can decrypt. The Secure Enclave, correctly, will never work headless — so this is how CI reads secrets without one.

A little history: how I got here

Nine years ago I wrote a little tool called sym and wrote a blog post about it.

I opened the post with a complaint that has aged distressingly well: keeping your secrets out of git, in a pile of .gitignored files, gives you a false sense of security and costs you everything good about version control. No history. No rollback. No diff. And the eternal team tax of "hey, can you AirDrop me the .env again? mine's three weeks stale."

The dream and ambition back then was not quite the same as today, but close: I just didn't want to have plain text secrets lying around in my repos, even if they were git-ignored. Ideally, I wanted to encrypt them and check them in. Pre-commit hooks existed back then, so committing a plaintext secret to git was less of an issue. But "keys to many kingdoms" just lying around on my file system in plain text, waiting for someone to break in and steal them — that bothered me, annoyingly and continuously. So I wrote sym — a symmetric encryption Ruby gem with a couple of macOS-specific twists. And for 2017 it wasn't so bad.

sym is a Ruby gem that used OpenSSL encryption underneath, so that part was solid. You generated a symmetric key, and you could encrypt your secrets and commit them to your repo as one text blob (encrypted data was also base64 encoded). You had one master key in 1Password, and when you "onboarded" someone, you'd grab that master key, and they would import that key using sym into OS X Keychain, in the process wrapping the key in an additional layer of password-protected encryption. Now you could git-commit the secrets, and to decrypt them on your machine required your password (whatever you assigned during the import). That was also a bonus. But a single master key was now the one thing that, if leaked, rendered the entire process completely pointless. And in that sense, some security people would say things like "Great tool, it gives you a warm and fuzzy feeling of protection, without actually providing it"...

It had a few conveniences too — you could request that your password be cached in a local memcached instance, although there I got lazy and cached it in clear text. Security via obscurity, right?

Still, it was way better than everyone sharing a plain text file with two dozen API keys that, if stolen, could have been used to rack up significant fees for the company, not to mention access some private data, which would have been way worse.

So? That was fine for 2017, and a few projects I was on adopted it. It did not become a hit, because shortly thereafter Rails launched credentials, which, although it lacked the convenience of the password-protected and Keychain-stored encryption key, ultimately allowed you to keep encrypted credentials in the repo. With the popularity of Rails, there was no competition.

Having recently been on a project where the .env file contained nearly one hundred various secrets and API keys, I thought to myself, maybe we can do better in 2026?

The research

After numerous conversations with Claude and ChatGPT about the best existing secrets-encryption tools for developers, and specifically — in the development environment — I kept coming back to the same question.

Production systems in the cloud have matured. If you're running on AWS, your keys are in KMS. If you're a Terraform nut, probably Vault. But locally, on your development MacBook, what is the preferred method to avoid keeping "keys to the kingdoms" lying around in plain text?

Surely, the tool that encrypts your dev secrets and does not depend on a single symmetric key has been written by now? Well, maybe it has. But the three of us (the two AI agents and myself) could not find anything that did what I wanted, with the ease of use I had in mind.

Having identified a potential hole in the developer tooling, and having collected information about the modern tools people use for encryption and key management, I concocted a plan. So over the last few days Claude and I built sopsy — a Rust-based CLI for doing what I wanted in 2017, but could not quite build then without hand-waving around the most important part.

A parade of excellent bugs

Now the fun part — and the honest part. I built sopsy over a handful of days pairing with an AI coding agent, and here's the detail I keep turning over: the agent wrote clean, well-tested code that passed its own suite green, and almost every real bug was found the moment a human actually used the thing. Not in CI. In my hands, on two real macOS accounts, with my actual thumb. That gap — between "the tests pass" and "it works for a person" — is the whole story of this section, and frankly the whole story of evals too. Determinism in the test suite is not correctness in the world.

Here are the ones worth your time.

1. The key that wasn't there

The first real run of sopsy init ended with the most confusing class of failure: success, then nothing worked. It generated my Secure Enclave identity, encrypted the file, reported victory — and then every decrypt failed with age: identity did not match any of the recipients.

The cause is a beautiful little misunderstanding about how age-plugin-se works. When it generates a key, it hands you back a long AGE-PLUGIN-SE-1… string. It is tempting — and wrong — to treat that string as a throwaway, because "the private key is in the Enclave, so who cares." But that string is the handle: the reference SOPS needs to ask the Enclave to unwrap anything. We were generating it and dropping it on the floor. The private key was safe in the Enclave, all right — and totally unaddressable, like a safe deposit box whose number you dropped in the toilet and accidentally flushed.

The fix is the distinction worth internalizing: the handle is not the secret. It only works on this device, behind this fingerprint, so it is perfectly safe to write to disk — and you must, or the Enclave key is useless. sopsy now persists it to the standard sops keys file and points every SOPS call at it. The private key still never leaves the chip. The map to it just needs to exist.

2. The `init` that ran forever

Early on, a teammate (me, in a different mood) ran sopsy init in the wrong directory — one that wasn't a fresh repo, so git happily walked up and decided the enclosing repository was my entire home folder. sopsy then tried to find every encrypted file to re-key it by... reading every file under the repo root. All of $HOME. It was still going, fans spinning, several minutes and a few gigabytes of node_modules later.

An unbounded "scan everything" is a bug that hides until someone points it at a big enough directory, and then it's a hang, not an error — the worst kind. The fix was to stop walking the filesystem and instead expand the declared encrypted-file globs directly (a * that never crosses a /), so the scan touches the directories the patterns actually name and nothing else. And init now refuses to silently adopt your home directory as a secrets repo — it stops and asks.

3. The secret that couldn't be committed (and then couldn't be shared)

This one was a two-part trap, and it's my favorite because each half was invisible without the other.

The default .gitignore sopsy wrote was a touch too greedy. It ignored .env.* to keep stray dotenv files out of git — sensible! — but .env.example.encrypted matches .env.* too. So the one file you wanted committed was quietly untracked. That's part one.

Part two: when a new member was approved, the re-key step enumerated files via git ls-files. Reasonable, except that command deliberately skips ignored files — so the encrypted artifact that part one had just hidden was also skipped during re-keying. The new member got added to .sops.yaml, everything reported success, and then they couldn't decrypt one of the files, because they'd never been added to it.

The lesson is a clean little invariant: encrypted artifacts are first-class committed files, full stop. The gitignore now explicitly rescues every *.encrypted no matter what, and re-keying enumerates secrets by globbing the working tree — not by asking git, which has opinions about what's worth mentioning. When two reasonable behaviors compose into one wrong behavior, the invariant was wrong or incomplete. That is where the bug was hiding.

4. `sudo` cannot touch the Enclave (and that's the point)

I tested the two-person flow with two macOS accounts on one laptop, and switched between them with sudo su - arc. Decryption failed every time, no Touch ID prompt, just the same flat "no identity matched." I was certain this was a sopsy bug. It is not a sopsy bug.

The Secure Enclave will only release a key to the user's active GUI login session. sudo, su, SSH — none of them can raise the biometric prompt, so the Enclave just says no. The instant I used Fast User Switching to actually log arc in at the screen, the prompt appeared, my thumb did its thing, and the secret opened.

This isn't a wart, it's the threat model doing its job. A key you can use over SSH is a key an attacker can use over SSH. Hardware-bound, presence-required decryption is exactly what you want for production secrets — and it's precisely why the portable break-glass and CI keys exist for the headless cases where the Enclave, correctly, won't play. Touch ID is per-account, too: each user enrolls their own fingerprint, falling back to that account's login password.

There were smaller ones — my own tests cheerfully writing fake keys into my real keystore, a Homebrew install shadowing the freshly-built cargo binary on $PATH so I kept testing the old bits — but the four above are the ones that taught me something. Every single one survived a green test suite and died on contact with a human. Annoying? Yes. Useful? Also yes.

sopsy in one glance

The whole feature set, for the skimmers:

One-command bootstrap — sopsy init writes .sops.yaml, .env.example, an encrypted .env.encrypted, .gitignore safety rules, and .sopsy.yml. Idempotent, so re-running is always safe.
Secure Enclave identities — your private key is generated inside Apple Silicon hardware, gated by Touch ID, and impossible to read or exfiltrate. --without-touch-id if you want it device-bound but prompt-free.
Self-service membership with an audit trail — sopsy join (alias request-access) records a pending request; any active member sopsy approves. .sopsy.yml keeps who asked, their username, when, and who approved them and when; sopsy recipient list prints it all.
Automatic re-keying — every add / remove / approve re-wraps all secrets via sops updatekeys, so the recipient set never drifts from the ciphertext.
Friendly edits + scriptable crypto — sopsy edit opens an encrypted file in your $EDITOR with auto file-type detection; sopsy encrypt / sopsy decrypt (and the secrets forms) do one-shot, direnv-friendly crypto to stdout or a file.
Break-glass + CI ceremonies — sopsy recipient break-glass mints an offline emergency key; sopsy recipient ci mints a portable CI decryption key (one SOPS_AGE_KEY secret and your pipeline can decrypt).
Tamper-evident config — a committed .sopsy.sha checksum over .sopsy.yml + the admin key is refreshed on write and verified on read, so hand-edits are surfaced, not silently trusted.
A CI gate that bites — sopsy check enforces hygiene invariants and exits non-zero on any tracked plaintext. It needs no key, so it runs on a Linux runner.
sopsy doctor — a grouped, colorful health report of your tools, Secure Enclave, Touch ID, and repo that you can paste straight into a GitHub issue.

Your turn

If you keep secrets in git today — or you've been getting away with a shared .env and quietly worrying about it — I'd genuinely love to hear how your team handles this. What are you using? What would stop you from committing encrypted secrets right next to the code? Install it (brew install kigster/tap/sopsy or cargo install sopsy), point it at a throwaway repo, and tell me what breaks.

Leave a comment below — bug reports, "you reinvented X," and "why not just use Y" are all equally welcome. And if it's useful to you, a star on GitHub helps more people find it.

Go encrypt something. Then try to commit it. That part is supposed to feel good now.

— Konstantin Gredeskoul

San Francisco, CA, June 28, 2026.

In the spirit of full disclosure: sopsy was written over a few days pairing with an AI coding agent, and the experience is the quiet subtext of this whole post. The agent was a frankly excellent programmer — clean Rust, real tests, all green. And it could not, on its own, find a single one of the bugs above, because every one of them required a human, a thumb, two login sessions, and a fair bit of swearing. The code was AI-built; the correctness and validation came from me yelling at Claude, compacting the context more than once, and generally being pretty annoyed with it. Keep a human in the loop, especially the one holding the finger and the fingerprint.

References

sopsy on GitHub — source, install instructions, and the owner/member guides.
sopsy on Crates.io — where you'd install it from.
SOPS · age · age-plugin-se — the three tools sopsy stands on.
Dead Simple Encryption with Sym — the 2016-era ancestor, and the problem statement that still holds.
Evals: The Unit Tests for the Non-Deterministic Parts of Your App — more on the gap between "tests pass" and "it works."

What 25 Years of Deterministic Software Engineering Taught Me About Building AI Systems

Konstantin Gredeskoul — Tue, 23 Jun 2026 16:49:46 +0000

One of the strangest things about AI engineering is that your test suite can be 100% green while your product is getting worse.

Traditional software taught us to think in absolutes:

pass/fail
correct/incorrect
deterministic/non-deterministic

AI systems force us to think in distributions, thresholds, confidence intervals, and trade-offs.

It’s a subtle shift, but it changes how you build, test, and deploy software.

I collected some of the lessons I’ve learned while building AI-powered applications and learning evals.

https://kig.re/2026/06/22/writing-evals-for-ai-powered-apps.html

Curious how others are approaching evaluation and regression testing in production AI systems.

And here is the summary of most recent research on the subject so you don't have to :-)

https://kig.re/2026/06/22/ai-digest-on-evals.html