The separation between “caught before” and “caught during” is the useful metric here. Runtime refusal is necessary, but pre-execution effect visibility changes whether a human can approve a plan before any side effect begins. I also appreciate including the two misses and harmless controls; without them the benchmark would reward a language for flagging everything.
A next test I would love to see is indirect authority: a safe-looking function calling a dependency whose declared effect budget changes between versions.
Thank you that's the distinction I'd want people to take from it. On indirect authority: the ratchet already fails a widening in a Velaris dependency because the baseline comparison is transitive (check_ratchet.py has that case), but a version-to-version diff of what an upgrade gained doesn't exist yet. It was next on the list; your comment moved it up. I'll add a category to the benchmark for it and link back here when it ships.
Shipped in 7.1.1. Your test is benchmark category 12, "indirect authority": three programs where the caller is unchanged and only the dependency's declared budget widened — net, a new host inside net, fs:write — all caught before running, plus a control that isn't flagged. velaris deps-diff is the tool; the honest limit in THREAT_MODEL.md is that for a non-Velaris package it sees only install scripts and declared dependencies. github.com/gowrishankar-infra/vela... . Thank you
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The separation between “caught before” and “caught during” is the useful metric here. Runtime refusal is necessary, but pre-execution effect visibility changes whether a human can approve a plan before any side effect begins. I also appreciate including the two misses and harmless controls; without them the benchmark would reward a language for flagging everything.
A next test I would love to see is indirect authority: a safe-looking function calling a dependency whose declared effect budget changes between versions.
Thank you that's the distinction I'd want people to take from it. On indirect authority: the ratchet already fails a widening in a Velaris dependency because the baseline comparison is transitive (check_ratchet.py has that case), but a version-to-version diff of what an upgrade gained doesn't exist yet. It was next on the list; your comment moved it up. I'll add a category to the benchmark for it and link back here when it ships.
Shipped in 7.1.1. Your test is benchmark category 12, "indirect authority": three programs where the caller is unchanged and only the dependency's declared budget widened — net, a new host inside net, fs:write — all caught before running, plus a control that isn't flagged. velaris deps-diff is the tool; the honest limit in THREAT_MODEL.md is that for a non-Velaris package it sees only install scripts and declared dependencies. github.com/gowrishankar-infra/vela... . Thank you