DEV Community

Cover image for Building on three week old models looks like gambling. It's the strictest engineering I do
Justin Bartak for Orbyt Labs

Posted on Originally published at orbytlabs.ai AI-assisted

Building on three week old models looks like gambling. It's the strictest engineering I do

Orbyt went from first commit to launch in 32 days with 243,000 lines and 4,124 tests, then kept getting rebuilt on models that had shipped only weeks earlier.

  • Published benchmarks measure yesterday's questions on everyone else's problems. My own eval suite, scored mechanically and run on every release, rules on a new model within hours: helps, hurts, or nothing.
  • Of 98 recorded incidents, 74 are now mechanized into permanent checks that no future release gets to un-learn. A finding that stays in someone's head restarts from zero on the next release.
  • My gates discarded 30 of the last 51 agent runs at the boundary. Every one was written down with its conditions, because an unrecorded dead end is a tax you pay annually.

Read the full article: Frontier Development Is a Discipline.

This summary was written with AI assistance (Claude) from the full article at https://www.orbytlabs.ai/blog/frontier-development.

Top comments (0)