What 25 Years of Deterministic Software Engineering Taught Me About Building AI Systems

Konstantin Gredeskoul — Tue, 23 Jun 2026 16:49:46 +0000

One of the strangest things about AI engineering is that your test suite can be 100% green while your product is getting worse.

Traditional software taught us to think in absolutes:

AI systems force us to think in distributions, thresholds, confidence intervals, and trade-offs.

It’s a subtle shift, but it changes how you build, test, and deploy software.

I collected some of the lessons I’ve learned while building AI-powered applications and learning evals.

Curious how others are approaching evaluation and regression testing in production AI systems.

And here is the summary of most recent research on the subject so you don't have to :-)

DEV Community: Konstantin Gredeskoul