Originally published on AI Tech Connect.
What you need to know Accuracy is a single-sample estimate. Reliability is a distribution. One number cannot describe the other. Consistency is measured with pass^k — the probability that all k independent runs of the same task succeed. It falls as k grows, which is exactly what an unsupervised agent experiences. Robustness is measured by perturbing inputs into semantically equivalent variants and watching the success rate move. Users do not speak in your canonical phrasing. Fault tolerance is measured by injecting tool and API failures deliberately. Rate limiting is the most damaging class and the one you will meet first. All three fit in CI if you gate on the aggregate metric against a recorded baseline rather than on individual runs. As of August 2026 this framing is becoming standard…
Top comments (0)