When your organisation funds AI work today, much of the budget and attention tends to go to AI engineering: assembling foundation models, retrieval, tools, and agents into products. That's usually the right place to start, because it shows quickly what a system can do. Once a system has to make a measurable decision in production, though, a different set of questions starts to matter. What counts as a good result? Which cases does the system get wrong? Do the numbers in the status report mean what everyone assumes they mean?
Those questions are the core of ML engineering. AI engineering is incomplete without ML engineering's habits, especially evaluation, and programmes that treat ML engineering as a first-class part of AI engineering are better placed to ship systems that hold up. That matters most at the handoffs, where an ML model's output feeds a generative AI system or a generative model's output feeds ML. This post sets out why, using five learning outcomes from a recent advisory engagement on a production ML system.
Two practices, one system
Chip Huyen's account of the AI engineering stack puts the difference well. AI engineering "differs from ML engineering in that it's less about model development, and more about adapting and evaluating models". Because foundation models can produce open-ended outputs that are harder to evaluate, she argues, "[t]his makes evaluation a much bigger problem in AI engineering" (Orosz and Huyen, 2025). That point supports the argument here: the evaluation discipline ML engineering built up is what AI engineering now needs more of. The same excerpt notes that foundation models let teams build the product first and invest in data and models once the product shows promise. That ordering is a strength, and it also means the evaluation work tends to arrive late, sometimes after commitments have already been made on the strength of a demo.
| AI engineering | ML engineering | |
|---|---|---|
| Starting point | Pre-trained foundation models and APIs | A specific task, its data, and its success criteria |
| Core activities | Composition, prompting, retrieval, tool use, and orchestration | Data and labels, evaluation design, training, calibration, deployment, and monitoring |
| Typical question | "Can we make the system do this?" | "How well does it do it, on which cases, and how do we know?" |
| Risk when it's the only practice | Impressive demos that don't hold up in production | Rigour applied to the wrong problem, or too slowly to matter |
Table 1. AI engineering and ML engineering compared.
The two practices meet at handoffs. Figure 1 shows the two that come up most: an ML model's scores, with a threshold the business has chosen, passed to a generative AI system that acts on them, and generated labels passed back into the data an ML model learns from. Both sides are measured against what ML engineering defines as good, and production feedback flows back into the data and the evaluation.
The engagement
The engagement was advisory, covering review, evaluation design, analysis, and recommendations for a production ML system, with the models built and trained by the system's owners. The system's output feeds an automated downstream process and is shown to end users. What stood out across the engagement was how many of the most important shifts in understanding came from evaluation and from tracing the system end to end, more than from changes to the model.
Lesson 1: a lenient headline metric can hide the real problem
Early reporting led with a lenient measure that gave credit for near misses: an output counted as correct if it partly matched its target. On that measure the model looked strong. Stricter measures already existed in the tooling but weren't the ones in the headline report.
Scored with one-to-one matching and a strict standard for what counts as correct, the same outputs told a different story, and the gap persisted across later models. Strict measures were added to the reporting. A strict standard can be harsh in its own way, so it helps to report both kinds of measure, choose thresholds deliberately, and know which question each number answers.
Evaluation design belongs to the engineering work, and it deserves the same scrutiny as the model. A single lenient metric can point effort at the wrong problem. In this case, tracing the outputs end to end (lesson 2) then showed that part of the gap sat in how outputs were exported, outside the model.
Lesson 2: two consumers, two outputs
Following the outputs from the model to the places they're used showed that exported results carried a fixed safety margin. Under the strict measure, that margin cost a lot of score, and a simulation on existing outputs showed that removing it would sharply improve the score with no retraining.
The margin met a real requirement: the downstream step needs it so that it still acts on the whole target when the model is slightly off. Removing it would have lifted the metric while reducing the coverage that step relies on. The proposal was two outputs for two consumers: precise outputs for scoring and display, and conservative outputs for the downstream step.
Before you change anything to improve a number, find out how each output is consumed. A change that lifts a metric can work against the product, and the people who run the downstream process often know why a design detail exists.
Lesson 3: measure what the user experiences
End users mainly see the result of the downstream step and see the model's raw outputs only indirectly, so what they experience is a better basis for acceptance. Building on that, the proposal was for acceptance measures tied to user outcomes: how completely the downstream step covers what it should, and how much it acts on beyond that.
Framed that way, the confidence threshold becomes a product decision between two kinds of error. A trade-off curve lets the business see what it gains and gives up at each setting and choose an operating point with that in view. The outcome measures would sit alongside the model-level measures that track the model. Each kind of measure answers a different question, and a release review benefits from both.
Operating points are business decisions. Part of the ML engineering job is to make the trade-off visible to the people who own that decision and leave the choice with them.
Lesson 4: an agreed gate turns opinions into experiments
An acceptance gate had been agreed, combining a recall measure, a strict accuracy measure, and a cap on false alarms, which puts the point from lesson 1 into practice.
Having the gate written down changed how decisions were made. Ideas were tested against the same measures, so debates became experiments. Experiments showed that several proposed approaches needed to be reworked, and with the gate agreed in advance, that was a normal outcome.
Lesson 5: verify the verifier, including with AI
The analysis work ran under a fixed review protocol, with key numbers re-derived independently from the raw outputs and an adversarial review of the main findings.
The protocol still let an error through. The review had run on an earlier draft, and the text that was shared differed from it: a statement about a fix held for some cases and not others. A blind review caught it, and the shared document and the accompanying message were corrected. The protocol now includes a blind review of the exact final version before anything is shared.
The same protocol had earlier surfaced existing code that, together with a data source already available, narrowed the scope of a proposed build. That helped turn the build into a proposal for a short, time-boxed test.
This is also where AI engineering and ML engineering meet in practice. AI agents are well suited to this kind of verification: re-deriving key numbers from raw outputs, measuring results against agreed outcome and acceptance measures, and giving a blind, adversarial review of a finding before it's shared. They can make verification faster and more thorough, and they can check more numbers than is practical by hand. Accountability for what is shared stays with people, and a verification protocol should assume errors will happen, including in the review itself, and check the exact artefact that goes out.
The benchmark and the training data need engineering too
Two smaller points from the engagement are worth adding, because both bear on the handoffs below.
A re-annotation substantially increased the number of labelled examples, likely in part because earlier labels were incomplete and in part because the labelling scope widened. That means some predictions scored as false alarms may have been correct. Benchmark label quality matters well beyond one project: a study of ten widely used test sets across image, text, and audio data estimated an average of at least 3.3% of test labels to be wrong, and the authors found that, on corrected labels, some lower-capacity models would outperform higher-capacity ones if the share of originally mislabelled test examples rose by about five to six percentage points (Northcutt, Athalye and Mueller, 2021). Treating a benchmark as a product, with versions and a changelog, makes it possible to say which version a result was measured on.
Runtime constraints made running several models at inference time impractical, so the chosen approach was a single model trained on the original labels plus labels fused from the outputs of several models. It performed about as well as a larger multi-model ensemble at the same false-alarm level, and a control experiment (the same model trained on the original labels alone) indicated that the gain came from the fused labels. Without that control run, the improvement could as easily have been credited to the architecture or to the training run itself. Labels produced by models are exactly what a generative AI system hands back to ML, which is where the next section picks up.
Handoffs between ML and generative AI
More systems now combine both kinds of model, and the handoffs between them are often where evaluation is thinnest. Each direction needs its own checks.
When ML outputs feed generative AI
A classifier, risk score, or extraction model often decides what a generative AI system sees or does: which requests an agent handles on its own, which documents reach a prompt, which cases go to an automated response. The ML model's threshold at that point is an operating point in the sense of lesson 3, and the agent inherits whatever errors sit above it. A few practices help at this boundary:
- Pass the score and the threshold along with the output, so the agent, and the person reviewing its work, can see how confident the upstream model was.
- Agree which cases go to a person, and treat that routing as part of the product.
- Measure the combined result. An ML model that passes its own gate can still feed an agent cases the agent handles badly.
One way to keep that decision visible is to declare it next to the action it gates. Arkhe, an open-source ontology language I maintain, takes that approach: a version-controlled file declares the entities an agent works with and the actions it may take, by whom, and with what audit trail, and compilers turn that into tool definitions for agent frameworks (Stevens, 2026). A threshold written as a guard on the agent's action sits in a file the business can review, and changing it goes through the same review as any other change.
Listing 1 shows the basic trade-off on ten invented cases. Raising the threshold from 0.5 to 0.9 sends fewer cases to the agent (90% falling to 30%) and more to people, and the error rate among the cases the agent handles falls from about a third to none. A sample this small is only an illustration; a real trade-off curve needs far more cases and should be measured on the combined system.
Python
# Invented scores for illustration only; not data from any system.
# Each case: (model score, whether the model's answer was actually correct).
CASES = [
(0.97, True), (0.93, True), (0.91, True), (0.86, False), (0.82, True),
(0.76, False), (0.71, True), (0.64, False), (0.58, True), (0.41, False),
]
def handoff_at(threshold: float, cases: list[tuple[float, bool]]) -> tuple[float, float]:
"""Cases at or above the threshold go to the agent; the rest go to a person.
Returns the share sent to the agent and the error rate among those cases."""
to_agent = [ok for score, ok in cases if score >= threshold]
share = len(to_agent) / len(cases)
error_rate = to_agent.count(False) / len(to_agent) if to_agent else 0.0
return share, error_rate
if __name__ == "__main__":
for t in (0.5, 0.7, 0.9):
share, err = handoff_at(t, CASES)
print(f"threshold {t:.1f}: {share:.0%} to the agent, {err:.0%} of those wrong")
Rust
// Invented scores for illustration only; not data from any system.
// Each case: (model score, whether the model's answer was actually correct).
const CASES: [(f64, bool); 10] = [
(0.97, true), (0.93, true), (0.91, true), (0.86, false), (0.82, true),
(0.76, false), (0.71, true), (0.64, false), (0.58, true), (0.41, false),
];
/// Cases at or above the threshold go to the agent; the rest go to a person.
/// Returns the share sent to the agent and the error rate among those cases.
fn handoff_at(threshold: f64, cases: &[(f64, bool)]) -> (f64, f64) {
let to_agent: Vec<bool> = cases
.iter()
.filter(|(score, _)| *score >= threshold)
.map(|(_, ok)| *ok)
.collect();
let share = to_agent.len() as f64 / cases.len() as f64;
let wrong = to_agent.iter().filter(|ok| !**ok).count();
let error_rate = if to_agent.is_empty() { 0.0 } else { wrong as f64 / to_agent.len() as f64 };
(share, error_rate)
}
fn main() {
for t in [0.5, 0.7, 0.9] {
let (share, err) = handoff_at(t, &CASES);
println!("threshold {t:.1}: {:.0}% to the agent, {:.0}% of those wrong", share * 100.0, err * 100.0);
}
}
Listing 1. The share of cases handed to an agent, and the error rate among them, at three thresholds on invented scores.
When generative AI feeds ML
The reverse handoff is increasingly common: labels generated by an LLM used to train a model, synthetic examples used to fill gaps in a dataset, or an LLM acting as a judge in an evaluation. These are model outputs, and they carry model errors into the data that everything else is measured against. The fused-label result above shows the pattern that helps. Check generated labels against a sample labelled by people before you rely on them, version them like any benchmark, and run a control experiment so you know whether they helped. An LLM judge needs the same treatment, with its agreement with people measured and tracked as models and prompts change.
What this means for agents
The disciplines above matter more once agents act on a model's outputs. A weak evaluation in a reporting pipeline produces a wrong number that someone may question. In an agentic system the same weakness can produce a wrong action, possibly before a person reviews it. Sculley et al. (2015) noted that "[a] model may directly influence the selection of its own future training data", and agents that act on the world, or generate the labels a model learns from, can widen that kind of feedback loop.
The learning outcomes above have direct counterparts in agent work. A task-success rate that counts partial completions behaves like a lenient headline metric. A tool output read by both an agent and a person has two consumers with different needs. The confidence level at which an agent acts without escalating to a person is an operating point, and the business should be the one choosing it.
Foundation models change where the effort goes. They don't remove the need for evaluation, data quality, or operating-point decisions, and EU law and US guidance point the same way. For high-risk AI systems, the EU AI Act will require "[t]he levels of accuracy and the relevant accuracy metrics" to be declared in their instructions of use once those obligations apply (European Parliament and Council, 2024, Article 15(3)), on a timetable the 2026 Digital Omnibus on AI pushed back (European Parliament and Council, 2026). In the US, the voluntary NIST AI Risk Management Framework asks that "AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s)" (National Institute of Standards and Technology, 2023, MEASURE 2.3). Both assume someone has decided which metrics matter, and the NIST text adds that they are demonstrated under conditions similar to deployment. That is the ML engineering question from Table 1.
On the governance side, the Trustworthy Agentic AI Blueprint and GATE, the open agent governance framework I maintain, describe wrapping the probabilistic model in a deterministic perimeter of governance, with controls at the tool boundary, where an agent's actions take effect. The evaluation work described here helps you know how far to trust what the model inside that perimeter produces.
Where to start
If you're about to scale an AI initiative, a useful first step is to ask whoever reports the headline metric how a correct result is defined, which benchmark version it was measured on, and who chose the operating point. Then list the handoffs between ML and generative AI in your system and ask who measures each one. If the answers are unclear, that is the work to fund before scaling.
Separately from the engagement above, if you'd like a second view on evaluation and production readiness for an AI or ML system, our Data & AI team works on machine learning in production and would be glad to talk it through.
Disclosure: GATE is authored and maintained by me personally rather than by Sakura Sky, and there is no paid tier, hosted version, or commercial product built on it. I co-authored the Trustworthy Agentic AI Blueprint with Sakura Sky. Arkhe (arkhelang) is also my personal open-source project, authored and maintained by me rather than by Sakura Sky, released under Apache-2.0, with no commercial product built on it. Agent governance is also my day job at Sakura Sky, which is a commercial interest worth stating.
References
European Parliament and Council, 2024. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Article 15: Accuracy, robustness and cybersecurity. AI Act Service Desk, European Commission. Available at: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-15 [Accessed 26 October 2026].
European Parliament and Council, 2026. Regulation (EU) 2026/1744 of the European Parliament and of the Council of 8 July 2026 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI). Official Journal of the European Union, L series, 24 July 2026. Available at: https://eur-lex.europa.eu/eli/reg/2026/1744/oj [Accessed 26 October 2026].
National Institute of Standards and Technology, 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Gaithersburg, MD: U.S. Department of Commerce. Available at: https://www.nist.gov/node/1731741 [Accessed 26 October 2026].
Northcutt, C.G., Athalye, A. and Mueller, J., 2021. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Track on Datasets and Benchmarks. Available at: https://arxiv.org/abs/2103.14749 [Accessed 26 October 2026].
Orosz, G. and Huyen, C., 2025. The AI Engineering Stack. The Pragmatic Engineer, 20 May. Available at: https://newsletter.pragmaticengineer.com/p/the-ai-engineering-stack [Accessed 26 October 2026].
Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F. and Dennison, D., 2015. Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NIPS 2015), pp. 2503-2511. Available at: https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf [Accessed 26 October 2026].
Stevens, A., 2026. Arkhe: An ontology language for AI systems. GitHub repository. Available at: https://github.com/arkhelang/arkhelang [Accessed 27 October 2026].
Top comments (0)