AI Accuracy Is Not the Problem. Decision Risk Is.

Why production AI systems need more than accurate models, and how to build automation that remains reliable when predictions are wrong.

There has been a lot of discussion about comparing Jev with traditional large language models (LLMs). Most comparisons focus on which model gives more accurate answers.

That is a useful test. But I think we're paying too much attention to the model and not enough to the system we're building around it.

Jev is interesting because it is designed for structured decisions. Instead of generating a long text response and asking developers to interpret it, it can return defined choices, probability distributions and confidence scores.

For some applications, this could make decision workflows simpler, faster or cheaper. Those benefits still need to be measured against alternatives using the same tasks, hardware or API conditions, and costs.

But both Jev and general-purpose LLMs can be used in production.

The bigger question is not which one wins a classification benchmark.

It's what happens when the model makes a wrong prediction.

1. A safer car doesn't guarantee a safe journey

Imagine comparing Volvo and Honda.

Suppose Volvo performs better in a particular crash-safety test. Does that mean every journey in a Volvo will be safer?

Not necessarily.

A crash test measures performance under specific conditions: impact speed, collision angle, vehicle configuration and test procedure.

Actual road safety depends on much more: speed limits, road design, traffic rules, driver behaviour, vehicle maintenance and emergency response.

A safer vehicle is valuable. But it is only one part of a wider safety system.

I think AI should be evaluated the same way.

A classification benchmark is like a controlled crash test. It tells us something important about the model under defined conditions.

It does not tell us whether the entire automation will be reliable under every condition it encounters in production.

This distinction is reflected in the NIST AI Risk Management Framework, which treats model accuracy as one part of overall system trustworthiness. Reliability also involves robustness, security, monitoring and the ability to respond to failures.

2. Why 99% accuracy can still be a problem

Imagine an AI system that categorizes 10,000 incoming emails every month.

Its classification accuracy is 99%.

That sounds excellent.

But 1% incorrect means approximately 100 emails are misclassified each month.

Now consider what those mistakes involve.

Mistake A: A promotional newsletter is classified as a customer support request.

An employee wastes a minute checking it.

Mistake B: A legal notice requiring a response within a fixed deadline is classified as an ordinary newsletter.

The email gets archived and nobody sees it.

Both count as one mistake in the accuracy metric.

But their consequences are completely different.

The cost of Mistake B could be far greater than the cost of Mistake A.

This is why optimizing only for overall accuracy can be misleading.

Google's official classification guidance gives a particularly clear example: if only 1% of cases belong to the positive class, a model that always predicts negative can reach 99% accuracy while failing to identify a single positive case.

The useful metrics depend on the problem.

For legal emails, the critical question might be:

Of all the emails that really required legal review, how many did the system successfully identify?

That is recall.

If 100 legal notices arrive and the system catches 98, legal-notice recall is 98%.

But the other two notices could still cause serious problems.

So even recall isn't enough to describe the whole system.

We also need to measure what happens after classification.

3. Probability alone doesn't tell us what action to take

Suppose a model returns these predictions:

Email	Probability of requiring attention	Consequence if missed
Routine customer enquiry	95%	Low
Potential legal deadline	30%	High

Should the system prioritize the first email simply because its predicted probability is higher?

Not automatically.

A low-probability event with serious consequences may deserve more attention than a high-probability event with minor consequences.

For a simplified case, we can calculate:

Expected loss = Probability of an event × Cost if the event occurs

If an event has a 10% probability and could cause £10,000 of damage, its expected loss is £1,000.

If another event has a 90% probability and could cost £10, its expected loss is £9.

These numbers are illustrations, not reliable estimates of actual legal or business costs.

And there is another problem: the model's probability may itself be wrong.

Even a well-calibrated model can make mistakes on individual examples or perform poorly when it encounters unfamiliar inputs.

This is why the NIST framework recommends considering the severity of different failures, rather than treating all prediction errors equally.

Instead of asking only:

"How confident is the AI?"

We should also ask:

"What action is it allowed to take, and what could happen if its prediction is wrong?"

4. Let's build an email automation system

Imagine a company has two departments:

• Customer Support (CS), which handles routine questions, product issues and refund requests.
• Legal, which handles contracts, legal notices, disputes and regulatory correspondence.

The company wants AI to process incoming emails, route them to the right department and eventually automate some responses.

A simple implementation might look like this:

Incoming email → AI classification → Send reply or archive

I would not build it this way.

Instead, I would separate the process into four parts:

Incoming email → AI prediction → Policy checks → Approved action

There should also be a human-review route, a record of every decision and a way to recover from mistakes.

The model's job is to describe the email.

The policy engine's job is to decide which actions are permitted.

These are different responsibilities.

Step 1: Define a small set of categories

Start with only the categories needed for the first useful workflow.

For example:

Category	Description
cs_general	Routine customer enquiries
cs_refund	Refunds, payments or account-related requests
legal	Legal notices, contracts, claims and disputes
unknown	Anything that does not clearly fit or requires additional review

The last category is essential.

Never force the model to choose a known category when the correct answer may be unknown.

Suppose an email says:

"We are contacting you regarding a matter that may require formal action. Please review the attached documents."

It might be a legal notice, a customer complaint, a scam or something else.

If the model must choose between cs_general, cs_refund and legal, it will select one even if none is clearly supported.

An unknown category gives the system a way to avoid premature decisions.

But adding unknown as a label is not enough. The application must also permit abstention when probabilities are weak, required information is missing or the email appears outside the supported workflow.

This idea has an established research basis. In machine learning, it is called classification with a reject option or selective classification.

A 2019 ICML paper on SelectiveNet studied how models can trade off the proportion of examples they classify against the risk of incorrect predictions.

The practical lesson is straightforward:

A model doesn't have to automate every input to be useful.

Step 2: Separate classification from action

Now imagine the AI produces this structured output:

{
  "category": "legal",
  "probability": 0.82,
  "requires_review": true,
  "reasons": [
    "Mentions contractual obligations",
    "References a response deadline"
  ]
}

The model has described what it believes.

It has not decided whether to archive, forward, reply or take another action.

A separate policy engine evaluates the result against business rules.

For example:

Condition	Approved action
Clear routine CS enquiry	Categorize and prepare a draft
CS enquiry eligible for an approved standard response	Auto-send only after separate validation
Refund request	Draft response; require human approval
Possible legal correspondence	Preserve and route for legal review
Unknown or ambiguous email	Keep visible; request review
AI processing failure	Preserve email and alert the operator

Notice that a legal email is not automatically answered even if the AI assigns it a 99% probability.

Its classification probability does not authorize the action.

Similarly, a refund request isn't automatically approved because the model is confident that the customer wants a refund.

Knowing what a user wants is not the same as having permission to do it.

This separation also protects against another common problem: malicious instructions embedded in emails.

The OWASP Top 10 for LLM Applications (2025) describes how giving AI agents unnecessary permissions can turn unexpected or manipulated model outputs into harmful actions.

An incoming email is untrusted content. It should never be able to authorize the software to access unrelated private information, transfer funds or send messages outside its permitted workflow.

Step 3: Don't rely on the classifier to enforce every safety rule

This is an important detail.

Suppose the classifier accidentally labels a legal email as cs_general, with 97% probability.

If the policy engine simply checks:

IF category = cs_general
AND probability > 0.95
THEN send automatically

the system might still send an inappropriate response.

We separated classification from policy, but our policy depends entirely on the same potentially wrong prediction.

We have not solved the problem.

To reduce this risk, some checks must operate independently of the model's chosen category.

For example:

• Check whether the sender belongs to a verified list of legal contacts.
• Detect configured legal case references or active disputes.
• Check whether the email is part of an existing legal thread.
• Restrict automatic sending to verified, low-risk reply templates.
• Block automation when there are attachments or conditions the workflow doesn't support.

These checks will not detect every possible legal email. They are additional controls, not guarantees.

For high-risk actions, approval and restricted permissions remain necessary.

A policy engine should not simply be a second AI model agreeing with the first. It needs enforceable rules and permissions that a mistaken prediction cannot override.

5. Start in shadow mode, not full automation

This is how I would actually develop the product.

The first version should not reply to customers, archive messages or modify anything.

It should only observe.

For each incoming email, the system records:

1. Predicted category and probability.
2. Whether the model selected unknown or requested review.
3. Which action the policy engine would have permitted.
4. What the employee actually did.
5. Whether the proposed action would have been appropriate.

This is shadow mode.

The AI operates alongside the existing human workflow, but its proposed actions are not executed.

Consider a hypothetical first batch of 1,000 emails:

Result	Number
Correctly classified	900
Incorrectly classified	50
Sent to unknown / review	50

Here, accuracy among the 950 classified emails would be approximately 94.7%.

Coverage—the fraction receiving a known classification—would be 95%.

But those numbers alone shouldn't determine whether we deploy.

We should inspect the 50 incorrect classifications.

Did the model mix up two routine customer enquiries?

Or did it miss five legal notices?

Those are very different outcomes.

We should also inspect the unknown emails and check whether the review process handled them correctly.

One important detail: simply counting human corrections is not enough. Employees may overlook errors themselves. Randomly sample apparently correct predictions and have them independently reviewed, including examples the model marked as low-risk.

The NIST AI RMF Playbook recommends comparing model behaviour against human baselines, collecting feedback and measuring how often people override AI decisions.

This is more useful than reporting a single accuracy number.

6. Expand automation one action at a time

After shadow testing, I would not immediately enable full email automation.

I would start with one narrow workflow.

Version 1: Categorization only

Allow the system to add internal labels to routine CS emails. Keep original emails accessible and make classifications reversible.

Version 2: Drafting

Let the AI draft responses for a small set of routine questions. A human reviews every message before sending.

Version 3: Limited automatic replies

Allow sending only for clearly defined, verified, low-risk cases—for example, acknowledging receipt of a support ticket with an approved template.

Version 4: Wider coverage

Add more categories and actions only after enough real-world examples have been reviewed and their error rates are acceptable.

There is no universal rule that says 95%, 98% or 99% confidence is sufficient for automation.

The required evidence depends on the particular action and the cost of mistakes.

This is also consistent with TypeSafe's confidence documentation. TypeSafe recommends different thresholds depending on the risk of the action, along with human review or fallback behaviour when uncertainty is high.

The model is one input to that decision—not the whole decision.

7. Measure system performance, not just model performance

Once automation is running, I would track six things.

Metric	What it tells us
Classification accuracy	How often known classifications are correct
Critical-case recall	How many genuinely important emails are identified
Unknown / review rate	How frequently the system declines to decide
Automation coverage	What proportion of emails receive an automatic action
Harmful action rate	How often automation causes an incorrect or consequential action
Recovery time	How quickly errors can be detected and corrected

I would also track costs, processing time and human-review workload.

The target isn't simply to maximize automation coverage.

A system that automates 60% of routine emails reliably may be more useful than one that automates 95% but introduces serious operational risk.

There is an important trade-off here: sending too many emails to human review also has a cost. It increases workload and may delay genuinely urgent messages.

The objective is to find the right balance using real data.

For instance, when reviewing 1,000 emails, a business might find that 400 belong to a narrow, repeatable support category.

If that workflow is sufficiently reliable, automating just those 400 cases could already deliver meaningful value.

There may be no reason to automate the remaining 600 immediately.

8. Use feedback to expand the system carefully

Every correction should improve our understanding of where the system works and where it fails.

Suppose a new type of email appears regularly:

"We need to update the data processing agreement for our enterprise contract."

Initially, the system routes these emails to unknown.

After reviewing enough examples, the team discovers that these messages form a distinct workflow that needs legal or compliance review.

Now there is evidence for adding a new category.

That is a better reason to expand the classifier than inventing 30 categories before the product has processed a single real email.

The development loop becomes:

Start small → Observe → Review mistakes → Update rules or models → Test again → Expand

Feedback can lead to several different improvements.

Sometimes the problem is the prompt. Sometimes it is the model. Sometimes the categories are poorly defined.

And sometimes the classification is correct, but the policy allows the wrong action.

These failures should be diagnosed separately.

Google's production ML guidance emphasizes monitoring changes in input data, predictions and production quality.

The NIST risk-management framework similarly recommends ongoing monitoring, user feedback, override mechanisms and recovery plans.

One more rule is important: don't automatically turn every user correction into a new production rule.

Corrections should be reviewed, tested against previous examples and evaluated on separate data before deployment. Otherwise, fixing one mistake can introduce several new ones.

9. What does the actual technical implementation look like?

The first version doesn't require a complicated agent framework.

For Gmail, a small backend can use the Gmail API's push notification mechanism to discover mailbox changes.

A queue processes new messages, stores the original identifiers and passes the required information to the model.

The model produces a structured prediction.

The policy engine checks that prediction against the company's rules and permissions.

A separate executor performs only approved actions.

Every decision is logged.

A minimal architecture looks like this:

Gmail API
   |
   v
Message Queue
   |
   v
Email Processor
   |
   v
AI Classifier
   |
   +--> Known category
   |        |
   |        v
   |    Policy Engine
   |        |
   |        +--> Approved safe action
   |        |
   |        +--> Human review
   |
   +--> Unknown / unsupported
            |
            v
        Human review

All paths --> Decision log --> Feedback

During shadow mode, the executor records proposed actions but does not perform them.

When live actions are enabled, access should be limited to exactly what each workflow requires. The classifier should not receive broad mailbox-writing permissions merely to classify incoming messages.

The system also needs basic operational safeguards: preventing duplicate actions, retrying failed processing, checking for missed mailbox notifications and providing a way to disable automation quickly.

For example, Google's Gmail documentation explicitly notes that push notifications can occasionally be delayed or dropped, so applications should also have a way to resynchronize mailbox history.

These details may sound less exciting than choosing an AI model, but they are part of what makes a production service dependable.

10. The real measure of a production AI system

I still think comparing Jev and LLM accuracy is valuable.

Better models can reduce errors. Faster and cheaper models can make automation economically practical. More useful probability outputs can help developers decide when to request human review.

But none of these properties removes the need for proper system design.

A model can be highly accurate while the application built around it remains fragile.

The right engineering question is not only:

"How often is this model correct?"

It is also:

"When it is wrong, what happens next?"

Can the system recognize when it doesn't have enough information?

Can it decline to make a decision?

Can it prevent an incorrect prediction from triggering an expensive action?

Can people review and correct its decisions?

Can the system improve using real operational feedback?

These questions matter more as AI moves from generating suggestions to taking actions on behalf of people and businesses.

A better model makes fewer mistakes. A better system limits what those mistakes can do.

That is where I think production AI engineering should focus.

---

References

1. NIST — AI Risk Management Framework 1.0 (2023). Official guidance on trustworthiness, reliability, safety and system-level risk.
   
   https://doi.org/10.6028/NIST.AI.100-1

2. Google — Classification: Accuracy, Recall, Precision, and Related Metrics. Explains why accuracy can be misleading and how to choose evaluation metrics based on the cost of mistakes.
   
   https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall

3. TypeSafe — Confidence. Documents probability distributions, confidence calculations and risk-dependent decision thresholds.
   
   https://docs.typesafe.ai/confidence

4. OWASP — LLM06:2025 Excessive Agency. Explains the risks of giving AI systems excessive permissions and autonomy.
   
   https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

5. Geifman & El-Yaniv (2019) — SelectiveNet: A Deep Neural Network with an Integrated Reject Option. ICML research on allowing models to decline uncertain predictions.
   
   https://proceedings.mlr.press/v97/geifman19a.html

6. NIST — AI RMF Playbook: Measure. Guidance on evaluation, human feedback and monitoring.
   
   https://airc.nist.gov/airmf-resources/playbook/measure/

7. Google — Productionization of ML Systems. Guidance on logging, monitoring and changes in production model behaviour.
   
   https://developers.google.com/machine-learning/managing-ml-projects/production

8. Google — Gmail API Push Notifications. Official documentation for implementing mailbox event processing.
   
   https://developers.google.com/workspace/gmail/api/guides/push