DEV Community: dreamyman

Outsourcing AI Projects? Don't Settle for Model Accuracy—Write These 4 Metric Categories Into Your Contract

dreamyman — Sat, 07 Nov 2026 04:37:24 +0000

Writing "Model accuracy shall not fall below the agreed threshold" in an AI outsourcing contract is virtually equivalent to having no acceptance criteria at all.

That sounds harsh, but the reality is straightforward: according to a RAND analysis of over 2,400 enterprise AI projects, 80.3% of projects failed to deliver their expected business value. Technical metrics pass inspection while business implementations fail—this happens all the time.

My view: model accuracy belongs in internal vendor testing and pre-launch evaluations. It should never serve as the primary acceptance criteria in a vendor contract. Contracts for outsourced AI projects must be built around four distinct categories of metrics: Business Outcomes, Scope Boundaries, Latency & Availability, and Error Handling & Retraining Liabilities.

Here is how to structure each one.

Model accuracy simply measures the percentage of predictions a model gets right. It tells you nothing about where errors occur or how expensive those errors are to your operation.

Imagine a manufacturing company outsourcing a visual quality control AI. The contract specifies: "Average accuracy on the test dataset shall meet vendor commitments." Once deployed, strong ambient light reflections on the shop floor trigger a wave of false alarms, flagging good parts as defective. To keep the line moving, operators bypass the system entirely, letting real defects slip through to customers. When confronted, the vendor points to the contract: the test set accuracy was met.

This example doesn't disprove the value of accuracy; it illustrates how aggregated test scores mask error distributions and real-world operational costs.

A secondary issue is data quality. Gartner research attributes 85% of AI project failures to poor data quality—issues that rarely show up on curated test sets and only surface after deployment. If your contract omits business outcomes and post-launch retraining liabilities, you will end up paying for a high model score that fails in production.

1. Business Outcomes: Stop Accepting Test Set Scores

Business outcome metrics must be drawn directly from your operational systems, not from static test datasets. Test sets are exam questions prepared by the vendor; your business systems represent the unpredictable reality of daily operations.

Take a mid-sized manufacturer outsourcing visual quality inspection as an example. Instead of model accuracy, the contract should specify:

Within an agreed post-launch evaluation window, the defect escape rate (missed defects) on Line A shall not exceed X%.
Manual review volume shall not exceed Y units per shift.
Production line downtime caused by false positives shall not exceed Z minutes.

The baseline numbers for X, Y, and Z should be measured from your existing operational data before project kickoff—never guessed off the top of your head.

If you are outsourcing an AI customer support system for cross-border e-commerce, relevant metrics might include:

The error rate in extracting buyer inquiry details.
The percentage of incoming orders requiring human takeover.
The frequency of repeated questions for the same issue.

The key is writing exact metric definitions, measurement methodologies, data sources, and evaluation windows directly into the contract.

Focusing on business outcomes also prevents the "Pilot Success, Scale Failure" trap—a common pattern analyzed in detail in AI Pilots Always Succeed, But Scale Fails: What's Missing Isn't the Model. Without concrete business metrics in the agreement, final acceptance easily devolves into vendor slide decks in a conference room.

2. Scope Boundaries: Lock Down the Perimeter

The most common source of friction in AI outsourcing isn't complete model failure—it is the scope dispute: "We never agreed to cover that scenario."

A model might perform impeccably on Assembly Line A, only to break down on Line B due to different lighting. An AI assistant might handle Chinese and English inquiries well, then garble Italian ones. Contracts that only specify overall accuracy implicitly assume complete coverage across all scenarios when none exists.

Your contract must explicitly define scope boundaries, including at least:

Product models or SKUs
Specific defect types or ticket categories
Supported languages and communication channels
Data input conditions (e.g., image resolution, lighting levels)
Hardware models or machine specs
Permissible operator behavior boundaries

The contract must also specify what happens when operational realities exceed these bounds. If a new defect type appears post-launch, is the vendor obligated to update the model under the existing agreement? If left unsaid, vendors will almost certainly quote it as a brand-new custom project. Your contract should explicitly distinguish between "in-scope optimization" and "out-of-scope additions", setting response times and costs for each.

Industry leaders already treat scenario adaptation capability as a core delivery benchmark. In September 2025, Alibaba Cloud earned the highest level (Level 5) certification for enterprise LLM delivery from the China Academy of Information and Communications Technology (CAICT), evaluating 34 core capabilities across requirements analysis, fine-tuning, scenario adaptation, and ongoing maintenance. Regardless of vendor size, demanding clearly defined scenario boundaries is a basic sanity check of vendor delivery maturity.

3. Latency & Availability: Don't Be Fooled by 99.9% Uptime

Many AI contracts include standard SLA language like "System availability shall not fall below 99.9%." While 99.9% sounds impressive, it offers very little protection for AI workloads.

Consider the public SLA for Azure OpenAI. While it guarantees 99.9% monthly availability, it strictly measures whether the API returned a response—not how fast that response arrived. Furthermore, the first 43.2 minutes of monthly downtime trigger zero financial credits. Beyond that threshold, credits are capped at 10% of the monthly bill, reaching 25% only if total downtime exceeds 7 hours and 12 minutes in a single month.

In other words, the issues that actually cripple business operations—high latency, rate-limiting, and severe queueing—often carry zero vendor financial penalty.

When mid-sized businesses implement AI, they typically rely on an integrator or contractor who routes requests to upstream cloud LLM providers. Uncovered latency risks from upstream vendors must be contractually absorbed by your integration partner or accounted for in pricing.

Define "Effective Availability" separately in your contract:

Latency Thresholds: Mandate that p95 response time shall not exceed X seconds (p95 latency is a percentile metric that prevents extreme delays from being hidden by average figures). If the threshold is exceeded continuously for an agreed duration, it counts as a performance failure.
Capacity & Concurrency: Explicitly define maximum supported concurrent requests, queueing logic, and graceful degradation plans during peak loads.
Financial Remedies: Ensure penalties are structured as direct cash refunds or invoice deductions rather than service credits restricted to future spend.

Latency isn't a minor technical detail; it directly impacts revenue. If a quality control AI goes from responding within seconds to keeping the line waiting, the line may not be able to wait for it, and the whole system may end up bypassed.

4. Error Handling & Retraining Liabilities: Clarify Ongoing Ownership

Once deployed, AI models inevitably encounter fresh data distributions and edge-case errors. Yet, small and mid-sized businesses rarely specify in their contracts what happens when a model makes mistakes after sign-off. As a result, the day after acceptance, every required tweak is billed as a paid change order.

Your contract should explicitly cover four key governance points:

Error Severity Classification:
- P0 (Critical): Safety incidents, legal or compliance violations, or significant customer loss.
- P1 (Major): Errors in core functionality.
- P2 (Minor): Cosmetic or low-impact anomalies. Set distinct response and resolution SLAs for each tier.
Data Drift Monitoring: Over time, operational data deviates from training sets, causing accuracy degradation. Specify which party is responsible for monitoring metric drift and how frequently performance reports must be delivered.
Retraining Triggers: When error rates cross agreed thresholds, or when new languages or defect types are added, require the vendor to deliver a remediation plan within a set number of business days. State clearly whether retraining costs are included in the initial contract or first-year maintenance agreement.
Data & Artifact Ownership: Ensure full ownership rights to training datasets, custom annotations, prompt templates, and evaluation sets. Confirm that these assets can be exported without lock-in if the contract terminates.

If your project involves an enterprise knowledge base or RAG (Retrieval-Augmented Generation) system, governance and update mechanics are even more critical. As noted in Knowledge Base Governance: Three Essential Mechanisms, a knowledge base doesn't stay alive just because it was built: it needs write incentives, clear ownership, and expiration rules. Failing to contractually mandate these processes means paying for a static system destined to fail.

When Does This Framework Not Apply?

Standardized SaaS Products: If you are buying an off-the-shelf SaaS tool rather than commissioning custom outsourcing, contract negotiation leverage is limited. However, you can still use these four categories to evaluate vendor suitability: Is the scope clearly bounded? Does the SLA cover latency? Is there a clear escalation path for performance regression?
Early-Stage Feasibility Pilots: For small, low-budget experimental pilots, demanding strict business outcome SLAs may be impractical. The goal of a pilot is testing feasibility. The contract should instead focus on pilot scope boundaries, fallback conditions, and explicit constraints preventing unvetted rollout to production.

An Unresolved Operational Challenge

Many SMBs lack historical baseline data and cannot quantify operational baselines when drafting the contract. You are left with two choices: pay upfront for an initial data audit, or insert a clause stating that baselines will be established within an agreed window post-launch.

I lean toward the latter, but it leaves the vendor room to argue, so it may need to be constrained through payment milestones.

This article doesn't provide a contract template. But the core logic behind these four metric categories is simple: Acceptance testing should prove that your business problem was solved—not just that a vendor's model looks good on paper.

References

Originally published on pierceliu.top.

Put an AI agent in one process and day one gives you two walls

dreamyman — Fri, 06 Nov 2026 08:13:48 +0000

Picture an ordinary request: have an AI pull data from thirty websites, clean it up, and turn it into a report. This is not a chat — one question, one answer, done. It runs for two hours straight, downloads files, eats memory, and executes code the AI writes for itself along the way.

Getting that working on your own laptop takes an afternoon. Turning it into a service several people use at once is where the difficulty starts.

This opens a six-part series on building an agent sandbox from scratch. Each post fixes one problem the previous one exposed, and the code lives in agent-sandbox-oss so you can run it yourself. Terms get explained where they first appear; no Kubernetes background required.

One thing about the code: it is a teaching skeleton. Real crawling is replaced by "sleep for a bit", and the cluster runs on a laptop. It is not meant to ship. Its value is that every crash in it is a crash real projects have.

The obvious way to write it, and the assumption hiding inside

The natural implementation: a web endpoint takes a task, starts working in the background, and keeps the progress in memory.

IN_MEMORY_DB = {}   # task progress lives here

async def run_data_research_task(task_id, target_pages, simulate_leak=False):
    for i in range(target_pages):
        await asyncio.sleep(0.5)                      # pretending to fetch a page
        IN_MEMORY_DB[task_id]["pages_crawled"] += 1   # note it down: one more page

Deployment is equally natural: one program, one container, a 128MB memory ceiling.

A container is best thought of as a disposable little server: the program and everything it needs, packed together, ready to start or destroy at any moment. The memory ceiling is the line drawn around it — cross it and the system shuts it down.

Nothing here is wrong. It runs, returns task IDs, reports progress. What it does is quietly assume that every task behaves.

In the real world it does not. Among thirty websites one page is always enormous, and AI-generated code eventually writes an infinite loop. The next two sections are what happens when that assumption breaks.

Wall one: one task goes down, everybody goes with it

The code has a switch that simulates hitting a giant page — every fetched page appends 20MB to memory. On the seventh page memory crosses the 128MB line and the system shuts the whole program down.

In Kubernetes (a system for running containers, K8s from here on) you can watch it happen:

kubectl get pod -l app=agent -w
# NAME                 READY   STATUS      RESTARTS
# agent-monolith-xxx   1/1     Running     0
# agent-monolith-xxx   0/1     OOMKilled   0      ← over the memory line, killed
# agent-monolith-xxx   1/1     Running     1      ← restarted seconds later, looks fine

OOMKilled means "used too much memory, terminated". Watching it restart on its own, the natural reaction is relief: the system healed itself.

Then you ask about a different task that was running at the same time — the well-behaved one, quietly fetching small pages:

curl localhost:8000/status/<some-other-task-id>
# {"detail":"Task not found. Data lost due to Pod crash!"}

It is gone too. An hour and forty minutes of progress, back to zero.

What got killed was not "the task that misbehaved" — it was the program holding all of the tasks. An analogy: ten people share a flat with one kitchen, one of them burns a pan and sets off the sprinklers. The sprinklers soak the whole flat. Everybody's dinner is ruined.

There is a worse detail. Because progress only existed in memory, after the restart you cannot even find out which page it had reached. K8s restarts protect whether the service can be reached, not how far the work got. It reopens the door; it does not give back your two hours.

I call it the monolith illusion: RESTARTS: 1 looks like a harmless number, and it costs every task that was running at the time.

Wall two: an agent executes code the AI just wrote

This is where agents differ most from ordinary backend services. An ordinary service runs code you wrote, reviewed and tested. An agent runs code a model produced seconds ago.

And whatever that program can reach is fully visible from inside the container:

cat /var/run/secrets/kubernetes.io/serviceaccount/token

That command reads the access credential the platform mounted for the program — effectively a key. One program means one shared set of files, one shared set of environment variables, one shared key. So code generated for task A can read the data task B collected, and the key sitting next to it.

No amount of care while coding fixes this. Inside one process there is no boundary — the operating system simply does not offer "parts of the same process, protected from each other".

What this step settles

The naive version is worth running by hand, because it disproves two very natural ideas at once:

"just give it more memory" — a 1GB ceiling only postpones the crash by a few dozen pages; the collateral damage is identical;
"just catch the error" — the kill comes from the operating system kernel, and the program never gets a chance to say a word.

Which leaves one road: give every task a container of its own. Sounds obvious enough.

The bill arrives in the next post: creating a container on demand leaves the user waiting 45 to 90 seconds.

In one line: inside one process there is no boundary. Isolation is not a performance optimisation, it is the precondition for offering this kind of service at all.

Code: agent-sandbox-oss/lab1 — a local cluster and one command is all it takes.

Originally published on pierceliu.top.