A shadow evaluation protocol for ERP agents

An ERP assistant can produce a convincing replenishment recommendation while relying on information that was unavailable when the decision mattered. Evaluating it against today's database can quietly reward hindsight. Before giving the assistant purchasing permissions, evaluate what it would have recommended from a time-bounded view of the business.

The protocol below is a proposed engineering method, not a report of measured results. It focuses on recommendation quality; transaction safety and authorization still require separate checks.

Freeze the evidence at decision time

Choose one workflow, such as suggesting replenishment for stocked maintenance parts. For each evaluation case, record the decision timestamp, inventory snapshot, open orders, supplier quotes and policy version. Keep both the effective date of each fact and the time your system actually received it.

A delivery notice entered on Friday must not appear in a replay of Thursday's decision, even if its effective date is Wednesday. Exclude later invoice corrections, retrospectively updated forecasts and notes that reveal the final outcome.

Store references to these snapshots with the case. Pin the retrieval configuration, prompt and model version where possible. If a provider cannot pin a model, record the returned version metadata and evaluation date. Otherwise, a changed answer may reflect changed evidence rather than improved reasoning.

Separate development cases from evaluation cases

Use one set of cases to refine prompts and retrieval, then reserve a later period for evaluation. Keep related events, such as revisions of the same purchase request, together so that an almost identical case does not appear on both sides.

The scikit-learn guidance on data leakage explains why information unavailable at prediction time creates overly optimistic evaluation. The same principle applies when an agent retrieves ERP records instead of reading a conventional feature table.

Sample ordinary requests and exceptions: unavailable suppliers, contradictory quantities, missing quotes and inventory corrections. Report how cases were selected. A handpicked collection of easy approvals cannot establish readiness for the whole workflow.

Review proposals without executing them

Run the assistant with read-only tools and route its proposed actions to an evaluation store. Use an isolated replay environment, not a live purchasing queue. Give reviewers the same time-bounded evidence and hide which system produced each proposal when practical.

For each case, classify the result as acceptable, requiring correction, requiring escalation or unsupported. Record the violated rule and the evidence used. Review disagreements separately; historical human decisions are useful comparisons, not automatically correct labels.

For context on the processes being evaluated, our overview examines how agentic AI changes ERP workflows. This protocol adds a concrete way to evaluate those proposed capabilities.

Measure usefulness alongside failure

Track unsupported recommendations per evaluated case, mandatory escalations missed, unnecessary escalations and reviewer minutes per case. Count skipped cases and tool failures explicitly. Break results down by supplier, exception type and purchasing category; an acceptable aggregate can conceal a weak subgroup.

Predefine acceptance criteria with the process owner before examining the reserved results. A read-only replay can assess recommendation quality, but it cannot establish realized savings or the effects of actions that were never taken.

After launch, compare incoming case distributions with the evaluation set. Google's model monitoring overview distinguishes changes between training and serving data from changes in serving data over time. Applied here, distribution checks signal when the evaluation may no longer represent current work. They should trigger investigation and fresh review, not automatic approval of a new model.