delivery-harness lets the rule engine set the payout and keeps the model advisory

#ai #llm #java #guardrails

One short line in the delivery-harness README captures a central design decision: compensation amounts are decided by the rule engine, never by the model. For a project that calls itself an AI harness, much of what the README describes reads like a fence around the model, keeping its output advisory while deterministic code holds the numbers.

Scope comes first. The README calls the repository an educational MVP and says it is not a production-ready delivery or compensation system. Its business tools return synthetic data, state is mostly held in memory, and there is no authentication, authorization, tenant isolation, or rate limiting. The authors warn against exposing it to the public internet, submitting real personal or order data, or executing compensation decisions without authentication, policy enforcement, and human approval. I read it as a reference implementation to study and run locally.

What stays deterministic

The README lists two synchronous workflows: abnormal-order analysis and compensation suggestion. Around them sit four locally registered tools covering orders, ETA, station capacity, and compensation rules, and each tool derives its result from its arguments. Synthetic orders vary by order ID across six named scenarios, one of which is delivered on time.

The README also describes a deterministic timeline that splits an order into dispatch wait, to-shop, merchant prep, and on-road legs. According to the README, that timeline is reported alongside the model's answer "as the baseline it has to beat."
On the compensation side, the rule engine owns the amount, and the README points to an architecture decision record for that choice. For DAMAGED complaints, the caller passes a damage_level of FULL or PARTIAL; without evidence, the policy returns zero and requires manual review. The property harness.compensation.approval-threshold defaults to 10.0, and payouts at or above that value require a human approver.

Failure handling follows the same split. When a completed workflow has an unavailable optional dependency, the README says the run is reported as DEGRADED, the deterministic result remains available, and the affected step is recorded as failed.

Where the model sits

The model is reached through an OpenAI-compatible chat-completions client behind an interface, which the README says makes the workflows testable without a model. The default HARNESS_LLM_ENDPOINT points at localhost on port 11434, qwen2.5:7b is the example model, and the Compose port binds to 127.0.0.1 only.

Knowledge retrieval is in-memory and lexical, scored by query-term overlap, with a rule and case base seeded at startup. The README flags ingested documents as a prompt-injection trust boundary, a habit worth copying into any harness that lets operators add knowledge.

The fields needs_human_review and approval_required are derived from guardrail outcome, parse success, model confidence, and baseline agreement. The README calls this reporting, not authority, and says nothing here executes a payment. Its advisory checker detects a small set of phrases and amount conditions, and the README states that it is not a policy engine.

Tests and evaluation

The suite has 112 tests, and both workflows run end to end against a stub model transport, so no test needs Ollama or a Docker daemon. The README names buildsDifferentEvidenceForDifferentOrders as the load-bearing test. It asserts that two different orders produce two different prompts, which the README calls the property every other measurement depends on.

Evaluation runs live in memory, and scorers report "not measured" instead of a perfect score when a case declares no expectation. The README describes the scorers as lexical and calibrated to catch regressions, not to certify quality, and says six synthetic cases cannot establish accuracy. It adds that modelVersion and promptVersion on a run are metadata labels that do not yet select a model or change the workflow prompt.

What is not implemented

The README also lists capabilities the project does not claim as implemented, a list worth checking if you are sketching an AI layer over an operational process like refunds or dispatch exceptions. It names autonomous agent planning, production tool integrations, durable persistence, vector embeddings and Milvus RAG, authentication, rate limiting, distributed tracing, and automatic compensation execution. The known limitations add that rules, documents, cases, evaluations, feedback, metrics, and traces are lost on restart, and that retrieval splits CJK text on whitespace and punctuation rather than segmenting it.

GitHub: https://github.com/diudiu-tech/delivery-harness

Curated by Agent Palisade — practical AI for small and mid-sized businesses.