We are building an MLOps validation agent that audits a model deployment manifest and current performance metrics, then returns a structured go/no-go recommendation. This helps ML engineers catch misconfigurations and data drift before a release reaches production.
What you'll need
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai
Step 1: Scaffold the mock deployment environment
I start with a realistic deployment manifest and a set of production metrics. Both are plain Python dictionaries so the tutorial is fully runnable without external services.
import json
DEPLOYMENT_MANIFEST = {
"model_name": "fraud-v2",
"version": "2.1.0",
"framework": "pytorch",
"gpu": "A100",
"replicas": 2,
"batch_size": 512,
"memory_limit": "16Gi",
"logging_level": "debug",
"pii_filter": False,
"rollback_on_error": False
}
CURRENT_METRICS = {
"accuracy": 0.912,
"baseline_accuracy": 0.934,
"latency_p99_ms": 1450,
"baseline_latency_p99_ms": 890,
"error_rate": 0.03,
"baseline_error_rate": 0.01,
"prediction_volume_24h": 4800000
}
Step 2: Define the agent's system prompt
The system prompt grounds the LLM as an MLOps auditor. It enforces structured JSON output and prevents vague prose.
SYSTEM_PROMPT = """You are an MLOps validation agent. Your job is to audit model deployment manifests and performance metrics. Follow these rules exactly:
1. Identify every misconfiguration or risk in the manifest.
2. Flag any metric that indicates model drift or degraded service health.
3. Return your findings as a JSON object with two keys: "manifest_issues" (list of strings) and "drift_flags" (list of strings).
4. If no issues are found, return empty lists.
5. Do not include markdown formatting, explanations, or conversational text outside the JSON."""
Step 3: Audit the deployment manifest for misconfigurations
This function sends the manifest to Oxlo.ai. I use llama-3.3-70b because it follows structured instructions reliably.
from openai import OpenAI
import json
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
def audit_manifest(manifest: dict) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Audit this deployment manifest:\n{json.dumps(manifest, indent=2)}"},
],
)
raw = response.choices[0].message.content
cleaned = raw.replace("
```json", "").replace("```
", "").strip()
return json.loads(cleaned)
manifest_result = audit_manifest(DEPLOYMENT_MANIFEST)
print(json.dumps(manifest_result, indent=2))
Step 4: Detect model drift from live metrics
Next, I send the metrics to Oxlo.ai and switch to deepseek-v3.2 for strong reasoning on numerical deltas. Because Oxlo.ai uses request-based pricing, sending large metric payloads does not increase the cost.
def check_drift(metrics: dict) -> dict:
response = client.chat.completions.create(
model="deepseek-v3.2",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Evaluate these production metrics for drift:\n{json.dumps(metrics, indent=2)}"},
],
)
raw = response.choices[0].message.content
cleaned = raw.replace("
```json", "").replace("```
", "").strip()
return json.loads(cleaned)
drift_result = check_drift(CURRENT_METRICS)
print(json.dumps(drift_result, indent=2))
Step 5: Generate the final go/no-go report
I combine the two JSON results into a final recommendation. If either list contains items, the deployment is blocked.
def generate_report(manifest_issues: list, drift_flags: list) -> str:
lines = [
"=== MLOps Validation Report ===",
"",
f"Manifest risks found: {len(manifest_issues)}",
]
for issue in manifest_issues:
lines.append(f" - {issue}")
lines.append("")
lines.append(f"Drift flags found: {len(drift_flags)}")
for flag in drift_flags:
lines.append(f" - {flag}")
lines.append("")
if manifest_issues or drift_flags:
lines.append("Recommendation: NO-GO. Fix the above before deploying.")
else:
lines.append("Recommendation: GO. No issues detected.")
return "\n".join(lines)
report = generate_report(
manifest_result.get("manifest_issues", []),
drift_result.get("drift_flags", [])
)
print(report)
Run it
Running the full script in one file produces output similar to this:
{
"manifest_issues": [
"logging_level is set to 'debug' in production, which may expose sensitive data and hurt performance",
"pii_filter is disabled, creating compliance risk",
"rollback_on_error is disabled, increasing blast radius on failure",
"replicas=2 with batch_size=512 may be aggressive for a 16Gi memory limit"
],
"drift_flags": []
}
{
"manifest_issues": [],
"drift_flags": [
"accuracy dropped from 0.934 to 0.912, exceeding typical 1% threshold",
"latency_p99 increased from 890ms to 1450ms, indicating potential throughput regression",
"error_rate tripled from 0.01 to 0.03"
]
}
=== MLOps Validation Report ===
Manifest risks found: 4
- logging_level is set to 'debug' in production, which may expose sensitive data and hurt performance
- pii_filter is disabled, creating compliance risk
- rollback_on_error is disabled, increasing blast radius on failure
- replicas=2 with batch_size=512 may be aggressive for a 16Gi memory limit
Drift flags found: 3
- accuracy dropped from 0.934 to 0.912, exceeding typical 1% threshold
- latency_p99 increased from 890ms to 1450ms, indicating potential throughput regression
- error_rate tripled from 0.01 to 0.03
Recommendation: NO-GO. Fix the above before deploying.
Next steps
Swap the mock dictionaries for live calls to your model registry and Prometheus endpoint. You can also wrap this script in a GitHub Actions job so every pull request to your serving config gets an automated Oxlo.ai audit.
Because Oxlo.ai charges a flat rate per request instead of per token, long manifests and detailed metric histories do not inflate costs the way token-based pricing would. For teams running hundreds of validation checks a day, that predictability matters. See https://oxlo.ai/pricing for details.
Top comments (0)