DEV Community

Ujjwal Dubey
Ujjwal Dubey

Posted on

The Logging Contract: One JSON Line per Automation Run, and a Script That Checks It

If someone else builds your automations, the most useful thing you can ask for is not documentation. It is a log line you can read. When we work as an agency we agree a logging contract up front, because AI automation agency builds you can debug later start with a predictable record of every run.

The contract

Every run of every automation writes exactly one JSON line when it finishes:

{"flow": "invoice-intake", "run_id": "2026-10-28T09:14:03Z-7f3a", "status": "ok",
 "input_ref": "email:msg-8812", "output_ref": "sheet:row-412", "human_review": false,
 "duration_ms": 1840, "error": null}
Enter fullscreen mode Exit fullscreen mode

Rules:

  • flow and run_id are always present.
  • status is one of ok, skipped, needs_review, failed.
  • input_ref points at the source item so a person can find it.
  • error is null unless status is failed.
  • No customer message text or personal data in the line, only references.

A checker you can run daily

import json, sys
from collections import Counter

REQUIRED = {"flow", "run_id", "status", "input_ref", "output_ref", "human_review", "error"}
STATUSES = {"ok", "skipped", "needs_review", "failed"}

def check(path):
    bad, counts = [], Counter()
    with open(path) as f:
        for n, line in enumerate(f, 1):
            try:
                rec = json.loads(line)
            except json.JSONDecodeError:
                bad.append((n, "not json")); continue
            missing = REQUIRED - rec.keys()
            if missing:
                bad.append((n, f"missing {sorted(missing)}"))
            if rec.get("status") not in STATUSES:
                bad.append((n, f"bad status {rec.get('status')}"))
            if rec.get("status") == "failed" and not rec.get("error"):
                bad.append((n, "failed without error"))
            counts[(rec.get("flow"), rec.get("status"))] += 1
    return bad, counts

if __name__ == "__main__":
    bad, counts = check(sys.argv[1])
    for (flow, status), c in sorted(counts.items(), key=str):
        print(f"{flow:20} {status:13} {c}")
    for n, msg in bad[:20]:
        print(f"line {n}: {msg}")
    sys.exit(1 if bad else 0)
Enter fullscreen mode Exit fullscreen mode

Why this belongs in the agreement

  • You can answer "did it run yesterday?" without asking anyone.
  • A count of needs_review per flow tells you whether human approval is still needed.
  • If you ever change agencies, the next team starts from the same record.

Put the contract in the statement of work, and ask for a sample log file before the build ends. More engineering notes live on the NxFlowAI engineering notes.

Top comments (0)