The report said: "Week 11 signup spike (+34%) likely driven by the March newsletter."
There was no March newsletter. We sent one in January. The 34% was real, the number behind it was correct, and the explanation next to it was invented. Same sentence. One part measured, one part made up, with no seam between them.
That is the failure I now build against. The tool I made for this is Weekly Ops Report: four small agents that pull your metrics, flag anomalies, and drop a ranked action list into an Excel workbook. Here is the part of that design I care about most.
How the wrong sentence got there
The pipeline runs four agents in sequence: data collection, cleaning, anomaly annotation, operations recommendations. The anomaly step found a genuine outlier. The recommendation step then wrote a plain-language cause for it, and nothing in that step was allowed to say "I don't know."
The number came from the data we collected. The explanation came from a model that had no changelog, no deploy log, no calendar, and no idea what we had shipped. It guessed a cause from the shape of a curve. Fluent, plausible, and probably wrong.
What I changed
Candidates get labelled as candidates. The output now reads "candidate explanation, not verified against a change log," and the phrasing is enforced, not suggested. An anomaly is a fact. Its cause is a hypothesis until a human checks a system I cannot see.
The report never invents a number. Every metric in the workbook traces back to a collected value. If something is missing it shows up missing, not as an estimate. That is the rule I would keep even if it made the report look less impressive.
Demo mode says simulated, everywhere. Before you connect a real source, the pipeline runs on documented simulated data and the workbook is labelled as such on the summary sheet. Not in a footer. On the sheet you screenshot into a slide.
That last one exists because I have watched a demo get screenshotted into a planning meeting. A chart that looks real with a number that looks real is the most expensive thing in the product.
The demo label, concretely
This is the whole mechanism, and it is about six lines:
def render_workbook(metrics, source_label):
# summary sheet carries the label, not the footer
ws["A1"] = "Weekly Ops Report"
ws["A2"] = f"Data source: {source_label}" # "SIMULATED - demo data"
ws["A3"] = f"Generated: {utcnow():%Y-%m-%d %H:%M} UTC"
for row in metrics:
# a missing value renders as missing. never as an estimate.
ws.cell(row=row, column="value",
value=metrics[row] if metrics[row] is not None else "- no data -")
The two lines that matter are the label on the sheet you screenshot, and the refusal to invent a value when a source returns nothing. Everything else is formatting.
The pipeline, honestly described
| Agent | Does | Does not do |
|---|---|---|
| Data Collection | Pulls metrics from your sources | Infer values that are absent |
| Data Cleaning | Detects anomalies, flags gaps | Fill gaps by guessing |
| Anomaly Annotation | Marks flagged values, attaches candidate causes | Present a candidate as a finding |
| Operations Recommendations | Ranks P0/P1/P2 actions | Invent a metric to justify a rank |
Four sheets come out: Data Overview, Trend Analysis, Anomaly Annotation, Operations Recommendations. The column worth looking at is the last one on the Anomaly sheet, the one that says "this explanation is a guess."
How the candidates get ranked
The ranking is where this design pays off, because a rank without provenance is just a confident list. Each recommendation carries the evidence that produced it:
| Rank | Action | Evidence required | If evidence is missing |
|---|---|---|---|
| P0 | Investigate the flagged metric | Anomaly confirmed in cleaned data | Cannot be raised to P0 |
| P1 | Check the deploy log for the window | A dated change in range | Listed as "check window" with no claim |
| P2 | Note for next cycle | Nothing urgent | Emitted, never as cause |
The rule is mechanical: a recommendation cannot sit above the evidence available for it. When the anomaly step found the week-11 spike, the newsletter candidate never cleared P1, because the March newsletter does not exist in any system the pipeline can read. It still got written down, as a candidate, attached to the spike, marked unverified. That is the difference between a hypothesis and a finding, and it is one line of enforcement rather than a paragraph of prompting.
The rule I now use for every AI feature
If the model cannot see the evidence, it does not get to state the conclusion.
This is not a prompt tweak. It is a structural decision about where the model sits: it reads collected values and produces language about them, and it never produces the values. Once a model can emit a number, you have no way to tell which part of a sentence is measured and which part is invented. Removing that ability is cheaper than auditing it later.
For a previous product I made the opposite mistake in a different place. I shipped a four-line regex that three separate checks passed, which took an image pipeline down for six hours and needed two characters to fix. I had optimised for "the check passed" over "the thing works." The same instinct, letting the automated layer look authoritative, is what produced a confident sentence about a newsletter that did not exist.
What it is not
It does not replace your BI tool, and it does not know why anything happened. It reads numbers you connect it to and tells you which ones look wrong, with the guess clearly marked as a guess.
Model-assisted narrative insights are a paid feature. Rule-based report generation with a fair-use daily cap is free. That split is deliberate: the rule-based path cannot fabricate a cause because it does not write causes.
The interface, prompts and report output are English in v1.x.
If your decision depends on the explanation rather than the number, the explanation needs a human who can open the change log. I built the part that can honestly say it does not know, which turns out to be the part that matters.
Top comments (1)
Yeah gluing a real spike to a cause you never measured is the Failure mode that looks polished.
What worked for me was making the answer open the definition it used and the source rows behind the number. If the rows don't support the story, leave the cause blank rather than invent one. Fluent text next to a correct total is still clean wrong.