Most AI Agents Can't Tell You If Their Output Is Any Good

An agent can generate text all day. It has no idea whether that text is readable, on-tone, or leaning positive vs negative.

That gap is dangerous the moment an agent starts producing customer-facing content — support replies, posts, summaries — instead of just answering you in a chat box. If nobody scores the output, "good enough" silently becomes "off-brand," and you find out from a screenshot someone sent you.

The problem: no quality signal in the loop

Most agent pipelines look like this:

prompt -> model -> output -> (ship it)

There's no checkpoint that asks: is this readable? is the sentiment right? what are the actual keywords being pushed? You're flying blind on the one thing that matters — whether the words are doing their job.

What I built: a text-insight checkpoint

I wired a lightweight analysis API into the agent loop so every generated piece of text gets scored before it ships. The API returns sentiment, readability, and keyword extraction in one call.

Here's the integration in Python:

import os, requests

TEXTINSIGHT_URL = "https://thebookmaster.zo.space/api/textinsight"


def score_text(text: str) -> dict:
    resp = requests.post(
        TEXTINSIGHT_URL,
        json={"text": text},
        timeout=10,
    )
    resp.raise_for_status()
    return resp.json()


def safe_to_publish(text: str) -> bool:
    result = score_text(text)
    readability = result.get("readability", {}).get("score", 0)
    sentiment = result.get("sentiment", {}).get("label", "neutral")
    # Block if too dense or sentiment is negative for a public post
    return readability >= 50 and sentiment != "negative"


draft = "Our new agent stack reduces latency by 40% while improving output quality."
if safe_to_publish(draft):
    print("ship it")
else:
    print("rewrite needed")

Now the loop is:

prompt -> model -> score_text() -> gate -> (ship | rewrite)

The gate is trivial to extend: reject profanity, enforce a max reading-grade level, require certain keywords, flag off-brand sentiment. It turns "I hope this is fine" into a measurable check.

Why this matters more than people think

Agents don't get embarrassed. They'll happily emit a wall of jargon or a tone-deaf reply and consider the task done. A text-insight checkpoint is the cheapest possible editor — it runs in milliseconds, costs pennies, and catches the failures that would otherwise reach a human.