Our studio runs a small pipeline that watches a freelance marketplace, drafts proposals, and sends them without a human in the loop for the easy cases. Before anything goes out, the text passes through a guard function. This week we audited what the pipeline had actually been saying to prospects, and the result changed how I think about validation layers.
The audit started because of a single phrase in our templates: "three recent client engagements, described without client data." It sounded solid. We went claim by claim through the material our scripts had been sending.
The first cited project was a security audit of a Windows installer. Real work, real findings, two NO-GO verdicts, a full test suite behind it. But it was our own internal work. No client existed. The second was a trademark watch service that "cross-references the client's trademark portfolio." That's the product we were pitching in an outreach campaign — measurements from our own ledger showed twenty-five sends and zero replies at the time. Pitching it as delivered client work was, at best, optimistic. The third claimed a Telegram bot had "89 tests." The package has 126 test functions today. The direction was true; the number was stale and hardcoded.
Nothing was invented. Everything was misleading. That distinction is the whole lesson.
The guard we trusted, guarda_texto, checks mechanics: no links, no payment keywords, length limits, accent normalization, forbidden project words. It rejected drafts that were dangerous or sloppy. It passed every draft above, because every claim was grammatical, format-safe, and free of banned strings. A validation layer only enforces what it can express, and ours could not express "is this claim true."
# what we had: mechanics only
def guarda_texto(texto):
return not re.search(r"(pix|comiss|\d+\s*reais)", texto, re.I) \
and LIMITE_MIN <= len(texto) <= LIMITE_MAX
When we sat down to fix it, someone proposed the expensive option first: a second model reviewing every generated proposal. We stepped back and ranked by cost. The cheapest fix was not a reviewer at all. It was rewriting the fixed templates to say "three of our own recent projects" instead of "client engagements," and deleting "the client's portfolio" from the trademark pitch. That removed the false implication at the source, in minutes, with no new moving parts.
The second fix targets the generated text, where templates can't help. Every experience claim now has to trace to an entry in what we call our house-proof file — a plain record of work we actually did, with its verifiable artifacts. The guard gains a second phase:
# what we added: claims must resolve
def guarda_afirmacoes(texto, prova_da_casa):
for afirmacao in extrair_afirmacoes_de_experiencia(texto):
if not prova_da_casa.existe(afirmacao):
return False, afirmacao # block, and say which claim failed
return True, None
A second-model review still has a place, but only after those two: it checks generated prose against a checklist where every line points back to the proof file. Reviewing text whose claims can't resolve anywhere just produces confident opinions.
The stale test count taught a third rule: never hardcode a number the artifact can produce. The count of test functions is one grep away from the package at send time. Any figure in sales copy that we don't recompute is a claim with an expiration date we can't see.
The same discipline showed up in the same audit on the metrics side. We had eight proposals out over three days and wanted to compute conversion per text variant, per price band, per channel. Eight samples is noise, and slicing noise by three variables produces decisions, not knowledge. The useful metric right now is binary and weekly: proposals sent, crossed with their status on the platform's own proposal page, and their age. We set ourselves a threshold — roughly thirty proposals — before cutting by any variable. Until then, the dashboard shows counts, not conclusions.
There was one more trap in that metrics plan worth mentioning. Our internal talk panel listed three conversations while the platform state file said eight. The listing is unstable; the proposal page is authoritative. We almost built a conversion funnel on top of the unstable one, because it was already in our database and the other source required another fetch. Convenience picks the data source unless you pick it deliberately first.
What I'm keeping from this week: guards split in two, one for mechanics and one for claims that resolve against a proof record; templates that say exactly what the work was, not what sounds bigger; numbers recomputed from the artifact instead of remembered; and metric plans that wait for enough samples before they pretend to explain anything. The pipeline sends the same volume as before. What changed is that now, when something false tries to get out, there is a layer that can actually notice.
Top comments (0)