DEV Community: CyprianTinasheAarons

The $239,000 Manual Handoff Tax: The ROI of Deep AI-to-ERP Integration 💸

CyprianTinasheAarons — Fri, 31 Jul 2026 09:17:40 +0000

A note on the numbers upfront, because I'd rather you catch this than a client does: $239,000 is illustrative. It's a composite built from published benchmarks on manual data-processing costs, applied to a mid-market company on a legacy ERP (SAP, NetSuite, or Dynamics). Not a universal constant. Recalculate it against your own client's headcount and error rates before you quote it out loud. The method's laid out below so you can defend it if someone pushes back.

Everyone's tuning prompts. Almost no one's fixing the handoff 🤔

Scroll through any AI strategy conversation right now, and it's all prompt engineering, model selection, fine-tuning, which LLM reasons best, which agent framework to standardize on. Easy stuff to obsess over, honestly; it's visible, it's measurable, it's fun to tinker with.

Here's the problem though. None of it matters if the AI's output still has to get manually copied, reformatted, and re-keyed into your ERP by a person at 6 pm reconciling the day's exceptions.

That's the handoff. The unglamorous seam between "the AI figured it out" and "the system of record actually has it." And it's exactly where the money leaks out.

Key Definitions 🧐

What's "the handoff"?

The manual step where a person moves AI-generated output into the actual system of record (SAP, NetSuite, Dynamics, whatever it is) because nothing automated does it for them.

What's an exception queue?

The pile of records the system genuinely couldn't resolve on its own and needs a human to look at. Before integration, this is most of the work. After, it's a five-minute review.

What the handoff really costs

The research on manual data processing is remarkably consistent, and none of it flatters the status quo:

Manual data entry runs an average of $15 per document once you count labor, error correction, and turnover. A mid-size accounts payable team processing 5,000 invoices a month is sitting at $75,000 a month before a single mistake even gets fixed.

Human error on manual entry runs 1% for skilled operators, up to 4% for average ones, per field. At 20 fields per invoice, that's 0.2 to 0.8 errors per document. Not a risk. A mathematical certainty at any real volume.

Organizations with 100+ employees spend $430,000 to $850,000 a year on manual document processing once you include the hidden stuff, error remediation at $25 to $150 a mistake, plus staff losing 20 to 40% of their time just searching for, verifying, or fixing document issues.

Poor data quality costs the average org $12.9 million a year, per Gartner. Over a quarter of organizations put their own losses above $5 million annually. 😬

None of that's actually an AI problem. It's a plumbing problem. And it's exactly the layer most AI rollouts skip, because a chatbot drafting a nice summary is a much easier demo than a pipeline writing clean, validated data straight into SAP.

Back of the envelope, for a mid-market company running a couple legacy systems with a team still reconciling exceptions by hand: a few FTEs' worth of manual reconciliation, plus error remediation on a modest error rate, plus the opportunity cost of good people doing data entry instead of their actual job, adds up fast into six figures. That's where $239,000 comes from. A realistic annual tax for a specific-sized company, not a number that applies evenly to everyone. Run it against your own client's headcount first.

Why the handoff is expensive to fix, and why most vendors underquote it 🎲

Custom AI-to-ERP integration work typically runs $10,000 to $35,000 for one well-scoped integration, which lines up with published benchmarks showing point-to-point ERP integrations (SAP S/4HANA to Salesforce, NetSuite to a custom warehouse system, that kind of thing) landing between $10,000 and $50,000+ depending on complexity.

Two things reliably push that number up if nobody scopes them correctly upfront.

1. Legacy system resistance. SAP, NetSuite, and Dynamics are the three we run into most on the mid-market side, and each one has its own friction. SAP specifically wants a CSRF token on every write operation plus an OAuth 2.0 client-credentials handshake most developers haven't touched before. Discovering and solving that alone typically eats 2 to 4 weeks that never show up in the original estimate. It's a big part of why 40 to 60% cost overruns are the norm, not the exception, on integration projects generally.

2. Data cleaning and compliance overhead. An AI system reads a mangled invoice or a half-structured shipping manifest just fine. Your ERP can't. Every field the AI pulls out has to be validated, mapped to the ERP's schema, and checked against whatever compliance rules govern that record type. That mapping work is where most of the real engineering hours go. Not the AI part.

The contrast is simple, honestly. Today, a person is the API between your AI tools and your ERP. After deep integration, they're not.

What six weeks actually looks like 🛣

Timelines vary a lot by system. Mid-market NetSuite integrations often go live in 4 to 6 weeks thanks to a more accessible API, multi-system builds across five or six applications typically run 8 to 16 weeks, and heavily customized SAP work can stretch past six months. A 6-week, three-phase engagement is realistic for a single-ERP, well-scoped project, and it's on the efficient end of the range on purpose.

Phase 1 — Discovery & data audit (Weeks 1-2)

Map every system currently in the handoff chain. Audit data quality at the source. Find the specific fields and record types causing the most manual rework.

Deliverable: a scoped integration plan with the real cost drivers named, not guessed at.

Phase 2 — Build & validation logic (Weeks 3-5)

Build the connection points. Write the validation and mapping logic between the AI output and the ERP schema. Handle authentication for whichever legacy system is involved.

Deliverable: a working integration in staging, tested against real historical data.

Phase 3 — Rollout & reliability testing (Week 6)

Move to production. Run parallel reconciliation against the old manual process to confirm accuracy. Hand off a monitoring setup so exceptions get flagged instead of buried.

Deliverable: a live integration plus a documented exception-handling process for the edge cases that still need a human. 🤗

The next step 🚀

If your team is still the API between your AI tools and your ERP, that's fixable and scoped. It's not a reason to sit on a bigger AI initiative waiting for someday.

Book an AI Reliability & Security Fit Call, and we'll walk through where your specific handoff is leaking time and money, and whether a direct integration is actually the right next move for you.

AI & Revenue Workflow Fit Call

cal.com

Thanks for reading this one.

The 40% Cancellation Trap: Why Agentic AI Projects Die in Production 🪤

CyprianTinasheAarons — Fri, 31 Jul 2026 09:06:26 +0000

Gartner reckons over 40% of agentic AI projects get canned by 2027. Not because the models are bad. Because nobody built the safety net.

Okay so that stat has been quoted so much it's basically lost its bite by now. But stick with me, because the reason buried inside it is the one most teams still aren't talking about: inadequate risk controls. Not cost. Not "unclear ROI." Risk controls.

If you're building agentic AI for a fintech or healthtech company, that line should worry you more than any of the others, because it's not a demo problem. It's a production problem. And in a regulated shop, production problems don't get killed in a budget review. They get killed in a compliance review.

Key Definitions 🧐

Before we get into why this happens, a few terms worth pinning down.

What's an agentic loop?

The retrieve → evaluate → retrieve-again cycle an agent runs when it's deciding whether it has enough information to answer. Sounds harmless. It's where most of the trouble starts.

What's Evidence Override?

When the retriever actually pulls the right document, hands it to the model, and the model just... doesn't use it properly. Leans on something more familiar instead. The evidence was there. The generation step ignored it.

What's cascading hallucination?

One wrong claim early in a multi-step reasoning chain becomes "context" for every step after it. Each step stays logically consistent with the step before, so the whole thing reads as confident and coherent, right up to the end — even though it was wrong two steps back.

The fear isn't "it doesn't work" 😬

Here's the actual fear, if you're honest about it: it's not that the agent gives a wrong answer. It's that the agent gives a wrong answer that looks right. Well formatted. Internally consistent. Confident. The kind of wrong that sails straight past a reviewer.

In a regulated environment that's not an embarrassing bug you patch on Monday. That's an incident report. That's an audit finding. That's a regulator asking a question you don't have a good answer to.

Pilots survive because someone is watching every single output closely. Production doesn't survive that same way, because nobody can watch every output closely at scale, and the agent has no internal sense of the difference between "confidently right" and "confidently wrong."

The failure everyone assumes vs. the one actually happening

Ask most teams why their RAG agent hallucinated and you'll hear "bad retrieval" almost every time. Wrong chunks got pulled. Fix the chunking, fix the embeddings, ship it.

Current research says that's usually the wrong diagnosis. Evidence Override — the generation-side failure, not the retrieval-side one — shows up several times more often than genuine retrieval failure. The retriever did its job. The correct evidence was sitting right there in context. The model just didn't weight it properly.

That distinction actually matters a lot, because the fix is completely different depending on which one you've got. Retrieval failure gets fixed with better chunking. Generation failure needs something that checks what the model did with the evidence it was handed. That's a validation problem. Not a search problem.

Why agents fail quietly, and APIs never do 🔗

A normal API call fails loudly. It errors out, it times out, you get a status code, you get a stack trace, you know something broke.

An agentic loop fails quietly. It fails by continuing. Without a hard stopping rule, the default answer to "do I have enough?" is always "get more" — so the loop retrieves again, escalates, retrieves again, burning tokens the whole way without necessarily getting any closer to correct.

Worse, the mistakes compound. An early hallucinated claim doesn't just sit there quietly; it becomes the foundation the next reasoning step builds on. The system isn't lying badly. It's reasoning cleanly from a bad premise, all the way to a confident, wrong, and completely traceable-looking conclusion.

Framework choice is a control surface, not a preference 🎲

This is where the tooling you pick actually matters, and it's worth being blunt about it.

CrewAI's default recovery behavior, when a tool call fails, is to retry with the same approach. That can loop forever unless you build custom callbacks yourself to stop it.

LangGraph takes a different route. Its interrupt-and-checkpoint model lets you pause the workflow at a defined point, wait on a human decision, and resume from that exact state. For a regulated environment where every AI decision needs an audit trail behind it, that's just a meaningfully better fit.

A quick composite example 🤓

(Built from patterns we see repeatedly across clients, not any one specific company, just to be clear.)

A healthtech intake agent classified incoming patient documents and routed them to the right internal system. Tested well. High accuracy in the pilot.

In production, an ambiguous document type triggered the "do I have enough to classify this?" loop. The agent re-queried its knowledge base four times, drifting the classification a little further each pass, until it confidently filed the document under the wrong compliance category.

No error thrown. Nothing broke. It just quietly did the wrong thing, cleanly, and moved on.

The reframe

Your agent didn't fail because it couldn't find the right answer. It failed because nothing was watching what it did with the answer once it had it.

That's the shift. The whole industry spent close to two years obsessing over retrieval — better chunking, better embeddings, better vector stores. Retrieval was never the whole problem. The generation step, and the agent's own decision loop, need the same level of scrutiny retrieval already got. Most architectures still don't give it to them.

The fix: a checkpoint, not a patch 🚀

The pattern that actually survives contact with production isn't one validation interceptor bolted between the retriever and the model. It's a checkpoint built directly into the orchestration graph, sitting at every point where the agent makes a decision that actually matters.

Firstly, notice the fallback isn't "retry." Retry is usually what caused the loop to begin with.

Next, the real fallback is a human review queue the agent escalates to on its own, once confidence drops below a threshold or the decision crosses a defined policy boundary — a compliance-sensitive classification, a transaction above a certain size, anything diagnosis-adjacent.

Lastly, the human reviewer sees exactly what the agent saw, approves or rejects it, and the workflow resumes from that exact checkpoint. Which is, not coincidentally, exactly what gives you the audit trail a regulator or an internal compliance team is eventually going to come asking for.

This is the "capability-deployment verification gap" that shows up over and over in the cancellation-risk research: agents that pass every pilot test but were never built with the escalation path production actually needs. The gap was never the model. It's the missing checkpoint. 🔗

Find out where your build actually sits 🔍

If you're not totally sure whether your architecture has this checkpoint, or whether it just has a retry loop dressed up in a checkpoint's clothes, that's exactly what our AI Reliability Scorecard is built to show you.

The Scorecard checks:

Whether failure gets caught at retrieval, at generation, or not until a human notices downstream
Whether you have an explicit stopping rule for agentic loops, or you're relying on implicit token/time limits and hoping
Whether low-confidence or policy-boundary decisions escalate to a real human review queue, or just retry
Whether every AI-driven decision in your pipeline leaves behind an auditable record of what was seen, decided, and overridden
How exposed your current architecture is to cascading hallucination across multi-step chains

Take it here: https://www.topiax.xyz/audit

What happens after: you get a scored breakdown by category, not one single grade, mapped straight to the gaps above, plus a short written read on which failure mode is the biggest production risk for your specific stack and industry. If your score flags real exposure, we'll follow up with an invite to a 20-minute call to walk through the checkpoint pattern for your architecture specifically. No pitch deck. Just your results and what we'd fix first.

Thanks for reading this one. If it was useful, share it with whoever on your team owns the "why did the agent do that" conversation. 🙏

The vibe code teardown: why your app can pass the demo and still fail security checks

CyprianTinasheAarons — Mon, 27 Jul 2026 08:28:23 +0000

Your app works.

The login flow works. The dashboard loads. The database returns the right records. The landing page looks good enough to share.

Then someone opens the browser developer tools.

They find a credential in the JavaScript bundle. They change an object ID in an API request and see another user's data. They query a Supabase table that was never protected with a row-level security policy.

The app was not broken in the way most people expect. It was broken while everything still looked finished.

That is the security problem with vibe-coded applications: a successful demo tells you that the feature works. It does not tell you whether the trust boundaries are correct.

The 45% number, stated accurately

Veracode's 2025 GenAI Code Security Report tested more than 100 large language models across Java, JavaScript, Python, and C#.

In its test set, 45% of generated code samples failed security tests and introduced an OWASP Top 10 vulnerability.

The Spring 2026 update used a continued testing framework and found that only about 55% of generation tasks produced secure code when the models received no security-specific guidance. In the other 45% of cases, the generated code introduced a known security flaw in the test task.

That does not mean 45% of all AI-built applications are vulnerable. The tests measured generated code in controlled tasks, not every application produced by every developer.

The result is still worth paying attention to:

AI coding tools have become much better at producing code that works. That improvement has not automatically made the code safe.

The gap matters because the most dangerous problems in a small web application are often not exotic. They are ordinary mistakes hidden behind a polished interface.

The demo is not the security boundary

AI coding tools are optimized to satisfy the request in front of them:

add authentication
connect the database
integrate payments
expose an API
create an admin dashboard
make the page work

Security depends on questions that may never appear in the original prompt:

Which user is allowed to read this record?
What happens if they change the ID in the request?
Which values can safely reach the browser?
Which database operations should the public client role perform?
Is this admin route protected outside the normal UI?
Does an error response reveal credentials or internal details?
Can a user perform an expensive action repeatedly?

A generated application can satisfy the visible feature request while leaving those questions unanswered.

That is how a working demo becomes a risky production system.

Here are three places I would check first.

1. The credential that ships to every browser

In a Next.js application, any environment variable prefixed with NEXT_PUBLIC_ is intended to be available to browser-side code.

That makes the prefix useful for public configuration. It also makes it dangerous when someone uses it for a secret.

This is a bad idea:

NEXT_PUBLIC_OPENAI_API_KEY=sk-live-example
NEXT_PUBLIC_STRIPE_SECRET_KEY=sk_live_example
NEXT_PUBLIC_DATABASE_PASSWORD=example

Those values can end up in the client bundle. Anyone who can load the application may be able to retrieve them.

The safe pattern is to keep secret credentials on the server:

OPENAI_API_KEY=sk-live-example
STRIPE_SECRET_KEY=sk_live_example

Then call the provider from a server-side route or server action.

There is an important distinction here: not every visible key is a secret.

For example:

Stripe publishable keys are designed to be public.
Supabase anon keys are designed to be public.
Stripe secret keys are not public.
Supabase service_role keys are not public.
Database passwords and private API tokens are not public.

The problem is not "a key appears in the frontend." The problem is that a credential with more authority than the browser should have crosses the client/server boundary.

What to check

Search the repository and the deployed JavaScript bundle for:

OpenAI or Anthropic API keys
Stripe secret keys
Supabase service-role keys
database credentials
private signing keys
unrestricted third-party API tokens
suspicious NEXT_PUBLIC_ variables

A source-only scan is not always enough. A value may be injected during the build and appear only in the generated client assets.

2. The database with no access policy

Supabase makes it fast to create tables, authentication, and a working frontend.

That speed creates a predictable failure mode: the application is wired together before the database access model is finished.

Row Level Security, or RLS, is the layer that controls which rows a database role can access. A table can exist, the frontend can query it, and the feature can appear complete while the table still lacks the policies that separate one user's data from another's.

The public Supabase anon key is not automatically a master key. It is meant to be public. The risk appears when that public client role has access to data that should have been protected.

A basic policy might look like this:

alter table public.notes enable row level security;

create policy "users can read their own notes"
on public.notes
for select
to authenticated
using (auth.uid() = user_id);

The exact policy depends on the application. The important point is that authentication alone does not prove authorization. A user can be logged in and still be unauthorized to read another user's record.

A real example

CVE-2025-48757 describes insufficient database Row Level Security policies in Lovable-generated sites through April 15, 2025. The NVD record says the condition could allow unauthenticated attackers to read or write arbitrary database tables.

The same record also notes that Lovable disputed the responsibility assigned to the platform and argued that individual customers were responsible for protecting their application data.

That disagreement is worth preserving because it points to the real lesson:

A generated app can look complete while its access-control model is still unfinished.

What to check

For every table:

Is RLS enabled?
Which roles can access it?
Are there explicit policies for select, insert, update, and delete?
Does every policy restrict access to the correct user, team, or organization?
What happens if an authenticated user changes an object ID?
Do tests run as a normal user, or only as an admin?

"RLS enabled" is not the same as "authorization is correct." The policies themselves need review.

3. The admin route nobody meant to publish

During development, an AI assistant may create routes such as:

/admin
/debug
/health/full
/swagger
/api/test

These routes can be useful while building. They become a problem when they reach production without an authentication and authorization check.

The same issue can appear in API handlers. The frontend may hide a button from ordinary users, but hiding a button is not authorization. A user can still call the endpoint directly.

A route should enforce access on the server, not depend on the interface behaving honestly.

What to check

Open the deployed application in a private browser window while logged out.

Test:

admin and dashboard routes
health and debug endpoints
API routes
file or export endpoints
password-reset and invitation flows
routes that accept a user ID, organization ID, or document ID

Then test horizontally:

Create two normal users.
Create a record for User A.
Send the request as User B.
Change the record ID.
Confirm that User B receives a denial rather than User A's data.

This class of bug is often called insecure direct object reference, or IDOR. It does not require a sophisticated exploit. It requires the application to trust an identifier supplied by the client.

A practical pre-launch security check

You do not need to begin with a full penetration test to catch the most obvious problems in a small Next.js or Supabase application.

Start with these checks.

1. Sweep for credentials

Scan:

the repository
.env files
Git history
build output
deployed client assets
CI logs
error-monitoring payloads

Remove exposed secrets and rotate them. Deleting a key from the latest commit does not invalidate a key that already exists in Git history or a deployed bundle.

2. Review database access

For each table, verify:

RLS status
policies for every operation
user and organization boundaries
behavior for unauthenticated requests
behavior for a different authenticated user
behavior when IDs are modified

Test the database with the same roles that real users will have. An admin dashboard can make a broken policy look correct because the admin has more access than ordinary users.

3. Test routes outside the interface

Use an incognito window and direct HTTP requests.

Do not rely on:

hidden buttons
client-side redirects
disabled form fields
checks that run only in React
route names that "look private"

The server must make the final authorization decision.

4. Check browser security configuration

Review:

HTTPS
cookie flags
HttpOnly
Secure
SameSite
Content Security Policy
frame protection
MIME-sniffing protection
permissive CORS
unnecessary X-Powered-By disclosure

These settings will not fix a broken authorization model, but weak deployment defaults can make other failures easier to exploit.

Try Vibe Scanner

I built Vibe Scanner as a read-only scanner for Next.js and Supabase repositories and deployed websites.

It is designed to catch common problems before they become someone else's discovery.

The scanner currently checks repository targets for issues such as:

hardcoded Stripe, OpenAI, GitHub, or AWS credentials
sensitive environment files
Supabase tables without matching RLS enablement
admin or dashboard pages without a recognized authentication check
Next.js API handlers without a recognized authentication check
wildcard CORS
missing baseline browser security headers
suspicious NEXT_PUBLIC_* variables

It can also scan a public URL for:

reachability
HTTPS transport
deployed security headers
wildcard CORS
cookie flags
X-Powered-By disclosure

The scanner is intentionally read-only. URL scans make a public GET request chain, follow redirects, and do not crawl the application, submit forms, or attempt exploitation.

Install and run it locally

git clone https://github.com/CyprianTinasheAarons/vibe-scanner.git
cd vibe-scanner

python3 -m venv .venv
source .venv/bin/activate

python -m pip install -e .
vibe-scanner /absolute/path/to/your/project

To scan a deployed URL:

vibe-scanner https://your-domain.com

To produce a machine-readable report:

vibe-scanner --json /absolute/path/to/your/project > report.json

The JSON output includes:

scanner version
target type
normalized target
severity counts
complete findings
remediation context

Secret values are not included in the findings.

The scanner is a first pass, not a security certificate. Authentication, RLS, CORS, and header checks use static heuristics. A warning needs to be confirmed in the context of the application. A clean scan does not prove that the application is secure.

It means the scanner did not find the specific patterns it checks for.

What the scanner cannot replace

Vibe Scanner does not replace:

a full penetration test
threat modeling
manual authorization testing
business-logic review
dependency risk analysis
compliance certification
an experienced security engineer reviewing the system

That distinction matters. Security tools become dangerous when their output is treated as certainty.

A scanner should reduce the chance of missing obvious problems. It should not create false confidence.

The rule I would use before shipping

If you built an application quickly with Cursor, Bolt, Lovable, v0, Claude Code, or another AI tool, ask one question before you send more traffic to it:

Which parts of this application are trusted because the server verified them, and which parts are trusted because the browser said they were true?

That question leads you to the places that deserve attention:

secrets
database policies
API authorization
object ownership
admin routes
deployment configuration

A working demo is a good start. It is not evidence that the application is ready for strangers, payments, or private data.

Need a human review?

Topiax is building tools and services for teams that want to move quickly without treating security as an afterthought.

For small Next.js and Supabase applications, Topiax offers a Ship Confidence Review:

48-hour turnaround
$750–$1,500 flat fee
focused review of secrets, authentication, authorization, RLS, API routes, and common deployment gaps
plain-English report with prioritized fixes
bounded review, not a penetration test or compliance certification

If you want a human to review the application, Comment SHIP.

Sources

The evaluation test 90% of AI engineers fail

CyprianTinasheAarons — Fri, 17 Jul 2026 09:24:51 +0000

I. Introduction: What Agent and RAG Evaluation Is

Have you ever built a system that worked flawlessly in testing but fell apart the moment it hit production?

Maybe you shipped an AI agent that answered everything correctly in your sandbox and even passed your evaluations, yet once users got hold of it, it started failing in subtle, expensive ways.

Or you wired a RAG component into a multi-agent system, watched it work perfectly in your test harness, and then realised in production that the RAG wasn't being referenced or used correctly at all.

The real cost of an unreliable agent

No matter how costly it is for LLMs to be unreliable in production, it can cost you:

Reputation
Your business
Time
Your valuable customers
And it can even lead to legal action

People still prefer to vibe-check, make weak evaluations, or follow a happy path when running their evaluations. This can cause an LLM to work very well in a test environment with predictable answers but fail in the real world. Here are some examples of when that failure happened.

II. When AI Failures Get Expensive: Real-World Case Studies

One wrong answer from a customer service bot was enough to end up in court.

Not even a working AI agent (or a "reliable" agent, an LLM, or a RAG) is safe from being costly in a production environment. Let me tell you about the Moffatt v. Air Canada case.

Air Canada had a system built on top of an LLM, mainly for customer service. After his grandmother died, Mr. Moffatt used Air Canada's chatbot to ask about bereavement fares. The chatbot told him he could book his flight at full price and apply for the discounted bereavement rate retroactively within 90 days. He did exactly that, and Air Canada refused the refund, pointing out that its actual policy didn't allow retroactive bereavement claims. Moffatt took Air Canada to the BC Civil Resolution Tribunal and won.

Another scenario: a lawyer wanted to use ChatGPT as a reference when putting together a court case. The references and case law it gave him were inaccurate and filled with hallucinations. It ended up costing that lawyer a fine of over $5,000. There are many cases like this. You can look up the link I've shared on some of the legal cases around hallucinations, and take a look at this graph:

Refer to https://www.damiencharlotin.com/hallucinations/?graphs=1

The number of hallucination-related legal cases keeps climbing every year.

III. The Promise of the Article

So what can I promise with this article?

You'll understand what causes LLM agents to hallucinate and why that can cost you real money.

You'll learn, step by step, how to build a more reliable AI agent and run meaningful evaluations, whether you're at a startup or a large engineering org.

You'll get practical tools and resources you can use to go deeper into LLM and agent evaluation.

Let's get started.

Why agent and RAG evaluation is harder than testing a single LLM

Testing a single LLM is relatively straightforward. Often, you can improve prompts or compare outputs against a known "gold" answer. That's manageable because an LLM is essentially a next-token predictor.

Testing multi-agent systems or RAG pipelines is much harder. They have multiple steps, dynamic behaviour, and sometimes autonomous decision-making. You don't always know the exact output you want, or even what the "correct" path to that output should be. There can be many valid paths through the system, which makes it much harder to design robust evaluations.

Still, we have to try because the consequences of shipping untested or poorly evaluated AI systems are now much higher.

Common Types of Failures

a. Agentic logic and execution failures

Three types worth calling out here:

Infinite action loop. Quite common. This is where an agent keeps calling a tool but keeps getting an error and keeps repeating that same failed call over and over, eating your token budget.
Type mismatch. The tool's parameters aren't being sent as expected. Maybe an email tool is expecting a string but keeps getting an integer, so the application hard-crashes or keeps failing. Imagine that happening in production.
State desynchronization and memory loss. In a complex multi-agent workflow, the agent drops critical context required to answer the prompt properly and ends up giving the user a vague, generic answer that ignores the full original prompt.

b. RAG & retrieval failures

The most common one is garbage in, garbage out. You've given it poor information, done a poor chunking strategy, or fed it poor data. You end up retrieving irrelevant documents, and sometimes it will say it doesn't know. Worse, it will confidently give you an answer that has nothing to do with what you expected.

An example: it gives you an answer for, say, a funeral policy FAQ, but the wrong answer (a hallucination), because even though you ingested the FAQs, they weren't chunked and vetted properly into the system.

Another common one is the "lost-in-the-middle" effect. This happens when you've provided a very large context, bloating the prompt. Due to that massive context window, the model will simply ignore some of the user's actual instructions and focus on either the very beginning or the very end of the prompt. Obviously, this results in an answer that doesn't quite make sense.

c. Silent failures

These are the worst-case-scenario type of failures, because sometimes you won't even notice them, especially when you're doing evaluations or vibe checks. They only become visible once, say, a power user runs into one, even though the system has already been in production for a while. Unlike traditional software, it won't throw a loud break. You'll probably get a status 200, and everything will look flawless.

The agent quietly executes every update, but maybe it's updating the wrong client record in the DB. Another scenario: it's supposed to send an email to Person A, but instead sends it to Person B, an unrelated prospect. These are the silent failures that are hardest to catch, because the system still "works." It's still sending; it's still updating. It's just doing it on the wrong data.

IV. Deep Dive: What Causes AI Agents to Fail

In practice, most failures come from a few recurring patterns:

Not testing at all
Testing only the happy path
Passing evals but still failing in production
Not having a continuous eval pipeline

1. Not testing at all

This doesn't necessarily mean you don't test at all. Most AI engineers (at least 90% of the engineers I've dealt with) put most of their effort into making sure everything works, and only a small fraction of their time, maybe 10% or less, goes toward testing.

What they usually do instead is a vibe check: throw in maybe 10-20 prompts and just see if the system produces an output. Sometimes they don't even check the quality of that output:

Is it fair?
Is it legally risky?
Is it following the prompt?
Is anyone scoring that response?

2. Testing only the happy path

The happy path is where tests actually happen, but they're only happy-path tests. There are no edge cases, no red teaming done, and the questions are not adversarial. Everything is quite easy. Things like "What's my name?" or "What is X?" Predictable stuff.

The happy path will always pass, and it's only when the edge cases show up in production that people panic.

3. Passing evals but still failing in prod

This is the scary one, because this is where you've put in the work and done proper evaluations in the past. So why is it still failing? This is where most AI engineers get caught out. A few reasons:

We might not be looking at the correct thing. We might just be looking at the final output. If it's an AI agent, you also want to look at whether it followed the right steps to get to that answer. "Correct" and "acceptable" might be different things.

This is what I call the wrong-path-to-right-answer problem: you got the right answer, but through the wrong process. For example, in a weather application, instead of using the weather API, the agent might hallucinate the answer or use a completely different tool to "find" the weather in San Francisco. Without cross-checking, you'd assume the agent is working correctly.

This is shortcut behavior. Without proper reasoning and the correct steps, it can still produce a correct answer. The problem is that it won't always produce the right answer because it isn't consistently using the tool you actually wanted it to use.

Same right answer, but only one of these two paths actually called the weather API.

The agent skips a required step. There might be steps the agent is required to follow. For example, before sending an email to a prospect or lead, you might want it to:

Find details about that prospect.
Call the function that pulls that prospect's profile.
Use a prompt that drafts and sends the email.
After sending, update the pipeline so the prospect moves from "lead" to "outbound lead."

The problem is that the agent might send the email and update everything else, but skip the step of actually finding the prospect's details first, so it's working off incomplete context. That can cost your business a lot if you're in the business of securing new prospects.

The high-confidence wrong answer. This is where the agent is very confident, but totally wrong. This ties back to RAG: if your data wasn't chunked properly, the retriever can still pull documents, and the agent can still treat them as relevant to the prompt, even when they're not.

Everything appears to work, but the answer is wrong. An example: a large organization with FAQ documents for different products, say motor insurance and microloans, where only the microloan documents were chunked and indexed properly. A customer service agent works across all of those entities at once could confidently give a wrong answer to someone asking about their motor insurance.

Next, I will walk you through how to design a eval workflow that catches these failure modes before they hit production.

V. The Ideal Workflow for Dealing with Agents and RAG

This is the workflow you'd use if you want to stop vibe-checking and actually turn your AI project from a hobby into a solution that has real impact in production and is highly reliable.

Six steps, one loop: evaluation is a flywheel, not a one-time checklist.

1. Build a custom evaluation dataset

It might be tempting to purchase an existing dataset or use an open-source one. The issue is these datasets aren't fully aligned with what you're building in most cases; they have different benchmarks. It's better to curate your own dataset from your own real inputs, or generate something synthetic based on your own knowledge base.

2. Turn traces into a dataset

If your application is already in production, for example a complex agent you're already running traces on using DeepEval, Langfuse, or LangSmith, you can use those traces to build a dataset that's more aligned with reality: the edge cases, what's been catastrophic, and what's succeeded.

3. Pick the right balance of eval metrics

Don't start off with more than 10 metrics in most cases. Pick just a few, because you're not trying to boil the ocean. Things worth considering:

Layer 1: deterministic metrics / code-level checks. Did it run? Did it return valid JSON? Was the latency low?
RAG-specific metrics. Did it retrieve the proper context, and was the answer faithful to it?
Layer 3: LLM as a judge. Using a separate, isolated model to score subjective quality against a strict rubric.

4. Run the evals

Now for the fun part: executing your evaluations systematically. Use different evaluation and orchestration tools (DeepEval, Ragas, LangSmith, etc.), depending on what you prefer and know, to run your target agent against the dataset you prepared.

5. Add evals to your pipeline (CI/CD)

Offline evaluations are great, but you also want online evaluations and a pipeline that treats your prompts and agent workflows like any other source code. When you change your model, say from GPT-5.5 to GPT-5.6, you want to be able to test things like faithfulness and see if the score dropped or improved, automatically, without human involvement, triggered on every PR.

6. Tracing feeding back into evals

This is what I call the flywheel. You want your production loop to be complete, able to update your dataset based on what's happening in real time, based on your traces. Rule number one: never let your evaluation dataset stay static.

Common LLM Eval Mistakes

Here are some common LLM evaluation mistakes teams make, especially early on. I call this the vibe-check trap.

Four traps that quietly sink most eval efforts before they even start.

Doing too much, too early. It's tempting to over-engineer the early stages of evaluation, 20 metrics, 100,000 synthetic data points, but that just burns your tokens and your team's engineering time. It's better to start small: 20-25 highly curated test cases with 3-4 high-value metrics tied to your actual business logic.
Focusing on the happy path (and ignoring operational realities). It's easy to write perfectly formatted questions with easy answers and test against those. We tend to forget that production gets messy, and end up testing the easy part while ignoring the adversarial edge cases. The agent looks great on paper, but the test isn't actually testing what will happen in production.
Not having a workflow, and relying on "vibe checks." This is the biggest and most common AI engineering trap: a handful of prompts tested by yourself in a playground, "yeah, looks good," then straight to production. No edge cases, no red teaming, just a subjective vibe check. It's unscalable, and it more or less guarantees silent failures down the line.
Having fixed evals that don't evolve with production. Production isn't just messy, it's dynamic and constantly changing. Models change, models drift, APIs change, user behavior changes. Testing the same 25-item dataset for a year after it's gone stale won't give you a sustainable long-term approach. What you need is a production-grade system that monitors change (those traces we talked about earlier) and feeds them back into the dataset.

VI. Different Tools You Can Use for Agent Evaluation

Three jobs, three tool categories: build it, score it, watch it.

Agent orchestration with built-in state: LangGraph (excellent for testing specific nodes), AutoGen, CrewAI.

RAG & agent metrics: Ragas, DeepEval, TruLens, LangSmith.

Agent tracing & observability: LangSmith (crucial for viewing multi-step agent thought processes), Datadog LLM Observability, DeepEval.

VII. Resources

LangSmith Evaluation - Docs by LangChain

docs.langchain.com

LangSmith evaluation allows for both offline and online evaluations. It's a good resource and tool if you're a fan of the LangChain ecosystem, like me.

Introduction | DeepEval - The LLM Evaluation Framework

DeepEval is the open-source LLM evaluation framework for testing and benchmarking LLM applications.

deepeval.com

DeepEval also provides great resources on the topic and tooling around LLM evaluation. I find it to be a very competent resource that will change how you see and do evaluations.

Further reading:

Chapter 3, Evaluation Methodology, in AI Engineering by Chip Huyen.
Chapter 10, Evaluating LLM Applications, in Prompt Engineering for LLMs by John Berryman & Albert Ziegler.
Chapter 7, Evaluating LLMs, in LLM Engineer's Handbook by Paul Iusztin & Maxime Labonne, for a practical, hands-on take.

References

Moffatt v. Air Canada, 2024 BCCRT 149 (CanLII). British Columbia Civil Resolution Tribunal. https://canlii.ca/t/k2spq
McCarthy Tétrault, "Moffatt v. Air Canada: A Misrepresentation by an AI Chatbot." https://www.mccarthy.ca/en/insights/blogs/techlex/moffatt-v-air-canada-misrepresentation-ai-chatbot
American Bar Association, "BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot," Business Law Today, February 2024. https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/
Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023). https://en.wikipedia.org/wiki/Mata_v._Avianca,_Inc.
Legal Clarity, "Mata v. Avianca: Fake Cases, ChatGPT, and Sanctions." https://legalclarity.org/what-happened-in-the-mata-v-avianca-case/
Damien Charlotin, AI Hallucination Cases Database (tracking legal cases involving generative AI hallucinations). https://www.damiencharlotin.com/hallucinations/?graphs=1
LangChain, LangSmith Evaluation Documentation. https://docs.langchain.com/langsmith/evaluation
DeepEval, Tutorial Introduction. https://deepeval.com/tutorials/tutorial-introduction
Huyen, Chip. AI Engineering, Chapter 3: Evaluation Methodology. O'Reilly Media.
Berryman, John & Ziegler, Albert. Prompt Engineering for LLMs, Chapter 10: Evaluating LLM Applications. O'Reilly Media.
Iusztin, Paul & Labonne, Maxime. LLM Engineer's Handbook, Chapter 7: Evaluating LLMs. Packt Publishing.

I Built a Payment Gate That Never Sees Your Balance 🔐⚡

CyprianTinasheAarons — Fri, 03 Jul 2026 20:48:47 +0000

Why Zero-Knowledge Payments Matter Right Now

"Send me your bank statement so I can confirm you can afford this."

I hear some version of that every time remittance compliance comes up in my work.

And honestly… I get why it exists. Nobody wants to release funds to a sender who can't cover them.

But here's the problem: proving you have enough money has always meant showing exactly how much money you have.

Your balance.
Your full statement.
Your whole financial life, just to move one payment.

You shouldn't have to show your hand to prove you can play the round.

With 24 hours left on the clock for Stellar Hacks: Real-World ZK, I shipped zkRemit Guard — a Stellar smart contract that releases an escrowed payment only after the sender proves, with math, that their balance clears the required amount.

The raw number never touches the chain.
Not once!

What You'll Build 🚀

A Noir circuit that proves balance >= required_amount without revealing balance
A Poseidon2 commitment that locks the proof to one specific balance, so it can't be faked after the fact
A Soroban contract (payment_gate) that escrows real tokens and only releases them after verifying the proof on-chain
Anti-replay binding so a valid proof from one transfer can never be reused on another
Pass and fail demo scripts that prove the gate actually rejects bad proofs, not just accepts good ones

Prerequisites

nargo 1.0.0-beta.9 (Noir's compiler — install via noirup)
bb 0.87.0 (Barretenberg, the UltraHonk proving backend — install via bbup)
stellar-cli ^3.2.0 for deploying and invoking Soroban contracts
Rust + the wasm32v1-none target
Docker, if you want a localnet before you touch testnet
Clone the repo https://github.com/CyprianTinasheAarons/zkremit-guard so you can follow along with full context

Step 1: Write the Circuit

This is the whole idea, in 14 lines:

fn main(
    balance: u64,
    salt: Field,
    transfer_id: pub Field,
    required_amount: pub u64,
    balance_commitment: pub Field,
) {
    assert(balance >= required_amount);

    let computed_commitment = Poseidon2::hash([balance as Field, salt, transfer_id], 3);
    assert(computed_commitment == balance_commitment);
}

balance and salt have no pub keyword. They never leave the sender's machine.

transfer_id, required_amount, and balance_commitment are the only three numbers that ever reach the chain.

Two asserts, two guarantees:

The balance actually clears the bar.
The balance used in this proof is the same one committed to earlier — not a bigger number invented on the spot.

Step 2: Bind the Secret to a Commitment

Here's the thing most people building their first ZK demo skip:

A private input with no binding is just a number nobody can check.

Before proving anything, the sender commits to their balance with a Poseidon2 hash — a one-way seal. salt keeps two people with the same balance from producing the same public commitment.

Prover.toml
  balance = "120"
  salt = "7"
  transfer_id = "1"
  required_amount = "50"
  balance_commitment = "1121312...5850"   # computed, not guessed

That commitment gets computed in Rust — using the exact same Poseidon2 hash the circuit and the contract both use — and written straight into Prover.toml. Three pieces of code, one shared source of truth.

Step 3: Compile and Prove

nargo check
nargo compile
nargo execute

bb prove --scheme ultra_honk --oracle_hash keccak \
  --bytecode_path target/reserve_threshold.json \
  --witness_path target/reserve_threshold.gz \
  --output_path target --output_format bytes_and_fields

bb write_vk --scheme ultra_honk --oracle_hash keccak \
  --bytecode_path target/reserve_threshold.json \
  --output_path target --output_format bytes_and_fields

nargo compiles your circuit and runs it once on real numbers.
bb generates the actual zero-knowledge proof — UltraHonk, the scheme Stellar's Protocol 26 host functions were built to verify cheaply on-chain.

Output: a proof, a vk (1,760 fixed bytes — the circuit's public fingerprint, reusable across every future transfer), and public_inputs.

Step 4: Build the Payment Gate as a State Machine

Every transfer moves through exactly one path:

PendingProof → ProofVerified → Released

Three functions drive it:

pub fn create_transfer(env: Env, sender: Address, recipient: Address, amount: i128, transfer_id: BytesN<32>) -> Result<(), PaymentGateError> {
    sender.require_auth();
    // ...escrows tokens, status = PendingProof
}

pub fn submit_proof(env: Env, sender: Address, transfer_id: BytesN<32>, required_amount: i128, balance_commitment: BytesN<32>, public_inputs: Bytes, proof_bytes: Bytes) -> Result<(), PaymentGateError> {
    // rebuild expected public inputs, reject any mismatch, then verify the proof
}

pub fn release_transfer(env: Env, sender: Address, transfer_id: BytesN<32>) -> Result<(), PaymentGateError> {
    // only pays out if status == ProofVerified
}

create_transfer locks tokens into escrow. Money moves out of the sender's wallet, but nowhere near the recipient yet.

Step 5: Wire in the Anti-Replay Check (The Part People Skip)

Before the contract touches any cryptography, it does something cheaper first:

let expected_public_inputs = expected_public_inputs(
    &env, &transfer_id, required_amount, &balance_commitment,
)?;

if public_inputs != expected_public_inputs {
    return Err(PaymentGateError::PublicInputsMismatch);
}

This rebuilds, byte-for-byte, what the public inputs should say for this specific transfer and rejects anything that doesn't match exactly.

Why does this matter?

A valid proof with no binding to a transfer ID is a proof anyone can replay anywhere.

This one line is what stops that.

Only after this check passes does verify_proof() run the real math — parsing the proof, rebuilding the Fiat-Shamir transcript, running sumcheck, and closing it out with a pairing check via Shplemini. If any of it fails, the transfer stays stuck in PendingProof. Money never moves.

Step 6: Run the Happy Path

./scripts/deploy_local.sh
STELLAR_NETWORK_NAME=local ./scripts/demo_pass.sh

Three contract calls, in order:

create_transfer → submit_proof → release_transfer

Sender escrows 50 tokens. Proof verifies. Recipient gets paid.

The sender's balance of 120 never appears anywhere on-chain — not in an event, not in storage, not in a log.

Step 7: Prove the Fail Path Actually Fails

This is the step that separates a real demo from a slide deck.

printf '\x01' | dd of=proof.bin bs=1 seek=100 conv=notrunc

One corrupted byte. Same transfer context. Run it:

create_transfer  → ✅ succeeds
submit_proof     → ❌ rejected
release_transfer → ❌ blocked (status still PendingProof)

Escrowed funds stay locked when the proof is bad. They don't pay out anyway "just in case."

That's the whole point of a proof gate — not that it accepts good proofs, but that it refuses bad ones under real economic stakes.

Step 8: Put a Control Panel in Front of It

I'm not going to pretend this is a full wallet-connected dApp — it isn't, and saying otherwise to judges is the fastest way to lose credibility.

ui/ is a Next.js page that streams the same shell scripts' output into the browser live. Click a button, watch create_transfer → submit_proof → release_transfer happen in real time instead of scrolling a terminal.

Not client-side proof generation. Not Freighter wallet integration. A control panel for a CLI-first demo. Said plainly, upfront, every time.

In Conclusion

If you're building anything that needs to prove a fact about private data a balance, a credential, an age, a KYC tier without leaking the underlying number, this is the shape of the answer: circuit proves the fact, contract checks the proof, chain never sees the secret.

Your Turn 👇

What's the first private fact you'd want a smart contract to verify without ever seeing it?

A balance? A credential? A KYC tier?

Drop it below 👇

Let's build the boring, load-bearing infrastructure nobody's hyping yet 😄

Resources

Stellar Developer Docs | Stellar Docs

Navigating the docs

developers.stellar.org

Noir | Noir Documentation

Noir is an open-source, Rust-influenced domain-specific language for writing privacy-preserving programs with zero-knowledge proofs, requiring no prior knowledge of the underlying mathematics or cryptography.

noir-lang.org

CyprianTinasheAarons / zkremit-guard

zkremit-guard

zkRemit Guard is a Stellar proof-gated escrow demo.

It demonstrates:

a Noir reserve-threshold proof
UltraHonk proof generation with nargo 1.0.0-beta.9 and bb 0.87.0
on-chain Soroban verification
a real escrowed token transfer that only releases after proof verification
pass/fail localnet and testnet demo flows

Status

The top-level MVP path is implemented and runnable.

Implemented:

circuits/reserve_threshold uses Poseidon2 commitment binding
contracts/payment_gate stores the VK at deploy time and verifies proofs on-chain
create_transfer escrows demo tokens into the contract
release_transfer pays escrowed tokens to the recipient
scripts/ runs local/testnet deploy and pass/fail demos

Still useful follow-ups:

replace demo token issuance with your intended production asset model
polish contract events and indexing
expand docs/UI beyond the CLI-first hackathon flow

Layout

zkremit-guard/
  README.md
  demo/
  circuits/
  contracts/
  scripts/
  ui/
  docs/

Suggested next steps

Build proof artifacts with ./scripts/build_proof.sh.
Deploy locally with ./scripts/deploy_local.sh.
Run the happy path with STELLAR_NETWORK_NAME=local ./scripts/demo_pass.sh.
Run the fail…

View on GitHub

roadmap.sh

Rust Tutorial

Rust is a modern systems programming language developed by the Mozilla Corporation. It is intended to be a language for highly concurrent and highly secure systems. It compiles to native code; hence, it is blazingly fast like C and C++.

tutorialspoint.com

[Boost]

CyprianTinasheAarons — Thu, 11 Jun 2026 08:21:47 +0000

CyprianTinasheAarons

Jun 10

Your AI Demo Is Not a Product!

#ai #startup #webdev #programming

3 min read

Your AI Agent Is Failing in Production

CyprianTinasheAarons — Wed, 10 Jun 2026 06:52:33 +0000

The demo is impressive. ✅

The demo works in your environment, with your data, with you watching. ✅

Production?

Silent failures. Cost overruns. Wrong tool calls. Stuck loops. No fallback. ❌

Agents in 2026: The Real Problem

Here is the thing most people are not talking about when they ship AI agents:

A demo agent and a production agent are completely different things.

A demo is: "watch this work once."

A production agent is: "what happens when it is wrong, stuck, expensive, over-permissioned, or called 10,000 times by real users?"

That second question is what separates a cool technical proof-of-concept from something a business can actually rely on.

Demos are not systems.

1️⃣ The 7 Things That Break in Prod

In every agent hardening sprint I run, the same failures show up:

Failure Mode	What It Costs
No logging	You have no idea what the agent did or why
No eval set	You cannot measure quality or catch regressions
Unlimited tool access	Agent calls tools it should never touch
No retry logic	Transient failures become permanent failures
No memory rules	Context leaks between sessions or inflates cost
No fallback path	Agent loops or crashes instead of escalating
No cost checks	1 misconfigured prompt → $400 API bill overnight

If your agent is in production with 3 or more of those missing — you are one bad prompt away from a very expensive incident.

2️⃣ The Production Hardening Checklist

Before you call an agent production-ready, run through this:

Eval set exists — at least 20 test cases covering happy path + edge cases
Structured logging — every tool call, every input, every output, every error — logged and searchable
Retry logic — transient API failures handled gracefully, not crashed
Tool limits — agent cannot call tools outside its defined scope
Memory rules — what carries over between sessions, what gets cleared, how context is compressed
Fallback paths — when the agent gets stuck or uncertain, it has an exit: escalate to human, return partial result, surface an error
Cost checks — token budgets enforced, alerts on spend spikes, expensive calls rate-limited
Human review gates — high-stakes decisions require confirmation before action That is not over-engineering. That is what makes an agent trustworthy enough to deploy.

3️⃣ The Eval Set Is the Most Skipped Step

I see this every time.

Founders ship agents without a single structured test case.

Then they notice inconsistent behavior in prod.

Then they fix one thing, break another, and have no way to tell whether the fix made things better or worse.

An eval set does not have to be complex. Start with:

5 happy-path inputs where the right answer is obvious
5 edge cases where the agent should gracefully fail or escalate
5 adversarial inputs where the agent should refuse or ask for clarification
5 cost-sensitive inputs where the expected response should be short 20 evals. Run them after every change. That is the minimum.

You prompt → agent responds → eval catches the regression → you fix it → you know the fix worked 🚀

4️⃣ The Cost That Sneaks Up on You

Here is the one most people learn the hard way:

An agent with 50+ tool calls per request, no cost checks, and no rate limits will hit a $1,000+ API bill in a weekend from legitimate-looking traffic.

Not a bug. Not a hack. Just: users engaging, agent running, costs accumulating silently.

The fix is boring:

Token budgets per request
Hard limits on tool call chains
Spend alerts at $50, $100, $250
Expensive tools gated behind confirmation That is infrastructure. Not rocket science. Just discipline.

The Real Bottom Line ⚡

Your agent working in a demo is not your agent working in production.

Production means: wrong inputs, repeated calls, unexpected users, cost pressure, and no one watching.

Harden it before you ship it. Evals, logging, retry logic, tool limits, memory rules, fallback paths, cost checks.

The $3,500–$12,000 hardening sprint is almost always cheaper than the incident that follows from skipping it.

Your Turn 👇

Which of the 7 failure modes is your current agent missing?

Or — have you had a prod incident that cost you time, money, or trust?

Drop the war story below 👇 — let's build the knowledge base together 😄

Proof Compounds. Claims Decay.

CyprianTinasheAarons — Wed, 10 Jun 2026 06:52:12 +0000

Here is the move most technical service providers miss:

Every project you deliver quietly dies inside a private folder.

Every project you deliver with receipts becomes a trust asset that sells the next sprint without you lifting a finger.

The Insight Almost No One Acts On

Delivery is not the end of marketing.

Delivery is where the next marketing asset is born.

The before/after screenshot. The launch-readiness report excerpt. The workflow map. The metric improvement. The buyer quote.

All of that is proof. And proof is the compound interest of service work.

Claims decay. Proof compounds.

1️⃣ What Proof Actually Looks Like

This is the proof asset menu. Every sprint should produce at least 1 item from this list:

Before/after screenshot — the most shareable format
Launch-readiness report excerpt — shows rigor and standard
Workflow map — visual, specific, credibility-dense
Dashboard screenshot — metrics that moved
Test checklist — shows what was verified, not just what was built
Client quote — even 1 sentence is worth 1,000 words of claims
Metric improvement — "response time dropped from 24 hours to 4 minutes"
Public teardown — anonymous version of the diagnosis
Case study — structured story: context → pain → fix → result
One-minute walkthrough video — screen-recorded, narrated, personal You do not need all of them. You need 1 per sprint.

2️⃣ The Case Study Structure That Sells

A case study is not a trophy. It is a reusable trust asset.

Use this structure every time:

1️⃣ Context — who had the problem? (anonymized if needed)
2️⃣ Pain — what was it costing them?
3️⃣ Hidden cause — what was really broken underneath?
4️⃣ Fix — what did you change, specifically?
5️⃣ Result — what improved? With a number.
6️⃣ Proof — what artifact backs it up?
7️⃣ Lesson — what should similar buyers do next?

That is 7 steps. The whole thing can fit in a LinkedIn post or a page section.

And here is the thing most people are not talking about: a case study with a specific number outperforms 10 posts about your process.

"Automated the follow-up system" — nice.

"Response time dropped from 24 hours to 4 minutes. Booked 3 additional calls in the first week." — that is a sprint someone will pay $3,500 for.

3️⃣ The Handover That Builds Trust

Never finish with "done."

Finish with:

What changed — be specific, not general
Why it matters — business impact, not technical detail
How to use it — what the buyer should do with it
What was tested — so they know the work was real
What still carries risk — honest, not alarming
What the next sprint should be — already positioning the next sale That is how delivery becomes trust — and trust becomes a referral, a case study, and a higher-priced next sprint.

4️⃣ The Proof Stack That Replaces Cold Outreach

Once you have 3 strong proof assets from 3 delivered sprints:

Your outbound changes completely.

Instead of: "Hi, I build AI automations."

You send: "Saw your app. Here's a launch-readiness report from a similar product I hardened last month. Want me to run the same audit on yours?"

The proof does the selling. You do the observing.

That is the machine: skill → sprint → proof → outbound that leads with evidence → easier close → another sprint → stronger proof.

It is infrastructure that compounds.

The Real Bottom Line ⚡

Every delivery that ends with "done" and a Slack message is a missed marketing opportunity.

Every delivery that ends with a report, a screenshot, a client quote, and a handover video is a trust asset that sells the next sprint without you having to start from zero.

Build the proof stack. Post it. Send it. Let it work while you sleep.

Your Turn 👇

What is the best proof asset you have created from a project — and did you post it publicly?

Drop it below 👇 — let's compare what has moved the needle 😄

The Sales Call Is Not a Performance

CyprianTinasheAarons — Wed, 10 Jun 2026 06:52:05 +0000

I have watched founders lose sales calls they should have won.

Not because they lacked skill. Not because the offer was wrong.

Because they walked in to prove they were smart — instead of finding out whether the pain was real.

Sales Is Diagnosis Plus Decision

The call is not there for you to pitch.

The call is there to find out:

Is the pain real?
Does the buyer have urgency?
Does the budget exist?
Can a fixed-scope sprint create a clear win? That is it. Four questions. Everything else follows from those.

Sales is not pressure. Sales is diagnosis plus decision.

1️⃣ The Call Structure That Works

Frame the call in the first 60 seconds:

"I'll understand the current state, ask what is costing you, then tell you whether a sprint makes sense. If it doesn't, I'll say so."

That sentence does 3 things:

Sets expectations — no pressure, no hard close
Signals competence — you have done this before
Removes the buyer's guard — they can be honest about what is broken Then run this flow:

1️⃣ Current state — what exists now?
2️⃣ Pain — what is broken or slow?
3️⃣ Cost — what does it cost in time, money, trust, or delay?
4️⃣ Urgency — why now?
5️⃣ Decision — who approves?
6️⃣ Success — what would make this worth paying for?
7️⃣ Close — recommend the sprint or walk away

2️⃣ The Questions That Reveal Money

These are the 6 questions I use to find whether a sprint is worth recommending:

"What happens if this stays broken for another 30 days?" — reveals urgency and cost
"What have you already tried?" — reveals how serious they are
"Where does the current process lose leads, users, time, or trust?" — reveals the money leak
"Who feels this pain most inside the business?" — reveals whether the buyer is also the decision-maker
"What would make this an obvious win?" — reveals success criteria before you price
"If we fixed only one thing first, what would matter most?" — reveals scope Listen for the answer with the money in it.

That is the thing you fix. That is what you price. That is the sprint.

3️⃣ How to Close Without Begging

Most founders either close too hard or too soft.

Too hard: "So are you ready to move forward today?"

Too soft: "Let me know if you want to chat more sometime."

Both lose.

Here is the close language that works:

"Based on what you told me, the best first move is not a huge rebuild. It is a fixed-scope sprint focused on the highest-risk bottleneck. I'll send a proposal with scope, timeline, price, what is excluded, and what success looks like."

No begging. No rambling. No discount panic.

Send the proposal within 24 hours of the call. Every hour you wait, urgency fades.

4️⃣ When to Walk Away

Not every call ends in a sprint.

Walk away when:

The pain is not expensive enough to justify action
The urgency is performed, not real — "we'll definitely do this next quarter"
The decision-maker is not on the call
The buyer wants open-ended help — not a defined outcome Walking away is not a lost sale. It is protecting your delivery capacity for buyers who have real pain and real urgency.

And honestly… a clean walk-away often triggers a re-engagement 2 weeks later when the problem gets worse.

The Real Bottom Line ⚡

The best sales call is the one where you ask the right questions, listen for the money, and either recommend a clear sprint — or tell them it is not the right fit.

No performance. No deck. No convincing.

Just diagnosis and decision.

That is how conversations become cash — without either person feeling gross about it.

Your Turn 👇

What is the hardest part of a sales call for you — the opening, the pricing, or the close?

Drop it below 👇 — I'm curious where the breakdown usually happens 😄

Outbound Without Begging

CyprianTinasheAarons — Wed, 10 Jun 2026 06:51:56 +0000

Cold outreach fails when it feels like a stranger asking for your time.

It works when it feels like a useful operator noticed a real problem and offered a small, low-risk next step.

The difference is almost always structure. Not charisma. Not volume.

Structure.

The Conversation I Keep Having

I see founders send 100 cold DMs with zero replies.

Then I read the messages.

"Hi [Name], I help businesses with AI automation. Would love to connect and explore synergies."

That message fails on every line:

No context — why this person, why now?
No observation — what did you actually notice?
No value — what is the useful thing?
No risk — what is the low-friction next step?
No diagnosis — you sound like you want something, not like you see something Here is the thing most people are not talking about: the DM that gets a reply is the one that feels like it was written about the specific person reading it.

1️⃣ The Outbound Formula

Every message that works uses this structure:

Part	Purpose
Context	Why this person, why now?
Observation	What did you actually notice about their product/page/content?
Risk or opportunity	What might be costing them that they haven't seen?
Useful next step	Checklist, teardown, quick audit — something useful
Light CTA	Easy to answer — not a marriage proposal

The goal is not to sell on the first message. The goal is to be useful enough that they want the next thing you send.

2️⃣ Three Messages That Actually Work

AI app launch opener:

"Saw your launch. Before adding more features, I'd check the hidden trust risks: auth, payments, logging, analytics, and onboarding. Want the launch-readiness checklist?"

GTM system opener:

"Your product looks useful, but the path from attention to booked calls feels thin. I can map the missing GTM system — want a quick look?"

Workflow automation opener:

"There's probably money leaking between first enquiry and follow-up. I can show you the 7-day missed-lead recovery workflow if helpful."

Notice what they all share:

Specific — not "I help with AI"
Observational — something you actually noticed
Useful — they get something even if they say no

- Low-friction — "want the checklist?" is a yes/no, not a commitment

3️⃣ The Follow-Up Cadence That Does Not Annoy

Most outbound fails not at the first message — but at the follow-up.

Either people never follow up at all.

Or they send "Just following up on my last message" — which is the DM equivalent of poking someone in the shoulder with no reason.

Here is the cadence that works:

Day 0 — contextual opener with observation and useful thing
Day 2 — one useful checklist or additional observation (no ask)
Day 5 — small proof or example: screenshot, result, before/after
Day 10 — close the loop cleanly: "Not a fit right now — happy to reconnect later."
Day 21 — re-engage only if there is a new trigger: new launch, new post, new product 5 touches. 21 days. Then move.

If someone is not ready, that is not a no. It is a timing mismatch. The pipeline exists to capture the right people at the right moment — not to convince the wrong people with more pressure.

4️⃣ The Comment That Opens the Conversation

Before the DM — the comment.

A thoughtful comment on the right post, published before the DM, warms the approach.

You are no longer a stranger. You are the operator who left the useful diagnosis in the thread.

That shift — from unknown to recognized — cuts the friction on the DM by half.

Comment first. DM second. Offer third.

You prompt → observation left in comment → DM sent with context → conversation starts → teardown offered → sprint sold 🚀

The Real Bottom Line ⚡

Outbound without begging is outbound that leads with value, not asks.

The message that gets a reply is the one where the reader thinks: "This person actually looked at my thing."

Do the research. Write the observation. Send the useful thing first.

Then ask for the conversation.

Your Turn 👇

What is the worst cold DM you have ever received — and what made it so bad?

Drop it below 👇 — let's learn from the horror stories 😄

The Million-Dollar Math Is Boring

CyprianTinasheAarons — Wed, 10 Jun 2026 06:51:47 +0000

A million dollars is emotional as a dream.

As math, it is boring. And that is exactly why most people never get close.

Break It Down

Here is the thing: $1M/year is not one big bet.

It is a machine. And machines are built from boring, repeatable components.

20 clients at $50,000? That is $1M.
100 clients at $10,000? That is $1M.
12 retainers at $4,000/month? That is $576k — plus 4 sprints at $10,000 each gets you to $616k. The question is not whether the number is possible.

The question is which machine can realistically produce it — from where you actually stand today.

1️⃣ The Practical Ladder

Here is how the staged path actually works for an AI service business:

Stage	What You Are Doing	Why It Matters
Stage 1	Sell fixed-scope sprints	Creates cash and proof
Stage 2	Turn repeated sprint work into templates, SOPs, automations	Reduces delivery time, increases margin
Stage 3	Sell retainers around highest-demand system	Predictable monthly cash
Stage 4	Productize repeated workflow into software or toolkit	Scalable without more hours
Stage 5	Scale the thing the market already proved it wants	Compound the machine

Notice what is missing from Stage 1.

There is no SaaS. No product. No cold paid traffic. No team.

Just skill, packaged cleanly, sold to people with money and a painful problem.

That is the fastest path — not the most glamorous one.

2️⃣ The Proof-of-Force Line

The first mission is not $1M.

The first mission is $10k/month — reliably, from sprint work.

Here is what that actually looks like:

2 × $1,500 teardown/audit packages = $3,000
2 × $3,500 implementation sprints = $7,000
2 × $5,000 launch/GTM sprints = $10,000
3 × $2,000 retainers = $6,000/month That is not the finish line. It is the proof-of-force line.

It proves the machine works. It funds the next iteration. It creates the case studies that make the next sprint easier to sell.

Then you go from $10k/month to $25k. Then $50k. Then you make the productization decision from a position of demand — not hope.

3️⃣ The Brutal Filter

Every week has to be measured against 5 buckets:

1️⃣ Creates qualified attention
2️⃣ Starts sales conversations
3️⃣ Improves the offer
4️⃣ Delivers paid outcomes
5️⃣ Turns work into proof

If the week was busy but produced none of those — that week does not count.

A week can feel intense and still generate no pipeline, no proposal, no proof, and no cash movement.

Drift is the enemy.

Not the lack of skill. Not the market. Drift.

4️⃣ What $10k/Month Actually Requires

People underestimate how few moving pieces this needs.

At $10k/month you need:

1 clear offer — buyers understand what they are purchasing in 10 seconds
1 functioning pipeline — at least 5 real conversations happening at any time
1 content system — useful posts going out daily, creating trust before the call
1 delivery standard — work produces proof, not just output
1 pricing boundary — no free overruns, no vague scope, payment before deep work That is 5 systems. Not 50.

And honestly… most founders skip all 5 and wonder why the number never moves.

The Real Bottom Line ⚡

The million-dollar path is not "learn more tools."

It is:

Pick a buyer with a painful, expensive problem → package one outcome they understand → build enough proof that trust starts before the call → create daily distribution → sell fixed-scope sprints → deliver with receipts → repeat until the market remembers what you are for.

That is the machine.

Build the machine.

Your Turn 👇

Where are you on the practical ladder right now?

Stage 1? Still figuring out the offer? Already at retainers?

Drop it below 👇 — let's compare notes 😄

Distribution Is the Moat — And Most Technical Founders Have None

CyprianTinasheAarons — Wed, 10 Jun 2026 06:51:39 +0000

Products are easier to build.

Workflows are easier to automate.

Content is easier to generate.

But trust is not easier. Attention is not easier. Buyer memory is not easier.

The Hard Truth

Here is the thing most people are not talking about in 2026:

The bottleneck is no longer the product. The bottleneck is whether the right buyer has seen your diagnosis 3 times in 2 weeks.

Because that is how trust is built. Not with one perfect post. With repeated, useful presence in the right feed.

Distribution is the moat.

1️⃣ Why "Staying Active" Is the Wrong Goal

Most founders post to stay active.

That is not a content strategy. That is anxiety dressed up as marketing.

Every post should do one of 3 things:

Make the buyer understand a pain they already have
Make the buyer trust your diagnosis of that pain
Move the buyer closer to a conversation A post about your tech stack? Probably none of those.

A post that says "Your AI app is not launch-ready until auth, payments, logging, and rollback are boring" — that does all 3.

2️⃣ The Five Content Pillars That Build Pipeline

Here is the system I use. 5 pillars. Everything maps to one of them:

Pillar	What It Signals
Launch risk	Why AI-built products break before production
GTM systems	How founders turn expertise into pipeline
Workflow automation	How businesses leak time and revenue
Proof and case studies	What changed before/after — with receipts
Founder operating lessons	The discipline behind building for money

Every post I write maps to one of these.

Not because it is tidy. Because each pillar speaks directly to a buyer who has a specific pain — and positions me as the operator who sees it clearly.

3️⃣ The Daily Format That Creates Pipeline

This is the actual weekly posting structure that works:

Monday — mistake post: a painful thing technical founders do wrong
Tuesday — teardown post: a real example dissected publicly
Wednesday — checklist: the 10-item audit your buyer needs
Thursday — before/after: what changed after a sprint, with specifics
Friday — offer or story: the sprint, the price, the outcome, the next step
Saturday — long guide: deeper into one of the 5 pillars
Sunday — metrics and lessons: what worked, what got cut, what moved That is 7 posts per week. Not all need to be long. Most should be short and sharp.

You prompt → post goes out → buyer reads → trust builds → conversation starts → sprint sells 🚀

4️⃣ The Comment That Builds More Credibility Than a Post

Here is the underrated move: a great comment on the right post outperforms a mediocre original post.

A good comment does one of 4 things:

Gives a sharper diagnosis than the original post offered
Adds a checklist the reader can use today
Names the hidden risk no one mentioned
Turns the idea into a concrete execution step Do not comment like a fan when you can comment like an operator.

"Great post!" — no pipeline.

"The real risk here is that broken auth after vibe-coding is invisible until a user gets locked out. Here's the 3-item check that catches it." — pipeline.

5️⃣ LinkedIn vs X — Different Games

LinkedIn is the boardroom. The buyer's professional identity lives there. Publish operator-level posts. Comment where founders and agency owners already gather. Use profile views as pipeline signals.

X is the radar. Early pain, builder language, launch signals, sharp opinions. Use it to spot emerging problems and test angles before committing to full posts.

Use both. Do not behave the same way on both.

The Real Bottom Line ⚡

The technical founder who posts useful, specific, buyer-facing content every single day will outcompete the quieter founder with a better product.

Every time.

Distribution is not optional support for a good product.

Distribution is how the market finds out a good product exists.

Your Turn 👇

Which of the 5 pillars is missing from your current content?

Or — what is stopping you from posting daily right now?

Drop it below 👇 — let's debug the system together 😄