DEV Community: Paul-S

What a customer support chatbot really costs per month (real API bill breakdown)

Paul-S — Thu, 22 Oct 2026 11:00:32 +0000

A chatbot can cost less than expected to generate answers and more than expected to retrieve the information behind them.

That sounds contradictory until you examine an actual workload.

This breakdown models a customer-support chatbot handling 50,000 conversations per month. It uses published API prices available in October 2026.

This is not presented as a private customer invoice. It is a reproducible estimate with every assumption exposed, so you can replace the numbers with your own.

Start With the Workload, Not the Model Price

Assume the chatbot handles:

50,000 conversations per month
Four generated replies per conversation
200,000 total model responses
1,800 input tokens and 250 output tokens for a normal reply
Five percent of requests routed to a stronger model
One knowledge-base search for each generated reply

The input includes system instructions, conversation history, customer information, and retrieved support content.

That last part matters. A chatbot with a growing conversation history can consume far more input tokens than a single-question demo.

The Model Bill: $147.95

For routine questions, the system uses GPT-6 Luna.

OpenAI’s published standard pricing lists GPT-6 Luna at $0.10 per million input tokens and $0.50 per million output tokens.

Ninety-five percent of 200,000 responses equals 190,000 Luna calls.

Those calls process:

342 million input tokens   = $34.20
47.5 million output tokens = $23.75

Luna subtotal              = $57.95

The remaining 10,000 requests are harder cases. They may involve conflicting policies, unusual account histories, or complicated troubleshooting.

Route those requests to GPT-6.1 Sol, currently priced at $2 per million input tokens and $10 per million output tokens.

Assuming 2,500 input tokens and 400 output tokens for each escalated request:

25 million input tokens = $50.00
4 million output tokens = $40.00

Sol subtotal            = $90.00

The combined generation bill is:

$57.95 + $90.00 = $147.95

Using the strongest model for every request would be simpler, but it would waste money on password resets, delivery questions, and other predictable tasks.

Model routing only works when evaluations prove that the cheaper model can handle those requests safely.

Retrieval Costs More Than Generation

A support chatbot normally searches product documentation, policies, previous tickets, or account information before answering.

OpenAI currently charges $2.50 per 1,000 file-search calls through the Responses API. Storage costs $0.10 per GB per day, with the first gigabyte free.

At 200,000 searches per month:

200,000 / 1,000 × $2.50 = $500

For a four-gigabyte knowledge base:

3 billable GB × $0.10 × 30 days = $9

Retrieval therefore adds $509, more than three times the model-generation bill.

This is the line item many estimates miss.

Not every message needs a new search. Greetings, confirmations, and follow-up questions can sometimes reuse existing context. Query classification and retrieval caching can reduce costs, but only if they do not introduce outdated answers.

The Infrastructure Still Has a Bill

The chatbot also needs an application around the model.

For this example, reserve:

Backend, database and queue: $140
Logs, traces and alerts:      $90

These are planning allowances rather than universal vendor prices.

Your number will depend on traffic patterns, retention periods, cloud provider, availability requirements, and how much conversation data you store.

Add a ten-percent allowance for failed requests, retries, traffic spikes, and evaluation runs:

Metered AI and retrieval: $656.95
Ten-percent headroom:      $65.70
Infrastructure:           $230.00

Estimated monthly total:  $952.65

That equals approximately $0.019 per conversation, or just under two cents.

Build Your Own Cost Calculator

Here is the complete Python calculation:

conversations = 50_000
replies = conversations * 4

luna_calls = int(replies * 0.95)
sol_calls = replies - luna_calls


def model_cost(
    calls,
    input_tokens,
    output_tokens,
    input_rate,
    output_rate,
):
    input_cost = (
        calls * input_tokens / 1_000_000 * input_rate
    )
    output_cost = (
        calls * output_tokens / 1_000_000 * output_rate
    )

    return input_cost + output_cost


luna = model_cost(
    luna_calls,
    input_tokens=1800,
    output_tokens=250,
    input_rate=0.10,
    output_rate=0.50,
)

sol = model_cost(
    sol_calls,
    input_tokens=2500,
    output_tokens=400,
    input_rate=2.00,
    output_rate=10.00,
)

file_search = replies / 1000 * 2.50
storage = (4 - 1) * 0.10 * 30

metered = luna + sol + file_search + storage
infrastructure = 140
observability = 90
headroom = metered * 0.10

monthly_total = (
    metered
    + infrastructure
    + observability
    + headroom
)

cost_per_conversation = monthly_total / conversations

print(f"Monthly cost: ${monthly_total:.2f}")
print(
    "Cost per conversation: "
    f"${cost_per_conversation:.4f}"
)

The output is:

Monthly cost: $952.65
Cost per conversation: $0.0191

Change the conversation volume, token counts, model rates, and retrieval frequency to estimate your own workload.

What This Estimate Does Not Include

The $952.65 total excludes:

Initial chatbot development
Customer-support platform fees
Human review
Security testing
Maintenance and evaluation
Engineering time spent investigating failures

Human escalation may eventually become the largest expense.

If five percent of 50,000 conversations require five minutes of agent review, that creates more than 208 hours of monthly support work.

A chatbot that routes uncertain cases correctly can still be valuable, but those hours belong in the business case.

Regional processing can also change the invoice. OpenAI currently applies a ten-percent uplift to eligible regional-processing and FedRAMP endpoints for newer models.

Ask for the Worksheet, Not One Monthly Number

When evaluating an AI chatbot development company, ask to see the assumptions behind its operating-cost estimate.

You should be able to identify:

Expected conversation volume
Model calls and token usage
Retrieval frequency
Escalation rate
Storage requirements
Failed calls and retries
Monitoring costs
Human-review workload

The model price is only one line on the invoice.

For this example, generating answers costs about $148. Finding the information behind those answers costs $509.

That is the cost relationship worth designing around.

The Chatbot Refunded Twice. The Retry Looked Successful

Paul-S — Tue, 20 Oct 2026 11:40:56 +0000

The production trace looked harmless:

14:03:11  Refund requested for order 4821
14:03:12  Payment provider called
14:03:22  Request timed out
14:03:24  Agent selected retry
14:03:26  Refund completed: $89

The chatbot reported success. The customer received confirmation. No exception remained open.

Two days later, finance found two $89 refunds.

The first payment request had not failed. Only the response had failed to reach the chatbot. When the agent retried, the payment provider processed the same business action again.

The retry looked successful because it was successful.

That was the bug.

This is a simplified composite example of a failure pattern that appears whenever an AI agent can perform consequential actions through APIs.

The HTTP Request Failed. The Refund Did Not.

Developers often treat a timeout as evidence that an operation failed.

It is only evidence that the client did not receive a response within the expected time.

Several things may have happened:

The provider rejected the request.
The provider received nothing.
The provider completed the refund but lost the response.
The response is still travelling through an overloaded network.
The provider is processing the request asynchronously.

The chatbot cannot distinguish these outcomes from a timeout alone.

A human support agent might pause, check the payment dashboard, and confirm the order status. An AI agent may interpret the timeout as permission to repeat the tool call.

This is why “retry on failure” is not a complete reliability policy.

The Model Should Not Create the Retry Identity

Suppose the agent calls this tool:

refundPayment({
  orderId: "4821",
  amount: 89
});

After a timeout, the model generates the same arguments and calls the tool again. The two calls look identical to us, but the payment system may see two independent refund requests.

The application should create a stable action identifier before execution:

type RefundCommand = {
  actionId: string;
  orderId: string;
  amount: number;
  requestedBy: string;
};

const command: RefundCommand = {
  actionId: "refund:4821:approved-request-7",
  orderId: "4821",
  amount: 89,
  requestedBy: "support-agent"
};

The important field is actionId.

It represents the business action, not one HTTP attempt. Every retry of that refund must reuse the same identifier.

Do not ask the language model to invent or remember this value. The surrounding application should generate it from trusted workflow state and persist it before calling the payment provider.

Idempotency Must Exist at More Than One Layer

If the provider supports idempotency keys, pass the stable identifier with every attempt:

async function issueRefund(command: RefundCommand) {
  return paymentProvider.refund(
    {
      orderId: command.orderId,
      amount: command.amount
    },
    {
      idempotencyKey: command.actionId
    }
  );
}

The first request may complete without returning a response. When the retry arrives with the same key, the provider should return the original result instead of creating another refund.

That protects the external transaction, but the application still needs its own record:

await db.refundCommands.create({
  data: {
    actionId: command.actionId,
    orderId: command.orderId,
    amount: command.amount,
    status: "PENDING"
  }
});

Place a unique constraint on actionId.

If two workers receive the same command simultaneously, only one should be allowed to create the execution record. The other should read the existing state instead of starting another action.

If an external API does not support idempotency, a database check alone may not be sufficient. Two processes can both check before either records completion. Use a lock, serialized queue, transactional outbox, or another mechanism that guarantees one executor for that business action.

“Completed” Should Not Be the Only Useful State

The chatbot’s original workflow understood three outcomes:

SUCCESS
FAILURE
RETRY

That was too simple.

The refund command needed states that described what the system actually knew:

PENDING
EXECUTING
CONFIRMATION_UNKNOWN
COMPLETED
REJECTED
REQUIRES_REVIEW

A timeout after sending the request should move the command to CONFIRMATION_UNKNOWN, not directly to FAILURE.

From there, the system can query the provider using the idempotency key, inspect a webhook, or send the case to a human. The chatbot should not independently decide that repeating a financial action is safe.

Test the Missing Response, Not Just the Failed Request

Most tests simulate an API rejecting a request before doing any work.

The more dangerous case is a successful operation followed by a lost response.

A useful integration test should reproduce this sequence:

The provider accepts the refund.
The provider records the transaction.
The client receives a timeout.
The workflow retries with the same action ID.
The provider returns the original result.
The ledger contains one refund.

Then run the test with two workers executing the same command concurrently.

The assertion that matters is not how many HTTP attempts occurred. It is how many financial transactions were created.

Retry Safety Is Part of Chatbot Quality

A 2026 Kore.ai survey of 408 enterprise IT and engineering leaders found that 79% of consequential autonomous actions required manual reversal. Among those reversals, 93% were described as costly or disruptive.

Better prompts cannot solve a missing transaction boundary.

If you are evaluating an AI chatbot development company, do not ask only whether its chatbot can process a refund. Ask what happens when the refund succeeds but its response disappears.

A reliable chatbot must know more than how to complete an action.

It must know when the action may already be complete.

Laravel AI SDK v1.0 Needs a Tenant Boundary

Paul-S — Thu, 15 Oct 2026 12:16:07 +0000

Authenticated tenant: 41
Authenticated user: 842
Agent tool: FindInvoices
Invoice returned: INV-1098
Invoice tenant: 17

The model found an invoice.

The tool executed successfully. No exception appeared in the logs. The response was factually correct.

It also exposed another customer’s data.

This is the dangerous kind of AI failure because every component appears to work. The problem is not hallucination. It is missing tenant isolation.

The SDK Did Exactly What We Asked

Laravel AI SDK v1.0 gives PHP developers a first-party system for agents, conversation storage, streaming, tool approvals and middleware.

It can decide when to call a tool and validate the arguments against a schema. Laravel also supports runtime provider configuration for tenant-specific API keys.

Those capabilities reduce integration work. They do not decide which records the current tenant is allowed to access.

Consider this tool call:

{
  "tool": "find_invoice",
  "arguments": {
    "customer_id": 842
  }
}

The arguments look valid. But customer_id is not an authorization boundary. IDs may collide across imported datasets, users may belong to multiple workspaces, and a malicious prompt may ask for a different account.

The tenant must come from authenticated application state, never from the prompt or model-generated arguments.

Make Tenant Context Impossible to Omit

Start by representing the current boundary explicitly:

final readonly class TenantContext
{
    public function __construct(
        public int $tenantId,
        public int $userId,
    ) {}
}

Resolve it after authentication and authorization:

$context = new TenantContext(
    tenantId: $request->user()->current_team_id,
    userId: $request->user()->id,
);

$response = (new SupportAgent($context))
    ->forUser($request->user())
    ->prompt($request->string('message'));

Calling forUser() is important because Laravel can associate stored conversations with a user. It is not the complete boundary when one user can access several tenants or workspaces.

The active tenant must still be authorized independently.

The Model Chooses the Tool, Not the Scope

Pass TenantContext into every data-access tool through its constructor:

use App\Models\Invoice;
use Illuminate\Contracts\JsonSchema\JsonSchema;
use Laravel\Ai\Contracts\Tool;
use Laravel\Ai\Tools\Request;
use Stringable;

final class FindInvoices implements Tool
{
    public function __construct(
        private TenantContext $context,
    ) {}

    public function description(): Stringable|string
    {
        return 'Find recent invoices for the authenticated account.';
    }

    public function schema(JsonSchema $schema): array
    {
        return [
            'status' => $schema->string(),
        ];
    }

    public function handle(Request $request): Stringable|string
    {
        $invoices = Invoice::query()
            ->where('tenant_id', $this->context->tenantId)
            ->where('user_id', $this->context->userId)
            ->when(
                $request['status'] ?? null,
                fn ($query, $status) => $query->where('status', $status)
            )
            ->select(['id', 'status', 'total', 'issued_at'])
            ->latest()
            ->limit(10)
            ->get();

        return $invoices->toJson();
    }
}

Notice what is missing from the schema: tenant_id.

The model cannot supply it, replace it or persuade the tool to ignore it. Even a successful prompt injection remains trapped inside the authenticated tenant boundary.

Laravel’s own guidance for production-safe database tools recommends the same underlying pattern: pass identity through application code, restrict visible columns, cap results and use a read-only database connection where possible.

Five Places Tenant Data Can Still Escape

Fixing the Eloquent query is necessary, but AI applications move data through more than one layer.

Check these boundaries:

Conversation history: Authorize both the user and active tenant before loading an existing conversation ID.
Vector retrieval: Store tenant_id with every chunk and require it as a server-side search filter.
Cache entries: Include the tenant, user, model and permission version in the cache key.
Queued agents: Serialize immutable IDs, then reload and reauthorize them when the worker executes.
Tool approvals: Bind an approval to the tenant, user, tool and original arguments. Reject it if any context changed while the request was waiting.

A cached answer generated for Tenant 41 must never be reusable by Tenant 17 simply because both asked the same question.

Write the Test Before the Prompt

The most useful test is not “Does the agent find an invoice?”

It is “Can the agent retrieve an invoice belonging to someone else?”

it('cannot return another tenant invoice', function () {
    $context = new TenantContext(
        tenantId: $this->tenantA->id,
        userId: $this->userA->id,
    );

    $foreignInvoice = Invoice::factory()->create([
        'tenant_id' => $this->tenantB->id,
    ]);

    $result = (new FindInvoices($context))
        ->handle(new Request(['status' => $foreignInvoice->status]));

    expect($result)->not->toContain($foreignInvoice->id);
});

Add adversarial cases for guessed IDs, reused conversation IDs, poisoned retrieval metadata, delayed approvals and queue jobs executed after membership is revoked.

A Tenant Boundary Is Application Code

Laravel AI SDK v1.0 makes agents feel native to Laravel. That is valuable because tools, queues, models and tests can remain inside the same application.

The abstraction should not hide the security model.

The best Laravel development company for a multi-tenant AI product should be able to identify where tenant identity originates, how it reaches every tool and what prevents the model from changing it.

A prompt can tell an agent to respect tenant boundaries.

Only application code can enforce them.

What tenant-isolation test would you run before connecting a Laravel agent to production data?

My MEAN Agent Remembered Everything. That Was the Bug

Paul-S — Tue, 13 Oct 2026 11:48:23 +0000

My MEAN Agent Remembered Everything. That Was the Bug

I reproduced the failure during a red-team test.

10:14  Tenant: northwind
User: Where should we send the replacement?

10:14  Agent memory retrieved:
“Deliver replacement orders to the Chicago warehouse.”

10:14  Source tenant: copper

The agent remembered a real instruction. It was relevant to the question and ranked highly in semantic search.

It also belonged to another customer.

This was not a hallucination. The model did not invent the warehouse. It accurately used information our application should never have placed in its context.

Nothing in the MEAN stack had crashed. Angular rendered the conversation, Express accepted the request, Node completed the workflow, and MongoDB returned a matching document.

The system was working exactly as we had designed it.

That was the bug.

We Had Built a Memory Bucket

The first version of the agent stored almost everything:

User messages
Conversation summaries
Tool responses
Customer preferences
Previous workflow results

Each record received an embedding. Before generating an answer, the Node service searched for semantically similar memories and added the strongest matches to the prompt.

The retrieval query considered relevance, but it did not enforce ownership strongly enough. We had treated memory as a smarter version of chat history.

It is not.

An audit record, a temporary workflow value, and a durable customer preference may all contain text, but they should not have the same retention period or retrieval rules.

Every Memory Needed a Receipt

We replaced the generic memory document with an explicit contract:

type AgentMemory = {
  tenantId: string;
  principalId: string;
  purpose: "current_task" | "approved_preference";
  source: "user" | "verified_tool";
  sensitivity: "standard" | "restricted";
  content: string;
  embedding: number[];
  createdAt: Date;
  expiresAt: Date;
  status: "active" | "revoked";
};

A memory could no longer exist simply because the model considered it useful.

The application had to know:

Who owned it?
Why was it stored?
Which verified source produced it?
How long could it influence the agent?
Could the user inspect and revoke it?

Anything that could not answer those questions stayed outside the agent’s retrievable memory.

We also separated four concepts that had previously shared one collection: short-lived conversation context, workflow state, approved preferences, and audit evidence.

Audit records remained available to authorized services, but they were no longer automatically fed back into the model.

We Added Three Gates

The repair was not another system prompt. It was three application-level gates.

1. The write gate

Only approved memory types could be persisted.

Raw tool output was not automatically trusted. A webpage, uploaded document, or CRM note could contain incorrect information or malicious instructions.

OWASP’s AI Agent Security Cheat Sheet identifies memory poisoning as a distinct risk and recommends validating content, isolating memory between users and sessions, and enforcing expiration and size limits.

2. The read gate

Every retrieval included identity and lifecycle filters before semantic ranking:

const memories = await Memory.aggregate([
  {
    $vectorSearch: {
      index: "agent_memory",
      path: "embedding",
      queryVector,
      numCandidates: 100,
      limit: 8,
      filter: {
        tenantId: { $eq: tenantId },
        principalId: { $eq: principalId },
        status: { $eq: "active" },
        expiresAt: { $gt: new Date() }
      }
    }
  }
]);

The corresponding vector index must define those fields as filters.

MongoDB’s multi-tenant Vector Search guidance recommends storing a tenant_id and applying it as a pre-filter so queries only return documents belonging to that tenant.

The important rule is simple: semantic relevance must never override authorization.

3. The forget gate

Temporary memories received an expiry date, backed by a MongoDB TTL index:

MemorySchema.index(
  { expiresAt: 1 },
  { expireAfterSeconds: 0 }
);

However, TTL deletion is asynchronous. MongoDB explains that expired documents can remain until its background process removes them.

That means TTL is useful for cleanup, but it is not an authorization boundary. The application must still exclude expired or revoked memory during every read.

A user pressing “Forget this” should affect retrieval immediately.

Angular Needed a Memory Interface

The backend changes exposed a product problem: users had no way to see what the agent remembered.

We added a small Angular memory panel showing:

The saved fact
Why it was remembered
Its source
Its expiry
A revoke action

This made incorrect memories observable before they silently influenced several future conversations.

It also changed our testing. We stopped asking only, “Did the agent remember the preference?” We added a second question:

Can it still retrieve that preference after the user changes tenants, revokes it, or reaches its expiry date?

That second question caught more important defects than the first.

Better Recall Is Not Always Better Memory

Agent benchmarks often reward successful recall. Production systems must also measure forbidden recall.

Can the agent avoid retrieving another tenant’s data? Can it forget expired workflow state? Can it distinguish verified preferences from untrusted text? Can it explain why a memory entered the prompt?

These boundaries deserve attention when teams hire MEAN stack developers for agentic applications. At Spaculus Software, we treat memory as governed application data, not an unlimited extension of the model’s context window.

The safest agent is not the one that remembers everything.

It is the one that can prove why each memory exists, who owns it, and when it must stop influencing decisions.

What would your agent need to forget before you trusted it with customer data?

My RAG Eval Passed. The Citations Were Still Wrong

Paul-S — Thu, 08 Oct 2026 11:52:23 +0000

My RAG Eval Passed. The Citations Were Still Wrong

Consider this small regression test:

Question:
What is the current refund period?

Retrieved:
POLICY-2025: Customers may request refunds within 30 days.
POLICY-2026: Customers may request refunds within 14 days.

Generated answer:
Customers may request refunds within 14 days. [POLICY-2025]

The answer contains the current policy.

The expected document was retrieved.

A citation was present.

Several common RAG checks could mark this response as successful. Yet the citation points to evidence that contradicts the answer.

That is a dangerous kind of failure because the response looks verifiable.

The Green Dashboard Missed a Separate Contract

RAG evaluation often combines retrieval and generation into a few broad scores:

Did search find a relevant document?
Was the response related to the question?
Was the answer grounded in the supplied context?
Did the model include a citation?

Those questions matter, but none proves that a specific citation supports the exact claim beside it.

The TREC 2025 RAG Track treats relevance, response completeness, attribution verification and agreement as separate evaluation layers. That separation is useful because citation quality can fail even when retrieval and generation appear healthy.

A citation is more than formatting. It is a contract between one claim and one piece of evidence.

Break the Answer Into Atomic Claims

Suppose a chatbot produces this response:

The Pro plan allows 20 users and includes priority support. [DOC-7]

That sentence contains at least two claims:

Claim 1: The Pro plan allows 20 users.
Claim 2: The Pro plan includes priority support.

DOC-7 might support the user limit while saying nothing about priority support. A sentence-level evaluator may give the entire sentence a passing score because part of it is supported.

A claim-level evaluator should test each statement independently.

Recent Q-CARE research follows a similar direction by decomposing answers into atomic claims and measuring whether each claim can be verified. Its evaluation correlated more closely with human judgments than several existing RAG metrics across the tested datasets.

Add a Citation Gate After Generation

The generation step should return structured claims instead of only formatted prose.

claims = [
    {
        "text": "The refund period is 14 days.",
        "source_id": "POLICY-2026",
        "source_version": "2026-09",
    }
]

A separate citation gate can then inspect each claim:

def evaluate_citations(claims, sources, supports):
    failures = []

    for claim in claims:
        source = sources.get(claim["source_id"])

        if source is None:
            failures.append({
                "claim": claim["text"],
                "reason": "source_not_found",
            })
            continue

        if source["version"] != claim["source_version"]:
            failures.append({
                "claim": claim["text"],
                "reason": "version_mismatch",
            })
            continue

        if not supports(source["content"], claim["text"]):
            failures.append({
                "claim": claim["text"],
                "reason": "claim_not_supported",
            })

    return failures

The supports function could use deterministic rules for exact values, a natural-language-inference model for descriptive claims or a calibrated LLM judge.

I would avoid allowing the same model to write the answer and approve its own citations without another check. Research on citation attribution has found that LLM judges can also reject valid evidence, especially when terminology varies.

High-risk claims may still need sampled human review.

Test Evidence Strength, Not Keyword Overlap

A source can be relevant while remaining too weak for the wording used.

Consider these two claims:

Evidence: The update may reduce processing time.

Claim A: The update may reduce processing time.
Claim B: The update will reduce processing time by 40%.

Both discuss processing time. Only Claim A is supported.

This failure is sometimes described as citation laundering: a related source makes an overstated claim appear trustworthy. FORCEBENCH research found that generic support prompts struggled to distinguish appropriately qualified claims from stronger unsupported versions.

Your evaluator should check numbers, dates, conditions, scope and certainty. Text similarity alone cannot do that reliably.

Put the Failure Into CI

Every citation bug found in production should become a regression case.

Useful tests include:

Two versions of the same policy
One source supporting only half a sentence
A correct fact attached to the wrong document
A citation pointing to a deleted or inaccessible source
Evidence using “may” while the answer says “will”
Two documents that disagree about the same value

For teams evaluating an AI chatbot development company, asking how citations are tested is more revealing than asking whether the chatbot “uses RAG.”

A healthy RAG system should prove four things independently: it retrieved useful evidence, produced a correct answer, covered the important claims and attached each claim to evidence that actually supports it.

My evaluation passed because it tested whether citations existed.

The better test asks whether every citation deserves to be there.

What citation failure would you add to this regression suite?

AI SaaS Free Trials Are Becoming an Abuse Surface

Paul-S — Tue, 06 Oct 2026 10:50:03 +0000

Consider this composite incident trace:

TRIAL INCIDENT 042

New accounts:          186
Unique email addresses: 186
Recognized devices:      4
High-cost generations: 912
Paid conversions:        0

Nothing was technically compromised.

No password was stolen. No database was breached. Every account used the signup form correctly and stayed within its advertised trial allowance.

The product still lost money.

Four devices had created 186 identities, consumed hundreds of expensive AI operations, exported the results, and disappeared. The free trial had become an unprotected compute endpoint.

The Trial Is Now Part of Your Attack Surface

A traditional SaaS trial might expose dashboards, sample data, and limited storage. Supporting another trial user may add almost no immediate infrastructure cost.

An AI product behaves differently.

Every prompt can trigger model inference, retrieval, embeddings, image generation, document parsing, external API calls, or agent tools. A user does not need to breach the system to create damage. They only need to repeatedly consume something expensive.

This problem is accelerating. Stripe reported that its models detected 6.2 times more abusive free trials between November 2025 and February 2026. The company also noted that AI businesses are particularly exposed because each generated output creates a direct compute cost.

That changes the engineering question.

It is no longer only:

How many free requests should a trial include?

It becomes:

How much unverified economic risk should one identity control?

Signup Validation Is Necessary but Insufficient

Email verification stops basic scripts. It does not establish that 20 verified email addresses represent 20 different people.

Attackers can rotate disposable inboxes, virtual phone numbers, payment cards, residential proxies, and browser profiles. Blocking one signal usually pushes the activity toward another.

Cloudflare’s account-abuse guidance recommends examining activity across users, IP addresses, devices, locations, browsers, logins, and registrations. The important word is across.

The application needs to connect events that appear legitimate when viewed separately.

For example, one account generating 30 reports may be an engaged prospect. Twenty accounts generating the same reports from related devices are a different pattern.

This is why a CAPTCHA at registration cannot be the entire defense. The system must continue evaluating behavior after access has been granted.

Give Every Trial a Risk Budget

A useful design is to separate the advertised trial allowance from an internal risk budget.

The allowance describes what a normal customer can use. The risk budget decides how much costly activity the system permits before requesting stronger verification.

A simplified policy could look like this:

def evaluate_trial(session):
    if session.linked_accounts >= 5:
        return "DENY_EXPENSIVE_ACTIONS"

    if session.estimated_cost_hour > 8:
        return "REQUIRE_PAYMENT_METHOD"

    if session.concurrent_jobs > 3:
        return "QUEUE_AND_REVIEW"

    if session.identity_risk == "medium":
        return "REDUCE_RATE_LIMIT"

    return "ALLOW"

These numbers are illustrative, not universal defaults. A coding assistant, video generator, and document-analysis platform have completely different cost profiles.

The important design choice is that the decision uses more than request count. It considers linked identities, estimated cost, concurrency, verification, and behavior.

Rate limiting one IP address is not enough when accounts can move between networks. Limiting one account is not enough when identities are inexpensive to create.

Add Friction to the Expensive Moment

Many teams put their strongest verification at signup. That can hurt conversion before the user has experienced any value.

A better approach is progressive friction.

Let a new user explore the interface or complete a small generation. Request additional verification when the user tries to run a GPU-heavy workflow, create an API credential, launch concurrent jobs, export large outputs, or consume more than a defined cost threshold.

The response does not always need to be “block the account.” The system can:

Reduce generation speed
Queue costly operations
Limit concurrency
Request a payment method
Require business-email verification
Route unusual activity for review

This preserves a low-friction path for genuine prospects while increasing the cost of repeated abuse.

Stop Measuring Trial Success by Signups

An abuse campaign can make a growth dashboard look healthy.

Registrations increase. Activation rises. Feature usage reaches a record level. Infrastructure spending follows it upward.

If the team measures only signups and activation, abusive activity can look like product-market traction.

AI SaaS teams should also monitor cost per verified trial, model spend per conversion, linked-account recurrence, challenge completion, failed payment patterns, and the percentage of usage created by newly registered accounts.

Stripe’s guidance warns that trial abuse can distort product analytics in addition to increasing infrastructure and API costs. Polluted data can lead the product team to prioritize workflows that genuine customers barely use.

Design Revocation Before Issuing Access

Blocking new requests is only part of containment.

A trial may already have API keys, active agent runs, scheduled jobs, generated download links, or connected third-party tools. When an account is restricted, those capabilities must be revoked together.

That means trial status should be checked when an expensive action executes, not only when the user logs in.

For teams evaluating a SaaS application development company, this is an important architecture question to raise. At Spaculus Software, we treat trial controls as part of entitlement, cost, and security design rather than a billing-screen setting.

A free trial should help a genuine user reach value quickly.

It should not provide anonymous, renewable access to your most expensive infrastructure.

If someone created 100 trial accounts tonight, which resource in your SaaS product would cost you the most?

The Flutter Upgrade Passed CI. Plugins Failed on Device

Paul-S — Wed, 30 Sep 2026 11:25:51 +0000

The upgrade pipeline looked perfect.

flutter analyze                         PASS
flutter test                            PASS
flutter build apk --release             PASS
flutter build ios --release --no-codesign PASS

The pull request was approved. The new build reached the testing team.

Then the device reports arrived.

Android: Push notifications never register
iOS: Sign-in returns to a blank screen
Both: Unit tests remain green

CI had not lied. It had successfully tested what we asked it to test.

The problem was what we never asked it to test.

A Green Build Does Not Prove Plugin Behaviour

Most Flutter applications are not entirely Flutter.

A plugin usually contains a Dart-facing API and a native implementation written in Kotlin, Java, Swift, or Objective-C. The two sides communicate through platform channels or native bindings.

That creates a longer execution path:

Dart code
   ↓
Platform channel
   ↓
Plugin registration
   ↓
Android or iOS lifecycle
   ↓
Native SDK

A Dart unit test may validate the first layer while never loading the remaining four.

Flutter’s documentation explains that plugin unit tests often replace the platform implementation with mocks. That is useful for testing application logic, but a mocked method channel cannot reveal that a native class failed to register or an operating-system callback stopped firing.

The test passes because the native failure never enters the test process.

Framework Upgrades Move Native Boundaries Too

A Flutter upgrade is rarely just a Dart SDK change.

On Android, it can affect the Android Gradle Plugin, Gradle itself, Java compatibility, Kotlin configuration, manifest handling, and plugin build scripts.

Recent Flutter migration guidance illustrates this clearly. Built-in Kotlin support changes how Android projects and plugins are configured, while add-to-app projects may require manual migration because Flutter cannot rewrite the native host application automatically.

On iOS, the move toward the UIScene lifecycle changes where application events are delivered. Plugins relying on older AppDelegate callbacks may compile correctly but fail when the application receives a URL, notification, or lifecycle event on a real device.

Compilation answers one question:

Can these files become an application?

It does not answer another:

Will every native capability behave correctly when the operating system invokes it?

Debug the Boundary, Not Just the Dart Exception

When a plugin fails after an upgrade, the Dart error is often the end of the trail rather than the beginning.

A MissingPluginException, frozen method call, or empty callback could originate from several places:

The plugin was not registered with the current engine.
The Android activity or iOS scene changed before a callback returned.
A required manifest entry, entitlement, or permission is missing.
The plugin supports debug builds but fails after release optimization.
Its native dependency is incompatible with the new build toolchain.
The plugin’s Dart package was updated without a matching native migration.

Start by reproducing the failure on one known device. Then inspect native logs through Android Studio, adb logcat, Xcode, or the macOS Console.

Follow one call across the boundary:

Did Dart send the method call?
Did the native handler receive it?
Did the operating system start the requested action?
Did the callback return?
Did Flutter receive the result?

This is usually faster than repeatedly changing Dart code around a native failure.

Put Plugin Workflows Into the Upgrade Gate

A better upgrade pipeline includes tests that launch the application on the platforms it supports.

Flutter’s integration_test package can run tests on emulators, simulators, physical devices, and device farms. Use it for critical workflows that depend on plugins, such as:

Authentication redirects
Camera and file access
Push-notification registration
Payments and in-app purchases
Deep links and app links
Location and background services

Do not test only the happy path.

Deny a permission and request it again. Put the app in the background during a callback. Launch it from a notification. Open a deep link after the process has been terminated. Run the signed release build, not only the debug build.

The minimum upgrade evidence should look closer to this:

Static analysis                 PASS
Dart unit tests                 PASS
Android release build           PASS
iOS release build               PASS
Android plugin workflows        PASS
iOS plugin workflows            PASS
Background and cold-start tests PASS

CI is still important. It simply needs coverage beyond compilation.

Change the Definition of “Upgrade Complete”

An upgrade is not complete when flutter build exits with code zero.

It is complete when the application’s important native paths have been exercised on Android and iOS, using the build modes and operating-system behaviours customers will actually encounter.

That may reveal that a plugin must be upgraded, patched, replaced, or temporarily pinned. It may also expose a missing native migration in the application itself.

This is the kind of evidence teams should expect from a hybrid app development company: not only a green pipeline, but a clear device matrix showing which plugin-backed workflows were tested.

The Flutter upgrade passed CI because the pipeline proved that the project could build.

The plugins failed because the application had not yet proved that it could run.

Which Flutter plugin has caused your most difficult device-only failure?

I Built a Laravel Agent That Must Ask Permission

Paul-S — Mon, 28 Sep 2026 11:59:36 +0000

The agent had already done the difficult work.

It found the customer’s order, checked the payment status, read the refund policy, and calculated the amount.

Then it proposed this:

Action: Issue refund
Order: ORD-1842
Amount: $129.00
Reason: Customer received a damaged product

The calculation looked reasonable. The customer qualified for a refund.

I still did not want the model pressing the final button.

A refund changes real business data and moves money. The agent could recommend the action, but a person needed to approve it.

The Dangerous Line Was One Method Call

My first version exposed an IssueRefund tool directly to the agent.

The instructions included a sentence like:

Always ask a manager before issuing refunds above $25.

That sounded safe until I considered what enforced it.

The same model deciding to issue the refund was also responsible for remembering when it needed permission. A changed prompt, incomplete conversation, or different model response could bypass that instruction.

The permission boundary belonged in PHP.

Laravel released AI SDK v1.0 on September 23, 2026. Among its production features is human approval for agent tools. An approvable tool pauses before its handle() method executes.

That was the behavior I needed.

Making the Refund Tool Approvable

Here is a simplified version of the tool:

<?php

namespace App\Ai\Tools;

use Illuminate\Contracts\JsonSchema\JsonSchema;
use Laravel\Ai\Approvals\Approval;
use Laravel\Ai\Concerns\InteractsWithApprovals;
use Laravel\Ai\Contracts\Approvable;
use Laravel\Ai\Contracts\Tool;
use Laravel\Ai\Tools\Request;
use Stringable;

class IssueRefund implements Approvable, Tool
{
    use InteractsWithApprovals;

    public function description(): Stringable|string
    {
        return 'Issue an approved refund for an existing order.';
    }

    protected function needsApproval(Request $request): Approval|bool
    {
        $amount = (int) $request['amount_cents'];

        return $amount <= 2500
            ? false
            : Approval::required(
                'Refunds above $25 require manager approval.'
            );
    }

    public function handle(Request $request): Stringable|string
    {
        return app(RefundService::class)->issueOnce(
            orderId: $request['order_id'],
            amountInCents: $request['amount_cents'],
            reason: $request['reason'],
        );
    }

    public function schema(JsonSchema $schema): array
    {
        return [
            'order_id' => $schema->string()->required(),
            'amount_cents' => $schema->integer()->min(1)->required(),
            'reason' => $schema->string()->required(),
        ];
    }
}

Refunds of $25 or less can continue automatically. Anything above that amount produces an approval request.

The threshold is ordinary application logic. The model cannot negotiate with it or reinterpret it.

A Pause Is a Real Workflow State

When the agent selects IssueRefund, Laravel returns the pending tool call instead of executing it.

$response = (new RefundAgent)
    ->forUser($user)
    ->prompt('Refund the damaged order ORD-1842.');

if ($response->hasPendingApprovals()) {
    foreach ($response->pendingApprovals as $approval) {
        // Display the tool name, arguments and reason.
    }
}

The approval screen should show the exact order, amount, reason, and requested action. A vague “Allow this agent to continue?” message does not give the reviewer enough information.

After the manager approves or rejects the request, the application resumes the conversation:

use Laravel\Ai\Approvals\Decision;
use Laravel\Ai\Approvals\Decisions;

$response = (new RefundAgent)
    ->continue($conversationId, as: $manager)
    ->prompt(Decisions::from([
        $toolCallId => Decision::approve(),
    ]));

Laravel also supports rejecting the request or editing its arguments before execution. The human tool approval documentation covers the full flow.

Approval Is Not Authorization

This was the most important lesson from the build.

An approval button does not prove that the current user is allowed to approve the refund.

Before resuming the agent, the application must verify that:

The conversation belongs to the correct account
The signed-in user has refund approval permission
The order still qualifies for a refund
The approved amount remains valid
The action has not already been completed

The refund service also needs an idempotency rule. If a queue retries or a user submits the approval twice, the customer should still receive only one refund.

The official documentation warns that paused turns are matched through the conversation and pending tool calls. The application is responsible for authorizing access before resuming them.

The Tests That Matter

A happy-path test proves that an approved refund can succeed. That is only the beginning.

I would also test these cases:

An unauthorized employee attempts to approve the refund
The order status changes while approval is pending
A manager edits the amount before approval
The same approval is submitted twice
The approval references an unknown tool-call ID
A rejected action returns a useful explanation
An old approval expires before execution

These tests check the business boundary around the model rather than the quality of its wording.

If you are comparing teams under a search such as best Laravel development company, ask how they would authorize, resume, audit, and retry this workflow. That answer reveals more than another framework quiz.

The agent can collect evidence and recommend a refund.

PHP decides whether the tool must pause. Laravel records the pending action. An authorized person makes the final decision. The service executes it once.

That separation is what made the agent safe enough to use.

Which action in your application should an AI agent never execute without approval?

Banks Know Their Customers. Now They Need to Know Their AI Agents.

Paul-S — Thu, 17 Sep 2026 09:58:56 +0000

“Who authorized this transaction?”

“The customer.”

“Which agent submitted it?”

“We do not record that separately.”

“Which tools was the agent allowed to use?”

“It used the customer’s session.”

“Can we revoke the agent without blocking the customer?”

Silence.

This is the architecture review many banks will eventually face.

Banks already know how to identify customers, employees, applications, and service accounts. AI agents introduce a less familiar actor: software that can interpret a goal, select tools, collect data, and initiate actions on someone else’s behalf.

Confirming the customer’s identity is no longer enough. The bank must also know which agent is acting, who owns it, what authority it received, and where that authority ends.

Know Your Agent Is a Chain of Trust

At the Global Fintech Fest 2026, State Bank of India Chairman CS Setty proposed a “Know Your Agent” framework covering agent identity, authentication, customer consent, transaction limits, audit trails, and revocation.

His warning was important. An autonomous error can travel through connected systems at machine speed. The same automation that reduces waiting time can also multiply the effect of a bad decision.

A banking agent should therefore never appear in a transaction as an invisible extension of the customer.

The trust chain should remain visible:

Customer → Consent → Agent → Policy Decision → Banking API

If one link is missing, the bank cannot confidently explain why an action happened.

Authentication proves which agent made the request. Authorization determines whether that particular agent may perform that particular action for that customer at that moment.

Banks need both.

Give the Agent a Passport, Not a Master Key

An agent identity should carry a small, verifiable description of its authority. Think of it as a temporary passport rather than a reusable API key.

Here is an illustrative manifest:

agent:
  id: card-dispute-prod
  owner: disputes-team
  acting_for: customer-8421

  permitted:
    - transactions.read
    - fee_reversal.draft

  prohibited:
    - money.transfer
    - beneficiary.create

  approval_required:
    - fee_reversal.execute

  expires_in: 5m

This information should not exist only inside the model’s prompt. Prompts can be altered, misunderstood, or exposed to injection attacks.

The passport must be issued, signed, and checked by infrastructure outside the model. Every banking API should evaluate it before accepting an action.

This also separates three identities that teams often combine:

The customer requesting help
The agent interpreting the request
The service executing the action

That separation makes investigation and revocation possible.

The NIST National Cybersecurity Center of Excellence is exploring similar questions around agent identification, authentication, least privilege, delegated access, human approval, and verifiable audit records. Its 2026 concept paper also considers established technologies such as OAuth, OpenID Connect, SPIFFE, and SCIM.

Banks may not need a completely new identity system. They may need to adapt proven identity controls to a new class of non-human user.

Authority Should Shrink as Consequence Grows

Not every agent action carries equal risk.

A banking assistant might operate in four lanes:

Explain: Describe a fee or summarize account activity.
Prepare: Collect details and draft a dispute.
Recommend: Suggest a decision to an employee.
Execute: Change data, move money, or approve an outcome.

Moving from explanation to execution should not simply unlock more tools. It should narrow the acceptable conditions.

A balance explanation may require authenticated account access. Adding a beneficiary should require fresh customer confirmation. Reversing a fee may need an employee’s approval. Moving money should require stricter limits, stronger authentication, and an independent fraud check.

My view is that banks should automate evidence gathering before automating irreversible decisions. It produces meaningful efficiency while keeping responsibility with a person who is accountable for the outcome.

Financial-Crime Work Shows a Safer Starting Point

FIS offers a useful example. Its Financial Crimes AI Agent is designed to assemble evidence from banking systems, evaluate activity, and surface higher-risk cases.

The investigator still controls the decision.

That division of work matters. The agent handles the repetitive search across disconnected systems. A qualified human evaluates the evidence and accepts responsibility for the conclusion.

This is a more defensible starting point than allowing a general-purpose agent to perform every available banking action simply because the customer is authenticated.

Revocation Should Be Boring

If an agent behaves unexpectedly, stopping it should not require disabling the customer’s account or rotating a shared production credential.

Agent authority should be short-lived and revocable independently.

A bank should be able to answer:

Which agent is active?
Who or what delegated its authority?
Which resources can it access?
What actions has it attempted?
When does its permission expire?
Can the bank terminate it immediately?

These are ordinary identity-management questions. What changes is the speed and flexibility of the software being controlled.

The Next Banking User May Not Be Human

This shift also changes how conversational banking products should be built.

For an AI chatbot development company, improving response quality is no longer the complete assignment. Once a chatbot can call tools and modify business systems, its identity, permissions, approval boundaries, and audit trail become part of the product.

At Spaculus Software, that means treating an AI agent as a governed system actor, not merely a conversational interface.

Banks have spent decades learning who their customers are. The next challenge is knowing which software is standing beside those customers, what it has been asked to do, and whether it should be allowed to continue.

If an AI agent contacted your banking API today, could your system identify the agent separately from the customer?

An AI Engineer’s Hardest Problem Is Knowing When the Model Should Stop

Paul-S — Tue, 15 Sep 2026 12:05:21 +0000

RUN 4821
Goal: Process customer refund
Order found: Yes
Refund policy found: Yes
Policy versions found: 2
Amounts calculated: $89 and $129
Tool retries: 4
Next action selected: Issue $129 refund

The agent found conflicting policies, calculated two different amounts, and still selected an action.

That last line might look like successful task completion.

It is actually the failure.

The agent should have stopped when it discovered that the evidence could not support one safe decision.

The Missing State in Most Agent Workflows

Many agent workflows define only two outcomes:

SUCCESS
FAILURE

But an agent can execute every tool successfully and still make the wrong decision. The API responded. The database was available. No exception occurred.

The uncertainty existed in the business context, not the infrastructure.

Production agents need more meaningful terminal states:

ASK_FOR_INPUT
WAIT_FOR_APPROVAL
ESCALATE_TO_HUMAN
DENY_ACTION
STOP_LIMIT_REACHED
COMPLETE

A stop is not necessarily a failure. Sometimes it is the most correct output the system can produce.

Why Another Prompt Will Not Fix This

A common response is to add instructions such as:

Only proceed when you are confident.

That delegates the boundary decision to the same probabilistic model creating the uncertainty.

The model may report high confidence even when its evidence is incomplete. It may interpret repeated tool calls as a reason to keep investigating. A different model version could interpret the instruction differently.

Prompts can guide reasoning. They should not be the only mechanism protecting consequential actions.

The application needs a deterministic gate between the model’s recommendation and the real operation.

Turn Stop Conditions Into Code

Here is a framework-independent example:

def next_state(run):
    if not run["authorized"]:
        return "DENY_ACTION"

    if run["conflicting_evidence"]:
        return "ESCALATE_TO_HUMAN"

    if run["missing_required_data"]:
        return "ASK_FOR_INPUT"

    if run["irreversible"] and not run["approved"]:
        return "WAIT_FOR_APPROVAL"

    if run["tool_attempts"] >= 6:
        return "STOP_LIMIT_REACHED"

    return "COMPLETE"

Notice what this function does not check: the model’s self-reported confidence.

It checks observable conditions such as authorization, missing data, conflicting evidence, reversibility, approval, and execution limits.

The model can propose the next action. The surrounding software decides whether that action is allowed.

OWASP calls the combination of excessive functionality, permissions, and autonomy Excessive Agency. Its recommended controls include least-privilege access, limited tools, independent authorization checks, and human approval for high-impact actions.

Break the Exit Paths Before Users Do

Happy-path tests ask whether an agent can complete a task.

Boundary tests ask whether it refuses to complete the wrong task.

A small pytest suite can make those decisions visible:

import pytest


@pytest.mark.parametrize(
    "change, expected",
    [
        ({"authorized": False}, "DENY_ACTION"),
        ({"conflicting_evidence": True}, "ESCALATE_TO_HUMAN"),
        ({"missing_required_data": True}, "ASK_FOR_INPUT"),
        (
            {"irreversible": True, "approved": False},
            "WAIT_FOR_APPROVAL",
        ),
        ({"tool_attempts": 6}, "STOP_LIMIT_REACHED"),
    ],
)
def test_agent_exit_paths(base_run, change, expected):
    run = {**base_run, **change}
    assert next_state(run) == expected

These tests are simple, but they force the team to define expected behavior before an unusual request reaches production.

The next layer is adversarial evaluation. Give the agent contradictory documents, expired authorization, unavailable tools, repeated timeouts, ambiguous user instructions, and requests that cross account boundaries.

Do not score only the final answer. Record whether the agent chose to continue, ask, pause, deny, or escalate.

The NIST AI Risk Management Framework recommends documenting knowledge limits, defining human oversight, testing under deployment-like conditions, and ensuring AI systems can fail safely when operating beyond their limits.

The Hiring Question I Would Ask

I would not evaluate an AI engineer only by asking which models or agent frameworks they have used.

I would give them the refund trace from the beginning of this article and ask:

Where should this workflow stop, and which component should enforce that decision?

A strong answer should discuss permissions, evidence quality, action reversibility, approval requirements, retry budgets, logging, and human handoff.

For teams planning to hire AI engineers, this boundary-first evaluation is more revealing than another prompt-engineering exercise. It is also how Spaculus Software approaches production AI workflows: the model reasons, but the system retains control.

The hardest part of agent engineering is not helping a model finish more tasks.

It is making sure the system recognizes the tasks it must not finish alone.

What stop condition would you add before allowing an AI agent to change real business data?

Your AI Engineer Is Debugging Workflows, Not Prompts

Paul-S — Fri, 11 Sep 2026 11:10:44 +0000

The support agent looked fine during testing.

It understood refund requests, found the correct order, explained the policy, and asked for confirmation before taking action.

Then it reached production and refunded the same order twice.

The first reaction was predictable: rewrite the prompt.

Add “never issue the same refund twice.” Put it in capital letters. Repeat it near the tool instructions. Lower the temperature.

None of that fixed the real problem.

The payment API had completed the first refund, but its response timed out. The workflow interpreted the missing response as failure and retried the tool call. The model followed its instructions both times.

This was not a prompt bug. It was a distributed-systems bug wearing an AI costume.

The Prompt Is Only One Part of the System

A demo often looks like this:

user request -> model -> answer

A production agent looks more like this:

request -> context -> model -> tool -> external API
-> state update -> model -> approval -> final response

Every arrow is another place where the workflow can fail.

The wrong customer record may enter the context. A tool may receive valid JSON with the wrong business meaning. Authentication may expire halfway through a run. A background worker may retry after the external action has already succeeded.

Prompt changes cannot repair those failures.

Tool Calls Create Real Side Effects

Generating a poor paragraph is inconvenient. Sending an email, cancelling an order, or updating a CRM record changes something outside the model.

That means consequential tools need the same protections as any other production service: input validation, authorization, timeouts, retry policies, idempotency, and audit logs.

A simplified refund tool might protect repeated calls with an action identifier:

async def refund_order(order_id: str, action_id: str):
previous = await actions.find(action_id)
if previous:
return previous.result

return await actions.run_once(
    action_id,
    lambda: payments.refund(order_id),
)

The important instruction is not hidden in the prompt. It is enforced by the application: one logical action should produce one external effect.

The production implementation still needs atomic storage and protection against concurrent requests, but the boundary is clear. The model can propose the action. The system decides whether that action is safe to execute.

Traces Are More Useful Than Chat Logs

A chat transcript shows what the user and model said. It rarely explains why an agent selected a tool, what arguments it generated, how long the API took, or which state was restored after a retry.

An end-to-end trace should connect the model turn, tool call, approval, API response, state change, and final answer under one run identifier.

The OpenAI Agents SDK tracing documentation reflects this approach. Its traces can capture generations, function calls, handoffs, guardrails, and custom events as parts of one workflow.

This makes a useful debugging question possible:

Where did the actual behavior first diverge from the expected behavior?

That is much more precise than asking why the model was “confused.”

Some Actions Should Pause

Not every decision should be automated simply because the agent can call the necessary tool.

Reading an order status may be low risk. Issuing a large refund, deleting a record, publishing content, or changing account permissions may require human approval.

The human-in-the-loop guidance for the OpenAI Agents SDK demonstrates a pause, approve or reject, and resume pattern for sensitive tool calls. The specific framework is optional. The control is not.

Experienced AI engineers define these boundaries before production incidents define them instead.

Evals Should Test the Workflow

Testing ten prompts against ten expected answers is useful, but it does not test the complete system.

Workflow evaluations should also ask whether the correct tool was selected, whether its arguments were valid, whether unauthorized data was excluded, whether duplicate actions were prevented, and whether low-confidence situations were escalated.

Every production failure should become a regression case. Over time, the evaluation set becomes a record of how the system has failed in the real world.

At Spaculus Software, this is why AI engineering work extends beyond model integration. Reliable systems require workflow design, tool boundaries, evaluation, observability, and safe recovery paths around the model.

The prompt still matters. It guides intent, tone, reasoning, and tool selection.

But when an AI application breaks in production, the most valuable engineer in the room is usually not asking, “How should we reword this?”

They are asking, “What exactly happened between the request and the result?”

What workflow failure has been hardest for you to diagnose?

I Connected an LLM to Production. Here’s What Broke

Paul-S — Tue, 08 Sep 2026 10:54:30 +0000

The demo looked ready.

The application could search company documents, answer support questions, classify requests, and create tickets through an API. It handled every test prompt we gave it.

Then real users arrived.

They asked incomplete questions. They pasted entire email threads. They used terms that did not exist in our documentation. Some requests matched several policies, while others required information the system could not access.

The LLM was still producing fluent answers. The workflow around it was falling apart.

The Output Was Valid Until It Wasn’t

During testing, the model returned predictable JSON:

{
"category": "billing",
"priority": "high",
"requires_human": true
}

Production inputs were less predictable. Sometimes the model changed a field name, returned an unsupported category, or added an explanation outside the JSON object.

A response that looks reasonable to a person can still break an application.

The first improvement was to treat model output as untrusted input. Every response had to pass schema validation before the application could use it.

from typing import Literal
from pydantic import BaseModel

class TicketAction(BaseModel):
category: Literal["billing", "technical", "account"]
priority: Literal["low", "medium", "high"]
requires_human: bool

def validate_action(model_output: dict):
return TicketAction.model_validate(model_output)

Structured output reduced parsing failures, but it did not prove that the selected category was correct. Format validation and business validation became separate steps.

RAG Retrieved Something Relevant but Not Correct

The knowledge base contained current policies, archived documents, internal notes, and several pages describing similar processes.

The retrieval system often returned a document related to the question. That did not mean it returned the document needed to answer it.

One customer asked about cancelling a subscription. The system retrieved an older cancellation policy because it shared more words with the question than the current policy did.

The solution was not simply increasing the number of retrieved chunks.

Documents needed version metadata, ownership, effective dates, and access rules. Archived content had to be removed from normal retrieval. The application also needed to recognise conflicting evidence and stop instead of asking the LLM to choose a convenient answer.

RAG improved the model’s access to information. It did not remove the need to manage that information.

Tool Calling Turned Small Errors Into Real Actions

A wrong answer is harmful. A wrong action can be worse.

Once the LLM could create tickets, update CRM records, and trigger notifications, every uncertain decision had operational consequences.

Retries created duplicate records. Incorrect parameters sent requests to the wrong queue. A broad service account gave the agent access to functions it never needed.

We reduced this risk by giving each tool one narrow purpose. Tool arguments were validated outside the model, write operations used idempotency keys, and high-impact actions required confirmation.

This follows the principle behind OWASP’s guidance on excessive agency: limit available tools, permissions, functionality, and autonomy.

The model could recommend an action. The application decided whether that action was allowed.

One User Request Became Eight Model Calls

The original demo made one request to an LLM.

The production version classified the user’s intention, rewrote the search query, retrieved documents, reranked the results, generated an answer, checked the answer, selected a tool, and summarized the result.

Each step appeared reasonable on its own. Together, they created noticeable latency and unpredictable costs.

Agents made the problem harder because the number of steps could change for every request. A simple question might finish immediately, while an ambiguous request could enter a loop of repeated searches and tool calls.

We added limits for execution time, model calls, retries, retrieved context, and total tokens. Smaller models handled basic classification, while stronger models were reserved for decisions that needed deeper reasoning.

The goal was not to minimize every model call. It was to ensure that each call justified its cost and delay.

Our Logs Said Everything Was Successful

The API returned 200. The workflow was still wrong.

Traditional logs showed that the request completed, but they did not explain which documents were retrieved, why a tool was selected, or where the answer changed.

Production debugging required a trace of the complete workflow. We recorded the prompt version, model version, retrieved document IDs, tool arguments, tool results, latency, token usage, validation failures, and final outcome.

Sensitive customer data was removed or masked before storage. Observability should help investigate failures without creating a new privacy problem.

The NIST Generative AI Profile also emphasizes ongoing monitoring, documented responsibilities, incident handling, and human oversight for generative AI systems.

The Evaluation Set Was Too Polite

Our early tests asked clear questions with known answers.

Real users did not behave like an evaluation dataset.

They submitted vague instructions, conflicting requests, spelling mistakes, old account details, pasted web content, and questions requiring permissions they did not have.

The evaluation set had to include those conditions. We added cases involving stale documents, conflicting sources, API timeouts, repeated actions, missing information, prompt injection, and requests that should be refused or escalated.

Every production failure became a new regression test.

That changed evaluation from a task completed before launch into a process that continued throughout the product’s life.

The LLM Was Only One Part of the Product

The biggest lesson was simple: connecting an LLM to an application is not the same as building a production AI system.

Reliable AI requires structured outputs, managed knowledge, secure tools, permission checks, human approvals, observability, cost controls, and realistic evaluations.

This is also what teams should examine when they plan to hire AI engineers. Prompting skills matter, but production AI development also requires backend engineering, API design, security, data management, testing, and operational thinking.

The LLM did not suddenly become less capable after deployment. Production simply exposed every assumption that the demo never tested.

What broke first when you moved an LLM application into production?