DEV Community: Paul-S

Python 3.15 Switched UTF-8 On. One Export Broke

Paul-S — Thu, 12 Nov 2026 09:47:02 +0000

The deployment looked harmless.

We upgraded a Windows export service from Python 3.14 to Python 3.15. The application started correctly, the tests passed, and the scheduled job produced its CSV at the usual time.

Then finance opened the file.

Customer ID,Name,City
1042,RenÃ©e,MontrÃ©al

The records were complete. The columns were in the right order. No exception had been logged.

The code had not changed. Its default encoding had.

What Changed in Python 3.15?

Python 3.15 was released on October 9, 2026. One of its important compatibility changes comes from PEP 686: UTF-8 mode is now enabled by default.

That means text operations without an explicit encoding now use UTF-8, including:

open()
standard input and output
pipes
filesystem encoding operations

Before Python 3.15, the default could depend on the operating system and locale. A Windows application might therefore write text using a local encoding such as Windows-1252.

This code contains that dependency:

import csv

def export_customers(customers, path):
    with open(path, "w", newline="") as file:
        writer = csv.writer(file)
        writer.writerow(["Customer ID", "Name", "City"])

        for customer in customers:
            writer.writerow([
                customer.id,
                customer.name,
                customer.city,
            ])

There is no encoding argument. The export contract is being decided by the Python runtime.

Python 3.15 now makes that decision consistently: UTF-8.

Why Did the Export Succeed?

The failure happened after Python wrote the file.

Consider the name Renée. Its é character is represented differently in UTF-8 and Windows-1252.

Python 3.15 wrote valid UTF-8 bytes. The legacy finance application still assumed Windows-1252, so it interpreted those bytes incorrectly and displayed RenÃ©e.

No encoding error occurred because both programs completed their own operations successfully. They simply disagreed about the file format.

This is why encoding bugs can survive unit tests. A test may confirm that the file exists and contains the correct number of rows without checking the bytes another system will receive.

Make Encoding Part of the Export Contract

The correct fix depends on the consumer.

If the receiving system requires Windows-1252, preserve that requirement explicitly:

with open(
    path,
    "w",
    encoding="cp1252",
    errors="strict",
    newline="",
) as file:
    writer = csv.writer(file)

Using errors="strict" is intentional. Replacing unsupported characters silently can damage customer names, addresses and product data.

If the consumer accepts UTF-8 but needs a byte-order mark to detect it correctly, use utf-8-sig:

with open(
    path,
    "w",
    encoding="utf-8-sig",
    newline="",
) as file:
    writer = csv.writer(file)

For ordinary UTF-8 integrations that do not require a byte-order mark, use encoding="utf-8".

The important part is that the decision lives in the code rather than the machine running it.

Test the File, Not Only the Rows

An export test should verify the encoding expected by the receiving application.

import codecs

def test_customer_export_uses_utf8_bom(tmp_path):
    output = tmp_path / "customers.csv"

    export_customers(
        customers=[
            Customer(
                id=1042,
                name="Renée",
                city="Montréal",
            )
        ],
        path=output,
    )

    raw = output.read_bytes()

    assert raw.startswith(codecs.BOM_UTF8)

    decoded = raw.decode("utf-8-sig")
    assert "Renée" in decoded
    assert "Montréal" in decoded

This test checks the actual integration boundary. It will fail if someone changes the encoding, removes the required byte-order mark or writes the file using a locale-dependent default.

A fixture should also include characters beyond basic ASCII. Tests containing only names such as John Smith cannot expose this class of problem.

Find Other Hidden Defaults Before Upgrading

Python can warn when code opens text files without specifying an encoding.

Run the test suite with:

python -X warn_default_encoding -m pytest

Or enable the same behavior through an environment variable:

PYTHONWARNDEFAULTENCODING=1 python -m pytest

Each EncodingWarning deserves review.

Configuration files, templates and application-owned data should usually specify UTF-8. A file that intentionally follows the operating system locale can use:

open(path, encoding="locale")

Python 3.15 can temporarily restore the earlier behavior with PYTHONUTF8=0 or -X utf8=0. Those options are useful for comparing behaviors during diagnosis. They should not become the permanent fix for an undefined file contract.

The Upgrade Lesson

Python 3.15 choosing UTF-8 by default is a sensible change. It removes a major source of cross-platform inconsistency.

It also exposes applications that allowed their environment to choose an integration format.

Before upgrading, inspect every place the application exchanges text with another system: CSV exports, fixed-width files, command-line pipes, email attachments, uploaded documents and legacy database imports.

For teams preparing to hire Python developers, encoding migrations are a useful interview topic. Strong candidates should discuss downstream consumers, byte-level tests and compatibility requirements rather than simply adding encoding="utf-8" everywhere.

Python 3.15 did not corrupt the export. It made an undocumented assumption visible.

Which text integration in your application still depends on the machine running it?

OpenAI Agents Can Swap Tools Mid-Session. Can Permissions Keep Up?

Paul-S — Thu, 05 Nov 2026 08:10:20 +0000

At 10:02, a support agent could search customers and read open cases.

At 10:06, the application added two tools to the same session:

issue_refund
close_account

The conversation did not restart. The customer context, earlier instructions, and read-only approval were still there.

The agent now had a different job without receiving a new identity.

OpenAI’s Python SDK 3.42.0, released on October 31, 2026, added agent-session tool settings. The current Agents API reference says an update can replace a session’s tools.

Functions and MCP servers change on the next model call, while built-in tools change by the next turn.

That creates a security question that cannot be solved inside the prompt:

When the available tools change, which earlier decisions are still valid?

A Tool Update Is a Replacement

The update operation does not append one harmless capability. The submitted array replaces the session’s tool list.

Omitting tools keeps the current list. Sending null or an empty array clears it.

An abbreviated request has this shape:

POST /v1/agents/sessions/session_123
Authorization: Bearer $OPENAI_API_KEY
Content-Type: application/json

{
  "agent": {
    "tools": [
      { "type": "function", "name": "search_cases" },
      { "type": "function", "name": "issue_refund" }
    ]
  }
}

Before the update, the agent could inspect a customer case.

After the update, it could move money.

If the application stores only session_id, it has no durable record showing that the session’s authority changed.

Permission Has Three Clocks

I find it useful to treat a live agent session as three clocks that can move independently.

The conversation clock contains what the user requested, what the model learned, and what earlier turns decided.

The capability clock represents the tools, schemas, and credentials currently available.

The authority clock represents the user’s current role, tenant, approval, and policy version.

A session is safe only when those clocks agree at execution time.

The conversation might say, “The supervisor approved this case,” while the tool arguments now refer to another customer.

A tool might still exist while its credential gains a wider scope.

Neither situation requires malicious behavior. Stale state is enough.

Bind Approval to a Capability Snapshot

An approval should cover one concrete action under one known capability set. It should not become permanent permission attached to an entire conversation.

One control is to create a fingerprint whenever the tool registry changes:

import hashlib
import json


def capability_fingerprint(tools: list[dict]) -> str:
    snapshot = [
        {
            "name": tool["name"],
            "schema": tool["schema"],
            "risk": tool["risk"],
            "credential_scope": tool["credential_scope"],
        }
        for tool in sorted(tools, key=lambda item: item["name"])
    ]

    encoded = json.dumps(
        snapshot,
        sort_keys=True,
        separators=(",", ":"),
    )

    return hashlib.sha256(encoded.encode()).hexdigest()

Store that fingerprint with the proposed action and approval:

approval = {
    "principal_id": "user_42",
    "tenant_id": "tenant_acme",
    "tool": "issue_refund",
    "arguments_hash": "sha256:7c1...",
    "capability_fingerprint": "sha256:a84...",
    "policy_version": 17,
    "expires_at": "2026-11-05T11:15:00Z",
}

Before execution, compare that record with the current identity, tenant, arguments, policy, and capability fingerprint.

A relevant mismatch should invalidate the approval and create a new review request.

This prevents an old decision from silently authorizing a changed system.

The Executor Still Owns the Boundary

The model may request issue_refund.

The tool handler must decide whether that request is allowed.

That decision should use server-derived identity and tenant context, not values supplied by the model. It should also enforce business limits and prevent duplicate execution through a stable operation ID.

OpenAI’s MCP guidance provides allowed_tools for restricting what an agent can discover and call. It also recommends keeping credentials outside reusable definitions and logs.

However, the application still owns resource-level authorization.

A tool allowlist answers:

Can this agent call the refund tool?

The business policy answers:

Can this user refund this order, for this tenant, for this amount, right now?

Both checks are necessary.

Test the Transition, Not Only Each Tool

Unit tests usually confirm that search_cases works and issue_refund rejects an unauthorized user.

You also need a test for the moment between those states:

def test_old_approval_fails_after_toolset_change():
    tools_v1 = [SEARCH_CASES, ISSUE_REFUND_V1]
    tools_v2 = [SEARCH_CASES, ISSUE_REFUND_V2]

    approval = approve(
        tool="issue_refund",
        capability_fingerprint=capability_fingerprint(tools_v1),
    )

    decision = authorize(
        approval=approval,
        requested_tool="issue_refund",
        current_fingerprint=capability_fingerprint(tools_v2),
    )

    assert decision == "REVIEW_REQUIRED"

Also test:

A removed tool
A changed argument schema
A narrower or broader credential
An expired approval
A tenant switch
A changed policy version

These transitions are where a valid session can become an invalid authority container.

One Current Documentation Warning

As of November 5, 2026, OpenAI’s API reference says session tools can be replaced. The Python SDK release notes also list agent-session tool settings.

However, the separate configuration guide still says tools cannot be updated through that endpoint.

Until the documentation is consistent, pin the SDK version, test the behavior in a disposable session, and recheck the API reference before deployment.

Tool Changes Are Policy Changes

Swapping tools mid-session can support phased workflows and keep one conversation alive across different tasks.

It should receive the same security treatment as changing an API permission or production role.

For teams planning to hire AI engineers, this also creates a useful interview exercise:

Give a candidate a read-only agent session, add a write tool, and ask which stored decisions must be invalidated.

Their answer will reveal more about production readiness than another prompt-engineering demonstration.

An agent session may keep its history when its tools change. Its authority should not silently survive a meaningful capability change.

What would your application invalidate when a live agent receives a more powerful tool?

The Agent Chose the Right Tool at the Wrong Time

Paul-S — Mon, 02 Nov 2026 10:39:14 +0000

The Agent Chose the Right Tool at the Wrong Time

Consider this fictional order-management trace:

10:02:11  Customer asks to change the delivery address
10:02:12  Agent reads order: PAID, version 17
10:02:13  Agent selects update_delivery_address
10:02:15  Warehouse starts packing: PACKING, version 18
10:02:17  Support manager approves the change
10:02:18  Agent updates the address
10:02:20  Shipping label still contains the old address

The model understood the request.

It selected the correct customer, order, and tool. The support manager even approved the action.

The problem appeared in the five seconds between selection and execution. By the time the tool ran, the order had entered a different business state.

This is a time-of-check to time-of-use problem inside an AI workflow.

A Correct Tool Call Can Become Stale

Agent evaluations often test tool selection like this:

User request: Change the delivery address
Expected tool: update_delivery_address
Actual tool: update_delivery_address
Result: PASS

That test proves the model can map an instruction to a function. It says nothing about whether the function remains safe to run later.

Real workflows introduce delays:

A person must approve the action.
Another service updates the record.
A background job changes its status.
A webhook arrives while the agent is paused.
The model retries after a timeout.
A queue resumes the run on another worker.

The selected tool may still be semantically correct while its original assumptions are no longer true.

Give Every Proposed Action a Shelf Life

A tool call that changes business data should carry more than a function name and arguments.

It should also contain the conditions under which the action was proposed:

from dataclasses import dataclass
from datetime import datetime


@dataclass(frozen=True)
class ActionProposal:
    tool: str
    entity_id: str
    expected_version: int
    allowed_states: tuple[str, ...]
    expires_at: datetime

For the address-change example, the proposal could say:

proposal = ActionProposal(
    tool="update_delivery_address",
    entity_id="order_4821",
    expected_version=17,
    allowed_states=("PAID", "QUEUED"),
    expires_at=approval_deadline,
)

The agent can recommend the action. The application records the state that made the recommendation valid.

Execution then becomes a separate decision.

Recheck State Inside the Tool Boundary

Do not rely on the model to inspect the order again and decide whether its earlier plan is still safe.

The tool executor should enforce the rule:

from datetime import UTC, datetime


def execute_address_change(proposal, new_address):
    order = orders.get_for_update(proposal.entity_id)

    if datetime.now(UTC) > proposal.expires_at:
        return {"status": "REPLAN", "reason": "proposal_expired"}

    if order.version != proposal.expected_version:
        return {"status": "REPLAN", "reason": "order_changed"}

    if order.status not in proposal.allowed_states:
        return {"status": "ESCALATE", "reason": "shipping_started"}

    orders.update_address_if_version(
        order_id=order.id,
        expected_version=proposal.expected_version,
        address=new_address,
    )

    return {"status": "COMPLETED"}

get_for_update() and update_address_if_version() are conceptual repository methods. In a real system, they could use a database transaction, row lock, or compare-and-swap operation.

The important part is where the checks happen: immediately before the write, inside the trusted execution layer.

Approval Does Not Last Forever

Current agent frameworks support pausing a run until a person approves a sensitive tool call. OpenAI’s human-in-the-loop documentation describes serializing a paused run and resuming it after the decision.

That solves one problem: the agent cannot perform the action before approval.

It does not prove that the world remained unchanged while approval was pending.

An approval should therefore apply to a specific proposal, entity version, parameter set, and time window. If any of those change, the previous approval should expire.

A manager approving a $50 refund should not accidentally approve a recalculated $500 refund. Approval for order version 17 should not authorize an operation against version 19.

Test the Delay, Not Only the Prompt

A useful test suite needs to change state between planning and execution.

Test cases should include:

Agent proposes action → entity changes → action is rejected
Agent proposes action → approval expires → workflow replans
Agent proposes action → duplicate callback arrives → one write occurs
Agent proposes action → permission changes → action is denied
Agent proposes action → parameters change → approval is requested again

These tests evaluate the system surrounding the model. They are often more valuable than adding another set of prompt examples.

OWASP describes excessive autonomy as part of its Excessive Agency risk. Its guidance recommends independent authorization and approval for high-impact operations.

Temporal checks add another boundary: authorization must still be valid at the moment of execution.

Log the Decision Boundary

When this failure reaches production, a simple “tool succeeded” log will hide the problem.

Record operational facts such as:

When the action was proposed
When approval was granted
When execution started
Which entity version was read
Which version existed at execution
Which precondition rejected or allowed the write
Whether the call was new, resumed, or retried

This creates an audit trail without storing private chain-of-thought. Engineers can reconstruct what the system observed and why the executor allowed the action.

The Engineering Question I Would Ask

When evaluating someone for an agent project, I would provide the opening trace and ask:

Where should this workflow recheck state, and what should happen when the state has changed?

A strong answer should cover versioning, transactions, expiring approvals, idempotency, retries, cancellation, and escalation. Framework knowledge matters, but production judgment is revealed at these boundaries.

For teams planning to hire AI engineers, this type of scenario is more informative than asking candidates to build another tool-calling demo. It reflects how Spaculus Software approaches workflow automation: models can propose actions, while the application decides whether those actions are still valid.

The agent chose the correct tool.

The system failed because it treated a correct decision as permanent.

Which state change could make a tool call unsafe in your current workflow?

Laravel AI Failover Worked. My JSON Contract Didn’t.

Paul-S — Thu, 29 Oct 2026 12:30:22 +0000

A Laravel AI failover can produce a surprisingly successful failure.

09:42:11  primary_provider  429 rate_limit
09:42:12  fallback_provider  200 success
09:42:12  application        Missing key: recommended_action

The primary provider was unavailable. Laravel moved the request to the backup provider, which returned a valid response.

Infrastructure saw success.

The application crashed.

The Prompt Was Pretending to Be a Contract

The agent had been instructed to return JSON:

Return JSON using this structure:

{
  "risk": "low|medium|high",
  "recommended_action": "approve|review|reject",
  "reason": "string"
}

The primary model usually followed that instruction.

The fallback model returned this instead:

{
  "risk_level": "medium",
  "action": "manual_review",
  "explanation": "The account requires additional verification."
}

Nothing about that answer was nonsensical. A person could understand it immediately.

PHP could not.

The parser expected recommended_action. The fallback supplied action. Even its value changed from review to manual_review.

Failover had restored availability without preserving substitutability.

Provider Failover Is Not Contract Failover

The Laravel AI SDK allows an ordered list of providers:

use Laravel\Ai\Enums\Lab;

$response = (new RefundDecision)->prompt(
    'Evaluate refund request RF-1084.',
    provider: [
        Lab::OpenAI,
        Lab::Anthropic,
    ],
);

If the primary provider is rate-limited or unavailable, the SDK can move to the next one.

That answers one question:

Can another provider complete the request?

It does not automatically answer another:

Will every provider behave identically enough for my application?

Providers differ in model behavior, supported tools, configuration options, and structured-output implementations. Laravel hides much of the transport complexity, but the application still owns its business contract.

The Laravel AI SDK’s recent changelog includes provider-specific structured-output fixes. That is a useful reminder that multi-provider compatibility must be tested, not assumed.

Replace JSON Instructions With a Schema

Laravel agents can implement HasStructuredOutput instead of asking for JSON in natural-language instructions.

<?php

namespace App\Ai\Agents;

use Illuminate\Contracts\JsonSchema\JsonSchema;
use Laravel\Ai\Contracts\Agent;
use Laravel\Ai\Contracts\HasStructuredOutput;
use Laravel\Ai\Promptable;

class RefundDecision implements Agent, HasStructuredOutput
{
    use Promptable;

    public function instructions(): string
    {
        return <<<'PROMPT'
        Evaluate the refund request.

        Return a risk level, a recommended action,
        and a short explanation based only on the
        supplied account and policy information.
        PROMPT;
    }

    public function schema(JsonSchema $schema): array
    {
        return [
            'risk' => $schema->string()->required(),
            'recommended_action' => $schema->string()->required(),
            'reason' => $schema->string()->required(),
        ];
    }
}

The response can then be accessed like an array:

$response = (new RefundDecision)->prompt(
    'Evaluate refund request RF-1084.',
    provider: [
        Lab::OpenAI,
        Lab::Anthropic,
    ],
);

$action = $response['recommended_action'];

This is stronger than “please return JSON” because the output structure becomes part of the agent definition.

It still should not be the final validation layer.

Convert Model Output Into a Domain Type

A string can satisfy the schema and still contain an unsupported business value.

The application should reject anything outside its own vocabulary:

enum RefundRisk: string
{
    case Low = 'low';
    case Medium = 'medium';
    case High = 'high';
}

enum RefundAction: string
{
    case Approve = 'approve';
    case Review = 'review';
    case Reject = 'reject';
}

final readonly class RefundResult
{
    public function __construct(
        public RefundRisk $risk,
        public RefundAction $action,
        public string $reason,
    ) {}

    public static function fromAiResponse($response): self
    {
        return new self(
            risk: RefundRisk::from($response['risk']),
            action: RefundAction::from(
                $response['recommended_action']
            ),
            reason: $response['reason'],
        );
    }
}

Now manual_review, approve_now, or an empty value fails before it reaches refund logic.

That failure is useful. It protects the business operation from a response that is valid JSON but invalid for the application.

Test Every Provider as a Primary Provider

A test that always uses the normal provider does not test failover.

Run a small contract suite against each configured provider independently:

$providers = [
    Lab::OpenAI,
    Lab::Anthropic,
];

foreach ($providers as $provider) {
    $response = (new RefundDecision)->prompt(
        'Account is seven days old. Policy requires review.',
        provider: $provider,
    );

    $result = RefundResult::fromAiResponse($response);

    expect($result->risk)->toBeInstanceOf(RefundRisk::class);
    expect($result->action)->toBe(RefundAction::Review);
}

Keep this separate from fast unit tests because it makes real provider calls. Run it in staging, before changing models, and after upgrading the SDK.

Also test what happens when validation fails. The safe behavior might be to retry once, send the request to manual review, or stop the workflow. It should not silently invent a default action.

A Backup Provider Is a Different Dependency

The important lesson is not that Laravel’s failover failed.

It worked exactly where it was supposed to work.

The mistake was treating two probabilistic providers as interchangeable implementations of the same deterministic interface.

If you are evaluating the best Laravel development company for an AI-enabled application, ask how the team tests structured outputs across every fallback provider. At Spaculus Software, provider availability and business-contract validity are treated as separate production concerns.

A 200 OK means the provider answered.

It does not mean your application received an answer it can safely use.

What contract would break first if your Laravel application switched AI providers today?

SaaS Rate Limits Break Differently With AI Agents

Paul-S — Mon, 26 Oct 2026 12:31:22 +0000

A production trace for one user request might look like this:

RUN 7F21
Goal: Prepare a renewal quote

Model calls:        4
Database queries:   7
CRM calls:          3
Pricing API calls:  2
CRM response:       429
SDK retry:          1
Agent retry:        1
Final status:       Unknown

The customer clicked one button.

The system performed eighteen operations, retried the same dependency at two different layers and no longer knew whether the CRM update had succeeded.

That is how rate limiting changes when an AI agent enters a SaaS product.

Traditional SaaS rate limiting controls how many requests a user or tenant can send. Agent rate limiting must also control how much work a single goal is allowed to create.

One Request Is No Longer One Unit of Work

A normal API route has a reasonably predictable cost.

POST /reports

It validates input, queries data, creates a report and returns a response.

An agent starts with an objective rather than a fixed execution path. It may search documents, ask the model to choose a tool, call an API, inspect the response and repeat the process.

Two users can submit the same request and create very different workloads.

One run may finish after three tool calls. Another may search five times, load more context after each result and repeatedly call a throttled service.

The user-level limit still sees one request.

The infrastructure sees a small distributed system.

A 429 Can Leave the Workflow Half Finished

In an ordinary request-response flow, 429 Too Many Requests usually means the requested operation was rejected.

That assumption becomes dangerous after an agent has already completed earlier steps.

Imagine an onboarding agent that:

Creates a customer account
Adds the customer to the CRM
Generates an invoice
Sends a welcome email

If the invoice API returns 429, the first two changes may already exist. Restarting the entire agent can create duplicate accounts or CRM entries.

The failure is no longer simply “rate limited.” It is partially completed.

This is why agent workflows need persistent step state:

CREATE_ACCOUNT     completed
UPDATE_CRM         completed
CREATE_INVOICE     waiting
SEND_EMAIL         not_started

Resume from the failed step. Do not ask the model to reconstruct progress from conversation history.

Rate-Limit the Run, Not Only the Endpoint

Each agent run should receive an execution budget before it starts.

from dataclasses import dataclass
from time import monotonic


@dataclass
class RunBudget:
    model_calls: int = 6
    tool_calls: int = 12
    external_writes: int = 2
    retries: int = 3
    deadline: float = monotonic() + 45

    def consume(self, resource: str, amount: int = 1):
        if monotonic() > self.deadline:
            raise RuntimeError("RUN_DEADLINE_EXCEEDED")

        remaining = getattr(self, resource)

        if remaining < amount:
            raise RuntimeError(f"{resource.upper()}_BUDGET_EXCEEDED")

        setattr(self, resource, remaining - amount)

This budget answers questions an endpoint limiter cannot:

How many times may the model reconsider its plan?
How many tools may one run call?
How many external writes may it perform?
How many retries are allowed across every layer?
How long can the run remain active?

The agent can choose its next action. The surrounding application decides whether enough budget remains to execute it.

Retries Need One Owner

Agent systems often contain several automatic retry layers.

The model retries a tool. The orchestration framework retries the model. The API client retries a 429. The queue worker retries the entire job.

Two retries at each of four layers can turn one failed call into many attempts.

OpenAI’s rate-limit guidance specifically warns that nested retry loops can multiply requests. It recommends respecting Retry-After, applying exponential backoff with jitter and limiting both attempts and total retry time. Unsuccessful requests can also count toward the rate limit.

Anthropic separately measures requests, input tokens and output tokens. Its documentation notes that short bursts can trigger throttling even when the apparent minute-level rate looks safe.

Choose one layer to own retries. Every other layer should report the failure upward.

Backoff Does Not Prevent Duplicate Actions

Backoff controls when another attempt happens. It does not make that attempt safe.

Before retrying a write, assign it a stable idempotency key derived from the agent run and intended business action:

renewal-run-7F21:create-invoice:customer-1842

The same logical action must reuse the same key, even when the model generates slightly different wording.

Stripe’s documentation explains that repeating a request with the same idempotency key can return the original result instead of creating another object.

AWS also recommends combining capped retries with exponential backoff, jitter and idempotent operations.

Measure Work Amplification

Requests per minute are no longer enough.

Track an amplification ratio for every agent workflow:

(model calls + tool calls + retries) / user requests

Also record the ratio by tenant, workflow version, model and failure type.

A sudden increase can reveal a broken stopping condition, poor retrieval, a looping planner or a dependency repeatedly returning incomplete data.

This metric connects reliability, cost and abuse detection. A tenant sending only ten requests could still consume more capacity than another tenant sending one thousand predictable API calls.

The Limit Is Part of the Product

Agent limits should not exist only as invisible infrastructure settings.

When a run exhausts its budget, the product needs a defined outcome: pause, request approval, schedule continuation, return partial work or escalate to a human.

For teams working with a SaaS application development company, this is an important architecture question. Ask how the system limits complete agent runs, prevents duplicate side effects and allocates usage fairly between tenants—not only how many requests its API gateway allows.

A rate limit should not merely stop traffic.

It should stop one uncertain goal from becoming an unlimited amount of work.

What does your SaaS product limit today: users, endpoints, tenants or complete agent runs?

What a customer support chatbot really costs per month (real API bill breakdown)

Paul-S — Thu, 22 Oct 2026 11:00:32 +0000

A chatbot can cost less than expected to generate answers and more than expected to retrieve the information behind them.

That sounds contradictory until you examine an actual workload.

This breakdown models a customer-support chatbot handling 50,000 conversations per month. It uses published API prices available in October 2026.

This is not presented as a private customer invoice. It is a reproducible estimate with every assumption exposed, so you can replace the numbers with your own.

Start With the Workload, Not the Model Price

Assume the chatbot handles:

50,000 conversations per month
Four generated replies per conversation
200,000 total model responses
1,800 input tokens and 250 output tokens for a normal reply
Five percent of requests routed to a stronger model
One knowledge-base search for each generated reply

The input includes system instructions, conversation history, customer information, and retrieved support content.

That last part matters. A chatbot with a growing conversation history can consume far more input tokens than a single-question demo.

The Model Bill: $147.95

For routine questions, the system uses GPT-6 Luna.

OpenAI’s published standard pricing lists GPT-6 Luna at $0.10 per million input tokens and $0.50 per million output tokens.

Ninety-five percent of 200,000 responses equals 190,000 Luna calls.

Those calls process:

342 million input tokens   = $34.20
47.5 million output tokens = $23.75

Luna subtotal              = $57.95

The remaining 10,000 requests are harder cases. They may involve conflicting policies, unusual account histories, or complicated troubleshooting.

Route those requests to GPT-6.1 Sol, currently priced at $2 per million input tokens and $10 per million output tokens.

Assuming 2,500 input tokens and 400 output tokens for each escalated request:

25 million input tokens = $50.00
4 million output tokens = $40.00

Sol subtotal            = $90.00

The combined generation bill is:

$57.95 + $90.00 = $147.95

Using the strongest model for every request would be simpler, but it would waste money on password resets, delivery questions, and other predictable tasks.

Model routing only works when evaluations prove that the cheaper model can handle those requests safely.

Retrieval Costs More Than Generation

A support chatbot normally searches product documentation, policies, previous tickets, or account information before answering.

OpenAI currently charges $2.50 per 1,000 file-search calls through the Responses API. Storage costs $0.10 per GB per day, with the first gigabyte free.

At 200,000 searches per month:

200,000 / 1,000 × $2.50 = $500

For a four-gigabyte knowledge base:

3 billable GB × $0.10 × 30 days = $9

Retrieval therefore adds $509, more than three times the model-generation bill.

This is the line item many estimates miss.

Not every message needs a new search. Greetings, confirmations, and follow-up questions can sometimes reuse existing context. Query classification and retrieval caching can reduce costs, but only if they do not introduce outdated answers.

The Infrastructure Still Has a Bill

The chatbot also needs an application around the model.

For this example, reserve:

Backend, database and queue: $140
Logs, traces and alerts:      $90

These are planning allowances rather than universal vendor prices.

Your number will depend on traffic patterns, retention periods, cloud provider, availability requirements, and how much conversation data you store.

Add a ten-percent allowance for failed requests, retries, traffic spikes, and evaluation runs:

Metered AI and retrieval: $656.95
Ten-percent headroom:      $65.70
Infrastructure:           $230.00

Estimated monthly total:  $952.65

That equals approximately $0.019 per conversation, or just under two cents.

Build Your Own Cost Calculator

Here is the complete Python calculation:

conversations = 50_000
replies = conversations * 4

luna_calls = int(replies * 0.95)
sol_calls = replies - luna_calls


def model_cost(
    calls,
    input_tokens,
    output_tokens,
    input_rate,
    output_rate,
):
    input_cost = (
        calls * input_tokens / 1_000_000 * input_rate
    )
    output_cost = (
        calls * output_tokens / 1_000_000 * output_rate
    )

    return input_cost + output_cost


luna = model_cost(
    luna_calls,
    input_tokens=1800,
    output_tokens=250,
    input_rate=0.10,
    output_rate=0.50,
)

sol = model_cost(
    sol_calls,
    input_tokens=2500,
    output_tokens=400,
    input_rate=2.00,
    output_rate=10.00,
)

file_search = replies / 1000 * 2.50
storage = (4 - 1) * 0.10 * 30

metered = luna + sol + file_search + storage
infrastructure = 140
observability = 90
headroom = metered * 0.10

monthly_total = (
    metered
    + infrastructure
    + observability
    + headroom
)

cost_per_conversation = monthly_total / conversations

print(f"Monthly cost: ${monthly_total:.2f}")
print(
    "Cost per conversation: "
    f"${cost_per_conversation:.4f}"
)

The output is:

Monthly cost: $952.65
Cost per conversation: $0.0191

Change the conversation volume, token counts, model rates, and retrieval frequency to estimate your own workload.

What This Estimate Does Not Include

The $952.65 total excludes:

Initial chatbot development
Customer-support platform fees
Human review
Security testing
Maintenance and evaluation
Engineering time spent investigating failures

Human escalation may eventually become the largest expense.

If five percent of 50,000 conversations require five minutes of agent review, that creates more than 208 hours of monthly support work.

A chatbot that routes uncertain cases correctly can still be valuable, but those hours belong in the business case.

Regional processing can also change the invoice. OpenAI currently applies a ten-percent uplift to eligible regional-processing and FedRAMP endpoints for newer models.

Ask for the Worksheet, Not One Monthly Number

When evaluating an AI chatbot development company, ask to see the assumptions behind its operating-cost estimate.

You should be able to identify:

Expected conversation volume
Model calls and token usage
Retrieval frequency
Escalation rate
Storage requirements
Failed calls and retries
Monitoring costs
Human-review workload

The model price is only one line on the invoice.

For this example, generating answers costs about $148. Finding the information behind those answers costs $509.

That is the cost relationship worth designing around.

The Chatbot Refunded Twice. The Retry Looked Successful

Paul-S — Tue, 20 Oct 2026 11:40:56 +0000

The production trace looked harmless:

14:03:11  Refund requested for order 4821
14:03:12  Payment provider called
14:03:22  Request timed out
14:03:24  Agent selected retry
14:03:26  Refund completed: $89

The chatbot reported success. The customer received confirmation. No exception remained open.

Two days later, finance found two $89 refunds.

The first payment request had not failed. Only the response had failed to reach the chatbot. When the agent retried, the payment provider processed the same business action again.

The retry looked successful because it was successful.

That was the bug.

This is a simplified composite example of a failure pattern that appears whenever an AI agent can perform consequential actions through APIs.

The HTTP Request Failed. The Refund Did Not.

Developers often treat a timeout as evidence that an operation failed.

It is only evidence that the client did not receive a response within the expected time.

Several things may have happened:

The provider rejected the request.
The provider received nothing.
The provider completed the refund but lost the response.
The response is still travelling through an overloaded network.
The provider is processing the request asynchronously.

The chatbot cannot distinguish these outcomes from a timeout alone.

A human support agent might pause, check the payment dashboard, and confirm the order status. An AI agent may interpret the timeout as permission to repeat the tool call.

This is why “retry on failure” is not a complete reliability policy.

The Model Should Not Create the Retry Identity

Suppose the agent calls this tool:

refundPayment({
  orderId: "4821",
  amount: 89
});

After a timeout, the model generates the same arguments and calls the tool again. The two calls look identical to us, but the payment system may see two independent refund requests.

The application should create a stable action identifier before execution:

type RefundCommand = {
  actionId: string;
  orderId: string;
  amount: number;
  requestedBy: string;
};

const command: RefundCommand = {
  actionId: "refund:4821:approved-request-7",
  orderId: "4821",
  amount: 89,
  requestedBy: "support-agent"
};

The important field is actionId.

It represents the business action, not one HTTP attempt. Every retry of that refund must reuse the same identifier.

Do not ask the language model to invent or remember this value. The surrounding application should generate it from trusted workflow state and persist it before calling the payment provider.

Idempotency Must Exist at More Than One Layer

If the provider supports idempotency keys, pass the stable identifier with every attempt:

async function issueRefund(command: RefundCommand) {
  return paymentProvider.refund(
    {
      orderId: command.orderId,
      amount: command.amount
    },
    {
      idempotencyKey: command.actionId
    }
  );
}

The first request may complete without returning a response. When the retry arrives with the same key, the provider should return the original result instead of creating another refund.

That protects the external transaction, but the application still needs its own record:

await db.refundCommands.create({
  data: {
    actionId: command.actionId,
    orderId: command.orderId,
    amount: command.amount,
    status: "PENDING"
  }
});

Place a unique constraint on actionId.

If two workers receive the same command simultaneously, only one should be allowed to create the execution record. The other should read the existing state instead of starting another action.

If an external API does not support idempotency, a database check alone may not be sufficient. Two processes can both check before either records completion. Use a lock, serialized queue, transactional outbox, or another mechanism that guarantees one executor for that business action.

“Completed” Should Not Be the Only Useful State

The chatbot’s original workflow understood three outcomes:

SUCCESS
FAILURE
RETRY

That was too simple.

The refund command needed states that described what the system actually knew:

PENDING
EXECUTING
CONFIRMATION_UNKNOWN
COMPLETED
REJECTED
REQUIRES_REVIEW

A timeout after sending the request should move the command to CONFIRMATION_UNKNOWN, not directly to FAILURE.

From there, the system can query the provider using the idempotency key, inspect a webhook, or send the case to a human. The chatbot should not independently decide that repeating a financial action is safe.

Test the Missing Response, Not Just the Failed Request

Most tests simulate an API rejecting a request before doing any work.

The more dangerous case is a successful operation followed by a lost response.

A useful integration test should reproduce this sequence:

The provider accepts the refund.
The provider records the transaction.
The client receives a timeout.
The workflow retries with the same action ID.
The provider returns the original result.
The ledger contains one refund.

Then run the test with two workers executing the same command concurrently.

The assertion that matters is not how many HTTP attempts occurred. It is how many financial transactions were created.

Retry Safety Is Part of Chatbot Quality

A 2026 Kore.ai survey of 408 enterprise IT and engineering leaders found that 79% of consequential autonomous actions required manual reversal. Among those reversals, 93% were described as costly or disruptive.

Better prompts cannot solve a missing transaction boundary.

If you are evaluating an AI chatbot development company, do not ask only whether its chatbot can process a refund. Ask what happens when the refund succeeds but its response disappears.

A reliable chatbot must know more than how to complete an action.

It must know when the action may already be complete.

Laravel AI SDK v1.0 Needs a Tenant Boundary

Paul-S — Thu, 15 Oct 2026 12:16:07 +0000

Authenticated tenant: 41
Authenticated user: 842
Agent tool: FindInvoices
Invoice returned: INV-1098
Invoice tenant: 17

The model found an invoice.

The tool executed successfully. No exception appeared in the logs. The response was factually correct.

It also exposed another customer’s data.

This is the dangerous kind of AI failure because every component appears to work. The problem is not hallucination. It is missing tenant isolation.

The SDK Did Exactly What We Asked

Laravel AI SDK v1.0 gives PHP developers a first-party system for agents, conversation storage, streaming, tool approvals and middleware.

It can decide when to call a tool and validate the arguments against a schema. Laravel also supports runtime provider configuration for tenant-specific API keys.

Those capabilities reduce integration work. They do not decide which records the current tenant is allowed to access.

Consider this tool call:

{
  "tool": "find_invoice",
  "arguments": {
    "customer_id": 842
  }
}

The arguments look valid. But customer_id is not an authorization boundary. IDs may collide across imported datasets, users may belong to multiple workspaces, and a malicious prompt may ask for a different account.

The tenant must come from authenticated application state, never from the prompt or model-generated arguments.

Make Tenant Context Impossible to Omit

Start by representing the current boundary explicitly:

final readonly class TenantContext
{
    public function __construct(
        public int $tenantId,
        public int $userId,
    ) {}
}

Resolve it after authentication and authorization:

$context = new TenantContext(
    tenantId: $request->user()->current_team_id,
    userId: $request->user()->id,
);

$response = (new SupportAgent($context))
    ->forUser($request->user())
    ->prompt($request->string('message'));

Calling forUser() is important because Laravel can associate stored conversations with a user. It is not the complete boundary when one user can access several tenants or workspaces.

The active tenant must still be authorized independently.

The Model Chooses the Tool, Not the Scope

Pass TenantContext into every data-access tool through its constructor:

use App\Models\Invoice;
use Illuminate\Contracts\JsonSchema\JsonSchema;
use Laravel\Ai\Contracts\Tool;
use Laravel\Ai\Tools\Request;
use Stringable;

final class FindInvoices implements Tool
{
    public function __construct(
        private TenantContext $context,
    ) {}

    public function description(): Stringable|string
    {
        return 'Find recent invoices for the authenticated account.';
    }

    public function schema(JsonSchema $schema): array
    {
        return [
            'status' => $schema->string(),
        ];
    }

    public function handle(Request $request): Stringable|string
    {
        $invoices = Invoice::query()
            ->where('tenant_id', $this->context->tenantId)
            ->where('user_id', $this->context->userId)
            ->when(
                $request['status'] ?? null,
                fn ($query, $status) => $query->where('status', $status)
            )
            ->select(['id', 'status', 'total', 'issued_at'])
            ->latest()
            ->limit(10)
            ->get();

        return $invoices->toJson();
    }
}

Notice what is missing from the schema: tenant_id.

The model cannot supply it, replace it or persuade the tool to ignore it. Even a successful prompt injection remains trapped inside the authenticated tenant boundary.

Laravel’s own guidance for production-safe database tools recommends the same underlying pattern: pass identity through application code, restrict visible columns, cap results and use a read-only database connection where possible.

Five Places Tenant Data Can Still Escape

Fixing the Eloquent query is necessary, but AI applications move data through more than one layer.

Check these boundaries:

Conversation history: Authorize both the user and active tenant before loading an existing conversation ID.
Vector retrieval: Store tenant_id with every chunk and require it as a server-side search filter.
Cache entries: Include the tenant, user, model and permission version in the cache key.
Queued agents: Serialize immutable IDs, then reload and reauthorize them when the worker executes.
Tool approvals: Bind an approval to the tenant, user, tool and original arguments. Reject it if any context changed while the request was waiting.

A cached answer generated for Tenant 41 must never be reusable by Tenant 17 simply because both asked the same question.

Write the Test Before the Prompt

The most useful test is not “Does the agent find an invoice?”

It is “Can the agent retrieve an invoice belonging to someone else?”

it('cannot return another tenant invoice', function () {
    $context = new TenantContext(
        tenantId: $this->tenantA->id,
        userId: $this->userA->id,
    );

    $foreignInvoice = Invoice::factory()->create([
        'tenant_id' => $this->tenantB->id,
    ]);

    $result = (new FindInvoices($context))
        ->handle(new Request(['status' => $foreignInvoice->status]));

    expect($result)->not->toContain($foreignInvoice->id);
});

Add adversarial cases for guessed IDs, reused conversation IDs, poisoned retrieval metadata, delayed approvals and queue jobs executed after membership is revoked.

A Tenant Boundary Is Application Code

Laravel AI SDK v1.0 makes agents feel native to Laravel. That is valuable because tools, queues, models and tests can remain inside the same application.

The abstraction should not hide the security model.

The best Laravel development company for a multi-tenant AI product should be able to identify where tenant identity originates, how it reaches every tool and what prevents the model from changing it.

A prompt can tell an agent to respect tenant boundaries.

Only application code can enforce them.

What tenant-isolation test would you run before connecting a Laravel agent to production data?

My MEAN Agent Remembered Everything. That Was the Bug

Paul-S — Tue, 13 Oct 2026 11:48:23 +0000

My MEAN Agent Remembered Everything. That Was the Bug

I reproduced the failure during a red-team test.

10:14  Tenant: northwind
User: Where should we send the replacement?

10:14  Agent memory retrieved:
“Deliver replacement orders to the Chicago warehouse.”

10:14  Source tenant: copper

The agent remembered a real instruction. It was relevant to the question and ranked highly in semantic search.

It also belonged to another customer.

This was not a hallucination. The model did not invent the warehouse. It accurately used information our application should never have placed in its context.

Nothing in the MEAN stack had crashed. Angular rendered the conversation, Express accepted the request, Node completed the workflow, and MongoDB returned a matching document.

The system was working exactly as we had designed it.

That was the bug.

We Had Built a Memory Bucket

The first version of the agent stored almost everything:

User messages
Conversation summaries
Tool responses
Customer preferences
Previous workflow results

Each record received an embedding. Before generating an answer, the Node service searched for semantically similar memories and added the strongest matches to the prompt.

The retrieval query considered relevance, but it did not enforce ownership strongly enough. We had treated memory as a smarter version of chat history.

It is not.

An audit record, a temporary workflow value, and a durable customer preference may all contain text, but they should not have the same retention period or retrieval rules.

Every Memory Needed a Receipt

We replaced the generic memory document with an explicit contract:

type AgentMemory = {
  tenantId: string;
  principalId: string;
  purpose: "current_task" | "approved_preference";
  source: "user" | "verified_tool";
  sensitivity: "standard" | "restricted";
  content: string;
  embedding: number[];
  createdAt: Date;
  expiresAt: Date;
  status: "active" | "revoked";
};

A memory could no longer exist simply because the model considered it useful.

The application had to know:

Who owned it?
Why was it stored?
Which verified source produced it?
How long could it influence the agent?
Could the user inspect and revoke it?

Anything that could not answer those questions stayed outside the agent’s retrievable memory.

We also separated four concepts that had previously shared one collection: short-lived conversation context, workflow state, approved preferences, and audit evidence.

Audit records remained available to authorized services, but they were no longer automatically fed back into the model.

We Added Three Gates

The repair was not another system prompt. It was three application-level gates.

1. The write gate

Only approved memory types could be persisted.

Raw tool output was not automatically trusted. A webpage, uploaded document, or CRM note could contain incorrect information or malicious instructions.

OWASP’s AI Agent Security Cheat Sheet identifies memory poisoning as a distinct risk and recommends validating content, isolating memory between users and sessions, and enforcing expiration and size limits.

2. The read gate

Every retrieval included identity and lifecycle filters before semantic ranking:

const memories = await Memory.aggregate([
  {
    $vectorSearch: {
      index: "agent_memory",
      path: "embedding",
      queryVector,
      numCandidates: 100,
      limit: 8,
      filter: {
        tenantId: { $eq: tenantId },
        principalId: { $eq: principalId },
        status: { $eq: "active" },
        expiresAt: { $gt: new Date() }
      }
    }
  }
]);

The corresponding vector index must define those fields as filters.

MongoDB’s multi-tenant Vector Search guidance recommends storing a tenant_id and applying it as a pre-filter so queries only return documents belonging to that tenant.

The important rule is simple: semantic relevance must never override authorization.

3. The forget gate

Temporary memories received an expiry date, backed by a MongoDB TTL index:

MemorySchema.index(
  { expiresAt: 1 },
  { expireAfterSeconds: 0 }
);

However, TTL deletion is asynchronous. MongoDB explains that expired documents can remain until its background process removes them.

That means TTL is useful for cleanup, but it is not an authorization boundary. The application must still exclude expired or revoked memory during every read.

A user pressing “Forget this” should affect retrieval immediately.

Angular Needed a Memory Interface

The backend changes exposed a product problem: users had no way to see what the agent remembered.

We added a small Angular memory panel showing:

The saved fact
Why it was remembered
Its source
Its expiry
A revoke action

This made incorrect memories observable before they silently influenced several future conversations.

It also changed our testing. We stopped asking only, “Did the agent remember the preference?” We added a second question:

Can it still retrieve that preference after the user changes tenants, revokes it, or reaches its expiry date?

That second question caught more important defects than the first.

Better Recall Is Not Always Better Memory

Agent benchmarks often reward successful recall. Production systems must also measure forbidden recall.

Can the agent avoid retrieving another tenant’s data? Can it forget expired workflow state? Can it distinguish verified preferences from untrusted text? Can it explain why a memory entered the prompt?

These boundaries deserve attention when teams hire MEAN stack developers for agentic applications. At Spaculus Software, we treat memory as governed application data, not an unlimited extension of the model’s context window.

The safest agent is not the one that remembers everything.

It is the one that can prove why each memory exists, who owns it, and when it must stop influencing decisions.

What would your agent need to forget before you trusted it with customer data?

My RAG Eval Passed. The Citations Were Still Wrong

Paul-S — Thu, 08 Oct 2026 11:52:23 +0000

My RAG Eval Passed. The Citations Were Still Wrong

Consider this small regression test:

Question:
What is the current refund period?

Retrieved:
POLICY-2025: Customers may request refunds within 30 days.
POLICY-2026: Customers may request refunds within 14 days.

Generated answer:
Customers may request refunds within 14 days. [POLICY-2025]

The answer contains the current policy.

The expected document was retrieved.

A citation was present.

Several common RAG checks could mark this response as successful. Yet the citation points to evidence that contradicts the answer.

That is a dangerous kind of failure because the response looks verifiable.

The Green Dashboard Missed a Separate Contract

RAG evaluation often combines retrieval and generation into a few broad scores:

Did search find a relevant document?
Was the response related to the question?
Was the answer grounded in the supplied context?
Did the model include a citation?

Those questions matter, but none proves that a specific citation supports the exact claim beside it.

The TREC 2025 RAG Track treats relevance, response completeness, attribution verification and agreement as separate evaluation layers. That separation is useful because citation quality can fail even when retrieval and generation appear healthy.

A citation is more than formatting. It is a contract between one claim and one piece of evidence.

Break the Answer Into Atomic Claims

Suppose a chatbot produces this response:

The Pro plan allows 20 users and includes priority support. [DOC-7]

That sentence contains at least two claims:

Claim 1: The Pro plan allows 20 users.
Claim 2: The Pro plan includes priority support.

DOC-7 might support the user limit while saying nothing about priority support. A sentence-level evaluator may give the entire sentence a passing score because part of it is supported.

A claim-level evaluator should test each statement independently.

Recent Q-CARE research follows a similar direction by decomposing answers into atomic claims and measuring whether each claim can be verified. Its evaluation correlated more closely with human judgments than several existing RAG metrics across the tested datasets.

Add a Citation Gate After Generation

The generation step should return structured claims instead of only formatted prose.

claims = [
    {
        "text": "The refund period is 14 days.",
        "source_id": "POLICY-2026",
        "source_version": "2026-09",
    }
]

A separate citation gate can then inspect each claim:

def evaluate_citations(claims, sources, supports):
    failures = []

    for claim in claims:
        source = sources.get(claim["source_id"])

        if source is None:
            failures.append({
                "claim": claim["text"],
                "reason": "source_not_found",
            })
            continue

        if source["version"] != claim["source_version"]:
            failures.append({
                "claim": claim["text"],
                "reason": "version_mismatch",
            })
            continue

        if not supports(source["content"], claim["text"]):
            failures.append({
                "claim": claim["text"],
                "reason": "claim_not_supported",
            })

    return failures

The supports function could use deterministic rules for exact values, a natural-language-inference model for descriptive claims or a calibrated LLM judge.

I would avoid allowing the same model to write the answer and approve its own citations without another check. Research on citation attribution has found that LLM judges can also reject valid evidence, especially when terminology varies.

High-risk claims may still need sampled human review.

Test Evidence Strength, Not Keyword Overlap

A source can be relevant while remaining too weak for the wording used.

Consider these two claims:

Evidence: The update may reduce processing time.

Claim A: The update may reduce processing time.
Claim B: The update will reduce processing time by 40%.

Both discuss processing time. Only Claim A is supported.

This failure is sometimes described as citation laundering: a related source makes an overstated claim appear trustworthy. FORCEBENCH research found that generic support prompts struggled to distinguish appropriately qualified claims from stronger unsupported versions.

Your evaluator should check numbers, dates, conditions, scope and certainty. Text similarity alone cannot do that reliably.

Put the Failure Into CI

Every citation bug found in production should become a regression case.

Useful tests include:

Two versions of the same policy
One source supporting only half a sentence
A correct fact attached to the wrong document
A citation pointing to a deleted or inaccessible source
Evidence using “may” while the answer says “will”
Two documents that disagree about the same value

For teams evaluating an AI chatbot development company, asking how citations are tested is more revealing than asking whether the chatbot “uses RAG.”

A healthy RAG system should prove four things independently: it retrieved useful evidence, produced a correct answer, covered the important claims and attached each claim to evidence that actually supports it.

My evaluation passed because it tested whether citations existed.

The better test asks whether every citation deserves to be there.

What citation failure would you add to this regression suite?

AI SaaS Free Trials Are Becoming an Abuse Surface

Paul-S — Tue, 06 Oct 2026 10:50:03 +0000

Consider this composite incident trace:

TRIAL INCIDENT 042

New accounts:          186
Unique email addresses: 186
Recognized devices:      4
High-cost generations: 912
Paid conversions:        0

Nothing was technically compromised.

No password was stolen. No database was breached. Every account used the signup form correctly and stayed within its advertised trial allowance.

The product still lost money.

Four devices had created 186 identities, consumed hundreds of expensive AI operations, exported the results, and disappeared. The free trial had become an unprotected compute endpoint.

The Trial Is Now Part of Your Attack Surface

A traditional SaaS trial might expose dashboards, sample data, and limited storage. Supporting another trial user may add almost no immediate infrastructure cost.

An AI product behaves differently.

Every prompt can trigger model inference, retrieval, embeddings, image generation, document parsing, external API calls, or agent tools. A user does not need to breach the system to create damage. They only need to repeatedly consume something expensive.

This problem is accelerating. Stripe reported that its models detected 6.2 times more abusive free trials between November 2025 and February 2026. The company also noted that AI businesses are particularly exposed because each generated output creates a direct compute cost.

That changes the engineering question.

It is no longer only:

How many free requests should a trial include?

It becomes:

How much unverified economic risk should one identity control?

Signup Validation Is Necessary but Insufficient

Email verification stops basic scripts. It does not establish that 20 verified email addresses represent 20 different people.

Attackers can rotate disposable inboxes, virtual phone numbers, payment cards, residential proxies, and browser profiles. Blocking one signal usually pushes the activity toward another.

Cloudflare’s account-abuse guidance recommends examining activity across users, IP addresses, devices, locations, browsers, logins, and registrations. The important word is across.

The application needs to connect events that appear legitimate when viewed separately.

For example, one account generating 30 reports may be an engaged prospect. Twenty accounts generating the same reports from related devices are a different pattern.

This is why a CAPTCHA at registration cannot be the entire defense. The system must continue evaluating behavior after access has been granted.

Give Every Trial a Risk Budget

A useful design is to separate the advertised trial allowance from an internal risk budget.

The allowance describes what a normal customer can use. The risk budget decides how much costly activity the system permits before requesting stronger verification.

A simplified policy could look like this:

def evaluate_trial(session):
    if session.linked_accounts >= 5:
        return "DENY_EXPENSIVE_ACTIONS"

    if session.estimated_cost_hour > 8:
        return "REQUIRE_PAYMENT_METHOD"

    if session.concurrent_jobs > 3:
        return "QUEUE_AND_REVIEW"

    if session.identity_risk == "medium":
        return "REDUCE_RATE_LIMIT"

    return "ALLOW"

These numbers are illustrative, not universal defaults. A coding assistant, video generator, and document-analysis platform have completely different cost profiles.

The important design choice is that the decision uses more than request count. It considers linked identities, estimated cost, concurrency, verification, and behavior.

Rate limiting one IP address is not enough when accounts can move between networks. Limiting one account is not enough when identities are inexpensive to create.

Add Friction to the Expensive Moment

Many teams put their strongest verification at signup. That can hurt conversion before the user has experienced any value.

A better approach is progressive friction.

Let a new user explore the interface or complete a small generation. Request additional verification when the user tries to run a GPU-heavy workflow, create an API credential, launch concurrent jobs, export large outputs, or consume more than a defined cost threshold.

The response does not always need to be “block the account.” The system can:

Reduce generation speed
Queue costly operations
Limit concurrency
Request a payment method
Require business-email verification
Route unusual activity for review

This preserves a low-friction path for genuine prospects while increasing the cost of repeated abuse.

Stop Measuring Trial Success by Signups

An abuse campaign can make a growth dashboard look healthy.

Registrations increase. Activation rises. Feature usage reaches a record level. Infrastructure spending follows it upward.

If the team measures only signups and activation, abusive activity can look like product-market traction.

AI SaaS teams should also monitor cost per verified trial, model spend per conversion, linked-account recurrence, challenge completion, failed payment patterns, and the percentage of usage created by newly registered accounts.

Stripe’s guidance warns that trial abuse can distort product analytics in addition to increasing infrastructure and API costs. Polluted data can lead the product team to prioritize workflows that genuine customers barely use.

Design Revocation Before Issuing Access

Blocking new requests is only part of containment.

A trial may already have API keys, active agent runs, scheduled jobs, generated download links, or connected third-party tools. When an account is restricted, those capabilities must be revoked together.

That means trial status should be checked when an expensive action executes, not only when the user logs in.

For teams evaluating a SaaS application development company, this is an important architecture question to raise. At Spaculus Software, we treat trial controls as part of entitlement, cost, and security design rather than a billing-screen setting.

A free trial should help a genuine user reach value quickly.

It should not provide anonymous, renewable access to your most expensive infrastructure.

If someone created 100 trial accounts tonight, which resource in your SaaS product would cost you the most?

The Flutter Upgrade Passed CI. Plugins Failed on Device

Paul-S — Wed, 30 Sep 2026 11:25:51 +0000

The upgrade pipeline looked perfect.

flutter analyze                         PASS
flutter test                            PASS
flutter build apk --release             PASS
flutter build ios --release --no-codesign PASS

The pull request was approved. The new build reached the testing team.

Then the device reports arrived.

Android: Push notifications never register
iOS: Sign-in returns to a blank screen
Both: Unit tests remain green

CI had not lied. It had successfully tested what we asked it to test.

The problem was what we never asked it to test.

A Green Build Does Not Prove Plugin Behaviour

Most Flutter applications are not entirely Flutter.

A plugin usually contains a Dart-facing API and a native implementation written in Kotlin, Java, Swift, or Objective-C. The two sides communicate through platform channels or native bindings.

That creates a longer execution path:

Dart code
   ↓
Platform channel
   ↓
Plugin registration
   ↓
Android or iOS lifecycle
   ↓
Native SDK

A Dart unit test may validate the first layer while never loading the remaining four.

Flutter’s documentation explains that plugin unit tests often replace the platform implementation with mocks. That is useful for testing application logic, but a mocked method channel cannot reveal that a native class failed to register or an operating-system callback stopped firing.

The test passes because the native failure never enters the test process.

Framework Upgrades Move Native Boundaries Too

A Flutter upgrade is rarely just a Dart SDK change.

On Android, it can affect the Android Gradle Plugin, Gradle itself, Java compatibility, Kotlin configuration, manifest handling, and plugin build scripts.

Recent Flutter migration guidance illustrates this clearly. Built-in Kotlin support changes how Android projects and plugins are configured, while add-to-app projects may require manual migration because Flutter cannot rewrite the native host application automatically.

On iOS, the move toward the UIScene lifecycle changes where application events are delivered. Plugins relying on older AppDelegate callbacks may compile correctly but fail when the application receives a URL, notification, or lifecycle event on a real device.

Compilation answers one question:

Can these files become an application?

It does not answer another:

Will every native capability behave correctly when the operating system invokes it?

Debug the Boundary, Not Just the Dart Exception

When a plugin fails after an upgrade, the Dart error is often the end of the trail rather than the beginning.

A MissingPluginException, frozen method call, or empty callback could originate from several places:

The plugin was not registered with the current engine.
The Android activity or iOS scene changed before a callback returned.
A required manifest entry, entitlement, or permission is missing.
The plugin supports debug builds but fails after release optimization.
Its native dependency is incompatible with the new build toolchain.
The plugin’s Dart package was updated without a matching native migration.

Start by reproducing the failure on one known device. Then inspect native logs through Android Studio, adb logcat, Xcode, or the macOS Console.

Follow one call across the boundary:

Did Dart send the method call?
Did the native handler receive it?
Did the operating system start the requested action?
Did the callback return?
Did Flutter receive the result?

This is usually faster than repeatedly changing Dart code around a native failure.

Put Plugin Workflows Into the Upgrade Gate

A better upgrade pipeline includes tests that launch the application on the platforms it supports.

Flutter’s integration_test package can run tests on emulators, simulators, physical devices, and device farms. Use it for critical workflows that depend on plugins, such as:

Authentication redirects
Camera and file access
Push-notification registration
Payments and in-app purchases
Deep links and app links
Location and background services

Do not test only the happy path.

Deny a permission and request it again. Put the app in the background during a callback. Launch it from a notification. Open a deep link after the process has been terminated. Run the signed release build, not only the debug build.

The minimum upgrade evidence should look closer to this:

Static analysis                 PASS
Dart unit tests                 PASS
Android release build           PASS
iOS release build               PASS
Android plugin workflows        PASS
iOS plugin workflows            PASS
Background and cold-start tests PASS

CI is still important. It simply needs coverage beyond compilation.

Change the Definition of “Upgrade Complete”

An upgrade is not complete when flutter build exits with code zero.

It is complete when the application’s important native paths have been exercised on Android and iOS, using the build modes and operating-system behaviours customers will actually encounter.

That may reveal that a plugin must be upgraded, patched, replaced, or temporarily pinned. It may also expose a missing native migration in the application itself.

This is the kind of evidence teams should expect from a hybrid app development company: not only a green pipeline, but a clear device matrix showing which plugin-backed workflows were tested.

The Flutter upgrade passed CI because the pipeline proved that the project could build.

The plugins failed because the application had not yet proved that it could run.

Which Flutter plugin has caused your most difficult device-only failure?