DEV Community: Dextra Labs

How to Build a Finance AI Agent With Audit Trails, Controls, and Human Oversight

Dextra Labs — Thu, 22 Oct 2026 04:26:17 +0000

Here’s a truth about building AI agents for finance that nobody puts in the tutorial: the agent intelligence is maybe 30 percent of the work. The audit trail, the access controls, the human oversight checkpoints, the compliance documentation, that’s the other 70 percent. And that 70 percent is what determines whether your agent runs in production or gets killed in the compliance review before it processes a single transaction.

We learned this the expensive way on our first finance AI agent deployment. Built a beautiful underwriting agent. Elegant reasoning. Clean tool use. Impressive accuracy on test data. The compliance team looked at it for two hours and rejected it because they couldn’t answer one question: “If a regulator asks why the agent recommended denying this loan application, can you show them the exact data and reasoning that produced that recommendation?”

We couldn’t. The agent worked. It just didn’t prove it worked. And in finance, working without proving it is the same as not working.

This article is the guide we wish existed when we started. It covers the audit trail architecture, the access control design, the human oversight implementation, and the code patterns that make a finance AI agent not just functional but auditable, controllable, and compliant.

If you’re building agents that touch money, customer data, or regulatory workflows, this is the engineering that separates a demo from a deployment.

The audit trail isn’t logging. It’s the primary deliverable

Most developers think of audit trails as application logging. Write some events to a log file. Query them when something goes wrong. That’s not what finance compliance requires.

A finance audit trail must capture what data the agent received, what reasoning it applied, what tools it called, what each tool returned, what decision it reached, how confident it was, and what action it took, for every single transaction. And it must store this immutably for seven to ten years in a format that a non-technical regulator can review.

Here’s the data model:

from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum
from typing import Optional
import hashlib
import json


class EvaluationType(Enum):
    DETERMINISTIC = "deterministic"
    AI_ASSISTED = "ai_assisted"
    HUMAN_REVIEWED = "human_reviewed"


class ActionType(Enum):
    AUTO_APPROVED = "auto_approved"
    AUTO_DENIED = "auto_denied"
    ESCALATED = "escalated"
    PENDING_REVIEW = "pending_review"


@dataclass
class AuditRecord:
    """Immutable record of every agent decision."""
    record_id: str
    timestamp: datetime
    agent_id: str
    agent_version: str

    # What was evaluated
    transaction_id: str
    transaction_type: str
    input_data_hash: str          # SHA-256 of exact input
    input_data_snapshot: dict      # Frozen copy of input state

    # How it was evaluated
    evaluation_type: EvaluationType
    rules_applied: list[str]       # Rule IDs for deterministic checks
    model_version: Optional[str]   # For AI-assisted evaluations
    prompt_hash: Optional[str]     # For reproducibility

    # What was concluded
    decision: ActionType
    confidence: Optional[float]
    reasoning_chain: list[dict]    # Step-by-step reasoning

    # What happened next
    escalated_to: Optional[str]
    human_reviewer: Optional[str]
    human_decision: Optional[str]
    human_override_reason: Optional[str]
    human_decision_timestamp: Optional[datetime]

    # Integrity
    previous_record_hash: Optional[str]  # Chain integrity
    record_hash: str = field(init=False)

    def __post_init__(self):
        """Compute hash for tamper detection."""
        content = json.dumps({
            'record_id': self.record_id,
            'transaction_id': self.transaction_id,
            'input_data_hash': self.input_data_hash,
            'decision': self.decision.value,
            'timestamp': self.timestamp.isoformat(),
            'previous_hash': self.previous_record_hash
        }, sort_keys=True)
        self.record_hash = hashlib.sha256(content.encode()).hexdigest()

Two fields in this model deserve specific attention.

input_data_hash and input_data_snapshot serve different purposes. The hash proves that the data hasn’t been modified since the decision was made. The snapshot preserves the exact data state at decision time, because the source data in the ERP may change after the agent evaluated it, and the regulator needs to see what the agent saw, not what the database currently contains.

previous_record_hash creates a chain where tampering with any historical record breaks the hash chain on every subsequent record. This isn’t blockchain, it’s a simple integrity mechanism that makes after-the-fact modification detectable.

The storage architecture matters:

class AuditStore:
    """Append-only audit storage with hot and cold tiers."""

    def __init__(self, hot_store, cold_store, retention_years=7):
        self.hot = hot_store      # PostgreSQL - last 90 days, fast queries
        self.cold = cold_store    # S3 + Parquet - long-term, cost-efficient
        self.retention = retention_years

    async def append(self, record: AuditRecord) -> str:
        """Write to hot store. Never update. Never delete."""
        # Verify chain integrity
        last_record = await self.hot.get_latest(record.agent_id)
        if last_record and record.previous_record_hash != last_record.record_hash:
            raise IntegrityError("Chain broken - previous hash mismatch")

        await self.hot.insert(record)
        return record.record_hash

    async def archive(self, older_than_days=90):
        """Move aged records to cold storage. Hot store stays fast."""
        records = await self.hot.get_older_than(older_than_days)
        await self.cold.write_parquet(records)
        # Mark as archived in hot store - never delete
        await self.hot.mark_archived(records)

The append-only constraint is non-negotiable. No updates. No deletes. No “soft deletes.” If a record needs correction, you append a correction record that references the original. The original stays forever.

The access control layer: least privilege isn’t optional

The agent’s service account should have exactly the permissions it needs and nothing more. This sounds obvious. In practice, it’s the control that most teams get wrong because scoping permissions precisely requires understanding every data access and system interaction the agent performs.

class AgentPermissions:
    """Define and enforce what the agent can access and do."""

    def __init__(self, agent_role: str):
        self.role = agent_role
        self.permissions = self._load_permissions(agent_role)

    def _load_permissions(self, role: str) -> dict:
        """Each agent role has explicit read/write scopes."""
        roles = {
            "ap_processor": {
                "read": [
                    "invoices",
                    "purchase_orders", 
                    "goods_receipts",
                    "vendor_master",        # Read vendor info
                    "approval_matrix",      # Read routing rules
                ],
                "write": [
                    "invoices.status",       # Update processing status
                    "invoices.match_result", # Record match outcome
                    "approval_queue",        # Route for approval
                ],
                "blocked": [
                    "payments.execute",     # Cannot trigger payments
                    "vendor_master.modify", # Cannot change vendor data
                    "general_ledger",       # No GL access
                    "payroll",              # No payroll access
                    "treasury",             # No treasury access
                ],
                "amount_ceiling": 100_000,  # Escalate above this
            },
            "underwriting_analyst": {
                "read": [
                    "loan_applications",
                    "credit_bureau_data",
                    "financial_documents",
                    "compliance_rules",
                ],
                "write": [
                    "applications.risk_assessment",
                    "applications.recommendation",
                    "escalation_queue",
                ],
                "blocked": [
                    "applications.approve",  # Cannot approve loans
                    "applications.deny",     # Cannot deny loans
                    "disbursement",          # No access to funds
                ],
                "amount_ceiling": None,  # All amounts escalate
            }
        }
        return roles.get(role, {"read": [], "write": [], "blocked": []})

    def check_access(self, operation: str, resource: str) -> bool:
        """Enforce permission before every system interaction."""
        if resource in self.permissions.get("blocked", []):
            raise PermissionDenied(
                f"Agent role '{self.role}' is blocked from '{resource}'"
            )

        allowed_resources = self.permissions.get(operation, [])
        if resource not in allowed_resources:
            raise PermissionDenied(
                f"Agent role '{self.role}' lacks '{operation}' on '{resource}'"
            )
        return True

The blocked list is as important as the read and write lists. Explicitly defining what the agent cannot access, even when those resources exist in the same system, creates a documented boundary that compliance teams can review. An AP processing agent that has no path to the payroll system, the treasury system, or the general ledger is an agent whose blast radius is bounded by design.

The amount_ceiling field is the operational authority limit. Transactions above this threshold escalate to a human regardless of how confident the agent is. The ceiling is a business decision, not a technical one, set by the finance leadership based on the organisation’s risk tolerance.

Human oversight: the checkpoint architecture

Human-in-the-loop for finance agents isn’t “a human can review if they want.” It’s a set of defined checkpoints where the agent must stop, present its work, and receive human authorisation before proceeding.

class OversightCheckpoint:
    """Defines when and how the agent pauses for human review."""

    def __init__(self, rules: list[dict]):
        self.rules = rules

    async def evaluate(
        self, 
        transaction: dict,
        agent_assessment: dict,
        audit_trail: list[AuditRecord]
    ) -> CheckpointDecision:
        """Determine if human review is required."""

        for rule in self.rules:
            if self._matches(rule, transaction, agent_assessment):
                return CheckpointDecision(
                    requires_human=True,
                    reason=rule["reason"],
                    reviewer_role=rule["reviewer_role"],
                    context=self._assemble_context(
                        transaction, agent_assessment, audit_trail
                    ),
                    sla_hours=rule.get("sla_hours", 24)
                )

        return CheckpointDecision(requires_human=False)

    def _matches(self, rule, transaction, assessment) -> bool:
        """Check if this transaction triggers a checkpoint."""
        checks = {
            "amount_exceeds": lambda: (
                transaction.get("amount", 0) > rule.get("threshold", 0)
            ),
            "confidence_below": lambda: (
                assessment.get("confidence", 1.0) < rule.get("threshold", 0.85)
            ),
            "new_counterparty": lambda: (
                transaction.get("counterparty_history_days", 999) < 
                rule.get("minimum_days", 90)
            ),
            "exception_type": lambda: (
                assessment.get("exception_type") in 
                rule.get("escalation_types", [])
            ),
            "regulatory_flag": lambda: (
                assessment.get("regulatory_flags", []) != []
            ),
        }

        check_fn = checks.get(rule["condition"])
        return check_fn() if check_fn else False

    def _assemble_context(self, transaction, assessment, trail) -> dict:
        """Give the human everything they need to make a decision."""
        return {
            "transaction_summary": transaction,
            "agent_recommendation": assessment.get("recommendation"),
            "agent_confidence": assessment.get("confidence"),
            "agent_reasoning": assessment.get("reasoning_chain"),
            "relevant_history": self._get_relevant_history(transaction),
            "applicable_rules": assessment.get("rules_checked"),
            "escalation_reason": assessment.get("escalation_reason"),
            "suggested_action": assessment.get("suggested_action"),
            "audit_trail": [r.record_id for r in trail[-5:]],  # Last 5 steps
        }


# Checkpoint rules configuration
CHECKPOINT_RULES = [
    {
        "condition": "amount_exceeds",
        "threshold": 100_000,
        "reason": "Transaction exceeds autonomous authority limit",
        "reviewer_role": "senior_finance",
        "sla_hours": 4
    },
    {
        "condition": "confidence_below",
        "threshold": 0.85,
        "reason": "Agent confidence below threshold",
        "reviewer_role": "domain_specialist",
        "sla_hours": 8
    },
    {
        "condition": "new_counterparty",
        "minimum_days": 90,
        "reason": "Counterparty has less than 90 days of history",
        "reviewer_role": "risk_analyst",
        "sla_hours": 24
    },
    {
        "condition": "regulatory_flag",
        "reason": "Transaction flagged for regulatory review",
        "reviewer_role": "compliance_officer",
        "sla_hours": 4
    },
]

The _assemble_context method is where the oversight becomes genuinely useful rather than merely procedural. A human reviewer who receives “please review this transaction” makes a worse decision than one who receives the transaction data, the agent’s recommendation, the reasoning chain, the applicable rules, and the suggested action. The context assembly is what makes human oversight efficient, the human starts with understanding, not investigation.

The sla_hours field is the detail that operational teams care about. A checkpoint without a response deadline creates a queue that grows without bound. The SLA ensures that escalated transactions are reviewed within a defined window, and transactions that aren’t reviewed within the SLA trigger secondary escalation.

Putting it together: the compliant agent loop

Here’s how the components compose into a complete agent execution loop:

class FinanceAgent:
    """A finance agent that's auditable, controlled, and supervised."""

    def __init__(self, tools, permissions, checkpoints, audit_store, llm):
        self.tools = tools
        self.permissions = permissions
        self.checkpoints = checkpoints
        self.audit = audit_store
        self.llm = llm

    async def process(self, transaction: dict) -> ProcessingResult:
        trail = []

        # Step 1: Deterministic compliance checks first
        compliance_result = await self._run_compliance_checks(transaction)
        trail.append(await self._audit(
            transaction, "compliance_check", 
            EvaluationType.DETERMINISTIC, compliance_result
        ))

        if not compliance_result.passed:
            return ProcessingResult(
                status="compliance_failed",
                details=compliance_result.failures,
                audit_trail=trail
            )

        # Step 2: AI-assisted assessment where judgment is needed
        assessment = await self._run_ai_assessment(transaction)
        trail.append(await self._audit(
            transaction, "ai_assessment",
            EvaluationType.AI_ASSISTED, assessment
        ))

        # Step 3: Check if human oversight is required
        checkpoint = await self.checkpoints.evaluate(
            transaction, assessment, trail
        )

        if checkpoint.requires_human:
            trail.append(await self._audit(
                transaction, "escalated",
                EvaluationType.AI_ASSISTED, 
                {"reason": checkpoint.reason}
            ))
            return ProcessingResult(
                status="pending_human_review",
                context=checkpoint.context,
                reviewer=checkpoint.reviewer_role,
                sla=checkpoint.sla_hours,
                audit_trail=trail
            )

        # Step 4: Execute within authority
        self.permissions.check_access("write", f"{transaction['type']}.status")

        if transaction.get("amount", 0) > self.permissions.permissions.get(
            "amount_ceiling", float("inf")
        ):
            trail.append(await self._audit(
                transaction, "amount_escalation",
                EvaluationType.DETERMINISTIC,
                {"ceiling": self.permissions.permissions["amount_ceiling"]}
            ))
            return ProcessingResult(
                status="escalated_amount_ceiling",
                audit_trail=trail
            )

        # Step 5: Execute and document
        action_result = await self._execute_action(transaction, assessment)
        trail.append(await self._audit(
            transaction, "action_executed",
            EvaluationType.DETERMINISTIC, action_result
        ))

        return ProcessingResult(
            status="completed",
            action=action_result,
            audit_trail=trail
        )

    async def _audit(self, transaction, step, eval_type, data) -> AuditRecord:
        """Every step gets an immutable audit record."""
        record = AuditRecord(
            record_id=generate_uuid(),
            timestamp=datetime.utcnow(),
            agent_id=self.agent_id,
            agent_version=self.version,
            transaction_id=transaction["id"],
            transaction_type=transaction["type"],
            input_data_hash=hash_data(transaction),
            input_data_snapshot=freeze(transaction),
            evaluation_type=eval_type,
            rules_applied=data.get("rules", []),
            model_version=self.llm.model_version if eval_type == 
                EvaluationType.AI_ASSISTED else None,
            prompt_hash=data.get("prompt_hash"),
            decision=data.get("decision", ActionType.PENDING_REVIEW),
            confidence=data.get("confidence"),
            reasoning_chain=data.get("reasoning", []),
            escalated_to=data.get("escalated_to"),
            human_reviewer=None,
            human_decision=None,
            human_override_reason=None,
            human_decision_timestamp=None,
            previous_record_hash=await self.audit.get_latest_hash(
                transaction["id"]
            )
        )
        await self.audit.append(record)
        return record

The execution order matters. Compliance checks run first, deterministic, binary, fast. If the transaction fails a regulatory requirement, there’s no point running the AI assessment. The AI assessment runs second, only for transactions that need interpretive evaluation. The checkpoint evaluation runs third, comparing the transaction characteristics and agent confidence against the oversight rules. The action execution runs last, only for transactions that passed compliance, received a confident assessment, and didn’t trigger a checkpoint.

Every step produces an audit record. The complete chain for a single transaction typically contains four to six records, each linked to the previous by hash. A regulator reviewing a decision can trace the complete chain: what data came in, what compliance checks passed, what the AI assessed and why, whether a human reviewed it, and what action was taken.

The testing that makes compliance teams comfortable

The code above gets you an architecture. The testing below gets you the approval to deploy it.

# Golden test set: known-outcome transactions for regression testing
class ComplianceRegressionSuite:
    """Run against every agent update, every prompt change, every model version."""

    def __init__(self, golden_set_path: str, drift_threshold: float = 0.02):
        self.golden_set = load_golden_set(golden_set_path)
        self.threshold = drift_threshold  # Max 2% decision drift

    async def run(self, agent: FinanceAgent) -> RegressionResult:
        results = []
        for case in self.golden_set:
            actual = await agent.process(case.transaction)
            results.append({
                "case_id": case.id,
                "expected_decision": case.expected_decision,
                "actual_decision": actual.status,
                "match": actual.status == case.expected_decision,
                "confidence": actual.audit_trail[-1].confidence
            })

        match_rate = sum(r["match"] for r in results) / len(results)
        drift = 1.0 - match_rate

        return RegressionResult(
            passed=drift <= self.threshold,
            match_rate=match_rate,
            drift=drift,
            failures=[r for r in results if not r["match"]],
            recommendation="BLOCK" if drift > self.threshold else "APPROVE"
        )

The golden test set contains 300 to 500 transactions with known correct outcomes, labelled by experienced finance professionals, not by the development team. Every agent update, every prompt change, and every model version upgrade runs against this set. If the decision outcomes drift by more than 2 percent from the baseline, the update is blocked.

This regression suite is what we point to when compliance asks “how do you know the agent still works correctly after a change?” The answer is: we run it against 500 transactions where we know the right answer, and we block any change that moves the outcomes by more than 2 percent.

For a deeper dive into the full development lifecycle, from workflow assessment through architecture selection, integration patterns, and deployment, our guide on how to build finance AI agents covers the engineering decisions we make before writing any agent code.

What most teams get wrong

Three patterns consistently cause compliance rejections or production failures.

Building the audit trail as an afterthought. If the audit architecture isn’t designed before the agent logic, the audit records won’t capture the right data at the right granularity. Retrofitting audit trails onto a working agent means re-engineering the decision flow to emit records at each step, which is more expensive than building it in from the start.

Testing with synthetic data only. Compliance teams want to see the agent’s performance on real transactions, transactions that humans have already processed with known correct outcomes. Synthetic test data proves the code works. Real test data proves the agent works on your actual business workflows.

Treating the human oversight as a failure mode rather than a design feature. The checkpoint architecture isn’t the system failing and asking for help. It’s the system correctly identifying which transactions need human judgment and routing them efficiently. The 20 to 30 percent of transactions that escalate aren’t a limitation, they’re the system working as designed.

The bottom line for finance developers

A finance AI agent that works but can’t prove it works is a demo, not a deployment. The audit trail, the access controls, and the human oversight checkpoints are what transform working code into a production system that compliance teams approve, regulators can examine, and operations teams trust.

Build the audit architecture first. Design the permissions before the agent logic. Define the oversight checkpoints with the compliance team, not for them. Test against real transactions with known outcomes. And treat every deployment as the beginning of a continuous proving process, not the end of a build project.

For teams ready to build finance agents that survive audit, the bespoke AI agent development practice at Dextra Labs designs and deploys agents with the compliance architecture described in this article as a foundational requirement, not an add-on. The audit trail, the access controls, and the oversight checkpoints are built into the agent from day one, because in finance, working without proving it is the same as not working.

Published by Dextra Labs - AI Consulting and Enterprise Agent Development

How to Manage Agent State and Memory in Production AI Systems

Dextra Labs — Wed, 14 Oct 2026 11:48:29 +0000

An AI agent processes a multi-step support request. It calls two APIs successfully, retrieves the customer's billing history, and begins preparing a resolution. Then the worker crashes before completing the third action. When the service restarts, the agent has lost its intermediate results and either starts the entire workflow over or produces an inconsistent response that contradicts what it already told the customer.

This is not a model problem. It is an infrastructure problem, and it shows up in three forms:

Conversation history and task context disappear between sessions.
Tool execution can fail or be interrupted halfway through a multi-step workflow.
Unbounded context accumulates over time, increasing latency, token consumption, and retrieval noise.

A production agent needs explicit mechanisms for preserving execution state and retrieving relevant memory. Without them, reliability degrades with every additional step in the workflow.

Agent State vs. Agent Memory: What is the Difference?

State represents the agent's current execution context: where it is in a workflow and what it has done so far. Memory preserves information that may be useful beyond the current step or session, such as user preferences, validated facts, or outcomes from previous runs. The boundaries between these concepts can shift depending on your framework and application design, but the distinction matters because each requires different storage, retrieval, and lifecycle management.

Conversation history is only one kind of memory. A production agent may also need structured facts, execution checkpoints, and retrieved knowledge from external sources. Each of these has different storage requirements and retention rules. This complexity increases significantly in multi-agent systems where multiple agents share context, coordinate tasks, and need consistent views of the same underlying state.

Design a Production AI Agent Memory Architecture

A practical architecture separates memory into three layers, each with distinct storage, retention, and access patterns. Understanding your LLM context window architecture is essential here, because what you load into the model's context at each step determines both cost and quality.

Working Memory for the Current Task

Holds the active goal, recent messages, intermediate outputs, and tool execution status. Keep this layer small enough for efficient orchestration and context construction. Working memory is reconstructible if lost, because its sources are the other two layers.

Persistent State for Checkpointing and Recovery

Stores versioned execution snapshots or equivalent durable records so a workflow can resume after interruptions, worker restarts, or human approval steps. This is the layer that prevents the crash-and-restart failure described in the opening.

Long-Term Memory for Cross-Session Retrieval

Stores selected user preferences, validated facts, and reusable task knowledge in structured records or searchable indexes. This memory persists across sessions and grows over the application's lifetime.

The data flow through these layers follows a consistent pattern:

Input → Load state → Retrieve memory → Execute tools → Persist updates → Return result

This sequence is conceptual. In practice, retrieval and persistence may occur at multiple points during execution, and the boundaries between layers depend on the complexity of the workflow.

Implement Persistent Agent State with Checkpointing

Checkpointing means saving recoverable execution state at defined boundaries so the agent can resume rather than restart. At minimum, a checkpoint should capture:

A stable run or thread identifier that ties the checkpoint to its workflow.
The current workflow step and task status.
Relevant intermediate results from completed actions.
Pending actions, retry counts, and timestamps.
A schema or workflow version needed for compatible restoration.

Here is a framework-neutral pattern in Python:

state = load_state(run_id)
result = execute_next_step(state)
state.update(result)
save_checkpoint(run_id, state)

Real implementations need transactional writes to avoid partial updates, serialization rules that handle complex objects, error handling for storage failures, and a durable backend such as PostgreSQL or a managed state store.

Saving a checkpoint does not automatically make external side effects safe to retry. If the agent issued a refund through a payment API before the crash, replaying from the checkpoint could issue a duplicate refund. Use idempotency keys, operation records, or reconciliation logic for any action that modifies external state. The checkpoint tells the agent where it was. The idempotency mechanism tells it whether the action already succeeded.

How to Manage Long-Term AI Agent Memory Without Context Bloat

An agent that injects its entire history into every model call will eventually hit context limits, increase latency, inflate token costs, and dilute the relevance of the information the model actually needs. Four techniques keep memory useful without letting it grow unbounded:

Summarization: Compress older conversation history while preserving key decisions, constraints, and unresolved tasks. Accept that summarization is lossy and keep the originals accessible for audit.
Selective memory writes: Save durable, useful facts rather than every message or tool output. A verified product preference is worth persisting; a routine acknowledgment message is not.
Retrieval on demand: Fetch relevant records based on the current task context instead of loading all stored memories into every prompt. Retrieval quality matters more than retrieval volume.
Expiration and revision: Remove outdated facts and update records when new evidence changes their validity. Memory without a lifecycle becomes a liability.

Consider a practical example. A customer support agent should retain a verified product preference and an unresolved case identifier across sessions, but it does not need to inject an entire six-month conversation history into every new interaction.

One important distinction: semantic retrieval (vector search over stored knowledge) and state restoration (loading a checkpointed workflow) solve different problems. Vector search can surface relevant context, but it should never be the sole mechanism for recovering the exact status of an in-progress task.

Memory quality depends on relevance, freshness, and retrieval accuracy, not simply the total amount stored.

How to Choose the Right Storage Backend for Agent State and Memory?

Storage selection should follow access patterns and reliability requirements rather than defaulting to a single database for every purpose.

These options coexist comfortably in production systems. Structured execution state might live in PostgreSQL, large documents and traces in object storage, and embeddings in a vector index for semantic retrieval. The combination gives each data type the storage characteristics it actually needs.

Avoid prescribing a single database for every agent. Durability, consistency, query shape, latency tolerance, and recovery requirements should drive the choice for each layer of your architecture.

How to Handle Memory Security, Multi-Tenant Isolation, and Data Retention

Prototype agents can get away with loose memory boundaries. Production systems cannot. Whether you are building internally or working with an AI agent company to deploy agents at scale, these engineering controls are non-negotiable:

Tenant isolation: Scope every state and memory lookup to the authenticated user, tenant, or permitted workspace. A missing tenant filter in a retrieval query is a data breach, not a bug.
Access control: Apply authorization checks before retrieving or modifying stored information. Memory retrieval should respect the same permission model as the rest of the application.
Data minimization: Avoid persisting credentials, unnecessary personal data, or sensitive tool outputs. Store what is needed for task execution and discard the rest.
Memory trust boundaries: Treat retrieved text as data, not as trusted instructions. A stored memory that says "skip authorization for this user" should never override system policy. Validate tool actions independently of what memory suggests.
Retention and deletion: Define expiration rules, auditability requirements, and deletion procedures that apply to checkpoints, summaries, and derived indexes. Users who request data deletion should not have their memories persist in a vector index.

A memory retrieval bug can expose another tenant's data even when the underlying model behaves correctly. Test these boundaries as rigorously as you test your authentication system.

Now Let’s Monitor and Test Agent State in Production

Persistence and memory only prove their value under real operational conditions. Monitor these areas continuously:

Checkpoint write and restore failures that could leave agents in an inconsistent state.
State size, storage growth, and retrieval latency over time.
Token consumption attributable to memory injection into model calls.
Stale-memory retrieval and cases where the agent acted on outdated information.
Recovery success rates after simulated process failures.

One useful failure-injection test: interrupt an agent after a tool call completes but before the checkpoint is saved, restart the worker, and verify that the agent resumes from the correct state without repeating the external action. If it retries the payment or the API call, your idempotency layer has a gap.

Also test concurrent updates to the same workflow, expired memory records that should no longer appear in retrieval, schema migrations across checkpoint versions, and tenant-boundary enforcement under realistic load.

Track both infrastructure reliability and memory quality. A checkpoint can restore perfectly while the retrieved information it surfaces is still wrong, outdated, or irrelevant to the current task.

Production AI Agent State Management: Final Checklist

Separate execution state from long-term memory with distinct storage and lifecycle rules.
Persist critical workflow progress to durable storage at defined boundaries.
Make external actions safe to retry using idempotency keys or operation records.
Bound context size and define explicit memory retrieval policies.
Enforce tenant isolation, access control, and data retention at every layer.
Test restoration, concurrent updates, and failure recovery under realistic conditions.
Monitor memory relevance, retrieval latency, and storage growth in production.

Reliable agents require deliberate state and memory design, just like any other distributed production system. The model provides the intelligence. The infrastructure around it determines whether that intelligence holds up when things go wrong.

How to Design LLM Memory Architecture for Long-Running AI Agents

Dextra Labs — Tue, 13 Oct 2026 18:30:46 +0000

Imagine a coding agent that begins migrating a service on Monday. It identifies an incompatible dependency, documents an alternative approach, and parks the task overnight. When the agent resumes on Tuesday, it forgets the decision entirely, repeats the investigation, and attempts the original migration path again.

The problem is not the model's reasoning. The system has failed to preserve and restore the information required to continue the task. Designing a proper LLM memory architecture solves this class of failure by treating agent memory as a structured engineering concern rather than an afterthought.

Context windows have finite capacity, and conversation context disappears when a session ends. Replaying full transcripts inflates token consumption and injects irrelevant noise. Summaries reduce size but can omit important details or preserve outdated assumptions. Conversation history records what happened, but it does not reliably establish the current state of a task.

Memory should preserve useful continuity rather than every token an agent has ever encountered. The design question is: what should the agent remember, where should it store that information, and how should it decide what to retrieve?

Separate Context, Memory, and Execution State

These three concepts are related but should not be treated as interchangeable. Conflating them leads to systems that either lose critical task progress or bloat the context window with irrelevant history.

In practice, this separation means:

Build the model's context dynamically from current instructions, authoritative task state, and relevant memories rather than replaying an entire transcript.
Persist execution state separately so a task can resume without relying on a generated conversation summary.
Record tool outcomes explicitly instead of inferring success from the agent's narrative about what it did.
Use durable checkpoints when an operation spans multiple steps or interacts with external systems.

A summary might say that a deployment was attempted. The execution record must establish whether it succeeded, failed, or remains unconfirmed. That distinction matters when the agent restarts.

Choose Memory Types Based on Access Patterns

Memory is best understood as a set of logical roles, not a requirement to deploy a separate database for each category. The intended use of each memory type should drive the implementation decisions.

Working memory is optimized for immediate access and can often be reconstructed if lost. It holds the scratchpad that the agent reasons against during a single execution cycle.

Episodic memory preserves historical evidence and helps agents learn from previous runs. When the coding agent encountered a missing staging secret last week, that episode becomes retrievable context for similar failures.

Semantic memory supports factual queries such as which service owns a particular endpoint or which team manages a given repository. These facts change infrequently and benefit from structured, queryable storage.

Procedural memory describes how to perform a task rather than merely recording that it happened. A validated deployment checklist or a tested rollback procedure belongs here.

A single PostgreSQL-backed system could store structured facts, task events, and workflow definitions, while a vector index supports semantic retrieval over selected records. The vector database is a retrieval mechanism, not automatically the authoritative source of every fact.

Query shape, consistency requirements, update frequency, durability, access controls, and retention needs should determine the storage design for each memory type.

Design the Memory Write Pipeline

Not every model-generated statement deserves to become persistent knowledge. A write pipeline transforms raw interactions into dependable

memories through five stages:

Capture: Record relevant messages, tool outputs, decisions, state transitions, and external events as they occur.
Extract: Identify candidate facts, task outcomes, user preferences, and potentially reusable procedures from the raw stream.
Validate: Check provenance, confidence, permissions, relevance, and whether the information merits long-term retention.
Deduplicate and reconcile: Identify equivalent records and resolve conflicting facts using explicit precedence rules.
Persist and index: Store validated records and update the retrieval index or cache.

Here is an illustrative schema for a stored memory record:

{
  "memory_id": "mem_1042",
  "type": "episodic",
  "entity_id": "project_42",
  "content": "Migration failed because the staging secret was missing.",
  "source_event_id": "event_884",
  "created_at": "2026-10-12T10:30:00Z",
  "confidence": 0.92
}

The source_event_id links the memory to the evidence it was derived from, so it can be audited or corrected later. The confidence value represents the extraction system's assessment, not proof that the claim is true. A production design would also require tenant scope, validity periods, versioning, retention rules, and correction status.

Key engineering decisions: use asynchronous extraction where latency matters, persist critical state transitions synchronously, make writes idempotent to avoid duplicates after retries, and always preserve source references so extracted memories remain traceable.

Build a Retrieval Pipeline That Fits the Context Budget

The retrieval pipeline determines how stored information reaches the model at inference time. In any serious AI agent orchestration framework, this pipeline is where memory becomes useful rather than just persistent.

New request or scheduled task
              ↓ 
Resolve identity and active task
              ↓ 
Fetch authoritative execution state
              ↓ 
Retrieve candidate memories
              ↓ 
Filter, rank, and deduplicate
            ↓ 
Apply freshness and access policies
             ↓ 
Fit results to the context budget
             ↓ 
LLM reasoning and tool execution

Each stage involves deliberate design choices:

Metadata filtering: Restrict candidates by tenant, user, project, task, timestamp, and memory type before ranking. This prevents cross-contamination and reduces the candidate set to something manageable.
Hybrid search: Combine lexical search for exact identifiers, error messages, or version numbers with semantic search for conceptually related information. Neither approach alone covers every retrieval need.
Reranking: Rank candidates by relevance, freshness, source authority, confidence, and task-specific importance. Similarity scores alone do not establish factual correctness.
Context budgeting: Reserve space for system instructions, the current request, execution state, and expected tool results before filling the remaining window with historical context.
Critical distinction: If a task depends on current deployment status or an account balance, query the authoritative system of record. Historical memory may explain what happened previously, but it should never override current state from the source of truth.

Handle Memory Updates, Conflicts, and Forgetting

Memory that persists for weeks or months introduces problems that short-lived conversation history never encounters.

Stale facts are inevitable when the world changes faster than memory updates. Attach timestamps and provenance to every record. Apply expiration or revalidation rules to information that changes frequently, such as environment configurations or team assignments.

Conflicting facts arise when multiple sources or sessions produce contradictory records. Define precedence rules based on source authority, version, and recency. Do not automatically assume that the newest model-generated statement is the correct one.

Memory bloat accumulates as completed episodes and low-value details pile up. Summarize completed task sequences, archive granular details, and retain links to the original events so the full history remains accessible if needed.

Corrections and deletion must propagate through primary storage, caches, and search indexes. Define how deleted or superseded records are excluded from retrieval so the agent does not resurface information that has been explicitly retracted.

Summarization is inherently lossy. Preserve original evidence whenever future debugging, auditing, or verification may require exact details.

Memory is a maintained information system, not an append-only collection of facts that can be trusted indefinitely.

Make Memory Safe and Recoverable in Production

Persistent memory improves agent continuity only when combined with reliable execution safeguards and access controls. Without these, memory becomes a vector for stale-state errors, unauthorized data access, and repeated side effects.

A production checklist should include:

Enforce access controls: Apply user-, tenant-, and project-level permissions before returning any memory to the model. Cross-tenant retrieval is a security incident, not a bug.
Treat retrieved content as untrusted: Historical messages and stored instructions must not override system policy or authorization checks.
Protect against repeated side effects: Use durable checkpoints, idempotency keys, and explicit operation states for payments, deployments, and other consequential actions.
Maintain auditability: Log memory writes, retrieval decisions, corrections, and source references.
Test recovery scenarios: Simulate worker restarts, duplicate events, stale facts, conflicting records, and cross-tenant retrieval attempts.

Consider a concrete failure: after a worker restart, the agent should inspect the persisted task state and the external tool's recorded outcome before retrying a deployment. If the outcome is unknown, it should reconcile the operation rather than blindly repeating it. An agent that reruns a payment because it lost track of the previous attempt is not a memory problem. It is a production incident.

Evaluate Memory Quality, Not Just Retrieval Speed

Build an evaluation suite around realistic multi-session workflows rather than isolated retrieval benchmarks. The metrics that matter measure whether memory actually improves task outcomes:

Recall and relevance: Was the necessary memory retrieved, and was it useful in context?
Freshness and correctness: Did the agent use the latest valid information rather than an outdated or superseded record?
Task continuity: Could the agent resume an interrupted workflow without repeating completed operations?
Efficiency: Tokens retrieved, retrieval latency, storage cost, and unnecessary writes that could have been avoided.
Safety: Unauthorized retrieval attempts, actions taken on stale state, and tenant-isolation failures.

Test with seeded facts, contradictory updates, long gaps between sessions, and interrupted workflows. Compare task outcomes against a baseline without persistent memory.

Optimize for correct task completion, not merely fast similarity search.

Store What Matters, Retrieve What Is Needed

Reliable LLM memory architecture separates current context, durable knowledge, past experiences, reusable procedures, and active execution state into distinct responsibilities with appropriate storage and retrieval strategies.

Start with the smallest design that satisfies the workload: durable task records, structured memory with explicit provenance, and targeted retrieval. Add vector search, specialized storage tiers, or more complex reconciliation only when evaluation demonstrates a measurable benefit. For teams building production-grade agent systems, working with experienced AI agent development services can help get the architecture right before memory debt accumulates.

The goal is not to make an agent remember everything. It is to help it resume the right work with the right information at the right time.

5 Coordination Failures That Can Break Multi-Agent AI Systems

Dextra Labs — Mon, 12 Oct 2026 08:29:13 +0000

A single agent fails in ways you can see. A multi-agent system fails in the spaces between the agents , and those failures are quiet, weird, and expensive.

Here’s the seductive pitch for multi-agent systems: instead of one giant agent trying to do everything, you build a team of specialists. One agent pulls the data, one assesses risk, one checks compliance, one makes the call. Each is simpler, easier to reason about, easier to test. Divide and conquer.

It’s a genuinely good idea, and it works , right up until the agents have to actually coordinate. Because the moment you have more than one agent, you’ve introduced a whole category of failure that doesn’t exist in a single-agent system: the failures that live between the agents. Not “this agent is wrong,” but “these two agents were each right and the system still did something insane.”

I want to walk through five of these coordination failures, because they’re the ones that quietly kill multi-agent projects in production, and they’re almost never what the architecture diagram prepares you for. I’ll use one running example to make them concrete: an SMB lending system , an multi-agent AI system that evaluates small-business loan applications, because money, compliance, and real consequences make the failure modes impossible to hand-wave.

Picture the setup. Five agents:

Intake , reads the application, extracts fields.
Data , pulls credit bureau, bank transactions, business records.
Risk , scores creditworthiness from the data.
Compliance , checks fair-lending rules, sanctions, policy limits.
Decision , an orchestrator that weighs it all and approves, declines, or escalates.

On the happy path, it’s beautiful. Now let’s break it, five ways.

Failure 1: The Telephone Game (Context Degradation)

In a single agent, all the context lives in one place. In a multi-agent system, context gets passed , and every handoff is a chance to lose or distort something.

In our lending system, Intake extracts “requested amount: $50,000” and passes a summary to Risk. But somewhere in the summarization , because agents often pass natural-language summaries, not structured data , “$50,000 requested, $200,000 annual revenue” becomes “strong revenue relative to request.” Risk scores on that framing. Compliance never sees the raw numbers, only Risk’s conclusion. By the time Decision acts, the actual figures have been three times abstracted into a vibe.

Each agent did its job correctly on the input it received. The failure is that the input degraded at each hop, like the children’s game of telephone. The output is confident and wrong, and worse, no single agent’s logs show the error , because no single agent made one.

The fix is discipline about what travels between agents. Pass structured, verifiable data (the actual numbers, with provenance), not prose summaries, for anything decision-critical. Let agents summarize for readability, but make the orchestrator able to see ground truth. The rule: a downstream agent should never have to trust an upstream agent’s interpretation when it could have the fact.

Failure 2: The Overlapping Mandate (Who Owns This Decision?)

This is the subtle one, and it’s a design failure disguised as a runtime bug.

Both Risk and Compliance, reasonably, have opinions about whether to decline a loan. Risk declines for creditworthiness; Compliance declines for policy. Fine , until a borderline case comes in where Risk says “approve, decent score” and Compliance says “decline, exceeds exposure limit for this sector.”

Who wins? If you never explicitly decided, the answer depends on accidental details , which agent the orchestrator happens to query last, how their outputs get merged, whether a later agent’s text overrides an earlier one. The system produces an answer, but not a principled one, and the same application could get a different result depending on ordering. In lending, that’s not just a bug , it’s a fair-lending problem, because inconsistent decisions on similar applicants is exactly what regulators look for.

The failure isn’t in either agent. It’s that two agents had overlapping authority over the same decision and nobody defined the precedence. This is the multi-agent version of an org chart where two people both think they own a call , and it fails the same way: in the ambiguity.

The fix is explicit decision rights. For every decision the system makes, exactly one agent (or one rule) must own the final say, and the precedence between agents must be defined, not emergent. Compliance overrides Risk, always, by design , written down, not left to query order. If you can’t say which agent wins a conflict, you don’t have an architecture yet; you have a race condition with opinions. (The same clarity you’d want when choosing types of AI agents for each role , a deliberative compliance agent and a fast reactive intake agent aren’t interchangeable, and their authority shouldn’t be either.)

Failure 3: The Cascade (One Wrong Agent Poisons the Chain)

In a pipeline, agents trust their inputs. That trust is the vulnerability.

Suppose the Data agent has a bad day , the bank-transaction API times out, and instead of failing loudly, it returns partial data (three months instead of twelve) without flagging it. Risk scores on three months, sees no red flags (because the problem months aren’t in the window), and passes a clean score downstream. Compliance sees a clean score. Decision approves.

One silent degradation at the start cascaded through every downstream agent, each of which behaved perfectly on the corrupted input it was handed. The error didn’t get caught because each agent trusted the one before it, and the bad data looked plausible all the way down.

This is the single most dangerous multi-agent failure, because it’s invisible. Nothing errored. Everything “worked.” The loan just shouldn’t have been approved.

The fix is that agents must distrust their inputs, and failures must be loud. Every agent validates what it receives against expectations (is this the full 12 months? does the data look complete?) and, critically, every agent must fail loudly , an upstream timeout should produce an explicit “incomplete data, cannot proceed,” never a silent partial result. Confidence and completeness must travel with the data, so a downstream agent knows it’s reasoning on 25% of the picture. In a chain, the most important thing an agent can say is “I’m not sure.”

Failure 4: The Infinite Handshake (Deadlocks and Loops)

When agents can call each other, they can also get stuck calling each other.

Our Decision orchestrator, facing a borderline case, asks Risk to re-evaluate with more weight on cash flow. Risk says “I need Compliance to confirm the sector classification first.” Compliance says “I need the Data agent to re-pull with the corrected business type.” Data re-pulls, Risk re-scores, Decision is still unsure, and asks Risk to re-evaluate again. Round and round. The application never resolves, tokens burn, and a human only notices when the latency alarm goes off or the bill does.

Or the subtler version: two agents each waiting for the other to go first, a classic deadlock, and the whole case just hangs.

Multi-agent systems, especially ones where the orchestrator can loop, are prone to exactly the pathologies distributed systems engineers have fought for decades , livelocks, deadlocks, infinite retries. The agents are each being “reasonable.” The system as a whole is stuck.

The fix is hard limits and a circuit breaker. Every workflow needs a maximum number of agent-to-agent round-trips, a maximum total time, and a maximum cost, after which it force-escalates to a human with whatever it has. No loop runs forever. An agent that’s been asked the same question twice should escalate, not re-answer. Treat “the agents couldn’t converge” as a first-class outcome that routes to a person , because an un-resolvable case resolved by a human in five minutes is infinitely better than one the system churns on for an hour and then gets wrong.

Failure 5: The Shared-State Collision (Two Agents, One Truth, No Lock)

The last one is pure distributed-systems gravity, and it bites the moment your agents share any mutable state.

Say two applications from the same business come in close together (a renewal and a new line), and both workflows touch a shared “total exposure to this borrower” record. Agent A reads exposure ($0 committed), decides there’s room, approves $50k. Agent B, running concurrently, also reads exposure ($0, because A hasn’t written yet), decides there’s room, approves another $50k. Now you’ve committed $100k against a $60k limit, and each agent was individually correct , they just both read stale state before either wrote.

This is the oldest bug in distributed computing , the race condition , and multi-agent AI systems reintroduce it enthusiastically, because agents act concurrently and the “state” they share (an exposure ledger, an inventory count, a rate limit) is exactly the kind of thing two agents will read-then-write without coordination.

The fix is the same as it’s always been: treat shared state as shared state. Anything multiple agents can read and modify needs proper concurrency control , locking, transactions, atomic updates , not hope. If two agents can act on the same record at once, assume they will, at the worst possible moment. The LLM part of your system is new; this part is a solved problem, and the solution is to stop pretending agents are the exception to fifty years of concurrency engineering.

The Thread Running Through All Five

Step back and notice what these have in common. Not one of them is “the AI wasn’t smart enough.” Every single failure is a coordination failure , context lost in a handoff, authority left ambiguous, bad data trusted, loops left unbounded, shared state left unlocked. The intelligence of the individual agents is almost irrelevant to whether the system works.

That’s the uncomfortable lesson of building real multi-agent systems: the agents are the easy part. Making several of them cooperate reliably, under real load, with real consequences, is a distributed systems problem wearing an AI costume. The field spent decades learning how to make independent processes coordinate without corrupting shared state or deadlocking each other. Multi-agent AI doesn’t get to skip that curriculum just because the processes now happen to reason in natural language.

So if you’re building one , or evaluating one someone else built , don’t start by asking how clever the agents are. Start by asking the five coordination questions: How does context travel, and does it degrade? Who owns each decision when agents disagree? What happens when one agent gets bad data? What stops the loop? And what locks the shared state?

Answer those well and you have a system that survives production. Skip them and you have a very impressive demo that will do something inexplicable the first busy Tuesday it meets the real world. The difference between those two outcomes is the entire job, and it’s the thing the serious AI agent builders spend most of their time on , not the agents, the coordination between them.

Multi-agent systems are powerful. They’re also distributed systems, with all the hard-won discipline that implies. Respect that, and the “team of specialists” pitch actually delivers. Ignore it, and the team of specialists turns into a committee that can’t agree, loses the paperwork, and occasionally approves a loan it shouldn’t.

Which of these has bitten you? For me the sneakiest is Failure 3 , the silent cascade , because nothing errors, everything “works,” and you only find out when the outcome is already wrong. What’s your worst coordination war story?

How to Build an Internal LLM Benchmark for Enterprise Model Selection

Dextra Labs — Fri, 09 Oct 2026 08:56:36 +0000

The public leaderboard says Model X is #1. Your production traffic disagrees. Here’s how to build the benchmark that actually predicts which model works for you.

Here’s a pattern we see constantly. A team needs to pick an LLM for a real product. They open a public leaderboard, see a model sitting proudly at the top, and reason, quite sensibly, “well, that’s the best one.” They wire it in. And then in production it’s mediocre at the one thing they actually needed it to do, while a model ranked eighth quietly would have crushed it.

The leaderboard wasn’t lying. It just wasn’t measuring your problem.

This isn’t a knock on public benchmarks. They’re genuinely useful for what they are: a broad, standardized signal about general capability. But “generally capable” and “good at extracting line items from your suppliers’ weirdly-formatted invoices” are different questions, and only one of them pays your bills. If you’re selecting a model for an enterprise workload, the benchmark that matters is the one you build yourself.

Let me walk through how to actually do that, what to measure, how to build the harness, and the traps that quietly ruin the result.

Why the Public Leaderboard Can’t Answer Your Question

Start with the uncomfortable truth about why a general LLM benchmark leaderboard can’t make your decision for you.

It measures the average task, not your task. A leaderboard score is an aggregate across a broad set of general challenges, reasoning, coding, trivia, writing. Your actual workload is narrow and specific: classify support tickets, draft compliance summaries, pull structured data from PDFs. A model can be excellent on average and weak on your narrow slice, and the aggregate score will never tell you which.

Your data isn’t in the test set, and might be the opposite of it. Public benchmarks run on public data, clean, well-formed, English-heavy, general-domain. If your inputs are messy internal documents, domain jargon, mixed languages, or formats no benchmark ever imagined, the leaderboard is testing a world that doesn’t resemble yours.

Contamination is real. Popular benchmark questions leak into training data. A model can score well partly because it has effectively seen the answers. Your internal tasks have no such leakage, which is exactly why they’re a truer test.

It ignores everything that isn’t accuracy. Latency. Cost per thousand calls at your volume. Whether it fails safely or confidently hallucinates. Whether it can be self-hosted for your compliance requirements. The leaderboard crowns a winner on quality alone; you’re optimizing a system with four or five constraints at once.

Preference ≠ fitness. Arena-style leaderboards rank models on which output humans prefer in blind comparison. Useful signal, but “which answer sounds nicer” is not “which model correctly applies your refund policy.” Likeability and correctness diverge more often than people expect.

None of this makes public leaderboards useless. It makes them a starting shortlist, not a decision. You use them to narrow fifty models to five. Then you build your own benchmark to pick the one.

What “Build Your Own Benchmark” Actually Means

It sounds heavier than it is. People hear “benchmark” and picture a research project. In practice, an internal LLM benchmark is four things:

A representative dataset of your real tasks, with known-good answers.
A scoring method that reflects what “correct” means for your use case.
A harness that runs every candidate model against that dataset, identically.
A scorecard that weighs quality against cost, latency, and your other constraints.

That’s it. You don’t need a PhD or a GPU cluster. You need a few hundred real examples, a clear definition of success, and a weekend of engineering. Let’s take each part.

Step 1: Build the Dataset From Your Actual Traffic

This is the part that determines whether the whole exercise is worth anything, so don’t shortcut it.

Pull real examples from your actual workload, support tickets you’ve already handled, invoices you’ve already processed, queries your users have actually typed. Not synthetic examples you invented, and not the clean happy-path cases. You want the real distribution, including the messy 20% that breaks things, because the messy 20% is where models actually differentiate.

How many? Fewer than you’d think. 100 to 300 well-chosen examples usually gives you a clear signal. The goal isn’t statistical perfection, it’s enough coverage of your real cases that the ranking is stable and the failures are visible.

The hard part is the known-good answers, the labels. For each example, you need the correct output a human expert would produce. This is tedious and it’s non-negotiable: a benchmark with no ground truth can only measure “which output looks plausible,” which is the exact trap we’re trying to escape. Have your domain experts label the set once. That labeled set becomes a durable asset, you’ll reuse it every time a new model drops.

Deliberately over-weight the edge cases. The ambiguous ticket, the invoice with the missing PO, the query in two languages. On the happy path most modern models are fine and indistinguishable. The edges are where the real ranking lives.

Step 2: Define What “Correct” Means For You

“Accuracy” is not one thing, and picking the wrong definition quietly wrecks the benchmark.

For a classification task (routing a ticket to the right queue), it’s clean: exact match against the label, measured as accuracy or F1. Easy to score, easy to automate.

For structured extraction (pulling fields from a document), score each field. Did it get the amount right? The date? The vendor ID? A model that nails 6 of 7 fields is very different from one that nails 3, and a single pass/fail hides that.

For open-ended generation (a drafted summary, a support reply), there’s no single right string, so you need one of:

Rubric scoring, a human rates each output against defined criteria (factually correct? complete? on-policy? right tone?).
LLM-as-judge, a strong model scores outputs against your rubric. Fast and cheap and genuinely useful, but validate it against human scores on a sample first, because judge models have biases (they tend to like longer answers, and they tend to favor outputs that sound like themselves). Trust it only after you’ve checked it agrees with your humans.

Whatever you choose, write the definition down before you run anything. Deciding what “good” means after you’ve seen the scores is how motivated reasoning sneaks in.

Step 3: Build the Harness

The engineering here is refreshingly boring, which is the point.

A simple loop: for each model in your shortlist, for each example in your dataset, send the same prompt, capture the output, the latency, and the token counts. Store everything. Score it. Aggregate.

A few things that separate a harness you can trust from one you can’t:

Hold the prompt constant across models, same instructions, same format, same few-shot examples. If you hand-tune the prompt for your favorite and use a lazy one for the rest, you’re benchmarking your prompt-writing, not the models.
Run each example more than once. LLMs are non-deterministic even at low temperature. Three runs per example and an average keeps a lucky or unlucky single roll from deciding your architecture.
Capture cost and latency as first-class metrics, not afterthoughts. Log tokens in and out per call; you’ll convert that to real money at your projected volume later.
Log every raw output. When a model scores surprisingly high or low, you’ll want to read what it actually produced. Half your real insight comes from reading the failures, not the scores.

Keep it in version control. When the next model launches, you re-run the same harness against the same labeled set and get an apples-to-apples answer in an hour, instead of re-litigating the whole decision from scratch.

Step 4: Score the Whole System, Not Just Quality

Here’s where most internal benchmarks still go wrong, they rank on quality and stop, reinventing the exact single-axis mistake the public leaderboard made.

A real enterprise decision is multi-constraint. Build a scorecard that holds them together:

Quality , your task-specific score from Step 2.
Cost , projected monthly spend at your real volume. A 2-point quality edge rarely justifies a 5x bill.
Latency , p50 and p95. The tail matters: a model that’s fast on average but occasionally stalls for ten seconds can be unusable in a live chat.
Failure behavior , when it’s wrong, is it wrong safely? A model that escalates or says “I’m not sure” beats one that confidently fabricates, even at equal accuracy.
Deployability , can it meet your data-residency and compliance constraints? Sometimes “must be self-hostable” silently eliminates the top three on quality, and that’s fine, it’s a real constraint.

Weight these for your situation , a high-volume support bot weights cost and latency heavily; a low-volume compliance tool weights quality and failure behavior above almost everything, and let the weighted scorecard pick the winner. The output isn’t “Model X is best.” It’s “Model X is best for us, at our volume, under our constraints,” which is the only ranking that was ever going to help you.

The Payoff, and Where It Leads

Do this once and you own something genuinely valuable: a labeled dataset and a repeatable harness that turn every future model release from an anxiety-inducing “should we switch?” into a one-hour measured answer. When the next frontier model drops with breathless leaderboard claims, you run it through your harness and know, for your workload, whether it’s worth the migration.

That capability compounds. It’s the same muscle you need for any serious custom LLM implementation, because selecting the model is step one of a longer job: the evals you build here become the regression tests that keep the system honest as you fine-tune, swap models, and scale. And once the model is chosen, the same measure-everything discipline is what separates an agent that demos well from one that survives production, which is the entire difficulty of real custom AI agent development.

The One-Paragraph Version

Public leaderboards answer “which model is generally good.” That is not your question. Your question is “which model is good at my task, at my volume, under my constraints, on my messy data”, and the only honest way to answer it is to build a small internal benchmark: a few hundred real labeled examples, a task-specific scoring method, a fair harness that holds the prompt constant and runs each case a few times, and a scorecard that weighs quality against cost, latency, failure behavior, and deployability. It’s a weekend of work that saves you from a six-figure mistake and keeps paying off every time a new model ships.

The leaderboard is where you start. The benchmark you build is how you actually decide.

Have you run your own internal model bake-off? I’m curious what surprised you most , for us it’s almost always that the “best” model on paper loses on the messy edge cases, and a cheaper mid-ranked model wins the real workload.

Building AI-Powered Payment Support Workflows for Modern Customer Service Teams

Dextra Labs — Wed, 30 Sep 2026 17:52:55 +0000

Most payment support automation stops at the chatbot layer. A customer asks about a failed charge, and the bot explains the refund policy. That is text generation, not workflow execution, and the gap between the two is where the real engineering challenge lives.

The interesting problem is not generating a response. It is building a system that can identify a customer, inspect payment records, check policy, execute an action through an API, verify the result, and update the support ticket, all within a controlled, auditable workflow.

In this blog, we walk through the architecture, integrations, failure handling, and security guardrails required to build AI-powered payment support workflows that actually do the work rather than just talk about it.

Payment Support Is a Workflow Problem, Not Just a Chatbot Problem

Consider a familiar support scenario: "I was charged twice. Can you refund one of the payments?"

A basic chatbot can explain the refund policy. An AI-powered workflow needs to actually identify the customer, inspect payment records, verify eligibility, check whether automation is permitted, execute the refund, update the support system, and respond with the outcome.

The difference becomes clear when you compare the two approaches. Traditional support follows a path where the customer message reaches an agent, who searches systems, checks the payment, takes the action, and updates the ticket. An AI-powered workflow follows a longer but more structured path: the customer request triggers intent detection, which pulls payment data, runs a policy check, executes the action, verifies the result, updates the ticket, and sends the response.

DEV readers will immediately recognize that the interesting engineering problem here is workflow execution, not text generation. As Dextra Labs' guide to AI agents for customer service explains, this execution-oriented pattern is increasingly central to how production support agents are being built.

Which Payment Support Tasks Are Worth Automating?

Not every payment conversation should be automated. The right starting point is repeatable, measurable workflows where the steps and decision logic are well defined.

Start with the tasks that have clear inputs, predictable steps, and measurable outcomes before expanding into more complex or ambiguous conversations.

Architecture of an AI Payment Support Workflow

This is where engineering gets interesting. A production AI agents for payment processing workflow typically follows a layered architecture where the customer message flows through intent classification, then into customer and payment context retrieval, through a policy and eligibility engine, into an action planner, out through payment, CRM, and ticketing APIs, through a verification step, into an audit log, and finally back as a customer response.

Each layer has a specific responsibility:

Intent layer: Identifies whether the request is a refund, failed payment, duplicate charge, invoice question, dispute, or something else.
Context layer: Retrieves customer, invoice, transaction, and subscription information from connected systems.
Policy layer: Determines what the agent is allowed to do based on business rules, thresholds, and customer state.
Action layer: Calls payment and business APIs to execute the permitted action.
Verification layer: Confirms that the action actually succeeded at the payment provider.
Audit layer: Records what happened, what was decided, and why.

A key engineering principle: the LLM should interpret requests and coordinate tools, while deterministic business rules enforce what the system is allowed to do. Policy enforcement belongs outside the probabilistic model.

Connecting the Agent to Payment and Support Systems

The agent becomes useful only when it can access the systems where the actual work happens.

Typical integrations:

Payment processor APIs (Stripe, Adyen, Razorpay, etc.)
Billing and subscription platform
CRM and customer database
Helpdesk and ticketing system
Order management system
Internal policy and knowledge base

A typical workflow calls a sequence like:

get_customer()
get_transactions()
check_refund_policy()
create_refund()
update_ticket()
send_customer_message()

Tools should expose narrow, permissioned operations rather than unrestricted database or payment access. For sensitive payment information, the workflow should use payment-provider tokens, IDs, and approved APIs rather than giving the LLM direct access to raw payment credentials.

Example: Automating a Duplicate-Charge Request

Rather than several shallow examples, here is one concrete workflow from start to finish.

Customer message: "I was charged twice for my subscription this month."

Workflow steps:

Identify the customer from the support session.
Retrieve recent successful transactions.
Compare amount, invoice, timestamp, and transaction status.
Determine whether the second charge is genuinely duplicated.
Retrieve the applicable refund policy.
Check the automated-refund threshold.
Execute the refund if permitted.
Verify the payment provider's response.
Add an internal ticket note.
Send the customer the outcome.

The decision structure looks something like:

json
{
  "intent": "duplicate_charge",
  "duplicate_confirmed": true,
  "refund_allowed": true,
  "requires_approval": false
}

Where Human Approval Should Stay in the Loop

Human-in-the-loop is not a weakness. It is an architectural control, and payment workflows need it in specific places.

Good candidates for human review:

High-value refunds above the automated threshold
Ambiguous duplicate-payment cases
Chargeback disputes requiring judgment
Suspected fraud
Policy exceptions
Account ownership uncertainty
Irreversible financial actions
Conflicting customer or payment data

The pattern is straightforward: low-risk requests that pass policy checks proceed to automated action, while high-risk or ambiguous requests route to human review. The agent should pause rather than guess when required information is missing or contradictory.

Designing for Failures, Retries, and Idempotency

Payment workflows cannot assume every API call succeeds. The system needs to handle:

Payment API timeouts
Duplicate webhooks
Partial workflow completion
Stale customer data
Failed refund attempts
CRM unavailability
Conflicting payment states
Network retries creating duplicate actions

Consider a refund request where the payment API times out. The system retries with an idempotency key, then verifies the actual transaction state before proceeding. If the refund succeeded, the ticket is updated.
If it failed, the case escalates to a human agent.

Idempotency, state management, retries, and explicit failure states are essential when an AI agent can trigger financial actions. Without them, a retry can become a double refund.

Security and Guardrails for Payment AI Agents

Keep this practical. The guardrails that matter most for payment workflows:

Least-privilege tool permissions
API-level authorization for every action
Refund and transaction limits
Customer identity verification before any action
Sensitive-data filtering (no raw card numbers in logs or LLM context)
Deterministic policy enforcement
Action logging with full traceability
Human approval thresholds
Kill switches for immediate workflow shutdown

A useful principle: never let the LLM be the final authority for a financial action. The model can interpret an instruction, but permissions and transaction constraints should be enforced by application code.

Measuring Whether the Workflow Actually Works

Move beyond "accuracy" and measure the entire workflow end to end.

An AI support workflow should be evaluated on successful task completion, not merely how convincing its responses sound.

Start With One Payment Workflow, Then Expand

The strongest implementation approach for custom AI agent development services in payment support follows a phased progression:

Phase 1: Automate read-only payment-status questions.
Phase 2: Add policy-driven, low-risk actions such as small refunds.
Phase 3: Introduce human approval for higher-risk actions.
Phase 4: Add proactive workflows such as failed-payment recovery and dispute preparation.
Phase 5: Continuously evaluate logs, failures, and customer outcomes.

This aligns with the broader 2026 pattern of starting with a narrowly defined workflow, connecting the required tools, and expanding only after the workflow is measurable and reliable.

Technical Due Diligence Errors That Can Kill an M&A Deal

Dextra Labs — Tue, 29 Sep 2026 10:50:36 +0000

If you've been an engineer long enough, one day someone from the business side will pull you into a call, share a private repo you've never seen, and ask a question that sounds simple and isn't:

"So… is this codebase okay? We're thinking of buying the company."

That question is technical due diligence. And the honest answer is almost never "yes" or "no", it's "okay for what, and at what cost?" Because the people asking are about to move real money based on your read of a system you've had for about a week.

Here's the thing nobody tells the engineer in that seat: a deal can survive a bad negotiation. It can survive a mediocre earn-out or a clumsy term sheet. What it often can't survive is a technology risk nobody saw until after signing.

A target looks great on paper, recurring revenue, customer growth, "proprietary" software, an AI story, a big engineering team. Then diligence (or worse, post-close reality) turns up the stuff that changes the math:

the architecture can't take the projected growth
source-code ownership is… complicated
the technical debt is really a rewrite wearing a trench coat
there are security holes that come with legal liability attached
cloud costs were modelled by an optimist
two engineers are load-bearing walls
the "AI" is a thin shell over someone else's API
integration is 3× harder than the deck implied

The whole job of technical due diligence is to surface that stuff before signing, while the buyer can still price it, negotiate it, or walk. TDD exists to reduce uncertainty before the deal depends on closing, not to document surprises after.

So here are the errors I see kill deals, roughly in the order they tend to happen.

First, what Technical DD actually is (it's not a code review)

Quick baseline, because half the technical due diligence mistakes below come from getting this wrong.

Technical diligence helps a buyer understand what they're actually acquiring, whether it works as represented, how well it scales, what the risks are, what remediation will cost, whether the tech supports the reason for the deal, and how painful integration will be.

And critically, it is not a code review. A real review spans the whole estate:

Code → Architecture → Infrastructure → Security → Data → IP → Engineering → Vendors → Roadmap

You're not grading a pull request. You're assessing an operating technology environment that a business is about to depend on. Keep that scope in your head, most of the failures below are really just people quietly shrinking it back down to "read the code."

1. Starting too late

The single most damaging error: bringing technical people in after the commercial assumptions are already locked.

By the time the engineer gets access, the deal team has often already agreed the valuation, drafted the integration plan, estimated synergies, and negotiated the big terms. Then diligence turns up something material, and now you're not informing the deal, you're detonating it.

Late findings trigger valuation renegotiation, extra warranties, escrow demands, integration-cost revisions, delays, and a lot of tense calls between buyer and seller who both thought this was done.

- TDD as a box to tick before close
+ TDD early enough that findings can still change the deal

If the diligence can't influence the terms, it's not diligence. It's archaeology with a deadline.

2. Treating TDD as a code review

Reading the source matters. It just doesn't tell you the whole story, and betting a deal on it alone is how buyers get blindsided.

A code-only pass reliably misses cloud architecture, infrastructure costs, cybersecurity, IP ownership, third-party dependencies, key-person risk, vendor contracts, data architecture, scalability, and integration complexity. You can have clean, elegant code and still be buying a disaster.

The mental model that fixes it:

Application + Infrastructure + Data + Security + People + IP + Vendors + Operations

You are acquiring an operating environment, not a git clone. The code is one file in the repo.

3. Not separating deal-killing risk from ordinary backlog

A report with 100 technical findings is not useful to a deal team. It's noise with a page count.

The only question that matters to the people signing is: which of these could actually move the transaction? So sort findings by business impact, not by how much they annoy you as an engineer:

Then translate each one into money:

Legacy architecture → modernization cost
Poor scalability → infrastructure investment
Security weakness → remediation plus potential liability
IP ownership gap → legal and transaction risk

That translation is what turns a technical report into a decision tool. A finding with no dollar figure and no severity is just trivia.

4. Underestimating technical debt

Every engineer instinctively over-indexes on debt because we're the ones who live with it. But not all debt is a deal-killer, and treating it that way makes you the person who tanks good deals over // TODO: refactor this.

The distinction that matters:

Manageable debt is documented, prioritised, understood by the team, and compatible with the roadmap. Fine. Normal. Every real codebase has it.

Material debt is undocumented, everywhere, actively blocking product work, causing reliability problems, and heading toward "we need to replace the architecture." That's the kind that shows up in the integration budget as a nasty surprise.

Questions worth asking the target's engineers directly:

How much debt is there, and where's it concentrated?
Why was it created? (Deliberate speed, or nobody was watching?)
Does it affect scalability?
Is remediation already budgeted?
Will it interfere with integration?
How many engineer-months to resolve the serious parts?

The answers tell you whether you're looking at a healthy fast-moving codebase or a bill nobody's added up yet.

5. Assuming they own what they built

A buyer will happily assume that acquiring a software company means acquiring all its software. That assumption has ended careers.

Audit ownership: employee IP assignments, contractor agreements, outsourced development, founder contributions, patents, proprietary algorithms, and who actually controls the repos.

Audit dependencies: open-source (and its licences, copyleft is not your friend at signing), commercial libraries, APIs, cloud providers, AI models, third-party data, software licences.

Why it reaches all the way to the closing table: an ownership or licensing gap can blow up the transaction representations, the warranties, the valuation, transferability, and the integration plan. "We built it" and "we own it" are different sentences, and diligence is where you find out which one is true.

6. Running the same generic checklist on every deal

A technology due diligence checklist gives you structure, and structure is good. But running the identical checklist on every acquisition means you spend time on low-impact boxes while missing the risks specific to this target's stack and business model.

Different deals need different depth:

SaaS → multi-tenancy, scalability, cloud economics, customer data isolation, recurring infra costs.

AI-native → model architecture, training data, model dependencies, inference costs, eval methodology, data rights, foundation-model reliance.

Fintech → security, data governance, regtech, legacy transaction systems, infrastructure.

Start from a standard framework, then customise it against the industry, the tech, the deal structure, the thesis, the regulatory exposure, and the integration goals. A checklist you didn't adapt is a checklist that's quietly looking in the wrong place.

7. Never checking the tech against the deal thesis

This is the M&A-specific miss, and it's a big one. Diligence should answer more than "is the technology healthy?" It has to answer: can this technology support the reason we're buying the company?

Same codebase, three different theses, three different reviews:

Growth / international scale → architecture, infrastructure, localization, data residency, reliability.
Cost synergy → application duplication, cloud spend, vendor contracts, infra consolidation.
Technology acquisition (acqui-hire / IP) → proprietary IP, code quality, technical differentiation, engineering capability, roadmap.

If you assess a cost-synergy deal like a growth deal, you'll write a beautiful report that answers the wrong question. Anchor the diligence to why the buyer is doing this at all.

8. Skipping cybersecurity and data risk

A product can pass every functional test and still be a liability the moment you own it.

Look hard at vulnerability management, IAM, encryption, secrets management (please, check for the .env committed in 2021), cloud security, incident response, data protection, backup/recovery, and monitoring.

And ask the pointed questions:

Have there been material security incidents, and were they disclosed?
Are there critical unresolved vulnerabilities?
Who has privileged access, and should they?
Is customer data actually protected, or just stored?
Are the controls appropriate for the industry, not just the stage?
Will security requirements jump after integration into a bigger, more-targeted company?

Post-close breaches are expensive in cash, churn, regulators, and reputation — all at once. This is not the section to skim.

9. Ignoring the bus factor

A platform can rest almost entirely on a few people, and that risk doesn't show up anywhere in the code.

Ask: who actually understands the critical systems? How dependent is everything on the CTO? Who maintains core infra? Which systems are undocumented? What happens if the two people who get it take the acquisition payout and leave? Can the buyer even retain them? Is the team big enough for the roadmap they're promising?

If one or two people are the only ones who understand a critical system, you're not acquiring a platform. You're acquiring their goodwill, on a month-to-month basis.

This matters most when the whole thesis depends on the target's tech keeping the lights on immediately after close, which is most of the time.

10. Treating AI like conventional software

If the target is AI-native, the standard software checklist isn't enough, there's a whole extra surface to assess: training data and its provenance, model ownership, foundation-model dependencies, inference costs, real-world model performance, evaluation methodology, fine-tuning, MLOps maturity, model security, and third-party API reliance.
And watch the economics especially, because this is where AI deals quietly break:

Impressive demo
  ├─ high inference cost per request
  ├─ expensive GPU footprint
  ├─ mediocre model efficiency
  └─ heavy dependency on one external provider
        └─ who can change pricing or pull access anytime
= margins that evaporate at scale

A model that dazzles in the demo can have unit economics that make the business model impossible at volume. AI-native deals may need reviewers who actually know model architecture, data pipelines, and inference economics, leading technical due diligence services for AI startups exist precisely to push the review past conventional code-and-architecture into the AI-specific risks.

11. Never translating findings into deal decisions

A technically brilliant report that a deal team can't act on has failed. The chain that makes it useful:

Finding → Business impact → Financial impact → Deal implication

Worked example:

Finding: legacy infrastructure can't support projected growth.
Business impact: major modernization required.
Financial impact: significant additional technology investment.
Deal implication: revisit the integration budget and the value-creation assumptions.

Every finding in the final report should answer six things: what matters, why it matters, how much it could cost, how fast it has to be fixed, who owns the fix, and whether it affects the transaction. If a finding can't answer those, it belongs in a backlog, not a deal memo.

The anti-error checklist

Turn the mistakes into a process:

- Start early before the assumptions harden.
- Define the thesis know what the tech has to deliver.
- Customise the scope industry, tech, transaction, risk.
- Assess the whole estate not just the code.
- Prioritise material risk separate critical from backlog.
- Quantify remediation cost, time, people, infra, integration.
- Connect to the deal valuation, terms, integration, post-close spend, risk allocation.

A pre-closing checklist you can actually paste into a doc

Technology

Architecture reviewed
Codebase assessed
Technical debt identified
Scalability tested
Infrastructure evaluated

Security

Vulnerabilities reviewed
Security controls assessed
Data protection evaluated
Incident history reviewed

IP & dependencies

IP ownership verified
Contractor agreements checked
Open-source dependencies reviewed
Third-party licences assessed
Critical vendor dependencies identified

Engineering

Team structure reviewed
Key-person dependencies identified
Technical leadership assessed
Retention requirements understood

Integration

Integration complexity estimated
Duplicate systems identified
Migration requirements mapped
Post-close roadmap established

A quick aside from the trenches: if your deal has any of the scary-shaped risks above, gnarly architecture, an AI stack, real security exposure, this is the point where a specialist earns their fee. Good technology due diligence partners like Dextra Labs can hand you a prioritised list of "this is a Tuesday fix" vs. "this is why the margins don't work," instead of a 60-item spreadsheet of undifferentiated red. A findings list with no priorities isn't diligence, it's console.error on the whole codebase.

The bottom line

Technical due diligence doesn't exist to make an acquisition look risk-free. Nothing makes an acquisition risk-free. Its job is to surface the material technology risks early enough that the buyer can understand them, price them, negotiate them, or fix them.

The biggest failure was never "we found technical debt." Every company has technical debt.

The biggest failure is finding out, the week after signing, that the technology is far more expensive, more fragile, harder to integrate, or more dependent on two irreplaceable people than anyone in the deal room assumed.

Find it early Price it honestly, then decide.

Building an AI-Powered Compliance Monitoring System for Financial Institutions

Dextra Labs — Thu, 24 Sep 2026 08:12:19 +0000

We are going to walk through the architecture of a compliance monitoring system we built for a mid-size bank. Not the strategy deck version. The engineering version, with the data pipeline design, the agent architecture, the audit trail implementation, and the parts that took three times longer than we estimated.

If you're an engineer building for financial services, compliance monitoring is probably the project your CTO will ask about next quarter. The regulatory pressure is intensifying, the manual monitoring approach is hitting its capacity ceiling, and the gap between what regulators expect and what batch-review processes deliver is widening every month.

The AI agents in finance space has matured past experimentation. Banks are running agent-based compliance systems in production. But the engineering content around how to build them is thin, most of what's published is either vendor marketing or regulatory theory. This is the builder's perspective.

The problem in engineering terms

Compliance monitoring in a financial institution means evaluating every transaction and operational event against applicable regulations, internal policies, and risk thresholds. The key word is every.

A mid-size bank processing 200,000 transactions daily against roughly 150 distinct regulatory rules across AML, sanctions, lending compliance, consumer protection, and internal policy creates a monitoring matrix of 30 million evaluations per day. The manual approach, sampling 5 percent of transactions for human review, covers 1.5 million of those evaluations. The other 28.5 million go unmonitored until something draws attention to them.

The engineering challenge isn't AI sophistication. It's building a system that performs 30 million evaluations daily with sub-minute latency on new transactions, maintains complete audit trails for every evaluation, handles rule changes without system downtime, distinguishes genuine violations from false positives at a rate that doesn't overwhelm the compliance team, and satisfies regulators that the monitoring is comprehensive, documented, and explainable.

That's a systems engineering problem with AI components, not an AI problem with systems requirements.

The four-agent architecture

The system runs four specialised agents coordinated through a central orchestration layer. Each agent has a bounded scope, defined inputs and outputs, and its own audit trail stream.

                    ┌─────────────────────┐
                    │   Orchestration      │
                    │   Engine             │
                    └──────────┬──────────┘
                               │
            ┌──────────────────┼──────────────────┐
            │                  │                   │
            ▼                  ▼                   ▼
   ┌────────────────┐ ┌────────────────┐ ┌────────────────┐
   │  Transaction    │ │  Regulatory    │ │  Policy        │
   │  Monitor       │ │  Change Agent  │ │  Compliance    │
   │  Agent         │ │                │ │  Agent         │
   └────────────────┘ └────────────────┘ └────────────────┘
            │                  │                   │
            └──────────────────┼──────────────────┘
                               │
                    ┌──────────▼──────────┐
                    │   Audit Trail       │
                    │   Engine            │
                    └─────────────────────┘

A fourth agent, the audit readiness agent, operates on a batch schedule rather than in real time, assembling examination evidence packages from the audit trail. I'll cover it separately because its architecture is fundamentally different from the real-time agents.

Agent 1: Transaction monitoring

The transaction monitor evaluates every transaction against the full applicable rule set in real time. This is the highest-throughput component and the one where architectural decisions have the most direct performance impact.

The naive approach, sending every transaction through an LLM for compliance evaluation, fails at this scale on three dimensions: latency (LLM inference adds 500ms to 2s per evaluation), cost (200,000 daily transactions at inference pricing burns through budget), and determinism (LLM outputs on clear-cut regulatory rules aren't perfectly consistent, which regulators won't accept).

The architecture that works splits the evaluation into two tiers.

The deterministic tier handles rules that have clear, binary answers. Transaction amount exceeds the $10,000 CTR reporting threshold, yes or no. Counterparty appears on the OFAC sanctions list, yes or no. Transaction geographic origin falls in a restricted jurisdiction, yes or no. These evaluations are rules-based, not AI-based. They run as a streaming rules engine against each transaction event.

# Deterministic compliance checks - no AI needed
class DeterministicComplianceEngine:
    def __init__(self, rule_set: RuleSet):
        self.rules = rule_set

    def evaluate(self, transaction: Transaction) -> list[ComplianceResult]:
        results = []
        for rule in self.rules.get_applicable(transaction):
            result = ComplianceResult(
                rule_id=rule.id,
                transaction_id=transaction.id,
                passed=rule.evaluate(transaction),
                evaluation_type="deterministic",
                timestamp=datetime.utcnow(),
                evidence=rule.get_evidence(transaction)
            )
            results.append(result)
        return results

The AI tier handles evaluations that require interpretation. Is this sequence of transactions consistent with normal business activity or does it suggest structuring? Does this customer's recent behaviour represent a legitimate change in financial patterns or a potential account compromise? Does this transaction narrative contain information that contradicts the stated transaction purpose?

These evaluations use an LLM with a structured prompt that includes the transaction data, the relevant regulatory context, and the customer's behavioural baseline. The output is structured, a classification, a confidence score, and a reasoning chain.

# AI-assisted compliance evaluation for pattern-based rules
class AIComplianceEvaluator:
    def __init__(self, model_client, prompt_templates):
        self.client = model_client
        self.templates = prompt_templates

    def evaluate_pattern(
        self, 
        transaction: Transaction,
        customer_history: CustomerHistory,
        rule: PatternRule
    ) -> ComplianceResult:
        prompt = self.templates.render(
            rule_type=rule.type,
            transaction=transaction.to_context(),
            history=customer_history.recent_summary(),
            regulatory_context=rule.regulatory_text,
            output_schema="classification, confidence, reasoning"
        )

        response = self.client.complete(
            prompt=prompt,
            temperature=0.1,  # Near-deterministic for compliance
            max_tokens=500
        )

        parsed = self.parse_structured_response(response)

        return ComplianceResult(
            rule_id=rule.id,
            transaction_id=transaction.id,
            passed=parsed.classification == "compliant",
            confidence=parsed.confidence,
            reasoning=parsed.reasoning,
            evaluation_type="ai_assisted",
            timestamp=datetime.utcnow(),
            model_version=self.client.model_version,
            prompt_hash=hash(prompt)
        )

The temperature=0.1 is deliberate and non-negotiable for compliance evaluations. We need near-deterministic outputs. We also store the prompt_hash and model_version in every evaluation result because regulators need to reproduce the conditions under which any specific evaluation was made.

The split between deterministic and AI tiers is the design decision that makes the system viable at scale. In our deployment, roughly 85 percent of evaluations are deterministic, clear rules applied to structured data. The remaining 15 percent go through the AI tier. This means LLM inference runs on 30,000 transactions per day rather than 200,000, which changes the cost and latency picture entirely.

Agent 2: Regulatory change monitoring

This agent scans regulatory publications and identifies changes relevant to the institution's operations. It's the most straightforward AI application in the system and the one that delivers value fastest.

The data pipeline ingests publications from the institution's applicable regulators, federal register entries, supervisory letters, guidance documents, enforcement actions, consent orders. For US banking, that's the OCC, FDIC, Federal Reserve, CFPB, and FinCEN at minimum, plus state regulators for the institution's operating jurisdictions.

# Regulatory change monitoring pipeline
class RegulatoryChangeAgent:
    def __init__(self, sources, relevance_model, institution_profile):
        self.sources = sources  # RSS feeds, API endpoints, scrapers
        self.model = relevance_model
        self.profile = institution_profile

    def scan(self) -> list[RegulatoryChange]:
        new_publications = []
        for source in self.sources:
            new_publications.extend(source.fetch_since_last_scan())

        relevant_changes = []
        for pub in new_publications:
            assessment = self.model.assess_relevance(
                publication=pub,
                institution_products=self.profile.products,
                institution_jurisdictions=self.profile.jurisdictions,
                institution_charter=self.profile.charter_type
            )

            if assessment.relevance_score > 0.6:
                change = RegulatoryChange(
                    source=pub.source,
                    publication_date=pub.date,
                    summary=assessment.summary,
                    affected_products=assessment.affected_products,
                    affected_rules=assessment.affected_rules,
                    recommended_actions=assessment.actions,
                    relevance_score=assessment.relevance_score,
                    urgency=assessment.urgency_classification
                )
                relevant_changes.append(change)

        return relevant_changes

The relevance model isn't trying to interpret the law. It's classifying whether a publication is relevant to this specific institution's products, services, and jurisdictions, and identifying which internal policies and monitoring rules might need updating. The compliance team makes the interpretive decisions, the agent surfaces what they need to see.

This agent runs on a scheduled basis, every six hours for federal sources, daily for state sources. The latency tolerance is hours, not seconds, which means it can use larger models with longer inference times for better classification quality.

Agent 3: Policy compliance monitoring

The policy compliance agent monitors internal operations against the institution's own policies and procedures, employee trading restrictions, information barriers, approval authority limits, customer communication standards.

The architecture mirrors the transaction monitor's two-tier approach. Deterministic checks for clear policy rules (approval authority thresholds, mandatory cooling-off periods, prohibited activity lists). AI-assisted evaluation for policies that require interpretation (communication tone compliance, conflict of interest assessment, information barrier monitoring across communication channels).

The data sources are broader than transaction data, email metadata, internal messaging, access logs, trading activity, document access patterns. The AI compliance monitoring architecture for policy compliance requires integration with communication platforms, HR systems, and access management infrastructure alongside the financial systems.

# Policy compliance - information barrier monitoring
class InformationBarrierMonitor:
    def __init__(self, barrier_config, communication_feed, model):
        self.barriers = barrier_config
        self.feed = communication_feed
        self.model = model

    def evaluate_communication(
        self, event: CommunicationEvent
    ) -> Optional[PolicyAlert]:
        # Deterministic check: are participants in restricted groups?
        sender_group = self.barriers.get_group(event.sender)
        recipient_groups = [
            self.barriers.get_group(r) for r in event.recipients
        ]

        barrier_crossed = any(
            self.barriers.is_restricted(sender_group, rg)
            for rg in recipient_groups
        )

        if not barrier_crossed:
            return None  # No barrier concern

        # AI assessment: does the communication content
        # contain material non-public information?
        content_assessment = self.model.assess(
            content_summary=event.content_summary,  # Never full content
            sender_role=event.sender_role,
            context="information_barrier_evaluation",
            output_schema="risk_level, reasoning, recommended_action"
        )

        if content_assessment.risk_level in ["high", "critical"]:
            return PolicyAlert(
                alert_type="information_barrier",
                severity=content_assessment.risk_level,
                participants=event.participants,
                reasoning=content_assessment.reasoning,
                recommended_action=content_assessment.recommended_action,
                timestamp=datetime.utcnow()
            )

A critical design note in the code above: the AI model receives content_summary rather than the full communication content. In regulated environments, the compliance monitoring system itself must respect data handling restrictions. The model assesses risk indicators, not reads everyone's email. This privacy-by-design approach is a regulatory requirement, not an engineering nicety.

The audit trail engine

The audit trail isn't a logging feature. It's the primary deliverable. Everything the system produces, every deterministic evaluation, every AI-assisted assessment, every alert generated, every human decision on an escalated case, flows into an immutable audit store.

# Audit trail, immutable, complete, reproducible
@dataclass
class AuditRecord:
    record_id: str
    timestamp: datetime
    event_type: str  # evaluation, alert, escalation, resolution
    agent_id: str

    # What was evaluated
    subject_id: str  # transaction_id, communication_id, etc.
    subject_data_hash: str  # Hash of the input data

    # How it was evaluated
    evaluation_type: str  # deterministic, ai_assisted
    rule_id: str
    model_version: Optional[str]  # For AI evaluations
    prompt_hash: Optional[str]  # For reproducibility

    # What was concluded
    result: str  # compliant, non_compliant, escalated
    confidence: Optional[float]
    reasoning: Optional[str]

    # What happened next
    action_taken: str
    human_reviewer: Optional[str]
    human_decision: Optional[str]
    human_decision_timestamp: Optional[datetime]

The subject_data_hash and prompt_hash fields are the reproducibility mechanism. If a regulator asks "why was this transaction evaluated as compliant on March 15th," the audit record contains the hash of the exact data that was evaluated and the exact prompt that was used. The original data can be retrieved and the evaluation can be reproduced with the same model version.

The audit store uses append-only storage, records are never modified or deleted. The retention period matches regulatory requirements, typically seven to ten years for banking. We use a combination of a hot tier (last 90 days in PostgreSQL for fast querying) and a cold tier (older records in S3 with Parquet format for cost-efficient long-term storage).

The audit readiness agent

The fourth agent operates on a batch schedule, weekly and on-demand, rather than in real time. It assembles examination evidence packages from the audit trail, organised against specific regulatory examination modules.

When an examination notice arrives, the compliance team specifies which examination modules apply. The agent queries the audit trail for the relevant time period, assembles the evidence for each module, transaction monitoring coverage statistics, alert volumes and resolution outcomes, rule change history, policy compliance metrics and produces a structured evidence package that the examination team can review.

This is the agent that turned six weeks of examination preparation into four days for one institution we worked with. The evidence already existed in the audit trail. The agent's job was assembly and formatting, not investigation.

The engineering lessons that cost us the most time

Three things took longer than estimated and are worth knowing before you start.

The deterministic rule engine was harder than it looked because regulatory rules aren't as deterministic as they appear in the regulation text. A rule that says "transactions exceeding $10,000" seems binary. Then you discover that the $10,000 threshold applies to aggregated transactions within a 24-hour window from the same customer, that the aggregation logic must account for transactions across multiple accounts held by the same beneficial owner, and that "same customer" has a specific legal definition that doesn't map cleanly to your customer ID field. What looked like a simple threshold check became a multi-step aggregation query with entity resolution.

Model consistency monitoring consumed more engineering effort than model development. In a compliance context, you need to prove that the AI tier produces consistent results over time, that the same input evaluated today produces the same result it would have produced last month. We built a regression testing pipeline that re-evaluates a golden test set of 500 labelled transactions against every model update and every prompt change. If the evaluation results drift beyond a defined threshold, the update is blocked. This pipeline took three weeks to build. It runs automatically and has blocked two updates that would have changed evaluation behaviour in ways the compliance team hadn't reviewed.

Integration with legacy communication systems for policy monitoring was the longest single engineering task. The bank's internal communication infrastructure included a modern email system, a legacy messaging platform, and a voice recording system with its own proprietary format. Building the ingestion pipeline that normalised communication metadata from all three sources into a format the policy compliance agent could evaluate took six weeks, longer than building the agent itself.

The operational architecture

┌──────────────────────────────────────────────────┐
│              Event Sources                        │
│  Core banking │ Payments │ CRM │ Comms │ Trading │
└───────────────────────┬──────────────────────────┘
                        │ (Event-driven)
                        ▼
┌──────────────────────────────────────────────────┐
│           Stream Processing Layer                 │
│  (Kafka / event bus - normalisation, routing)    │
└───────────────────────┬──────────────────────────┘
                        │
         ┌──────────────┼──────────────┐
         ▼              ▼              ▼
┌──────────────┐ ┌────────────┐ ┌────────────────┐
│ Deterministic│ │ AI-Assisted│ │ Policy         │
│ Rules Engine │ │ Evaluator  │ │ Monitor        │
│ (85% volume) │ │ (15% vol)  │ │ (async)        │
└──────┬───────┘ └─────┬──────┘ └───────┬────────┘
       │               │               │
       └───────────────┼───────────────┘
                       ▼
┌──────────────────────────────────────────────────┐
│           Audit Trail Engine                      │
│  (Append-only │ Immutable │ 7-10yr retention)    │
└───────────────────────┬──────────────────────────┘
                        │
         ┌──────────────┼──────────────┐
         ▼              ▼              ▼
┌──────────────┐ ┌────────────┐ ┌────────────────┐
│ Alert Queue  │ │ Dashboard  │ │ Audit Readiness │
│ (compliance  │ │ (real-time │ │ Agent (batch    │
│  team)       │ │  metrics)  │ │  assembly)      │
└──────────────┘ └────────────┘ └────────────────┘

The stream processing layer is the foundational infrastructure investment. We use Kafka for event ingestion because the throughput requirements (200,000+ events daily with sub-minute processing latency) exceed what REST-based architectures handle gracefully. Each source system produces events to Kafka topics. The normalisation layer transforms source-specific event formats into the common evaluation schema.

The deterministic and AI-assisted evaluation paths are separate consumers from the same Kafka topics. The routing decision, which transactions need AI evaluation versus deterministic-only, is made by a lightweight classifier at the stream processing layer based on transaction characteristics (amount, type, counterparty risk level, customer risk profile).

What this costs to build and run

Development timeline: approximately five months for the core system (transaction monitoring, audit trail, basic regulatory change monitoring). An additional two months for policy compliance monitoring and the audit readiness agent. Total: seven months from kickoff to full production deployment.

Infrastructure costs: approximately $4,000 to $7,000 per month for a mid-size deployment. This covers the Kafka cluster, the inference compute for the AI tier, the PostgreSQL hot tier, and the S3 cold storage. The LLM inference costs are the largest variable component, they scale with the volume of transactions routed to the AI tier.

The cost comparison against manual monitoring: a twelve-person compliance team doing sample-based review costs roughly $1.2 to $1.8 million annually in fully loaded compensation. The agent-based system costs roughly $250,000 to build and $60,000 to $85,000 annually to operate. The system monitors 100 percent of transactions. The manual team monitored 5 percent. The math isn't close, and it gets more favourable as transaction volume grows because the system's cost scales sub-linearly while the manual team's cost scales linearly.

For financial institutions ready to build compliance monitoring that evaluates every transaction against every applicable rule with complete audit trail documentation, the AI agent development partner practice at Dextra Labs covers the full stack, from stream processing architecture through deterministic rule engines, AI-assisted evaluation pipelines, audit trail implementation, and the regulatory change monitoring that keeps the rule set current as regulations evolve.

The compliance monitoring problem is fundamentally an engineering problem. The AI components are important but bounded, they handle the 15 percent of evaluations that require interpretation. The other 85 percent is deterministic rules applied at streaming scale with complete auditability. Building for compliance means building for both, and knowing which evaluation type applies to which rule.

Published by Dextra Labs, AI Consulting and Enterprise Agent Development

The Shift from AI Insights to AI Actions in Finance

Dextra Labs — Wed, 23 Sep 2026 07:47:05 +0000

There's a line in finance technology that most teams don't realise they've been standing on one side of for the past decade.

On one side: AI that tells you things. A model that predicts which invoices will be paid late. A dashboard that flags anomalous transactions. A forecast that estimates next quarter's revenue. A risk score that classifies a loan application.

On the other side: AI that does things. An agent that processes the invoice, chases the payment, and reconciles the outcome. A system that detects the anomalous transaction, investigates it, assembles the evidence, and routes a case to the investigator with full context. An underwriting pipeline that reads the application, extracts the financials, calculates the ratios, checks compliance, and produces a recommendation with a documented reasoning chain.

The first category, traditional AI in finance, has been in production for years. It works. It's valuable. It gives finance teams better information to act on.

The second category, agentic AI, is what's changing the operational economics of finance right now. It doesn't give teams better information. It handles the work that teams used to do with that information.

The distinction matters architecturally because the two categories are built differently. And it matters operationally because AI agents across finance operations are delivering ROI measured in headcount reallocation and cycle time compression, not in "better dashboards."

This article walks through the architectural shift from insights to actions, what changes in the system design, what new components are required, and what the code looks like when you move from a model that scores to an agent that executes.

What traditional AI in finance actually looks like

Traditional AI in finance is predominantly predictive and classificatory. The model receives structured input, produces a score or a classification, and hands it to a human who decides what to do.

A credit scoring model is the canonical example:

# Traditional AI: predict, score, hand to human
class CreditScorer:
    def __init__(self, model):
        self.model = model

    def score(self, application: dict) -> CreditScore:
        features = self.extract_features(application)
        probability = self.model.predict_proba(features)

        return CreditScore(
            applicant_id=application["id"],
            score=probability[1],  # P(default)
            risk_tier=self.classify_tier(probability[1]),
            timestamp=datetime.utcnow()
        )

    def extract_features(self, application: dict) -> np.array:
        return np.array([
            application["debt_to_income"],
            application["credit_utilization"],
            application["payment_history_months"],
            application["employment_years"],
            # ... 30+ features manually extracted upstream
        ])

The model does one thing well: given a set of pre-extracted features, it produces a probability of default. That probability is valuable. An underwriter who has it makes better decisions than one who doesn't.

But look at what the model doesn't do. It doesn't extract the features, someone upstream manually pulled the debt-to-income ratio from bank statements, calculated credit utilisation from bureau data, and assembled the feature vector. It doesn't check compliance, someone downstream verifies the application against regulatory requirements. It doesn't produce a recommendation, someone interprets the score in the context of credit policy. It doesn't document its reasoning in a way a regulator can audit, the score is a number without an explanation.

The model is one step in a multi-step workflow where humans perform every other step. The model made that one step better. It didn't change the workflow.

A fraud detection model follows the same pattern:

# Traditional AI: flag, score, queue for human review
class TransactionFraudModel:
    def __init__(self, model, threshold=0.7):
        self.model = model
        self.threshold = threshold

    def evaluate(self, transaction: dict) -> FraudAlert | None:
        features = self.extract_features(transaction)
        fraud_probability = self.model.predict_proba(features)[1]

        if fraud_probability > self.threshold:
            return FraudAlert(
                transaction_id=transaction["id"],
                score=fraud_probability,
                alert_type="model_triggered",
                timestamp=datetime.utcnow()
            )
        return None

The model evaluates one transaction against one set of features at one point in time. It doesn't maintain a behavioural model across months of activity. It doesn't map relationships between accounts. It doesn't investigate the alert, an analyst does that. The model produces the flag. Humans do everything around it.

This is the insight paradigm. AI makes the information better. Humans do the work.

What the shift to actions looks like architecturally

The agentic paradigm replaces the single-step prediction with a multi-step execution loop. The agent doesn't just score, it reasons, plans, uses tools, handles exceptions, and produces a complete outcome.

The architectural difference is the reasoning loop. Traditional AI is function call: input → output. Agentic AI is a loop: observe → think → act → observe → think → act, continuing until the goal is achieved or the agent recognises it needs to escalate.

Here's the same underwriting workflow as an agent:

# Agentic AI: observe, reason, act, repeat until goal is met
class UnderwritingAgent:
    def __init__(self, llm, tools, compliance_rules, confidence_threshold=0.85):
        self.llm = llm
        self.tools = tools
        self.rules = compliance_rules
        self.threshold = confidence_threshold
        self.audit_log = []

    async def process_application(self, application_id: str) -> UnderwritingResult:
        # Step 1: Gather and extract documents
        docs = await self.tools.fetch_documents(application_id)
        extracted = await self.extract_all_documents(docs)

        # Step 2: Verify completeness
        missing = self.check_completeness(extracted, required_docs=["bank_statements",
            "tax_returns", "business_registration", "financial_statement"])

        if missing:
            await self.tools.request_documents(application_id, missing)
            self.audit_log.append(AuditEntry(
                step="document_collection",
                action="requested_missing_documents",
                details=missing,
                timestamp=datetime.utcnow()
            ))
            return UnderwritingResult(status="pending_documents", missing=missing)

        # Step 3: Calculate credit metrics
        metrics = self.calculate_metrics(extracted)

        # Step 4: Run compliance checks
        compliance = self.run_compliance_checks(extracted, metrics)

        # Step 5: Produce risk assessment
        assessment = await self.assess_risk(extracted, metrics, compliance)

        # Step 6: Route based on confidence
        if assessment.confidence >= self.threshold and compliance.all_passed:
            return UnderwritingResult(
                status="auto_recommended",
                risk_tier=assessment.risk_tier,
                metrics=metrics,
                compliance=compliance,
                reasoning=assessment.reasoning,
                audit_trail=self.audit_log
            )
        else:
            return UnderwritingResult(
                status="escalated_to_human",
                risk_tier=assessment.risk_tier,
                metrics=metrics,
                compliance=compliance,
                reasoning=assessment.reasoning,
                escalation_reason=self.get_escalation_reason(assessment, compliance),
                audit_trail=self.audit_log
            )

    async def extract_all_documents(self, docs: list[Document]) -> ExtractedData:
        """Extract data from each document using semantic understanding"""
        extracted = ExtractedData()

        for doc in docs:
            result = await self.tools.extract_document(
                document=doc,
                required_fields=self.get_required_fields(doc.type)
            )

            # Flag low-confidence extractions for human verification
            for field, value in result.fields.items():
                if value.confidence < 0.88:
                    result.flags.append(f"Low confidence on {field}: {value.confidence}")

            extracted.add(doc.type, result)
            self.audit_log.append(AuditEntry(
                step="document_extraction",
                document_type=doc.type,
                fields_extracted=len(result.fields),
                flags=result.flags,
                timestamp=datetime.utcnow()
            ))

        return extracted

    def calculate_metrics(self, data: ExtractedData) -> CreditMetrics:
        """Deterministic calculations, no AI needed"""
        return CreditMetrics(
            dscr=data.net_operating_income / data.total_debt_service,
            dti=data.total_monthly_debt / data.gross_monthly_income,
            ltv=data.loan_amount / data.collateral_value,
            current_ratio=data.current_assets / data.current_liabilities,
            cash_flow_trend=self.calculate_trend(data.monthly_cash_flows)
        )

    def run_compliance_checks(self, data: ExtractedData, 
                               metrics: CreditMetrics) -> ComplianceResult:
        """Rules-based compliance - deterministic, auditable"""
        results = []
        for rule in self.rules.get_applicable(data.loan_type):
            passed = rule.evaluate(data, metrics)
            results.append(ComplianceCheck(
                rule_id=rule.id,
                rule_description=rule.description,
                passed=passed,
                evidence=rule.get_evidence(data, metrics),
                timestamp=datetime.utcnow()
            ))
            self.audit_log.append(AuditEntry(
                step="compliance_check",
                rule_id=rule.id,
                passed=passed,
                timestamp=datetime.utcnow()
            ))

        return ComplianceResult(checks=results, all_passed=all(r.passed for r in results))

The code makes the architectural shift visible. The traditional model was a single function call: features in, score out. The agent is a multi-step workflow: gather documents, extract data, verify completeness, calculate metrics, check compliance, assess risk, and route based on confidence, with an audit trail documenting every step.

The five capabilities that separate agents from models

The underwriting agent code above demonstrates five capabilities that traditional models don't have. These five capabilities are what define the shift from insights to actions.

Tool use. The agent calls external systems, document storage, extraction services, the ERP, the compliance database. Traditional models receive pre-processed input. Agents fetch and process their own input by interacting with the systems where the data lives.

# Tool use: agent interacts with external systems
class AgentTools:
    async def fetch_documents(self, app_id: str) -> list[Document]:
        """Pull documents from the origination system"""
        return await self.origination_client.get_documents(app_id)

    async def extract_document(self, document: Document, 
                                required_fields: list) -> ExtractionResult:
        """Semantic extraction, understands content, not positions"""
        return await self.extraction_service.extract(
            document=document,
            fields=required_fields,
            return_confidence=True
        )

    async def request_documents(self, app_id: str, missing: list):
        """Send targeted document request to borrower"""
        await self.communication_service.send_document_request(
            application_id=app_id,
            missing_documents=missing,
            template="targeted_request"  # Specific, not generic
        )

State management

The agent maintains state across the workflow, what documents have been received, what data has been extracted, what compliance checks have passed, what the risk assessment concluded. Traditional models are stateless: same input, same output, no memory of what happened before.

Confidence-based routing

The agent knows when it's confident enough to recommend and when it should escalate. The confidence threshold is a design parameter that determines the system's autonomy boundary. Traditional models produce scores that humans interpret. Agents produce scores and act on them according to defined rules.

Exception handling

When the agent encounters something unexpected, a missing document, a low-confidence extraction, a compliance check that fails, it doesn't stop. It classifies the exception, determines the appropriate response, and either handles it within its authority or escalates with context. Traditional models have no concept of exceptions. They produce output regardless of input quality.

Audit trail generation

Every step the agent takes, every tool call, every extraction, every calculation, every routing decision, is logged with timestamps, inputs, and reasoning. The agentic AI for finance and accounting architecture treats the audit trail as a first-class output, not an afterthought. In regulated environments, the audit trail is often more valuable than the processing itself because it's what regulators actually examine.

Why the shift is happening now

Three technical developments converged to make the action paradigm viable.

Foundation models with tool-use capability. LLMs that can reason about when to call a tool, what inputs to provide, and how to interpret the results are the core enabler. Two years ago, getting a model to reliably call the right tool with the right parameters in a multi-step workflow required extensive prompt engineering and fragile orchestration. In 2026, tool calling is a native capability with structured outputs that integration code can rely on.

Infrastructure for agent orchestration. Frameworks for managing agent state, coordinating multi-step workflows, handling retries and errors, and maintaining audit trails have matured from experimental libraries to production-grade infrastructure. The orchestration layer that was custom-built for early deployments is now available as reusable architecture.

Cost reduction at inference scale. Processing 200,000 transactions daily through an LLM was economically impractical two years ago. The combination of tiered models (using expensive models only where judgment is required and cheap models or deterministic rules for everything else), prompt caching, and inference optimisation has reduced the per-transaction cost to levels where production-scale agent deployment is financially viable.

The hybrid architecture that actually works

The practical agent architecture in finance isn't "AI does everything." It's a deliberate split between what requires AI and what doesn't.

# The hybrid: deterministic where possible, AI where necessary
class HybridFinanceAgent:
    def __init__(self, rules_engine, ai_evaluator, tools):
        self.rules = rules_engine      # Fast, deterministic, auditable
        self.ai = ai_evaluator         # Interpretive, contextual, slower
        self.tools = tools

    async def process(self, event: FinanceEvent) -> ProcessingResult:
        # Deterministic first: apply all clear rules
        rule_results = self.rules.evaluate(event)

        # If rules are conclusive, no AI needed
        if rule_results.is_conclusive():
            return ProcessingResult(
                decision=rule_results.decision,
                evaluation_type="deterministic",
                reasoning=rule_results.evidence,
                confidence=1.0  # Rules are binary
            )

        # AI for the ambiguous remainder
        context = await self.tools.assemble_context(event)
        ai_result = await self.ai.evaluate(
            event=event,
            context=context,
            applicable_rules=rule_results.inconclusive_rules
        )

        return ProcessingResult(
            decision=ai_result.classification,
            evaluation_type="ai_assisted",
            reasoning=ai_result.reasoning,
            confidence=ai_result.confidence,
            model_version=self.ai.model_version,
            context_hash=hash(str(context))
        )

This hybrid pattern appears in every production finance agent we've built. The deterministic engine handles 75 to 85 percent of evaluations, the clear-cut cases where rules produce binary answers. The AI engine handles the remaining 15 to 25 percent, the cases where interpretation, context, and judgment are genuinely required.

The split matters for three reasons. Cost, deterministic evaluations are essentially free compared to LLM inference. Speed, rules execute in milliseconds versus seconds for AI evaluation. Auditability, deterministic evaluations produce provably consistent results that regulators find easier to examine than probabilistic AI assessments.

The agents that try to run everything through AI are more expensive, slower, and harder to audit than hybrid architectures that reserve AI for the work that actually needs it. This echoes what we've seen across every domain: the best agent systems use less AI than you'd expect, applied precisely where it creates value.

The operational results

The shift from insights to actions produces measurable operational changes that the insight paradigm couldn't deliver.

Cycle time compression. When the agent handles document collection, extraction, calculation, and compliance, instead of just scoring, the end-to-end processing time drops by 80 to 90 percent. An underwriting workflow that took 11 days drops to 31 hours. An AP invoice that took 11 days to approve processes in under 3 days. The time compression comes from eliminating the human steps between the AI steps, not from making the AI step faster.

Volume capacity multiplication. When 70 to 78 percent of transactions process without human intervention, the team's capacity for judgment work multiplies by 3 to 4x without adding headcount. One underwriting team went from 14 people processing all cases to 3 people handling a higher volume of complex cases.

Error rate reduction. Agents that apply compliance checks and matching logic with deterministic consistency produce 70 to 85 percent fewer errors than human teams performing the same checks at volume. The consistency isn't AI, it's the deterministic engine running the same rules every time without fatigue or attention drift.

Audit trail completeness. Traditional workflows produce audit trails assembled after the fact, someone reconstructing what happened from email threads and system logs. Agent workflows produce audit trails generated during the processing, every step documented as it occurs. The shift from retrospective to contemporaneous documentation is what changes compliance posture.

Where the shift is heading

The current generation of finance agents handles single-domain workflows, underwriting, AP processing, compliance monitoring, fraud detection. Each agent operates within one bounded domain with its own tools, rules, and escalation logic.

The next evolution is cross-domain agent orchestration, systems where a lending agent's credit assessment informs a fraud agent's risk scoring, where a compliance agent's regulatory change detection triggers a policy agent's rule update, where a customer operations agent's case resolution feeds back into a risk model's training data.

This cross-domain orchestration is where the compound value lives. Individual domain agents deliver linear improvements. Connected agents across domains deliver compounding improvements because intelligence flows between them.

That orchestration layer is the engineering challenge worth watching and building toward, in 2026.

For finance teams ready to make the shift from AI insights to AI actions, the custom AI agent development company practice at Dextra Labs builds the hybrid architectures described in this article, deterministic engines for the clear-cut evaluations, AI for the interpretive work, and the orchestration layer that connects them into workflows that execute rather than advise. Whether the starting point is underwriting, AP, compliance, or fraud detection, the architectural pattern is consistent and the engineering path is proven.

The insight era gave finance teams better information. The action era gives them back their time.

Published by Dextra Labs - AI Consulting and Enterprise Agent Development

Why SMBs Choose Custom AI Agent Development Over Off-the-Shelf Platforms

Dextra Labs — Tue, 22 Sep 2026 05:26:59 +0000

There is no shortage of AI agent platforms in 2026. A business can spin up an agent quickly, connect a few apps, and automate a basic workflow without building an entire AI stack from the ground up.

The catch is that a successful first demo is not the same thing as a production-ready system. What works in a five-minute walkthrough often starts to strain the moment real business complexity shows up.

An SMB usually discovers, a little further in, that its workflow actually depends on proprietary business rules, internal databases, multiple APIs, custom approval logic, customer-specific context, human escalation, and permissions with auditability. A generic agent rarely accounts for all of that.

The question is not whether an off-the-shelf AI agent can automate something. It is whether it can automate the way your business actually works.

Off-the-Shelf AI Agents vs. Custom AI Agents: What's the Difference?

First, a misconception worth clearing up. Custom does not mean building an AI model or an agent framework from zero.

In 2026, custom AI agent development usually means combining existing models, agent SDKs, databases, APIs, and orchestration frameworks into a system designed around one company's workflow. The engineering work is increasingly about integration, guardrails, and evaluation rather than reinventing the underlying AI infrastructure.

The trade-off comes down to one line. Off-the-shelf platforms optimize for speed to first deployment. Custom agents optimize for fit with the workflow.

Why SMBs Are Outgrowing "Plug-and-Play" AI

This is less about the shortcomings of any single platform and more about a natural progression that many SMBs go through as their needs mature.

1. The workflow isn't standard

An accounting firm, a logistics company, and a SaaS business may all say they want a "customer support agent," but the actual workflows behind that phrase are completely different.

2. Existing systems don't always fit the platform

SMBs often run a mix of CRM, ERP, spreadsheets, internal databases, proprietary applications, and SaaS tools, and the agent has to work coherently across all of them.

3. Business logic becomes more important

Simple instructions eventually turn into real conditional chains, where if something happens the agent checks one thing, compares another, requests an approval, updates a record, and notifies someone.

4. AI needs business context

Generic knowledge is not enough once the agent has to understand internal policies, customer history, or proprietary processes.

5. The agent becomes part of the product

Once customers or employees rely on it, the company needs far more control over how it behaves, how it fails, and how it evolves.

The Five Reasons SMBs Choose Custom AI Agent Development

This is the heart of the matter, so each reason is worth looking at on its own terms.

1. Custom Workflows, Not Generic Automation

Off-the-shelf platforms generally work best when the workflow fits the capabilities they have already designed. Step outside those capabilities and you start bending your process to suit the tool.

Custom agents let developers model the actual process instead, moving through trigger, understanding, retrieval, decision, action, verification, and escalation in whatever shape the business needs. The agent follows business-specific logic rather than forcing the business to redesign itself around the platform.

2. Deep Integration With Existing Systems

A useful SMB agent often has to move through several systems in a single workflow. It might read from the CRM, pull records from a customer database, check the ERP or inventory system, call an internal API, consult a knowledge base, and finally update a notification or ticketing system.

The value here is not the raw number of integrations. It is whether the agent can use those systems coherently during one continuous workflow, rather than treating each as a disconnected lookup.

3. Proprietary Data Becomes an Advantage

Generic AI knows general information. That is useful, but it is also available to everyone else.

A custom agent can reason over the things that are specific to your company:

SOPs and product documentation
Customer history and contracts
Internal policies and operational data

This is exactly where RAG, structured data retrieval, and context engineering become important. The competitive advantage often is not the model. It is the context the model can reliably use.

4. Greater Control Over Permissions and Actions

Consider an agent that can read customer data, issue refunds, modify account information, and send emails. Should it really have unrestricted access to all four? Almost certainly not.

A custom architecture lets developers define the boundaries precisely:

Which tools the agent can access
Which actions require approval
Which users can trigger actions
Which data the agent can retrieve
What gets logged

This matters more and more as agents move from answering questions to taking actions, which is why identity, access control, and auditability sit at the center of serious agent architecture.

5. The Agent Can Evolve With the Business

An SMB's processes rarely stay still. A new CRM, a new pricing model, a new approval process, a new product line, or a new compliance requirement can all land within a single year.

A custom agent can be engineered around an architecture that evolves with those changes, rather than waiting on a platform's roadmap to catch up to where the business already is.

The Hidden Cost of "Cheap" AI Platforms

It is tempting to treat the monthly subscription as the price of the platform. In practice, the real cost is spread across several areas that do not appear on the pricing page.

A platform can look cheaper at the start and still become expensive once every additional workflow demands workarounds, premium connectors, or platform-specific logic.

To be fair, custom development is not automatically cheaper either. It carries real upfront engineering cost, and pretending otherwise would be dishonest. The point is not that custom always wins on price, but that the true comparison is total cost over time, not the headline subscription.

When Off-the-Shelf Platforms Actually Make More Sense

Custom is not always the answer, and it would be misleading to suggest otherwise. Off-the-shelf platforms are a genuinely sensible choice in plenty of situations, including when the workflow is simple, the integrations are already supported, the risk is low, the business logic is straightforward, speed matters more than customization, the agent is mainly internal, and the process is unlikely to change much.

The right question is not "Build or buy?" It is "Where does the workflow stop fitting the product?"

## A Better Approach: Start With One Workflow

Rather than debating the whole strategy at once, the most productive move is to prove the idea on a single workflow. Here is a practical six-step way to do that.

1. Pick one workflow:

Do not start with "let's build an AI employee." Start with something concrete, like "let's automate invoice reconciliation."

2. Map the current process:

Document the trigger, the inputs, the decisions, the actions, the output, and the exceptions, in that order.

3. Identify system dependencies:

List every API, database, CRM, ERP, knowledge source, and human approval the workflow touches.

4. Define agent permissions:

Decide exactly what the agent can read, decide, write, and execute, and where the limits sit.

5. Build evaluation scenarios:

Test normal cases, missing information, incorrect inputs, edge cases, tool failures, and ambiguous requests.

6. Measure the outcome:

Track task completion, error rate, human escalation, latency, and cost per task.

Done this way, you learn whether custom development is worth it in one workflow before committing to it across the business.

What a Custom SMB AI Agent Stack Can Look Like

It helps to picture the stack as a path a request travels through. A user or business event reaches the agent interface, which hands off to an orchestration layer, which calls an LLM or reasoning model, draws on a context and retrieval layer, invokes tools and APIs, connects to the underlying business systems, passes through validation and human approval, and finally produces an action.

In practice, that stack might combine model providers, agent frameworks, RAG and vector search, structured databases, REST APIs, MCP and tool interfaces, authentication, observability, and evaluation pipelines. None of these has to be built from scratch.

The developer takeaway is straightforward. Custom development is less about building every component yourself and more about choosing the right components and composing them around the workflow.

The Production Problems You Don't See in the Demo

The gap between a prototype and a production system is where most of the real work lives, and it is easy to underestimate from a polished demo.

A two-hour agent demo can prove that something is possible. It cannot prove that the system is reliable enough to run a business workflow unattended, and that distinction is the whole game when moving a custom agent from prototype into production.

Custom Doesn't Mean "Build Everything From Scratch"

The myth: custom AI means training your own foundation model and maintaining an enormous AI infrastructure stack.

The reality: a custom agent can lean on proven components, combining open-source frameworks, commercial models, APIs, vector databases, cloud infrastructure, and existing enterprise systems.

The customization does not happen at the model layer. It happens in workflow design, tool selection, context management, business rules, permissions, evaluation, user experience, and integration architecture. In other words, the modern approach to custom AI models vs off-the-shelf is largely about composition, not reinventing foundational AI technology.

A Practical Build-vs-Buy Checklist for SMBs

Before committing either way, it helps to answer five honest questions.

1. Does the workflow differentiate our business? If yes, customization may matter.

2. Does the agent need deep access to internal systems? If yes, integration architecture becomes critical.

3. What happens if the agent makes a mistake? Higher-impact actions require stronger controls.

4. Will the workflow change frequently? If yes, flexibility becomes more valuable.

5. Do we need ownership and control? Consider data, infrastructure, model choice, observability, and vendor dependency.

From there, a simple decision path tends to emerge. A standard workflow with low risk and supported integrations points toward starting with a platform. A custom workflow with deep integrations and meaningful business impact points toward custom development. And a mix, where some workflows are standard and others are differentiated, points toward a hybrid approach.

The Emerging SMB Model: Buy the Commodity, Build the Differentiator

The most balanced model in 2026 is not a binary choice at all. It is a deliberate split between what to buy and what to build.

SMBs do not have to choose between platforms and custom development wholesale. A practical architecture uses off-the-shelf tools for commodity automation while reserving custom engineering for the workflows that genuinely create differentiation. That is a more realistic stance than treating custom development as a universal replacement for platforms.

Final Takeaway: The Best Agent Is the One That Fits the Workflow

Off-the-shelf AI agent platforms have lowered the barrier to experimentation, and that is a real gain. SMBs can now automate tasks that used to require substantial engineering effort.

As a workflow becomes more valuable, complex, or differentiated, though, the limits of a generic platform tend to become more visible. That is usually the moment the conversation shifts toward custom work, and a capable AI agent development company can help a business decide which workflows are worth owning and which are fine to rent.

Custom AI agent development gives businesses greater control over how the agent reasons, what data it sees, which tools it can use, what actions it can take, and how failures are handled. The goal is not to build AI because custom sounds more impressive. The goal is to build when the workflow itself is worth owning.

Buy the commodity. Build the workflow that makes your business different.

Voice AI vs Chat AI: How to Choose the Right Customer Support Channel

Dextra Labs — Mon, 21 Sep 2026 04:42:47 +0000

Choosing an AI support channel is not simply a matter of deciding whether your customers would rather talk or type. That framing feels intuitive, but it skips the question that actually determines success, which is what the customer is trying to accomplish in the first place.

Voice and chat each have real strengths, and they do not overlap as much as they seem to. Voice is genuinely useful for urgency, complexity, emotion, and hands-free interactions, while chat is better for speed, asynchronous support, sharing links and screenshots, and absorbing high volumes of routine queries. The right choice follows the customer journey rather than the novelty of the technology.

The market reflects this split rather than a winner. Voice AI now handles around 19% of inbound contact-center volume in 2026, up from just 6% in 2024 according to Forrester Wave research, yet chat remains the workhorse for high-volume digital support. The best AI customer support channel is the one that matches the task, the customer context, and the level of action required, and the framework below will help you find it.

Voice AI vs Chat AI: What Is the Difference?

Before comparing the two, it helps to define each one clearly, since both get lumped together under "conversational AI" even though they behave quite differently in practice.

What Is Voice AI for Customer Support?

Voice AI uses speech recognition, language understanding, reasoning, and text-to-speech to hold a spoken conversation with a customer. Its common capabilities include:

Answering inbound calls
Identifying customer intent
Verifying customer information
Retrieving account details
Updating records
Scheduling appointments
Routing complex cases to human agents

Modern voice AI is far more than an IVR menu, but it still lives or dies on strong latency control, interruption handling, and reliable fallback logic, because a spoken conversation punishes hesitation far more than a text one does.

What Is Chat AI for Customer Service?

Chat AI operates through websites, mobile apps, messaging platforms, and customer portals, meeting customers in the digital spaces they already use. Its typical capabilities include:

Answering FAQs
Searching knowledge bases
Tracking orders or tickets
Collecting structured information
Sharing links and documents
Creating or updating support tickets
Escalating conversations to human agents

Voice AI vs Chat AI: What is the Core Difference Between Them?

The cleanest way to see the contrast is side by side:

Start With the Customer Journey, Not the AI Channel

The most reliable way to choose badly is to pick a channel first and fit the workflow to it afterward. Map the support workflow first, and the right channel tends to reveal itself.

1. Identify the Type of Customer Request

Start by sorting your support requests into categories, because different types behave very differently across channels:

Simple information requests
Transactional requests
Multi-step troubleshooting
Sensitive or high-risk issues
Emotionally charged complaints
Requests requiring human judgment

2. Map the Systems the Agent Must Access

Then get honest about what the agent has to reach in order to actually resolve the request, rather than just talk about it. Ask:

Does the agent need access to a CRM?
Can it retrieve order or account information?
Does it need to create tickets?
Must it trigger refunds, replacements, or cancellations?
Are identity verification and permissions required?
What happens when an API fails?

Example: Order Delivery Support

The same workflow can suit either channel depending on the customer's situation.

Chat AI may be the better fit when:

Customers simply want to check an order status
The agent can hand over a tracking link
Customers need to upload an image or share written details
The interaction can happen asynchronously

Voice AI may be the better fit when:

A delivery is urgent
The customer is frustrated or confused
Several details must be clarified conversationally
The customer is unable or unwilling to type

The takeaway is that the decision is not "voice or chat?" It is "which channel helps this customer complete this task with the least friction?"

When Voice AI Is the Better Customer Support Channel

There are clear situations where voice creates more value than chat, and they usually share a common thread of urgency or human nuance. This is where AI agents for customer service earn their place on the phone rather than the screen.

1. High-Urgency Support

Voice shines when the customer needs help right now:

Service outages
Travel disruptions
Medical appointment scheduling
Payment or account access issues
Time-sensitive delivery problems

2. Complex or Multi-Step Conversations

Voice reduces the friction of entering long explanations, which matters most when customers need to describe a problem in their own words rather than squeezing it into a text box.

3. Emotionally Charged Interactions

Customers often prefer speaking when they are frustrated, anxious, or dealing with something serious, and a well-built voice agent can acknowledge that emotion and route the conversation to a human when the moment calls for it.

4. Hands-Free or Accessibility-Driven Use Cases

Voice is valuable when customers are driving, working with their hands, have limited typing ability, or interact through a phone-first support environment where calling is simply the natural choice.

5. Important Voice AI Limitations

Voice is not a free win, and it carries real constraints worth planning around:

Speech recognition can fail with accents, background noise, or poor connections
Long pauses and latency make conversations feel unnatural
Customers may repeat themselves if the agent loses context
Sensitive actions require strong verification and confirmation
Voice interactions are harder to scan or review than text transcripts

When Chat AI Is the Better Customer Support Channel

Chat is the right first channel far more often than the excitement around voice suggests, especially for high-volume digital support. The economics back this up, since chat resolutions average roughly $0.41 each against about $1.18 for voice AI and $7.40 for a human agent.

1. High-Volume, Repeatable Questions

Chat excels at the steady stream of predictable queries:
Password reset instructions

Product availability
Shipping policies
Billing FAQs
Return and warranty information
Basic troubleshooting

2. Support That Requires Links, Images, or Documents

Chat is often more effective when customers need to work with visual or written material, such as when they need to:

Open a help article
Follow step-by-step instructions
Upload a screenshot
Share an order number
Review a policy
Complete a form

3. Asynchronous Customer Support

Chat lets a customer leave a message, return later, and review the conversation without repeating every detail, which fits the way people actually manage their time.

4. Lower-Friction Self-Service

Chat can be embedded directly into a website, app, or portal, letting customers start support without dialing a number or waiting in a queue, which removes a real barrier to getting help.

5. Important Chat AI Limitations

Chat has its own failure modes, and pretending otherwise leads to a frustrating deployment:

Customers may abandon conversations when responses are slow or generic
Text-only interaction can be frustrating for genuinely complex problems
Poorly designed chatbots create repetitive loops
Chat agents still need reliable integrations and human escalation
A chat interface does not automatically make the underlying agent intelligent

Voice AI vs Chat AI: Compare the Total Cost, Not Just the Tool Price

The price of the AI model is only a fraction of what each channel actually costs to run, and a fair voice agent vs chatbot comparison has to weigh the full stack rather than the tool price alone. Each channel carries a different set of expenses underneath the interface.

Voice AI Cost Factors

Telephony and carrier fees
Inbound and outbound call minutes
Speech-to-text processing
Text-to-speech generation
Real-time infrastructure
Call recording and transcription
Human transfer costs
Monitoring and quality assurance

Chat AI Cost Factors

Platform subscription
Message or conversation volume
Model and token usage
Knowledge-base indexing
Premium integrations
Agent seats and escalation workflows
Analytics and conversation storage
Support and customization fees

The Cost per Resolved Issue Matters More

Rather than comparing sticker prices, measure what it actually costs to

solve a customer's problem:

Cost per resolved interaction
Average handling time
Escalation rate
Repeat contact rate
Customer satisfaction
Revenue or retention impact

A cheaper interaction is not automatically the better one, because a low per-message cost means little if it generates repeat contacts or quietly pushes every hard case to a human agent.

Integration and Workflow Depth Matter More Than the Interface

Voice and chat are ultimately just customer-facing interfaces, and the real value of an agent comes from what it can do behind them. An agent that sounds wonderful but cannot act is still a dead end for the customer.

That value depends on integration with the systems where the work actually happens:

CRM systems
Help desks
Order management platforms
Billing and payment systems
Scheduling tools
Identity and authentication systems
Internal knowledge bases
Communication platforms

Read-Only vs Action-Oriented Agents

There is a meaningful difference between an agent that can only retrieve information and one that can actually change something.

Read-only examples:

Checking order status
Explaining a policy
Retrieving account information

Action-oriented examples:

Rescheduling an appointment
Creating a support ticket
Initiating a replacement
Updating customer information
Escalating based on risk or sentiment

Permissions and Guardrails

An agent should never receive unrestricted access to every system, no matter how capable it is, which is why an experienced custom AI agent development company designs boundaries alongside capability:

Role-based permissions
Identity verification
Approval requirements for high-risk actions
Audit logs
Tool-level access controls
Human confirmation for irreversible actions

Human Handoff Is a Requirement, Not a Backup Plan

Both voice AI and chat AI need a clearly designed escalation path from the start, because the handoff is where trust is either preserved or lost. Treating it as an afterthought shows up in exactly your worst customer moments.

When Should the AI Escalate?

Some situations should trigger a handoff almost every time:

The customer explicitly requests a human
The agent fails repeatedly
The issue involves a vulnerable customer
A refund, cancellation, or financial action is high-risk
The customer expresses severe frustration
Required information is missing
The agent detects its own uncertainty

What a Good Handoff Includes

When the agent escalates, the human should inherit the full picture rather than a blank slate:

The conversation transcript or call summary
Customer identity and account context
Actions already attempted
Relevant system errors
The reason for escalation
A recommended next step

Avoid Repeating the Customer's Story

A poor handoff forces the customer to start over, which is one of the fastest ways to turn a recoverable situation into a lost one. A well-designed handoff preserves context and makes the transition feel like one continuous conversation rather than a cold restart.

Security, Privacy, and Reliability Considerations

Each channel carries its own risks, and they are different enough that a single security checklist will miss things. It helps to look at them separately and then at what they share.

1. Voice AI considerations:

Call recording consent
Voice data retention
Caller authentication
Spoofing and impersonation risks
Background noise and misrecognition
Secure transfer to human agents

Chat AI considerations:

Sensitive information sitting in chat logs
Prompt injection through uploaded content
Unauthorized account access
Data retention and deletion
Exposure of internal knowledge
Unsafe links or generated instructions

Shared requirements: both channels should support encryption in transit and at rest, access controls, auditability, data minimization, monitoring and alerting, defined retention policies, tested fallback procedures, and clear ownership of customer data.

Security here is not a checkbox exercise, though. The controls you actually need depend on your industry, geography, the data involved, and the actions the agent is allowed to perform, so treat this as a design input rather than a form to sign at the end.

Should You Use Voice AI, Chat AI, or Both?

For many businesses the honest answer is both, arranged so that each channel does what it is best at. An omnichannel approach is less about offering more channels and more about placing them intelligently.

Use Chat as the First Layer When

Most requests are simple and repeatable
Customers need links, documents, or screenshots
The business receives high digital traffic
Customers prefer asynchronous support
The organization wants to deflect routine tickets

Use Voice as the First Layer When

Customers primarily contact support by phone
Issues are frequently urgent or complex
The service involves emotional or sensitive situations
Customers need conversational clarification
The support environment is hands-free or phone-first

Use Both When the Journey Has Multiple Stages

A layered journey often looks like this:

The customer starts in chat.
Chat AI identifies the issue and gathers basic information.
The customer requests a call, or the system detects complexity.
Voice AI continues with the existing context intact.
A human agent takes over if the issue requires real judgment.

The design principle that matters here is continuity across channels, not simply the number of channels you offer.

A Practical Framework for Choosing the Right AI Support Channel

To apply all of this to your own business, evaluate each support workflow against the factors below rather than making one company-wide decision.

Score the Workflow Before Selecting the Technology

Rather than committing across the whole company, run a small pilot on a single workflow and test it against real numbers:

Resolution rate
Escalation rate
Incorrect tool calls
Average handling time
Customer satisfaction
Cost per resolved case
Failure recovery
Context preservation during handoff

Test the Channel With Real Customer Conversations

A polished demo is not evidence, because it is built to succeed. A meaningful pilot deliberately includes the messy reality your customers will bring:

Historical customer conversations
Misspellings and incomplete information
Accents and background noise for voice
Angry or confused customers
Unexpected requests
API failures
Authentication failures
Human escalation
Multiple turns and follow-up questions

For voice, test interruption handling, latency, pronunciation, silence, and transfer quality. For chat, test context retention, response usefulness, retrieval accuracy, and the ability to avoid repetitive loops.

The line worth remembering is this. A demo shows what the system does when the conversation goes as planned, while a pilot shows how it behaves when the customer does not.

Conclusion: Choose the Channel That Helps Customers Resolve Issues

Voice AI and chat AI are not competing technologies in every situation, and treating them as rivals leads to worse decisions than treating them as tools with different jobs.

Choose voice when real-time conversation, urgency, complexity, or emotion is central to the interaction. Choose chat when customers need speed, written information, asynchronous support, or digital self-service. Use both when customers naturally move between channels during a single support journey. The final decision should rest on workflow fit, integration depth, security, cost per resolution, and the quality of your human handoff.

So start with one high-volume support workflow, test both channels where it is practical to do so, and measure completed resolutions rather than chatbot deflection alone.

What It Costs to Build an AI Agent, From POC to Production

Dextra Labs — Fri, 18 Sep 2026 16:17:26 +0000

Ask three teams what it costs to build an AI agent and you will get three completely different numbers, often an order of magnitude apart. This is not because someone is lying to you, but because the phrase "AI agent" is hiding two very different things inside the same words.

A $15K proof of concept and a $150K production system might both be described as "an AI agent," yet they are not the same purchase at all. A POC typically proves that an agent can complete one workflow using sample data. Production, on the other hand, requires a great deal more:

Real integrations
Authentication
Evaluation
Guardrails
Monitoring
Failure handling
Security
Deployment
Ongoing maintenance

The current market spread reflects exactly this gap. Published 2026 estimates run from roughly $10K to $25K for focused POCs up to $50K to $150K and beyond for production agents, with enterprise systems extending well past that depending on complexity.

The thesis worth holding onto is simple. The expensive part of an AI agent is rarely making it work once, because the real cost lives in making it work reliably every single time.

How Much Does It Cost to Build an AI Agent in 2026?

Let us put the answer on the table before explaining it, because you came here for numbers rather than suspense. The figures below are planning ranges rather than universal market prices, since scope and the very definition of "production" vary enormously between teams.

Treat these as a map rather than a quote, because published 2026 estimates vary widely and any honest vendor will narrow the range only after understanding your scope.

The distinction that explains most of the price jump is the difference between two questions:

-POC asks: Can the agent do this?
-Production asks: Can it do this reliably, securely, repeatedly, at scale, with real users and real systems?

That shift from "can it" to "can it every time" is where the money goes, and it is the single most useful lens for reading any proposal you receive.

What Does an AI Agent POC Actually Cost?

A focused POC usually lands somewhere between $10K and $30K, but the number moves depending on what you pack into it. Understanding that scope is the difference between a POC that clarifies your decision and one that quietly turns into a half-built product.

A focused POC usually includes:

One defined workflow
One primary model
Limited data
One or two integrations
A basic interface
Initial evaluation
Basic logging
Human review

It usually does not include:

Full production security
Complex multi-system integrations
Extensive monitoring
High availability
Large-scale load testing
Full compliance architecture

The difference in ambition becomes obvious with an example. A ticket-triage agent that reads support tickets, classifies them, retrieves relevant knowledge, and drafts a response is a fundamentally simpler thing than an agent that can resolve the ticket outright, update the CRM, issue a refund, and trigger downstream workflows. The first proves feasibility, while the second is a production system wearing a POC's clothing.

The principle to keep in mind is that a good POC should reduce uncertainty rather than pretend you have already built the final product. Current 2026 guidance increasingly treats the POC as a feasibility and cost-validation exercise, not a miniature production deployment, and that framing will save you a great deal of money.

Why the Jump From POC to Production Costs So Much More

This is the heart of the whole question, because the leap from a working demo to a trustworthy system is where most of the engineering budget actually goes. It concentrates in four places.

1. Integrations Become Real

POC: a mock API or a single limited connector that behaves nicely.
Production: CRM, ERP, ticketing, internal APIs, identity, and databases, all at once.

Each of those connections adds engineering, authentication, permissions, error handling, testing, and ongoing maintenance, which is why current industry estimates consistently name integrations as one of the largest cost drivers of the entire project.

2. Evaluation Becomes Continuous

POC: "it worked in the demo."
Production: "it passed 1,000+ evaluation cases and continues to pass them after every change."

That shift brings golden datasets, regression tests, edge cases, hallucination testing, tool-use evaluation, and model comparison, all maintained over time rather than run once and forgotten.

3. Guardrails Become Engineering

In a prototype, safety is mostly a matter of watching the agent closely. In production, safety becomes real engineering work:

Permission boundaries
Validation
Human approval
Fallbacks
Retry limits
Audit logs

4. Reliability Becomes a Requirement
Production means the system has to stay up and behave predictably, which brings its own layer of work:

Monitoring
Alerts
Error handling
Observability
Uptime
Incident response

The blunt way to summarize this whole section is that production cost is largely the price of everything that happens when the happy path stops working.

The 7 Biggest Factors That Drive AI Agent Development Cost

If you want to understand your own likely budget, these are the seven levers that move it the most. This section is also where learning how to build AI agents economically really comes down to knowing which of these you can simplify.

1. Workflow Complexity

One simple workflow costs less than an agent coordinating fifteen steps across multiple systems, because every additional step is another place to reason, fail, and test.

2. Number of Integrations

More tools mean more of everything that costs money:

Connectors
Permissions
Failure points
Testing

CRM, ERP, email, payment, ticketing, databases, and internal APIs each add their own version of this list.

3. Data Complexity

Clean, structured data is cheap to work with. Cost rises sharply with:

PDFs
Emails
Unstructured documents
Conflicting knowledge
Legacy databases

4. Agent Architecture

Single-agent workflows are generally simpler than the alternatives, each of which multiplies both engineering and evaluation effort:

Multi-agent systems
Planner/executor architectures
Long-running agents
Agent-to-agent coordination

5. Model Choice

Different tasks may require different models, and production architecture does not mean using the most expensive model everywhere. A routing strategy can use:

A smaller model for classification
A stronger model for reasoning
A specialized model for extraction

6. Security and Compliance

Costs climb when you need PII handling, encryption, audit trails, role-based access, HIPAA/PCI/SOC 2 controls, and data residency. Some 2026 estimates put compliance-heavy projects 15 to 25% or more above a comparable base build, though the real figure depends heavily on your scope.

7. Expected Usage

A low-volume internal agent and a customer-facing agent handling a million interactions a month have completely different operating economics, and that expected volume shapes both the architecture and the running bill.

The Hidden Cost: Running the AI Agent After Launch

Here is the part that catches teams off guard. Your build quote is only the opening line of the bill, because an agent in production keeps spending money every day it runs.

The monthly costs stack up across several categories at once:

- Model/API usage: input tokens, output tokens, tool calls, reasoning loops
- Infrastructure: compute, databases, vector storage, queues, hosting
- Observability: logs, traces, evaluation, monitoring
- Maintenance: prompt updates, knowledge updates, integration changes, model changes, regression testing
- Human oversight: review, exception handling, operations

Recent industry coverage makes exactly this point, noting that total cost of ownership extends well beyond the initial build into model usage, maintenance, monitoring, and organizational costs. The market data backs it up, with maintenance now commonly running 15 to 25% of the build cost each year and three-year TCO frequently landing at 1.5 to 2 times the original build.

A useful way to hold all of this in your head is a single formula:

3-Year AI Agent TCO = Build + Infrastructure + Model/API Usage + Monitoring + Maintenance + Human Operations

Skip any of those terms and your budget is fiction.

How Much Does It Cost to Run an AI Agent Each Month?

This deserves its own answer because operating cost is a genuinely separate question from build cost. The ranges below are planning figures rather than fixed market rates, since your actual bill depends heavily on how the agent is used.

Simple internal agent: $300–$1,500/month
Production workflow agent: $1,500–$5,000+/month
High-volume or multi-agent system: $5,000–$15,000+/month

The major variables are interaction volume, token consumption, model choice, the number of tool calls, context length, infrastructure, and monitoring requirements.

The key insight from current 2026 discussions is that development complexity and operating cost are separate variables. A technically complex agent with low usage can easily cost less to run than a simple customer-facing agent handling enormous volume, which is why you have to model both independently.

Where AI Agent Projects Usually Go Over Budget

Budget overruns tend to follow a predictable script, and recognizing the plot early is how you avoid it. Here are the five places projects most reliably slip.

1. The scope starts too broad: "Build us an AI agent for sales" is a wish, not a specification, and a vague brief invites an expensive, unfocused build.

2. Integrations are underestimated: The model is ready in a week. The ERP integration is not, and that gap is where timelines quietly double.

3. Evaluation is added too late: Testing becomes far more expensive when the entire system is already built and you are retrofitting checks onto finished code.

4. Production requirements appear after the POC: Security, SSO, logging, auditability, and uptime often surface only once everyone assumes the hard part was done.

5. The team optimizes the model instead of the workflow: The cheapest model is not useful if the architecture requires ten unnecessary calls.

The honest takeaway is that most overruns do not come from the model suddenly becoming expensive. They come from discovering too late what "production-ready" actually requires.

How to Reduce AI Agent Development Costs Without Building a Toy

Cutting cost does not have to mean cutting corners, as long as you are disciplined about where you economize. These seven habits keep a build lean without leaving you with a prototype that cannot grow up.

1. Start with one workflow. Do not build a platform before proving a single use case.

2. Use existing foundation models first. Do not fine-tune unless there is a demonstrated reason.

3. Limit integrations in the POC. Prove the core workflow before connecting everything.

4. Build evaluation early. Catch bad architecture before production engineering begins.

5. Use model routing. Do not send every task to your most expensive model.

6. Separate POC from production architecture. The POC should answer feasibility questions, while production solves reliability and scale.

7. Put a budget cap on inference. An agent stuck in a loop should never be able to generate an unlimited bill.

Build vs Buy: Is Building an AI Agent Even Worth the Cost?

Before you commit a budget, it is worth asking whether you should build at all, and the honest answer is that it depends on your workflow. This is the real substance of the build vs buy AI agents decision, and treating it seriously earns you credibility rather than costing it.

Build when:

The workflow is highly proprietary
Deep system integration is required
You need control over architecture and data
Existing products cannot handle the workflow

Buy when:

The use case is standardized
A mature product already exists
Customization requirements are limited

Partner when:

You need custom development
Internal engineering capacity is limited
Multiple enterprise systems must be integrated
You need help moving from POC to production

The point that ties it together is one many vendors will not tell you. The cheapest way to build an AI agent is sometimes not to build one at all.

A Practical AI Agent Budgeting Framework

Before you request a single vendor quote, walk through these six steps so you arrive at the conversation with a real budget rather than a hopeful guess. Each step sharpens the number and exposes the assumptions hiding inside it.

1. Define the workflow. What exactly will the agent do?
2. Count the systems. How many APIs and tools does it need?
3. Estimate volume. How many tasks or interactions per day?
4. Define autonomy. What can it read, recommend, execute, and escalate?
5. Define production requirements. Security, monitoring, compliance, uptime.
6. Estimate TCO. Work through POC → Production → Monthly Operations → Year 1 → 3-Year TCO.

Here is the test that makes this framework worth using. If a vendor gives you a single cost to build an AI agent without asking these questions first, the number is not a budget yet. It is a guess dressed up as a quote.

The Cost of an AI Agent Is the Cost of Making It Reliable

A POC proves possibility, and production proves reliability, and the gap between those two words is where most of the engineering cost quietly lives. Once you see the problem that way, the strange spread in vendor quotes stops being mysterious and starts being informative.

So do not budget only for the model or the first working demo. Budget for the integrations, the evaluation, the security, the monitoring, the inference, the maintenance, and the people responsible for keeping the whole thing reliable over time.

The right question was never "how much does it cost to build an AI agent?" The better question, the one that actually protects your budget, is "how much will it cost to make this agent reliable enough to trust with real work?"