DEV Community: Subodh Kc

Where Should Your Company Use AI? A Decision Framework

Subodh Kc — Thu, 15 Oct 2026 15:18:03 +0000

A team can fill a whiteboard with AI ideas in an hour.

Customer service chatbot. Sales assistant. Internal knowledge bot. Meeting summaries. Automated reporting. Proposal generation. An agent that handles part of operations.

The list is easy.

The decision is not.

Most of those ideas start with the technology and work backward until someone finds a problem that seems to fit. That can produce an interesting demo. It does not tell us whether anything important in the business should actually change.

I prefer to start somewhere else.

The first AI question is not what can be automated. It is what should change.

That sounds like a small distinction. In practice, it changes the entire investigation.

Instead of searching the company for places to install AI, start by looking for operating pressure. Where are customers waiting? Where does skilled capacity disappear into preparation and coordination? Where does information exist but fail to arrive when a decision has to be made? Where does work lose context at a handoff? What useful capability is not happening at all because it has historically been too expensive?

Only after that do I ask whether AI belongs there.

And even then, "AI can do it" is not enough.

A worthwhile opportunity has to survive a harder test: it has to create a meaningful change, beat the simpler alternatives, fit the nature of the work, respect the right authority boundaries, produce evidence we can measure, and become a system somebody can actually operate.

That is the difference between collecting AI ideas and making an AI decision.

Start with the business, not the model

When someone asks where a company should use AI, I do not think the best starting point is a catalog of use cases.

A better investigation begins with the operating system of the business.

Customer and Employee Wait States

A customer may be waiting for an answer. An employee may be waiting for information. A manager may be waiting for analysis. A deal may be waiting for the one person who knows how to interpret an issue.

Waiting is useful because it exposes friction without presuming a solution.

The question is not yet whether a model can answer faster. The question is why the work is waiting and what would change if it stopped.

Capacity Lost to Preparation

Repetitive work is the obvious version, but it is not the only one.

A highly paid employee may spend hours gathering information before using five minutes of actual judgment. A technical specialist may repeatedly explain the same context to different teams. A manager may spend more time preparing the decision than making it.

The opportunity may not be "replace the task." It may be to remove the preparation that keeps a person from doing the part of the job that actually requires that person.

Information Trapped Across Systems

Companies often have more information than they can use.

It sits across documents, conversations, tickets, emails, systems and individual experience. The constraint is not necessarily a lack of data. It can be the inability to bring the right evidence together when somebody needs to act.

That is a very different problem from "we need an internal chatbot."

Expensive Handoffs Between Departments

A process can work well inside each department and still fail between them.

Context gets lost. Someone re-enters information. Another person has to interpret what the first person meant. Work stops while ownership changes.

A handoff can be a technology problem. It can also be a process problem, an ownership problem, or an information design problem. We should know which one before introducing AI.

Quality Variation Beyond Judgment

Two people receive the same information and produce very different outcomes.

Sometimes that variation represents legitimate judgment. Sometimes it exposes a process that depends too heavily on memory, individual experience or inconsistent access to information.

AI may help. A clearer standard may help more.

The variation is the signal. AI is only one possible intervention.

Expertise as a Bottleneck

Some work stops because the next step depends on a specialist.

That might be a lawyer, engineer, analyst, senior operator, product expert, sales leader or simply the one person who knows how a system really works.

The opportunity is not automatically to automate the expert. It may be to let more of the organization arrive at the expert with the evidence already gathered, routine questions already resolved, and the real judgment call clearly isolated.

That protects expertise instead of pretending it is unnecessary.

Capabilities That Were Not Economical Before

This is the category I find more interesting than repetitive work.

A company may want every inbound interaction answered, every account researched, every document reviewed, every anomaly inspected, every proposal adapted, or every employee to have access to specialist-level assistance.

Historically, the economics may have made that impossible.

AI can change that boundary.

That is different from automation. It is not only doing existing work cheaper. It can make a previously unavailable capability possible.

This is where some of the more valuable opportunities hide because nobody is complaining about a broken workflow. The workflow does not exist yet.

Turn the pressure into a change

Once an operating pressure appears, I still do not call it an AI use case.

I try to describe the change first.

"Build a customer service bot" is a technology idea.

"Reduce the number of customers who wait until the next business day for a basic answer" is a business change.

"Use AI for reporting" is a technology idea.

"Give an operating leader the evidence needed for a weekly decision without spending six hours assembling it" is a business change.

"Create a knowledge assistant" is a technology idea.

"Reduce how often work stops because only one experienced employee knows where the answer is" is a business change.

The second version gives us something important that the first does not.

It gives us something to investigate and eventually measure.

The Opportunity Card

Before discussing models, vendors or architecture, capture the opportunity in a simple record:
QuestionWhat we need to knowWhat is happening today?The current workflow or operating conditionWho experiences the problem?Customer, employee, operator, manager, partnerHow often does it happen?Frequency and volumeWhat is the current baseline?Time, cost, quality, delay, capacity, variation or errorWhat should change?The desired operating outcomeWhy does the change matter?Revenue, capacity, quality, experience, risk or a new capabilityWho owns the outcome?The person accountable if it improves or does notWhat evidence is missing?Assumptions we have not proved yet
Notice what is absent.

There is no model.

There is no vendor.

There is no architecture.

There is not even an assumption that AI is the answer.

That is intentional.

AI has to beat the simpler intervention

One of the easiest mistakes in an AI program is comparing AI with the status quo.

That is the wrong comparison.

If the current process is painful and an AI prototype looks better, the project can appear compelling.

But perhaps the real alternatives are a better form, a cleaner process, an integration between two systems, a deterministic rule, ordinary software automation, or simply removing a step that should not exist.

So the first gate I apply is simple:

Can a simpler intervention solve enough of the problem?

If it can, AI has to demonstrate why the additional complexity is justified.

This matters because every AI system introduces another operating surface.

Something has to provide context. Access has to be controlled. Outputs have to be evaluated. Failure has to be handled. Costs change with usage. Models and vendors change. Somebody has to decide what happens when the system is uncertain or wrong.

That does not mean AI should be avoided.

It means complexity has to earn its way into the system.

Is the work actually shaped for AI?

If the opportunity survives the simpler-alternative test, then I look at the nature of the work.

Some problems are well suited to current AI capabilities because they involve messy language, interpretation, classification, synthesis, generation, pattern recognition, variable inputs or decisions where useful evidence is distributed across a lot of information.

Other problems are better expressed as deterministic software.

If the correct behavior can be written as:

If X happens, always do Y.

then an AI model may be unnecessary.

If the work is closer to:

Understand what this person is asking, retrieve the relevant context, apply several constraints, produce a useful response, know when confidence is inadequate, and escalate.

then the AI case becomes more interesting.

The distinction matters because "AI" is not one capability.

A useful AI system can include models, deterministic rules, retrieval, APIs, permissions, workflow logic, human approvals and conventional software.

The goal is not to find a place for a model.

The goal is to design the smallest system capable of producing the desired change.

Then ask what the AI is allowed to do

This is where many use-case frameworks become too shallow.

They ask whether an idea has value and whether it is feasible.

I think there is another question that matters just as much:

What authority are we giving the system?

An AI system that drafts something for a person to review is not the same system as one that communicates directly with a customer.

A system that recommends a refund is different from one that issues it.

A system that finds information is different from one that modifies the source.

A system that proposes an appointment is different from one that changes a calendar.

A system that summarizes a decision is different from one that makes it.

The technology may look similar. The operating consequences are not.

For any meaningful use case, I want to know:

What can the AI see?
What can it infer?
What can it say?
What systems can it reach?
What can it change?
What can it trigger?
What requires human approval?
Who is responsible when an action crosses that boundary?

This is one place where opportunity discovery intersects with work I have done on Cognitive Systems Management and the Instruction Stack Audit Framework.

The point is not to turn every opportunity conversation into a governance exercise.

It is to recognize that an AI capability eventually becomes part of a larger system of instructions, data, permissions, tools, people and decisions.

In ISAF, I use a similar principle for accountability: if behavior matters, we should be able to trace where relevant instructions entered the system and who owns the layer that influenced the outcome.

At the opportunity stage, the executive version of that question is simpler:

If we give this system authority, will we still understand who controls what?

If we cannot answer that, we do not yet understand the system we are proposing.

Failure tells us more than the happy path

I also want to know what happens when the system is wrong.

Not because every AI application is dangerous.

Because failure exposes the real architecture.

Consider a system answering a basic internal policy question.

A wrong answer might create minor confusion and be corrected quickly.

Now consider a similar model communicating contractual information to a customer, making a hiring recommendation, approving a financial action or controlling part of an operational process.

The probability of an incorrect output might be similar.

The consequence is not.

That changes the evidence, controls and human authority we need.

So I do not think an opportunity should be evaluated only on expected value.

It should also be understood through its failure path.

What happens if the system is wrong?

Can the action be reversed?

Will anyone know?

Can the system fail safely?

Can a human intervene?

Can we identify which data, instruction, tool call, rule or human decision produced the behavior?

If the answer to that last question is no, production may become difficult long before a demo reveals the problem.

If we cannot measure the change, we do not have a decision yet

Another weakness in AI opportunity discussions is vague success.

"Improve productivity."

"Enhance customer experience."

"Use AI to make employees more efficient."

These may describe the intent. They do not give us a decision threshold.

Before committing to an AI initiative, I want a baseline.

How long does this take today?

What does it cost?

How often does it fail?

How much work waits?

How much skilled capacity is being consumed?

How much variation exists between operators?

What demand goes unanswered?

Then we can define what meaningful improvement would look like.

Maybe the proposed system reduces response time from hours to minutes.

Maybe it gives ten employees access to knowledge that previously depended on two specialists.

Maybe it shortens a research process without reducing decision quality.

Maybe the expected productivity benefit disappears after we include review time.

All of those are useful outcomes because they tell us something.

A negative finding can be valuable too.

It may save an organization from scaling an idea that looked impressive in a demo but did not change the economics of the work.

A score comes later

Use-case prioritization matrices are useful.

I use structured evaluation too.

But I do not think a weighted score should be allowed to rescue an idea that fails a fundamental decision gate.

A project might have high expected value, strong executive enthusiasm and easy technical feasibility.

That still does not make it a good AI project if ordinary automation solves the problem better.

Or if the system needs access the organization cannot responsibly provide.

Or if the outcome cannot be measured.

Or if nobody will own the system once the project team leaves.

A score is most useful after qualification, when several legitimate opportunities need to be compared.

Before that, the harder questions matter more.

Current enterprise guidance already does useful work around use-case discovery, repetitive work, skill bottlenecks, ambiguity, impact-versus-effort prioritization and workflow mapping. I see this framework as a complement to that work, not a replacement for it. The additional question I want to force earlier is whether the opportunity deserves to become an AI system at all.

This is also where I depart from a common impact-versus-effort approach. Impact and effort are useful portfolio dimensions, but they answer a later question: which qualified opportunities deserve attention first? They do not tell us whether a candidate should have entered the portfolio in the first place.

The AI Opportunity Decision Map

I move an opportunity through six decisions.

1. Find the pressure

Identify what is waiting, constrained, fragmented, inconsistent, expensive, underused or currently impossible.

Do not name the technology yet.

2. Define the change

Describe the operating outcome before describing the solution.

Establish a baseline and an outcome owner.

If we cannot explain what should be different, the idea is not ready.

3. Test the simpler intervention

Ask whether process change, existing software, deterministic automation, integration or removal of unnecessary work solves enough of the problem.

AI has to beat the simpler intervention, not merely the current pain.

4. Qualify the AI fit and authority

Determine what AI contributes that matters.

Then define what information it needs, what it may do, what remains human and what failure would mean.

This is where a feature idea begins to become a system decision.

5. Decide what must be proved

List the assumptions that would change the decision if they turn out to be wrong.

Define the baseline, success threshold, failure criteria and evidence needed before a larger commitment.

The test is not "can we make a demo work?"

The test is "what evidence would justify the next decision?"

6. Test whether the system can be operated

Ask who owns:

quality
access
exceptions
cost
monitoring
vendor and model changes
human escalation
failure recovery
improvement after deployment

A project that has no operating owner is not production-ready, even if the prototype is excellent.

That sequence mirrors how I think about AI more broadly:

Research, Reframe, Prove and Decide, Architect, Mobilize, Improve.

Research the situation.

Reframe the problem before choosing the technology.

Prove what matters and make the decision.

Architect the system, including authority and failure paths.

Mobilize the people, ownership and controls around it.

Improve it from production evidence.

The idea is not to slow AI down.

It is to avoid spending speed on the wrong decision.

One example from building a voice AI system

When I worked on Kestrel Voice, "build an AI voice agent" was never a useful enough problem statement.

The technology was obvious.

The business problem was more interesting.

What should happen when a business cannot answer every customer interaction?

A voice model that can carry a conversation solves only part of that.

The useful outcome may require the system to understand why someone called, answer from approved information, collect what the business needs, book something, transfer the conversation when appropriate, preserve the result, and make sure somebody knows what happened.

Once you look at the whole operating outcome, the architecture changes.

Correct escalation can matter more than sounding perfectly human.

Business rules have to exist before they can be automated.

Cost, monitoring and failure recovery become product features instead of afterthoughts.

That experience reinforced something I now use as a general test:

The use case is not the AI capability. The use case is the operating change the capability has to produce.

"AI voice agent" names a technology.

"Make sure valuable customer demand does not disappear when the team cannot answer" describes something worth investigating.

One produces a demo.

The other can produce an operating decision.

The same opportunity can end in different decisions

A good investigation does not have to result in "deploy AI."

I would expect opportunities to leave this process in several different states.

Explore

The pressure appears important, but important assumptions remain.

We need more evidence before deciding whether an AI intervention is worth testing.

Prove

The opportunity is strong enough to justify a bounded evidence test.

The next step should be designed around the assumptions that could kill the idea.

Pursue

The opportunity has enough evidence and operating clarity to move into solution design, vendor selection or architecture.

This does not mean "build immediately." It means the problem has earned the next commitment.

Not AI

The problem matters, but another intervention is better.

That can be a successful result.

Wait

AI may fit later, but capability, economics, data, authority, ownership or another dependency is not ready.

A Wait decision should include a revisit trigger.

Stop

The opportunity does not justify further investment.

A company that can stop weak AI ideas is in a better position to fund the strong ones.

The one-page AI Opportunity Card

For each candidate, I would preserve the decision in a record like this:
FieldDecision purposeOperating pressureWhat is happening that deserves attention?Affected people or processWho experiences it?BaselineWhat happens today?Desired changeWhat should be different?Business consequenceWhy does the change matter?Outcome ownerWho is accountable for the result?Simpler interventionCan process, software or rules solve it first?AI-specific advantageWhat does AI contribute that matters?Required context or dataWhat must the system know or access?Authority boundaryWhat may AI recommend, communicate, decide, change or trigger?Human authorityWhat remains explicitly human?Failure pathWhat happens when it is wrong or unavailable?TraceabilityCan behavior be traced to inputs, instructions, tools and owners?Success evidenceWhat baseline and threshold would prove value?Operating ownerWho owns quality, cost, exceptions and change in production?Key assumptionsWhat remains unproved?DecisionExplore, Prove, Pursue, Not AI, Wait, or StopRevisit triggerWhat new evidence would change the decision?
That last field matters.

AI capability changes. Vendor economics change. Regulations change. Internal data improves. Integration becomes easier. A "Wait" decision today may become a strong opportunity later.

A decision framework should preserve that context instead of forcing every idea into approved or rejected.

After an opportunity survives

Finding the opportunity is only the first decision.

If it survives, a different set of questions begins.

Should we buy a product, configure something we already have, integrate a vendor capability, build a differentiated layer ourselves, or wait?

If we need a proof of concept, what exactly must it prove?

If a vendor is involved, what dependency are we accepting along with the capability?

What architecture preserves the right authority boundaries?

What must be observable once the system is live?

Those deserve separate treatment because each is a substantial decision on its own.

The important part is that we do not jump to them too early.

A company will always be able to find more places where AI could be used than it has the capacity to pursue.

The harder question is which part of the business is worth changing, whether AI materially improves the intervention, and what evidence would justify the next commitment.

If those answers are still unclear, the next step is not implementation.

It is investigation.

Companion Artifact: AI Opportunity Decision Map

The complete decision flow in one view:

FIND THE PRESSURE Waiting, capacity, information, handoffs, variation, expertise bottleneck, unserved demand, previously uneconomic capability
DEFINE THE CHANGE Current baseline to desired operating outcome to accountable owner
TEST THE SIMPLER INTERVENTION Process change? Existing software? Deterministic automation? Integration? Remove the work? If a simpler intervention solves enough of the problem, the answer is NOT AI. If not, continue.
QUALIFY AI FIT AND AUTHORITY What does AI uniquely contribute? What can it see? What can it say? What can it change? What remains human? What happens when it is wrong? Can the behavior be traced?
DEFINE THE EVIDENCE GATE Baseline, hypothesis, success threshold, failure threshold, key assumptions. Insufficient evidence means EXPLORE. Enough to test means PROVE.
OPERATING READINESS Owner, monitoring, cost, access, exceptions, escalation, change, fallback, improvement. Ready means PURSUE. Not ready but potentially viable means WAIT. Not justified means STOP.

Source and methodology notes

This article is intentionally not a list of AI use cases. Current enterprise guidance already covers common AI opportunity discovery patterns such as repetitive work, skill bottlenecks, ambiguity, use-case gathering, impact-versus-effort prioritization and workflow mapping. The framework above adds a different layer: operating pressure first, simpler alternatives before AI, explicit authority and failure boundaries, evidence gates, traceability, and production ownership.

For context on existing enterprise use-case guidance, OpenAI's Identifying and scaling AI use cases covers repetitive low-value work, skill bottlenecks, ambiguity, use-case primitives, impact-versus-effort prioritization and department workflow mapping. OpenAI's How enterprises are scaling AI emphasizes workflow design, ownership, quality before scale, governance as an enabler and protecting human judgment.

Internal methodology references: Cognitive Systems Management (CSM) supplies compatible operating principles around ownership, risk, monitoring and continuous operation. The Instruction Stack Audit Framework (ISAF) supplies compatible principles around instruction traceability, layer ownership and accountability. The AI Opportunity Decision Map is a separate executive opportunity framework authored for this article. It is not part of ISAF.

FAQ

How is this framework different from an AI use case prioritization matrix?

A prioritization matrix is useful after qualification, when several legitimate opportunities need to be compared. This framework handles the earlier question: whether a candidate should enter the portfolio at all. It applies decision gates before scoring.

Should every AI opportunity go through all six decisions?

Yes for any opportunity that would require production resources. For very small experiments, a compressed version still helps: identify the pressure, test the simpler intervention, and define what would count as evidence.

What is the most common mistake in AI use case discovery?

Starting with the technology instead of the operating pressure. Teams search the company for places to install AI rather than looking for what should change. That produces demos, not decisions.

When should a company decide Not AI for an opportunity?

When a simpler intervention solves enough of the problem. Process change, existing software, deterministic automation, or removing an unnecessary step may be the better answer. Not AI is a successful result when it saves the organization from unnecessary complexity.

How does this connect to ongoing AI advisory?

The framework identifies qualified opportunities. For recurring decisions across multiple opportunities, ongoing advisory through AI Advisor for Business provides continuous judgment support. For assessing one specific workflow, the AI Opportunity and Workflow Assessment applies this methodology to a real opportunity.

Trying to determine whether a workflow is actually worth pursuing with AI?

The AI Opportunity and Workflow Assessment examines one opportunity, the current workflow, the alternatives, the evidence and the path forward before you commit to implementation.

Originally published at subodhkc.com. Explore KestrelVoice AI Voice Operations. Follow for more on AI governance, enterprise architecture, and compliance engineering.

Is Your AI Pilot Production-Ready? 12 Checks Before You Ship

Subodh Kc — Tue, 13 Oct 2026 15:18:53 +0000

The Gate That Separates Pilot From Production

Most AI pilots die in the gap between demo and production. The demo works on clean data in a controlled environment. Production introduces adversarial inputs, multi-tenant data isolation requirements, regulatory evidence obligations, and latency budgets that the demo never faced. In my former role as an AI Strategy Core Team Lead at a Fortune 100 company, I managed 53 enterprise applications through this gap. The pilots that survived shared one trait: they passed a structured readiness gate before touching production traffic.

This is not a generic checklist. Each check below maps to a specific failure mode I have seen in production AI systems, the architectural control that prevents it, and the evidence artifact required for audit. If your pilot cannot produce these artifacts, it is not ready for production.

Check 1: Data Provenance and Lineage Verification

Every input feature must trace back to a verified source with a documented transformation pipeline. In production, RAG poisoning occurs when untrusted content enters the knowledge base without lineage controls. The readiness check requires a data lineage map showing every source, transformation step, and access control point.

Failure mode: Untrusted data enters the pipeline through a vendor API or user-generated content, poisoning model outputs.

Architectural control: Implement a data provenance layer that tags every input with source, timestamp, and trust score. Use row-level security to isolate tenant-specific data. For RAG systems, enforce a content quarantine where new documents are validated against schema and content policies before entering the vector database.

Evidence artifact: Data lineage diagram showing source-to-feature mapping, with access control points marked. See secure enterprise RAG architecture for the reference pattern.

Check 2: Model Robustness Under Adversarial Load

Stress testing in staging does not predict production behavior. Production traffic includes prompt injection attempts, malformed inputs, and edge cases that staging never sees. The readiness check requires adversarial testing results showing model behavior under attack scenarios.

Failure mode: Model produces harmful outputs when subjected to prompt injection, jailbreak attempts, or distribution shift.

Architectural control: Run the model through an adversarial test suite before deployment. Include prompt injection patterns from the OWASP Top 10 for LLMs, data exfiltration attempts, and model extraction attacks. Measure the rejection rate for each attack class. Set a threshold: if the model fails more than 2% of adversarial tests, it does not pass this gate.

Evidence artifact: Adversarial test report with pass/fail rates per attack class, signed by the security team.

Check 3: Multi-Tenant Data Isolation

If your AI system serves multiple tenants, data isolation is not optional. Cross-tenant data leakage is a regulatory violation under GDPR, HIPAA, and most enterprise contracts. The readiness check requires proof that Tenant A cannot access Tenant B's data through any query path.

Failure mode: RAG retrieval returns documents from the wrong tenant. Vector search bypasses row-level security because the embedding model operates on shared indexes.

Architectural control: Implement tenant-scoped vector indexes or enforce row-level security at the retrieval layer. Test isolation by attempting cross-tenant queries with Tenant A credentials against Tenant B data. The test must fail 100% of the time. For shared infrastructure, use namespace isolation in the vector database and verify that the retrieval filter chain enforces tenant boundaries. See implementing RAG row-level security for multi-tenant AI for the implementation pattern.

Evidence artifact: Isolation test results showing cross-tenant query rejection rate (must be 100%).

Check 4: Runtime Monitoring and Drift Detection

A model that passes pre-deployment testing can drift in production. Input distributions shift, model behavior changes, and degradation is often invisible until a user complains. The readiness check requires a deployed monitoring stack with defined alert thresholds before the pilot goes live.

Failure mode: Model quality degrades silently over weeks. The team discovers the issue when a customer reports incorrect results, by which point the damage is done.

Architectural control: Deploy three monitoring layers before production traffic: (1) input distribution monitoring to detect data drift, (2) output quality monitoring to detect response degradation, and (3) latency monitoring to detect infrastructure issues. Set alert thresholds based on baseline measurements from staging. Use runtime verification tools to capture deterministic checks on probabilistic outputs.

Evidence artifact: Monitoring dashboard URL, alert configuration with thresholds, and escalation runbook.

Check 5: Latency Budget Compliance

Every AI system has a latency budget. For voice agents, the budget is 800ms round-trip before callers notice degradation. For RAG systems, the budget is typically 2-3 seconds before users abandon the query. The readiness check requires measured latency at p50, p95, and p99 under production-equivalent load.

Failure mode: System works in staging with 10 concurrent users. In production with 500 concurrent users, latency spikes to 12 seconds and users abandon the system.

Architectural control: Run load testing at 2x expected peak traffic. Measure p50, p95, and p99 latency for the full request path: input processing, model inference, retrieval, response generation. If p99 latency exceeds the budget, optimize before deployment. Common fixes: implement response caching, use a model router to send simple queries to faster models, or pre-compute embeddings for high-frequency queries.

Evidence artifact: Load test report showing p50/p95/p99 latency at 2x peak load, with pass/fail against the latency budget.

Check 6: Fallback and Escalation Architecture

Production AI systems fail. The question is not whether, but when, and what happens when they do. The readiness check requires a documented fallback chain: what happens when the model is unavailable, when latency exceeds thresholds, and when outputs fail quality checks.

Failure mode: Model API goes down. The AI system returns errors to users. No fallback exists. The pilot is labeled a failure and the project dies.

Architectural control: Implement a three-tier fallback chain: (1) cached response for common queries, (2) deterministic rule-based response for queries the model cannot handle, (3) human escalation for high-stakes decisions. For voice agents, this means IVR fallback, SMS escalation, and human handoff. See why AI voice agents fail in production for the reference architecture.

Evidence artifact: Fallback flowchart showing the decision tree for model unavailability, latency timeout, and quality failure.

Check 7: Compliance Evidence Pipeline

Under TRAIGA, the EU AI Act, and NYC Local Law 144, you must produce evidence that your AI system was tested, monitored, and governed. A compliance dashboard that shows current status is not evidence. Evidence is an immutable, timestamped record of what happened, when, and who approved it. The readiness check requires a deployed evidence pipeline before production traffic.

Failure mode: Regulator requests evidence of bias testing. The team has a dashboard showing current bias metrics, but no historical record of when tests were run, what the results were, or who signed off. The organization faces a regulatory penalty for insufficient documentation.

Architectural control: Implement an immutable audit trail that captures every model decision, input, output, and human override. Use append-only storage (not a mutable database). Capture: timestamp, user ID, model version, input hash, output, confidence score, and any human review action. See AI governance and evidence architecture for the reference implementation.

Evidence artifact: Evidence pipeline architecture diagram, sample audit trail export, and retention policy documentation.

Check 8: Security Threat Model and Penetration Test

AI systems have unique attack surfaces that traditional security testing misses. Prompt injection, model extraction, data exfiltration through crafted queries, and adversarial inputs that bypass content filters are all AI-specific threats. The readiness check requires a threat model that covers these attack vectors and a penetration test that validates the defenses.

Failure mode: Attacker uses prompt injection to extract system prompts, customer data, or model weights. The team did not test for this because traditional pen testing does not cover AI-specific attack vectors.

Architectural control: Build a threat model using the NIST AI RMF framework. Include: prompt injection, data exfiltration, model extraction, denial of service through expensive queries, and supply chain attacks via model weights or plugins. Run a penetration test that attempts each attack class. Document the defenses and their effectiveness.

Evidence artifact: Threat model document, penetration test report with attack-by-attack results, and remediation plan for any failed tests.

Check 9: Human-in-the-Loop Decision Points

Not every AI decision should be automated. High-stakes decisions (financial, legal, healthcare) require human review. The readiness check requires a documented decision framework that specifies which outputs are automated, which require human review, and which require human approval before action.

Failure mode: AI system automates a decision that should have had human review. The decision causes harm. The organization is liable because there was no human checkpoint.

Architectural control: Define a decision matrix mapping output risk levels to review requirements: (1) low risk: automated, (2) medium risk: human review after action, (3) high risk: human approval before action. Implement the matrix as a policy layer in the application, not as a documentation note. The policy layer must be testable and auditable.

Evidence artifact: Decision matrix document, policy implementation code, and test results showing that high-risk outputs are blocked until human approval.

Check 10: Cost and Resource Budget

AI systems have variable costs that can spike unpredictably. A prompt that costs $0.01 in staging can cost $5.00 in production if a user crafts a query that triggers a long chain of tool calls. The readiness check requires a cost model showing projected monthly spend at expected traffic, and a circuit breaker that cuts off spending before it exceeds the budget.

Failure mode: Pilot goes live. A user discovers a query pattern that triggers expensive model calls. The team gets a $50,000 API bill in the first week.

Architectural control: Implement a cost monitoring layer that tracks spend per user, per query, and per model. Set a circuit breaker that rate-limits or degrades service when spend exceeds 80% of the daily budget. Build a cost model that projects monthly spend at expected traffic, and validate it against the first 48 hours of production traffic.

Evidence artifact: Cost model spreadsheet, circuit breaker configuration, and 48-hour cost validation report.

Check 11: Rollback and Version Management

When a production AI system degrades, you need to roll back to the previous version quickly. This requires versioned models, versioned prompts, versioned retrieval indexes, and a rollback procedure that takes minutes, not hours. The readiness check requires a tested rollback procedure.

Failure mode: New model version degrades output quality. The team tries to roll back but discovers the previous model version was overwritten, the prompt template was changed without version control, and the vector index was rebuilt with different embeddings. Rollback takes 6 hours instead of 6 minutes.

Architectural control: Version every component: model weights, prompt templates, retrieval indexes, and configuration. Store previous versions with retention policies. Test the rollback procedure by deploying a new version, then rolling back, and measuring the time. The rollback must complete in under 15 minutes.

Evidence artifact: Version management policy, rollback test results with measured time, and rollback runbook.

Check 12: Documentation and Operational Runbook

The team that builds the pilot will not be the team that operates it in production. The readiness check requires operational documentation that a new on-call engineer can follow to diagnose and resolve issues without contacting the original builders.

Failure mode: Pilot goes live. The original builders move to the next project. Production issue occurs at 2 AM. The on-call engineer has no documentation, no runbook, and no idea how the system works. Mean time to resolution: 4 hours.

Architectural control: Produce three documents before production: (1) system architecture diagram showing all components and data flows, (2) operational runbook with step-by-step procedures for common incidents, (3) on-call guide with escalation contacts and decision trees. Have a new engineer read the documentation and attempt to diagnose a simulated incident. If they cannot, the documentation is insufficient.

Evidence artifact: Architecture diagram, operational runbook, on-call guide, and the results of the new-engineer documentation test.

The Gate Decision

These 12 checks are not suggestions. They are the gate between pilot and production. If any check fails, the pilot does not go live. If the team cannot produce the evidence artifact for any check, the pilot does not go live.

In my Fortune 100 experience, this gate prevented at least 4 deployments that would have caused production incidents. In each case, the team believed the system was ready because the demo worked. The gate revealed what the demo could not: adversarial vulnerabilities, missing isolation controls, absent evidence pipelines, and undocumented rollback procedures.

If you are deploying high-stakes enterprise AI and need an independent systems advisor to audit your readiness before regulators or production traffic do, schedule a strategic evaluation.

FAQ

What is the most common reason AI pilots fail the readiness gate? Missing evidence pipelines. Teams build the system but do not build the audit trail. When regulators or internal auditors request evidence, there is nothing to show.

How long should the readiness check process take? For a well-architected system, 1-2 weeks. For a system that has not considered these checks, 4-8 weeks. The checks are not the bottleneck. The architectural work to satisfy them is.

Can these checks be automated? Checks 1, 3, 5, 8, and 11 can be partially automated with test suites and monitoring tools. Checks 2, 4, 6, 7, 9, 10, and 12 require human judgment and architectural review. Do not attempt to fully automate the gate.

What happens if a check fails? The pilot does not go to production. The team addresses the failure, re-runs the check, and re-presents at the gate. There is no partial pass.

Originally published at subodhkc.com. Explore KestrelVoice AI Voice Operations. Follow for more on AI governance, enterprise architecture, and compliance engineering.

Is AI-Generated Frontend Code Hurting Your SEO?

Subodh Kc — Thu, 08 Oct 2026 15:17:22 +0000

An AI assistant can now produce a complete, working Next.js page in minutes: components, API calls, navigation, interactions. It looks done. It renders in a browser. And parts of it are invisible to Google.

Code that works correctly for a human visitor is not necessarily structured correctly for a search crawler. That gap is what this whitepaper examines.

This whitepaper examines an architectural risk observed during the development of an AI-assisted frontend: important public content was being fetched or constructed in the browser rather than included in the initial HTML response. Despite conventional SEO work, performance optimization and manual indexing requests, some pages took more than two weeks to appear in Google's index.

Key finding: When titles, body content, links or structured information depend on browser-side JavaScript, crawlers must perform additional rendering work before they can fully understand the page. This creates an avoidable crawlability dependency that can delay or prevent indexing.

Published: July 29, 2026 · Author: Subodh KC · Publication type: Technical position paper

Abstract

The observation does not prove that client-side rendering was the sole cause of the delay. Google itself states that crawling may take anywhere from several days to several weeks and that requesting indexing does not guarantee immediate inclusion.

It does, however, expose a meaningful engineering risk. When titles, body content, links or structured information depend on browser-side JavaScript, crawlers must perform additional rendering work before they can fully understand the page. Google supports JavaScript rendering, but rendering is a separate processing stage and may not occur immediately. Other crawlers may have different or undocumented JavaScript capabilities.

The appropriate conclusion is not that JavaScript or Client Components are inherently harmful. The conclusion is that public, search-critical information should be available in the server-delivered HTML whenever practical.

For a complementary perspective on making websites accessible to AI agents and crawlers, see our related article on llms.txt and AI-ready website architecture.

1. The Initial Observation

The website examined in this case was developed using an AI coding copilot and a modern Next.js frontend architecture.

Standard SEO controls had been implemented, including:

Page titles and descriptions
Sitemap generation
Robots directives
Canonical URLs
Page-performance improvements
Internal links
Manual indexing requests through Google Search Console

Nevertheless, newly published pages were not consistently indexed within the expected timeframe. Some manually submitted pages took more than two weeks to be crawled or indexed.

A technical inspection found that portions of the public content were dependent on client-side execution. In several cases, the server returned a partial interface or loading state, while the meaningful content appeared only after JavaScript executed and completed an API request.

This established a plausible technical mechanism, but not standalone proof of causation. Google identifies several other reasons that pages may not be indexed, including content quality, duplicate or canonicalized pages, crawl discovery, server capacity, robots directives and general crawl prioritization.

The finding should therefore be treated as an architectural risk requiring controlled validation.

2. What Client-Side Rendering Actually Means

Client-side rendering occurs when the browser receives a limited HTML document and relies on JavaScript to fetch data or construct the meaningful page content.

A simplified client-rendered sequence is:

The server returns an HTML shell.
The browser downloads JavaScript.
React initializes.
A browser-side API request begins.
The API returns the page data.
React inserts the content into the document.

By contrast, server-rendered or statically generated pages include their primary content in the initial HTML response.

Google processes JavaScript pages through crawling, rendering and indexing stages. It initially retrieves the page, extracts available links and then sends eligible pages to its rendering system. Pages whose content is already present in the response can be understood without depending on browser-side data loading.

Important Next.js Distinction

The "use client" directive does not automatically make a page invisible to crawlers.

Next.js can pre-render Client Components into HTML for the initial page load. The actual risk appears when important content depends on:

useEffect
Browser-only APIs
SWR or TanStack Query requests without server-provided initial data
User clicks
Scroll events
Local storage or session state
Client-only dynamic imports
Suspense fallbacks that replace primary content
Metadata inserted after page load

Next.js documentation confirms that Client Components may still be pre-rendered. Therefore, counting "use client" directives is not a reliable SEO audit by itself.

The correct audit question is:

Is the page's important public content present in the HTML returned by the server?

3. Why Client-Only Content Creates SEO Risk

3.1 Rendering Introduces an Additional Dependency

Google can execute JavaScript using an evergreen Chromium renderer. However, rendering is a distinct processing stage, and a page may remain in the rendering queue until resources become available.

Server rendering does not guarantee instant indexing. It removes one dependency from the indexing process.

3.2 Interaction-Dependent Content May Never Load

Search crawlers generally do not click buttons, complete forms or scroll through a page as a human user would.

Content loaded only after a click, swipe, typing event or scroll action may therefore remain undiscovered. Google specifically recommends that primary content should not depend on user interaction to load.

3.3 Non-Crawlable Navigation Weakens Discovery

A visual control is not automatically a crawlable link.

Google generally discovers links through anchor elements containing an href attribute. A button that calls router.push() may work for users but does not provide the same explicit crawl path as an anchor or Next.js <Link> component.

For example:
// Weak discovery pattern
<button onClick={() => router.push('/products/42')}>
View product
</button>
A crawlable alternative is:
import Link from 'next/link';

<Link href="https://subodhkc.com/products">
View product
</Link>
Programmatic routing remains appropriate for application actions. It should not replace semantic links used to navigate between public pages.

3.4 Client-Managed Metadata Can Arrive Too Late

Titles, descriptions, canonical tags and Open Graph metadata should be included in the document head generated by the server.

In Next.js App Router applications, metadata should normally be defined through the metadata export, metadata files or generateMetadata. Next.js then generates the corresponding head elements.

Updating document.title inside an effect may change what a user sees in the browser, but it creates unnecessary dependence on JavaScript and may produce incomplete previews for services that inspect the initial response.

3.5 Excessive JavaScript Affects Users as Well as Crawlers

Large JavaScript bundles, extensive hydration and unnecessary frontend libraries can degrade responsiveness and loading performance.

Core Web Vitals are among the signals used by Google's ranking systems. However, poor metrics should not be described as producing an immediate or automatic ranking penalty. Google evaluates many signals, and relevance can outweigh page-experience deficiencies.

The stronger argument is that excessive client-side work creates both usability and crawl-processing costs. This is especially relevant for privacy-first AI tools that need to be discoverable.

4. Common AI-Assisted Frontend Failure Patterns

AI coding tools are not inherently hostile to SEO. However, they optimize for completing the requested feature and may select technically functional patterns without considering crawler behavior unless SEO architecture is explicitly included in the instructions.

Common risks include:

Client-Side API Fetching for Primary Content

'use client';

useEffect(() => {
fetch('/api/article')
.then(response => response.json())
.then(setArticle);
}, []);
If the initial HTML contains only a loading indicator, the article depends entirely on JavaScript rendering.

Interaction-Gated Content

{isOpen && <p>{technicalDescription}</p>}When the content is not present until a user clicks, crawlers may never receive it.

A semantic alternative for basic expandable content is:
<details>
<summary>Technical specifications</summary>
<p>{technicalDescription}</p>
</details>
The text remains in the server-delivered HTML while the browser provides native interaction.

Infinite Scroll Without Crawlable Pagination

Infinite scrolling may improve user experience, but every meaningful result should also have a stable URL and a crawlable paginated path.

JavaScript-Only Navigation

Buttons and clickable containers should not replace anchors for navigation between indexable pages.

Root-Level Client Components

Placing "use client" at a high level can expand the client-side dependency boundary and increase the JavaScript shipped to the browser. Interactive components should normally be isolated as deeply as practical.

This is an architectural guideline, not a rule that every Client Component is harmful. For a deeper understanding of how AI systems interact with website content, read our analysis of llms.txt for AI agent discovery.

5. A Server-First Next.js Pattern

Public pages should resolve their essential content before returning the response.
// app/products/[id]/page.tsx

import type { Metadata } from 'next';
import Link from 'next/link';
import { notFound } from 'next/navigation';

type Product = {
id: string;
name: string;
description: string;
};

type ProductPageProps = {
params: Promise<{ id: string }>;
};

async function getProduct(id: string): Promise<Product | null> {
const response = await fetch(
https://api.example.com/products/${encodeURIComponent(id)},
{
next: { revalidate: 3600 },
}
);

if (response.status === 404) return null;
if (!response.ok) throw new Error('Unable to retrieve product');

return response.json();
}

export async function generateMetadata(
{ params }: ProductPageProps
): Promise<Metadata> {
const { id } = await params;
const product = await getProduct(id);

if (!product) {
return {
title: 'Product not found',
robots: { index: false },
};
}

return {
title: ${product.name} | Example Company,
description: product.description.slice(0, 160),
alternates: {
canonical: /products/${product.id},
},
};
}

export default async function ProductPage({
params,
}: ProductPageProps) {
const { id } = await params;
const product = await getProduct(id);

if (!product) notFound();

return (
<main>
<article>
<h1>{product.name}</h1>
<p>{product.description}</p>
</article>

  &lt;Link href="https://subodhkc.com/products"&gt;
    View all products
  &lt;/Link&gt;
&lt;/main&gt;

);
}
Pages in the Next.js App Router are Server Components by default, while current dynamic-route parameters are provided asynchronously. Client Components can then be introduced for specific interactive functions without moving the page's primary content into the browser.

6. The 2 MB Fetch Limit

HTML size is a legitimate but uncommon risk.

As of March 2026, Google states that Googlebot fetches up to 2 MB for an individual non-PDF URL, including response headers. Content beyond that limit is not retrieved, rendered or indexed. Referenced JavaScript and CSS resources are fetched separately and receive their own limits.

This could become relevant when AI-generated code introduces:

Large inline JSON objects
Base64-encoded images
Extensive inline CSS
Embedded scripts
Oversized navigation structures
Repeated generated markup

Most normal pages will remain far below 2 MB. The limit should be presented as an edge case rather than a primary explanation for routine indexing delays.

7. AI Search and llms.txt

Server-readable content has value beyond conventional search because different crawlers and agents may possess different rendering capabilities. Google itself notes that not every bot can execute JavaScript.

However, it is not currently defensible to claim that ChatGPT Search, Claude, Perplexity or all other AI systems categorically ignore JavaScript. Their official crawler documentation identifies crawler names, access controls and robots directives, but does not establish a universal rendering model.

Similarly, llms.txt should not be presented as an established AI-search ranking standard.

The original llms.txt project describes itself as a proposal intended to provide LLM-friendly site summaries and documentation links. For a complete analysis, see our detailed guide on what llms.txt is and why every AI-ready website should publish it.

Google explicitly states that:

Google Search does not use llms.txt.
It does not improve Google rankings.
It does not improve visibility in Google's generative AI features.
It may still be maintained for other tools that independently choose to use it.

Therefore, llms.txt may be treated as optional machine-readable documentation infrastructure following the llms.txt format specification, particularly for developer documentation or controlled agent workflows. It should not replace server rendering, crawlable navigation, structured content, sitemaps or conventional SEO.

8. How to Test the Hypothesis Properly

A credible paper must distinguish personal observation from measured evidence.

The following controlled study would validate or reject the proposed relationship.

Hypothesis

Pages whose important content is absent from the initial HTML response will experience lower raw-HTML content visibility and may experience slower discovery or indexing than comparable server-rendered pages.

Test Groups

Create at least 20 matched page pairs:

Group A: Primary content server-rendered
Group B: Identical content fetched after page load through JavaScript

Keep the following approximately equal:

Content length
Topic
Publication date
URL depth
Internal-link count
Metadata
Sitemap inclusion
Canonical configuration

Measurements

For every page, record:

Publication timestamp
Initial HTML response
Percentage of primary text present in raw HTML
Number of internal links present in raw HTML
First verified Googlebot server-log request
Search Console crawl status
First detected indexing date
Rendered HTML reported by URL Inspection
JavaScript bundle size
LCP and INP measurements

Evidence Threshold

The paper should claim a causal indexing effect only if the server-rendered group consistently outperforms the client-only group while other variables remain controlled.

Without this experiment, the strongest supportable conclusion is:

Client-only rendering creates an avoidable crawlability dependency and should be treated as a technical SEO risk, particularly when public content depends on browser-side fetching or user interaction.

9. Practical Audit Checklist

Developers should test every public route using the following controls:

curl -L https://example.com/pageConfirm that the response includes:

The primary heading
Main page copy
Canonical URL
Page description
Important internal links
Structured data
Indexing directives

Additional validation should include:

Viewing the page with JavaScript disabled
Comparing raw HTML with the rendered DOM
Using Google Search Console URL Inspection
Reviewing server logs for verified crawler activity
Testing status codes and redirect chains
Confirming robots and noindex rules
Checking canonical consistency
Verifying sitemap inclusion
Testing mobile and desktop output
Confirming that pagination has stable URLs

The curl test is valuable, but it is not a complete crawler simulation. Google can render JavaScript, while raw curl cannot. The test identifies dependence on rendering; it does not by itself prove that Google cannot index the page.

For organizations building AI systems with compliance requirements, this audit should be integrated into your AI compliance governance workflow. If you need expert guidance on implementing server-first architectures, contact Subodh KC for advisory services.

Conclusion

AI-assisted development has lowered the cost of producing sophisticated web interfaces. It has not removed the need for architectural review.

The meaningful SEO risk is not AI-generated code itself, React, Next.js or the "use client" directive. The risk is allowing critical public information to exist only after JavaScript executes, an API responds or a user performs an action.

A server-first architecture provides a more resilient default:

Render primary public content on the server.
Generate metadata through framework-supported server APIs.
Use crawlable anchors for navigation.
Give paginated content stable URLs.
Isolate interactivity to focused components.
Keep HTML and JavaScript payloads controlled.
Verify what the server sends, not merely what the browser displays.

Server rendering will not guarantee indexing or rankings. It removes avoidable technical uncertainty and ensures that search engines, social preview systems and crawlers with varying capabilities receive a meaningful representation of the page.

That is the defensible lesson from the case: AI coding assistants can accelerate implementation, but they should not be allowed to make rendering architecture decisions without explicit crawlability requirements. To learn more about how AI systems discover and process web content, explore our llms.txt implementation guide and our AI architecture and governance services.

References

Google Search Central, Understand the JavaScript SEO Basics.
Google Search Central, Ask Google to Recrawl Your URLs.
Google Search Central, How Google Search Works.
Google Search Central, Inside Googlebot: Crawling, Fetching and the Bytes We Process.
Google Search Central, Optimizing for Generative AI Features.
Next.js Documentation, Server and Client Components.
Next.js Documentation, Metadata and Dynamic Routes.
OpenAI, Overview of OpenAI Crawlers.
Perplexity, Perplexity Crawlers.
llms.txt Project, The /llms.txt File Proposal.

Use the Architecture Decision Master Sheet

Originally published at subodhkc.com. Follow for more on AI governance, enterprise architecture, and compliance engineering.

Low-Latency AI Voice Pipelines with Modal, FastAPI, Postgres

Subodh Kc — Tue, 06 Oct 2026 15:17:16 +0000

The hard part of a production voice agent is not connecting speech recognition to an LLM. It is deciding what must happen immediately, what can wait for reasoning, where session state belongs, and what the system should do when one dependency stops behaving.

Updated August 8, 2026

Most AI voice tutorials end at approximately the same place.

Audio comes in.

Speech becomes text.

The text goes to a language model.

The model returns an answer.

Text-to-speech turns the answer back into audio.

It looks something like this:

Caller
↓
Speech-to-Text
↓
LLM
↓
Text-to-Speech
↓
Caller

That is enough to prove the concept.

It is not enough to run a production voice system.

The problems begin when several things happen at once.

A caller interrupts the assistant while it is speaking.

Another customer calls while the first model request is still running.

One tenant has a calendar integration and another does not.

The model provider slows down.

The database is temporarily unreachable.

An appointment API returns successfully but the acknowledgement is lost.

A configuration changes while a call is already active.

A new container starts cold.

Someone asks a question that can be answered in 30 milliseconds, but the architecture sends it through a multi-second reasoning path anyway.

The naive pipeline has only one route:

everything
↓
LLM

A production architecture needs considerably more judgment.

The architecture I have come to prefer separates the real-time media path, the orchestration path, and the persistent control plane. It also treats graceful degradation as part of the normal design rather than something added after the first outage.

The combination of FastAPI, Modal and Postgres/Supabase can work well for this kind of system.

But they should not be treated as three interchangeable pieces of a stack.

FastAPI is the application and ASGI framework.

Modal is the execution and scaling environment.

Postgres is the durable system of record.

Putting each one in the right part of the architecture matters more than simply having all three.

The architecture I would not build

A tempting first implementation looks like this:

Incoming audio
↓
FastAPI WebSocket
↓
Write audio event to Postgres
↓
Load tenant configuration from Postgres
↓
Send transcript to LLM
↓
Write LLM response to Postgres
↓
Read latest state from Postgres
↓
Call booking tool
↓
Write result
↓
Generate speech

It is logically clean.

It is operationally expensive.

Every additional round trip sits inside the user's conversational turn.

That means database latency, model latency, tool latency and network latency all begin accumulating in series.

A better mental model is:

                REAL-TIME MEDIA PLANE

Caller
↓
Audio ingress
↓
Turn / speech detection
↓
Streaming transcription
↓
In-memory session orchestrator
↓
Routing decision
↓
Response generation
↓
Speech synthesis
↓
Caller

                   CONTROL PLANE

Authentication
Tenant resolution
Agent configuration
Tool permissions
Fallback policy
Rate limits
Session policy

                  PERSISTENCE PLANE

Postgres / Supabase

Tenant configuration
Call/session record
Configuration versions
Tool execution ledger
Transcript/event history
Bookings / outcomes
Audit metadata

The key idea is simple:

Postgres should know what happened. It does not need to sit between every two milliseconds of what is happening.

That distinction alone can remove a surprising amount of unnecessary latency.

What FastAPI should do in this architecture

FastAPI is well suited to the orchestration boundary because it sits on ASGI and supports both normal HTTP endpoints and WebSockets.

FastAPI's own documentation supports WebSocket routes directly. (fastapi.tiangolo.com)

For a voice platform, I would typically use FastAPI for several different interface types:

HTTP webhooks
|
+-- call started
+-- call completed
+-- recording available
+-- provider status
+-- booking callback

WebSockets
|
+-- media stream
+-- browser voice session
+-- live transcript / state

Normal APIs
|
+-- configuration
+-- session history
+-- tools
+-- admin actions

The important word here is orchestration.

FastAPI should not become the place where every expensive computation is performed synchronously.

If your WebSocket event loop is also doing CPU-heavy processing, blocking model SDK calls and large database operations, the gateway that is supposed to coordinate the conversation can become the bottleneck itself.

That is the same architectural problem I discussed in the context of FastAPI WebSocket 1006 failures: slow upstream work and an event loop that has actually stopped making progress are different failure classes.

Where Modal fits

Modal changes the deployment model.

It can serve a FastAPI application directly as an ASGI app:

@modal.asgi_app()
def fastapi_app():
return web_app

Modal's current documentation specifically supports FastAPI and other ASGI applications, and its ASGI-hosted functions support WebSockets. (modal.com)

That matters for AI voice because the same environment can also give you access to:

autoscaled compute;
CPU and GPU functions;
secrets;
container lifecycle hooks;
concurrency controls;
warm containers;
region selection;
independently scaled functions.

But "serverless" does not mean "latency disappears."

There are several Modal behaviors I would account for explicitly before putting a voice path into production.

Cold starts are part of the latency budget

When no ready container exists for a Modal Function, Modal starts one.

Modal calls this a cold start and identifies two main sources of additional latency:

waiting for a container to become available;
initialization work inside the newly started container.

(modal.com)

For a batch job, that can be irrelevant.

For voice, the user is already waiting.

Current Modal exposes three particularly useful controls:

min_containers

buffer_containers

scaledown_window

min_containers maintains a warm baseline.

buffer_containers gives the autoscaler spare capacity while traffic is active.

scaledown_window controls how long idle containers remain alive before scaling down; the current default maximum idle period is 60 seconds if you do not change it. (modal.com)

There is a direct tradeoff:

more warm capacity
↓
lower cold-start probability
↓
higher idle cost

There is no correct universal number.

For a production voice system, I would derive it from call-arrival patterns and P95/P99 startup latency rather than setting min_containers=10 because somebody's tutorial did.

Modal WebSockets have one non-obvious consequence

Modal documents a WebSocket connection as a single Function call for the lifetime of that connection. It also supports concurrent inputs so multiple WebSockets can be handled by the same container. (modal.com)

That has an operational consequence.

Modal Functions currently have a default execution timeout of 300 seconds, although the timeout can be configured from one second up to 24 hours. (modal.com)

So if your WebSocket represents a phone call that may reasonably last longer than five minutes, I would not leave the Function timeout implicit.

I would configure it deliberately.

The exact value depends on your call/session policy.

A 30-minute maximum conversation and a 24-hour browser connection are very different products.

A production-shaped Modal + FastAPI entry point

A simplified version could look like this:

import modal

from fastapi import FastAPI, WebSocket, WebSocketDisconnect

app = modal.App('voice-orchestrator')

image = (
modal.Image.debian_slim()
.pip_install('fastapi[standard]')
)

web_app = FastAPI()

@web_app.websocket('/v1/voice')
async def voice_session(websocket: WebSocket):
await websocket.accept()

try:
    while True:
        event = await websocket.receive()

        # Decode and route the event.
        # Keep expensive/blocking work off the ASGI loop.

except WebSocketDisconnect:
    pass

@app.function(
image=image,

# Illustrative values only.
# Benchmark these for your workload.
min_containers=1,
buffer_containers=1,
scaledown_window=300,

# Explicitly choose a session lifetime appropriate
# for your product rather than depending on defaults.
timeout=3600,

)
@modal.concurrent(
max_inputs=20,
target_inputs=10,
)
@modal.asgi_app()
def api():
return web_app

The numbers above are examples, not production recommendations.

20 simultaneous WebSocket sessions may be trivial for one workload and disastrous for another.

Modal's concurrency documentation makes the underlying behavior clear: individual containers can handle multiple inputs concurrently, while demand beyond the configured level causes Modal to scale additional containers. target_inputs can also influence when that scale-out begins so traffic is less likely to wait for cold containers. (modal.com)

Benchmark the work that actually happens inside your connection.

Geography matters more than many voice-agent diagrams admit

This is easy to overlook.

Suppose:

Caller Dallas
Modal runtime Oregon
Postgres Virginia
LLM another region
TTS another region

No individual component may be "slow."

The architecture has simply created several cross-country round trips inside each conversational turn.

Modal currently routes Function inputs through us-east by default, although it supports compute-region selection and now has regional routing in beta for selected routing regions. Modal specifically calls out latency-sensitive applications and proximity to external databases as reasons to control region placement. (modal.com)

Supabase also deploys a project into a primary region and recommends choosing a region close to users for performance. (supabase.com)

For voice, I would draw the physical architecture, not only the logical one.

Then measure the actual network legs.

Model benchmarks do not include the latency you created by placing the system three regions apart.

Postgres should be the memory of the system, not its nervous system

Postgres is excellent at things a voice application needs:

tenant configuration;
durable call records;
tool execution history;
bookings;
agent versions;
permissions;
transcripts;
audit events;
analytics.

It is much less attractive as a per-audio-frame message bus.

I do not want this:

audio frame
↓
INSERT
↓
audio frame
↓
INSERT
↓
token
↓
INSERT

inside the live turn.

Instead:

Real-time state
=
in-memory session state

Durable state
=
Postgres

Then persist at useful boundaries:

session started

turn completed

tool requested

tool succeeded / failed

booking confirmed

fallback activated

session ended

You can still stream operational state elsewhere when required.

Just do not make the database a mandatory hop for every audio or token event unless you have a specific reason.

Supabase changes the Postgres operational model, but not the principle

Supabase gives each project a full Postgres database and layers products such as Auth, Realtime and Storage around it. (supabase.com)

For backend connections, Supabase currently recommends different connection modes depending on workload:

direct connections for long-lived backend clients where connectivity permits;
session-mode pooling for persistent backends on IPv4-only networks;
transaction-mode pooling for temporary/serverless-style connections;
dedicated transaction pooling as another option on applicable plans.

(supabase.com)

That distinction matters with an autoscaling Modal application.

A small number of warm, long-lived containers behaves differently from hundreds of containers appearing and disappearing during a spike.

Do not let every new container create an uncontrolled collection of Postgres connections.

Capacity-plan the database as part of the autoscaling design.

The multi-tenant problem starts before the first query

In a SaaS voice system, a call does not simply belong to "the application."

It belongs to a tenant.

So the very beginning of the session should resolve:

connection
↓
trusted channel / number / token
↓
organization
↓
voice agent
↓
approved configuration version
↓
tool permissions

I do not want arbitrary downstream code repeatedly accepting:

organization_id = request.organization_id

and trusting it because the caller supplied one.

The tenant boundary should be established from authenticated or otherwise trusted context.

Then it should follow the session.

Put tenant identity on the durable records

A minimal relational model might include:

organizations

organization_members

voice_agents
organization_id

voice_agent_versions
organization_id

phone_channels
organization_id

call_sessions
organization_id
voice_agent_version_id

conversation_turns
organization_id
call_session_id

tool_executions
organization_id
call_session_id

bookings
organization_id

For user-facing access, Supabase's Postgres Row Level Security can enforce authorization directly at the database row boundary.

Supabase recommends enabling RLS on exposed schemas and describes policies as effectively adding authorization predicates to every query. (supabase.com)

A simplified membership policy could look like:

create policy 'members can read organization calls'
on call_sessions
for select
to authenticated
using (
exists (
select 1
from organization_members m
where m.organization_id = call_sessions.organization_id
and m.user_id = auth.uid()
)
);

The exact policy needs to match your schema and indexes.

The important point is that the tenant boundary is not merely a React filter.

For a deeper treatment of multi-tenant data isolation patterns, see implementing RAG row-level security for multi-tenant AI.

One Supabase detail can undo an otherwise good tenancy design

Server-side Supabase credentials deserve special attention.

Supabase explicitly documents that a client using the service-role authorization bypasses Row Level Security. Its newer secret-key model serves the same category of trusted backend use and must not be exposed to customers. (supabase.com)

So:

RLS enabled

does not automatically mean:

every backend query is tenant-isolated

If your trusted voice backend operates with a credential that bypasses RLS, your application code now owns tenant authorization for those queries.

That is a major architectural boundary.

I prefer either:

scoped database access where RLS remains meaningful; or
an explicit trusted-backend authorization layer that validates the resolved tenant before every privileged operation.

What I would not do is assume the word "RLS" somewhere in the architecture means the multi-tenant problem is solved.

Snapshot the voice-agent configuration when the call begins

This is a small design choice that becomes extremely important later.

consider a call begins at 10:02.

At 10:04 an administrator changes:

system prompt

business hours

transfer policy

model

temperature

booking rules

Should the call that started at 10:02 immediately inherit all of those changes?

Sometimes yes.

Usually I would rather have a consistent session.

At admission:

call begins
↓
resolve tenant
↓
resolve active agent version
↓
snapshot configuration
↓
session uses version 184

Persist:

call_session.voice_agent_version_id = 184

Now when someone asks:

Why did the assistant say this?

you can answer with the configuration that actually governed that call.

Not whatever happens to be current today.

This is as useful for debugging as it is for governance.

The three-layer routing pipeline

This is the part of the architecture that made the largest conceptual difference for me.

A voice turn should not automatically enter the most expensive and least predictable reasoning path.

I think of the router in three layers.

                 USER EVENT
                     |
                     v
          +---------------------+
          | 1. SESSION CONTROL  |
          +----------+----------+
                     |
                not handled
                     v
          +---------------------+
          | 2. BOUNDED FAST PATH|
          +----------+----------+
                     |
                not handled
                     v
          +---------------------+
          | 3. LLM REASONING    |
          +---------------------+

This is an architecture pattern, not a feature supplied automatically by FastAPI, Modal or Postgres.

Each layer exists for a different reason.

Layer 1: deterministic session controls

Some events should never wait for an LLM.

Examples include:

WebSocket disconnected

call ended

DTMF event

hard timeout reached

authentication failed

session cancelled

explicit tool acknowledgement

provider status event

rate limit exceeded

These are state-machine events.

They belong in deterministic application logic.

For example:

if event.type == 'disconnect':
await close_session()
return

if event.type == 'dtmf' and event.digit == '0':
await request_human_transfer()
return

if session.deadline_expired():
await initiate_timeout_flow()
return

One qualification matters here.

A caller saying:

"Stop."

is not automatically deterministic if you are relying on speech recognition to detect the word.

STT itself may be probabilistic.

The deterministic piece starts after the event has been classified into a trusted session command.

That sounds pedantic until a voice agent hangs up because it hallucinated the word "goodbye."

Layer 2: the bounded fast path

A large fraction of customer interactions do not require general reasoning.

Examples might include:

What time do you close?

What is the address?

Do you service this ZIP code?

Do you have my appointment?

Can I leave a callback number?

What is the status of my booking?

Some of those can be answered from authoritative structured data.

A good fast path might use:

known command
↓
validated lookup
↓
templated response

or:

retrieval
↓
high-confidence approved content
↓
short response

instead of:

construct massive context
↓
send to strongest reasoning model
↓
wait

The goal is not to avoid LLMs.

It is to avoid paying reasoning latency for a problem that does not require reasoning.

Fast path does not mean "hard-code every conversation"

There is an easy mistake to make here.

A developer hears "deterministic fast path" and creates:

if 'hours' in transcript:
return BUSINESS_HOURS

That becomes fragile almost immediately.

Natural language still needs interpretation.

The architecture should distinguish:

bounded answer source

from:

perfectly deterministic intent recognition

The source of truth can be deterministic while the route into it uses a classifier, a small model or semantic matching.

For higher-impact actions, require stronger evidence.

For example:

"What time do you close?"
↓
low-risk FAQ fast path

is very different from:

"Cancel every appointment I have."
↓
identity + explicit confirmation
↓
authorized tool

Latency optimization must not erase authorization.

Layer 3: real-time LLM reasoning

The general reasoning path is where the LLM earns its cost.

This is the route for turns requiring:

synthesis across several facts;
ambiguous conversational context;
multi-step planning;
tool selection;
qualification;
complex natural-language explanation.

The LLM should receive enough context to work.

Not every object the company has ever stored.

A good request envelope might contain:

tenant policy

agent configuration version

conversation state

bounded relevant history

retrieved knowledge

allowed tools

authorization context

deadline

fallback policy

Notice what is in that list:

deadline

The reasoning engine does not own unlimited conversational time simply because the WebSocket is still open.

A routing function should return more than an answer

Conceptually:

from dataclasses import dataclass
from enum import Enum

class Route(Enum):
CONTROL = 'control'
FAST = 'fast'
REASONING = 'reasoning'

@dataclass
class RouteDecision:
route: Route
reason: str
confidence: float | None = None

async def choose_route(event, session) -> RouteDecision:

control = match_session_control(event, session)

if control:
    return RouteDecision(
        route=Route.CONTROL,
        reason=control,
    )

fast_match = await find_bounded_answer(
    event=event,
    session=session,
)

if fast_match.is_acceptable:
    return RouteDecision(
        route=Route.FAST,
        reason=fast_match.source,
        confidence=fast_match.confidence,
    )

return RouteDecision(
    route=Route.REASONING,
    reason=&#39;requires_general_reasoning&#39;,
)

The important thing is not this exact Python.

It is making the route observable.

Later you can measure:

What percentage of turns use the fast path?

How often does the fast path fall back?

Which route produces poor outcomes?

How much latency does each route add?

Without an explicit route decision, "the AI was slow" is almost impossible to decompose.

Do not put Postgres in the routing hot loop unless the query is necessary

Suppose the fast path requires:

business opening hours

Do you need to query Postgres every time the caller asks?

Probably not.

You can load the tenant configuration when the session begins:

session started
↓
load immutable config snapshot
↓
keep in memory
↓
serve repeated bounded reads

For data that genuinely changes during the call, query the authoritative source.

The rule I use is:

cache configuration; verify transactions.

Business hours can be cached.

Whether a specific appointment slot is still available should probably come from the scheduling authority.

Those are different consistency requirements.

The most dangerous latency optimization is caching the wrong truth

consider this:

10:00:00
availability = 2 PM

10:00:05
another customer books 2 PM

10:00:08
voice agent uses cached availability

10:00:10
"You're booked for 2 PM."

Fast.

Wrong.

For transactional actions, the authoritative tool must win.

The LLM can suggest:

2 PM looks available

but your system should only say:

You're booked for 2 PM

after the authoritative booking operation succeeds.

That distinction is not about latency.

It is about truth.

Side effects need idempotency

Voice systems are full of retries.

Networks fail.

Providers retry webhooks.

Users repeat themselves.

A model retries a tool.

The booking succeeds, but the acknowledgement times out.

Now the orchestrator thinks it failed.

If you blindly retry:

book 2 PM

you may create two appointments.

For side-effecting tools, I want:

session_id
turn_id
tool_name
idempotency_key
request_hash
status
external_reference

persisted in a durable execution ledger.

Conceptually:

create table tool_executions (
id uuid primary key,
organization_id uuid not null,
call_session_id uuid not null,
idempotency_key text not null,
tool_name text not null,
status text not null,
external_reference text,
created_at timestamptz not null default now(),

unique (organization_id, idempotency_key)

);

Now a retry can ask:

Did this logical operation already execute?

before reproducing the side effect.

Graceful degradation without idempotency can create a more serious incident than the one you were trying to survive.

Four levels of graceful degradation

A production voice system should have an answer to:

What happens when the LLM is unavailable?

"Return HTTP 500" is not a conversational strategy.

I prefer explicit degraded modes.

Normal operation might look like:

NORMAL

Primary reasoning
RAG
Tools
Full conversation policy

If that path fails, move through controlled levels.

Degradation Level 1: fail over the reasoning provider

The primary model is unavailable or exceeds its deadline.

If your risk, privacy and commercial requirements permit it:

Primary model
X
↓
Secondary approved model/provider

The important word is approved.

A fallback provider should not suddenly receive data your organization deliberately prohibited from leaving another boundary.

The secondary path needs its own:

model configuration
tool policy
data policy
timeout
evaluation history

Provider redundancy is useful only if the backup path is genuinely operational.

Degradation Level 2: remove free-form reasoning

If general model generation is not trustworthy or available:

Reasoning unavailable
↓
Bounded knowledge + authoritative tools

The assistant may still be able to:

report business hours;
provide a verified address;
check an appointment;
create a callback request;
route a call;
perform an explicitly authorized deterministic workflow.

It should stop pretending it can handle open-ended questions.

A user message can be honest:

"I'm having trouble with some of my answering capabilities right now, but I can still check your appointment or take a callback request."

That is better than generating confident nonsense through a damaged reasoning path.

Degradation Level 3: capture and route

If even the bounded answer path is unavailable:

Capture:
name
number
reason
urgency
preferred callback time

Then:

transfer
OR
queue for callback
OR
create service request

This is not a glamorous AI experience.

It can still preserve the business outcome.

For many service businesses:

accurately capture lead

is preferable to:

keep talking until something breaks

Degradation Level 4: fail safely

Sometimes there is no trustworthy action remaining.

Database unavailable.

Tool authorization cannot be verified.

Telephony state is inconsistent.

Session integrity is uncertain.

At that point:

do not guess
do not retry destructive tools
do not fabricate confirmation

Escalate, offer a safe alternate contact path, or terminate the automated interaction cleanly.

A production system needs the ability to say:

"I'm unable to complete that action right now."

Failure disclosure is part of reliability.

The degradation ladder should be a state machine

I would not scatter fallback behavior across dozens of exception handlers.

Represent it:

             NORMAL
                |
    primary path unavailable
                v
          DEGRADED_1
       secondary model
                |
          unavailable
                v
          DEGRADED_2
    bounded answers / tools
                |
          unavailable
                v
          DEGRADED_3
      capture + route
                |
          unsafe/unavailable
                v
          DEGRADED_4
          fail safely

Persist the transitions:

session_id
from_mode
to_mode
reason
dependency
timestamp

Now a reliability report can answer:

How often do callers experience degraded mode?

instead of simply:

Uptime was 99.98%.

Those are different measures.

Not every failure should trigger model failover

This is another place where broad except Exception logic becomes dangerous.

If the primary LLM returns:

401 Unauthorized

do not automatically send the user's content to another provider.

If the request was blocked because:

tenant is suspended

a model failover is wrong.

If the tool failed because:

authorization denied

a retry against another tool is wrong.

Fallback logic should understand failure classes.

For example:

TIMEOUT
-> eligible for provider failover

PROVIDER_5XX
-> perhaps eligible

RATE_LIMIT
-> eligible if fallback approved

AUTH_FAILURE
-> do not reroute

POLICY_BLOCK
-> do not reroute

INVALID_TOOL_REQUEST
-> repair or reject

TENANT_NOT_FOUND
-> terminate

DATA_BOUNDARY_VIOLATION
-> fail closed

Graceful degradation should preserve policy.

Not bypass it.

The latency budget should be measured per stage

One metric matters more than "LLM latency":

How long from the user's end of speech until useful response audio begins?

That is the conversational latency the user experiences.

Break it down:

Instrument each boundary.

I would record at least:

speech_end_at

transcript_ready_at

route_selected_at

llm_requested_at

llm_first_event_at

tool_started_at

tool_completed_at

tts_requested_at

tts_first_audio_at

response_playback_at

Now:

"The model felt slow."

can turn into:

STT 180 ms
routing 22 ms
DB 310 ms
LLM 890 ms
tool 2300 ms
TTS 190 ms

The slowest thing may not be the model.

Measure P95 and P99, not just the demo

A prototype tells you:

this worked once

Production needs to tell you:

this usually works within the expected latency

I would monitor:

end-of-speech to first response audio;
model time to first event/token;
TTS time to first audio;
tool latency;
database query latency;
event-loop lag;
cold-start frequency and duration;
routing distribution;
fallback activation rate;
abandoned generation rate;
WebSocket abnormal-close rate;
transfer success;
booking success;
duplicate side-effect attempts.

For multi-tenant systems I would add one metric with a very different target:

cross-tenant data exposure

Target:

zero

A fast voice assistant with weak tenant isolation is not production-ready.

Cold-start latency needs to be visible as its own metric

Averages hide it.

consider:

95 calls:
700 ms response startup

5 calls:
4.5 sec response startup

Your average may still look respectable.

Those five callers experience a different product.

Log:

container_warm: true / false

or the closest runtime evidence you can derive, along with Modal task and region metadata.

Modal exposes runtime information such as MODAL_REGION, MODAL_TASK_ID, image ID and cloud provider in the container environment. (modal.com)

That gives you enough context to investigate whether a latency anomaly clusters around:

cold container
region
image revision
deployment

instead of guessing.

Keep model initialization out of the first user turn

Modal supports @modal.enter lifecycle handlers for one-time initialization when a container starts. It specifically calls out model loading and expensive package initialization as uses. (modal.com)

For an inference class:

@app.cls(...)
class ModelRuntime:

@modal.enter()
def load(self):
    self.model = load_model()

@modal.method()
async def generate(self, request):
    ...

That still contributes to cold-start time when the container first appears.

But it avoids accidentally reloading the same model for every user turn.

For latency-critical workloads, combine:

container initialization
+
warm capacity
+
startup profiling

rather than optimizing only the inference call.

Separate the gateway from GPU inference when the workloads scale differently

The FastAPI gateway and the model do not necessarily belong in the same Modal Function.

They have different scaling characteristics.

The gateway may need:

many lightweight concurrent connections

while a local model may need:

few expensive GPU containers

That suggests:

          FASTAPI / ASGI
      lightweight orchestration
              |
        async remote call
              |
              v
         GPU FUNCTION
      inference / reasoning

Modal Functions scale independently, and its asynchronous API allows a coroutine to invoke remote functions without blocking the async caller. (modal.com)

This separation is especially useful if:

you self-host a model;
GPU capacity is expensive;
media connections outnumber simultaneous generations;
the inference layer needs a different concurrency policy.

Do not put a GPU behind every open socket if only a fraction of those sockets are actively generating.

The database and the gateway may need a different topology too

A Modal container can be ephemeral.

A Postgres connection is stateful.

Those facts can collide.

If each scale-up creates:

50 containers
x

20 DB connections

1,000 connections

the autoscaling tier has transferred its problem into Postgres.

Use a bounded application pool and choose Supabase's direct/pooler mode deliberately.

For bursty serverless-style connections, Supabase specifically recommends transaction pooling. For long-lived backend processes, direct or session-mode connections may make more sense depending on network constraints and connection budgets. (supabase.com)

Autoscaling compute and fixed-capacity databases have to be designed together.

Supabase Realtime can help dashboards, but I would not put it in the audio path

Suppose an operator dashboard needs to show:

call connected

AI speaking

appointment booked

transfer requested

call ended

Supabase Realtime can be useful there.

Supabase currently recommends Broadcast as the more scalable and secure approach for many realtime database-change scenarios, while Postgres Changes provides a simpler model with different scaling characteristics. (supabase.com)

That is a good control-plane use case.

I would not turn this into:

audio
↓
Postgres
↓
Realtime
↓
voice runtime

The operator's dashboard and the caller's audio stream have very different latency requirements.

Do not force them through the same path simply because both are "real-time."

Voice session state should have an owner

A call may have:

tenant
agent version
caller
conversation history
current speaker
active generation
current tool
pending confirmation
degradation level
transfer state
deadline

Where does that state live?

If the answer is:

some of it in Redis
some in Postgres
some in a Modal global
some in the WebSocket handler
some in the LLM messages

you will eventually get state disagreements.

For each state field, define:

authoritative owner
durability
lifetime
recovery behavior

For example:

State
Owner
Durability

Active audio buffer
session runtime
ephemeral

Current speaking state
session runtime
ephemeral

Agent configuration version
Postgres to snapshot
durable

Booking
scheduling system
authoritative external

Tool attempt
Postgres ledger
durable

Transcript
session + durable sink
eventually durable

Degradation mode
runtime + event log
recoverable

Call outcome
Postgres
durable

This is not glamorous architecture work.

It prevents extraordinarily confusing failures.

Barge-in is a state transition, not simply another prompt

A live voice user will interrupt.

If the system is speaking and new speech begins, the orchestration path may need to:

detect new user turn

cancel/stop TTS playback

possibly cancel generation

invalidate queued audio

preserve relevant context

begin listening

resume reasoning

If you treat the interrupt as:

append new text to chat history

you can end up playing stale audio after the user has already changed the conversation.

That is why I think of real-time voice more as a distributed state machine than a chatbot with a microphone attached.

Cancellation needs to cross every layer

Suppose the user interrupts while an expensive reasoning request is running.

You stop TTS.

Good.

But what happens to:

LLM generation?

tool execution?

database operation?

queued audio?

background task?

If they all continue, the user experiences one conversation while the backend continues another.

The correct cancellation boundary depends on whether the action is reversible.

For generation:

cancel if supported

For a tool that may have already committed:

do not assume cancellation means rollback

That is exactly why tool execution needs idempotency and durable state.

The voice stack should have two clocks

I like separating:

TURN DEADLINE

from:

SESSION LIFETIME

A call may last 20 minutes.

One reasoning turn should not automatically be permitted to consume those 20 minutes.

The turn has:

STT budget
routing budget
reasoning budget
tool budget
TTS budget

The session has:

overall call lifetime
idle policy
authentication lifetime
resource limits

If those clocks are not distinct, a stuck turn can effectively own the entire connection.

Avoid one global timeout

This:

TIMEOUT = 30

is attractive.

It is usually ambiguous.

Thirty seconds for what?

database query?

LLM generation?

tool call?

session idle?

WebSocket heartbeat?

HTTP request?

Modal Function?

Each timeout should answer a specific operational question.

For example:

MODEL_FIRST_EVENT_DEADLINE

TOOL_EXECUTION_DEADLINE

DATABASE_QUERY_DEADLINE

SESSION_IDLE_DEADLINE

CALL_MAX_DURATION

Once a timeout has a name, its fallback behavior becomes easier to design.

A production routing envelope

I would want every turn to carry something close to:

@dataclass
class TurnContext:
organization_id: str
session_id: str

agent_version_id: str

allowed_tools: set[str]

degradation_mode: str

turn_deadline: float

request_id: str

trace_id: str

Then:

reasoning
retrieval
tool calls
persistence

all receive the same context.

That avoids rediscovering tenant identity from untrusted parameters deep inside the call path.

It also makes logs traceable.

One trace ID should cross the whole call

A voice incident becomes painful when every service uses its own identifier.

I would propagate:

organization_id
call_session_id
turn_id
trace_id

across:

telephony/media provider

FastAPI

Modal Function

STT

LLM

tool execution

TTS

Postgres

A useful event:

{
"event": "reasoning_fallback",
"organization_id": "org_...",
"session_id": "call_...",
"turn_id": "turn_...",
"trace_id": "trace_...",
"from": "primary_llm",
"to": "bounded_fast_path",
"reason": "first_event_deadline_exceeded"
}

Now an incident can be reconstructed.

Without correlation, you are reading five log systems and hoping the timestamps line up.

Do not log everything simply because voice is hard to debug

A voice system handles unusually sensitive data.

Potentially:

phone numbers

names

addresses

appointment details

health information

financial details

raw audio

transcripts

credentials spoken accidentally

Observability is important.

Unlimited logging is not.

Separate:

operational metadata

from:

conversation content

and define retention independently.

Do not put full transcripts into every exception log simply because it is convenient.

Modal Secrets belong at the runtime boundary

Modal provides Secret objects specifically for injecting credentials into Functions as environment variables rather than hard-coding them into images or source. (modal.com)

For example:

@app.function(
secrets=[
modal.Secret.from_name('voice-production'),
],
)
def runtime():
...

I would still keep secrets scoped.

One enormous secret bundle containing:

database admin
telephony
LLM provider
calendar
CRM
payments

means every function receiving it has all of those capabilities.

Prefer the smallest credentials required by each execution surface.

Dev and production should not share one mutable runtime

Modal Environments isolate applications and resources such as Secrets between environments, making them suitable for dev/prod separation. (modal.com)

That matters for voice.

A developer testing:

transfer()

should not accidentally trigger the production transfer target because both deployments shared the same secrets and state.

Environment boundaries should include:

Modal app

Secrets

database project/schema

telephony configuration

model/provider keys

tool endpoints

where practical.

"Environment" should describe an actual isolation boundary.

Not a string in a log message.

The architecture should fail locally before it fails conversationally

Here is the principle I keep returning to.

A production voice agent has enough uncertainty already.

The control plane should remove uncertainty where it can.

Before a call begins, validate:

tenant exists

agent version exists

channel belongs to tenant

configuration is valid

required credentials exist

tool policy is loadable

fallback policy exists

Then admit the session.

Do not wait until minute four of a customer conversation to discover:

there is no calendar credential

if the system could have detected that before answering the call.

The four things I would benchmark before calling the stack production-ready

Not "Does the demo work?"

These four.

Normal latency

Under expected traffic:

How long does a normal conversational turn take?

Break it down by stage.

Burst behavior

When ten calls become fifty:

Does latency degrade gradually,
or does the system fall off a cliff?

Watch Modal queueing, cold starts, external APIs and database connections.

Dependency failure

Deliberately remove:

primary LLM
database
booking API
TTS

one at a time.

Does the system enter the intended degraded state?

Tenant isolation

Attempt:

tenant A session
↓
tenant B identifier

across every privileged route.

The expected result is not:

usually blocked

It is:

blocked

every time.

A CTO should care about degradation rate as much as uptime

consider two systems.

System A:

99.99% process uptime

but:

8% of calls spend >10 seconds waiting for response

4% silently enter broken booking path

System B:

99.9% process uptime

but:

dependency failures trigger a clear fallback
calls continue in capture-and-route mode
no false booking confirmations

Which system is more reliable?

Infrastructure uptime alone cannot answer.

For real-time AI I would put these on the executive dashboard:

successful conversation rate

P95 turn latency

P99 turn latency

fallback activation rate

fallback recovery rate

tool success rate

false confirmation incidents

transfer success

abandoned calls during AI wait

cold-start exposure

tenant-isolation failures

That is closer to the actual product.

Why the standard "LLM wrapper" architecture fails

A wrapper tutorial usually assumes:

input
↓
model
↓
output

A production voice system actually behaves more like:

                 SESSION STATE
                      |
                      v

         ^                    ^

   TENANT CONTROL         DURABLE EVIDENCE
     Postgres                Postgres

The LLM is one component.

That is the architectural shift.

Where Kestrel Voice changed my view of this

Working on Kestrel Voice made this distinction much more concrete for me.

The public product spans AI phone answering, appointment booking, RAG-powered answers, transcripts, call intelligence and integrations with external business systems. (kestrelvoice.com)

Once a voice system crosses that many boundaries, "the model answered correctly" is only one success condition.

The interaction can still fail because:

the booking tool did not confirm

the media connection disappeared

the wrong tenant configuration loaded

the fallback repeated a side effect

the caller interrupted stale TTS

the database could not persist the outcome

the secondary provider violated a policy boundary

That changed the way I think about voice architecture.

The most important component is not always the model.

It is often the orchestration layer deciding what is allowed to happen next.

For a deeper look at the product that shaped this thinking, see the KestrelVoice architecture detailed look.

The real three-tier performance strategy

If I had to reduce the entire architecture to one diagram, it would be this:

                INCOMING TURN
                     |
                     v
          +---------------------+
          | CAN APP LOGIC      |
          | HANDLE IT SAFELY?  |
          +---------+----------+
                    |
             yes    |    no
             v      |     v
      CONTROL ACTION|  BOUNDED FAST PATH?
                    |        |
                    |   yes  |  no
                    |    v   |   v
                    |  FAST  | GENERAL
                    | ANSWER | REASONING
                    |        |
                    +----+---+
                         |
                         v
                     VERIFIED
                      ACTION
                         |
                         v
                        TTS

The performance insight is not:

use a smaller model.

It is:

do not invoke general reasoning when the problem has a safer, faster bounded path.

Frequently Asked Questions

Can Modal host a FastAPI voice agent?

Yes. Modal can serve ASGI applications through @modal.asgi_app(), including FastAPI, and Modal's ASGI-hosted Web Functions support WebSockets. Modal also supports input concurrency so a container can handle multiple simultaneous WebSocket connections. (modal.com)

Does Modal support WebSockets?

Yes. Modal currently supports the WebSocket protocol for @modal.asgi_app, @modal.wsgi_app and @modal.web_server functions. Modal documents each WebSocket as one Function call per connection. It currently limits individual WebSocket messages to 2 MiB and does not support RFC 8441 WebSockets over HTTP/2 or the permessage-deflate extension. (modal.com)

Should I use `modal.fastapi_endpoint` for a voice WebSocket?

For an application with actual WebSocket routes, use a FastAPI ASGI application exposed through @modal.asgi_app() rather than treating the connection like an ordinary fastapi_endpoint. Modal explicitly documents WebSocket support for its ASGI/web-server hosting modes. (modal.com)

How do I reduce Modal cold-start latency for a voice agent?

Measure first, then consider min_containers, buffer_containers, a longer scaledown_window, smaller initialization work and lifecycle initialization through @modal.enter. Modal documents these as controls for reducing queueing and container-initialization latency. They trade cost against latency. (modal.com)

What is Modal's default Function timeout?

Current Modal Functions have a default execution timeout of 300 seconds. Modal allows configured timeouts from one second to 24 hours. Since Modal documents each WebSocket connection as one Function call, long-lived voice or browser sessions should use an explicitly chosen Function timeout appropriate to the session policy. (modal.com)

Should Postgres store live voice audio frames?

Usually I would keep raw real-time media out of the transactional Postgres hot path. Postgres is better suited to durable session, configuration, transcript, tool and outcome records. Raw recordings or media are generally better handled through media/object storage designed for that payload, while Postgres stores references and metadata.

Why use Supabase/Postgres in an AI voice platform?

Postgres is useful as the durable control plane: organizations, users, voice-agent versions, sessions, tools, bookings, transcripts and audit metadata. Supabase adds Auth, Realtime and managed Postgres capabilities on top of that foundation. (supabase.com)

How should I handle multiple tenants in a voice-agent database?

Resolve tenant context from trusted authentication or channel information, include the organization identifier on tenant-owned data and enforce authorization at every access layer. With Supabase, RLS can provide row-level controls for user-facing access. Be aware that trusted service-role credentials can bypass RLS, so privileged backend paths require their own explicit authorization controls. (supabase.com)

Should every voice-agent query go through Postgres RLS?

That depends on the connection model. RLS is highly useful for user-scoped Supabase access. A trusted backend using credentials that bypass RLS must enforce tenant isolation itself or use a more constrained database role/access pattern. Do not assume RLS protects a query that is deliberately running with bypass privileges.

Should my AI voice agent send every question to the LLM?

No. Session-control events should be handled deterministically, and bounded questions can often use authoritative data or a constrained fast path. General LLM reasoning is most useful for turns that genuinely require synthesis, ambiguity resolution or planning.

What is a fast path in an AI voice architecture?

A fast path is a bounded response route that avoids full general reasoning when the answer can be obtained from an approved source or deterministic workflow. Examples include business hours, location information, appointment lookup or structured routing. The route may still use probabilistic language classification, so the confidence and risk of the action should determine when it is safe to use.

What should happen if the LLM provider goes down?

Design this before production. A useful degradation sequence is: approved secondary model/provider, bounded deterministic or retrieval/tool capabilities, capture-and-route mode, and finally safe escalation or termination when no trustworthy operation remains.

Should I automatically retry failed booking tools?

Not blindly. A booking may have succeeded even if the acknowledgement failed. Side-effecting operations should use idempotency keys and a durable execution ledger so retries can determine whether the logical operation already occurred.

How should I measure AI voice latency?

Measure the user-facing turn from end of speech to first useful response audio, then decompose it into speech recognition, routing, retrieval/database, LLM time to first event, tool execution and TTS time to first audio. Track P95 and P99 as well as averages.

Does Supabase Realtime belong in the live audio path?

I would generally use it for control-plane experiences such as dashboards, call-state notifications or database-driven updates rather than as the audio transport itself. Supabase currently recommends Broadcast for scalable database-change notification use cases, while Postgres Changes is simpler but has different scaling characteristics. (supabase.com)

Should Modal and Supabase run in the same region?

For latency-sensitive applications, keeping compute near users and data can remove avoidable network delay. Modal supports region selection and beta regional request routing, and Supabase offers multiple database deployment regions. Benchmark the actual network path rather than assuming geography is negligible. (modal.com) (supabase.com)

Production Checklist: Modal + FastAPI + Postgres AI Voice

Before calling the architecture production-ready:

[ ] Separate media, orchestration and persistence paths
[ ] Host WebSocket routes through an ASGI application
[ ] Keep blocking work off the FastAPI event loop
[ ] Measure Modal cold-start latency
[ ] Configure warm capacity from traffic data
[ ] Configure WebSocket Function timeout deliberately
[ ] Benchmark per-container WebSocket concurrency
[ ] Align compute, database and model geography
[ ] Bound Postgres connection pools
[ ] Resolve tenant before loading session configuration
[ ] Test tenant isolation across every privileged endpoint
[ ] Understand which backend credentials bypass RLS
[ ] Snapshot the agent configuration version at session start
[ ] Keep high-frequency audio/token state out of Postgres hot loops
[ ] Handle session controls before general reasoning
[ ] Create a bounded fast path for authoritative answers
[ ] Reserve general LLM reasoning for turns that need it
[ ] Verify transactional actions before confirming them to callers
[ ] Make side-effecting tools idempotent
[ ] Give each turn an explicit deadline
[ ] Give the session a separate lifetime
[ ] Implement an approved provider failover path
[ ] Implement bounded degraded operation
[ ] Implement capture-and-route fallback
[ ] Implement a fail-safe state
[ ] Persist degradation transitions and reasons
[ ] Propagate one trace ID across the full interaction
[ ] Measure end-of-speech to first response audio
[ ] Measure P95/P99, not just averages
[ ] Load-test concurrent calls
[ ] Chaos-test LLM, DB, tool and TTS failures
[ ] Verify cancellation and barge-in behavior

A low-latency voice system is not created by finding the fastest model and putting a WebSocket in front of it.

It comes from removing unnecessary work from the critical path.

Keep session state close to the live connection.

Keep durable truth in Postgres.

Keep tenant boundaries explicit.

Handle deterministic controls before reasoning.

Use the fast path when the answer is bounded.

Use the LLM when reasoning is genuinely required.

Verify side effects before telling a caller they happened.

And when the intelligence layer fails, degrade into something smaller and trustworthy rather than something equally conversational but less reliable.

That is the part most voice-agent demos do not show.

It is also the part that decides whether the architecture still works after the demo traffic becomes real traffic.

Continue Reading

Originally published at subodhkc.com. Explore KestrelVoice AI Voice Operations. Follow for more on AI governance, enterprise architecture, and compliance engineering.

Texas HB 149 vs the EU AI Act: What Engineers Must Build

Subodh Kc — Thu, 01 Oct 2026 15:18:33 +0000

This article is an engineering and governance analysis, not legal advice. Legal applicability and interpretation should be reviewed with qualified counsel.

The Engineering Problem Underneath Two Different Laws

Production AI does not stand still after a compliance checklist is completed. The model changes. A vendor replaces an underlying model. Retrieval data changes. A prompt changes. An engineer adds another tool. Someone increases an autonomy setting. A team begins using the same system for a use case that was never part of the original review.

The application that legal approved six months ago may no longer be the application running today.

That is why engineering leaders should approach laws such as the Texas Responsible Artificial Intelligence Governance Act, or TRAIGA, enacted through Texas HB 149, and the EU AI Act as specifications that eventually have to meet a running system. Not as two collections of forms.

Both laws, despite their structural differences, ultimately require the same engineering capabilities: know what AI you deployed, why it is allowed to operate, what changed, what evidence exists, and whether the system is still behaving inside its approved boundaries. For a broader framework on production AI governance, see seven layers of AI compliance.

Where the Laws Actually Stand in August 2026

Texas HB 149 took effect on January 1, 2026. TRAIGA applies broadly to persons who conduct business in Texas, provide products or services used by Texas residents, or develop or deploy AI systems in the state. (Texas Legislature Online)

The EU AI Act has followed a phased implementation. Prohibited practices and AI-literacy provisions began applying in February 2025. Governance rules and obligations for providers of general-purpose AI models began applying in August 2025. Article 50 transparency requirements became applicable on August 2, 2026. (Digital Strategy in Europe)

The EU's July 2026 AI Omnibus changed the high-risk timeline. High-risk AI systems in areas such as employment, education, biometrics, and critical infrastructure are now scheduled for their high-risk requirements from December 2, 2027. High-risk systems embedded in regulated products have until August 2, 2028. (Digital Strategy in Europe)
RequirementStatus as of Aug. 8, 2026Texas TRAIGA / HB 149*Effective since Jan. 1, 2026EU prohibited AI practicesApplicableEU AI-literacy obligationsApplicableEU GPAI provider obligationsApplicableEU Article 50 transparencyApplicable from Aug. 2, 2026*EU Annex III high-risk regimeDec. 2, 2027EU high-risk in Annex I regulated productsAug. 2, 2028
An engineering team that believes the entire EU high-risk regime went live in August 2026 may build the wrong remediation plan. Urgency should come from what actually applies, not from a headline.

Texas and Europe Are Solving Different Regulatory Problems

Calling TRAIGA the Texas version of the EU AI Act is convenient and misleading. The EU AI Act creates a broad risk-based regulatory framework with four categories: prohibited uses, high-risk systems, systems subject to transparency obligations, and minimal or no-risk systems. (Digital Strategy in Europe)

TRAIGA does not create an equivalent general-purpose risk-tier system. Texas instead combines broad applicability, specific prohibited uses, particular disclosure requirements, biometric provisions, anti-discrimination rules, government-specific restrictions, an Attorney General enforcement mechanism, a cure process, and liability protections tied in part to testing and recognized risk-management frameworks.

The first mistake in a compliance architecture would be trying to force Texas systems into an EU-style high-risk or low-risk classification and assuming the classification itself satisfies Texas law. It does not.
DimensionTexas TRAIGA / HB 149EU AI ActPrimary structureProhibitions, duties, enforcement tied to specific usesBroad risk-based frameworkEffective stateIn force since Jan. 1, 2026Phased application underwayGeneral AI risk tiersNo equivalent EU-style tier systemProhibited, high-risk, transparency, minimalGovernment AI disclosureRequired for covered governmental agenciesDepends on applicable AI Act categoryHealthcare AI disclosureSpecific Texas disclosure requirementDepends on AI Act and other applicable EU lawDiscriminationIntentional unlawful discrimination prohibitedFundamental-rights protections in high-risk regimeCure processStatutory 60-day opportunity after AG noticeNo equivalent universal 60-day cure rightMain enforcerTexas Attorney GeneralNational authorities, AI Office, EDPSMaximum penaltiesTexas statutory civil penalty rangesUp to 35M EUR or 7% for certain violations

Texas HB 149 Is Narrower Than People Assume

TRAIGA's disclosure obligation illustrates how summaries can mislead. The law specifically requires a covered governmental agency making an AI system available to interact with consumers to disclose that the consumer is interacting with AI. It also establishes disclosure requirements when AI is used in connection with covered healthcare services or treatment. (Texas Legislature Online)

That is materially different from a universal commercial chatbot disclosure requirement. The EU transparency regime is broader in different directions. As of August 2, 2026, Article 50 includes obligations around certain direct AI interactions and AI-generated or manipulated content. (Digital Strategy in Europe)

TRAIGA also does not prohibit AI bias in the abstract. The statute prohibits developing or deploying an AI system with the intent to unlawfully discriminate against a protected class. Disparate impact alone is not sufficient by itself to demonstrate intent. (Texas Legislature Online) An engineering dashboard claiming a fairness metric outside threshold equals a TRAIGA violation would be inventing a legal standard the statute does not contain.

The 60-Day Cure Period Makes Auditability Strategically Valuable

TRAIGA's cure process is one of its most operationally important provisions. If the Texas Attorney General determines that a person has violated the law, the AG must provide written notice. The AG generally may not bring an action before the 60th day after that notice. If the organization cures the violation, it must provide a written statement, supporting documentation, and information about changes made to internal policies to prevent recurrence. (Texas Legislature Online)

Read those requirements from an engineering perspective: identify the violation, remediate it, produce supporting documentation, change internal controls, and demonstrate how recurrence will be prevented. That is an incident-response workflow.

Sixty days is generous only if you already know which system is implicated, who owns it, what version is deployed, which model is running, what data it uses, what controls were active, what changed, and how to reproduce the behavior. If your organization has to discover all of that after receiving the notice, much of the 60-day window will be spent reconstructing its own architecture.

When investigating a complaint, the Texas AG may request information including the AI system's purpose, deployment context, training data, input categories, outputs, performance metrics, known limitations, post-deployment monitoring, user safeguards, and oversight processes. (Texas Legislature Online) That is close to a specification for the minimum AI system record you should maintain internally.

Texas Enforcement and Penalties

TRAIGA gives the Texas Attorney General primary enforcement authority and does not create a private right of action. For uncured violations, the law provides different civil-penalty ranges: $10,000 to $12,000 for each violation a court determines to be curable, $80,000 to $200,000 for each violation determined to be uncurable, and $2,000 to $40,000 per day for a continuing violation. The AG can also seek injunctive relief and recover specified costs. (Texas Legislature Online)

The EU AI Act's penalty structure is on a different scale: up to 35 million EUR or 7% of worldwide annual turnover for certain prohibited-practice violations, up to 15 million EUR or 3% for a range of other obligations, and up to 7.5 million EUR or 1% for supplying incorrect or misleading information to authorities. Lower comparative thresholds apply to SMEs. (Digital Strategy in Europe)

The numbers differ. The architectural lesson does not. Evidence matters.

consider two companies receive a Texas notice on the same day. Company A has no AI inventory, shared API keys, unversioned prompts, unknown model revisions, no deployment evidence, no red-team history, and no owner map. Company B has a system inventory, deployment history, versioned control results, known owners, model lineage, evaluation history, a change log, and documented monitoring. Both receive the same statutory period. They do not have the same amount of time. Company B spends its time correcting the problem. Company A spends its time discovering what exists. Compliance engineering converts regulatory response from archaeology into incident management.

NIST Gives Texas Organizations Something Concrete

TRAIGA contains an important connection to the NIST Artificial Intelligence Risk Management Framework: Generative AI Profile. The law's liability section addresses situations in which a defendant discovers a violation through mechanisms including feedback, adversarial or red-team testing, or an internal review process associated with substantial compliance with the current NIST Generative AI Profile or another nationally or internationally recognized AI risk-management framework. (Texas Legislature Online)

Substantial compliance with NIST does not mean every action your company takes is automatically TRAIGA-compliant. The statutory protection is more specific and should be interpreted with counsel. But Texas has explicitly connected recognized AI-risk-management practices, internal review, and testing to its liability framework. That is a significant engineering signal.

NIST does not describe the AI RMF as a checklist. The framework is organized around four functions: GOVERN, MAP, MEASURE, and MANAGE. NIST explicitly says the activities are not necessarily sequential checklist steps. Risk management is intended to be continuous across the AI lifecycle. (NIST AI Resource Center)

That makes it more useful for engineering. GOVERN defines who owns the system and which policies apply. MAP documents what the system does and who is affected. MEASURE evaluates whether identified risks are materializing. MANAGE determines which controls are required and what happens when they fail. For a deeper operational framework, see the HAIEC modular governance framework.

The EU AI Act Asks for a More Formal Lifecycle

When the relevant high-risk requirements begin applying under the updated timeline, providers of covered high-risk systems face a considerably more prescriptive structure. The Commission summarizes requirements including risk assessment and mitigation, data-quality requirements, activity logging, technical documentation, information for deployers, human oversight, robustness, cybersecurity, and accuracy. (Digital Strategy in Europe)

Providers must retain certain technical documentation for ten years, and automatically generated logs under their control must generally be kept for at least six months unless other applicable law provides differently. (EUR-Lex) The Regulation also requires providers to establish and document post-market monitoring designed to collect and analyze performance information across the system's lifetime so continued compliance can be evaluated.

Those are not merely policy-writing requirements. They create system-design consequences: logging architecture, retention policies, documentation pipelines, and monitoring systems must all be built to satisfy these obligations.

The Inventory Is the Foundation

Before building an AI compliance CI/CD gate, build an AI inventory. A minimal system record should be machine-readable, versioned, and attributable:
system_id: customer-support-assistant
owner:
business: customer_operations
technical: platform_ai
jurisdictions:

texas
european_union role: texas: deployer: true developer: false eu: provider: false deployer: true purpose: approved:
- answer_customer_questions
- retrieve_order_status prohibited_use:
autonomous_credit_decision
employment_screening model: provider: example-provider model_id: model-x revision: "2026-07-21" data: personal_data: true biometric_data: false retrieval_sources:
- approved_support_kb human_oversight: escalation_required: true monitoring: latency: enabled safety_evaluations: enabled output_sampling: enabled evidence: risk_review_id: RR-2026-184 deployment_id: DEP-8371 If the organization cannot answer which AI systems are currently affecting Texans or EU users, the rest of the compliance program is operating on assumptions. For production readiness fundamentals, see 12 production readiness checks for AI pilots.

Compliance Becomes Architecture: Moving Governance Into the Deployment Path

Most companies add AI governance alongside their CI/CD pipeline as a separate process. Legal writes a PDF. Engineering deploys code. The two systems barely interact.
AI SYSTEM RECORD
|
v
CODE --> BUILD --> CONTROL EVALUATION
|
+--> Security tests
+--> Risk tests
+--> Disclosure requirements
+--> Human-oversight controls
+--> Model/version checks
+--> Jurisdiction rules
+--> Evidence generation
|
v
RELEASE GATE
/ \
PASS FAIL
| |
v v
DEPLOY REMEDIATE
|
v
POST-DEPLOY MONITORING
|
v
VERSIONED EVIDENCE
Neither TRAIGA nor the EU AI Act mandates this exact CI/CD architecture. The benefit is that legal requirements stop existing only as prose. They become testable conditions in the deployment path.

Deterministic Gates vs Legal Judgments

Some compliance questions can become deterministic gates. Suppose a healthcare AI service in Texas requires a disclosure before the relevant service is provided. A deployment can fail if healthcare_ai = true and disclosure_enabled = false. That is deterministic. No language model needs to decide whether it feels compliant.

But not every legal question should become a Boolean. Texas's prohibition on intentionally using AI to unlawfully discrimination cannot be reduced to if fairness_score > 0.91: traiga_compliant = True. Instead, the system can verify that required governance steps were performed: approved use case, protected class review completed, fairness evaluation completed, legal review approved, model version tested. The tool proves the workflow happened. It does not pretend the mathematical test conclusively answered the legal question.

The Right Unit of Automation Is the Control, Not the Law

Instead of one giant EU_AI_ACT = PASS or TRAIGA = 92% without being able to explain the denominator, the architecture should trace a chain:
Law
|
v
Requirement
|
v
Applicability decision
|
v
Control
|
v
Technical check
|
v
Evidence
For example: EU Article 50, then AI-interaction transparency applies, then UI disclosure required, then disclosure test, then PASS, then screenshot plus test result plus build hash. The compliance conclusion remains a governance decision. The technical result becomes reproducible.

A deterministic gate for a healthcare AI disclosure in Texas might look like:
control_id: TX-AI-DISCLOSURE-HEALTHCARE

applies_when:
jurisdiction: texas
healthcare_ai: true

require:
disclosure_enabled: true
disclosure_version: approved
disclosure_event_test: passed
Now a deployment fails if healthcare_ai = true and disclosure_enabled = false. No language model needs to decide whether it feels compliant. A lawyer or governance owner determines the rule. The pipeline verifies whether the agreed technical condition exists.

The Deployment Gate I Would Build

def evaluate_release(system, deployment, controls):
failures = []
if not system.owner:
failures.append('AI system owner missing')
if not deployment.model_revision:
failures.append('Model revision not recorded')
for control in controls.required_for(system):
result = control.evaluate(system=system, deployment=deployment)
evidence.store(result)
if result.blocking and not result.passed:
failures.append(control.id)
if failures:
return {'decision': 'BLOCK', 'reasons': failures}
return {'decision': 'ALLOW', 'evidence_id': evidence.bundle_id}
The release decision and the evidence supporting it should be generated from the same control execution. Not reconstructed months later.

Do Not Let Compliance Automation Blindly Stop Deployment

A governance gate that can block production becomes a production dependency. It needs its own availability design, versioning, rollback, exception process, separation of duties, change approval, and audit history. Suppose a compliance rule is incorrectly configured and blocks every production deployment on Friday afternoon. You now have a governance incident.

An exception should not mean turning compliance off. It should mean: named owner, reason, approver, scope, expiration, and compensating control. That is how ordinary enterprise change management already works. AI governance does not need to reinvent it.

Where AI Can Still Help

None of this means compliance engineering should avoid AI. AI can classify systems for review, summarize regulatory changes, suggest likely control mappings, analyze policy documents, help investigators search logs, identify missing evidence, generate test cases, and assist with red-team exploration.

But three responsibilities should remain distinct: AI assists interpretation, deterministic system enforces approved control, and qualified human owns legal conclusion. Conflating them is how a compliance assistant quietly becomes a compliance authority.

Drift Detection: Configuration Before Statistics

At least four things can drift: model drift (provider version changes), data drift (input distribution changes), behavior drift (outputs change under same evaluation), and configuration drift (prompts, tools, retrieval sources, safety controls change).

For many LLM applications, the highest-value monitoring does not require a sophisticated ML drift algorithm. Start with configuration drift:
Yesterday:
model = x-2026-07
prompt_hash = A84...
retrieval_index = v12
tools = [search, booking]
temperature = 0.2

Today:
model = x-2026-08
prompt_hash = 9BC...
retrieval_index = v13
tools = [search, booking, refund]
temperature = 0.7
That is a material change detected deterministically. A deployment gate can require rerunning evaluations when configuration changes. In the EU high-risk framework, post-market monitoring must actively and systematically collect and analyze performance data across the system's lifetime. (EUR-Lex) Drift detection supports that process. It is not synonymous with it.

Evidence Should Be Generated When the System Changes

A common compliance architecture produces evidence once per audit. That is backwards for software. Most meaningful evidence is easiest to produce when the change happens. At deployment time, capture: system identifier, application version, model provider and revision, prompt and configuration version, tool and retrieval configuration, control-set version, test results, required disclosures, approval identities, deployment timestamp, artifact hashes, exceptions, and known limitations.

A stronger evidence artifact tells you: control ID, result, system, deployment, commit hash, model version, rule version, test file, timestamp, executor, and artifact hash. The question becomes answerable precisely: this exact deployment passed this exact version of the control at this time. That is the difference between documentation and provenance.

The Board Does Not Need Another AI Risk Score

Most AI governance dashboards present a single composite score: AI Governance Score: 87. That number does not tell a board member anything useful. It does not tell a CTO what to fix. It does not tell an auditor what was checked. It does not tell a regulator what evidence exists.

A stronger dashboard answers specific questions: Which systems are deployed in Texas? Which have completed disclosure tests? Which have pending exceptions? Which have unresolved drift alerts? Which have overdue evaluations? Which have missing owners? Which have model changes since last approval?

The composite score is a communication artifact. The specific answers are an operating artifact. Boards need both. Engineering teams need the second one.

Not Every Model Update Should Be Treated Equally

A model version change from x-2026-07 to x-2026-08 might be a patch release with no behavioral change or a major release with new capabilities. A prompt change might add a clarification or remove a safety instruction. A retrieval index update might add new documents or remove a critical policy document.
change_classification:
model:
patch: rerun_safety_eval
minor: rerun_safety_eval + fairness_eval
major: full_reapproval
prompt:
cosmetic: log_only
behavioral: rerun_eval
safety_control_removed: full_reapproval
retrieval:
add_documents: log_only
remove_documents: rerun_eval
policy_change: full_reapproval
The point is not the exact classification. The point is that a deployment gate should treat material changes differently from cosmetic ones. A blanket rule that blocks every change creates noise. A blanket rule that allows every change creates risk. The classification itself should be governed and versioned.

A Practical 10-Control Starting Set

AI inventory completeness: Every production AI system has owner, purpose, jurisdiction, model, data classes, deployment context.
Role classification: For each jurisdiction, determine developer, provider, deployer, government entity, or healthcare provider status.
Approved-use boundary: The system has explicitly allowed and prohibited uses.
Model provenance: Record provider, model, and version.
Material-change detection: Detect changes to models, prompts, tools, retrieval, or safety configuration.
Required disclosure test: Where applicable, verify actual user-facing behavior.
Evaluation evidence: Required safety and performance tests execute before deployment.
Human oversight: Where a workflow requires human control, prove the control exists.
Post-deployment monitoring: Track metrics and limitations tied to the system's risk profile.
Incident and remediation history: When a control fails, record what happened, what changed, who approved it, and what proves remediation.

These ten controls will not make an organization automatically compliant. They will make it substantially easier to operate a serious compliance program. For immutable audit trail patterns, see implementing immutable audit trails for SOC 2 AI compliance.

A Cross-Jurisdictional Control Architecture

Organizations operating in both Texas and the EU should not build two separate compliance programs. They should build one control layer with jurisdiction-specific applicability rules.
SHARED AI SYSTEM RECORD
|
v
SHARED CONTROL LAYER
/ \
TEXAS RULES EU RULES
| |
TX disclosure EU transparency
TX cure process EU high-risk
TX AG inquiry EU post-market
| |
+--------+-----------+
|
v
SHARED EVIDENCE STORE
|
v
JURISDICTION-SPECIFIC
REPORTING OUTPUTS
The controls are shared. The applicability decisions differ. The evidence is shared. The reporting outputs differ. This avoids duplicating engineering work while respecting that the legal requirements are not identical.

What Changes Between Texas and Europe

Both jurisdictions ultimately require you to know what AI you operate, why it is permitted, what changed, and what evidence exists. The emphasis differs:
CapabilityTexas emphasisEU emphasisInventory and ownershipAG inquiry responseTechnical documentationRisk managementNIST alignment, testing, internal reviewFormal risk management systemDisclosureGovernment and healthcare contextsArticle 50 transparency, broader scopeMonitoringPost-deployment, feedback, red-team testingPost-market monitoring, systematic collectionEvidence retentionSupporting documentation for cure10-year documentation, 6-month logsEnforcementAG, cure process, civil penaltiesNational authorities, AI Office, tiered fines
NIST can become the common operational language. TRAIGA explicitly references the NIST Generative AI Profile. The EU AI Act references risk management systems and conformity assessment but does not mandate NIST specifically. An organization that builds its control architecture around NIST GOVERN, MAP, MEASURE, and MANAGE can map those controls to both Texas and EU requirements without maintaining two separate frameworks.

Compliance Becomes More Difficult When AI Becomes Agentic

Most current AI compliance assumes a relatively static system: one model, one prompt, one retrieval source, one deployment. Agentic AI systems break that assumption. An agent might call multiple models, use different tools depending on context, modify its own prompts, select retrieval sources dynamically, and chain calls in ways that were not explicitly designed.

The compliance question shifts from did we approve this system? to is this system still operating within its approved boundaries? That requires runtime monitoring, not just deployment-time gates. The system graph, not the spreadsheet, becomes the compliance artifact.

The Future Compliance Artifact Is a Deployment Graph

When an auditor asks show me the evidence for system X on date Y, the answer should not require reconstructing a year of Slack messages and Jira tickets. The answer should be a query against a deployment graph: system, version, model, controls, results, evidence, exceptions, approvals, and timeline.

That graph is built incrementally. Every deployment adds nodes. Every control execution adds edges. Every exception adds metadata. The graph becomes the living compliance record. The audit becomes a traversal.

My Test for Whether an AI Control Belongs in CI/CD

Can the condition be evaluated deterministically from system configuration, deployment metadata, or test results?
Does failing the condition create a real compliance or safety risk, not just a documentation gap?
Can the check be versioned, reviewed, and rolled back like any other code change?
Can an exception be granted with named owner, reason, approver, scope, and expiration?
Does the check produce evidence that an auditor or regulator would accept?

If the answer to all five is yes, the control belongs in the deployment gate. If not, it belongs in a governance workflow that feeds into the gate but does not block it directly.

The Real Difference Between a Policy Program and a Compliance System

A policy program says: We have a policy that requires AI risk assessment before deployment. A compliance system says: Deployment DEP-8371 passed control TX-AI-DISCLOSURE-HEALTHCARE v3.2 at 2026-08-08T14:03:22Z, evidence stored as EV-4471, approved by j.smith, model revision x-2026-08, no active exceptions.

The first statement is necessary. The second statement is what an auditor, a regulator, and a court will actually ask for.

The 90-Day CTO Playbook

Days 1-30: Know what exists. Build the inventory. For every production AI system, identify owner, use, geography, role, model and provider, affected people, data, autonomy, integrations, and existing controls. Do not begin by buying a compliance dashboard. Begin by establishing ground truth.

Days 31-60: Map obligations to controls. Work with legal, security, and product to create the chain: Requirement, then Applicability, then Control, then Owner, then Evidence. Start with currently applicable requirements and the highest-risk use cases.

Days 61-90: Move controls into engineering. Add model and configuration versioning, deployment-linked evaluations, disclosure tests, change detection, approval gates, exceptions, evidence generation, and post-deployment alerts.

The result after 90 days will not be finished compliance. That is the wrong target. The result should be a system that makes continued compliance easier to operate.

The Strongest Compliance Architecture

Texas HB 149 and the EU AI Act are not two versions of the same law. TRAIGA is comparatively targeted: specific duties, prohibited practices, AG enforcement, a defined cure process, and an unusual connection between testing and recognized AI risk-management frameworks. The EU AI Act is broader lifecycle regulation with a formal risk-based structure, transparency rules already applicable, GPAI obligations underway, and a high-risk regime whose major technical requirements now arrive later under the 2026 AI Omnibus.

Strip away the legal vocabulary and look at both from the system architecture layer. The questions converge: What AI do we operate? Why is this use permitted? Which model and configuration are actually deployed? Which requirements apply? Which controls enforce them? What changed? Did the required tests run? What happened after deployment? Can we reproduce the evidence? Who approved the exceptions?

Those questions cannot be answered reliably by an annual spreadsheet. They have to become part of the engineering lifecycle. The strongest AI compliance architecture is not the one with the longest checklist. It is the one that can connect a legal requirement to a technical control, a technical control to a specific deployment, and that deployment to evidence you can still reproduce when somebody finally asks for it.

If you are operating AI systems across Texas and EU jurisdictions and need an independent systems advisor to audit your compliance architecture, schedule a strategic evaluation.

Originally published at subodhkc.com. Learn more about the HAIEC AI Governance Platform. Follow for more on AI governance, enterprise architecture, and compliance engineering.

FastAPI WebSocket 1006 in LLM Streaming: Why and How to Fix

Subodh Kc — Tue, 29 Sep 2026 15:18:09 +0000

The Symptom: WebSocket Closed Before the LLM Finished

Your FastAPI WebSocket opens normally. The client sends a prompt. The model begins a long inference operation. Nothing comes back for a while. Then the browser reports:
WebSocket closed
code: 1006
reason: ""
Code 1006 is not a close code sent by FastAPI. RFC 6455 reserves 1006 for abnormal closure where a proper WebSocket Close frame was not received. An endpoint must not put 1006 into a Close control frame. (RFC Editor)

That means 1006 tells you the connection disappeared without a clean handshake. It does not tell you which layer disappeared first.

The Diagnostic Fork: Four Failure Classes

Before changing any timeout, determine which of these four failures you actually have:
Browser / Mobile / Voice Client
|
v
CDN / Edge
|
v
Load Balancer / WAF

The fastest debugging signal is timing. Measure the gap between last successful traffic and disconnect:
PatternInvestigate FirstDisconnect varies wildlyNetwork, client state, process healthConsistently around 20-40 secPing/pong and WebSocket liveness configConsistently around 60 secProxy or load-balancer idle policyOnly during synchronous model callsEvent-loop blockingOnly under high concurrencyEvent-loop lag, CPU saturation, backpressureServer logs clean close but client says 1006Trace intermediary or networkServer reports ping timeoutTransport latency or event-loop responsiveness

Why the Common Fix Is Wrong

Search for this problem and you will find a confident explanation:

FastAPI automatically kills WebSockets when the LLM takes longer than 10 seconds.

That is wrong. FastAPI does not have a built-in 10-second LLM generation timeout. Current Uvicorn defaults are 20 seconds for --ws-ping-interval and 20 seconds for --ws-ping-timeout. (Uvicorn)

The common proposed fix is a JSON heartbeat daemon:
async def websocket_ping_daemon(websocket):
while True:
await asyncio.sleep(5)
await websocket.send_json({"type": "ping"})
This has good instincts but three critical problems. First, {"type": "ping"} is an application data message, not a WebSocket protocol Ping control frame. Second, if the event loop is blocked by a synchronous call, the heartbeat cannot run either. Third, it does not address proxy or load-balancer idle timeouts that operate independently of application-level traffic.

Class 1: You Are Blocking the Event Loop

This is the failure that most closely matches the original theory. Consider two operations that both take 20 seconds:

Genuinely async - event loop stays responsive

result = await async_llm_client.generate(prompt)

Blocking - event loop frozen for 20 seconds

result = synchronous_llm_client.generate(prompt)
With a genuinely asynchronous operation, await gives control back to the event loop. During those 20 seconds, the event loop services other WebSockets, ping/pong handling, timers, and HTTP requests. A synchronous blocking call inside an async handler freezes everything. (Python documentation)

The same principle applies to time.sleep() versus asyncio.sleep(). The problem is not sleep. The problem is blocking the event loop.

Fix A: Use the async SDK

async for token in async_model.stream(prompt):
await websocket.send_text(token)
Long latency is not automatically event-loop starvation. A model can take 30 seconds before producing output without freezing the event loop, provided the operation is implemented asynchronously.

Fix B: Wrap synchronous SDKs with asyncio.to_thread

result = await asyncio.to_thread(
blocking_model_call,
prompt,
)
Python documents asyncio.to_thread() specifically for synchronous I/O that would otherwise block the event loop. (Python documentation)

Fix C: Move CPU-bound work to a process pool

For local inference, embedding post-processing, or expensive parsing, asyncio.to_thread() is limited by CPython's GIL. Use a process pool or a dedicated compute service. Your WebSocket process should not quietly become your compute scheduler. For production voice and real-time AI patterns, see voice agent architecture at Kestrel Voice.

Class 2: Uvicorn WebSocket Ping Timeout

Your model request may be correctly asynchronous and the process responsive, but the WebSocket transport still fails ping/pong health checks. Server logs may show:
websockets.exceptions.ConnectionClosedError:
sent 1011 (internal error) keepalive ping timeout;
no close frame received
The websockets library documents this as a keepalive timeout. (websockets)

Parameter Breakdown

ParameterDefaultControlsWhen to Change--ws-ping-interval20.0sHow often Uvicorn sends WebSocket Ping framesIncrease if ping traffic is excessive on constrained networks--ws-ping-timeout20.0sHow long Uvicorn waits for a Pong before closingIncrease only after confirming event loop is healthy--timeout-keep-alive5.0sHTTP keep-alive timeout (not WebSocket)Do not change for WebSocket 1006 issuesNGINX proxy_read_timeout60sTime NGINX waits between reads from upstreamIncrease for long LLM generation gapsAWS ALB idle timeout60sTime ALB waits with no traffic from either sideAlign with your maximum expected silent period
Do not confuse --timeout-keep-alive with WebSocket ping settings. They are separate timeout classes documented separately by Uvicorn. (Uvicorn settings)

Class 3: Proxy or Load-Balancer Idle Timeout

If disconnects happen consistently near 60 seconds, inspect infrastructure before rewriting the application. NGINX defaults proxy_read_timeout to 60 seconds. (Nginx) AWS Application Load Balancers default their idle timeout to 60 seconds. (AWS Documentation)

A WebSocket-aware NGINX configuration:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

location /stream {
proxy_pass http://fastapi_upstream;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 300s;
}
The websockets project notes that real HTTP infrastructure commonly closes connections after idle periods in the 30-120 second range. (websockets) When a disconnect occurs at a repeatable infrastructure-sized interval, inspect infrastructure first. For broader production architecture patterns, see production RAG architecture patterns.

Production Configuration: Async Streaming with Serialized Heartbeat

Once the event loop is healthy, add an application heartbeat for session liveness and proxy traffic. Serialize all outbound writes with a lock to prevent interleaved frames:
import asyncio
import time

from fastapi import FastAPI, WebSocket, WebSocketDisconnect

app = FastAPI()

async def send_json_safe(websocket, lock, payload):
async with lock:
await websocket.send_json(payload)

async def application_heartbeat(websocket, lock, interval=15.0):
while True:
await asyncio.sleep(interval)
await send_json_safe(websocket, lock, {
"type": "heartbeat",
"timestamp": time.time(),
})

@app.websocket("/stream")
async def stream(websocket: WebSocket):
await websocket.accept()
send_lock = asyncio.Lock()
heartbeat_task = asyncio.create_task(
application_heartbeat(websocket, send_lock)
)
try:
while True:
prompt = await websocket.receive_text()
async for token in stream_model_async(prompt):
await send_json_safe(websocket, send_lock, {
"type": "token", "payload": token,
})
await send_json_safe(websocket, send_lock, {
"type": "complete"
})
except WebSocketDisconnect:
pass
finally:
heartbeat_task.cancel()
await asyncio.gather(heartbeat_task, return_exceptions=True)
For high-concurrency systems with multiple producers, use a bounded asyncio.Queue(maxsize=128) with a single writer task. This gives explicit backpressure instead of unbounded buffering. FastAPI's WebSocket API raises WebSocketDisconnect when a receive operation detects the closed connection. (FastAPI) Use that event to cancel upstream generation and log structured metadata.

What Not to Deploy

Do not increase timeouts to hide event-loop starvation

If the event loop is blocked for 45 seconds and you increase --ws-ping-timeout to 60, you have hidden the defect, not fixed it. A timeout should represent your operating policy, not camouflage.

Do not send JSON heartbeats and call them Ping frames

{"type": "ping"} is an application data message. Native WebSocket Ping/Pong uses protocol control frames handled by Uvicorn's transport layer. Application heartbeats solve a different problem: session liveness, UI status, and keeping intermediaries active.

Do not conflate session lifetime with generation lifetime

A WebSocket can live for hours. That does not mean one LLM request should. Add a separate generation deadline:
async with asyncio.timeout(90):
async for token in stream_model_async(prompt):
...
This lets you fail one generation cleanly while keeping the user's session alive. For governance frameworks around production AI deadlines and evidence, see seven layers of AI compliance.

Diagnostic Decision Tree

Start: WebSocket closed with 1006
|
+-- Check server logs for "keepalive ping timeout"
| |
| +-- YES: Transport latency or event-loop lag
| | +-- Is LLM call async? NO -> Fix: use async SDK or to_thread
| | +-- Is LLM call async? YES -> Increase ws-ping-timeout
| | (only after measuring event-loop lag)
|
| +-- NO: Check disconnect timing pattern
| |
| +-- Consistent ~60s -> Check proxy_read_timeout,
| | ALB idle timeout
| |
| +-- Only during sync calls -> Event-loop blocking
| |
| +-- Random -> Network, client, process restart
| |
| +-- Under concurrency -> CPU saturation, backpressure

Verification Checklist

Capture the exact client close code and timestamp
Check server-side disconnect or error logs for ping timeout
Search async handlers for time.sleep(), requests.get(), or sync SDK calls
Measure event-loop lag with a monitoring task (see below)
Verify Uvicorn --ws-ping-interval and --ws-ping-timeout values
Confirm --timeout-keep-alive is not being changed for WebSocket issues
Inspect NGINX proxy_read_timeout and ALB idle timeout
Test slow generation deliberately: 5s, 15s, 30s, 60s, 120s with async calls
Test under production-like concurrency
Test client reconnection and process restarts
Confirm generation cancellation fires on client disconnect

Event-loop lag monitor for production:
async def monitor_event_loop_lag(interval=0.5, warning_threshold=0.25):
loop = asyncio.get_running_loop()
expected = loop.time() + interval
while True:
await asyncio.sleep(interval)
now = loop.time()
lag = now - expected
if lag >= warning_threshold:
logger.warning("event_loop_lag_seconds=%.3f", lag)
expected = now + interval
If a blocking call freezes the event loop, this task freezes too. When scheduling resumes, the delay becomes observable. In production, export this as a metric rather than relying on logs alone.

What to Monitor for a Production LLM Streaming Service

MetricWhy It MattersAbnormal WebSocket closure rate (1006)Track separately from normal 1000 closuresEvent-loop lag (p95, p99)Earliest signal of event-loop starvationTime to first model eventDistinguishes transport health from generation healthMaximum inter-event gapDetects when a stream goes silent unexpectedlyPing timeout countTransport health metricAbandoned generation rateCompute cost after client disconnectOutbound queue depthEarliest backpressure signalReconnection rateExposes transport problems before users complain
A WebSocket service can remain technically up while a significant portion of long-running sessions are unusable. API uptime of 99.9% does not capture that. For production AI operations and deployment patterns, see 12 production readiness checks for AI pilots.

Liveness Is Not Progress

A heartbeat proves the session is alive. It does not prove the model is making progress. A mature streaming system tracks three separate timers:
Transport liveness
|
+-- Is the WebSocket alive? (ping/pong, proxy idle)

Generation liveness
|
+-- Has upstream produced meaningful activity? (TTFT, inter-event gap)

Application deadline
|
+-- How long are we willing to keep this request alive? (product policy)
Keepalive can turn a failed request into a connection that remains successfully stuck forever. Separate the timers. Cancel or account for orphaned model generations when the client disappears. At enterprise scale, uncancelled generations become spend.

If you are deploying real-time LLM streaming and need an independent systems advisor to audit your WebSocket architecture, schedule a strategic evaluation.

Originally published at subodhkc.com. Learn more about the HAIEC AI Governance Platform. Explore KestrelVoice AI Voice Operations. Follow for more on AI governance, enterprise architecture, and compliance engineering.

vLLM CUDA OOM at Startup? Two Bugs, Two Opposite Fixes

Subodh Kc — Thu, 24 Sep 2026 15:17:33 +0000

Two Different OOM Errors, Two Different Fixes

You downloaded the model. The weights fit on the GPU. vLLM starts loading. Then, before the first prompt reaches the server, the engine dies.

Depending on your vLLM version and where initialization failed, the useful part of the traceback may look like this:
ValueError: No available memory for the cache blocks.
Try increasing gpu_memory_utilization when initializing the engine.
Or you may see the more familiar:

torch.OutOfMemoryError: CUDA out of memoryThese errors look similar in logs. They are not fixed the same way. If you blindly lower gpu_memory_utilization from 0.90 to 0.80, you may fix a CUDA OOM while making a KV cache capacity error worse. If you blindly raise it, you can do the opposite. The right fix depends on which part of vLLM's startup memory calculation failed.

Current vLLM code explicitly recommends increasing gpu_memory_utilization for KV cache errors and either increasing utilization or decreasing max_model_len when the requested sequence length will not fit. (vLLM GitHub)

This article walks through that calculation, the Docker-specific checks worth doing, and the vLLM settings that actually matter for production deployment. For broader production AI architecture patterns, see secure enterprise RAG architecture.

Why the Widely Copied Fix Is Partly Wrong

A common Stack Overflow recommendation looks like this:
LLM(
model=model_id,
gpu_memory_utilization=0.80,
block_size=8,
swap_space=4,
enforce_eager=True,
disable_custom_all_reduce=True
)
Several parts of that configuration deserve correction.

gpu_memory_utilization=0.80 can help if vLLM is attempting to use too much VRAM and CUDA itself runs out during profiling. It is not a universal fix. If vLLM is telling you the KV cache has no available memory, lowering utilization gives vLLM an even smaller memory budget. Current vLLM documentation lists the default as 0.92, not 0.90 as older tutorials claim. (vLLM Docs)

block_size=8 is not a generic OOM remedy. Current vLLM documentation does not define a universal static block-size default. The value is resolved according to platform and cache configuration. vLLM sizes the KV cache primarily from its available memory budget. Reducing block granularity does not reduce total KV-cache memory. (vLLM Docs)

swap_space=4 is deprecated and ignored in current vLLM's LLM constructor. Modern vLLM exposes separate mechanisms for model-weight CPU offloading and KV-cache offloading instead. (vLLM Docs)

disable_custom_all_reduce=True disables vLLM's custom all-reduce kernel and falls back to NCCL. That can be useful for compatibility issues, but current documentation does not describe it as an OOM-control mechanism. (vLLM Docs)

Do not tune five unrelated flags because one Stack Overflow answer happened to start successfully with them. Diagnose which memory boundary failed first.

What vLLM Does With Your GPU During Startup

Stop thinking about VRAM as a single number. If nvidia-smi says 24564 MiB total, that does not mean vLLM has 24 GB available for the KV cache. The GPU must accommodate several consumers:
+-----------------------------------+
| Total GPU VRAM |
| (e.g. 24564 MiB = ~24 GB) |
+-----------------------------------+
| Model weights |
| (e.g. 8B params x 2 bytes = 16GB)|
+-----------------------------------+
| Runtime / non-PyTorch allocs |
+-----------------------------------+
| Peak activation memory |
+-----------------------------------+
| CUDA graph memory |
+-----------------------------------+
| KV cache |
| (what remains after profiling) |
+-----------------------------------+
| Safety / unused headroom |
+-----------------------------------+
vLLM's GPU worker performs a profiling run because it cannot know all of those values from model parameter count alone. The engine loads the model, executes a profiling pass with dummy inputs to estimate non-KV peak memory, then determines how much remains for the KV cache. Recent vLLM versions also account for estimated CUDA graph memory in this calculation. (vLLM Docs)

The startup profiling sequence looks like this:
vLLM Startup Memory Profiling
|
v
Load model weights onto GPU
|
v
Run profiling pass with dummy inputs
(measures peak activation memory)
|
v
Estimate CUDA graph memory
(v0.21.0+ accounts for this)
|
v
Calculate remaining budget:
gpu_memory_utilization x total VRAM
- weights - activations - graphs
= available KV cache memory
|
v
Allocate KV cache blocks
|
+----+----+
| |
OK FAIL
| |
v v
Server "No available memory
starts for the cache blocks"
Conceptually, the calculation is: requested vLLM memory budget minus model, activation, runtime, and CUDA graph memory equals available KV-cache memory. This mental model is more useful than treating GPU memory as a single pool.

The Diagnostic Table

The distinction between failure patterns will save more time than any magic 0.80 setting:
Error PatternWhat FailedFirst Actionstorch.OutOfMemoryError: CUDA out of memoryPhysical CUDA allocationFree GPU memory, lower utilization, reduce context/batch, test eager modeNo available memory for the cache blocksRequested vLLM budget leaves no KV capacityIncrease utilization if real headroom exists; otherwise reduce model/memory needsKV cache needed ... larger than availablemax_model_len exceeds KV capacityReduce max_model_len; increase available KV memory where safeOOM during CUDA graph captureGraph/runtime overheadTry --enforce-eager or smaller graph configurationCannot re-initialize CUDA in forked subprocessCUDA multiprocessing initializationFix process model; do not treat as ordinary OOMContainer cannot see GPUNVIDIA runtime configurationValidate --gpus, NVIDIA_VISIBLE_DEVICES, driver

The Two Failure Paths

The most important distinction in vLLM startup OOM is which memory boundary failed. These two failure paths require opposite fixes in some cases:
Failure A: CUDA allocation OOM Failure B: KV cache capacity error
| |
v |
"torch.OutOfMemoryError: "No available memory for
CUDA out of memory" the cache blocks"
| |
v v
GPU ran out of physical vLLM budget too small for
memory during allocation required KV cache capacity
| |
v v
FIX: Lower utilization FIX: Increase utilization
(give vLLM smaller target) (if real VRAM is available)
OR: Free GPU from other procs OR: Reduce max_model_len
OR: Reduce max_model_len OR: Use FP8 KV cache
OR: Disable CUDA graphs OR: Tensor parallelism
| |
v v
More headroom = safer More budget = more cache
but less cache capacity but less safety headroom
Lowering gpu_memory_utilization helps Failure A but worsens Failure B. Raising it helps Failure B but worsens Failure A. Read the error message before touching any setting.

Step 1: Verify Docker GPU Access

A Docker container does not automatically give vLLM a private pool of VRAM. The NVIDIA Container Toolkit's --gpus and NVIDIA_VISIBLE_DEVICES controls determine which GPUs are visible. (NVIDIA Docs)

Start on the host:

nvidia-smiThen check inside your vLLM container:

docker exec -it <vllm-container> nvidia-smiIf the server has not started yet, validate GPU access independently:
docker run --rm --gpus all \
nvidia/cuda:12.8.0-base-ubuntu22.04 \
nvidia-smi
What matters is the memory picture immediately before vLLM starts. If another Python process, notebook, or neighboring container owns several gigabytes of VRAM, changing vLLM's internal cache settings does not reclaim that memory. For production container deployment patterns, see building internal AI applications with Streamlit, RAG, and MCP.

Step 2: Confirm Model Weights Fit

Paged KV caching cannot rescue a model whose weights consume all available VRAM. A rough unquantized weight calculation: parameter count times bytes per parameter. An 8-billion-parameter model in BF16 is approximately 16 GB of weights. That is before runtime memory, activations, CUDA graphs, and KV cache.

If model weights leave almost no GPU headroom, tuning block_size is not the fix. Your options: use a smaller model, use an appropriate quantized model, use tensor parallelism across multiple GPUs, or offload model weights to CPU. vLLM's memory-conservation guide recommends quantization and tensor parallelism for this class of problem. (vLLM Docs)

Step 3: Reduce max_model_len Before Chasing Obscure Flags

Modern open-weight models advertise context windows of tens or hundreds of thousands of tokens. That does not mean your deployment needs to reserve enough KV-cache capacity to serve that entire context. If your application only expects 4K prompts, launching with a much larger maximum imposes memory requirements you do not need.
vllm serve "$MODEL" \
--max-model-len 4096 \
--max-num-seqs 1
vLLM explicitly recommends reducing max_model_len and max_num_seqs to reduce memory consumption. (vLLM Docs) The production value should come from your workload requirements, not from whichever number makes the server boot.

Step 4: Tune gpu_memory_utilization Based on the Error

For a true CUDA OOM, a conservative diagnostic run:
vllm serve "$MODEL" \
--gpu-memory-utilization 0.80 \
--max-model-len 4096 \
--max-num-seqs 1
If that starts successfully, the previous configuration was operating too close to the device's usable memory boundary. You can then increase utilization gradually while monitoring startup logs.

But if the result is No available memory for the cache blocks, the smaller memory target left too little room for the cache. Do not keep marching utilization downward. You are shrinking vLLM's cache budget. Either reduce the requested context or, if nvidia-smi confirms genuine unused VRAM, allow vLLM a larger fraction. The error text matters. Read it.

Step 5: Use enforce_eager for CUDA Graph Memory Issues

vLLM uses CUDA graphs for performance, and graphs consume GPU memory. The project's memory-conservation documentation explicitly recommends enforce_eager=True to disable CUDA graph capture completely. (vLLM Docs)
vllm serve "$MODEL" \
--max-model-len 4096 \
--gpu-memory-utilization 0.80 \
--enforce-eager
If eager execution starts but the normal configuration does not, CUDA graph memory is part of the boundary you are crossing. Treat eager mode as a diagnostic and capacity lever, not an automatic production default. Disabling graphs can reduce inference performance. For runtime verification of production AI systems, see llmverify.

Step 6: Consider FP8 KV Cache for KV Capacity Bottlenecks

vLLM supports quantized KV cache formats including FP8 on supported hardware. FP8 KV-cache quantization can significantly reduce the cache memory footprint, allowing more tokens or greater concurrency. (vLLM Docs)
vllm serve "$MODEL" \
--kv-cache-dtype fp8
Quantization strategy and scale calibration affect accuracy. vLLM's documentation recommends dataset-based calibration when appropriate. Use KV-cache quantization because you have measured a KV-capacity bottleneck and tested model quality, not simply because an OOM appeared.

The Production Tuning Order

When vLLM will not start, change things in this order:
PriorityCheck or ChangeWhy1nvidia-smiEstablish actual free VRAM2Pin vLLM versionDefaults and memory behavior change between versions3Confirm model fitsNo cache tuning fixes oversized weights4Set realistic max_model_lenDirect control over KV requirement5Reduce max_num_seqsLowers batch/concurrency memory6Tune gpu_memory_utilization by error typeControls executor budget7Try enforce_eagerTests whether CUDA graphs are part of the boundary8Quantize model or KV cacheReduces the relevant memory footprint9Tensor parallelismSpreads model across GPUs10CPU/KV offloadingTrades performance for capacityLastRandom block-size or all-reduce flagsUsually solving a different problem
That order deliberately favors explanations over superstition.

A Production Debugging Decision Tree

What Not to Deploy

Do not build production logic that catches a CUDA OOM and instantiates another vLLM engine with increasingly arbitrary settings:
try:
llm = LLM(gpu_memory_utilization=0.80)
except RuntimeError:
llm = LLM(gpu_memory_utilization=0.70)
That pattern hides the distinction between OOM classes and makes deployment behavior dependent on runtime failure. A server that silently falls back to 0.70 because 0.80 failed today will behave differently tomorrow when a neighboring container frees 2 GB of VRAM. The 0.70 configuration was never validated. It was just the next number down.

Use explicit configuration validated before the service receives traffic:
import os
from vllm import LLM

MODEL_ID = os.environ["MODEL_ID"]

llm = LLM(
model=MODEL_ID,
gpu_memory_utilization=float(
os.getenv("VLLM_GPU_MEMORY_UTILIZATION", "0.80")
),
max_model_len=int(
os.getenv("VLLM_MAX_MODEL_LEN", "4096")
),
max_num_seqs=int(
os.getenv("VLLM_MAX_NUM_SEQS", "1")
),
enforce_eager=os.getenv(
"VLLM_ENFORCE_EAGER", "false"
).lower() == "true",
)
The values should come from capacity testing for your model and hardware. A server should not silently decide that because 80% failed today, 70% is suddenly the correct production configuration. For governance of production AI deployments, see how to secure and govern AI systems.

Startup OOM vs Runtime KV Cache Pressure

Do not confuse startup OOM with runtime KV-cache pressure. At runtime, concurrent requests and long contexts consume KV-cache capacity. vLLM may preempt requests and recompute when cache pressure becomes high rather than crashing. The performance-tuning guide recommends increasing available KV cache, decreasing maximum batch parameters, or distributing the model across more GPUs when preemption becomes frequent. (vLLM Docs)

A server that cannot initialize at all is a different problem. The distinction matters when searching logs because both involve KV cache and memory.

Final Checklist: vLLM OOM Before the First Request

Confirm the exact vLLM version
Run nvidia-smi on the host
Run nvidia-smi inside the container
Check for other GPU processes
Confirm model weights realistically fit
Set max_model_len to the application's real requirement
Reduce max_num_seqs while diagnosing
Identify CUDA OOM vs KV-cache-capacity error
Tune gpu_memory_utilization in the correct direction
Test enforce_eager if graph memory is implicated
Consider quantization if weights are the problem
Consider FP8 KV cache if KV capacity is the problem
Consider tensor parallelism for multi-GPU systems
Consider CPU/KV offloading only with measured tradeoffs
Load-test the final configuration after startup succeeds

The one setting that does not belong at the top of that list is block_size. And changing gpu_memory_utilization from 0.90 to 0.80 does not instantly fix vLLM. Sometimes it does. Sometimes the correct move is exactly the opposite. The error message tells you which problem you actually have.

If you are deploying self-hosted LLM infrastructure and need an independent systems advisor to audit your production readiness before traffic hits, schedule a strategic evaluation.

FAQ

How do I fix vLLM CUDA out of memory during startup? First check actual GPU usage with nvidia-smi. Then identify whether CUDA itself failed an allocation or vLLM calculated insufficient KV-cache capacity. For true CUDA OOMs, reducing gpu_memory_utilization, max_model_len, or max_num_seqs can help. For insufficient KV cache errors, reducing max_model_len or increasing the vLLM memory budget when physical VRAM is available is usually more appropriate.

Should I set gpu_memory_utilization from 0.90 to 0.80? Not automatically. Lowering it provides additional GPU headroom and may solve actual CUDA allocation OOMs. But it also reduces the memory budget available to vLLM's KV cache. If the error says the KV cache is too small, lowering utilization may make the problem worse.

What is the default gpu_memory_utilization in vLLM? It is version dependent. Many older vLLM releases documented a default of 0.90, while current vLLM documentation lists 0.92. Always check the documentation matching your installed version.

Does lowering block_size fix a vLLM OOM? Not reliably. Current vLLM does not use one universal static block-size default, and total KV-cache capacity is primarily driven by available memory. Block size controls cache organization, not total memory.

Does enforce_eager reduce vLLM memory? It disables CUDA graph capture, which consumes additional GPU memory. This can reduce memory requirements, but the amount saved is model- and hardware-dependent and may come with a performance cost.

Can FP8 KV cache reduce vLLM memory? Yes, on supported hardware. vLLM documents FP8 KV-cache quantization as a way to significantly reduce KV-cache memory and support more tokens or concurrency. Accuracy and calibration should be tested before production use.

Does Docker cause vLLM CUDA OOM errors? Not inherently. Docker and the NVIDIA Container Toolkit control which GPUs are exposed to the container. You still need to inspect actual VRAM consumption and other processes using the same device.

What is PagedAttention in vLLM? PagedAttention is the memory-management approach introduced with vLLM for handling the KV cache in blocks, inspired by virtual-memory paging. It reduces KV-cache waste and enables efficient sharing and batching. It cannot make a configuration fit when the model, runtime, and required KV cache exceed physical GPU capacity. (arXiv)

Originally published at subodhkc.com. Follow for more on AI governance, enterprise architecture, and compliance engineering.

Claude Code MCP "Failed to Connect": Fix stdio, ENOENT, Timeout

Subodh Kc — Tue, 22 Sep 2026 15:17:17 +0000

The Error Is a Status, Not a Root Cause

You added a local MCP server to Claude Code. The configuration looks reasonable. The Python script works from Terminal. The database is running. Then:
$ claude mcp list

secure-local-database:
✅ Failed to connect
For a local MCP server using stdio, Claude Code is not opening a TCP connection to your Python process. It launches the server as a subprocess and communicates through standard input and standard output. A local MCP server can fail to connect even though no network connection was ever attempted. (MCP Python SDK)

Claude Code can show MCP servers in several states: Connected, Needs authentication, Failed to connect, Pending approval, and Rejected. A server showing Failed to connect means Claude Code could not establish the configured MCP integration. It does not tell you which architectural layer failed. Project-scoped servers can also remain at Pending approval until the workspace and server configuration are explicitly trusted. (Claude Code Docs)

The debugging principle: do not start at the server code. Start at the first boundary Claude Code actually attempted to cross.
Configuration
|
v
Trust / approval
|
v
Process or network transport
|
v
MCP protocol
|
v
Tool discovery
|
v
Tool execution
|
v
Downstream system
Prove those layers in order and most MCP failures stop looking mysterious.

Identify the Transport Before Touching Anything

MCP supports different transport models. The two that matter for this article:
Local stdio MCPRemote HTTP MCPClient launches a subprocessServer is already running elsewhereCommunication uses stdin/stdoutCommunication uses HTTPExecutable path mattersURL, DNS and routing matterOS permissions matterTLS, proxy and firewall matterstdout is protocol trafficHTTP body/headers carry protocolENOENT is meaningfulConnection refused/HTTP status is meaningfulLocal environment variables matterOAuth/HTTP credentials often matter
The current MCP Python SDK describes stdio as the default transport for local servers and Streamable HTTP as the deployment transport. SSE remains available for compatibility but is not recommended for new servers. (MCP Python SDK)

Before searching for MCP connection refused, ask whether there is actually a network connection here. For a normal local Claude Code stdio server, the answer is no.

Failure Class 0: The Server Is Waiting for Approval

Run claude mcp list. If you see Pending approval, your Python server has not failed. Claude Code has not authorized that project MCP configuration to run.

Project-scoped MCP servers live in the repository's .mcp.json. Claude Code requires approval for those servers in interactive sessions because a repository should not be able to clone itself onto a developer's machine and silently instruct Claude Code to execute an arbitrary local process. A repository cannot simply commit its own approval and bypass an untrusted-workspace decision. (Claude Code Docs)

Open Claude Code interactively with claude, review the workspace trust prompt, then check again. For details: claude mcp get secure-local-database.

This behavior is not developer inconvenience. It is a supply-chain control. A project-level MCP configuration can contain a command that executes software on an employee workstation. Requiring trust before that configuration runs prevents version-controlled configuration from becoming an automatic local code-execution mechanism. An enterprise should preserve that boundary rather than training developers to bypass it.

Failure Class 1: Claude Code Cannot Find the Executable

You can run python3 /opt/company-mcp/server.py from your shell. Claude Code fails. Your interactive shell may initialize PATH modifications, pyenv, conda, uv, nvm, fnm, Volta, Homebrew, or corporate shell scripts before you type a command. The process launching your MCP server may see a different environment.

Anthropic explicitly documents the spawn ... ENOENT failure class when Claude Code cannot find the executable configured for a stdio MCP server, and recommends supplying the full executable path. (Claude Code Docs)

On Linux or macOS: which python3, which uv, which node. On Windows: where.exe python. Then configure the resolved executable:
{
"mcpServers": {
"secure-local-database": {
"type": "stdio",
"command": "/home/me/audit-mcp/.venv/bin/python",
"args": ["/home/me/audit-mcp/server.py"]
}
}
}
Do not solve PATH problems by inventing a smaller PATH. A common recommendation is "PATH": "/usr/local/bin:/usr/bin:/bin". Sometimes that works. Sometimes it removes the exact runtime you need. The better production pattern is: pinned runtime, pinned environment, absolute executable, absolute server path. This also makes the MCP deployment reproducible on another developer machine or CI runner.

Failure Class 2: OS Refuses to Run the Executable

If you launch a Python script directly, check ls -l /opt/company-mcp/server.py and make sure the file is executable with a valid interpreter line. A cleaner configuration uses the interpreter as the executable:
{
"command": "/opt/company-mcp/.venv/bin/python",
"args": ["/opt/company-mcp/server.py"]
}
Check directory traversal permissions: ls -ld /opt and ls -ld /opt/company-mcp. A file can be readable while a parent directory remains inaccessible. This becomes particularly relevant on hardened Linux workstations, mounted enterprise volumes, WSL, containers, and corporate endpoint-management environments. A permission failure is an operating-system boundary. Changing MCP protocol settings will not fix it.

Failure Class 3: The Process Starts and Crashes Immediately

Claude Code successfully launches Python. Python starts. Then something throws: ModuleNotFoundError, ImportError, KeyError, certificate error, database authentication error. From Claude Code's perspective, the MCP server disappeared. From Python's perspective, it crashed before it ever became a usable server.

The highest-value diagnostic is simple: run exactly the configured command yourself. If .mcp.json says "command": "/home/me/mcp/.venv/bin/python" with "args": ["/home/me/mcp/server.py"], run /home/me/mcp/.venv/bin/python /home/me/mcp/server.py. A healthy stdio server run manually may appear to hang silently because it is waiting for a host to send protocol traffic over stdin. There is no port to open and no server listening banner. (MCP Python SDK)

If your process immediately exits instead, find the startup exception first.

Failure Class 4: stdout Corrupts the stdio Protocol

The server can appear healthy while corrupting the protocol. For a stdio MCP server, stdout is the protocol wire. The MCP transport specification reserves it for MCP messages. Local diagnostic logs should go to stderr.

Use Python logging instead of print:
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(name)

logger.info("Starting database MCP server")
The current Python SDK moves the wire to a private descriptor during serving and diverts flushed stdout to stderr, but output flushed to stdout before serving begins still lands on the wire. A dependency that prints during import can contaminate the transport without your server ever calling print(). (MCP Python SDK)

Do not use sys.stdout.flush() as a protocol fix. If invalid output has already been written, flushing ensures those invalid bytes reach the protocol stream. The fix is to not use stdout for non-protocol output.

Failure Class 5: Missing Environment Configuration

Database MCP servers commonly depend on DATABASE_URL or API_TOKEN. Your shell may have that variable. Claude Code's MCP subprocess may not. Project .mcp.json supports environment-variable expansion using ${VAR} and ${VAR:-default} in commands, arguments, environment values, URLs, and HTTP headers. (Claude Code Docs)
{
"mcpServers": {
"secure-local-database": {
"type": "stdio",
"command": "/absolute/path/to/.venv/bin/python",
"args": ["/absolute/path/to/server.py"],
"env": {
"DATABASE_URL": "${MCP_DATABASE_URL}"
}
}
}
}
Then outside version control: export MCP_DATABASE_URL='postgresql://readonly_user:...@127.0.0.1:5432/audit' and run claude.

If ${MCP_DATABASE_URL} is not defined and has no fallback, the configuration does not fail to load. Claude Code reports a missing-variable warning and leaves the literal expression unexpanded. That can create a confusing second-order failure if the downstream library attempts to interpret the placeholder as a real connection value. Treat an unresolved secret placeholder as a configuration failure even if Claude Code technically loads the file.

Do not debug secrets by printing them. Log bool(os.getenv("DATABASE_URL")) instead of the value itself. A troubleshooting log can outlive the incident and turn a short connection problem into a credential exposure. The MCP security guidance consistently emphasizes reducing credential exposure and limiting server privileges. (MCP Security)

Failure Class 6: The Server Is Slow, Not Broken

A server can be correct and still take too long to become available. Current Claude Code gives us several timing concepts that should not be confused.
ParameterDefaultWhat It ControlsWhen to ChangeMCP_TIMEOUT30000 (30s)Timeout for an individual MCP server startup attemptIncrease only after confirming startup correctness; investigate why startup is slow firstMCP_CONNECT_TIMEOUT_MS5000 (5s)Blocking connection window when non-blocking is disabled or server uses alwaysLoadIncrease if legitimate servers need more blocking time during startupMCP_CONNECTION_NONBLOCKING1 (on)Whether startup waits for MCP servers before first querySet to 0 to restore blocking 5-second waitMCP_DISCOVERY_CACHE1 (on)Cross-process MCP discovery cache for remote serversSet to 0 to force every server to connect at startup
For a diagnostic test: MCP_TIMEOUT=60000 claude. But increasing the timeout should not become your permanent architecture without asking why startup is slow. A healthier server does lazy initialization: process starts, MCP becomes available, tool gets invoked, expensive dependency is initialized if needed. (Claude Code Env Vars)

Failure Class 7: Wrong Transport in Configuration

This configuration is incomplete:
{
"mcpServers": {
"company-api": {
"url": "https://example.com/mcp"
}
}
}
Claude Code interprets entries without a type as stdio configuration. A URL-only server needs an explicit HTTP type: "type": "http". Claude Code reports a configuration error when a server has a URL but no type and tells the user to specify http, sse, or ws. (Claude Code Docs)

The MCP 2026-07-28 Protocol Change

MCP through the 2025-11-25 generation used an initialization lifecycle built around initialize / initialized and protocol-level sessions with Mcp-Session-Id. MCP 2026-07-28 removed that architecture. The new core protocol removed the handshake exchange and the protocol-level session. Requests are self-contained, and an optional server/discover RPC can be used when a client wants capability information in advance. (MCP Blog)
2025-era mental model:
connect -> initialize -> session -> requests

2026 model:
request carries protocol/client metadata
-> server processes request
The current Python SDK v2 implements the new protocol while retaining compatibility with older clients and servers. Its client can probe modern discovery and fall back to legacy initialization when speaking to an older server. (MCP Python SDK v2)

For enterprise platform teams, this changes deployment architecture. Modern requests can be handled without requiring a protocol-level sticky session. A request can land on different server instances behind ordinary load balancing because the protocol version, client information, and capabilities travel with the request. (MCP Blog)

Remote MCP can fail at the gateway even when the server is healthy. The current MCP 2026-07-28 HTTP example uses headers including MCP-Protocol-Version, Mcp-Method, and Mcp-Name. If an intermediary strips or mishandles headers, expects obsolete session behavior, or applies the wrong authentication policy, the MCP server can be completely healthy while the client still fails. Capture the request at the server boundary and verify the properties you expect to survive.

Why the 2026 Protocol Change Matters to Infrastructure Teams

For an individual developer, removing the handshake may sound like protocol trivia. For an enterprise platform team, it changes deployment architecture.
Before (2025-era):

Client
|
v
Load Balancer
|
v
Sticky Server A
|
v
Protocol session state

After (2026-07-28):

Client
|
v
Load Balancer
/ | \
A B C
Application-level state can still exist. The protocol simply no longer requires its own implicit server session. For organizations standardizing MCP gateways, this makes ordinary horizontal scaling easier. It also creates a new migration concern: your proxies and gateways need to understand the traffic your new clients are actually sending.

Authentication Failures Are Different from Connection Failures

For remote MCP servers, 401 and 403 are meaningful. Claude Code currently recognizes those responses as authentication-related states and can mark a remote MCP server as needing authentication. It also supports OAuth flows for remote MCP integrations. (Claude Code Docs)

This is different from connection refused and different again from server returned 500. A useful remote diagnostic path is:
DNS resolves?
|
v
TCP/TLS succeeds?
|
v
HTTP endpoint exists?
|
v
Authentication succeeds?
|
v
MCP protocol accepted?
|
v
Tool discovered?
|
v
Tool succeeds?
When all of that gets collapsed into "MCP handshake," teams end up adjusting protocol code to solve identity problems.

The Database Can Be Broken While MCP Is Healthy

The most important distinction for secure local database MCP projects: a real database connection refusal can happen at the final boundary, and that is not a local MCP transport error.
Claude Code
|
| stdio healthy
v
MCP server
|
| TCP connection fails
v
PostgreSQL
From the same environment where the MCP process runs, test PostgreSQL independently: psql "$DATABASE_URL" -c 'select 1;'. Then verify host, port, database name, TLS requirements, CA/client certificates, username, password, database role, Docker network, SSH context, WSL boundary, and firewall rules. Once that works, test the MCP tool. This separation prevents a simple database listener problem from turning into a protocol rewrite.

Cursor Has the Same Architectural Problem

Cursor also supports MCP and can hit many of the same subprocess, environment, and downstream-resource failures. Current Cursor uses agent as the primary CLI entry point; cursor-agent remains available as a backwards-compatible alias. Current CLI releases expose MCP management through /mcp commands. (Cursor CLI Changelog)

The important point is not that Claude Code and Cursor store every setting identically. Their local MCP failure tree is structurally similar: can host resolve command, can process start, can MCP transport work, can tool be discovered, can tool reach target. If a server works in one MCP client but not another, compare the host environment, configuration, and runtime location before rewriting the server.

Remote Development Changes What localhost Means

This is a classic distributed-systems trap. You configure postgresql://127.0.0.1:5432/audit. It works on your laptop. Then you run your MCP server in a remote development environment. Now 127.0.0.1 means the remote machine, not your laptop.

The same confusion can appear with Cursor Remote SSH, dev containers, Codespaces-like environments, WSL, Docker, and remote agents. Before investigating MCP, draw the topology:
Where does the AI client execute?
Where does the MCP process execute?
Where does the database execute?
What machine does "localhost" refer to?
That one diagram can save an hour.

Secure Database MCP Architecture

The easiest database MCP tool to expose is query(sql). It is also one of the broadest interfaces you can hand to an autonomous agent. Prefer narrower tools: get_recent_errors(service, since, limit), get_trace(trace_id), get_failed_jobs(since, limit). The tool schema becomes one security boundary.

Claude Code's own current MCP documentation uses PostgreSQL through DBHub as an example and explicitly recommends a read-only database user so Claude's queries cannot modify the database. (Claude Code Docs)
Claude Code
|
v
Approved MCP server
|
v
Narrow tool
|
v
Read-only DB role
|
v
Specific schema / views
Local stdio has a smaller remote attack surface than exposing a public HTTP listener. That does not make the server harmless. MCP's security guidance warns that local MCP servers can run with the privileges of the client process and recommends sandboxing and minimal default privileges. (MCP Security)

Claude Code's permission rules apply to tools, including MCP tools, before they execute. Claude Code's built-in OS sandbox applies to Bash commands and their child processes. It does not automatically mean every arbitrary MCP process has been placed inside the same operating-system sandbox. (Claude Code Sandboxing) Those are complementary controls. For a sensitive MCP server, launch it through an isolation mechanism appropriate to the threat model rather than assuming MCP permissions alone provide process containment.

The Enterprise Issue Is Larger Than One Developer Configuration

A developer sees Failed to connect. A CTO should see a new integration surface. MCP can connect AI agents to databases, source control, observability, CI/CD, cloud infrastructure, ticketing, communication systems, internal documents, and customer systems. That creates enormous productivity potential. It also means the AI tooling layer can become a new control plane across systems that were previously isolated from each other.

The enterprise problem is therefore not merely how to get MCP working. It is how to make MCP operable, governable, and auditable at scale.

A CTO-Level MCP Control Model

I would divide enterprise MCP governance into seven controls.

1. Server Provenance

Know who supplied each server. A production inventory should record server name, publisher, repository, version, transport, deployment location, business owner, technical owner, and review status. Installing an MCP package is equivalent to adding executable integration code to the environment. Treat it that way.

2. Capability Scope

Inventory not just servers but exposed tools. A server named database tells you almost nothing. These do: read_orders, create_ticket, deploy_service, delete_branch, run_sql. The security-relevant unit is the capability.

3. Identity and Authorization

Do not reuse application superuser credentials. Prefer agent identity, then MCP-specific service account, then minimum role, then minimum schema/API scope. For remote MCP, integrate authentication with enterprise identity rather than distributing long-lived secrets where possible.

4. Environment Isolation

Determine where MCP actually executes: developer workstation, remote host, container, enterprise gateway, or cloud worker. Then define filesystem, network, and secret boundaries accordingly.

5. Tool Approval Policy

Not every tool should have the same execution policy.
CapabilityDefault PostureRead documentationAuto-allowRead logsAuto-allow with data policyQuery read-only analyticsAuto/limitedCreate issueAllow with scopeModify repositoryReview depending on branchTrigger deploymentApprovalModify infrastructureStrong approvalWrite production databaseExceptionalDelete resourcesExplicit human approval
Cursor's current agent controls already apply approval logic to MCP tool calls, and Claude Code similarly supports tool-level permission controls. (Cursor Auto-review)

6. Auditability

For a sensitive MCP tool call, an enterprise should be able to reconstruct: who initiated the session, which agent/model was involved, which MCP server/version handled it, which tool was invoked, with which normalized parameters, which identity executed the downstream action, what resource was affected, what result was returned, and whether a human approval was required. Without that evidence, MCP may improve productivity while reducing accountability. That is a bad trade.

7. Lifecycle Governance

MCP servers change. Dependencies change. Tool schemas change. Permissions expand. Teams forget old integrations. Treat MCP like any other software supply chain: approve, version, monitor, review permissions, patch, retire.
Approve
|
v
Version
|
v
Monitor
|
v
Review permissions
|
v
Patch
|
v
Retire
Claude Code itself now supports managed organizational MCP configuration, including a centrally deployed server set and allowed/denied server controls. Cursor has similarly moved toward centrally distributed Team MCPs through organization marketplaces. That is where the enterprise market is heading: MCP is moving from individual developer configuration toward governed organizational infrastructure.

The Connection Error Can Be a Useful Security Signal

There is a tendency in development to treat every permission error as friction. MCP is one place where that instinct can be dangerous.

If the server cannot read a sensitive directory, the right fix is not automatically chmod -R 777. If the database rejects the agent, the right fix is not automatically giving it the application's production-owner credential. If the repository is waiting for MCP approval, the right fix is not automatically disabling workspace trust.

The acceptance criteria should be two things: integration works, and integration still has only the authority it needs. A fix that satisfies the first and breaks the second is not a production fix.

A Minimal MCP Server Is Your Best Diagnostic Tool

When a database MCP server refuses to connect, temporarily remove the database. The current stable MCP Python SDK is v2. Its high-level server class is MCPServer; the former v1 FastMCP name was changed as part of the v2 migration. (MCP Python SDK)
from mcp.server import MCPServer

mcp = MCPServer("transport-test")

@mcp.tool()
def health() -> dict[str, str]:
return {"status": "ok"}

if name == "main":
mcp.run()
With no transport argument, mcp.run() uses stdio. Prove Claude Code can reach health(). If that works, your transport is not the problem. Then add the next dependency. That turns one opaque failure into a sequence of small proofs.

Use MCP Inspector before blaming Claude Code: uv run mcp dev server.py. The Inspector launches the server and connects over stdio much like a real host would. (MCP Python SDK) If the Inspector cannot talk to the server, changing Claude Code's authentication or workspace settings is unlikely to fix the server implementation.

Production Configuration Pattern

{
"mcpServers": {
"audit-database": {
"type": "stdio",
"command": "/absolute/path/audit-mcp/.venv/bin/python",
"args": ["/absolute/path/audit-mcp/server.py"],
"env": {
"DATABASE_URL": "${MCP_AUDIT_DATABASE_URL}"
}
}
}
}
The desirable properties are not the exact filenames. They are: explicit transport, explicit runtime, explicit server path, no password committed to Git, dedicated credential, narrow database privileges, reproducible environment. Those characteristics make troubleshooting easier and reduce security ambiguity at the same time.

What Not to Deploy

Avoid configurations that accumulate years of unexplained workarounds:
{
"env": {
"PATH": "/usr/local/bin:/usr/bin:/bin",
"PYTHONPATH": "/random/dependencies",
"DATABASE_URL": "postgresql://admin:password@prod/db",
"MCP_TRANSPORT_MODE": "stdio",
"SOME_FLAG_FROM_A_GITHUB_ISSUE": "true"
}
}
The danger is not only that a setting is wrong. Future engineers cannot distinguish required architecture from historical superstition. A good MCP deployment should be explainable field by field. Do not add MCP_TRANSPORT_MODE=stdio unless your server actually consumes it. There is no universal MCP environment variable with that meaning. Do not add PYTHONPATH unless you can explain why. A reproducible Python server should have its dependencies installed into its environment.

The Troubleshooting Ladder

Identify stdio vs remote HTTP
Run claude mcp list
Resolve Pending approval / Rejected first
Confirm configuration type and scope
Confirm the executable exists (which / where.exe)
Use absolute runtime and server paths
Run the configured command manually
Confirm the process stays alive
Keep non-protocol output off stdout
Test a one-tool server (remove database)
Test with MCP Inspector (uv run mcp dev)
Check environment-variable warnings
Never print secrets while debugging
Check MCP_TIMEOUT only after startup correctness
Confirm MCP protocol/SDK generation
Inspect proxy/gateway behavior for remote MCP
Test the downstream database/API separately
Use a dedicated least-privilege identity
Prefer narrow tools over arbitrary privileged actions
Test one complete tool call from the real client ## Symptom-to-Root-Cause Mapping SymptomStart HerePending approvalWorkspace trust / project MCP approvalRejectedMCP approval settingsspawn ... ENOENTExecutable path / PATHOS permission deniedFile or directory permissionsProcess exits instantlyImport/config/runtime startupWorks in shell, fails in Claude CodeEnvironment, executable, cwd, secretsInspector cannot connectMCP server/transport implementationInvalid protocol datastdout contaminationSlow server eventually worksStartup initialization / timeoutMissing-variable warning.mcp.json environment expansionURL server treated incorrectlyMissing type: "http"Remote 401 / 403Authentication/authorizationMCP works, DB failsDatabase/network/credential layerWorks locally, fails remotelyRuntime topology / meaning of localhost The most useful way to think about an MCP connection failure is not Claude Code cannot connect, change MCP settings. It is: did Claude trust the configuration, could it launch or reach the server, did the transport work, did the protocol work, was the tool discovered, did identity and authorization work, did the downstream system work. Once those boundaries are separated, most MCP handshake failures become ordinary engineering problems with much smaller search spaces.

For CTOs, the broader conclusion: MCP is becoming an execution bridge between AI agents and enterprise systems. Connection reliability matters, but the real architectural requirement is controlled connectivity. The right process, using the right identity, reaching the right resource, with the smallest necessary authority and enough evidence to reconstruct what happened afterward.

The Broader Architectural Lesson

Model Context Protocol is often presented as a connector standard. That description is accurate but incomplete. Once an AI agent can use MCP to cross from reasoning into databases, source repositories, cloud accounts, deployment systems, ticketing, observability, and business applications, MCP becomes part of the organization's execution architecture.

That means MCP deserves the same disciplines we already apply to APIs and privileged integration services: identity, authorization, least privilege, change management, observability, supply-chain review, isolation, incident response, and auditability.

The protocol makes integrations easier to create. It does not make their authority less consequential.

Final MCP Connection Checklist

[ ] Identify stdio vs remote HTTP
[ ] Run claude mcp list
[ ] Resolve Pending approval / Rejected first
[ ] Confirm configuration type and scope
[ ] Confirm the executable exists
[ ] Use an absolute runtime path where practical
[ ] Use an absolute server path
[ ] Run the configured command manually
[ ] Confirm the process stays alive
[ ] Keep non-protocol output off stdout
[ ] Test a one-tool server
[ ] Test with MCP Inspector
[ ] Check environment-variable warnings
[ ] Never print secrets while debugging
[ ] Check MCP_TIMEOUT only after startup correctness
[ ] Confirm MCP protocol/SDK generation
[ ] Inspect proxy/gateway behavior for remote MCP
[ ] Test the downstream database/API separately
[ ] Use a dedicated least-privilege identity
[ ] Prefer narrow tools over arbitrary privileged actions
[ ] Test one complete tool call from the real client
[ ] Preserve approval, audit and rollback boundaries
If you are deploying MCP integrations and need an independent systems advisor to audit your agent-to-database control boundaries, schedule a strategic evaluation.

Originally published at subodhkc.com. Follow for more on AI governance, enterprise architecture, and compliance engineering.

What Is llms.txt? Why Every AI-Ready Website Should Publish It

Subodh Kc — Sun, 20 Sep 2026 06:14:01 +0000

The next important visitor to your website may never see your homepage.

It may be an AI coding assistant working inside Windsurf or Cursor. It may be a chatbot researching vendors, an automated compliance assessment, a procurement agent, a custom RAG application, or an AI assistant retrieving information for a user.

These systems do not navigate websites the way humans do. They extract information from JavaScript-heavy pages, incomplete navigation, duplicated marketing content, ambiguous links, or hundreds of URLs competing for limited context.

That is the problem llms.txt is designed to solve.

llms.txt is a lightweight Markdown file that gives AI systems a curated map of a website: what the site represents, which pages are authoritative, and where the most useful machine-readable content can be found.

My position: Websites already publish interfaces for humans, search engines, applications, accessibility tools, and APIs. Publishing an interface for AI agents is the logical next layer.

Published: July 25, 2026 · Last verified: July 25, 2026

What is llms.txt?

llms.txt is a proposed machine-readable publishing convention introduced by Jeremy Howard in September 2024. The file is placed at the root of a domain:

https://example.com/llms.txtIt provides:

A concise description of the website
Important context about the company, product, or project
Curated links to authoritative pages
Descriptions explaining what each page contains
Optional links that agents can skip when context is limited

The original proposal was created because language models increasingly rely on website information, while complex HTML, navigation, advertising, scripts, and limited context windows make websites inefficient to process. It uses Markdown because the format is readable by humans, language models, parsers, and traditional tools. [1]

A simple implementation looks like this:

Example Company

> Example Company provides AI vendor-risk assessment and governance tools.

Important context:

Assessments are based on publicly available evidence.
Results support, but do not replace, human security and legal review.
The methodology page is the authoritative source for scoring logic.

Core Resources

Product overview: Capabilities, intended users, and product limitations.
Assessment methodology: Evidence collection, scoring, and confidence methodology.
Security: Security architecture and data-handling practices.
Documentation: Implementation and integration guidance.

Optional

Company blog: Educational articles and company perspectives. This is not intended to replace the website. It is a machine-readable map of the website.

The argument for llms.txt starts with how AI systems work

An AI product such as ChatGPT is not simply one model reading the entire internet. A modern AI information pipeline contains several separate components:

A crawler or search index discovers content.
A retrieval system finds possible sources.
A ranking layer selects relevant pages or passages.
A context-building layer prepares information for the model.
The language model generates an answer based on the supplied context.

The language model is probabilistic. The retrieval path does not have to be.

A website owner cannot control every decision the model makes, but the owner can reduce ambiguity about:

Which pages exist
Which pages are important
Which source is canonical
What each page contains
Which content is optional
Where clean Markdown versions can be found

That is the real value of llms.txt.

OpenAI's own documentation separates its search crawler, OAI-SearchBot, from GPTBot and user-triggered agents. OAI-SearchBot surfaces websites in ChatGPT Search, while GPTBot is associated with content that may be used for model development. This illustrates the distinction between the model and the systems that discover and retrieve website information. [2]

llms.txt belongs primarily in that discovery and retrieval layer.

A deterministic map for a probabilistic consumer

The strongest way to understand llms.txt is not as an SEO file. It is a deterministic content interface.

Traditional websites make humans infer structure through navigation bars, dropdown menus, visual cards, footer links, buttons, page hierarchy, branding, and layout. An AI retrieval system may instead encounter:

Client-side rendering
Repeated navigation text
Multiple versions of similar pages
Weak internal links
Orphan pages
Tracking parameters
Unclear canonical sources
Large amounts of marketing text
Important content hidden behind interactive components

When the system cannot confidently determine what exists, it may search repeatedly, guess URLs, retrieve the wrong page, or miss the information entirely. Sites assembled by AI coding tools can carry the same opacity as a hidden SEO risk in AI-assisted frontend development.

llms.txt reduces that search space. It says: here is what this website represents, here are the authoritative resources, here is what each resource contains, and here are the pages you can ignore when context is limited.

That is a practical design pattern whether the consumer is ChatGPT, Claude, a coding agent, an internal copilot, or a custom application.

What happened in my AI Vendor Risk Assessment tool

My support for llms.txt comes from building, not from following an SEO trend.

I built an AI Vendor Risk Assessment tool that analyzes company websites and collects evidence related to security, privacy, AI usage, data retention, subprocessors, compliance frameworks, model providers, governance controls, and contractual limitations.

During testing, the retrieval workflow repeatedly failed to find important pages. This occurred even on some of my own websites, where I already knew the pages existed.

The pages were public and accessible. A human could eventually locate them. But some were buried in navigation, weakly linked, or difficult for the scraper to identify as authoritative.

I added those pages to llms.txt and configured the retrieval process to consult the file. The number of missed pages and unnecessary navigation attempts dropped.

That result does not depend on waiting for a search engine to adopt a new ranking rule. It demonstrates an immediate architectural use case: when an AI application is designed to consult llms.txt, the file makes website discovery more deterministic.

The same principle applies to vendor assessments, compliance scanners, support agents, product-research agents, documentation assistants, procurement systems, IDE assistants, and custom RAG applications. A website that provides an authoritative map is easier to consume than one that requires every system to reconstruct its structure independently.

The evidence: agents perform better when they are given a map

In July 2026, Mintlify published an open-source benchmark comparing four ways of serving documentation to AI agents: HTML, plain Markdown, Markdown linking to llms.txt, and Markdown with the complete llms.txt file embedded.

The benchmark covered 2,400 runs across 20 documentation sites using Claude Code and Codex.

When agents received a link to llms.txt, average failed URL requests dropped from 2.23 per task with HTML to 0.11. That represents roughly 90% fewer dead URLs, alongside fewer wasted fetches and reduced token usage.

Answer accuracy remained broadly similar. The improvement was navigation efficiency: agents reached the correct information faster and with less waste. Mintlify replicated the pattern across additional models, where failed requests again fell close to zero. The benchmark and implementation were published openly. [3]

The practical finding: Markdown alone was not enough. The map was the valuable part.

Chrome is already aligning the web toward agents

The strongest signal is not a prediction from an SEO influencer. It is Chrome incorporating agent-readiness into Lighthouse.

Chrome's Agentic Browsing documentation now describes llms.txt as an emerging convention that provides a machine-readable summary of a website for language models and AI agents. Chrome states that without the file, agents may spend more time crawling a website to understand its structure and primary content. Lighthouse now checks whether the file can be retrieved and whether it follows the proposed format. [4]

This matters because Chrome is not treating agent compatibility as a theoretical marketing topic. It is beginning to operationalize it as a website-quality concern.

The broader Agentic Browsing work also evaluates accessibility for agents, layout stability, machine-readable form descriptions, registered WebMCP tools, and agent interaction reliability. The direction is clear: future websites will not only be read by agents — they will increasingly be operated by them.

Major developer platforms are building AI-readable documentation

The convention is being adopted by platforms whose documentation is heavily consumed by coding agents.

Vercel publishes llms.txt and llms-full.txt, teaches developers how to implement both, and describes llms.txt as the selective navigation layer while llms-full.txt provides complete documentation context. Vercel specifically recommends using its machine-readable resources with ChatGPT, Claude, Gemini, Cursor, and Windsurf. [5]

Cloudflare documentation now exposes site-level llms.txt, product-level llms.txt, llms-full.txt, Markdown versions of individual pages, content negotiation for machine consumers, and agent-specific guidance embedded into documentation pages. Cloudflare's pages explicitly direct agents away from context-heavy HTML and toward Markdown and the relevant llms.txt index. [6]

Mintlify automatically generates both files, hosts them at root and .well-known locations, and advertises them using HTTP response headers so agents can discover them without already knowing their paths. [7]

This is what first-mover adoption looks like. It starts with developer documentation, where machine consumption is obvious, and expands into other information-heavy industries.

llms.txt is not only for developer documentation

Documentation is the first clear use case, not the last. The original proposal explicitly identifies possible applications across software projects, businesses, legislation, personal websites, ecommerce, education, and professional profiles. [1]

SaaS platforms

An AI agent can quickly identify product capabilities, pricing, integrations, security documentation, API references, limitations, and changelogs.

Compliance and vendor-risk platforms

A machine-readable map can identify assessment methodologies, framework mappings, security controls, evidence requirements, privacy policies, data-retention rules, subprocessors, and product limitations.

Ecommerce websites

Agents can locate product categories, return policies, warranty information, shipping policies, product specifications, and compatibility information.

Professional and personal websites

An agent can understand professional background, areas of expertise, projects, publications, speaking topics, services, and contact information.

Universities and public institutions

AI systems can locate programs, admission requirements, courses, policies, research, and public resources.

The underlying need is the same: reduce the cost and ambiguity of machine retrieval.

What about sites whose llms.txt files receive no traffic?

A large Ahrefs server-log study found that most published llms.txt files received no requests during the study period. It also found that AI systems generally did not probe blindly for a missing /llms.txt file. When files were fetched, agents often reached them through a link, an index, or an explicit instruction. [8]

That finding strengthens the implementation case rather than eliminating it. The correct lesson is: publishing the file silently is incomplete. Discovery must also be designed.

A sitemap that is never submitted or linked is less useful. An API that nobody knows exists receives no calls. A security policy hidden in an unlinked PDF will not guide an assessment. The same principle applies to llms.txt.

A complete implementation should:

Publish /llms.txt.
Publish /llms-full.txt where appropriate.
Link the files from documentation and relevant HTML pages.
Advertise them through HTTP Link headers.
Provide Markdown versions of important pages.
Reference the file in agent instructions and integration documentation.
Configure custom agents and scrapers to check it first.
Monitor server logs and downstream retrieval behavior.

The future advantage will not go to the site that merely owns the file. It will go to the site that builds the entire machine-readable content path.

llms.txt versus llms-full.txt

The easiest way to understand the distinction: llms.txt is the map. llms-full.txt is the compiled reference.
FilePrimary functionBest usellms.txtLists and describes authoritative resourcesSelective discovery and navigationllms-full.txtCombines page content into one fileFull-context ingestionIndividual .md pagesProvides clean content for one pageTargeted retrievalsitemap.xmlLists indexable website URLsSearch-engine discoveryrobots.txtCommunicates crawler access preferencesAccess governanceStructured dataDescribes entities and page meaningSearch and machine interpretationMCP serverExposes live resources and toolsDynamic agent access and actions

Use llms.txt when:

The website has many pages.
The agent should retrieve only what it needs.
Context efficiency matters.
Content changes frequently.
Individual Markdown endpoints are available.

Use llms-full.txt when:

The content set is reasonably contained.
A user or agent needs a complete reference.
The file will be attached to an AI project.
The content supports documentation, compliance, or research.
One-fetch ingestion is more useful than selective retrieval.

For larger sites, provide both.

The official llms.txt structure

The original proposal defines a specific order: [1]

Optional byte-order mark
H1 containing the site or project name
Blockquote containing a short summary
Optional paragraphs or lists providing context
H2 sections containing lists of links
Optional ## Optional section for secondary resources

Only the H1 is required, but the descriptions and curated sections create most of the practical value.

Enhanced production template

Company or Project Name

> A concise, factual explanation of what the company, website, or project does.

Canonical domain: https://example.com
Primary language: en-US
Last reviewed: 2026-07-25
Content scope: Public product, technical, security, and policy information.

Important context:

State what the product does.
Identify the intended user.
Explain which page is authoritative for methodology or product claims.
State relevant limitations.
Clarify whether the content is informational, contractual, technical, or legal.
Avoid promotional claims that cannot be independently supported.

Company and Product

Company overview: Company identity, leadership, purpose, and operating scope.
Product overview: Product capabilities, intended users, and limitations.
Use cases: Supported applications and implementation scenarios.

Methodology

Methodology: How evidence, analysis, and results are produced.
Scoring: Scoring rules, confidence levels, and interpretation guidance.
Limitations: Known exclusions and conditions affecting results.

Security and Privacy

Security: Security architecture and control environment.
Privacy: Personal-data collection, use, retention, and rights.
Subprocessors: Third-party service providers and their roles.
Data retention: Retention and deletion practices.

Technical Documentation

Getting started: Initial setup and onboarding.
API reference: Authentication, endpoints, requests, and errors.
Architecture: System components and data flows.
Changelog: Current releases and deprecations.

Commercial and Support

Pricing: Current plans and included capabilities.
Support: Support channels and escalation procedures.
Contact: Official company contact information.

Optional

Blog: Educational content and company perspectives.
News: Announcements and media coverage.
Case studies: Examples of reported customer outcomes. ## Recommended llms-full.txt structure

llms-full.txt has become a widely implemented companion convention, although the original proposal more specifically describes generated context files named llms-ctx.txt and llms-ctx-full.txt. [1]

A production-quality llms-full.txt should contain cleaned, authoritative content rather than an uncontrolled dump of every page.

Company Name: Full AI-Readable Reference

> Consolidated public documentation for AI assistants, agents, retrieval systems, and human readers.

Canonical domain: https://example.com
Generated: 2026-07-25T12:00:00-05:00
Content version: 1.0
Language: en-US
Source index: https://example.com/llms.txt
Scope: Public authoritative content only

Company Overview
Product Overview
Methodology
Security
Privacy
Compliance
Technical Documentation
Pricing and Support
Limitations
Frequently Asked Questions

Company Overview

Source: https://example.com/about
Last updated: 2026-07-20

[Clean Markdown content from the authoritative company page.]

Product Overview

Source: https://example.com/product
Last updated: 2026-07-22

[Complete product description, capabilities, intended users, and exclusions.]

Methodology

Source: https://example.com/methodology
Last updated: 2026-07-24

[Complete methodology, evidence hierarchy, scoring rules, confidence levels,
human-review requirements, and known limitations.]

Exclude from llms-full.txt

Do not include: cookie banners, navigation, repeated footers, tracking parameters, search-results pages, duplicate landing pages, outdated announcements, empty category pages, private customer content, security secrets, unreviewed user-generated content, or unsupported promotional claims.

The goal is not maximum volume. The goal is maximum useful signal per token.

Related Resources

Explore AI Systems Research

llms.txt should be part of a broader AI SEO and GEO stack

Generative Engine Optimization should not be reduced to one file. A strong AI-readable website includes several coordinated layers:

1. Crawlability

Important pages must return clean server responses, use canonical URLs, and remain accessible to the crawlers and agents you intend to support.

2. Clear entities

The company name, product names, founder identity, categories, locations, and capabilities should remain consistent across the site and external references.

3. Extractable answers

Pages should contain direct definitions, factual statements, tables, FAQs, limitations, and concise summaries that can be accurately quoted.

4. Original evidence

Research, methodologies, benchmarks, frameworks, case studies, and firsthand experience give AI systems a reason to use the site as a source rather than merely summarize it.

5. Third-party authority

Independent references, media coverage, directories, GitHub repositories, reviews, community discussion, and citations strengthen the broader entity record.

6. Structured data

Use appropriate Schema.org markup for articles, organizations, people, software applications, products, FAQs, breadcrumbs, and services.

7. Machine-readable resources

Publish llms.txt, llms-full.txt, Markdown page versions, OpenAPI specifications, RSS feeds, XML sitemaps, structured data, and MCP resources where applicable.

8. Measurement

Monitor AI crawler requests, user-triggered agent requests, access to .md pages, access to llms.txt and llms-full.txt, brand mentions, AI citations, retrieval accuracy, missed-page rates, failed URL requests, and downstream conversions.

SEO makes content discoverable in search. GEO makes content understandable and usable inside generated answers. Agent readiness makes the website navigable and actionable by machines. llms.txt sits at the intersection of all three.

Traffic is no longer limited to search-result clicks

The traditional journey was: search query → search result → website visit.

The emerging journeys look different:
Question in an IDE â†’ documentation retrieval â†’ product adoption

Chatbot request â†’ source retrieval â†’ brand recommendation

Vendor-risk assessment â†’ policy discovery â†’ procurement review

AI agent â†’ product documentation â†’ API implementation

Internal copilot â†’ knowledge retrieval â†’ employee action
Some of these interactions produce website visits. Others produce brand exposure, product inclusion, technical adoption, vendor shortlisting, evidence collection, API usage, agent actions, or later human visits.

Vercel already positions llms-full.txt as context that users can provide to ChatGPT, Claude, Gemini, Cursor, and Windsurf. Cloudflare actively routes machine consumers toward Markdown and site-level documentation indexes. [9]

The website is no longer only a destination. It is becoming a knowledge source inside other interfaces.

The first-mover advantage

The first-mover advantage is not simply having a file before competitors. It is gaining experience with the machine-facing layer of your website.

Early adopters can begin learning which pages agents fail to discover, which descriptions improve retrieval, which file structures reduce errors, which content is frequently requested, whether agents prefer individual pages or complete context, how quickly machine-facing information becomes stale, how AI systems interpret company and product entities, how agent traffic differs from search traffic, how to secure machine-facing instructions, and how to connect static content with APIs and MCP tools.

The implementation cost is low when the files are generated from an existing content source. The strategic upside is asymmetric. If adoption develops slowly, the organization still gains a structured, curated content map usable by its own applications. If adoption accelerates, the organization already understands how to publish, monitor, secure, and optimize information for agents.

Chrome, Vercel, Cloudflare, Mintlify, coding agents, and custom AI applications are already moving in that direction. [4]

How to implement llms.txt correctly

Publish both files

/llms.txt
/llms-full.txt

Provide Markdown pages

/about.md
/product.md
/security.md
/docs/getting-started.md

Advertise the files through HTTP headers

Link: </llms.txt>; rel="llms-txt", </llms-full.txt>; rel="llms-full-txt"
X-Llms-Txt: /llms.txt
Mintlify currently uses this approach to improve file discovery. [7]

Link from visible pages

Add an "AI and Agent Resources" section to documentation footers, API pages, developer resources, security pages, GitHub READMEs, and integration instructions.

Configure your own applications to use it

A custom retrieval workflow should:

Check /llms.txt.
Parse its sections and descriptions.
Identify relevant authoritative pages.
Retrieve the required Markdown content.
Fall back to a sitemap or conventional crawl when needed.
Record failed and successful retrieval paths.

Version-control it

Treat llms.txt as production content. Monitor for unauthorized changes, broken links, stale claims, redirects, unexpected external links, prompt-injection language, and conflicts with human-facing pages.

How to test whether llms.txt is working

Run a controlled comparison.

Baseline

Ask an agent questions requiring information from deeply nested pages without providing llms.txt.

Intervention

Give the same agent access to llms.txt.

Measure

Track relevant-page discovery, failed URLs, number of pages fetched, retrieval time, input tokens, answer completeness, source accuracy, stale information, and human correction required.

Repeat

Use multiple models, multiple question phrasings, and multiple runs. Generative answers vary. Your evaluation method should not depend on one prompt or one successful response.

Frequently asked questions

What is llms.txt?

llms.txt is a Markdown file that gives AI agents and retrieval systems a curated overview of a website and links to its most important resources.

Is it llm.txt or llms.txt?

The proposed filename is llms.txt, with an "s." The singular llm.txt is a common typo and search variation.

What is llms-full.txt?

llms-full.txt consolidates the substantive content of multiple pages into one machine-readable file that can be supplied directly to an AI assistant or retrieval system.

Does llms.txt replace sitemap.xml?

No. A sitemap broadly lists indexable URLs. llms.txt curates important resources, explains what they contain, and can direct agents toward Markdown versions designed for efficient retrieval.

Does llms.txt replace robots.txt?

No. robots.txt communicates access preferences. llms.txt communicates context and resource structure.

Is llms.txt useful for AI SEO and GEO?

Yes, as part of a broader machine-readable publishing strategy. It helps agents and retrieval systems understand which content exists and where authoritative information can be found.

Will llms.txt help AI agents navigate a website?

Evidence from a 2,400-run Mintlify benchmark showed that providing agents with a link to llms.txt substantially reduced failed URL requests and retrieval waste. [3]

Should a small website create llms.txt?

Yes, when the website contains important product, professional, policy, compliance, or documentation content. A small file can be maintained with minimal effort.

Should every website create llms-full.txt?

Not necessarily. It is most valuable when the authoritative content can be consolidated without creating an excessively large or repetitive file.

How often should the files be updated?

Update them whenever important pages, product capabilities, policies, methodologies, pricing, or canonical URLs change. Automatic generation from the site's source content is preferable.

How do agents discover llms.txt?

Agents may find it through direct links, HTTP headers, documentation instructions, a user-provided URL, custom application logic, or platform integrations. Publishing the file and actively exposing it is more effective than leaving it unlinked.

Can llms.txt contain instructions?

It can contain factual interpretation guidance and context. Avoid manipulative language, unsupported claims, or instructions attempting to override an agent's policies.

Is llms.txt only for developers?

No. It can support ecommerce, education, professional profiles, SaaS products, compliance systems, legislation, public institutions, and any website whose content may be consumed by AI applications.

Conclusion

The web was originally built for humans. Search engines added another audience, and websites responded with robots.txt, XML sitemaps, canonical tags, metadata, and structured data.

AI agents are now becoming another audience. They operate through chatbots, IDEs, custom applications, retrieval systems, compliance tools, procurement workflows, and agentic browsers. They need clean content, clear authority, efficient discovery, and less ambiguity.

llms.txt gives them a map. llms-full.txt gives them a consolidated reference.

The value is not that one file magically changes a probabilistic model. The value is that it makes the information supplied to that model more deliberate.

My own vendor-risk assessment tool found pages more reliably after I introduced a curated llms.txt path. Mintlify's benchmark found that agents reached documentation with far fewer failed requests when given the same kind of map. Chrome now recognizes the convention as part of its Agentic Browsing work. Major developer platforms are actively publishing and teaching machine-readable documentation.

The signals are moving in one direction. Websites are becoming interfaces for machines as well as people. The companies that begin designing that interface now will understand it before everyone else is forced to catch up.

A deterministic map for probabilistic systems is not a gimmick. It is good infrastructure.

Author

Subodh KC is an AI Systems Architect and Governance Expert. Former Fortune 100 AI Strategy CTL. Founder of HAIEC — integrated AI Ethics & Compliance. 16+ years building production AI systems from startups to global enterprise. His work includes LLMVerify, the HAIEC compliance platform, and an AI Vendor Risk Assessment system designed to discover, analyze, and validate public evidence about AI vendors.

Sources

Originally published at subodhkc.com. Learn more about the HAIEC AI Governance Platform. Follow for more on AI governance, enterprise architecture, and compliance engineering.

How to Detect Cross-Tenant Data Leakage in MCP Servers and Multi-Tenant SaaS

Subodh Kc — Fri, 07 Aug 2026 03:42:01 +0000

The Hidden Security Gap in Multi-Tenant MCP Servers

When you build a multi-tenant SaaS application or an MCP (Model Context Protocol) server that serves multiple organizations, cross-tenant data leakage is one of the most dangerous vulnerabilities you can ship. A single missing organizationId filter in a database query can expose one tenant's data to another — and traditional security scanners like Snyk, Semgrep, and CodeQL don't catch these patterns.

That's why I built mcp-tenant-isolation — a static analysis scanner with 57 deterministic rules specifically designed to catch tenant isolation failures in multi-tenant codebases.

What Is Tenant Isolation?

Tenant isolation ensures that data belonging to one organization (tenant) is never accessible to another. In a multi-tenant SaaS app, every database query, cache read, and file access must be scoped to the current tenant's organizationId.

The most common failure looks like this:

// VULNERABLE: No organizationId filter
const users = await prisma.user.findMany({
  where: { role: 'admin' }
});

// SECURE: Tenant-scoped query
const users = await prisma.user.findMany({
  where: { role: 'admin', organizationId: ctx.orgId }
});

It looks obvious in isolation. But in a codebase with 100+ API routes, dozens of lib functions, and complex middleware chains, missing tenant filters are easy to miss in code review and impossible for traditional SAST tools to detect.

Why Traditional Scanners Miss This

Tools like Snyk and Semgrep are excellent at detecting:

SQL injection
XSS
Dependency vulnerabilities
Secret leakage

But they don't understand tenant context. They don't know that organizationId is the tenant boundary. They don't track which functions require tenant guards. They can't tell you that prisma.user.findMany({ where: { role: 'admin' } }) is missing a critical tenant filter.

mcp-tenant-isolation fills this gap with 57 rules across 7 categories:

Rule Categories

Category	Rules	What It Detects
Database Queries (DBQ)	12	Missing `organizationId` in `findMany`, `findUnique`, `updateMany`, `deleteMany`, `groupBy`, `aggregate`
Schema (SCH)	6	Missing tenant columns, missing composite indexes, missing RLS policies
File Storage Isolation (FSI)	4	Unscoped S3/Blob keys, missing tenant prefix in file paths
Cache Key Scoping (CKS)	5	Redis/memoization keys without tenant prefix
IDOR	8	Insecure direct object references — accessing resources by ID without ownership check
Logging (LOG)	6	Missing tenant context in structured logs
MCP-Specific (MCP)	15	Tool visibility, cache prefix, session binding, credential vault isolation, prompt injection surface

Quick Start

Install

npm install -g mcp-tenant-isolation

Scan your codebase

mti scan ./src

Output

The scanner produces a clear pass/fail verdict with detailed findings:

╔══════════════════════════════════════════╗
║  MCP Tenant Isolation Scanner v1.6.2     ║
║  Verdict: FAIL                           ║
╚══════════════════════════════════════════╝

Findings: 47 active, 12 suppressed, 3 baseline

DBQ-001  HIGH   app/api/users/route.ts:23
         Missing organizationId in prisma.user.findMany()
         Remediation: Add organizationId to the where clause:
           where: { ..., organizationId: ctx.orgId }

IDOR-003 HIGH   app/api/documents/[id]/route.ts:45
         No ownership check after findUnique by ID
         Remediation: Verify result.organizationId === ctx.orgId

CI/CD Integration

GitHub Actions (Pre-built Action)

- uses: subodhkc/mcp-tenant-isolation@v1
  with:
    path: ./src
    format: sarif

Manual npx

- name: Run tenant isolation scan
  run: npx mcp-tenant-isolation scan ./src --format sarif --output scan.sarif

- name: Upload to GitHub Code Scanning
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: scan.sarif

SARIF output integrates directly with GitHub Advanced Security — findings appear in your repo's Security tab alongside CodeQL results.

MCP Server Integration

The package includes an MCP server with 4 tools that AI agents (Claude Desktop, Cursor, Cline) can use:

{
  "mcpServers": {
    "mcp-tenant-isolation": {
      "command": "npx",
      "args": ["-y", "mcp-tenant-isolation", "mcp"]
    }
  }
}

Tools available:

scan — Run tenant isolation scan on a codebase
rules — List all 57 rules with descriptions and remediation hints
config — Get/set scanner configuration
version — Get scanner version info

This means you can ask Claude: "Scan my src directory for tenant isolation issues" and get structured findings with remediation hints — directly in your chat.

Supported Frameworks

ORMs: Prisma, Drizzle, raw SQL (Knex, pg, mysql2)
Web frameworks: Next.js (App Router & Pages), Express, Fastify
Cache: Redis (ioredis, redis), in-memory memoization
Storage: S3, Vercel Blob, local filesystem
Auth: NextAuth, custom JWT, session-based

Output Formats

Format	Use Case
Terminal	Local development with pass/fail verdict
JSON	Programmatic consumption, custom dashboards
SARIF 2.1.0	GitHub Code Scanning, Azure DevOps
AI JSON	AI agent consumption with remediation hints
Markdown	Shareable reports for PRs and team review

Suppression and Baselines

Not every finding is actionable immediately. The scanner supports:

Suppressions — .mtirc.json config to suppress specific findings with reasons
Baselines — Mark existing findings as known debt, only fail CI on new issues
Rule packs — Enable/disable rule categories per project

{
  "suppress": [
    {
      "rule": "DBQ-001",
      "file": "app/api/public/badges/route.ts",
      "reason": "Public route, no tenant context needed"
    }
  ]
}

Real-World Results

I ran the scanner against a production Next.js SaaS codebase with 200+ API routes:

47 active findings — including 23 missing tenant filters, 8 IDOR risks, 5 unscoped cache keys
12 suppressed — public routes with legitimate no-tenant-context access
3 baselined — known debt scheduled for next sprint
Scan time: 4.2 seconds for 50,000+ lines of TypeScript

Why This Matters for MCP Server Developers

If you're building MCP servers that serve multiple tenants (organizations, teams, users), tenant isolation is the security boundary. The MCP protocol doesn't enforce tenant isolation — it's up to your implementation.

The 15 MCP-specific rules check for:

Tool visibility — Are all tools visible to all tenants? Should some be org-scoped?
Cache prefix — Is the MCP response cache keyed by tenant?
Session binding — Is the MCP session bound to a specific tenant?
Credential vault — Are per-tenant credentials isolated in the vault?
Prompt injection surface — Does the server expose tenant context to prompt injection?

Production RAG Architecture Patterns for Hybrid Search

Subodh Kc — Fri, 31 Jul 2026 11:40:30 +0000

Introduction to Production RAG Architecture Patterns

In the evolving landscape of enterprise AI, ensuring robust governance and compliance while harnessing the power of hybrid search is crucial. The Retrieval-Augmented Generation (RAG) architecture serves as a bridge, integrating large language models with external knowledge sources, enabling organizations to deliver accurate and relevant AI-driven solutions effectively. This blog post delves into practical implementation strategies for establishing RAG architecture patterns tailored for hybrid search environments.

Understanding RAG Architecture

RAG architecture combines generative models with retrieval methods, allowing AI systems to access the latest information and provide contextually relevant responses. This synergy is particularly beneficial for enterprises looking to incorporate AI solutions while adhering to compliance regulations such as HIPAA and ISO standards.

Key Components of RAG Architecture

Retrieval Component: This part retrieves relevant documents or data from external sources, ensuring the generative model is grounded in accurate information.
Generation Component: The generative model processes the retrieved information to produce human-like text outputs.
Feedback Loop: Integrating user feedback to refine the model's performance over time, ensuring relevance and accuracy.

Implementing RAG Architecture Patterns

To successfully implement RAG architecture patterns in your hybrid search system, consider the following steps:

Step 1: Define Use Cases

Identify specific use cases where AI can add value. For instance, legal document automation, as discussed in our post on Legal Document Automation for New York Law Firms in 2026, can benefit from RAG architecture by ensuring compliance while generating accurate legal documents.

Step 2: Establish Data Sources

Determine the data sources necessary for retrieval. This may include internal databases, public APIs, or industry-specific knowledge bases. Ensure these sources comply with relevant regulations such as HIPAA for healthcare applications or AI governance frameworks.

Step 3: Choose the Right Technology Stack

Select the appropriate tools and frameworks to build your RAG system. Consider open-source libraries like Hugging Face’s Transformers for model implementation alongside robust retrieval systems such as Elasticsearch or Pinecone.

Step 4: Develop the Retrieval System

Build a retrieval system that efficiently processes queries. For instance, use vector databases to enhance search capabilities, ensuring quick access to relevant documents that can be fed into the generative model.

Step 5: Implement the Generation Component

Integrate a generative model that can interpret the retrieved data and produce accurate outputs. Train the model using domain-specific data to improve its understanding and contextual relevance.

Step 6: Ensure Compliance and Security

Implement strict data governance policies to adhere to compliance requirements such as HIPAA or ISO 42001. Regularly audit the system for vulnerabilities, incorporating security measures at every stage of the architecture.

Step 7: Monitor and Optimize Performance

Establish metrics to monitor the system's performance and implement a feedback loop. Use user analytics to refine both the retrieval and generation components, enhancing overall accuracy and relevance.

Practical Example: RAG in Legal Tech

Consider a legal tech firm leveraging RAG architecture to automate document generation. By integrating a retrieval system that sources legal precedents, the firm ensures compliance with regulations while reducing the time necessary for legal drafting. The generative model tailors documents based on the retrieved information, ensuring accuracy and legal soundness.

Conclusion and Takeaway

Establishing a robust RAG architecture for hybrid search can significantly enhance your enterprise AI initiatives. By following the outlined steps, CTOs, CISOs, and AI program leaders can implement a compliant and secure system that improves decision-making and operational efficiency. Start by defining your use cases today and choose the right technologies to drive your AI strategy forward.

FAQs

What is RAG architecture?

RAG architecture combines retrieval and generative models, enabling AI systems to access external information for accurate and contextually relevant outputs.

How can RAG architecture improve compliance?

By integrating real-time data retrieval with generative models, RAG architecture can ensure that outputs adhere to compliance standards, reducing legal risks.

What technologies are best for implementing RAG systems?

Open-source frameworks like Hugging Face’s Transformers and retrieval systems such as Elasticsearch are popular choices for implementing RAG architectures.

Can RAG architecture be applied in healthcare?

Yes, RAG architecture can enhance AI applications in healthcare by ensuring that model outputs comply with HIPAA regulations while providing accurate information.

How do I measure the performance of a RAG system?

Establish metrics such as retrieval accuracy, user satisfaction, and output relevance, and implement a feedback loop for ongoing optimization.

Originally published at subodhkc.com. Learn more about the HAIEC AI Governance Platform. Follow for more on AI governance, enterprise architecture, and compliance engineering.

Legal Document Automation for New York Law Firms in 2026

Subodh Kc — Thu, 30 Jul 2026 11:35:09 +0000

Legal Document Automation for New York Law Firms in 2026

Which legal document automation providers lead the New York market?

Three platforms stand out for New York law firms and legal departments evaluating automated legal documents in 2026: LEAP Legal Software, Litify, and Legal Outsourcing 2.0. Each addresses a distinct operational profile.

LEAP Legal Software integrates case management, document automation, legal accounting, billing, and AI legal tools into a single platform, purpose-built for small to medium law firms that need one system to run their practice.
Litify delivers a legal operations platform covering intake management, matter management, document management, time and billing, spend management, reporting, workflow automation, and AI-powered legal insights, targeting enterprise legal teams and larger firms.
Legal Outsourcing 2.0 takes a different path entirely: technology-enabled outsourcing combining experienced US-licensed attorneys with ISO 27001 certified infrastructure to handle litigation document review, contract extraction and analysis, and data breach notification review at pricing roughly half the cost of comparable US services.

The efficiency case for all three rests on the same foundation. Automated contract creation eliminates the manual drafting cycle, reduces transcription errors, and produces documents that carry a consistent clause structure across every matter. For New York firms operating under state-specific compliance requirements, that consistency is not a convenience. It is a liability control measure.

How do these providers compare on services, pricing, and fit?

New York legal professionals evaluating document automation software need to match vendor capabilities to their actual workflow, not just their wish list. The table below maps each provider across the dimensions that matter most for a purchasing decision.

Provider	Services Offered	Specialization	Pricing Model	Unique Positioning	Rating
LEAP Legal Software	Case management, document automation, legal accounting, billing, AI legal tools	Small to medium law firms, all-in-one practice management	Subscription per user	Single platform for full practice management with embedded AI	4.6★ (244 reviews)
Litify	Intake, matter management, document management, billing, spend management, reporting, workflow automation, AI insights	Enterprise legal teams, large law firms, legal departments	Enterprise subscription	Real-time workflow visibility with AI-driven legal analysis across complex operations	4.1★ (28 reviews)
Legal Outsourcing 2.0	Litigation document review, contract extraction and analysis, data breach notification review, legal innovation solutions	Legal process outsourcing, litigation support, contract review	All-in pricing, ~50% below US market rates	ISO 27001 certified; US-licensed attorneys plus technology for high-volume, compliance-grade review	5★ (1 review)

LEAP Legal Software suits firms that want to stop managing five separate tools. Its embedded AI handles document generation inside the same environment where attorneys track matters and run billing, which means no data re-entry and no context switching. For a 5–20 attorney firm in New York, that integration alone recovers meaningful time each week.

Litify is built for operations where workflow visibility matters as much as document output. Legal departments managing high case volumes across multiple practice lines benefit from its reporting and spend management modules, which give general counsel a real-time picture of where matters stand and what they cost.

Legal Outsourcing 2.0 fits a different buyer: firms that need high-volume document review or contract analysis without building internal capacity. Its ISO 27001 certification addresses the data security scrutiny that New York firms face under state bar ethics guidance on cloud and third-party data handling.

How do you implement legal document automation without losing control?

Deployment is where most automation projects either take hold or quietly collapse. The sequence matters.

Onboarding phases to plan for:

Template mapping (one-time): During initial setup, your existing clause language, conditional logic, and formatting standards get configured into the platform. This phase typically runs one to two hours and determines the quality of every document the system produces afterward.
Integration with existing systems: Connecting your law firm automation tools to your case management system, CRM, and billing platform creates a single source of truth. When client data updates in one place, all associated documents cascade automatically, eliminating the version-control problems that plague manual workflows.
Workflow building and testing: Before going live, map the decision logic for your most common document types. Test conditional clauses against real matter scenarios, not hypothetical ones.
Human review gates: Successful automation balances AI generation with governed workflows that route documents through attorney review before delivery. Every output needs an audit trail. A document that cannot be defended in a disciplinary proceeding is worse than one that was never automated.

Common pitfalls:

Firms that route every template change through IT create a bottleneck that kills adoption. Platforms built on low-code editors let attorneys update clause logic themselves, keeping the system current without a development queue.
Pricing for document automation software ranges from free AI generators for basic use to enterprise subscriptions with advanced workflow and integration features. Implementation costs and ongoing support fees are often underestimated in initial budgets.

Pro Tip: Before signing any vendor contract, ask specifically whether the platform processes your documents on private managed infrastructure or routes data through a shared public AI model. Attorney-client privilege does not survive on infrastructure that retains or trains on your client data. This is a non-negotiable due diligence question for any New York firm.

Subodhkc brings AI governance to legal automation

Legal document automation does not end at document generation. The harder problem is governing the AI that produces those documents: ensuring outputs are auditable, compliant with New York's AI-specific regulations including NYC Local Law 144, and defensible under professional responsibility standards.

Subodhkc addresses that layer directly. Where LEAP, Litify, and Legal Outsourcing 2.0 handle the document workflow, Subodhkc provides the AI governance architecture that sits beneath it: deterministic compliance frameworks, audit trail design, and risk controls that make AI-generated legal work defensible at the system level, not just the document level. Its HAIEC platform applies structured governance logic to AI operations, giving legal teams the technical evidence trail that regulators and bar associations increasingly expect.

For New York legal professionals who have already chosen or are evaluating a document automation platform, Subodhkc's advisory and governance services answer the question those platforms leave open: how do you prove the AI did what it was supposed to do? That is the accountability layer the market is still building toward, and it is where Subodhkc operates. Start with a governance assessment at subodhkc.com/haiec.

Key Takeaways

The strongest legal document automation strategy pairs the right platform with a governed AI architecture that produces auditable, defensible outputs.

Point	Details
Match provider to firm size	LEAP suits small to medium firms; Litify targets enterprise legal teams; Legal Outsourcing 2.0 fits high-volume review needs.
ISO 27001 certification matters	Legal Outsourcing 2.0 holds ISO 27001 certification, a concrete compliance signal for firms handling sensitive data under New York bar ethics guidance.
Template mapping is foundational	A one-time onboarding session configures clause logic and formatting; the quality of this phase determines every document the system produces.
Governed workflows protect defensibility	AI generation paired with human review gates and a full audit trail keeps automated documents compliant and professionally defensible.
Subodhkc adds the governance layer	Subodhkc's HAIEC platform provides deterministic AI governance and audit trail architecture for legal teams operating automated document workflows.