DEV Community: MrĐức Nguyen

Your LLM API Returns 200 OK. So Why Is Your AI Application Still Broken?

MrĐức Nguyen — Tue, 22 Sep 2026 16:29:40 +0000

AI applications rarely fail in the way traditional software fails.

Sometimes the API returns 200 OK — but the answer is wrong.

Sometimes latency looks acceptable — until one prompt suddenly consumes 10× more tokens.

Sometimes a RAG pipeline technically works — but retrieval quality quietly gets worse.

And sometimes users report:

“The AI feels different today.”

Where do you even start investigating?

That’s why I think AI observability is becoming one of the most important operational capabilities for production LLM systems.

Traditional monitoring tells you:

CPU
Memory
HTTP errors
Infrastructure availability

But production AI requires another layer of visibility:

Prompt → Model → Retrieval → Tokens → Latency → Cost → Output → User session

Without that context, debugging an AI application can quickly turn into guesswork.

Over the past months, I’ve spent a lot of time working with Langfuse and LLM observability for real AI environments — looking at traces, generations, sessions, metadata, latency, token usage, errors, RAG behavior, and operational security signals.

One lesson kept coming back:

Collecting traces is easy. Knowing what to monitor is harder.

So I organized that experience into the:

Langfuse AI Observability Bundle 2026

The goal is simple:

Help teams move from:

“We installed Langfuse.”

to:

“We actually know how to use observability to operate an AI system.”

The bundle is designed around practical questions such as:

What metadata should every AI trace contain?
How should applications, environments, users, and sessions be identified?
Which latency and token metrics actually matter?
How do you investigate abnormal LLM behavior?
How do you monitor RAG and retrieval workflows?
How can traces support incident investigation?
What should operations and security teams look for in production AI logs?
How do you build a repeatable observability workflow instead of manually opening random traces?

The bigger idea behind the bundle is that AI observability isn't just monitoring.

It sits at the intersection of:

LLMOps + Security + Reliability + Cost Management + Troubleshooting

And as AI systems move from experiments into production, that intersection becomes increasingly important.

A chatbot demo can survive with console logs.

A production AI service used by hundreds or thousands of users cannot.

If you're building or operating systems using:

LLM APIs
RAG
AI agents
Enterprise chatbots
Document AI
AI gateways
Multi-model applications

then observability should probably be designed into the architecture — not added only after the first production incident.

I packaged my notes, operational patterns, and reusable resources into the Langfuse AI Observability Bundle 2026 for engineers, AI teams, security practitioners, and organizations building production LLM systems.

You can check it out here:

https://techsavant013.gumroad.com/l/langfuse-ai-observability-bundle-2026

But regardless of whether you use the bundle, I’d strongly recommend asking one question before your next AI system goes live:

If this application starts producing bad answers tomorrow, will your team actually know why?

The Hidden Cost of AI Automation Projects: Scope Creep Is Eating Your Margin

MrĐức Nguyen — Tue, 22 Sep 2026 16:15:34 +0000

The Hidden Cost of AI Automation Projects: Scope Creep Is Eating Your Margin

A client says:

“We just need a simple AI chatbot.”

You estimate the effort.

The price looks reasonable.

Everyone agrees.

Then development starts.

A few days later:

“Can it also connect to our CRM?”

Then:

“We need document uploads too.”

Then:

“Could we support another language?”

Then:

“Can you keep monitoring it after launch?”

None of these requests sounds unreasonable on its own.

But together, they can completely change the economics of the project.

And that taught me something important:

One of the hardest parts of AI automation isn't building the system. It's defining what you're actually agreeing to build.

The problem isn't always estimation

When an AI automation project loses margin, it's easy to blame inaccurate development estimates.

Sometimes that's true.

But I've found another problem to be just as important:

The original commercial boundary was too vague.

Consider a basic knowledge assistant.

The original project might assume:

One data source
One language
One interface
A defined number of documents
Basic testing
Initial deployment

But several weeks later the project may include:

CRM integration
Additional document sources
Multilingual support
More users
Increased LLM/API consumption
Monitoring
Prompt tuning
Additional testing
Ongoing support

Technically, each change might be small.

Commercially, the accumulated impact may be significant.

I started separating six decisions

Instead of treating a proposal as just a description of what I'll build, I started looking at it as six connected decisions.

1. Scope

What exactly will be delivered?

Not:

“Build an AI assistant.”

But something closer to:

“Build a knowledge assistant that retrieves information from the approved internal document repository and provides answers through a web interface.”

Specificity matters.

2. Exclusions

This is surprisingly important.

Examples:

Additional integrations
Custom mobile applications
Data cleansing outside an agreed volume
Additional languages
Third-party subscription fees
Production support after the agreed period

An exclusion isn't there to make the proposal defensive.

It's there to remove ambiguity.

3. Pricing assumptions

The implementation fee isn't the only number that matters.

For AI projects I also want to understand:

Development effort
Infrastructure cost
Model/API cost
Third-party platform fees
Expected support effort
Contingency
Target margin

This becomes especially important when recurring services are involved.

A project can look profitable at launch but become much less attractive after several months of support and increased usage.

4. Usage assumptions

AI systems have variable operating costs.

That's different from many traditional software projects.

Suppose your monthly price assumes a certain volume of:

LLM tokens
OCR pages
Automation executions
Vector database operations
Storage
External API requests

If usage triples, somebody absorbs that cost.

The proposal should make clear who.

5. Acceptance criteria

One of the most dangerous sentences in a project is:

“We'll know when it's finished.”

That's not an acceptance criterion.

Instead, define observable conditions.

For example:

Integration successfully connects to the agreed CRM environment.
Documents in supported formats can be processed.
Required workflow steps execute successfully.
Agreed test scenarios pass.
Client representatives complete acceptance testing.

The exact criteria depend on the project.

The important part is agreeing on them before delivery.

6. Change requests

This is where everything connects.

Imagine the original proposal includes one CRM integration.

The client later asks for a second platform.

You now have a simple question:

Does this request fall inside the agreed scope?

If not, it becomes a change request.

Then you can evaluate:

New request → additional effort → additional cost → updated timeline

That conversation is much easier than arguing about what everyone remembered from a meeting three weeks earlier.

A simple structure I now use

I've gradually settled on this workflow:

Client Brief

↓

Scope

↓

Exclusions

↓

Cost & Usage Assumptions

↓

Implementation + Recurring Pricing

↓

Acceptance Criteria

↓

Change Request Process

The individual pieces aren't revolutionary.

The value comes from connecting them.

For example:

If you change the scope, the estimated effort may change.

If effort changes, pricing changes.

If API usage changes, recurring cost changes.

If acceptance criteria change, implementation effort may change.

Everything is connected.

The $2,000 project that quietly becomes a $4,000 project

Imagine you've quoted an AI automation project at $2,000.

During delivery, the client requests:

Another integration
More document formats
Two additional revision rounds
Additional prompt tuning
Post-launch monitoring

Each request sounds like:

“Just one small change.”

Maybe each one really is small.

But five small changes can easily become another 20–30 hours of work.

If those hours aren't priced, your effective rate drops quickly.

That's why I increasingly think of scope as a financial control, not just project documentation.

The same problem exists with monthly services

Recurring AI services introduce another challenge.

Imagine charging:

$300/month

while your underlying monthly costs are:

LLM/API usage: $70
Automation platform: $40
Infrastructure: $25
Support effort: $60

Your contribution before other overhead is:

$105

Now imagine usage doubles.

API cost becomes $140.

Support increases to $90.

Suddenly the economics look very different.

A monthly service therefore needs more than a monthly price.

It needs assumptions.

This eventually became a toolkit

After repeatedly thinking through these same questions, I started turning the process into reusable documents and calculators.

Eventually that became something I call the:

AI Automation Proposal & Pricing Kit

It contains worked examples for projects such as:

Customer-support knowledge assistants
Lead intake and CRM automation
Invoice extraction with human review

I also built a small offline pricing lab for testing different assumptions around:

Implementation cost
Contingency
Margin
Monthly expenses
Usage growth
Discounts
Support

The goal wasn't to create another generic proposal template.

There are plenty of those already.

The goal was to connect:

Scope + pricing assumptions + recurring costs + acceptance + change management

into one workflow.

If you're curious, I've published the toolkit here:

https://techsavant013.gumroad.com/l/ai-automation-proposal-pricing-kit

But the larger lesson is independent of any template:

Before pricing an AI project, price the uncertainty around it too.

The clearer the commercial boundary is before development starts, the easier almost every conversation becomes afterward.

I'm curious how others handle this

For freelancers, consultants, and agency owners working on AI automation:

Which part causes you the most difficulty?

Estimating implementation effort?
Scope creep?
API/LLM usage costs?
Pricing ongoing support?
Getting clients to approve change requests?

I'd be interested to compare approaches.