DEV Community: Serhii Panchyshyn

How to Roll Out an Internal AI Product Without Lying to Yourself

Serhii Panchyshyn — Mon, 13 Apr 2026 23:53:55 +0000

I've helped teams roll out AI products for the past two years.

The same failure pattern shows up almost every time.

They build something that demos well. Leadership gets excited. They ship it to 50 users in week one. Within two weeks, trust is destroyed and the project gets shelved 😅

The teams that succeed do something different. This is the playbook I walk clients through now.

The problem I see everywhere

Most teams measure AI rollouts wrong.

They track one number. "Accuracy" or "user satisfaction" or something equally vague. The number looks good. They ship broadly. Then real users hit edge cases, the agent hallucinates, and suddenly everyone thinks "AI doesn't work for us."

The issue isn't the AI. The issue is they never built the infrastructure to see what was actually happening.

You can't improve what you can't observe. And most teams can't observe anything.

The rollout framework that works

Here's what I advise now. Nine steps, usually 6-8 weeks before external users.

Step 1: Start with 3 users, not 30

Every team wants to move fast. "Let's get feedback from the whole department!"

I push back hard on this.

More users means more noise. You can't inspect every trace. You start pattern-matching on vibes instead of data.

The right first cohort:

3 people who actually need the tool for real work
Different roles (support, ops, sales)
Direct channel to the eng team

One client started with 30 users. Couldn't keep up. Rolled back to 5. Found more bugs in one week than the previous month.

// What I recommend tracking for each early user
interface EarlyUserContext {
  userId: string;
  role: string;           // "support", "ops", "sales"
  primaryUseCase: string; // "answer customer questions"
  feedbackChannel: string; // direct line to eng team
}

Step 2: Instrument everything before anyone touches it

This is where most teams cut corners. They want to ship. Observability feels like overhead.

It's not optional.

Before the first user session, you need to answer these questions from your traces:

What query did the user send?
What tools did the agent consider?
Which tool did it pick and why?
What context was in the window?
What was the final response?
Did the user accept, edit, or reject it?

I've seen teams ship without trace logging. They have no idea why things fail. They guess. They tweak prompts randomly. Nothing improves.

// Minimum viable trace structure
interface AgentTrace {
  runId: string;
  userId: string;
  query: string;
  toolsConsidered: string[];
  toolSelected: string;
  contextSummary: string;
  response: string;
  userFeedback: "accepted" | "edited" | "rejected" | null;
  latencyMs: number;
}

LangSmith, Langfuse, whatever. The tool matters less than having something.

Step 3: Review every trace for the first week

Yes, every single one.

This is where you learn what's actually broken. Not what you assumed was broken.

I sit with clients and review traces together. Same patterns show up:

Wrong tool selection: Agent picked searchOrders when it should have picked searchShipments
Missing context: Agent couldn't answer because the right doc wasn't retrieved
Hallucinations: Agent made up data that doesn't exist
Premature stopping: Agent gave up too early
Slow responses: Anything over 10 seconds feels broken

Create a simple spreadsheet. Log every failure. Categorize them.

Run ID	Failure Type	Root Cause	Fix
abc123	Wrong tool	Vague tool name	Renamed function
def456	Hallucination	No source doc	Added missing doc
ghi789	Slow response	Too much context	Scoped retrieval

After one week, you'll have a clear picture. This spreadsheet becomes your roadmap.

Step 4: Fix perception before prompts

Here's the insight that saves teams weeks of wasted effort:

90% of early failures come from three sources:

Bad tool names and descriptions
Missing or wrong context
Retrieval pulling irrelevant docs

These aren't prompt problems. They're perception problems.

I tell clients: the agent can only do the right thing if it can see the right things.

// Before: I see this constantly
const tool = {
  name: "handleData",
  description: "Handles data operations"
}

// After: Clear enough for the model to reason about
const tool = {
  name: "createShipmentFromOrder",  
  description: "Creates a new shipment record from an existing order. Requires orderId. Returns shipmentId and tracking number."
}

One client renamed 12 tools in week one. Tool selection accuracy went from 60% to 87%. No prompt changes.

Step 5: Build evals from your failures

Don't build generic evals. Build evals from the specific failures you observed.

Every row in that failure spreadsheet becomes a test case.

// Example eval case from a real client failure
const evalCase = {
  id: "shipment-status-check",
  query: "What's the status of order 12345?",
  expectedTool: "getShipmentByOrderId",
  expectedBehavior: "Return actual status from database",
  failureWeObserved: "Agent said 'delivered' without checking",
  groundTruth: "in_transit"
}

One team I worked with had 47 eval cases after two weeks. All from actual user sessions. All testing things that actually broke.

Generic benchmarks tell you nothing. Failure-driven evals tell you everything.

Step 6: Measure the right things separately

This is where most teams lie to themselves.

They compute one accuracy number. "We're at 85%!" Leadership is happy. But 85% of what?

I push clients to measure these separately:

interface AgentMetrics {
  // Did we pick the right tool?
  toolSelectionAccuracy: number;

  // Did we retrieve relevant docs?
  retrievalRecall: number;

  // Did the final answer match ground truth?
  answerCorrectness: number;

  // Did we cite the right sources?
  groundingAccuracy: number;

  // Did the user accept the response?
  userAcceptanceRate: number;
}

You can have 95% tool selection and 40% answer correctness. That means retrieval or synthesis is broken.

You can have 90% answer correctness and 60% user acceptance. That means the answer is technically right but useless in practice.

Separate metrics tell you where to focus. One number tells you nothing.

Step 7: Expand slowly with permission gates

After 2 weeks with 3 users, you might be ready for 10.

Don't flip a switch. Add gates.

const canUseAgent = (user: User): boolean => {
  // Phase 1: Named early adopters
  if (ROLLOUT_PHASE === 1) {
    return earlyAdopters.includes(user.id);
  }

  // Phase 2: Specific teams
  if (ROLLOUT_PHASE === 2) {
    return user.team === "support" || user.team === "ops";
  }

  // Phase 3: Everyone
  return true;
}

Each phase should last at least a week. Each phase needs its own baseline metrics.

If metrics drop when you expand, you've found a gap. That's good. That's the system working.

Step 8: Watch for drift

The first week is not representative.

Early users are curious. They ask simple questions. They're forgiving.

By week 4, they're using it for real work. Queries get harder. Edge cases appear. Patience drops.

I tell clients to track metrics weekly, not just at launch:

Week 1: 87% tool accuracy, 72% answer correctness
Week 2: 85% tool accuracy, 75% answer correctness  
Week 3: 83% tool accuracy, 71% answer correctness
Week 4: 79% tool accuracy, 68% answer correctness  ← investigate

If metrics drift down, dig into traces. Usually it's new use cases, missing docs, or users learning to ask harder questions.

Step 9: Know when you're actually ready

I've seen teams ship too early and destroy trust. I've also seen teams wait forever and never ship.

Here's what ready looks like:

✅ Tool selection accuracy > 90%

✅ Answer correctness > 80%

✅ User acceptance rate > 75%

✅ p95 latency < 8 seconds

✅ No hallucinations in last 100 traces

✅ You've handled the top 10 failure modes

Not ready:

❌ Still finding new failure categories weekly

❌ Metrics vary wildly day to day

❌ Users work around the agent instead of using it

❌ You can't explain why it fails when it fails

The outcome when this works

Teams that follow this playbook:

Ship with confidence, not hope
Have real data to show leadership
Know exactly where to focus engineering effort
Build user trust instead of destroying it

The teams that skip steps end up with shelved projects and skeptical users. I've seen it enough times to know.

The checklist

Week 0: Instrument everything
Week 1: 3 users, review every trace, build failure spreadsheet
Week 2: Fix perception issues (tools, context, retrieval)
Week 3: Build evals from failures, establish baselines
Week 4: Expand to 10 users, new roles, new use cases
Week 5: Fix new failures, update evals
Week 6: Expand to full internal team
Week 7+: Monitor drift, harden edge cases
When metrics stabilize: Consider external rollout

The boring work is the real work. Instrument first. Start small. Review everything. Fix perception before prompts. Measure the right things separately. Expand slowly.

Your agent is only as good as your willingness to watch it fail and fix what you find.

If you're rolling out an AI product and want a second set of eyes on your approach, I help teams get this right. DM me on X or LinkedIn.

Stop Prompting. Start Engineering Perception.

Serhii Panchyshyn — Mon, 13 Apr 2026 23:43:02 +0000

I've watched teams spend weeks rewriting the same system prompt.

Different phrasings. More examples. Clearer instructions. The agent still picks the wrong tool. Still hallucinates. Still feels broken.

Then they rename six functions and accuracy jumps 30%.

This pattern shows up constantly. The model doesn't care how clever your prompt is. It cares about what it can see.

The problem I see everywhere

Teams treat prompts like magic spells. Say the right words, get the right output.

But agents aren't following instructions. They're making predictions based on everything in context. The tool names. The API responses. The error messages. The structure of your data.

That's perception. And it matters way more than your system prompt.

Most teams optimize the wrong layer. They iterate on prompts for weeks while their tool names are handleData and processRequest. The model has no chance.

Here are 10 patterns I've seen work across the past two years of helping teams build production agents 💪

1. Tool names are the real prompt

Bad tool names are invisible to the model.

I audit client codebases and find this constantly:

// ❌ The model has no idea what this does
async function handleRequest(data: unknown) { }

// ✅ Now it knows exactly when to use this
async function createShipmentFromOrder(orderId: string) { }

One client had 47 tools. Half had names like processData or executeAction. The model was guessing.

We renamed 12 functions. Tool selection accuracy went from 60% to 87%. No prompt changes.

2. Tool descriptions matter more than you think

The model reads descriptions to decide which tool to pick.

I tell clients: write descriptions like you're onboarding a new developer. Because you are.

// ❌ Vague description
const tool = {
  name: "searchRecords",
  description: "Search for records in the system"
}

// ✅ Specific description with constraints
const tool = {
  name: "searchShipments",
  description: "Search shipments by tracking number, origin, destination, or date range. Returns max 50 results. Use filters to narrow results before searching."
}

Specific descriptions reduce wrong tool selection by 30-40% in my experience.

3. Passing everything into context is lazy

I've reviewed architectures where teams dump entire conversation histories into context. 20 turns. 50 tool results. Everything.

The model drowns.

What works:

Last 3 turns by default
Relevant retrieved docs only
Structured summaries instead of raw data

Less context. Better decisions. Faster responses.

One team cut their context by 60% and saw answer quality improve. Counter-intuitive until you realize the model was distracted by noise.

4. Scoped retrieval beats broad retrieval

Early RAG implementations pull from everywhere. The whole knowledge base. 200+ docs. The model has no idea which ones matter.

I push clients toward module-level filtering. If someone asks about shipments, only retrieve shipment docs.

// ❌ Retrieve from everything
const docs = await retriever.search(query);

// ✅ Scope to relevant module
const docs = await retriever.search(query, { 
  module: detectModule(query),
  maxResults: 5 
});

Recall goes up. Hallucinations go down. Should be the default from day one.

5. Structured outputs prevent downstream chaos

If another agent or system consumes your output, structure it.

// ❌ Free text response
"I found 3 shipments that match. The first one is #12345 going to Chicago..."

// ✅ Structured response
{
  "shipments": [
    { "id": "12345", "destination": "Chicago", "status": "in_transit" },
    { "id": "12346", "destination": "Denver", "status": "delivered" }
  ],
  "total": 3,
  "hasMore": true
}

Unstructured responses compound errors. Each downstream consumer has to parse and guess. I've seen entire pipelines break because one agent returned prose instead of JSON.

6. Silent failures are invisible failures

The model can't fix what it can't see.

I audit error handling in every client codebase. Same pattern:

// ❌ Silent failure
if (!hasPermission) {
  return null
}

// ✅ Loud failure
if (!hasPermission) {
  return {
    error: "PERMISSION_DENIED",
    message: "User lacks 'shipments.create' permission",
    requiredPermission: "shipments.create",
    suggestedAction: "Request access from workspace admin"
  }
}

Explicit errors let the agent reason about what went wrong. And let you debug faster.

7. Real system state beats assumed state

I watched an agent confidently tell a user their shipment was delivered.

It wasn't. The agent assumed based on typical timelines. It never checked the actual record.

This happens when teams don't pass real state:

// ❌ Agent has to guess
const context = {
  orderId: "12345"
}

// ✅ Agent knows the truth
const context = {
  shipment: {
    id: "12345",
    status: "in_transit",  // actual current status
    lastUpdate: "2024-01-15T10:30:00Z",
    currentLocation: "Memphis hub"
  }
}

Agents will make up state if you don't give them real state. Always.

8. Specialized agents beat one generalist

I've seen teams try to build one agent that handles everything. Customer questions. Data entry. Workflow automation. Reports.

It's mediocre at all of them.

The pattern that works:

One agent for Q&A using org context
One agent for record operations with strict schemas
One agent for document extraction with specialized prompts

Each one is easier to eval. Easier to constrain. Easier to improve.

Generalist agents are harder to debug and harder to trust. I push clients toward decomposition early.

9. Guardrails should block bad things, not useful things

I've seen guardrails so aggressive they blocked legitimate business operations.

"Can you help me set up a webhook?" → BLOCKED (mentions code execution)

"What's the API endpoint for shipments?" → BLOCKED (mentions API)

The users stopped trusting the product. Not because the AI was bad. Because the guardrails were dumb.

Narrow guardrails work better. Be specific about what's actually dangerous. Allow everything else.

10. Audit perception before rewriting prompts

When a client tells me their agent is underperforming, I ask these questions first:

Can it see the right tools? Are names and descriptions clear?
Can it see the right context? Or is it drowning in noise?
Can it see real state? Or is it guessing?
Can it see errors? Or do failures happen silently?

Nine times out of ten, the problem is perception. Not the prompt.

The outcome when you get this right

Teams that engineer perception instead of prompts:

Stop the endless prompt iteration cycle
Get measurable accuracy improvements in days, not months
Build agents that actually work in production
Have clear debugging paths when things break

The teams that keep tweaking prompts stay stuck. I've seen it enough times to know.

The mental model shift

Prompt engineering asks: "How do I word this better?"

Perception engineering asks: "What does the agent need to see to make a good decision?"

One has diminishing returns after a few iterations.

The other compounds as your system improves.

Stop rewriting prompts. Start auditing what your agent can perceive.

Rename tools for clarity
Scope your context
Pass real state
Make errors loud
Use specialized agents

Your agent is only as good as what it can see 👀

If you're building agents and want a second set of eyes on your architecture, I help teams get this right. DM me on X or LinkedIn.

My First RAG System Had No Evals. 40% of Answers Were Wrong.

Serhii Panchyshyn — Mon, 13 Apr 2026 20:58:06 +0000

When I started building production RAG systems, I noticed something: nobody was measuring retrieval quality.

Teams would ship a system, ask users if it "felt good," and move on. No metrics. No baseline. No way to know if changes actually helped.

So I started measuring everything. And the first thing I discovered: most RAG failures aren't LLM failures. They're retrieval failures.

The documents that could answer the question aren't making it into the context window. The LLM is being asked to answer questions without the information it needs. No wonder it hallucinates.

Here's what I've learned about measuring and fixing RAG systems after building them for B2B SaaS companies.

The metric that actually matters: Recall@k

Before I measure anything else on a new RAG system, I measure Recall@k.

Recall@k answers a simple question: "Of all the documents that should have been retrieved, what percentage actually made it into the top k results?"

def recall_at_k(retrieved_ids: list, relevant_ids: list, k: int) -> float:
    """What % of relevant docs are in the top k results?"""
    top_k = set(retrieved_ids[:k])
    relevant = set(relevant_ids)

    if not relevant:
        return 1.0

    return len(top_k & relevant) / len(relevant)

On systems I've audited, Recall@10 is often around 60%. That means 40% of the time, the document that could answer the question isn't even in the context. The LLM never had a chance.

Here's the math that drives everything:

P(correct answer) ≈ P(correct context retrieved)

If the right chunks aren't retrieved, the LLM can't answer correctly. This is why I always measure retrieval separately from answer quality. Otherwise you're debugging the wrong layer.

You can start measuring today

You don't need production traffic to build evals. Generate synthetic test data from your corpus:

def generate_synthetic_evals(chunks: list) -> list:
    """Generate question-answer pairs from your chunks."""
    eval_pairs = []

    for chunk in chunks:
        response = llm.generate(f"""
Generate 3 questions that this text can answer.
Make them specific. "What is this about?" doesn't test retrieval.

Text:
{chunk.text}

Return JSON: [{{"question": "...", "chunk_id": "{chunk.id}"}}]
""")

        eval_pairs.extend(parse_json(response))

    return eval_pairs

50-100 questions is enough to establish a baseline. Run your retriever, measure Recall@10, write down the number. Now you can actually tell if changes help.

The two fixes that consistently move the needle

I've tried a lot of retrieval improvements. Most make marginal differences. Two consistently deliver results.

Fix 1: Hybrid search

Embeddings are great at semantic similarity. "How do I reset my password?" matches "Steps to recover account access" even though they share no keywords.

But embeddings are weak on:

Numbers: They don't understand that 49 is close to 50
Exact match: Product codes, IDs, ticker symbols
Rare terms: Domain jargon not in the training data

BM25 (keyword search) catches what embeddings miss. Combine them:

def hybrid_search(query: str, k: int = 10) -> list:
    """Combine embedding search and BM25 using RRF."""

    embedding_results = embedding_index.search(query, k=20)
    bm25_results = bm25_index.search(query, k=20)

    # Reciprocal Rank Fusion
    scores = {}
    rrf_k = 60

    for rank, doc_id in enumerate(embedding_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (rrf_k + rank + 1)

    for rank, doc_id in enumerate(bm25_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (rrf_k + rank + 1)

    ranked = sorted(scores.keys(), key=lambda x: scores[x], reverse=True)
    return ranked[:k]

Typical improvement: 5-15% recall boost depending on query mix.

Fix 2: Add a reranker

Embedding models are bi-encoders. They encode query and documents separately, then compare. Fast, but imprecise.

Cross-encoders (rerankers) look at the query and document together. Slower, but much more accurate. Use them as a second pass:

def search_with_rerank(query: str, k: int = 5) -> list:
    """Retrieve broadly, then rerank precisely."""

    # Cast a wide net
    candidates = hybrid_search(query, k=20)

    # Rerank with cross-encoder
    pairs = [(query, get_content(doc_id)) for doc_id in candidates]
    scores = reranker.score(pairs)

    # Return top k after reranking
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [doc_id for doc_id, score in ranked[:k]]

Typical improvement: another 5-10% on top of hybrid search.

Combined, these two fixes often take a system from 60% to 80% recall. That's the difference between "works sometimes" and "works reliably."

Chunking decisions that make or break retrieval

Your chunking strategy matters more than your embedding model choice. A few things I always check:

The "it" problem

Chunks that start with "It also supports..." or "This feature allows..." are useless on their own. The word "it" has no meaning without the previous chunk.

Fix: Prepend context to every chunk.

def chunk_with_context(doc) -> list:
    chunks = []

    for section in doc.sections:
        # Prepend document and section info
        context = f"Document: {doc.title}\nSection: {section.header}\n\n"

        for chunk_text in split_section(section.content):
            chunks.append({
                "content": context + chunk_text,
                "metadata": {
                    "doc_title": doc.title,
                    "section": section.header
                }
            })

    return chunks

Other chunking rules I follow

Never split mid-table. A row without headers is meaningless.
10-20% overlap between consecutive chunks.
Test multiple chunk sizes (256, 512, 1024 tokens). Optimal depends on your queries.

The workflow I use on every RAG project

Week 1-2: Establish baseline

Parse documents (test multiple parsers for PDFs)
Chunk with context headers
Generate 50-100 synthetic eval questions
Build basic retriever
Measure Recall@10
Write down the number

Week 2-4: Apply standard fixes

Add hybrid search (BM25 + embeddings)
Add reranker
Measure again
Compare to baseline

Week 4+: Debug specific failures

Break down recall by query type
Find worst-performing segment
Fix that segment
Measure again

The key: measure after every change. If you can't see improvement in numbers, you're guessing.

When to measure answer quality

Only after retrieval is solid.

Once Recall@10 is above 80%, start measuring end-to-end:

def eval_answer(question: str, answer: str, context: list) -> dict:
    """Use LLM-as-judge for answer evaluation."""

    result = llm.generate(f"""
Evaluate this answer. Return JSON:
- correct: true/false (factually accurate)
- grounded: true/false (supported by the context)
- complete: true/false (addresses the full question)

Context: {format_context(context)}
Question: {question}
Answer: {answer}
""")

    return parse_json(result)

But if retrieval is broken, this eval is noise. You're just measuring how well your LLM fills in gaps it shouldn't have to fill.

The takeaway

RAG quality is retrieval quality.

Before you touch your prompts:

Generate synthetic evals from your corpus
Measure Recall@10
Add hybrid search
Add a reranker
Fix your chunking
Measure again

The fixes are straightforward. The impact is not.

This is Part 1 of a series on production AI systems. Next: how to know when to fix your prompts vs. build an evaluator.

About me

I help B2B SaaS companies ship production AI in 6 weeks.

If you're building RAG and want a second set of eyes, I do free AI Teardowns — a 30-45 min video showing exactly where your pipeline is breaking and how to fix it.

No pitch. Just clarity.

AI Implementation for B2B SaaS | AnimaNova Labs | AnimaNova Labs

Ship production AI features in 6 weeks. For B2B SaaS companies who need AI but can't hire fast enough. No $300K engineer. No 6-month timeline.

animanovalabs.com