DEV Community: Valenx

What is agentic RAG?

Valenx — Mon, 09 Nov 2026 05:07:08 +0000

What the interviewer is testing

Whether you can distinguish agentic retrieval from a fixed pipeline and explain its benefits, costs, and production controls.

Answer framework

Agentic RAG replaces one fixed retrieve-then-generate pass with a model-controlled retrieval loop.
The model decides when and what to search and can issue several queries during that loop.
It reads the returned results and may search again when it still needs evidence.
This supports multi-hop questions, but it can increase cost and make evaluation harder.
Cap the number of steps and log each retrieval so the decision path can be inspected.

Sample answer

I would define agentic RAG as a retrieval-augmented generation system where the model controls part of the retrieval loop. In a basic RAG pipeline, I rewrite a query once, retrieve documents, and pass them to the model. In an agentic version, the model can decide whether it needs to search, choose what to look for, issue multiple queries, read the results, and decide whether to search again before it writes the answer.

That extra control is useful for questions that cannot be answered from one retrieval pass. For a multi-hop question, I might retrieve evidence about one entity, notice that another fact is missing, and issue a second query that connects the two. I can also ask for a narrower search if the first results look weak. The system therefore treats retrieval as an action the agent can iterate on, while the model uses the retrieved passages as evidence for the final response.

The tradeoff is that this can make the system more expensive, slower, and harder to evaluate than a fixed pipeline. Additional searches add retrieval overhead, and model calls to plan searches or process results can increase token cost. A model may also loop without converging. In production, I would cap the number of steps, log every query and retrieved document, and preserve a trace so I can diagnose both the answer and the path. I would test it on multi-hop tasks against a simpler baseline, measuring success as well as steps, cost, and latency rather than assuming the agentic design wins.

Likely follow-ups

What distinguishes agentic RAG from a multi-query RAG pipeline? A multi-query pipeline runs a preset query expansion, while agentic RAG lets the model choose later searches after reading the results of earlier ones.

When is agentic RAG worth the additional cost? I would favor it when a single retrieval pass cannot connect the evidence needed for a multi-hop task, and when testing shows better task success despite the extra steps.

How do I prevent the retrieval loop from running indefinitely? I would set a step cap, define stopping conditions, and monitor repeated queries and step counts in production logs.

Common mistakes

Calling any pipeline with several retrievals agentic without a model-controlled retrieval loop.
Assuming more searches will always produce a better answer.
Running the agent without a step cap or stopping condition.
Evaluating only the final answer without retaining retrieval logs and decision traces.

This article was prepared with AI assistance and reviewed before publishing. The original lives on the free question bank:

https://johnnnmai.github.io/ai-engineer-interview-questions/questions/what-is-agentic-rag/?utm_source=devto&utm_medium=community&utm_campaign=what-is-agentic-rag

How do you monitor an LLM system in production?

Valenx — Mon, 09 Nov 2026 04:59:10 +0000

What the interviewer is testing

Whether you can define request-level logging, multi-step tracing, operational alerts, quality monitoring, and safe handling of production data.

Answer framework

Log each request with its prompt version, model, token usage, latency, cost, and outcome.
Trace multi-step calls so the order of model, retrieval, and tool operations is visible.
Build dashboards and alerts for error rate, p95 latency, cost spikes, and quality signals.
Sample conversations for review and track user feedback to identify quality problems.
Redact PII before information is written to logs and restrict access to sensitive records.

Sample answer

I would monitor an LLM system at three levels: individual requests, multi-step traces, and aggregate service health. For every request, I would log a request identifier, the prompt or template version, model name, token counts, latency, cost, and a structured outcome. Those fields let me connect a user report to the exact call that produced it. I would avoid storing unnecessary prompt content, and I would redact personally identifiable information before any text reaches a log or observability tool.

For an agent or multi-step workflow, request logs are not enough. I would add distributed traces that show each model call, tool call, retrieval step, input, output, and error in the order they occurred. Dashboards would track error rate, p95 latency, token usage, and cost, while alerts would fire for sustained error increases, p95 latency regressions, and abnormal cost spikes. I would be careful to make alerts actionable so responders know which service and version are affected.

Operational logs and traces alone do not establish whether an output was useful. I would sample conversations for periodic human review, track thumbs-up, thumbs-down, and other user feedback, and add quality signals such as schema validity, refusal rates, or retrieval hit rates where they apply. I would segment these metrics by prompt version and model so I can detect a quality regression that averages hide. Finally, I would document retention limits, verify that redaction works, and restrict access to sensitive logs.

Likely follow-ups

Which production signals should trigger alerts? I would prioritize sustained error-rate increases, p95 latency regressions, abnormal cost spikes, and sharp changes in quality signals. Each alert should identify the affected service and version.

How do I monitor quality when there are too many conversations to review? I would combine sampled human review with user feedback and targeted automatic checks such as schema validity, refusal rates, or retrieval hit rates, then segment the results by model and prompt version.

How should monitoring handle sensitive user data? I would redact PII before logging, minimize the content retained, restrict access, and define retention limits. I would also test that the redaction works across prompts, traces, and error messages.

Common mistakes

Logging only final responses without prompt version, model, token usage, latency, cost, or outcome.
Using aggregate dashboards without request-level logs and traces for multi-step workflows.
Sending an alert for every isolated fluctuation instead of sustained changes that need attention.
Storing raw PII in logs because the observability tool is considered internal.

This article was prepared with AI assistance and reviewed before publishing. The original lives on the free question bank:

https://johnnnmai.github.io/ai-engineer-interview-questions/questions/how-to-monitor-llm-in-production/?utm_source=devto&utm_medium=community&utm_campaign=how-to-monitor-llm-in-production

How do you evaluate an AI agent?

Valenx — Mon, 09 Nov 2026 04:57:29 +0000

A common interview mistake is to evaluate an AI agent by asking whether it produced a good answer once. That tests a demo, not a system.

A stronger answer explains how to define success, build an evaluation set, compare outputs against evidence, and monitor behavior in production. The interviewer is usually testing whether you can turn an open ended AI problem into a measurable engineering process.

Before naming metrics, clarify the agent's job. An agent that retrieves documents, calls tools, and completes a workflow has more evaluation dimensions than a single turn chatbot.

The interviewer is likely looking for four things:

Can you define task success from the user's perspective?
Can you separate intermediate behavior from the final answer?
Can you design reliable test cases and review procedures?
Can you detect regressions when prompts, models, tools, or retrieval settings change?

You do not need to list every possible metric. Connect each check to a specific failure mode.

Start with the intended outcome. An illustrative practice example is an agent that answers questions from an approved document set, selects a tool, and completes a multi step workflow.

Possible failures include:

Unsupported claims
Missing required information
Incorrect tool selection
Invalid tool arguments
Repeated actions
Failure to stop
Unnecessary latency or cost
Sensitive data exposure

This shows that evaluation begins with system behavior, not a metric dashboard.

Use real or carefully constructed examples covering normal requests, ambiguous requests, edge cases, and known incidents. Freeze a version of the set so that changes can be compared consistently.

For retrieval augmented generation, bind each expected claim to an approved source where possible. For tool use, record the expected tool, arguments, order of operations, and stopping condition.

The final answer matters, but it is not enough. Inspect the trajectory as well.

For every check, define a pass condition:

Task completion passes only when all required steps are completed
Tool use passes only when the selected tool and arguments match the case requirements
Groundedness passes only when every factual claim maps to a retrieved source
Citation alignment passes only when the cited source supports the associated claim
Safety passes only when the response avoids the defined prohibited behavior
Efficiency passes when the trajectory stays within the case's step, latency, or cost limit

For subjective qualities, use a rubric with clear criteria and illustrative examples. Human review can validate the rubric, while automated checks can run more frequently.

Freeze the reference documents and review important claims against them. Ask:

Is the claim supported by the retrieved source?
Does the source actually answer the question?
Did the agent add details that are not present?
Did it ignore conflicting or newer evidence?
Can another reviewer reproduce the judgment?

If a factual claim cannot be mapped to an approved source, mark it unsupported and record the failure category. This makes evaluation more concrete than asking whether an answer feels good.

Offline tests do not cover every production condition. Log inputs, outputs, tool calls, errors, latency, and cost in a privacy aware way. Sample interactions for review and track new failure patterns.

When the system changes, rerun the frozen evaluation set and compare claim level, trajectory level, and task level results. Investigate regressions before accepting the new baseline.

A weak answer says, “I would use accuracy and user feedback.” That is too vague because it does not define accuracy, identify the reference, or explain how feedback maps to a failure mode.

A stronger answer says what success means, which examples represent the task, how outputs are compared to sources, and how regressions are caught.

This article was prepared with AI assistance and reviewed before publishing. The original lives on the free question bank:

https://johnnnmai.github.io/ai-engineer-interview-questions/questions/how-to-evaluate-an-ai-agent/?utm_source=devto&utm_medium=community&utm_campaign=how-to-evaluate-an-ai-agent