DEV Community

Manohar Papasani
Manohar Papasani

Posted on

You ship agent to production & 3 days later internal user says it's not answering well - next move?

Asked in an AI agent developer interview.

What they are really testing: whether you treat a vague complaint as a debugging problem backed by evidence, not a prompt-tweaking exercise, and whether you can protect users while you investigate.

A strong answer:

  • Stabilize first: determine whether the complaint is isolated or systemic. If it looks like a regression, roll back the last change (prompt version, model version, tool config or retrieval index) behind a feature flag.
  • Make runs observable: trace every interaction with prompt version, model, retrieved context, tool calls with arguments and results, latency and token usage. Without this you are guessing.
  • Reproduce and classify: pull the failing sessions, replay them against the same inputs, and bucket the failures as retrieval miss, wrong tool or bad arguments, instruction-following error, stale or missing data, or provider-side model drift.
  • Turn failures into a versioned regression suite: combine deterministic checks (schema valid, required tool called, citation present) with graded checks for answer quality. Calibrate any LLM-as-judge against a sample of human labels before trusting its scores.
  • Fix at the right layer: retrieval problems call for chunking changes, metadata filters or reranking, not a longer prompt. Tool problems call for stricter schemas and validation. Prompt changes come last, gated by the eval suite and a shadow or A/B run.
  • Close the loop: add in-product feedback, alert on thumbs-down rate and tool error rate, and write a short postmortem that adds the failure modes to the eval set.

Common mistakes:

  • Rewriting the system prompt immediately without reproducing the failure, which often fixes one example and breaks three others.
  • Judging the fix on a handful of anecdotes instead of a labelled set with a baseline score.
  • Blaming the model without checking data freshness, retrieval results or what changed in the last deployment.

How would you structure the first 24 hours after a complaint like this, and what would you check before touching the prompt?


Originally shared on LinkedIn.

An AI interview question every day, leadership for tech people and hands-on AI learning:

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.