Asked in an AI agent developer interview.
What they are really testing: whether you treat a vague complaint as a debugging problem backed by evidence, not a prompt-tweaking exercise, and whether you can protect users while you investigate.
A strong answer:
- Stabilize first: determine whether the complaint is isolated or systemic. If it looks like a regression, roll back the last change (prompt version, model version, tool config or retrieval index) behind a feature flag.
- Make runs observable: trace every interaction with prompt version, model, retrieved context, tool calls with arguments and results, latency and token usage. Without this you are guessing.
- Reproduce and classify: pull the failing sessions, replay them against the same inputs, and bucket the failures as retrieval miss, wrong tool or bad arguments, instruction-following error, stale or missing data, or provider-side model drift.
- Turn failures into a versioned regression suite: combine deterministic checks (schema valid, required tool called, citation present) with graded checks for answer quality. Calibrate any LLM-as-judge against a sample of human labels before trusting its scores.
- Fix at the right layer: retrieval problems call for chunking changes, metadata filters or reranking, not a longer prompt. Tool problems call for stricter schemas and validation. Prompt changes come last, gated by the eval suite and a shadow or A/B run.
- Close the loop: add in-product feedback, alert on thumbs-down rate and tool error rate, and write a short postmortem that adds the failure modes to the eval set.
Common mistakes:
- Rewriting the system prompt immediately without reproducing the failure, which often fixes one example and breaks three others.
- Judging the fix on a handful of anecdotes instead of a labelled set with a baseline score.
- Blaming the model without checking data freshness, retrieval results or what changed in the last deployment.
How would you structure the first 24 hours after a complaint like this, and what would you check before touching the prompt?
Originally shared on LinkedIn.
An AI interview question every day, leadership for tech people and hands-on AI learning:
Top comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.