DEV Community

Cover image for I Review My AI Assistant's Logs Every Week, With an Agent. Here's the Process.
Nunc
Nunc

Posted on

I Review My AI Assistant's Logs Every Week, With an Agent. Here's the Process.

#ai #agents #observability #llm

We run an internal AI assistant on top of our 20-year-old helpdesk system. Production since April, 15 active users across five roles, from developers to operators to project managers. The model matters less than people think. What actually improves the assistant week over week is a boring process: a structured weekly review of its logs. This post describes that process, the metrics we track, and two real failures the review caught that I would never have found otherwise.

What gets logged

Two levels. A summary log with one line per request:

[2026-07-20 16:46:52] user=xxx timing=99926ms status=ok
  model=... turns=13 tools=12 cost=$0.25
  profile=xxx ftok=1820ms query="have we ever solved..."
Enter fullscreen mode Exit fullscreen mode

And a full JSONL transcript per session: every user turn, every answer, plus which context was injected and where it came from. The provenance part was added later, after a review showed we couldn't tell why the assistant said something. If you're building an assistant, log answer provenance from day one.

The weekly review is run by an agent

Reading a week of transcripts by hand is not sustainable, so the review itself is an agent task. I say "do the review", and the agent figures out which period is not yet covered, pulls the logs from the server, and works through every session. Not a sample. All of them.

The review has three outputs:

  1. Session quality review. The agent grades answers, and the important part: it re-verifies factual claims against the actual database. If the assistant told a user "this fix was installed at customer X in May", the review checks whether that's true.
  2. Improvement proposals. Concrete and prioritized, each with the problem, evidence (session IDs, quotes), a proposed fix, and an effort estimate. Each proposal enters a backlog with a status: proposed, approved, implemented, rejected, or watch.
  3. User profiles. Usage patterns per user feed personalization (more below).

Diagram of the weekly review loop: a summary log with one line per request and JSONL transcripts with every turn and injected context both feed a weekly review agent that works through every session rather than a sample; the agent produces three outputs, session quality grades with claims re-checked against the database, improvement proposals with problem and evidence and fix and effort, and user profiles that go into the assistant's prompt; proposals land in an improvement backlog with statuses proposed, approved, implemented, rejected and watch, and an orange arrow loops from the backlog back to the review agent, where the next review verifies every implemented fix

The next review then verifies the previous round: did the implemented fixes actually stop the failure they targeted? Several times the answer was "partially", and the item went back into the backlog. Without that verification step, an improvement backlog turns into a feel-good list.

The scorecard

Every review adds one row to a long-running scorecard. Same metrics, same methodology, so trends are visible: requests, users, sessions, error rate, median and p95 response time, corrections, factual errors, false negatives, security events, user feedback, average session grade.

A recent week looked like this: 137 requests, 12 users, 44 sessions, 0 errors, median response 99 seconds, p95 262 seconds, average session grade 4.5 out of 5 at 100% review coverage.

One week on the scorecard, shown as eight metric tiles: 137 requests, 12 active users, 44 sessions, 0 errors, 99 seconds median response, 262 seconds p95 response, average session grade 4.5 out of 5, and 100 percent review coverage

Two practical lessons from maintaining it. First, write down the counting methodology, because "how many sessions" turns out to have edge cases (we now count a session as a transcript file with at least one real user turn, feedback-only files excluded). Second, mark methodology changes in the scorecard itself. Otherwise a metric jump reads as regression when it's actually deeper measurement.

Two failures the review caught

The leaking limit banner. One afternoon our primary auth token kept hitting a rate limit, and a fallback took over. The failover worked, but the review found that at least 12 answers shown to 4 users had an English "You've hit your limit" banner and fragments of the failed first attempt glued to the top of the final answer. Users saw it and said nothing. Nobody reports weirdness in a tool they don't fully trust yet, which is exactly why you read the logs.

The adopted false premise. A user asked about a project, and their question contained a wrong assumption about which customer group a code referred to. The assistant took the premise at face value and confidently attributed everything to the wrong company. The transcript grade was fine (the user was happy!), but the facts were wrong. The fix was a prompt rule: verify the entity behind a code against the database before building an answer on it, even when the user asserts it. You will not catch this class of failure from user feedback, because the user is the source of the error.

Two kinds of user profiles

Each user has a profile the assistant loads into its system prompt: role, which projects they may see, typical questions, preferred answer depth. A developer asking about a package gets code references. A project manager asking the same thing gets a business summary. An operator asking about an error code gets the past-ticket solution plus a suggestion for which developer to assign.

The part that took iteration: we keep two separate profile sets. Analytical profiles are our internal understanding of each user, built from the log reviews. Agent profiles are the distilled version the assistant actually loads. The analytical profile is the raw material, the agent profile is the product. Mixing them was a mistake we made first: internal observations about a user do not belong in a prompt.

One rule I'd flag for anyone doing this: personalization must stay a default, not a cage. A non-technical user who asks a technical question gets the technical answer. And log review being about improving the assistant, not evaluating employees, is something you say out loud and write down before someone discovers their conversations are being read.

Wrapping up

An AI assistant in production is a product, and it needs the product loop: instrumentation, periodic review, prioritized backlog, verification that fixes worked. Doing the review with an agent makes 100% coverage affordable. And the highest-value finds are the ones no user will ever report: answers that were wrong while everyone was satisfied.

Top comments (0)