Don’t Trust the Answer, Trust the Log: Lessons from Building an AI Expense Auditor

Yogendra singh — Sat, 17 Oct 2026 12:24:39 +0000

canonical_url:https://valentiuskryptix.com/3-surprising-failures-from-testing-my-ai-auditing-agent/

A chatbot answers questions. An agent is different: it is given a goal and a toolbox, then decides for itself which tool to call, what to pass in, and when to stop. For my “Agentic AI Workflow” milestone during an internship at Valentius Kryptix, I built one, and I set myself a harder requirement than “it works.” I had to prove it.

The project

ExpenseBot reviews employee expense claims against an invented company policy. It is written in Python with LangGraph and has three tools: a policy search, a currency converter with fixed rates, and a calculator. The calculator parses expressions into a syntax tree and allows only numbers and basic operators, instead of using eval, because model output feeding into code should be treated as untrusted input.

A system prompt acts as the agent’s job description: look up the rule, convert foreign currency, do all arithmetic through the calculator, and if no policy matches, say so and refer the user to Finance instead of inventing a rule.

Take three hotel nights at 200 euros each. The agent finds the $250 international cap, converts to $216 per night, sees it is under the cap, calculates 648, and approves the claim. Every number comes from a tool call, so it can be traced in the log.

How I tested it

I wrote 12 test claims with expected results before running anything. A test passed only if the final answer was correct and the agent had used the tools that answer depends on.

Three lessons from failures

The bug was in the tools, not the prompt. A taxi claim was told no policy existed. My instinct was to rewrite the prompt, but the log showed my search scoring treated “domestic” as important as “taxi,” so three rules tied and the taxi rule lost. Fixing the search fixed the problem.
A right answer can hide a wrong process. A smaller model returned correct totals but did the multiplication itself instead of calling the calculator. I counted those as failures. A system that sometimes guesses right is not one that computes.
The plumbing matters too. Rate-limit errors were being recorded as test failures, so I made the script save progress and resume later. GitHub also blocked a push because of a real API key in a template file, a good reminder to rotate any exposed credential, not just hide it.

What the numbers say

The agent scored 10 to 11 out of 12 depending on the run. Those scores shift even at temperature zero, so I treat them as a rough range. Twelve cases are enough to find bugs, not to claim general reliability. The prototype also uses keyword search, invented policy data, and handles one claim at a time, so embedding-based search and multi-expense claims are my next steps.

Takeaway

Read the tool-call log before touching the prompt, test the process as well as the output, and report your results with the failures and limits included.

👉 Read the full article: 3 Surprising Failures from Testing My AI Auditing Agent

Code and test cases: https://github.com/Yogendra404/expense-agent

Building a RAG App That Knows When to Say "I Don't Know"

Yogendra singh — Thu, 08 Oct 2026 06:58:08 +0000

Building a RAG App That Knows When to Say "I Don't Know"

Large language models are impressive, but they answer confidently even when they have no idea. For my internship milestone at Valentius Kryptix, I set out to fix that by building a Retrieval-Augmented Generation (RAG) app: an assistant that answers only from a document I give it. RAG works by looking up the relevant text first and then asking the model to answer from it, which keeps answers grounded and checkable.

What I built

A small web app where you can chat with a set of machine learning study notes stored as a PDF. Ask "What test RMSE did the random forest reach?" and it replies from the notes and shows the source passage. Ask something unrelated, like the capital of France, and it politely says it could not find that in the documentation. You can try it here: https://rag-study-assistant-fro0.onrender.com/

How it works

Chunk: the PDF text is split into pieces of about 800 characters with a small overlap, so ideas are not cut in half.
Embed: each chunk is turned into a vector with a small open-source embedding model that runs locally, so no paid API is needed.
Index: the vectors are stored in FAISS for fast similarity search.
Retrieve: the user's question is embedded the same way, and the four closest chunks are fetched.
Generate: those chunks are sent to an LLM through Groq's API, with strict instructions to answer only from the context.

The backend is FastAPI, the interface is a single HTML page, and the whole thing is deployed on Render.

Three takeaways

Retrieval quality decides everything. Chunk size, overlap and a similarity threshold determine whether the app answers correctly or admits it does not know. A chunk that is too small loses context, and one that is too large buries the answer. Tuning the threshold is what made off-topic questions get refused.

Prompts are part of the product. A clear system prompt, two few-shot examples and structured JSON output (an answer, an "answerable" flag and source numbers) made responses consistent and easy to cite.

Shipping teaches the most. Keeping API keys in environment variables, showing errors clearly in the interface, and debugging problems that only appeared on the server taught me more than any tutorial.

What's next

I want to try chunking by headings, support several documents at once, and measure answer quality with a small set of test questions.

Read the full article: 3 Powerful Lessons from Building My First RAG-Powered AI App

DEV Community: Yogendra singh

Don’t Trust the Answer, Trust the Log: Lessons from Building an AI Expense Auditor

Building a RAG App That Knows When to Say "I Don't Know"

Building a RAG App That Knows When to Say "I Don't Know"

What I built

How it works

Three takeaways

What's next