What 22 Days of Building AI Systems Taught Me: Grounding, Evals, and Control

I started this learning sprint with a familiar but incomplete idea: an LLM was a powerful chatbot, and the hard part of using one was writing a clever prompt.

Twenty-two days later, I have a different model. An LLM is a probabilistic component inside a larger software system. A useful AI feature needs the same things other production software needs: clear inputs, constrained behavior, observability, and a way to tell whether a change made it better or worse.

This is the path I took: fundamentals, local experiments, a RAG system over my own notes, automated evaluation, and finally a controlled workflow that can choose from a small set of safe tools.

Start with the mental model, then test it

The first few days were theory: tokens, context windows, temperature, embeddings, and the Transformer pipeline. I wrote explanations in my own words, then tested myself on them.

That test was useful because several of my confident explanations were wrong.

• I had treated RAG as primarily a privacy or local-hosting pattern. Its core purpose is more direct: retrieve the relevant facts at request time so the model does not have to receive an entire corpus in its prompt.
• I had confused embeddings with the mechanism that predicts the next token. Embeddings represent tokens or text as vectors; the Transformer layers and attention operate on those representations; an output layer produces the next-token distribution.
• I had inverted the practical token estimate. In English, one word is roughly $1.33$ tokens, not the other way around. That matters for context limits and cost estimates.

The first lesson was uncomfortable but durable: AI concepts become useful only after they survive a concrete explanation and a falsifiable experiment.

Local inference made the abstractions real

I used Ollama and a small local model to make API concepts visible. The model streamed its answer token by token, so latency stopped being an abstract metric. I also ran a simple memory test: I told the model a fact, then made a later request without resending the earlier conversation.

It did not remember.

That was the cleanest demonstration of LLM statelessness I could ask for. A chat application appears to have memory because the application sends previous messages back with each new request. The model only knows what is present in the current context window.

That same fact later shaped the agent workflow. Conversation memory is not magic or a model setting; it is deliberate state management.

I also measured tokenization across English, code, and Hindi. Code used about $2.2$ tokens per word-like unit, and Hindi about $3.4$, compared with roughly $1.33$ for English. Prompt size is not neutral: the same product can have different latency and cost characteristics depending on the language and content it handles.

Building RAG exposed the retrieval problem

The next stage was a RAG system. I first built semantic search with local all-minilm embeddings, then combined retrieval with generation. Instead of matching keywords, the system embedded a user question and compared that vector with document vectors using cosine similarity.

The important constraint is that queries and documents must use the same embedding model. Vectors from different models occupy incompatible spaces, so comparing them has no useful meaning.

I then pointed the system at my own learning notes. The pipeline became:

1. Split markdown notes into paragraph-sized chunks.
2. Filter short, low-signal chunks.
3. Embed and store each chunk.
4. Retrieve the closest chunks for a question.
5. Give only those chunks to the generation model, along with a request to answer from the supplied context.

That system indexed 467 chunks from ten days of notes. It also immediately taught me that RAG quality is not a single model-quality problem.

A query about the temperature setting for coding retrieved a chunk about a probability table rather than the practical recommendation of 0.1-0.3. The generation model was not the only thing that could be wrong; the system had fetched the wrong evidence.

I added source tracking, similarity-score visibility, a minimum score threshold, source deduplication, and a limit on chunks from any one file. Chunking became a real engineering parameter: chunks that are too large dilute meaning, while chunks that are too small lose the surrounding explanation.

Persistent storage mattered too. Creating embeddings for the notes took roughly 8-10 minutes on the first run; reloading a JSON cache took less than a second. I later replaced that manual cache with a ChromaDB collection configured for cosine distance.

A better generator did not fix retrieval

For generation, I moved from a local phi3:mini model to Groq's hosted llama-3.1-8b-instant, while keeping retrieval local. The response quality improved, especially on questions where the retrieved context was correct but the smaller model was weak at following it.

This made an important distinction concrete:

• Retrieval answers: did the system find the right evidence?
• Generation answers: did the model faithfully and clearly use that evidence?

They need separate debugging paths. Better generation cannot repair missing or irrelevant retrieved context, and perfect retrieval does not guarantee a faithful answer.

I also asked the model for strict JSON containing an answer and a confidence value. Structured output is not just neat formatting. It lets the next part of a system route, validate, store, or reject a response without parsing a prose paragraph.

"Seems to work" is not an evaluation strategy

The turning point came with a 20-question RAG evaluation harness. Each case defined a question, an expected answer concept, and expected source files. The runner checked both answer matching and whether the correct note had actually been retrieved, then wrote detailed results and returned a failing exit code below an $80\%$ threshold.

The first run passed only 1 of 20 questions: $5\%$.

That was not discouraging. It was the first reliable signal I had. The failures found a configuration mistake that casual manual testing had missed: a skip list excluded the Day 11 note containing a major part of the RAG and chunking explanation.

I used source-level retrieval traces to investigate, refined expectations where the initial checks were either too strict or too loose, re-indexed the right content, and repeatedly reran the same suite. The final validated result was 16 of 20: $80\%$.

The number is not a universal quality score, and it does not mean the system is production-ready. It is a baseline tied to a small, explicit set of questions. Its value is that the next prompt, retrieval, model, or corpus change can be compared against something more useful than intuition.

From one model call to a controlled workflow

The final phase was a small tool-using workflow. I began with a deterministic loop: a maximum of eight steps, stop conditions, retry logic for transient failures, and a structured StepLog for every decision.

Then I added a real planner using llama-3.1-8b-instant, but did not let model text directly become arbitrary program behavior. The planner can select only one of four actions:

read_progress_files
summarize_status
check_git_status
ask_clarification

An executor handles those approved actions. A deterministic fallback planner takes over if the model is unavailable or returns something outside the allowed set. The workflow logs the selected action, result, planner source, and retry count.

For example, a simulated temporary failure records that an action succeeded on its second attempt rather than hiding that instability behind a final success message. At the end of each run, the step trace is persisted as timestamped JSON, so the terminal output becomes a reviewable artifact.

Finally, I replayed prior intent/action pairs as real chat messages when the planner chooses the next action. The model remains stateless; the application supplies the run history. This preserved the original guardrails while giving the planner context about the steps already taken.

What I would carry into a real product

This journey changed my definition of an AI feature. The model is necessary, but it is not the system.

• Use retrieval when answers must be grounded in changing or private information.
• Treat chunking, source selection, and model behavior as independently testable concerns.
• Create an evaluation set before making many prompt changes. A bad baseline is still valuable evidence.
• Give models narrowly defined action spaces, enforce them in code, and preserve a deterministic fallback for important paths.
• Log the decision path, not merely the final answer. AI failures are often visible in how the system arrived at an output.
• Manage conversation history explicitly, because no LLM call carries state by itself.

I began by learning what an LLM predicts. I ended this phase building systems around what it cannot guarantee: grounded knowledge, deterministic tool permissions, memory, and proof that changes improve the behavior I care about.

Next, I am deciding whether to harden the workflow further with more tools and stop conditions, or apply these patterns to a small AI feature in a real product. The second path is probably the real test: a controlled demo is useful, but user workflows are where the constraints become honest.

What is the first AI-system behavior you would put under an evaluation before trusting it?