DEV Community: Jonathan Westerfield

The Start of My RAGgedy Journey

Jonathan Westerfield — Sat, 18 Jul 2026 00:59:40 +0000

Big Howdy,

I'm so tired of hearing all the hype about AI.

Don't get me wrong, I think AI is pretty cool and I use it all the time for various tasks throughout the day, but I hate AI marketing.

What do I mean by that? Whenever there is any hype, advertising, or corporate shilling about AI, it's always vague. Talks about agents, multi-agent systems, RAG pipelines, etc. are everywhere, and yet the specifics are always just out of reach.

Even when I try to dig into deeper technical articles, the handwaving usually begins almost immediately. Several architecture diagrams and acronyms later, I still do not have a much better idea of what is actually happening (although, to be fair, I wasn't exactly looking that hard).

I’m going to fix that.

For myself, at least.

First Stop: RAG

My first target is RAG. It's been out for quite a while now and I've been hearing about it for forever. It also sounds like enough progress has been made that the problem has effectively been solved. Best practices have been established and it's pretty commonplace now, so it's a great place to start.

But what is RAG?

RAG stands for Retrieval-Augmented Generation, which is... not a helpful name if you are just learning about this for the first time like I am. However, if you use AI at all, you've already been using a form of it without even knowing it.

RAG is simply a pattern:

Retrieve relevant external information.
Add that information to the AI model’s working context.
Generate an answer grounded in it.

That's it.

When you use an AI assistant and it goes and searches the internet for relevant information to give you an answer, that's a form of RAG. That doesn't mean your AI assistant is a RAG system, but it can perform RAG functions.

But this STILL sounds a bit hand-wavy to me. How does this work in real life?

The Part I Kept Missing

Context windows are massive these days, but I have seen technical documents that are literally more than 3,000 pages long and full of dense technical information. Even if one technically fits inside a model’s context window, sending the entire document with every question would be slow, wasteful, and probably not very good at finding the exact paragraph I need.

But an LLM would be perfect for working with documents like that. If I am reading a specification for something like High Bandwidth Memory, it would be fantastic to ask:

What does each bit in this register represent? If a bit maps to a different register, show me that too.

That would be infinitely more convenient than manually jumping around a massive PDF trying to find every related table, footnote, and register definition.

That's where the "retrieval" part of a RAG system comes in. Somehow, I need to structure and index the data, search it based on the user's question, and then give the relevant results to the LLM.

That's the part I'm going to figure out.

Enter Librarian

My project idea for learning this is to create an assistant that specifically answers questions about my personal ebook library. I call it Librarian (original, I know). Librarian will ingest my ebooks, many of which are several megabytes in size (although the size comes from images embedded in the file, not the text itself), structure the data, then search across it when I prompt it.

The goal is to support questions like:

"What is this book about?"
"What does this author have to say about this topic?"
"Compare how these books discuss suffering."
"Recommend a fantasy book with war themes."
"Where does this book explain this specific idea?"

The important part is that Librarian should not simply produce a plausible answer. It should retrieve evidence from my books and cite the passages it used.

Of course, describing the finished product is the easy part. I still need to figure out what happens between dropping an ebook into a folder and getting a useful answer back.

How do I extract and structure the text? How do I make thousands of pages searchable? How does the system decide which passages are relevant? How do I preserve enough information to cite the book, chapter, and section an answer came from?

And most importantly, how do I know the answer is actually based on my books instead of something the model made up? The engineer in me can't stand relying on nondeterministic output without something I can measure, so I need a way to tell whether the system actually works and quantify how well it works.

These are the parts I want to understand.

I have already started building Librarian, and the next few posts will follow the project as I work through them. The next post will cover the high-level design: what components I think the system needs, what each one is responsible for, and how information will move from an EPUB file to an answer.

After that, I’ll start digging into the individual pieces.

You can follow the project here:

JonathanGWesterfield/Librarian

I Failed: A Post-Mortem

Jonathan Westerfield — Tue, 14 Jul 2026 04:00:04 +0000

King Yemma and KY are pseudonyms. I have changed or generalized names and a few identifying details to keep internal information private.

Big Howdy,

I want to write about a project that, on paper, went pretty well.

The Project

Our server fleet had a major problem with dirty reboots. Think “Blue Screen of Death,” but on a much more expensive computer. Sometimes it happened when testing servers before putting them into production, which was annoying but manageable. Other times it happened in production while a customer was using the server, which was considerably less manageable and considerably more expensive for everyone involved.

The fleet health team was already investigating these failures, but there were a lot of them and the logs were a pain to work through. The servers are wildly complex, meaning there could be hundreds of reasons for one to crash like this.

To scale to the amount of work, I was assigned to onboard our servers to an internal tool called King Yemma. Dragon Ball reference, anyone? From here on out, I am calling it KY.

I was the only person from my team assigned to this work, so I worked directly with both the team that owned KY and the fleet health team.

The idea was simple enough: KY would pull the relevant logs for a dirty reboot, look for known failure patterns, and categorize the ticket. If it found a known issue, it could close the ticket automatically. If it found something new, it could help point the investigation in the right direction by tagging the ticket based on other rules in KY. The original team that built it was already using it with great success.

There was one problem: KY had not been built for our servers.

Getting KY Working

I worked with the team that owned KY to make it understand our systems. I added our log groups, pointed it at the right resources, and taught it where to find our dirty reboot tickets. I also added a feature to parse the logs and pull out the component that appeared to have failed, which was one of the requests that came down while I was working on it.

After that, I started working with fleet health to add fingerprints.

A fingerprint was basically a known pattern in the logs that mapped to a known failure. Generally, this meant doing a regex or pattern match on one or more log lines. The goal was to get enough of these into KY that it could automatically close tickets for problems that had already been seen and understood. If it did not have an exact match, KY could at least add tags to the ticket to point engineers in the right direction.

This meant I had to spend a lot of time looking through dirty reboot logs. Don’t get me wrong, the fleet health team worked with me and provided fingerprints when they found things too. It is not like they simply expected a software engineer to suddenly start doing a fleet health or hardware engineer’s job overnight. However, I still needed to do my part to speed up development.

I would find something in the logs that looked suspicious, ask someone from the fleet health team whether it was actually important (it rarely was), and then repeat. Turns out there are a lot of weird warnings and error messages that pop up during runtime that are apparently... completely meaningless?

Still, I got KY working on our systems. Together with the fleet health team, I added numerous fingerprints that could automatically close tickets. It also helped identify new buckets of failures for the team to investigate.

KY had another useful feature too. Someone could give it a server and a time range, and it would pull the relevant logs into one place on their local machine. Then they could use grep, awk, sed, or whatever else they wanted without waiting on CloudWatch or manually jumping between log groups. This meant KY could be useful for more than just fleet health. Everyone in the org could use it for their own debugging.

I cannot take credit for implementing that feature, but I CAN take credit for exposing it in the CLI for a normal user to call. I even gave a short presentation about it to the org.

At this point, it sounds like a smashing success, right?

If only it actually was...

So Much Work, So Little Value...

The actual goal for this whole boondoggle was to get the fleet health team to adopt KY as part of their normal investigation process for dirty reboots. The goal was to reduce the time they spent digging through logs and make the whole investigation process less painful and more automated. After all, there were only a handful of fleet health engineers and literally thousands of servers.

Unfortunately, they thought it was only slightly better than useless and decided manually searching through CloudWatch was faster.

And, honestly... they were right.

KY ended up being useful mostly after a problem had already been diagnosed. It could automatically close tickets for known failures, which was helpful, but it was not the investigation tool we all hoped it would become.

I had built something that worked well, but I had not built something my customer wanted to use.

What I Got Wrong

Looking back, my mindset, along with my lack of awareness, was a major part of the problem.

First, I did not understand our system very well at the time, and I did not understand the customer use case or how the fleet health team operated nearly as well as I thought I did.

Second, I was under a lot of organizational pressure at the time. I thought I had made a mistake that severely affected my standing, and was worried that KY was a test of whether I could recover. That fear turned out to be mistaken, but it changed how I approached the work.

As a result, I honestly hated this project. I hated reading logs that I did not understand. I hated programming Python when my normal language was Lua, since the languages are just similar enough that I would constantly confuse the syntax (this was before the days of AI). It frustrated the hell out of me every day.

But I wanted to perform well, show progress, and get to something I could point at and say, “look, I did the work.” Mostly, I just didn't want to get fired, which I was convinced would happen if I so much as sneezed too loudly.

The easiest way to do that was to add fingerprints. More fingerprints meant more tickets could be categorized automatically. More fingerprints meant more pull requests for the codebase. More fingerprints gave me a metric.

The problem was that the metric did not matter.

The fingerprints the fleet health team actually needed were often more complicated than simple log matching. Sometimes they depended on values that had to be decoded or interpreted in a particular way. Sometimes the decoded value had to come from a very specific component to have meaning. Sometimes specific values needed to come from multiple specific components within a specific time range.

The more useful work would have been making KY flexible enough for fleet health to create and manage those fingerprints themselves. I did some of that, but it was clearly secondary to adding fingerprints.

I was trying to do their job for them instead of building the thing that would help them do their job better.

I also did not understand the fleet health team’s workflow well enough. I talked to them, got feedback, and worked with them on fingerprints, but I was still mostly seeing their work from the outside. I understood the inputs and outputs more than I understood the actual process, or the system itself.

I should have spent more time sitting with them while they investigated failures. I should have asked my manager to loan me to the team for a bit so I could understand how they debugged, what information they needed first, and where the existing process was actually slow.

Instead, I focused on the work that felt safest.

What I Learned

There was also a more personal lesson in it for me. At the time, I was still learning how to operate in an environment where I had no solid footing. There was a lot of information, a lot of system complexity, and no obvious answer. These servers were complicated enough that a single machine was basically its own distributed system. There were hardware blocks, firmware, software, logs from different sources, and a bunch of interactions I did not understand yet, many of which I never figured out.

When it came to logs, I would sometimes feel so overwhelmed that I would shut down and not see the obvious line saying, “Error! I’m literally the root cause!”

When I felt overloaded, I had a tendency to narrow my focus until I found something concrete that I could make progress on. In this case, that became fingerprints.

Fingerprints were real work. They were useful. But they were also safer than going deeper into the uncomfortable question of whether KY itself was solving the right problem.

The project taught me a lot about our systems, and that knowledge was extremely useful later when I was writing tests and assisting with, and eventually driving, future investigations. It also gave me better relationships with both the KY team and the fleet health team, so I did gain a lot from it.

However, it was still a failure. After two months of work, I had nothing enduring to show for it because it was quickly forgotten. Even the log pulling feature was never really adopted after I gave that presentation on it.

KY failed to accomplish its most important goal because I optimized for what I could measure instead of what the customer actually needed. I was so worried about proving that I was doing good work that I stopped asking whether I was doing the right work.

My First Post

Jonathan Westerfield — Sat, 11 Jul 2026 19:25:39 +0000

Big Howdy,

I’m starting a blog.

That sentence feels weirdly dramatic for what this is to me.

The plan is to write about engineering, personal projects, things I’m learning, things I thought I understood but absolutely did not understand, and the occasional reflection after I’ve spent way too long staring at a problem that was probably my fault the whole time.

I’ve wanted to write more for a while, but I kept running into the same problem: a lot of engineering writing feels a little too polished; almost a little fake (I use semi-colons, no it's not AI).

Not bad necessarily. Just polished in a way where the author always seems to have known the correct abstraction, the correct database, the correct deployment strategy, and the correct lesson before the story even started.

I am not that guy.

Most of my actual engineering experience has been much messier.

I'll start with some assumptions, plow ahead thinking I know what I'm doing and then hit a wall realizing that the foundation of my idea was flawed because of... my assumptions. You know what they say about assumptions right? From there I take a step back, untangle the mess and get it right on the 4th try. If it was easy, I probably wouldn't want to do it anyway.

This is all well and good, but the real reason I want to write is because I learn better when I explain things.

If I can write down what happened, what confused me, what I tried, what worked, and what I’d do differently next time, then maybe the lesson actually sticks. And if it helps someone else avoid one of my mistakes, even better. I can’t promise the mistakes will be impressive, but they will be honest.

The posts will probably be pretty short. I’m not trying to write a novel every time I discover a new edge case. That said, I reserve the right to accidentally write too much if I get interested in something. This has happens more than I'd like to admit...

The topics will likely be a mix of:

personal projects
debugging stories
engineering tradeoffs
tools I’m trying
things I learned the hard way
reflections on work, burnout, confidence, and getting better at this without pretending it’s all obvious

I’m not writing this because I think I’m some grand authority on software engineering. I’m very much still learning; sometimes even successfully. Sometimes through doing something dumb and then having to deal with the consequences (honestly the most impactful learning).

But I do think there’s value in writing from that place.

Not the “here are the ten rules for building perfect software” place.

More like the “here’s what I was trying to do, here’s where I got confused, here’s what finally clicked, and here’s the part I still don’t fully understand” place.

That feels more useful to me. At least it feels more real.

So that’s what this blog is going to be. Small posts about engineering and projects, written as honestly as I can manage, without trying to sound smarter than I am, because I'm not that smart :)

We’ll see how it goes.