DEV Community: Jonathan Westerfield

Good Engineers Have Opinions

Jonathan Westerfield — Wed, 14 Oct 2026 15:24:29 +0000

One trait I have started noticing in good engineers, regardless of seniority, is that they are opinionated.

Not argumentative for the sake of it. Not the person who turns every code review into a hostage situation because someone used a different library. I mean they have a point of view. They have thought about the tradeoffs, they can say what they think we should do, and they are willing to stand behind it.

That matters more than I used to think.

An opinion gives you somewhere to start

Engineering is full of choices where there is no universally correct answer. Which abstraction is worth adding? Should we optimize now or leave it alone? Is a new dependency worth the complexity? Should we build a more flexible system, or solve the actual problem in front of us with something boring?

You need to make a call eventually.

People with opinions have some set of principles, experiences, and assumptions that helps them make those calls. Maybe they strongly prefer simple systems. Maybe they are suspicious of clever abstractions. Maybe they have been burned by a particular failure mode before. Whatever it is, they have axioms that guide their actions instead of reacting to every new suggestion as if it arrived from nowhere.

That does not mean their instincts are always right. Nobody's are. But it is much easier to have a useful discussion with someone who says, “I think we should do this, and here is why,” than someone who never has a position at all.

Strong opinions should be loosely held

The important qualifier is that a good engineer can change their mind.

“Strong opinions, loosely held” is a cliché at this point, but I think it survives because it is useful. You should be able to advocate for a design and explain why you believe in it. Then, if someone finds a flaw, brings better data, or understands part of the system that you do not, you should be able to update your opinion without treating it like a personal defeat.

That is different from being wishy-washy. The goal is not to avoid taking a side until there is perfect information. The goal is to make the best call you can with what you know, and stay open to being wrong.

Some of the best technical discussions I have seen have been between people with real, competing opinions. They were useful because both people were willing to put their reasoning on the table. The disagreement was about the work, not about who needed to win.

It makes people trust you

I think people tend to trust a clear point of view, especially when it is backed by reasoning.

If you consistently explain what you think, why you think it, and what evidence would change your mind, people can understand how you make decisions. Even when they disagree with you, they know where you stand. That makes working together easier.

The opposite is harder. If someone refuses to take a position on anything, or changes direction depending on who spoke last, it is difficult to know whether they have actually evaluated the problem. It can feel like there is no one to rely on when a decision needs to be made.

Being opinionated does not mean being loud. Some of the most opinionated engineers I know are quiet. But when it is time to make a call, they can clearly say what they believe and defend it without getting defensive.

It helps when things go wrong

I also think having a point of view gives people a bit more mental fortitude when things get difficult.

When a project is failing, a production issue is happening, or someone strongly disagrees with your proposal, it is easy to get pulled around by the urgency of the moment. Engineers with clear principles seem less reactive. They can take in new information without losing the thread of what they are trying to accomplish.

Again, that is not stubbornness. Sometimes the new information means the plan needs to change. But it is easier to adapt deliberately when you have a reasoned starting point than when you are just responding to the loudest person in the room.

Trying to find that part of myself again

Over my career, I have been wrong plenty of times. Some of those times damaged my confidence more than they probably should have.

My reaction was to stop having opinions. I used to be very strongly opinionated, but after enough misses, I started treating every opinion as evidence that I was about to be exposed as stupid. So I started looking for consensus and peace instead. I would only give my opinion when someone pressed me for one, and even then it was usually weak and flimsy. The moment someone more experienced spoke up, I would let it blow over, even when I still did not agree.

That felt safer, but I do not think it made me a better engineer. It just meant I was less willing to put my reasoning on the table where it could be discussed, challenged, or improved.

I am trying to regain some of that confidence now. The stakes are usually not that high. Having an opinion is not the same as wearing a billboard that says I am stupid if I turn out to be wrong. It is making the best call I can with what I know, explaining why, and being willing to update it when someone shows me something I missed.

I do not need to have an opinion about every tiny thing, and I do not want to become difficult or attached to being right. But a good engineer has opinions. I am trying to find that part of myself again.

What I Notice About Really Good Senior Engineers

Jonathan Westerfield — Tue, 06 Oct 2026 15:20:19 +0000

Throughout my time at Amazon, I have worked with senior/principal engineers who made what seemed impossible look almost annoyingly normal.

Not because the work was actually normal. Sometimes they were dealing with a production issue, a weird bit of infrastructure, a project that had been stuck for weeks, or an ambiguous request that everyone had interpreted differently. They just had a way of taking the situation apart, finding the important part, and making it possible to move again.

I want to be careful about what I can claim here. I am not a staff or principal engineer and I have never been their manager. I cannot really evaluate how good they are at managing budgets, setting organization-wide strategy, or doing the invisible business work that probably matters a lot at that level.

But I have worked beneath some very good senior engineers, and I have noticed some common traits. The technical depth is obvious. The less obvious part is how much of their value comes from making other people more effective.

They still genuinely enjoy learning

This one stood out to me more than I expected.

The best senior engineers I have worked with are usually buried in work. They have a pile of responsibilities, people asking them questions, systems that only they seem to understand, and the occasional mysterious incident that appears at the worst possible time.

And yet, they still like learning new things.

Not in the performative “I read every new framework announcement” sense. More in the sense that they are curious. They want to understand why something works, try a tool, poke at an unfamiliar system, or figure out whether an idea from another field applies here.

I think that curiosity is part of how they got so good in the first place. Experience clearly matters, but experience by itself can turn into doing the same thing for ten years. The senior engineers who stand out seem to keep adding new material to work with.

Their technical depth is real, but it is also connected

Yes, the obvious one: they know a lot.

They have usually seen more systems fail, more strange edge cases, more migrations, more shortcuts that came back to haunt someone, and more versions of “this should be simple” than I have. They can often explain what is happening several layers below the thing I was looking at.

But the really useful part is not just that they can remember obscure details. It is that they can connect ideas that do not initially look related.

They might use an operating-systems concept to reason about a distributed system. Or recognize that a problem that looks like an application bug is actually a queueing problem, a data-model problem, or an incentive problem. Or they know some bizarre shell trick that saves everyone ten minutes a day and somehow turns it into a reliable little tool instead of a cursed one-liner nobody can maintain.

I watched this happen once with an issue that had been sitting around for about two months. A senior engineer heard about what another team in a completely different part of the organization was building. They spent a little time playing with that team's code and realized that the approach they were using could also work in our codebase. It was not an obvious connection, and I do not think anyone on our team would have thought to look there. But that small bit of exploration gave us a way through an issue that nobody had managed to tackle.

They come at problems from odd angles because they have more angles available.

That feels like one of the biggest differences to me. A less experienced engineer may know the tools directly in front of them. A great senior engineer has enough experiences stored away that they can reach sideways and ask, “What does this problem resemble? Could I pull the answer from someplace else?”

They can explain things without making you feel dumb

Deep knowledge is helpful. Being able to share it is a separate skill.

The senior engineers I have learned the most from can explain a complicated concept to different people at different levels. They can give a stakeholder the actual tradeoff without burying them in implementation details. They can walk a junior engineer through the mechanics. They can talk to another experienced engineer without slowing the conversation down.

Most importantly, they do not make questions feel expensive.

There are people who are brilliant and still intimidating enough that everyone works around them. You hesitate to ask for help because you expect the interaction to be painful, so small misunderstandings survive longer than they should. That is not a great setup for a team, no matter how technically capable the person is.

The best people I have worked with somehow do the opposite. They make you feel like the question is reasonable, even when the answer is “this is weird and I also had to learn it the hard way.” That makes it much easier to learn, and it makes the whole team faster over time.

They are force multipliers, not just problem solvers

This is probably the trait I admire most.

Senior engineers are often the people who can take on the hardest work themselves. But the most effective ones do not just collect the hard work and disappear into it. They take a large, unclear problem and turn it into pieces other people can own.

That might mean clarifying the actual goal, drawing the boundaries between components, making a small technical decision that unblocks three people, or handling a tiny but horrible issue that has been wasting everyone else's time. Sometimes the highest-leverage thing they do is not glamorous at all. It is finding the one permissions issue, flaky deployment step, unclear API contract, or missing bit of documentation that has quietly stopped other work from moving.

They also seem comfortable letting other people take real ownership. Instead of keeping the interesting parts for themselves, they create a path for someone else to do them successfully. That requires trust, patience, and a good sense of where help is actually needed.

In practice, this can make them look like they are doing less than the person who is heads-down implementing a huge feature. I think they are often doing something harder: increasing the amount of good work the whole group can get done.

They sometimes end up doing manager-shaped work

There is a weird space between pure implementation and management, and good senior engineers frequently seem to occupy it.

They help untangle priorities. They make sure people know what needs to happen next. They resolve disagreements before they become months of low-grade frustration. They coach people through a problem instead of just fixing it for them. They notice when someone is blocked, overloaded, or about to build the wrong thing for a perfectly understandable reason.

I do not mean that engineers should quietly become unpaid managers. Teams still need actual managers, and management is its own demanding job.

But from below, it is very noticeable when a senior engineer can provide some of that direction and stability without turning every technical conversation into a power struggle. It makes a team feel much less chaotic.

They have the people skills to make their technical judgment usable

This is related, but I think it deserves its own point.

Someone can be technically extraordinary and still be difficult to work with. Maybe they are dismissive, impossible to approach, combative in every disagreement, or so scary that nobody brings them bad news until the problem has become much worse.

That limits how much influence their technical judgment can have.

The people who move into real positions of trust seem to pair technical credibility with basic human skills: listening, being direct without being cruel, admitting uncertainty, and treating people like adults. They can disagree without making the disagreement personal. They can say “I think this is a bad idea” and still leave room for a useful conversation.

It sounds painfully obvious when written out. It is much less common in practice than I would have guessed.

They know when the technically interesting answer is not the right answer

The best senior engineers I have worked with like experimenting. They are curious about tools and architecture and would probably enjoy building the more elegant version of a system.

They are also unusually good at asking whether that version actually solves the problem the business has.

Sometimes the right answer is the new tool. Sometimes the right answer is a boring database, a smaller design, a manual process for now, or simply not building the thing at all. The point is not that business needs should always beat technical concerns. Technical debt, reliability, and maintenance costs are business concerns too, even if they do not show up in a spreadsheet immediately.

The useful skill is being able to hold both sides at once: understand the technical tradeoff deeply enough to see the future cost, while staying grounded in what the company or user actually needs right now.

They do not wait to be handed every problem

The senior engineers I have watched most closely do not need a ticket to notice that something is wrong.

They seem to have a working model of where the business is going, what the team is trying to accomplish, and what is likely to break on the way there. Then they go looking. They ask questions, notice rough edges, and take care of problems before they become urgent enough for somebody to officially assign them.

That does not mean randomly refactoring everything in sight. I think it comes back to judgment. They have enough context to distinguish between a personal annoyance and something that will genuinely slow down the team or the business.

At their best, the only direction they need is the direction the business is moving. They can identify the important technical work from there.

They have enough trust to get things done across the company

Finally, there is social capital.

The strongest senior engineers tend to know a lot of people, but it is not just a giant collection of Slack contacts. They have earned trust across teams. People know they understand their systems, keep their word, explain their reasoning, and will not throw someone under the bus when a project gets difficult.

That makes it much easier for them to get things done. They can ask the right person for context. They can bring two groups together before a decision hardens into a conflict. They can influence a plan without needing formal authority over everyone involved.

“Playing politics” has a bad reputation, and sometimes for good reason. But there is a non-sleazy version of it: understanding people, incentives, and relationships well enough to make useful work happen. I do not think you can be effective at a senior level without some version of that skill.

What I am taking from this

When I think about becoming a better engineer, my first instinct is usually to focus on technical depth. Learn another system. Get better at debugging. Read more. Build more things. That is still important, obviously. The people I am describing are technically strong for a reason.

But when I take a step back, I do not think the answer is just becoming the person who knows the most things.

I am trying to get closer to the other traits too: staying curious even when I am busy, explaining things clearly, paying attention to what is blocking other people, and understanding enough of the wider context to work on the problems that actually matter. A lot of that happens adjacent to writing code. It means talking to people, learning how other teams work, writing things down, and noticing friction before it turns into a fire.

I am definitely not there yet. But I find that a more useful goal than simply trying to become a deeper technical specialist. The engineers I admire most do not just have more answers. They make it easier for everyone around them to find answers and get useful work done.

That is the kind of engineer I am trying to become.

Why I Chose SQLite Over Postgres

Jonathan Westerfield — Wed, 16 Sep 2026 14:56:00 +0000

I have been building a little project called Librarian: a local-first assistant for my EPUB collection. The basic idea is that I can point it at a folder of books, have it extract and chunk the text, create embeddings locally, and then ask questions with citations back to the source.

At some point I had to choose where all of that stuff would live: book metadata, chapters, raw chunks, embeddings, summaries, tags, and ingestion state.

I chose SQLite.

Not because Postgres is bad. Postgres is great. I have actually been spending time building out a PostgreSQL backend course for myself because I want to get much better at using it. It has better tooling for concurrent users, a real server process, pgvector, mature migrations, replication options, and basically every feature I could want if this became a normal cloud application.

But Librarian is not a normal cloud application.

What Librarian actually is

The important detail is that Librarian is designed around one person's local book library.

The EPUBs live in a folder on my computer. The app parses them locally, keeps the source text and metadata locally, and runs embeddings through a local model. The resulting database is mostly there to make my own collection searchable.

There is no multi-user app. There are no user accounts. There is no fleet of web servers racing to update the same rows. Ingestion is pretty bursty: I add a book, the program does a bunch of work, then I mostly read and search the results. That is a very different workload from, say, an API that has a bunch of people constantly writing to it.

SQLite is a pretty natural fit for that. It is a single local file, it is boring, and it has basically no operational ceremony. I do not need to run another database container, keep credentials around, think about a network connection, or wonder why my personal book search tool cannot start because a separate service decided to have a bad day.

It is also fast. Like, surprisingly fast. For ordinary relational queries over one person's local library, SQLite is not some sad little temporary database that needs to be replaced the second a project becomes real. It is embedded in the application, avoids a network round trip, and handles this kind of read-heavy local workload with very little drama. The slow part I have hit is vector search across enough embeddings, not SQLite doing normal database things. I have a hunch that bottleneck was actually due to computing cosine similarity across the embeddings (using Numpy), not the database reads.

I like Postgres, but using Postgres for this situation is like using a bazooka to hunt squirrels. It's just overkill.

Starting simple was the actual requirement

The first version of Librarian needed to prove that the whole retrieval loop was useful:

Can I ingest messy EPUB files reliably?
Can I preserve enough chapter and source information for citations?
Are local embeddings good enough to find useful passages?
Does asking questions across my own books actually feel helpful?

None of those questions get meaningfully easier because I started with a production-grade database server.

SQLite gave me a place to store the book records, raw text, chunks, vectors, summaries, and job state while I figured out the more interesting problems. Once the basic loop worked, I could see the actual bottleneck: scanning vectors in SQLite gets slow enough to be annoying. That is a retrieval-index problem, not a sign that every bit of data needs to graduate to Postgres. That is why I am adding OpenSearch as a local hybrid-search index for vector, keyword, and filtered retrieval.

That is also the part I like about the choice. SQLite is still the source of truth. I can improve the part that needs improvement without prematurely turning the entire project into a small distributed system that I now have to babysit.

If this were a cloud product, I would choose Postgres

If Librarian needed to put the book data in the cloud, I would probably choose Postgres. That is a hypothetical, though, not a roadmap item.

That would be a totally reasonable product decision. A hosted service that lets people use their library from multiple devices, sync their data, have accounts, share collections, or access it from a web app has very different needs. At that point, a managed Postgres database would give me durable remote storage, concurrent access, backups, migrations, observability, and a much more normal path to operating the thing.

SQLite would start getting awkward because its biggest strength here is that the data and the application are sitting together on one person's machine.

But I don't want to build that. I made Librarian to learn how RAG actually works: ingestion, chunking, embeddings, retrieval, citations, and the rest of the weird little pipeline. It also does the useful thing I wanted it to do, which is let me search and ask questions about my own book collection. That fulfills the need.

I am not interested in turning this particular project into a hosted backend with accounts and cloud sync.

Putting the actual book data in the cloud would also be a terrible idea from a copyright-risk perspective.

SQLite is not a permanent moral commitment

I think people sometimes talk about database choices as if they are picking a sports team. I am not on Team SQLite (I do like it though). I am on Team “use the thing that matches the project you are actually building.”

For a local-first, single-user tool that is still proving its core loop, SQLite is the right amount of database. It keeps the project easy to start, easy to inspect, and easy to move around. If the requirements change, I can change the storage architecture too.

If I ever build a different product that genuinely needs to host data in the cloud, Postgres will be waiting for me. Librarian does not need to become that product.

I Violated an API Contract by Fixing a Timeout Bug

Jonathan Westerfield — Sat, 05 Sep 2026 17:02:02 +0000

Apparently Contracts Should Be Honored

When I think of contracts, I think of scary documents that limit my freedom, usually in exchange for something I want. Documents like EULAs, user agreements, divorce papers, and prenups are just a few examples.

However, these contracts also provide a sense of comfort. If the terms serve you, then you take comfort in knowing that the contract is in your favor and the other party is forced to uphold it. Even if the terms are bad, you at least know what you're getting.

The common theme is trust in the contract.

The contract is sacred, and both parties must uphold their end of the deal. Break that agreement, and suddenly the contract is just an ideal on a worthless piece of paper, or a text document file. When this trust is broken, everything falls to pieces. People get mad and things start breaking, emotionally, spiritually, financially, or otherwise.

This holds true for API contracts. The provider agrees to define and uphold a specific interface for other developers to use. This not only tells developers how to actually use the API, but it also guarantees that they don't need to worry about the contract suddenly, magically changing under their feet. Any changes to this contract must involve all parties, providers and users, to a reasonable degree. At the very least, users need to be notified of the change and given a window to adjust.

Unplanned changes bring chaos, so who on Earth would violate this sacred trust?

The Setup

I implemented a CoAP handler in Lua on our servers. It's a simple framework that takes CoAP requests, calls a library on the server, and returns the results. If there are no results, it returns nothing.

The key thing is that each request to the server happens in two stages:

A POST request to a corresponding endpoint with the information needed to execute the action.
A GET request with the request ID from the POST to get the result.

For long-running actions, you would poll until the request succeeded. If an action takes too long, there is a default timeout built into the framework to kill the server process, which would be reflected in the GET request.

The issue is that I didn't implement that part correctly.

A Faulty Timeout Mechanism

The timeout mechanism didn't measure time correctly. I incorrectly started the timer at the wrong point in the request flow. The result was that the effective timeout was slightly longer than the default of five seconds. On the server side, the fix was pretty simple: just move the start of the timer to the correct place in the code flow.

But what about the client side?

This framework was unique in that I controlled both the client- and server-side code. I could quickly check for all calls to the CoAP endpoints, make my server-side change, and bundle that with client-side fixes too. I could also easily run tests to make sure nothing broke. Easy, right?

So that's what I did. I made the timer actually time things correctly, tested the changes against our codebase on a known-good server, raised a PR, and pushed the change once it was approved.

The whole thing took me a little over a day. This was back before my team used AI, so I felt pretty good about how speedy the whole affair was.

Why Do I Hear Police Sirens?

One small problem: suddenly our release was blocked.

Before we release code changes to production, the whole org runs tests on our servers to ensure everything works. However, every once in a while, sporadically, a request from my framework would randomly time out.

A typical release cycle can take two whole days, and multiple teams from all over our business unit put their changes into this release. Let's just say that since there are so many chefs in this kitchen, tensions run high and any release blockers cause people to get really testy. This is not a chill release cycle. It's akin to a full-on production issue.

The engineer on call pinged me and let me know about the problem. I wracked my brain for possible causes and easy fixes, but nothing came to mind.

What we did know was that it was related to my commit. For production issues, the first lever to grab is a rollback. In this case, since we knew it was my commit causing this, I quickly pushed a revert to unblock the release. No sense in trying to debug during a production event.

It was quick to stop the bleeding, but I'll admit that pushing that revert still embarrassed me. I was committed to figuring out what went wrong.

To Catch a Killer (Bug)

Now that the release was unblocked and things had calmed down, it was time to get to the bottom of this.

I went back to my code and performed my sanity checks. I jumped on a known-good server and ran my tests again.

Everything passed. That's annoying.

I did code profiling to check the hotspots in my code. I ran requests in a loop to see if any of them returned slightly slower than the others. Nothing worked. The requests still ran normally and returned without timing out. I even grabbed the exact server that failed during the release and ran the tests again. Everything passed...

If the server that presented the problem wasn't able to reproduce the issue, I was out of options. I had hit the limit of my knowledge.

A Break in the Case

Since I ran out of leads, I did what any engineer does: I asked other engineers. I talked to—and at—people about my problem to see if anyone could come up with a reason this could have happened. No one had any clue. They were mostly hardware engineers, so that figures. This wasn't their realm of knowledge.

Then I ran across our principal engineer at lunch and gave him the same spiel. He thought about the problem for a bit and asked me something that hit me like a truck.

"Do you know you changed the API contract?"

I told him, "Surely not, right?" I didn't change how developers interacted with the APIs. I didn't change parameters or return values. The API contract was completely unchanged, right?

Wrong.

He told me that even changing the timeout functionality is technically changing the contract. It's not as obvious as changing the structure of an API, but it's still an unplanned, uncoordinated change.

But that's not a problem, right? I controlled both the client and server code. I was the provider and user, so I should have been able to change it with impunity.

What he said next changed the whole picture.

"This is the problem with unplanned changes. You can't know how they will affect the system. Generally, when you make an API change, you add optional parameters or, preferably, version the API to a V2 for others to use. Any change to the existing API opens you up to a world of pain, so you shouldn't make those changes unless you absolutely have to, even in your case where you control both worlds. In this case, it sounds like maybe the servers got bogged down and slowed down just enough to cross the five-second limit; the NEW, SHORTER limit you set."

My Smoking Gun

It was so obvious. I can't believe I missed it.

When I was doing my investigation, I was calling my framework on known-good servers.

I had never considered using bad servers or intentionally making lots of simultaneous calls to bog down the server. The compute on these servers was tiny, so this is not hard to do. I originally chose the five-second timeout because that was the upper limit on how long a call originally took. I ran numbers to calculate that timeout, but didn't think about that when I changed the timeout mechanism. Nor did I seriously consider just how much I was shortening the effective timeout of a request.

I had three things to do:

Confirm that bogging down the server could cause the timeout to be breached.
Fix my API contract by changing the timeout to what it was before.
See if I needed a longer default timeout than the original timeout in case the server got bogged down. Slow servers are not actually code bugs. These things happen in production, and the code should handle them gracefully.

All three were simple.

First, I wrote a simple script to make 100 calls to the endpoints and see if any of them timed out. I also helped out by running other heavy processes on the server to further burden the resource-constrained CPU. This immediately surfaced and confirmed the issue.

Great.

Second, I made calls to the endpoints on both the new and old code to see how long each request took. The difference was when the start time occurred in the code flow. I could take the difference in those request times and add it to the default timeout value, effectively restoring its old implicit value. That gave me a correct timeout mechanism without randomly timing out requests that used to work.

A quick PR later, and we were back at it. The code ran through the release testing and got pushed into production. No problem.

Penance and Reflection

So obvious, and I somehow missed it.

I had committed the cardinal sin of API contracts and got burned. Changing an API contract doesn't have to mean something as explicit as changing parameters or return values.

I made what I thought was a tiny cleanup change, but the timeout was part of the deal whether I had written it down that way or not. Nothing about the endpoint changed, but on a sufficiently grumpy server, requests that used to succeed could now fail. API contracts are also the assumptions someone can build code around, including the ones you don't know exist yet.

I learned that with a blocked release and a reverted PR. Now, whenever I make changes, I assume existing server behavior is part of the contract.

We add new features. We don't change existing ones.

The Start of My RAGgedy Journey

Jonathan Westerfield — Sat, 18 Jul 2026 00:59:40 +0000

Big Howdy,

I'm so tired of hearing all the hype about AI.

Don't get me wrong, I think AI is pretty cool and I use it all the time for various tasks throughout the day, but I hate AI marketing.

What do I mean by that? Whenever there is any hype, advertising, or corporate shilling about AI, it's always vague. Talks about agents, multi-agent systems, RAG pipelines, etc. are everywhere, and yet the specifics are always just out of reach.

Even when I try to dig into deeper technical articles, the handwaving usually begins almost immediately. Several architecture diagrams and acronyms later, I still do not have a much better idea of what is actually happening (although, to be fair, I wasn't exactly looking that hard).

I’m going to fix that.

For myself, at least.

First Stop: RAG

My first target is RAG. It's been out for quite a while now and I've been hearing about it for forever. It also sounds like enough progress has been made that the problem has effectively been solved. Best practices have been established and it's pretty commonplace now, so it's a great place to start.

But what is RAG?

RAG stands for Retrieval-Augmented Generation, which is... not a helpful name if you are just learning about this for the first time like I am. However, if you use AI at all, you've already been using a form of it without even knowing it.

RAG is simply a pattern:

Retrieve relevant external information.
Add that information to the AI model’s working context.
Generate an answer grounded in it.

That's it.

When you use an AI assistant and it goes and searches the internet for relevant information to give you an answer, that's a form of RAG. That doesn't mean your AI assistant is a RAG system, but it can perform RAG functions.

But this STILL sounds a bit hand-wavy to me. How does this work in real life?

The Part I Kept Missing

Context windows are massive these days, but I have seen technical documents that are literally more than 3,000 pages long and full of dense technical information. Even if one technically fits inside a model’s context window, sending the entire document with every question would be slow, wasteful, and probably not very good at finding the exact paragraph I need.

But an LLM would be perfect for working with documents like that. If I am reading a specification for something like High Bandwidth Memory, it would be fantastic to ask:

What does each bit in this register represent? If a bit maps to a different register, show me that too.

That would be infinitely more convenient than manually jumping around a massive PDF trying to find every related table, footnote, and register definition.

That's where the "retrieval" part of a RAG system comes in. Somehow, I need to structure and index the data, search it based on the user's question, and then give the relevant results to the LLM.

That's the part I'm going to figure out.

Enter Librarian

My project idea for learning this is to create an assistant that specifically answers questions about my personal ebook library. I call it Librarian (original, I know). Librarian will ingest my ebooks, many of which are several megabytes in size (although the size comes from images embedded in the file, not the text itself), structure the data, then search across it when I prompt it.

The goal is to support questions like:

"What is this book about?"
"What does this author have to say about this topic?"
"Compare how these books discuss suffering."
"Recommend a fantasy book with war themes."
"Where does this book explain this specific idea?"

The important part is that Librarian should not simply produce a plausible answer. It should retrieve evidence from my books and cite the passages it used.

Of course, describing the finished product is the easy part. I still need to figure out what happens between dropping an ebook into a folder and getting a useful answer back.

How do I extract and structure the text? How do I make thousands of pages searchable? How does the system decide which passages are relevant? How do I preserve enough information to cite the book, chapter, and section an answer came from?

And most importantly, how do I know the answer is actually based on my books instead of something the model made up? The engineer in me can't stand relying on nondeterministic output without something I can measure, so I need a way to tell whether the system actually works and quantify how well it works.

These are the parts I want to understand.

I have already started building Librarian, and the next few posts will follow the project as I work through them. The next post will cover the high-level design: what components I think the system needs, what each one is responsible for, and how information will move from an EPUB file to an answer.

After that, I’ll start digging into the individual pieces.

You can follow the project here:

JonathanGWesterfield/Librarian

I Failed: A Post-Mortem

Jonathan Westerfield — Tue, 14 Jul 2026 04:00:04 +0000

King Yemma and KY are pseudonyms. I have changed or generalized names and a few identifying details to keep internal information private.

Big Howdy,

I want to write about a project that, on paper, went pretty well.

The Project

Our server fleet had a major problem with dirty reboots. Think “Blue Screen of Death,” but on a much more expensive computer. Sometimes it happened when testing servers before putting them into production, which was annoying but manageable. Other times it happened in production while a customer was using the server, which was considerably less manageable and considerably more expensive for everyone involved.

The fleet health team was already investigating these failures, but there were a lot of them and the logs were a pain to work through. The servers are wildly complex, meaning there could be hundreds of reasons for one to crash like this.

To scale to the amount of work, I was assigned to onboard our servers to an internal tool called King Yemma. Dragon Ball reference, anyone? From here on out, I am calling it KY.

I was the only person from my team assigned to this work, so I worked directly with both the team that owned KY and the fleet health team.

The idea was simple enough: KY would pull the relevant logs for a dirty reboot, look for known failure patterns, and categorize the ticket. If it found a known issue, it could close the ticket automatically. If it found something new, it could help point the investigation in the right direction by tagging the ticket based on other rules in KY. The original team that built it was already using it with great success.

There was one problem: KY had not been built for our servers.

Getting KY Working

I worked with the team that owned KY to make it understand our systems. I added our log groups, pointed it at the right resources, and taught it where to find our dirty reboot tickets. I also added a feature to parse the logs and pull out the component that appeared to have failed, which was one of the requests that came down while I was working on it.

After that, I started working with fleet health to add fingerprints.

A fingerprint was basically a known pattern in the logs that mapped to a known failure. Generally, this meant doing a regex or pattern match on one or more log lines. The goal was to get enough of these into KY that it could automatically close tickets for problems that had already been seen and understood. If it did not have an exact match, KY could at least add tags to the ticket to point engineers in the right direction.

This meant I had to spend a lot of time looking through dirty reboot logs. Don’t get me wrong, the fleet health team worked with me and provided fingerprints when they found things too. It is not like they simply expected a software engineer to suddenly start doing a fleet health or hardware engineer’s job overnight. However, I still needed to do my part to speed up development.

I would find something in the logs that looked suspicious, ask someone from the fleet health team whether it was actually important (it rarely was), and then repeat. Turns out there are a lot of weird warnings and error messages that pop up during runtime that are apparently... completely meaningless?

Still, I got KY working on our systems. Together with the fleet health team, I added numerous fingerprints that could automatically close tickets. It also helped identify new buckets of failures for the team to investigate.

KY had another useful feature too. Someone could give it a server and a time range, and it would pull the relevant logs into one place on their local machine. Then they could use grep, awk, sed, or whatever else they wanted without waiting on CloudWatch or manually jumping between log groups. This meant KY could be useful for more than just fleet health. Everyone in the org could use it for their own debugging.

I cannot take credit for implementing that feature, but I CAN take credit for exposing it in the CLI for a normal user to call. I even gave a short presentation about it to the org.

At this point, it sounds like a smashing success, right?

If only it actually was...

So Much Work, So Little Value...

The actual goal for this whole boondoggle was to get the fleet health team to adopt KY as part of their normal investigation process for dirty reboots. The goal was to reduce the time they spent digging through logs and make the whole investigation process less painful and more automated. After all, there were only a handful of fleet health engineers and literally thousands of servers.

Unfortunately, they thought it was only slightly better than useless and decided manually searching through CloudWatch was faster.

And, honestly... they were right.

KY ended up being useful mostly after a problem had already been diagnosed. It could automatically close tickets for known failures, which was helpful, but it was not the investigation tool we all hoped it would become.

I had built something that worked well, but I had not built something my customer wanted to use.

What I Got Wrong

Looking back, my mindset, along with my lack of awareness, was a major part of the problem.

First, I did not understand our system very well at the time, and I did not understand the customer use case or how the fleet health team operated nearly as well as I thought I did.

Second, I was under a lot of organizational pressure at the time. I thought I had made a mistake that severely affected my standing, and was worried that KY was a test of whether I could recover. That fear turned out to be mistaken, but it changed how I approached the work.

As a result, I honestly hated this project. I hated reading logs that I did not understand. I hated programming Python when my normal language was Lua, since the languages are just similar enough that I would constantly confuse the syntax (this was before the days of AI). It frustrated the hell out of me every day.

But I wanted to perform well, show progress, and get to something I could point at and say, “look, I did the work.” Mostly, I just didn't want to get fired, which I was convinced would happen if I so much as sneezed too loudly.

The easiest way to do that was to add fingerprints. More fingerprints meant more tickets could be categorized automatically. More fingerprints meant more pull requests for the codebase. More fingerprints gave me a metric.

The problem was that the metric did not matter.

The fingerprints the fleet health team actually needed were often more complicated than simple log matching. Sometimes they depended on values that had to be decoded or interpreted in a particular way. Sometimes the decoded value had to come from a very specific component to have meaning. Sometimes specific values needed to come from multiple specific components within a specific time range.

The more useful work would have been making KY flexible enough for fleet health to create and manage those fingerprints themselves. I did some of that, but it was clearly secondary to adding fingerprints.

I was trying to do their job for them instead of building the thing that would help them do their job better.

I also did not understand the fleet health team’s workflow well enough. I talked to them, got feedback, and worked with them on fingerprints, but I was still mostly seeing their work from the outside. I understood the inputs and outputs more than I understood the actual process, or the system itself.

I should have spent more time sitting with them while they investigated failures. I should have asked my manager to loan me to the team for a bit so I could understand how they debugged, what information they needed first, and where the existing process was actually slow.

Instead, I focused on the work that felt safest.

What I Learned

There was also a more personal lesson in it for me. At the time, I was still learning how to operate in an environment where I had no solid footing. There was a lot of information, a lot of system complexity, and no obvious answer. These servers were complicated enough that a single machine was basically its own distributed system. There were hardware blocks, firmware, software, logs from different sources, and a bunch of interactions I did not understand yet, many of which I never figured out.

When it came to logs, I would sometimes feel so overwhelmed that I would shut down and not see the obvious line saying, “Error! I’m literally the root cause!”

When I felt overloaded, I had a tendency to narrow my focus until I found something concrete that I could make progress on. In this case, that became fingerprints.

Fingerprints were real work. They were useful. But they were also safer than going deeper into the uncomfortable question of whether KY itself was solving the right problem.

The project taught me a lot about our systems, and that knowledge was extremely useful later when I was writing tests and assisting with, and eventually driving, future investigations. It also gave me better relationships with both the KY team and the fleet health team, so I did gain a lot from it.

However, it was still a failure. After two months of work, I had nothing enduring to show for it because it was quickly forgotten. Even the log pulling feature was never really adopted after I gave that presentation on it.

KY failed to accomplish its most important goal because I optimized for what I could measure instead of what the customer actually needed. I was so worried about proving that I was doing good work that I stopped asking whether I was doing the right work.

My First Post

Jonathan Westerfield — Sat, 11 Jul 2026 19:25:39 +0000

Big Howdy,

I’m starting a blog.

That sentence feels weirdly dramatic for what this is to me.

The plan is to write about engineering, personal projects, things I’m learning, things I thought I understood but absolutely did not understand, and the occasional reflection after I’ve spent way too long staring at a problem that was probably my fault the whole time.

I’ve wanted to write more for a while, but I kept running into the same problem: a lot of engineering writing feels a little too polished; almost a little fake (I use semi-colons, no it's not AI).

Not bad necessarily. Just polished in a way where the author always seems to have known the correct abstraction, the correct database, the correct deployment strategy, and the correct lesson before the story even started.

I am not that guy.

Most of my actual engineering experience has been much messier.

I'll start with some assumptions, plow ahead thinking I know what I'm doing and then hit a wall realizing that the foundation of my idea was flawed because of... my assumptions. You know what they say about assumptions right? From there I take a step back, untangle the mess and get it right on the 4th try. If it was easy, I probably wouldn't want to do it anyway.

This is all well and good, but the real reason I want to write is because I learn better when I explain things.

If I can write down what happened, what confused me, what I tried, what worked, and what I’d do differently next time, then maybe the lesson actually sticks. And if it helps someone else avoid one of my mistakes, even better. I can’t promise the mistakes will be impressive, but they will be honest.

The posts will probably be pretty short. I’m not trying to write a novel every time I discover a new edge case. That said, I reserve the right to accidentally write too much if I get interested in something. This has happens more than I'd like to admit...

The topics will likely be a mix of:

personal projects
debugging stories
engineering tradeoffs
tools I’m trying
things I learned the hard way
reflections on work, burnout, confidence, and getting better at this without pretending it’s all obvious

I’m not writing this because I think I’m some grand authority on software engineering. I’m very much still learning; sometimes even successfully. Sometimes through doing something dumb and then having to deal with the consequences (honestly the most impactful learning).

But I do think there’s value in writing from that place.

Not the “here are the ten rules for building perfect software” place.

More like the “here’s what I was trying to do, here’s where I got confused, here’s what finally clicked, and here’s the part I still don’t fully understand” place.

That feels more useful to me. At least it feels more real.

So that’s what this blog is going to be. Small posts about engineering and projects, written as honestly as I can manage, without trying to sound smarter than I am, because I'm not that smart :)

We’ll see how it goes.