DEV Community: Jonathan Westerfield

I Violated an API Contract by Fixing a Timeout Bug

Jonathan Westerfield — Sat, 05 Sep 2026 17:02:02 +0000

Apparently Contracts Should Be Honored

When I think of contracts, I think of scary documents that limit my freedom, usually in exchange for something I want. Documents like EULAs, user agreements, divorce papers, and prenups are just a few examples.

However, these contracts also provide a sense of comfort. If the terms serve you, then you take comfort in knowing that the contract is in your favor and the other party is forced to uphold it. Even if the terms are bad, you at least know what you're getting.

The common theme is trust in the contract.

The contract is sacred, and both parties must uphold their end of the deal. Break that agreement, and suddenly the contract is just an ideal on a worthless piece of paper, or a text document file. When this trust is broken, everything falls to pieces. People get mad and things start breaking, emotionally, spiritually, financially, or otherwise.

This holds true for API contracts. The provider agrees to define and uphold a specific interface for other developers to use. This not only tells developers how to actually use the API, but it also guarantees that they don't need to worry about the contract suddenly, magically changing under their feet. Any changes to this contract must involve all parties, providers and users, to a reasonable degree. At the very least, users need to be notified of the change and given a window to adjust.

Unplanned changes bring chaos, so who on Earth would violate this sacred trust?

The Setup

I implemented a CoAP handler in Lua on our servers. It's a simple framework that takes CoAP requests, calls a library on the server, and returns the results. If there are no results, it returns nothing.

The key thing is that each request to the server happens in two stages:

A POST request to a corresponding endpoint with the information needed to execute the action.
A GET request with the request ID from the POST to get the result.

For long-running actions, you would poll until the request succeeded. If an action takes too long, there is a default timeout built into the framework to kill the server process, which would be reflected in the GET request.

The issue is that I didn't implement that part correctly.

A Faulty Timeout Mechanism

The timeout mechanism didn't measure time correctly. I incorrectly started the timer at the wrong point in the request flow. The result was that the effective timeout was slightly longer than the default of five seconds. On the server side, the fix was pretty simple: just move the start of the timer to the correct place in the code flow.

But what about the client side?

This framework was unique in that I controlled both the client- and server-side code. I could quickly check for all calls to the CoAP endpoints, make my server-side change, and bundle that with client-side fixes too. I could also easily run tests to make sure nothing broke. Easy, right?

So that's what I did. I made the timer actually time things correctly, tested the changes against our codebase on a known-good server, raised a PR, and pushed the change once it was approved.

The whole thing took me a little over a day. This was back before my team used AI, so I felt pretty good about how speedy the whole affair was.

Why Do I Hear Police Sirens?

One small problem: suddenly our release was blocked.

Before we release code changes to production, the whole org runs tests on our servers to ensure everything works. However, every once in a while, sporadically, a request from my framework would randomly time out.

A typical release cycle can take two whole days, and multiple teams from all over our business unit put their changes into this release. Let's just say that since there are so many chefs in this kitchen, tensions run high and any release blockers cause people to get really testy. This is not a chill release cycle. It's akin to a full-on production issue.

The engineer on call pinged me and let me know about the problem. I wracked my brain for possible causes and easy fixes, but nothing came to mind.

What we did know was that it was related to my commit. For production issues, the first lever to grab is a rollback. In this case, since we knew it was my commit causing this, I quickly pushed a revert to unblock the release. No sense in trying to debug during a production event.

It was quick to stop the bleeding, but I'll admit that pushing that revert still embarrassed me. I was committed to figuring out what went wrong.

To Catch a Killer (Bug)

Now that the release was unblocked and things had calmed down, it was time to get to the bottom of this.

I went back to my code and performed my sanity checks. I jumped on a known-good server and ran my tests again.

Everything passed. That's annoying.

I did code profiling to check the hotspots in my code. I ran requests in a loop to see if any of them returned slightly slower than the others. Nothing worked. The requests still ran normally and returned without timing out. I even grabbed the exact server that failed during the release and ran the tests again. Everything passed...

If the server that presented the problem wasn't able to reproduce the issue, I was out of options. I had hit the limit of my knowledge.

A Break in the Case

Since I ran out of leads, I did what any engineer does: I asked other engineers. I talked to—and at—people about my problem to see if anyone could come up with a reason this could have happened. No one had any clue. They were mostly hardware engineers, so that figures. This wasn't their realm of knowledge.

Then I ran across our principal engineer at lunch and gave him the same spiel. He thought about the problem for a bit and asked me something that hit me like a truck.

"Do you know you changed the API contract?"

I told him, "Surely not, right?" I didn't change how developers interacted with the APIs. I didn't change parameters or return values. The API contract was completely unchanged, right?

Wrong.

He told me that even changing the timeout functionality is technically changing the contract. It's not as obvious as changing the structure of an API, but it's still an unplanned, uncoordinated change.

But that's not a problem, right? I controlled both the client and server code. I was the provider and user, so I should have been able to change it with impunity.

What he said next changed the whole picture.

"This is the problem with unplanned changes. You can't know how they will affect the system. Generally, when you make an API change, you add optional parameters or, preferably, version the API to a V2 for others to use. Any change to the existing API opens you up to a world of pain, so you shouldn't make those changes unless you absolutely have to, even in your case where you control both worlds. In this case, it sounds like maybe the servers got bogged down and slowed down just enough to cross the five-second limit; the NEW, SHORTER limit you set."

My Smoking Gun

It was so obvious. I can't believe I missed it.

When I was doing my investigation, I was calling my framework on known-good servers.

I had never considered using bad servers or intentionally making lots of simultaneous calls to bog down the server. The compute on these servers was tiny, so this is not hard to do. I originally chose the five-second timeout because that was the upper limit on how long a call originally took. I ran numbers to calculate that timeout, but didn't think about that when I changed the timeout mechanism. Nor did I seriously consider just how much I was shortening the effective timeout of a request.

I had three things to do:

Confirm that bogging down the server could cause the timeout to be breached.
Fix my API contract by changing the timeout to what it was before.
See if I needed a longer default timeout than the original timeout in case the server got bogged down. Slow servers are not actually code bugs. These things happen in production, and the code should handle them gracefully.

All three were simple.

First, I wrote a simple script to make 100 calls to the endpoints and see if any of them timed out. I also helped out by running other heavy processes on the server to further burden the resource-constrained CPU. This immediately surfaced and confirmed the issue.

Great.

Second, I made calls to the endpoints on both the new and old code to see how long each request took. The difference was when the start time occurred in the code flow. I could take the difference in those request times and add it to the default timeout value, effectively restoring its old implicit value. That gave me a correct timeout mechanism without randomly timing out requests that used to work.

A quick PR later, and we were back at it. The code ran through the release testing and got pushed into production. No problem.

Penance and Reflection

So obvious, and I somehow missed it.

I had committed the cardinal sin of API contracts and got burned. Changing an API contract doesn't have to mean something as explicit as changing parameters or return values.

I made what I thought was a tiny cleanup change, but the timeout was part of the deal whether I had written it down that way or not. Nothing about the endpoint changed, but on a sufficiently grumpy server, requests that used to succeed could now fail. API contracts are also the assumptions someone can build code around, including the ones you don't know exist yet.

I learned that with a blocked release and a reverted PR. Now, whenever I make changes, I assume existing server behavior is part of the contract.

We add new features. We don't change existing ones.

The Start of My RAGgedy Journey

Jonathan Westerfield — Sat, 18 Jul 2026 00:59:40 +0000

Big Howdy,

I'm so tired of hearing all the hype about AI.

Don't get me wrong, I think AI is pretty cool and I use it all the time for various tasks throughout the day, but I hate AI marketing.

What do I mean by that? Whenever there is any hype, advertising, or corporate shilling about AI, it's always vague. Talks about agents, multi-agent systems, RAG pipelines, etc. are everywhere, and yet the specifics are always just out of reach.

Even when I try to dig into deeper technical articles, the handwaving usually begins almost immediately. Several architecture diagrams and acronyms later, I still do not have a much better idea of what is actually happening (although, to be fair, I wasn't exactly looking that hard).

I’m going to fix that.

For myself, at least.

First Stop: RAG

My first target is RAG. It's been out for quite a while now and I've been hearing about it for forever. It also sounds like enough progress has been made that the problem has effectively been solved. Best practices have been established and it's pretty commonplace now, so it's a great place to start.

But what is RAG?

RAG stands for Retrieval-Augmented Generation, which is... not a helpful name if you are just learning about this for the first time like I am. However, if you use AI at all, you've already been using a form of it without even knowing it.

RAG is simply a pattern:

Retrieve relevant external information.
Add that information to the AI model’s working context.
Generate an answer grounded in it.

That's it.

When you use an AI assistant and it goes and searches the internet for relevant information to give you an answer, that's a form of RAG. That doesn't mean your AI assistant is a RAG system, but it can perform RAG functions.

But this STILL sounds a bit hand-wavy to me. How does this work in real life?

The Part I Kept Missing

Context windows are massive these days, but I have seen technical documents that are literally more than 3,000 pages long and full of dense technical information. Even if one technically fits inside a model’s context window, sending the entire document with every question would be slow, wasteful, and probably not very good at finding the exact paragraph I need.

But an LLM would be perfect for working with documents like that. If I am reading a specification for something like High Bandwidth Memory, it would be fantastic to ask:

What does each bit in this register represent? If a bit maps to a different register, show me that too.

That would be infinitely more convenient than manually jumping around a massive PDF trying to find every related table, footnote, and register definition.

That's where the "retrieval" part of a RAG system comes in. Somehow, I need to structure and index the data, search it based on the user's question, and then give the relevant results to the LLM.

That's the part I'm going to figure out.

Enter Librarian

My project idea for learning this is to create an assistant that specifically answers questions about my personal ebook library. I call it Librarian (original, I know). Librarian will ingest my ebooks, many of which are several megabytes in size (although the size comes from images embedded in the file, not the text itself), structure the data, then search across it when I prompt it.

The goal is to support questions like:

"What is this book about?"
"What does this author have to say about this topic?"
"Compare how these books discuss suffering."
"Recommend a fantasy book with war themes."
"Where does this book explain this specific idea?"

The important part is that Librarian should not simply produce a plausible answer. It should retrieve evidence from my books and cite the passages it used.

Of course, describing the finished product is the easy part. I still need to figure out what happens between dropping an ebook into a folder and getting a useful answer back.

How do I extract and structure the text? How do I make thousands of pages searchable? How does the system decide which passages are relevant? How do I preserve enough information to cite the book, chapter, and section an answer came from?

And most importantly, how do I know the answer is actually based on my books instead of something the model made up? The engineer in me can't stand relying on nondeterministic output without something I can measure, so I need a way to tell whether the system actually works and quantify how well it works.

These are the parts I want to understand.

I have already started building Librarian, and the next few posts will follow the project as I work through them. The next post will cover the high-level design: what components I think the system needs, what each one is responsible for, and how information will move from an EPUB file to an answer.

After that, I’ll start digging into the individual pieces.

You can follow the project here:

JonathanGWesterfield/Librarian

I Failed: A Post-Mortem

Jonathan Westerfield — Tue, 14 Jul 2026 04:00:04 +0000

King Yemma and KY are pseudonyms. I have changed or generalized names and a few identifying details to keep internal information private.

Big Howdy,

I want to write about a project that, on paper, went pretty well.

The Project

Our server fleet had a major problem with dirty reboots. Think “Blue Screen of Death,” but on a much more expensive computer. Sometimes it happened when testing servers before putting them into production, which was annoying but manageable. Other times it happened in production while a customer was using the server, which was considerably less manageable and considerably more expensive for everyone involved.

The fleet health team was already investigating these failures, but there were a lot of them and the logs were a pain to work through. The servers are wildly complex, meaning there could be hundreds of reasons for one to crash like this.

To scale to the amount of work, I was assigned to onboard our servers to an internal tool called King Yemma. Dragon Ball reference, anyone? From here on out, I am calling it KY.

I was the only person from my team assigned to this work, so I worked directly with both the team that owned KY and the fleet health team.

The idea was simple enough: KY would pull the relevant logs for a dirty reboot, look for known failure patterns, and categorize the ticket. If it found a known issue, it could close the ticket automatically. If it found something new, it could help point the investigation in the right direction by tagging the ticket based on other rules in KY. The original team that built it was already using it with great success.

There was one problem: KY had not been built for our servers.

Getting KY Working

I worked with the team that owned KY to make it understand our systems. I added our log groups, pointed it at the right resources, and taught it where to find our dirty reboot tickets. I also added a feature to parse the logs and pull out the component that appeared to have failed, which was one of the requests that came down while I was working on it.

After that, I started working with fleet health to add fingerprints.

A fingerprint was basically a known pattern in the logs that mapped to a known failure. Generally, this meant doing a regex or pattern match on one or more log lines. The goal was to get enough of these into KY that it could automatically close tickets for problems that had already been seen and understood. If it did not have an exact match, KY could at least add tags to the ticket to point engineers in the right direction.

This meant I had to spend a lot of time looking through dirty reboot logs. Don’t get me wrong, the fleet health team worked with me and provided fingerprints when they found things too. It is not like they simply expected a software engineer to suddenly start doing a fleet health or hardware engineer’s job overnight. However, I still needed to do my part to speed up development.

I would find something in the logs that looked suspicious, ask someone from the fleet health team whether it was actually important (it rarely was), and then repeat. Turns out there are a lot of weird warnings and error messages that pop up during runtime that are apparently... completely meaningless?

Still, I got KY working on our systems. Together with the fleet health team, I added numerous fingerprints that could automatically close tickets. It also helped identify new buckets of failures for the team to investigate.

KY had another useful feature too. Someone could give it a server and a time range, and it would pull the relevant logs into one place on their local machine. Then they could use grep, awk, sed, or whatever else they wanted without waiting on CloudWatch or manually jumping between log groups. This meant KY could be useful for more than just fleet health. Everyone in the org could use it for their own debugging.

I cannot take credit for implementing that feature, but I CAN take credit for exposing it in the CLI for a normal user to call. I even gave a short presentation about it to the org.

At this point, it sounds like a smashing success, right?

If only it actually was...

So Much Work, So Little Value...

The actual goal for this whole boondoggle was to get the fleet health team to adopt KY as part of their normal investigation process for dirty reboots. The goal was to reduce the time they spent digging through logs and make the whole investigation process less painful and more automated. After all, there were only a handful of fleet health engineers and literally thousands of servers.

Unfortunately, they thought it was only slightly better than useless and decided manually searching through CloudWatch was faster.

And, honestly... they were right.

KY ended up being useful mostly after a problem had already been diagnosed. It could automatically close tickets for known failures, which was helpful, but it was not the investigation tool we all hoped it would become.

I had built something that worked well, but I had not built something my customer wanted to use.

What I Got Wrong

Looking back, my mindset, along with my lack of awareness, was a major part of the problem.

First, I did not understand our system very well at the time, and I did not understand the customer use case or how the fleet health team operated nearly as well as I thought I did.

Second, I was under a lot of organizational pressure at the time. I thought I had made a mistake that severely affected my standing, and was worried that KY was a test of whether I could recover. That fear turned out to be mistaken, but it changed how I approached the work.

As a result, I honestly hated this project. I hated reading logs that I did not understand. I hated programming Python when my normal language was Lua, since the languages are just similar enough that I would constantly confuse the syntax (this was before the days of AI). It frustrated the hell out of me every day.

But I wanted to perform well, show progress, and get to something I could point at and say, “look, I did the work.” Mostly, I just didn't want to get fired, which I was convinced would happen if I so much as sneezed too loudly.

The easiest way to do that was to add fingerprints. More fingerprints meant more tickets could be categorized automatically. More fingerprints meant more pull requests for the codebase. More fingerprints gave me a metric.

The problem was that the metric did not matter.

The fingerprints the fleet health team actually needed were often more complicated than simple log matching. Sometimes they depended on values that had to be decoded or interpreted in a particular way. Sometimes the decoded value had to come from a very specific component to have meaning. Sometimes specific values needed to come from multiple specific components within a specific time range.

The more useful work would have been making KY flexible enough for fleet health to create and manage those fingerprints themselves. I did some of that, but it was clearly secondary to adding fingerprints.

I was trying to do their job for them instead of building the thing that would help them do their job better.

I also did not understand the fleet health team’s workflow well enough. I talked to them, got feedback, and worked with them on fingerprints, but I was still mostly seeing their work from the outside. I understood the inputs and outputs more than I understood the actual process, or the system itself.

I should have spent more time sitting with them while they investigated failures. I should have asked my manager to loan me to the team for a bit so I could understand how they debugged, what information they needed first, and where the existing process was actually slow.

Instead, I focused on the work that felt safest.

What I Learned

There was also a more personal lesson in it for me. At the time, I was still learning how to operate in an environment where I had no solid footing. There was a lot of information, a lot of system complexity, and no obvious answer. These servers were complicated enough that a single machine was basically its own distributed system. There were hardware blocks, firmware, software, logs from different sources, and a bunch of interactions I did not understand yet, many of which I never figured out.

When it came to logs, I would sometimes feel so overwhelmed that I would shut down and not see the obvious line saying, “Error! I’m literally the root cause!”

When I felt overloaded, I had a tendency to narrow my focus until I found something concrete that I could make progress on. In this case, that became fingerprints.

Fingerprints were real work. They were useful. But they were also safer than going deeper into the uncomfortable question of whether KY itself was solving the right problem.

The project taught me a lot about our systems, and that knowledge was extremely useful later when I was writing tests and assisting with, and eventually driving, future investigations. It also gave me better relationships with both the KY team and the fleet health team, so I did gain a lot from it.

However, it was still a failure. After two months of work, I had nothing enduring to show for it because it was quickly forgotten. Even the log pulling feature was never really adopted after I gave that presentation on it.

KY failed to accomplish its most important goal because I optimized for what I could measure instead of what the customer actually needed. I was so worried about proving that I was doing good work that I stopped asking whether I was doing the right work.

My First Post

Jonathan Westerfield — Sat, 11 Jul 2026 19:25:39 +0000

Big Howdy,

I’m starting a blog.

That sentence feels weirdly dramatic for what this is to me.

The plan is to write about engineering, personal projects, things I’m learning, things I thought I understood but absolutely did not understand, and the occasional reflection after I’ve spent way too long staring at a problem that was probably my fault the whole time.

I’ve wanted to write more for a while, but I kept running into the same problem: a lot of engineering writing feels a little too polished; almost a little fake (I use semi-colons, no it's not AI).

Not bad necessarily. Just polished in a way where the author always seems to have known the correct abstraction, the correct database, the correct deployment strategy, and the correct lesson before the story even started.

I am not that guy.

Most of my actual engineering experience has been much messier.

I'll start with some assumptions, plow ahead thinking I know what I'm doing and then hit a wall realizing that the foundation of my idea was flawed because of... my assumptions. You know what they say about assumptions right? From there I take a step back, untangle the mess and get it right on the 4th try. If it was easy, I probably wouldn't want to do it anyway.

This is all well and good, but the real reason I want to write is because I learn better when I explain things.

If I can write down what happened, what confused me, what I tried, what worked, and what I’d do differently next time, then maybe the lesson actually sticks. And if it helps someone else avoid one of my mistakes, even better. I can’t promise the mistakes will be impressive, but they will be honest.

The posts will probably be pretty short. I’m not trying to write a novel every time I discover a new edge case. That said, I reserve the right to accidentally write too much if I get interested in something. This has happens more than I'd like to admit...

The topics will likely be a mix of:

personal projects
debugging stories
engineering tradeoffs
tools I’m trying
things I learned the hard way
reflections on work, burnout, confidence, and getting better at this without pretending it’s all obvious

I’m not writing this because I think I’m some grand authority on software engineering. I’m very much still learning; sometimes even successfully. Sometimes through doing something dumb and then having to deal with the consequences (honestly the most impactful learning).

But I do think there’s value in writing from that place.

Not the “here are the ten rules for building perfect software” place.

More like the “here’s what I was trying to do, here’s where I got confused, here’s what finally clicked, and here’s the part I still don’t fully understand” place.

That feels more useful to me. At least it feels more real.

So that’s what this blog is going to be. Small posts about engineering and projects, written as honestly as I can manage, without trying to sound smarter than I am, because I'm not that smart :)

We’ll see how it goes.