Apparently Contracts Should Be Honored
When I think of contracts, I think of scary documents that limit my freedom, usually in exchange for something I want. Documents like EULAs, user agreements, divorce papers, and prenups are just a few examples.
However, these contracts also provide a sense of comfort. If the terms serve you, then you take comfort in knowing that the contract is in your favor and the other party is forced to uphold it. Even if the terms are bad, you at least know what you're getting.
The common theme is trust in the contract.
The contract is sacred, and both parties must uphold their end of the deal. Break that agreement, and suddenly the contract is just an ideal on a worthless piece of paper, or a text document file. When this trust is broken, everything falls to pieces. People get mad and things start breaking, emotionally, spiritually, financially, or otherwise.
This holds true for API contracts. The provider agrees to define and uphold a specific interface for other developers to use. This not only tells developers how to actually use the API, but it also guarantees that they don't need to worry about the contract suddenly, magically changing under their feet. Any changes to this contract must involve all parties, providers and users, to a reasonable degree. At the very least, users need to be notified of the change and given a window to adjust.
Unplanned changes bring chaos, so who on Earth would violate this sacred trust?
The Setup
I implemented a CoAP handler in Lua on our servers. It's a simple framework that takes CoAP requests, calls a library on the server, and returns the results. If there are no results, it returns nothing.
The key thing is that each request to the server happens in two stages:
- A POST request to a corresponding endpoint with the information needed to execute the action.
- A GET request with the request ID from the POST to get the result.
For long-running actions, you would poll until the request succeeded. If an action takes too long, there is a default timeout built into the framework to kill the server process, which would be reflected in the GET request.
The issue is that I didn't implement that part correctly.
A Faulty Timeout Mechanism
The timeout mechanism didn't measure time correctly. I incorrectly started the timer at the wrong point in the request flow. The result was that the effective timeout was slightly longer than the default of five seconds. On the server side, the fix was pretty simple: just move the start of the timer to the correct place in the code flow.
But what about the client side?
This framework was unique in that I controlled both the client- and server-side code. I could quickly check for all calls to the CoAP endpoints, make my server-side change, and bundle that with client-side fixes too. I could also easily run tests to make sure nothing broke. Easy, right?
So that's what I did. I made the timer actually time things correctly, tested the changes against our codebase on a known-good server, raised a PR, and pushed the change once it was approved.
The whole thing took me a little over a day. This was back before my team used AI, so I felt pretty good about how speedy the whole affair was.
Why Do I Hear Police Sirens?
One small problem: suddenly our release was blocked.
Before we release code changes to production, the whole org runs tests on our servers to ensure everything works. However, every once in a while, sporadically, a request from my framework would randomly time out.
A typical release cycle can take two whole days, and multiple teams from all over our business unit put their changes into this release. Let's just say that since there are so many chefs in this kitchen, tensions run high and any release blockers cause people to get really testy. This is not a chill release cycle. It's akin to a full-on production issue.
The engineer on call pinged me and let me know about the problem. I wracked my brain for possible causes and easy fixes, but nothing came to mind.
What we did know was that it was related to my commit. For production issues, the first lever to grab is a rollback. In this case, since we knew it was my commit causing this, I quickly pushed a revert to unblock the release. No sense in trying to debug during a production event.
It was quick to stop the bleeding, but I'll admit that pushing that revert still embarrassed me. I was committed to figuring out what went wrong.
To Catch a Killer (Bug)
Now that the release was unblocked and things had calmed down, it was time to get to the bottom of this.
I went back to my code and performed my sanity checks. I jumped on a known-good server and ran my tests again.
Everything passed. That's annoying.
I did code profiling to check the hotspots in my code. I ran requests in a loop to see if any of them returned slightly slower than the others. Nothing worked. The requests still ran normally and returned without timing out. I even grabbed the exact server that failed during the release and ran the tests again. Everything passed...
If the server that presented the problem wasn't able to reproduce the issue, I was out of options. I had hit the limit of my knowledge.
A Break in the Case
Since I ran out of leads, I did what any engineer does: I asked other engineers. I talked to—and at—people about my problem to see if anyone could come up with a reason this could have happened. No one had any clue. They were mostly hardware engineers, so that figures. This wasn't their realm of knowledge.
Then I ran across our principal engineer at lunch and gave him the same spiel. He thought about the problem for a bit and asked me something that hit me like a truck.
"Do you know you changed the API contract?"
I told him, "Surely not, right?" I didn't change how developers interacted with the APIs. I didn't change parameters or return values. The API contract was completely unchanged, right?
Wrong.
He told me that even changing the timeout functionality is technically changing the contract. It's not as obvious as changing the structure of an API, but it's still an unplanned, uncoordinated change.
But that's not a problem, right? I controlled both the client and server code. I was the provider and user, so I should have been able to change it with impunity.
What he said next changed the whole picture.
"This is the problem with unplanned changes. You can't know how they will affect the system. Generally, when you make an API change, you add optional parameters or, preferably, version the API to a V2 for others to use. Any change to the existing API opens you up to a world of pain, so you shouldn't make those changes unless you absolutely have to, even in your case where you control both worlds. In this case, it sounds like maybe the servers got bogged down and slowed down just enough to cross the five-second limit; the NEW, SHORTER limit you set."
My Smoking Gun
It was so obvious. I can't believe I missed it.
When I was doing my investigation, I was calling my framework on known-good servers.
I had never considered using bad servers or intentionally making lots of simultaneous calls to bog down the server. The compute on these servers was tiny, so this is not hard to do. I originally chose the five-second timeout because that was the upper limit on how long a call originally took. I ran numbers to calculate that timeout, but didn't think about that when I changed the timeout mechanism. Nor did I seriously consider just how much I was shortening the effective timeout of a request.
I had three things to do:
- Confirm that bogging down the server could cause the timeout to be breached.
- Fix my API contract by changing the timeout to what it was before.
- See if I needed a longer default timeout than the original timeout in case the server got bogged down. Slow servers are not actually code bugs. These things happen in production, and the code should handle them gracefully.
All three were simple.
First, I wrote a simple script to make 100 calls to the endpoints and see if any of them timed out. I also helped out by running other heavy processes on the server to further burden the resource-constrained CPU. This immediately surfaced and confirmed the issue.
Great.
Second, I made calls to the endpoints on both the new and old code to see how long each request took. The difference was when the start time occurred in the code flow. I could take the difference in those request times and add it to the default timeout value, effectively restoring its old implicit value. That gave me a correct timeout mechanism without randomly timing out requests that used to work.
A quick PR later, and we were back at it. The code ran through the release testing and got pushed into production. No problem.
Penance and Reflection
So obvious, and I somehow missed it.
I had committed the cardinal sin of API contracts and got burned. Changing an API contract doesn't have to mean something as explicit as changing parameters or return values.
I made what I thought was a tiny cleanup change, but the timeout was part of the deal whether I had written it down that way or not. Nothing about the endpoint changed, but on a sufficiently grumpy server, requests that used to succeed could now fail. API contracts are also the assumptions someone can build code around, including the ones you don't know exist yet.
I learned that with a blocked release and a reverted PR. Now, whenever I make changes, I assume existing server behavior is part of the contract.
We add new features. We don't change existing ones.
Top comments (0)