DEV Community: Chris Price

A spooky tale of overprovisioning with Amazon DynamoDB and Redis

Chris Price — Wed, 08 May 2024 21:50:59 +0000

(NOTE: This was originally published in October 2022 🙂👻)

Halloween is around the corner. Buckle up for a spooky engineering ghost story.

A few years ago, I worked as a software engineer at a large company building a video streaming service. Our first customer was a major professional sports league who would be using our service to broadcast a livestream of their games once a week to millions of viewers; an opportunity that was both exciting and terrifying!

‍When we signed on to the project, our service didn’t actually exist yet. But the league’s broadcast schedule certainly did. 🙂 The launch date was rock solid, and the service had to be able to handle all traffic being sent to us.

‍Is this where the scary part of the story begins? Nope! We had a fantastic engineering team and an architecture design we believed in. The schedule was tight, but we were confident we’d be able to hit our launch date. We put our heads down and got to work.

‍A few weeks before the first broadcast, we were feeling pretty good. The service was built, sans some finishing touches. The team was in the home stretch of load testing to make sure the service would hold up to the traffic at game time, and everything was business as usual.

‍But then…

‍We got our first realistic sample data set from our customer, and we integrated it into our load tests. It did not go smoothly. Based on our budget and our estimates for how much data we would need to store, we had configured a maximum read and write capacity for DynamoDB. But during the load test, we found that we were dramatically exceeding that capacity and running into DynamoDB throttles. Our service failed. Hard.

Be afraid. Be very afraid.

Uh oh. It’s only a few weeks until our first broadcast, and we have a major problem. In our architecture design, there were data we needed to store for each individual viewer watching the broadcast to keep track of where they were in the stream. We had decided to store this data in DynamoDB. After investigating the traffic that the broadcaster was sending us, we discovered the size of the payload for each viewer might be up to 10x larger than our estimates. This required 10x the IOPs on DynamoDB—and 10x the costs!

‍Our workload was very write-heavy. Some napkin math based on the observed 10x increase in data made it clear that storing it in Dynamo would put us far over budget. These data were ephemeral, so we decided that we could move them out of DynamoDB and into a cache server. We did some quick research on our options and decided to move forward with a managed Redis solution.

‍Managed Redis services have some nice benefits in that you aren’t explicitly responsible for provisioning and operating the individual nodes in your cache cluster. But, you *are* explicitly responsible for determining how many nodes you need in your cache cluster, and how big they need to be.

‍The next step was to write code to simulate the load that we would put on the Redis cluster, and run it... over and over again. We tested different sizes of nodes. We tested different cluster sizes. We tested different replication configurations. We tested. A lot.

‍All this writing of synthetic load tests to size a caching cluster was not work that we had accounted for in our engineering plans. Experimenting with different sizes (and types) of cache nodes, monitoring them to ensure they weren’t overloaded during the test runs… These tasks were expensive and time consuming—and largely ancillary to the actual business logic of the service we were trying to build. None of them were especially unique to us. But we still had to allocate precious engineering resources to them.

‍After a week, we had nailed down the sizing and configuration for our cluster, still racing against the clock. After another week, we had completed the work to migrate that part of our code off of Dynamo onto the Redis cluster.

‍And the service was up and running again.

It’s alive! It’s aliiive!

We did it! The first broadcast went smoothly. As with any major software project, after observing it in action in the real world, we learned some lessons and found some things to improve, but the viewers had a good viewing experience. We rolled out some of those improvements during the subsequent weeks, and before we knew it, the season was well underway. Victory!

Until…

‍About a month into the season, we got our AWS bill. To say that it caused us a fright would be an understatement. The bill was… HUGE! What the heck happened?!

‍## It’s coming from inside the house!

Because of our architecture, we knew that the biggest chunk of our bill was going to come from DynamoDB. But we had done a reasonable job of estimating that cost based on our DDB capacity limits. So why was the AWS bill so high?

‍It turns out that the culprit was our Redis clusters. In retrospect, it was predictable, but we had been so busy just trying to make sure that things were operational in time to meet our deadlines, we hadn’t had time to do the math.

‍To meet the demands of our peak traffic during the games, we had been forced to create clusters with 90 nodes in them—in every region that we were broadcasting from. Plus, we needed each node to have enough RAM to store all the data we were pumping into them, which required very large instance types.

‍## Is this place haunted?

Very large instance types that provided the amount of RAM we needed happened to also come with high numbers of vCPUs. Redis is a single-threaded application, meaning it can only take advantage of one vCPU on each node in the cluster, leaving the remaining vCPUs almost 100% idle.

‍So there we were, paying for boatloads of big 16-vCPU instances, and we were guaranteed each one of them would never be using more than about 6% of the CPU it had available. Believe it or not, this wasn’t even the worst of it.

‍The peak traffic we would experience during the sports broadcasts dwarfed the traffic we were handling during any other window of time. So not only were we forced to pay for horsepower that we weren’t even fully utilizing during the games, but we were paying for these Redis clusters 24 hours a day, seven days a week, even though they were effectively at 0% utilization outside of the 3-hour window each week when we were broadcasting the sporting events.

‍And then the season ended and we had no more sports broadcasts for 6 months. So now those clusters were sitting at approximately 0% utilization 24-7.

‍Okay, fine. Problem identified. All we had to do was fix it and get our cloud bill under control!

‍## A horde of zombie… engineers!

Well, it turns out fixing our spend on our Redis clusters was much easier said than done. The managed Redis service didn’t have any easy, safe way to scale the clusters up and down. And because Redis clients handle key sharding on the client side, they have to be aware of the list of available servers at any given time, meaning that scaling the cluster in or out carries a high risk of impacting cache hit rate during the transition, and thus would need to be managed very carefully.

‍These were solvable problems. Throw enough engineers at something, and anything is possible, right? They could update all of the code so that it writes to two different clusters during a scaling event and have reads fail over from the new cluster to the old one for cache misses during the transition. Then, they could scale down by adding a second, smaller Redis cluster alongside the giant one needed for peak traffic. They could definitely handle the work of meticulously monitoring the behavior of the new code while the new cluster was brought online, and they could decide when it’s safe to begin the teardown of the old cluster. Oh, and they can kick that off and meticulously monitor it to make sure that goes smoothly.

‍So sure, our team was capable of doing that twice a week: once when we needed to scale up in preparation for the sports broadcast, and again when we needed to scale down to save costs after the event.

‍But that would be a ton of work. Now we were forced to do some math on how much we were paying those engineers vs. how much we were paying for the overprovisioned Redis clusters.

‍And then there’s the opportunity cost: none of this cluster scaling nonsense had any unique business value for us, and we had a limited number of engineers available to work on delivering features actually unique to our business and provide actual customer-facing value to our users.

‍I bet you can guess where we landed. Yep. We never reached a point where we felt like we could justify the engineering cost it would take to try to solve this problem when there were so many more valuable customer projects our engineers could be doing—projects which would actually move the business forward and win us new customers.

‍So we just kept paying. For something we weren’t using.

‍At a certain point, if our business was struggling, we might have been forced to allocate the engineering resources to solving this problem in order to reduce our spending and balance the budget. But this would have been a sign that we were in trouble.

‍And I don’t know how you feel about the cloud services your team spends money on, but I consider it pretty scary that a cloud service can make it so complicated for you to get a fair bill—a bill where you are paying a fair amount for what you are actually using, and not paying a ton of money for resources that are sitting idle—that you will only be able to make time for it if you’ve gotten into a desperate situation.

‍It’s a great business model for the cloud service provider. Not a great business model for the customer.

‍It doesn’t have to be this way.

‍## Momento Cache: All treat, no tricks!

The horrific tale you’ve just read was a large part of the inspiration for us to build Momento’s serverless caching product. One of the best things about serverless cloud services is the fair pricing model: pay for what you use and nothing more. Why should we settle for less with caching?

‍With Momento, you get a dead-simple pricing policy based strictly on how many bytes you send to and receive from your cache. We don’t think you should have to pay more if those bytes are all transferred within a 3-hour window or are evenly distributed over the course of a week or a month. As far as we’re concerned, you should be able to read and write your cache when you need it. That’s it. Plain and simple.

‍Of course, serverless doesn’t stop there. We manage all of the tricky stuff on the backend for you. If your traffic increases and your cache needs more capacity, that’s on us. If your traffic decreases, you shouldn’t have to pay the same amount of money for your low-traffic window as you did for your high-traffic window. And you most certainly shouldn’t have to pay for 15 idle CPU cores on a bunch of nodes in a caching cluster just because you needed more RAM.

‍So: stop letting cloud services trick you into paying for caching capacity that you aren’t using, and see what a treat it is to work with Momento today! You can create a cache—for free—in less than five minutes. If it takes more than five minutes, let us know, and we’ll send you some Halloween candy.

‍Visit our getting started guide to try it out, and check out our pricing page to see how we make sure you get what you pay for.

‍Happy Halloween! 👻

Moving your bugs forward in time

Chris Price — Wed, 08 May 2024 21:14:50 +0000

Over the course of my years as a software engineer, I’ve slowly become more curmudgeonly deliberate about how to structure a codebase, and how to gauge its relative success.

In my early days I was myopically focused on what the code can do today. It was all about speed, and cranking out code as fast as possible. Tests were a nice-to-have. “Works on my machine” was a reasonable acceptance criteria. And I’m not sure I even knew the definition of the word “maintainable”.

Those times were great fun. And I considered myself to be a pretty 1337 coder.

Then I watched several codebases I’d worked on grind to an eventual halt because no one could understand them, or they were too hard to extend or debug, or they were so fragile that people were afraid to change anything about them lest they introduce a crazy bug that would explode after being deployed to production.

Old man yells at cloud

Now, I’m older and I’ve worked with a lot of different engineers on a lot of different codebases. Now, I have a very different opinion. Now, “maintainability” is one of the most important words in my vocabulary. Now, I care so much more than I did before about what the code will be able to do tomorrow. And perhaps most importantly, I care so much more about what $nextEngineer will be able to make this code do tomorrow than about what I myself might be able to do.

These are the things that allow your software to survive and thrive beyond the early days. The things that ensure that your business will be able to continue to grow and evolve at the same pace a year from now that it can today, that it won’t get bogged down by an un-maintainable, un-extensible code foundation that drags your engineering team’s velocity down towards zero.

The skills, experience and foresight that are required to ensure a maintainable codebase, to be a force multiplier that ensures that a breadth of current and future engineers will be able to achieve sustainable, high velocity working on the code—these are the traits that are at the top of my list when evaluating engineering candidates nowadays. It’s not how many lines of code you can produce nor how quickly—it’s how well those lines of code will hold up as the foundation for your business that grows and evolves over time.

Boiling it down

Occasionally I reflect back on this mindset transition and try to distill my thoughts on maintainability down into something concrete that I can try to communicate to other engineers. And if I had to choose one overarching theme—that I could boil down to a single sentence—that best captures the spirit of what I believe about maintainability these days, it would be this:

Structure your code so that you will catch your bugs at compile time, rather than at run time.

Move your bugs forward in time. There is no single thing that you can do that will have a more sustained impact on the medium-to-long-term velocity of your team than this.

‍For the rest of this post I’ll list off some more tactical examples of things that you can do towards this goal. Savvy readers will note that these are not novel ideas of my own, and in fact a lot of the things on this list are popular core features in modern languages such as Kotlin, Rust, and Clojure. Kotlin, in particular, has done an amazing job of emphasizing these best practices while still being an extremely practical and approachable language.

‍So, credit where it is due: the brilliant language designers of these and other languages deserve all of the kudos for bringing these ideas to the foreground of the software engineering zeitgeist. Today, I’m just here to sing their praises. 🙂

(Side note: spending some time writing code in a variety of new languages is a really amazing way to broaden your horizons and challenge your beliefs about software engineering best practices. You’ll find that it’s often much easier than you think to apply lessons learned from a foundational feature of one language to another language that doesn’t explicitly provide or emphasize that feature. I haven’t written Clojure in several years now, but I firmly believe that the time I spent writing in that language did more to improve my skills as a software engineer than anything else I’ve ever done.)

‍And now, without further ado, let’s get into it.

5 patterns and language features to help catch bugs earlier

1. Static types

This one can be a tough pill to swallow for folks who love Python, Ruby, Clojure, and other dynamically typed languages. And I might never convince some of you of this point. But this is a thing that has burned me enough times over the years that I don’t think I’ll ever change my mind on it.

‍Part of the allure of dynamically typed languages is that if you don’t have to spend time on all of the ceremony of defining types and declaring them on all of your method signatures, you can code more quickly and spend your time thinking about the business logic instead of the object model. And you can write more flexible, re-usable functions that operate on data rather than operating on types.

‍There’s some truth to those arguments, especially in the early prototyping phase of a project. But what I’ve repeatedly seen is that once a codebase in a dynamically typed language grows beyond a certain size, it becomes harder and harder to reason about it and maintain it. Over and over again I’ve seen cases where a well-meaning developer working on one part of the code passes the wrong object type to a function in another part of the code that they weren’t the original author of, and then when that function call occurs, the app crashes.

‍The worst part of this is that that error happens at run time. If you’ve already deployed the code to production without catching this bug, you may have a customer-facing outage on your hands, and now you have to go through a fire drill to rollback the change or push a hotfix. Depending on how bad it was, this may cost you customers. Even in the best case scenario it’s stressful—and it has a high opportunity cost, as you have to pull some of your engineering team off to fight the fire.

‍Some argue that if such a bug makes it through to production, that is a sign that you didn’t add enough test coverage to ensure that the function would only be called with the correct argument types. My response to that is: yes, if you are diligent enough about test coverage, and you don’t make any mistakes in the test code itself, you might be able to avoid shipping most bugs of this classification. But testing is an art form in and of itself, and every one of your engineers must achieve a certain level of proficiency at it in order to clear this bar. And even if they do, we all still make mistakes from time to time.

A compiled language with a static type system guarantees that you will avoid shipping this type of bug to production.

No ifs, ands, or buts about it. And it does not rely on the varying experience levels of your engineers - if they write some code that has a bug like this in it, it won’t compile, and it will never even make it into a PR. No matter how good or bad the test coverage is. Let’s offload this category of work to compilers instead of putting it on our engineers!

‍I’ve become more and more convinced of this over time, to the point where I won’t even advocate for dynamically typed languages for prototypes anymore. Prototypes very often end up being promoted to products, if for no other reason than the code is already written. But if you’re going to promote a prototype to a product then you really ought to have sufficient test coverage to make sure your product is reliable, and at that point you’ll probably have invested the same amount of engineering effort that you would have put into building the prototype in a statically typed language like Kotlin or Go in the first place.

‍I’m not here to tell people which languages they should love. But if you do find yourself writing production code in a dynamically typed language like Python, Ruby, or JavaScript, I would give serious consideration to opting into the type-checking tools that have become available in those ecosystems. In Python, consider requiring type hints and adding mypy checks to your CI to move your type safety bugs forward in time. For JavaScript, consider incrementally shifting to TypeScript. For Ruby, try out the RBS type annotation system that was added in Ruby 3.0.

2. Null safety

Now we’ll get into some (hopefully) less controversial territory. You’ve probably heard the line about null references being a billion-dollar mistake. And if you’ve worked in a language that doesn’t provide compile-time null safety, you’ve surely encountered your fair share of silly bugs resulting in null pointer exception crashes at run time, or code that is littered with boilerplate null checks at the beginning of every single function call, or both.

‍Thankfully the trend in modern languages is to help us move these bugs forward in time by giving us a way to declare variables that are not allowed to be null. This is a core feature of C#, Kotlin, and TypeScript, among others. In Java, you can use Optional instead of allowing null. So we can let the compilers do this work for us.

‍In general if you find yourself using nullable variables these days, it might be a code smell. See if there is a different way you can structure the code to avoid it, and if not, see if your language of choice has any mechanism for compile-time/build-time null safety tooling.

3. Immutable variables and data structures

This one takes some getting used to and may be hard to believe at first, but consider this: there are precious few places in your code where you actually need any of your variables or types to be mutable.

‍The first time someone told me this was when I was learning Clojure, where it was a matter of necessity because it’s very difficult to even express a mutable object. I found the idea quite implausible. But once I opened my mind to it and got a little practice, I realized that it was true.

‍Immutable variables are an incredibly powerful way to improve code maintainability. Here’s why: when you are an engineer working on a code base that you aren’t entirely familiar with, and you encounter a line of code that defines an immutable variable whose type is an immutable data structure, you know all that you will ever need to know about that variable after reading that one line of code. Because it is guaranteed not to be changed anywhere else in the program.

‍Contrast this with mutable variables and mutable data structures: after reading a line of code where one of these is instantiated, if I want to reason about the state of that variable 100 lines farther down in that same code, there are a ton of things I have to consider:

Were there any statements in between that might have modified it?
Was that object passed by reference to any functions that might have mutated it?
If so, do I need to go examine the source code of all of those functions to make sure I know what the state will be?
Does my program have more than one thread, and if so, do any other threads have a reference to this object that might have allowed them to mutate it while I was working with it?

There is so much hidden complexity that comes along with mutable state. If I have to do the amount of reasoning described above for every line of code that deals with a mutable object, and I can instead write the same program in a way that only uses immutable variables and data structures, the reduction in complexity is astounding. And the corresponding increase in engineering velocity and maintainability is as well.

‍Many languages have a way to define immutable local variables these days (e.g. Kotlin val, TypeScript const). Many also have a way to define immutable data structures (e.g. Kotlin data class, C# record struct). Lean into these where you can.

‍Most engineers I work with are sold on this idea fairly easily, except for when dealing with collections. We are so used to writing loops that build up arrays or maps, it’s really hard to get used to the idea that this can be done without a mutable data structure and without a loop. But it can! Almost all languages these days have some flavor of functional programming tools for operating on collections (map, filter, reduce/fold, etc.). These can take some getting used to but they are well worth the price of admission.

‍The reduce / fold operation in particular can be a bit of a learning curve but it is the key to eliminating the need for mutable collections in your code. It will allow you to re-write code that looks like this:

val pepperNames = listOf("jalapeno", "habanero", "serrano", "poblano")
val pepperNameLengths = mutableMapOf()
for (pepperName in pepperNames) {
    pepperNameLengths[pepperName] = pepperName.length
}
// from here forward we need to be cognizant about the pepperNameLengths map being mutated!‍

without the mutable map:

val pepperNameLengths: Map = pepperNames.fold(mapOf()) { accumulator, pepperName ->
    accumulator + (pepperName to pepperName.length)
}
// no mutable map to worry about here!

4. Persistent collections (aka immutable collections)

When a coworker originally told me that I should be using immutable collections, my instinct was that this was impractical due to performance concerns and memory usage. If I represent a map as an immutable collection, and then somewhere in my code I need to add or modify a key in it, doesn’t that mean copying the entire data structure in order to obtain the version that contains my modification? Isn’t that crazy expensive?

‍Well, it turns out: no. As long as you are using persistent collections.

‍I first encountered this concept in Clojure, and I highly recommend Rich Hickey’s fantastic talk on the topic. The tl;dr is that:

A persistent data structure is guaranteed to be immutable, but provides modifier functions (put, add, remove etc.) that will produce another persistent data structure with the same immutability guarantees.
Under the hood, these data structures are implemented as trees, and when you want to modify a single item, you can do so by creating a new tree that shares almost all of the nodes of the original tree. You only need to copy and replace the small set of nodes in the tree on the path to the item you are modifying. In efficient implementations, this means you almost never need to clone more than about 4 nodes in the tree even if it has millions of nodes. The rest can be shared, which is efficient in terms of both memory usage and performance.

‍Many languages now have “persistent collections” or “immutable collections” libraries (e.g. Java PCollections, C# Immutable Collections, etc.) that do all of the heavy lifting for you. You interact with them just like normal collections, but you get all of the benefits of immutability while still maintaining great performance.

‍This concept is amazingly powerful, especially in concurrent programs. It means that you can pass a reference to a collection around anywhere you like in your program, and many threads can consume it at the same time with no locking or any other concerns about the collection being modified by another thread. You’ll be amazed at how much simpler this can make some of your application code! And at how nice it is to stop needing to worry about lock contention.

5. Algebraic data types and exhaustive pattern matching

These go by many names in different languages. In Kotlin they are called sealed classes. In many languages this may end up being just a special flavor of polymorphism. I think of them as an enumeration of types, where you know at compile time all of the types in the enumeration, but each type in the enumeration can have its own discrete properties, methods, etc.

‍It’s easiest to explain via a specific example. I’ll use an example from the Momento Cache API, since that’s something I’ve been working on a lot lately.

‍When you make a call to the get method on a Momento client object to retrieve a value from your cache, the response may be one of three very different types:

A cache hit, in which case you will also get back the value that was retrieved from the cache
A cache miss, in which case there will be no cache value.
An error, if something went wrong with the request, in which case you might get an error code and an error message.

‍Without algebraic data types, a common way to try to represent this situation in code might be to provide a GetResponse object, with a status enum property that could be used to identify whether the response was a HIT, MISS, or ERROR. The object would also need fields to hold the various data that is relevant for each of those cases: e.g. value, errorCode, errorMessage. Those fields would have to be nullable or optional, because they would only be available conditionally, depending on which type of response we got. Something like this:

enum class GetResponseStatus {
    HIT, MISS, ERROR
}
data class GetResponse(
    val status: GetResponseStatus,
    val value: String?,
    val errorCode: Int?,
    val errorMessage: String?
)

This is not an awful way to define this API, but it has one big drawback: it is the developer’s responsibility to write code that checks all of the conditions correctly, and if there is a bug in the code it will only surface at run time. For example, if you write code that assumes the response was a HIT without checking, and you try to access the value property, you will get a null pointer exception at run time if the response was not actually a HIT. (In the Kotlin code snippet above, because of Kotlin’s null-safety rules, you’d be forced to write some code to deal with the possibility of those values being null, but in other languages that wouldn’t necessarily be the case. The point remains that it is the developer’s responsibility to reason about which of these fields might be null and when.)

Algebraic data types provide a much nicer way to specify this API, without exposing any nullable fields at all. Here’s how this might look using Kotlin’s sealed classes:

sealed interface GetResponse {
    data class Hit(val value: String) : GetResponse
    object Miss : GetResponse
    data class Error(val errorCode: Int, val errorMessage: String) : GetResponse
}

Now we have a discrete class for each of the three cases, and each of those three classes has only the properties that are relevant to it. And they are no longer nullable.

A developer would access the appropriate class via pattern matching. In Kotlin, this is done via the when expression:

val getResponse: GetResponse = cacheClient.get("myCacheKey")
when (getResponse) {
    is GetResponse.Hit -> {
        println("Cache hit! ${getResponse.value}")
    }
    GetResponse.Miss -> { 
        println("Cache miss!")
    }
    is GetResponse.Error -> {
        println("Error! ${getResponse.errorMessage}")
    }
}

This approach is really nice because it removes the burden of knowledge from the developer for questions like “in which cases will value be available?” The value property only exists on the Hit class, so we get compile-time enforcement that it can’t be accessed unless the result was a Hit. We have once again moved our bugs forward in time!

The other great thing about this approach is that, in languages like Kotlin, the pattern matching expression is exhaustive. This means that the compiler is smart enough to know whether you have handled all of the possible cases in your when expression, and fail to compile if you have not. Imagine a scenario where you have several of these when expressions scattered around a large code base, and an engineer is working on a new feature that involves adding an additional type of GetResponse to the sealed class. Without the exhaustive pattern matching, the engineer would be responsible for identifying every place in your code that is interacting with a GetResponse, and making sure that it appropriately handles the new type of response. Otherwise what do we end up with? A bug that isn’t exposed until run time.

‍But with exhaustive pattern matching, once the new type is added, the code won’t compile until we’ve updated all of the places in the code that need to be updated to account for it. Win!

Closing thoughts

The key to building a solid foundation for your software and sustaining high velocity for your engineering team for the life of your product is to make sure your codebase is maintainable. It’s crucially important that future engineers are able to ramp up on the code quickly and safely. Thankfully, trends in modern programming languages are giving us more and more tools to achieve that, and to move entire classes of bugs forward in time from run time to compile time. This also saves us a ton of engineering time that we don’t need to spend writing tests to prove that we haven’t introduced these classes of bugs. (Don’t get me wrong: tests are still very important! But it’s so nice not to have to write tests around the behavior of nullable properties or other such mundane things that aren’t actually related to your business.)

‍The strategies above have proved especially valuable for me in the projects that I’ve worked on in recent years. I hope you’ll find them valuable too!

‍If you have any other favorite approaches for moving bugs forward in time, I would love to hear about them. Send them my way on Twitter (@cprice404) or LinkedIn—or join the Momento Discord and start a discussion!