DEV Community: Corey Cleary

Practice System Design interview question walkthrough

Corey Cleary — Tue, 25 Mar 2025 00:00:00 +0000

Originally published at coreycleary.me. This is a cross-post from my content blog. I publish new content every week or two, and you can sign up to my newsletter if you'd like to receive my articles directly to your inbox! I also regularly send cheatsheets and other freebies.

Tech job interviews are notoriously difficult. You never know what questions they are going to ask you - are they going to ask about obscure language trivia, have you build some UI component while doing a screenshare, ask you some algorithm puzzles that have no relevance to the job day-to-day, or ask you to design an architecture for some theoretical problem?

There are a million things they could ask you, and unfortunately you don't have much control over that. You can look on Glassdoor to see what the company tends to ask, but even then it's a gamble.

Personally I'm not a fan of how most technical interviews are done. Trivia questions aren't relevant and even experienced candidates often forget simple things all the time. Screensharing/livecoding some feature is more realistic, but time-consuming and nerve wracking. And Leetcode-style/algorithm puzzle questions you will almost never encounter in the real programming world, not to mention those can require weeks or months of studying - not something I personally have time for.

The one interview question type that I think is actually relevant are System Design questions. Relevant, because discussing with your future team members different approaches to building out a new feature or system is what you do everyday on the job. Sometimes these interview questions are very formal, where you have a list of requirements, and other times they're more informal, like a back and forth discussion. Something like "we have this current problem we are working through solving, how would you go about it?", and you talk through different options.

Although those are both different styles - and sometimes you're not designing a whole system but instead a slice of it, like a feature - for the sake of conciseness I am calling both of them "System Design".

With the job market being as tough as it is right now, having peace of mind that you have gotten practice in going over these types of questions will go a long way in reducing your pre-interview anxiety.

And if you don't have a ton of experience designing systems/doing architecture work - maybe you have a couple of years experience and now you're looking to move to a more Senior-level role - learning how to go about designing a feature or system is extremely useful.

Maybe you have helped out Architects or Senior developers on your team before, but didn't quite understand why certain decisions were made, or how they even came up with their approach.

No matter what scenario you find yourself in - the key to both is being exposed to lots of patterns and company architectures. Once you have seen enough, the pattern recognition starts to kick in and you can more easily identify which patterns to apply to which problems. While it's always best to gain this experience in the "real world", there is only so much you can learn at any given company, so being exposed to these patterns is key.

With enough exposure to different patterns and architectures, you can ace the System Design interview and/or learn enough to move from junior-mid to senior.

This is a continuous learning process, but a continuous learning process starts with step 1, which is why I am going to cover a system design interview question I received in the "real world" recently when interviewing.

This post will accomplish two main things:

get practice with an interview question (if you are actively applying/interviewing)
learn a few patterns/approaches you can apply in your work (if you are just trying to level up design/architecture knowledge)

The interview question

I picked this particular question because in addition to it being one I was recently asked in a real interview, it also:

is a common question asked in interviews
is a problem that allows us to go over a few different intersting achitectural topics

The question asked was "how would you go about designing a food delivery service like Grubhub/DoorDash?"

When I mentioned the formal vs. informal distinction before, this one definitely fits in the informal category. Which has some challenges, because we don't have a clearly defined scope up front. But for purposes of this post it's actually helpful because it presents an opportunity to go over how to define and narrow scope, which we will do now.

Step 1: narrowing scope and clarifications

The way I broke down this problem is first identifying the core pieces. You get live updates for where your driver is, so there is a real time component needed here. A restaurant needs to manage their "inventory". The service needs to efficiently route drivers.

Think about how you use an app like Grubhub, and think about how restaurants and drivers might use it. This process will quickly help you realize there are lots of pieces involved:

users must be able to register
users must be able to set favorites/bookmark restaurants
users must be able to setup payment options
users must be able to search for restaurants
the service must determine which drivers to allocate to a food pickup and delivery
the service must serve data to the front-end containing restaurant details
the service must prevent orders from being made before or after a restaurant's opening/closing hours
the service must allow the user to track their order status and delivery in real-time
the service must send text or push notifications when food has been picked up, when it's on the way, and when it has been delivered
restaurants must be able to specify menu items and prices
restaurants must be able to mark items as "sold out" (either manually or using a quantity available) and the service must not allow a user to order those
the service must provide directions for the driver - to the restaurant and to the customer

... and there are probably many more components/features we could think of.

But think of this a "whittling down" from open and abstract, to something clearer and clearer. It's not necessary to go so deep that you are essentially doing requirements gathering, but enough you can identify the high-level building blocks.

For purposes of a 30 minute (maybe 1 hour) long conversation though, going over all the above would take way too much time. A lot of these components are important but kind of standard to all apps. Things like registration, user settings, billing/payments, etc. So we can make an assumption those aren't really important to go over for purposes of an interview question.

But these elements do seem important, because they are relatively unique to a food delivery service when compared to some other app/service:

~~users must be able to register~~ (Common to most apps)
~~users must be able to set favorites/bookmark restaurants~~ (Common to most apps)
~~users must be able to setup payment options~~ (Common to most apps)
users must be able to search for restaurants
the service must determine which drivers to allocate to a food pickup and delivery
~~the service must serve data to the front-end containing restaurant details~~ (Common to most apps)
the service must prevent orders from being made before or after a restaurant's opening/closing hours
the service must allow the user to track their order status and delivery in real-time
the service must send text or push notifications when food has been picked up, when it's on the way, and when it has been delivered (Likely less important, given we will discuss real-time updates)
restaurants must be able to specify menu items and prices (Likely less important)
restaurants must be able to mark items as "sold out" (either manually or using a quantity available) and the service must not allow a user to order those (Likely less important, and more of an "implementation detail")
the service must provide directions for the driver - to the restaurant and to the customer (Important but likely too big for current scope)

So in my interview I went over this list (all items) and then asked if the narrowed down scope (everything in bold) was a correct assumption, and they confirmed. Not only is this useful for limiting the discussion, but it also shows the interviewer that you have enough experience to even think about all the pieces potentially involved, even though you won't be discussing all of them.

Coming up with a design

Now we start on the design work. In my interview I didn't go through each component in a specific order, unless they were related to each other.

For purposes of this post, the real-time update component is an interesting one to think through so let's start with that.

Feature - the service must allow the user to track their order status and delivery in real-time

Either WebSockets or SSE (Server-Sent Events) are viable options for handling the real-time aspect of updating the user on their order and delivery status.

A discussion comparing and contrasting these two technologies/patterns is an entire post by itself, but generally speaking here is a comparison:

WebSockets are bi-directional, meaning both client and server can communicate with each other. Whereas SSE is uni-directional - only the server can send data to the client.
WebSockets are low latency, because a connection stays open and there is no additional overhead required to re-open connections. SSE connections do automatically reconnect, however.
WebSockets can maintain millions of connections, but they need stateful scaling. The server needs to keep track of the connection state of each client connected to it. SSE on the other hand, since it uses HTTP which is stateless, can scale more easily.
For load balancing requests, WebSockets require a proxy that supports WebSockets, whereas SSE's since they just use HTTP, and load balancer will work.
Some ISP's will block WebSocket connections, so some users may not be able to connect depending on their ISP's rules.

With that in mind, you could use:

WebSockets:
- for the bi-directional capability of a driver being able to send updates to the server, and the server pushing those updates to the user
SSE:
- you want to account for users having bad network connections where it drops frequently (SSE's will auto-reconnect)
- the server is the only one that needs to push updates (updates to the user like "Preparing food", "Out for delivery", "Delivered")

In an interview situation, if you have two potential options or solutions, you don't have to necessarily propose just one as the "right" option. For example, you can mention the above two options and discuss tradeoffs between the two. This is actually a good thing to do because most of the job day-to-day is thinking through and discussing tradeoffs.

In our case, you could probably argue for either option. The bi-directionality of WebSockets is likely to be useful, but that comes with more infrastructural and configuarion costs. On the other hand, if you don't need that SSE's are more "lightweight" and you have less to think about when it comes to scaling and handling dropped connections.

One last thing - in theory you could also use short polling, which is where the client makes HTTP requests for status updates every few seconds. The positives of this approach are:

the implementation is only on the app/client code, instead of also needing server code to support this
no connection management to keep track of

However, this approach is inefficient for a service and app that works at scale, because now you will have many more requests made to your REST API servers handling these update requests (let's say one request every 5 seconds, which is 12 req/min, times potentially hundreds of thousand of clients making these requests). Short polling is also not real-time, which may not matter in the food delivery case because a user doesn't need precisely real-time. But because of this inefficiency of putting more load on the servers, WebSockets or SSE are a better option.

Interviewer follow-up question

Let's say that you did choose WebSockets. How would you handle reconnecting disconnected clients - OR - if their connection is completely blocked, as by their ISP?

I got this question in my interview, and my answer was that while theire WebSocket connection reconnects you could use a fallback polling until the WebSocket becomes reconnected. The tradeoff there is that you then sacrifice some REST API server efficiency (as we went over in the short polling section just above), but that you then have a good fallback experience for the user, so that they can still receive updates.

Feature - users must be able to search for restaurants

A user is going to want to be able to search by restaurant name, restaurant type, food item, by location, etc.

To support this feature, there are many options out there but Elasticsearch or PostgreSQL "Full Text Search" are good options. Similar to WebSockets vs. SSE, there are tradeoffs with both approaches and neither is the "correct" choice. But there are some clear use cases for when you would want to use one over the other.

Both Postgres and Elasticsearch are very commonly used technologies, so with either choice you won't have to worry about finding developers with lack of knowledge in either or a poorly maintained solution (no development, no bugfixes, etc). While Elasticsearch is a dedicated search solution, Postgres also has search functionality built-in (despite being a dedicated database solution).

Postgres Full Text Search (referred to as "FTS" going forward) gives us some of the features of a search engine, and which we can make use of in designing the restaurant and search feature of the food delivery service. The topic of Postgres FTS could be covered in a post by itself but at a high level, here are the benefits this feature provides:

Basic search by terms
Ranking support
You don't need an additional technology you have to add to your infrastructure and then maintain if you are already using Postgres

The drawbacks are that:

It doesn't support typo correction
Fuzzy searching is not as robust as it is with Elasticsearch
It doesn't support more robust searches (like by location, ratings, delivery time, etc.)

Elasticsearch, on the other hand:

Requires additional infrastructure to be added to your stack and maintained
Supports handling typos
Supports fuzzy searches/matches
Supports more advanced searching (like by location, ratings, delivery time, etc.)

As with the real-time feature, we again come to tradeoffs. In this case you can propose either. I lean towards Elasticsearch because a user is going to want to search by restaurant name, food, location and get those results back sorted/ranked by restaurant rating, delivery time, etc. Elasticsearch just supports this better than Postgres FTS, which you can do a lot with but is still more basic.

NOTE: Look for a post on Postgres FTS coming in the future!

Interviewer follow-up question

Is there a way you could take a hybrid approach and use both?

One approach would be to start with Postgres FTS and if you run into scaling issues or find there are features you need to add that it doesn't support very well, you can migrate to Elasticsearch. Another approach would be to use FTS for some search categories that are more basic (like a restaurant manager searching for their own menu items to update via their management dashboard), and Elasticsearch for other that are more complex (like user searches).

How would Elasticsearch ingest restaurant data?

The restaurant details - location, hours, menu items, prices, etc - are likely going to be stored in the database via some management/dashboard UI the restaurant uses to set these details. Everytime there is an update, Elasticsearch could then ingest the new data and store it in a format that is designed for searchability.

This conveniently skips over how thinking through how Elasticsearch would know there is an update - for example, would this be accomplished via a cron job? Is there something like a webhook or emitted event from the database that lets Elasticsearch know to ingest the new data? Because this post is already getting kind of long I am going to skip over the details on that, but this is a good practice to answer in more detail on your own if you are preparing for an interview.

Feature - the service must determine which drivers to allocate to a food pickup and delivery

In order to determine which drivers to allocate to pickup and delivery a customer order, we need to know three things:

pickup address (restaurant location)
driver's current location
delivery address

And then the selection algorithm would need to essentially triangulate between those three.

Because this is a System Design question and not a coding/algorithm question, you probably won't be asked what that algorithm would be. But figuring out how to store those pieces of information is relevant to designing a system and, thus, worth talking through.

Fortunately, the pickup and delivery addresses are static, and input by the restaurant and by the custom. So we only need to store those in a restaurants table and a customers table (or have some foreign key relationship to an addresses table).

But figuring out and storing the driver's location is more challenging, because of the dynamic nature of their coordinates. Because they are pretty much always driving, their location is going to be always changing.

A potential solution to this is Geohashing. Using this technique, you spatially index coordinates of drivers by encoding the latitude and longitude coordinates into a hash.

For example:

Location	Latitude	Longitude	Geohash (Precision 5)
Restaurant 1	40.7128	-74.0060	dr5r7
Driver 1	40.7139	-74.0075	dr5r7
Driver 2	40.7291	-73.9995	dr5r8

In this example, because the Geohash of Driver 1 and Restaurant 1 match, we know that Driver 1 is closer to the restaurant than Driver 2.

I mentioned above that we already have the restaurant's and customer's addresses. We would need to take those, get the latitude/longitude coordinates (likely using some lookup service), and then store as a Geohash. This would likely happen asynchronously using some queueing system to create a job for this work, since it doesn't need to happen immediately. With each of the three data points we need as a Geohash, we can then triangulate.

We can either store these hashes in a database, or we can use Redis' Geohashing feature to accomplish the storage piece of this. Redis handles the hashing part of it for you, and provides a nice GEOSEARCH feature that lets you search by a radius. Too much to go into in this post, but you can checkout the link above to learn more about it.

You may be wondering - because the driver's location is dynamic and always updating, how do we handle updating it? The Geohash would therefore also always be changing.

There are a couple approaches to this we could take. One would be for the driver's version of the delivery app to update their location every 30 seconds or so (or whatever timeframe would not overwhelm or Geohash storage solution). Another would be for the driver's app to track when it's gone more than X miles since the last Geohash update, or to even detect when it's sitting idle and to not send updates then. Ultimately, this logic best lives on the client.

And the client needs to make sure it's not overwhelming the server with updates. Fairly easily accomplishable.

This is another potential use case for WebSockets - in this case the driver app would open a WebSocket connection to our delivery service and pipe updates. This would be closer to "real-time" than the suggestions above, but then again, we get back to tradeoffs. How precisely real-time does this need to be, and is the increased load handling of hundreds of thousands (or however many drivers are active) worth it?

Interviewer follow-up question

What if a driver is not active or offline? How can we be sure they aren't selected for a pickup/delivery?

Rather than go over this, I'm going to leave this as a good practice exercise.

Feature - the service must prevent orders from being made before or after a restaurant's opening/closing hours

While in the context of a System Design interview, this leans more towards the "implementation detail"/programming side of things, I think in the context of a food delivery service it's a fairly important part of the system. In my interview I briefly discussed this with the interviewers, so I'm going to go over that here.

An approach you could take here is to store each restaurant’s operating hours in a table, and then for each order validate it against the stored hours before confirming the order. Because this table would be hit often, it's best to index it, most likely on the hours column, in order to not slow down performance.

The REST endoint for placing an order would:

get the opening/closing hours of the restaurant from the table
check if the time the order was placed is greater than the closing hours (or less than the opening hour)
allow if not, reject if so

On the front-end you would do this same check, and ideally you would already have the restaurant's operating hours fetched so you don't need to make an additional server-side call. But you still need this validation on the back-end too, as someone could be hitting the endpoint directly rather than through the front-end.

And alternative to checking the database would be to store the restaurant hours in a cache, that way you reduce the load on your database (which is going to be serving up lots of other query responses).

Interviewer follow-up question

What if you had multiple orders placed around the same time for an item that is out of stock, or had one item remaining? How would you handle that

I'm going to leave this as a practice question you can work through yourself!

Remaining things we didn't cover

Other things you could be asked about are:

scaling/load balancing
infrastructure setup
security
caching
- what are things you would cache?
- what are things you wouldn't?

These things tend to come up in interviews but are fairly standard across all architectures, so when you feel comfortable discussing approaches for handling for one particular service/company, you should feel comfortable applying it to others.

Summary

If you have a technical interview coming up, use the scope defined in this post to practice how you would go about designing it, using the patterns and approaches I discussed, and/or come up with your own design.

Instead, if you are looking to learn more about architecture, use the patterns and design discussed to apply to the next project you are working on or as a springboard for learning more about any of the given topics!

There isn’t much content out there that dives into advanced topics for Node.js developers or for front-end developers transitioning to back-end development, which is popular given that front-end devs already know JavaScript. Most resources focus on beginner-level material or generic concepts but leave a gap for those looking to advance their knowledge and/or career.

This post is part of my effort to fill that gap. While it’s not yet part of a formal series, I plan to write more articles like this one in the future, because architecture concepts are part of that "advanced" knowledge.

If you’ve ever felt the pain of not having enough material to guide you through these advanced topics, you’re not alone. My goal is to provide actionable insights and examples that you can apply directly to your work or interview prep.

Stay tuned for more posts like this, and if you’re interested in learning about these topics as I publish them, here's that signup link again!

Testing REST endpoints with fake HTTP statuses

Corey Cleary — Tue, 18 Jun 2024 00:00:00 +0000

Sometimes when developing a service that calls an external REST API or testing calling an endpoint from a front-end, you'll need to test what happens when that API returns different HTTP status codes. You can do this using tests, and mock/stub a 404, 500, etc. But often it's very helpful to see what happens when a specific response status is returned.

Or imagine your QA team needs to manually test what happens when a 500 is returned. For example, they need to confirm an error page shows if the app receives an error response. In that scenario you can't (purely) rely on stubbed automated tests.

Anytime you have to stub responses, that setup can be tricky and can slow you down. You could go an alternative route and setup your own server that returns 404, 401, 500, etc. And then you could manually swap out the endpoints that get called to point to your "stubbed" service, or using some complicated proxy setup to forward the requests. But that also takes time, adds complexity, and will slow you down.

Rather than having to spin up your own server or only handle this via tests, you can use free 3rd party services that do this for you! This will make seeing how your code handles different HTTP response statuses, with scenarios like the below much easier:

Node (or any language) REST API -> external endpoint
UI/Front-end -> REST API

Using a 3rd party service

One of the best tools I've found for testing out different responses from a service is httpstat.us.

This service will let you call pretty much any standard HTTP response status as an endpoint, and get that response code returned. For example, calling https://httpstat.us/503 (with GET/POST/etc) will return a 503 Service Unavailable response.

I've used this for testing that a generic "hit an error" page/component gets triggered when the app hits any 500-level errors in an HTTP interceptor. This is an especially good use case for QA teams, because if they are testing this scenario, and you don't want to bring down your server in a dev/UAT environment they can just swap out your server endpoint for something like https://httpstat.us/500 and test that way (ideally the URL is already defined as an environment variable to avoid a code change).

You can also simulate slowness/latency by adding a sleep query param like https://httpstat.us/503?sleep=5000. This is very useful for testing how the UI behaves when having to wait for a slow endpoint. For example, you test using this delay and realize that one component is blocking loading of data for several other components until the first (slow endpoint) response is received.

Lastly, I also use this when I'm working on a frontend feature and the backend endpoint isn't ready yet, and I don't need a response body, just the status.

For POST/PATCH/PUT requests you can send data in the request body, but it will just be ignored. Be careful with this, because although they claim they don't log any data it's hard to verify that and know for sure. So just don't send any sensitive customer data or anything like that.

Using this service on the backend

I went over two use cases on the frontend above, now will go over a use case for using https://httpstat.us/ on the backend.

Often your backend service will need to communicate with external third party services via REST, and this is where this mocked service can come in handy. Because while you will likely have some integration or end-to-end tests for your backend service that test how it handles receiving different HTTP responses, you will also likely want to "smoke" test this manually.

So if I'm calling the Google Maps API as part of my backend service, I could swap the URL out for https://httpstat.us/ with whatever HTTP response code I want to test and confirm my code behaves as expected, does error handling properly, etc.

Similarly, let's say there is another team within the company that is working on an endpoint that is not quite ready yet, but all I need is a 200 or 201 response code and not the resposne body. Instead of holding up my development work waiting for them, I can swap it out with the mocked service.

Summary

There are probably several tools that do what httpstat.us does, but I've found it works well and has most if not all HTTP response codes covered.

To summarize, I use it for the following scenarios:

For QA testing when they need to test something that is hard to simulate (like the server being down in a dev or UAT environment)
When I'm working on a FE feature and the BE endpoint isn't ready yet, or when the BE service relies on another service that isn't ready yet
For testing how the FE (or BE) handles a slow endpoint
When I'm working on a BE feature and want to manually test how it handles different response statuses, as opposed to it being handled through integration tests
When I want to test how the BE handles different response statuses from a third party service

Next time you run into one of the scenarios above, consider using a tool like httpstat.us to assist with your development work and/or testing!

Love JavaScript but still getting tripped up by unit testing, architecture, etc? I publish articles on JavaScript and Node every 1-2 weeks, so if you want to receive all new articles directly to your inbox, here's that link again to subscribe to my newsletter!

Learning a codebase by reading tests

Corey Cleary — Mon, 15 Apr 2024 00:00:00 +0000

Getting up-to-speed on a new codebase - whether you're diving into an open source library or you just joined a new company and you're learning their code - is always a bit daunting... Where do you start? What file do you open first? What is the code even doing?

There are lots of ways to go about this, but one of the best ways I've found and how I tackle learning any new project is by reading the tests that accompany the code.

Advantages of learning via this approach

Documentation is the usual place most developers will start when learning a codebase. This is definitely not a bad place to start, and it can help you to get a high-level understanding. But there are a few advantages of reading tests as "documentation".

Often documentation will become out-of-date, but unit tests generally won't. Since tests are (hopefully) run as part a CI build and deployment, if a function changes and breaks the test the CI build will fail. Which means tests have to be more up-to-date than documentation, which is static.

Another major advantage is that tests are about inputs and outputs, usually more written from an "outcome" perspective. Most modern test frameworks have you write a test with an expected input(s), pass those to whatever it is that's being exercised/tested, and confirm an expected output. Reading through tests like this can help you understand how data is passed around, gets modified, passed to other functions, etc.

And 90% of the time when you're trying to understand a codebase, you're trying to understand what a given function expects and what it either returns or what action(s) it takes. Tests will help you understand that.

Similar to how "Test-driven development" is about writing the test to write the code, this is about reading the test to understand the code.

Starting with higher-level tests

Usually the way I will do this is start with reading the end-to-end and integration tests (if they exist, ideally both do), since they are written to test things at a more "high level" and less granular than unit tests. These tests will exercise a larger amount of code paths, and are usually written in more "business requirements" or "feature requirements" language. This helps you understand how the module or REST API/service is used/consumed, and get a sense of what each feature does.

Reading unit tests

Next I will go a level deeper/more granular and start reading the unit tests. For these tests, a great way to go through these is to have the test file and source file open side-by-side in your IDE.

Generally but not always, unit tests will be grouped by function or at least more correlated to a couple functions (in contrast to integration and end-to-end tests which will hit many functions and harder to trace back to a specific part of the code). This makes having the source file open side-by-side with the tests easier to start getting an understanding of what the code is doing. You can see expected inputs, follow those through the function, see how they're used or manipulated, then see what the expected output should be.

Test files

Usually codebases will have a test file name that corresponds to a source file, something like books.service.js and books.service.test.js. But in the case of integration or end-to-end tests, they most likely won't. This makes locating the unit test that goes along with a given source file easy to find and open up next to your source.

Summary

Next time you need to get up to speed on a new codebase, try the method discussed above. It's always helped me learn much more quickly and given me a "roadmap" of where/what to start focusing on.

Making test stub setup effortless using Sinon

Corey Cleary — Fri, 03 Feb 2023 02:10:38 +0000

When you're writing tests, it's often the general setup like creating fake data and setting up stubs/mocks/fakes that can hold you back. Sometimes to the point where you end up skipping writing the tests altogether...

This setup can feel tedious and creates more work to do on top of having to write the rest of the test, come up with the assertion, and make sure you're covering reasonable test scenarios.

But by using the testing library Sinon - and having a few real-world/practical examples - test setup can become effortless.

I'm going to go over one of those examples in this post to help explain how to write stubs/fakes so you can quickly write your tests and get back to writing application code.

The unit under test

The unit under test here - the getBinarySliceFromFile() function - returns a slice of a binary file based on positional arguments.

export const getBinarySliceFromFile = (start, offset, fileName) => {
  const fileBinary = fs.readFileSync(`./binaries/${fileName}`)
  return fileBinary.subarray(start, start + offset)
}

readFileSync() returns the file in binary/Buffer format.
start here is the starting point to slice the file, so you could slice from any arbitrary point
offset is sort of like the end point to slice, but rather than an explicit end, it's used along with start to define the end

The issue is that in this function it's reading from the file system via the Node fs module. We want to test the output of this function, so we need to be able to control what fileBinary is. And we can't hardcode the directory path + filename.

To do this, we can stub out / fake the fs module. This will allow us to set specifically what to return from the fs.readFileSync() function, which is what we want.

The test code

describe('getBinarySliceFromFile()', () => {
  afterEach(() => {
    sinon.restore()
  })

  it('should return a slice of the binary starting from an index point', () => {
    const index = 1
    const offset = 1

    // fake a binary
    const fakedFileBinary = Buffer.alloc(2)
    // output of the below is <Buffer 81 0b>
    fakedFileBinary.writeUInt16LE(2945)
    // create the Sinon fake, so that it returns the fake binary above
    sinon.replace(fs, 'readFileSync', sinon.fake.returns(fakedFileBinary))

    const slice = getBinarySliceFromFile(index, offset)

    expect(slice.length).to.equal(1)
    // notice that the below is the last block in <Buffer 81 0b>, which proves the "slicing" works
    expect(slice).to.deep.equal(Buffer.from([0x0b]))
  })
})

The fake setup in the code above happens here:

    // fake a binary
    const fakedFileBinary = Buffer.alloc(2)
    // output of the below is <Buffer 81 0b>
    fakedFileBinary.writeUInt16LE(2945)
    // create the Sinon fake, so that it returns the fake binary above
    const readFileSyncFake = sinon.fake.returns(fakedFileBinary)
    sinon.replace(fs, 'readFileSync', readFileSyncFake)

On the first two lines, it's nothing Sinon-specific but just assigning a Buffer (which is binary data) to a variable and then writing data to that Buffer.

The next part is where Sinon comes into play:

    const readFileSyncFake = sinon.fake.returns(fakedFileBinary)
    sinon.replace(fs, 'readFileSync', readFileSyncFake)

This is using the Sinon fakes API, which are similar to stubs and likely what you are more familiar with hearing. According to the Sinon docs, "Fakes are alternatives to the Stubs and Spies, and they can fully replace all such use cases."

sinon.fake.returns lets us setup the fake using the simulation of the file binary. And the .replace() function lets us specify the fs method to "overwrite" - in this case readFileSync - and instead return the fake.

That code is pretty simple to setup but very powerful. Now we don't have to modify the function definition for getBinarySliceFromFile() and can instead call it as usual. When it gets called, when it comes time to hit the fs code Sinon will have swapped that out with the fake. And you can get an output from getBinarySliceFromFile() to then make assertions on. A pretty effortless way of doing it.

Test cleanup

You may have noticed there is an afterEach() block defined at the beginning of the test suite:

  afterEach(() => {
    sinon.restore()
  })

Doing this will reset Sinon and the defined fakes after each test, which is a good practice for test "cleanliness" so you're starting from a clean slate with each test and there aren't leftover fakes from previous tests that never got reset.

Note that fakes are available in Sinon with version 5 and up. If you are using a version below that I would recommend upgrading, but if you can't do that you can use the stub API that Sinon provides, and is very similar.

This vs. passing in as an argument

One thing we could have done and avoided Sinon and fakes entirely would have been to pass the file binary in as an argument to the function, like so getBinarySliceFromFile(start, offset, fileBinary) (note the last arg was changed from the file name to the file binary). If we did that it would let us pass in a simulated binary (just like how we set it up using a Buffer above) in the tests rather than using Sinon.

This can be a valid way, but I don't think it's great to do that here. Any code that calls getBinarySliceFromFile() will now have to call fs.readFileSync() and pass the result of that to getBinarySliceFromFile(), which starts to break encapsulation. Whereas if we keep fs.readFileSync() in the function body, consumers of getBinarySliceFromFile() only need to pass in a file name, and that function is responsible for fetching the file binary. This is better encapsulation in my opinion.

Summary

The next time writing tests is holding you back because either you don't know how to setup fakes/stubs, or the setup is just too tedious, use Sinon and the example test code above to help guide you and make it very easy to get those tests written!

And if you're feeling like you don't know where to even begin with real testing examples and patterns, the below posts will help you with that:

In what layer does request validation go in a Node REST API?

Corey Cleary — Thu, 12 Jan 2023 21:05:10 +0000

In a Node REST API, there are multiple different layers where you could conceivably put request validation, the code where you verify that the request is "valid" and has the correct fields, checks that those fields are the expected types, etc.

In a layered API that roughly follows a web tier, service tier, and data tier structure, it makes more sense to do the request validation in the web tier, since it is the one that deals with the HTTP request. If the request is invalid, it doesn't make sense to call all the other layers of code. (Note: typically you will have some kind of model or query validation "further down" the stack, but that is a different type of validation than what we're talking about here.)

But the web tier is made up of middleware and controllers/router handlers. Which of these should you put your request validation in? With all the different types of logic in a REST API, knowing what code goes where can be confusing with lots of conflicting information out there.

For a recap on the difference between middleware and controllers, check out this post.

Best place

I think the conclusive place to put request validation is in the middleware. You usually want to bail as early as possible if you are dealing with an invalid request, as opposed to letting code seep "further down" to execute. Controllers/route handlers are closest to middleware - so it may not seem like a big deal to do the validation there - but it's still "one layer" deeper than just bailing early from the middleware.

You also keep your controllers thinner and cleaner by keeping request validation out of them, and you end up having better separation of concerns. Middleware is responsible for the validation, and controllers are responsible for routing the request where it needs to go after it's been validated.

You can also better reuse the validation logic in other routes if it's separated into middleware rather than baked into a controller.

Below is pseudo-code for what this would look like:

const app = express()

// the schema to use for validating the request
const orderSchema = {
    type: 'object',
    required: ['total', 'items'],
    properties: {
        total: {
            type: 'number',
            minimum: 0.5,
        },
        items: {
            type: 'array',
      uniqueItems: true,
      items: {
        type: 'object',
        properties: {
          itemId: {
            type: 'integer'
          },
          quantity: {
            type: 'integer'
          }
        }
      }
        }
    }
};

app.post(
    '/order', // route name/definition
    validate({ body: orderSchema }), // validation middleware
    createOrderController // controller
)

In this case, validate() is a JSON Schema middleware library function, and that Schema is defined for an order. If the request that the REST API receives is invalid - for example, maybe it's missing the total field - then the request will stop and return, skipping the controller.

Summary

Next time you are setting up request validation, put it into a middleware function before your controllers/route handlers. This way the code will exit as early as possible and you won't spend valuable server resource time and compute power, and will return a response to the client as quickly as possible.

What is the difference between Middleware and Controllers in Node REST APIs?

Corey Cleary — Thu, 15 Dec 2022 02:04:15 +0000

When you're writing code dealing with the request lifecycle, what is the difference between middleware and controllers? What code goes in each? They both touch the "request object" and do things with it, so it seems like there might be overlap or controllers might be redundant to have.

But they are distinctly different, and understanding the differences in the use cases for each logic type is important in structuring maintainable and scalable applications. Let's examine the distinctions between the two and what code goes where, so the next time you are working on an API you can build it sustainably.

Note: this post focuses on middleware and controllers from a NodeJS perspective but is applicable to other languages and frameworks.

Middleware vs. Controllers

What's immediately confusing is that a controller can be considered a type of middleware. (And to add to the confusion, "controllers" are sometimes referred to as "route handlers".) But middleware - compared to controllers - handles HTTP-specific concerns as well as concerns that are common to every (or most) request. "HTTP concerns" are things like handling CORS, parsing the body, cookies, etc. "Common concerns" are things like request validation, sessions, request logging using something like morgan, and authentication and authorization. Middleware is basically agnostic of application logic.

All of the above you don't want in your controllers. Why? Because controllers are responsible for routing a given request where it needs to go for "processing". They don't directly deal with application logic, but send the request to code that does. Controllers should be fairly "thin" and not do a whole lot. So, a "book order" controller would take the request object after it's already been "pre-processed" by middleware and "successfully passed" it, pull out what data it needs from either the query string or body and send it to the service layer/domain logic layer to execute the business logic.

Middleware functions are typically (although not always) passed through by all HTTP requests and do not fulfill the request, unless it's a case of returning an error response early. It's the controller that ultimately fulfills the request with a successful response. They are usually are specific to a single endpoint (like /items, /orders, etc).

Example code

Controller code will look something like this:

const express = require('express')
const router = express.Router()

// Controller
const createOrder = async (req, res, next) => {
  // grab what we need from the request...
  const {customerId, orderTotal, orderItems, paymentDetails} = req.body

  try {
    // ...then route it to the appropriate business-logic-processing functions...
    const customerData = await getCustomerData(customerId)
    await processOrder(orderTotal, orderItems, paymentDetails, customerData)
    await sendConfirmationEmailToCustomer(customerId, orderItems)

    // ...then fulfill the request with the response object
    res.sendStatus(201)

    return
  } catch (err) {
    res.sendStatus(500)

    return
  }
}

// Route definition
router.post('/order', createOrder)

module.exports = router

Above you will notice that the controller grabs what we need from the request, routes it to the appropriate business-logic-processing function(s), and finally fulfills the request via the response (res) object.

Then the chain of middleware will look similar to this, usually called either app.js or server.js:

const express = require('express')
const app = express()
// middleware library imports
const bodyParser = require('body-parser')
const cookieParser = require('cookie-parser')
const passport = require('passport') // for auth

const routes = require('./routes') // this is a .js file that contains all your route definitions mapped to their respective controllers

// here is the middleware being chained together
app.use(bodyParser.json())
app.use(bodyParser.urlencoded({ extended: true }))
app.use(cookieParser)
app.use(passport.initialize())

// routes defined last in the chain of middleware
app.use('/api', routes)

In the above, we chain the middleware together, using app.use(), before we have the routes (which map to controllers) at the end of the chain, so that the request has to pass through the middleware in order before even hitting the controllers.

Note: if I need to build custom middleware I will generally add a directory called middleware/ to be at the same level as the controllers/ directory, and put my custom middleware functions there.

Summary

While it may seem confusing at first, the difference between these two "layers" / logic types are generally fairly clear, so it should make sense now as to what code goes where. Occasionally, you may run into a scenario where it's less black and white, but then you just have to decide what makes sense for where to put the code based on you specific application requirements.

Why does this matter? If it's not obvious, this separation of concerns is necessary for developing a maintainable codebase and API. You will save yourself a lot of painful refactoring in the future if you can achieve this separation, because as your code grows and you add more developers to the team things can start to get out of hand quickly.

Multiple assertions per test may actually be OK

Corey Cleary — Wed, 30 Nov 2022 00:00:00 +0000

In this post I want to examine the claim that "you should only ever have one assertion per test". This is oft-repeated to the point it is basically considered a rule or best practice at this point. But is it actually a good practice that should be followed universally?

Knowing or learning how to write good tests for your code can be confusing enough - you have to know when and how to use test doubles, write meaningful test cases and descriptions, when to stub a function vs use a real database/API, etc.

Having a bunch of so-called rules people have made up on top of that makes it all even more confusing. And there are a lot of different interpretations on what this "rule" even means. So I want to look at that statement further and discuss some instances when it might not make sense, and what the original rule should was (likely) meant to intend in the first place since it seems to be repeated often in developer-land.

A bad test

First let's look at an example of a test that has a lot of assertions for a single test case. The code we want to test is:

class User {
  constructor(userName) {
    if (!userName) {
      throw Error('A username must be provided')
    }

    this.userName = userName
  }

  getFormattedUserName() {
    // a simplified example just for demonstration purposes, you likely wouldn't be appending a string to a username
    const identifier = 'usr'
    return `${this.userName}-${identifier}`.toUpperCase()
  }
}

And the bad test is:

import mocha from 'mocha'
import {expect} from 'chai'

it('should return a correctly formatted username', () => {
  expect(new User()).to.throw()

  const user = new User('testcaseuser')
  expect(user).to.be('TESTCASE-USR')
})

This is an example where multiple assertions are bad, because it's essentially two tests in one: 1) that the class instantiated without a user name string should throw an error and 2) that the formatting works as expected. But the test case - "should return a correctly formatted username" - means only the second assertion is irrelevant.

So having multiple assertions here is bad because of an irrelevant assertion.

The fix

It can be helpful to think about what the output for your function is that you are testing. The output of getFormattedUserName() is not a thrown Error if the username is not passed in when instantiating the User class. Instead, it is a formatted username. With that in mind, here is an example of the refactored tests. The obvious thing to do here is to limit the assertions to just testing the formatting, like so:

it('should return a capitalized username', () => {
  const user = new User('testcase456')
  expect(user).to.match(/TESTCASEUSER456/)
})

it('should return a username with an appended identifier', () => {
  const user = new User('testcase456')
  expect(user).to.match(/-USR/)
})

The original test was just one test, but in this case we broke it out into two test cases in order to test more expected output.

So now we have one assertion per test, but those tests are fairly closely related. What if we did this?

it('should return a correctly formatted username', () => {
  const user = new User('testcase456')
  expect(user, 'should capitalize username').to.match(/TESTCASE456/)
  expect(user, 'should append username').to.match(/-USR/)
})

We can add an assertion description as the second argument to expect(), which allows us to have multiple assertions per test and which makes it clearer what the issue is if the assertion fails. (NOTE: using Mocha and Chai here, but most test frameworks should also have this ability to add an assertion description)

This code is helpful to discuss a certain point - that "no multiple assertions per test" makes more sense / may be clearer as "no multiple aspects/concepts per test". I would argue that this example above, where we have one test case - a discrete aspect/concept - with multiple assertions is actually OK. Those assertions are both testing closely related formatting concepts, which can be tested with two "sub-assertions".

Also, one of the reasons "no multiple assertions per test" has been touted as a rule is because if the assertion fails, you won't know why necessarily, but here we get around that issue by having assertion descriptions.

In the second example (multiple assertions) we only have to instantiate User once, wherease with two tests we have to do it twice. User here is so simple it probably doesn't matter, but you can imagine a scenario where the test setup is more complex. More than one assertion per test may make sense in a case like this where otherwise you’d have to copy most of test A again to make test B

Having this test as-is above vs. breaking it out into two tests - honestly I think both are OK. At a certain point it gets into splitting hairs, or more of a readability thing. But I wanted to include the test above to show that multiple assertions per test is not always bad.

Note: it should be pointed out that some test runners will skip the rest of the assertions if one fails and go to the next test, and we mitigate this a bit by having clear assertion descriptions like above, but it's worth pointing out.

Single responsibility principle

I'll jump into some other examples of where multiple assertions may make sense in a bit, but want to discuss one more thing related to when this rule does make sense.

If you find yourself having to continually add assertions to your test case, it might be a sign your "unit under test" is doing too much and is violating the "Single responsibility principle" (SRP), that states "every class, module, or function in a program should have one responsibility/purpose in a program." This is a key sign that you might need to refactor your function, and is a good example of too many assertions being a code smell that you can use to reevaluate the code you're testing.

This code smell is a sign you should break that function down into multiple ones. You can keep the same test, but treat it more as an "integration" test, and include more "scoped" smaller unit tests after you've broken that into separate tests.

Now that we've covered the general intent of the rule/principle in question and shown some examples of bad tests that likely led to this becoming a rule, let's look at some other types of examples where it's more nuanced and where multiple assertions may make more sense.

REST API testing

One use case where you probably should use multiple assertions for your test case is if you are testing a REST API. Imagine you are testing an API endpoint and you want to test a successfull response is returned with expected data:

describe('GET /v1/accounts', () => {
  it('should return all the accounts', () => {
    return api.get('/v1/accounts')
      .expect(200)
      .expect('Content-Type', 'application/json')
      .expect(res => {
        expect(res.body.data).to.deep.equal({
          accounts: [
            {id: 1, name: 'John'}
            {id: 2, name: 'Jack'}
          ]
        })
      })
  })
})

In this example, the case we are testing is that the endpoint returns all expected accounts. As part of that, it has assertions on a couple of the response headers, in addition to an assertion for the actual response body/accounts.

At first glance this appears to be in violation of the "one assertion per test" principle, so you might be inclined to rewrite the tests like so:

describe('GET /v1/accounts', () => {
  it('should return 200 OK', (done) => {
    api.get('/v1/accounts')
      .expect(200, done)
  })

  it('should return Content-Type application/json', (done) => {
    api.get('/v1/accounts')
      .expect('Content-Type', 'application/json', done)
  })

  it('should return accounts', (done) => {
    api.get('/v1/accounts')
      .expect(res => {
        expect(res.body.data).to.deep.equal({
          accounts: [
            {id: 1, name: 'John'}
            {id: 2, name: 'Jack'}
          ]
        })
      })
  })
})

This gets us to one assertion per test, but does it make sense?

Firstly, you now have to execute the GET call multiple times, which adds additional test execution time / latency to your tests. Additionally, API tests by nature are closer to integration tests since their execution ends up hitting multiple parts of the code, so it's not implausible in these scenarios to need more than one assertion to test the output / result of what you are testing.
Also, those headers could be considered to be part of the response body, and thus part of a "singular" test case.
Lastly, test headers by themselves don't exactly constitute a "unit" to be tested - I'd rather test all the headers together with a case that tests for appropriate header responses. Sure, if one of those headers changes and it breaks the test, depending on your test runner the rest of the assertions may not run, but you will know right away which header change broke the test.

So in this scenario I think the first example test is acceptable rather than testing each individual header individually.

An alternative - which would still include multiple assertions per test - would be to test the response body in one test case and the headers (multiple assertions) in another test case, like so:

describe('GET /v1/accounts', () => {
  it('should return the appropriate headers', () => {
    return api.get('/v1/accounts')
      .expect(200)
      .expect('Content-Type', 'application/json')
  })

  it('should return all accounts', () => {
    return api.get('/v1/accounts')
      .expect(res => {
        expect(res.body.data).to.deep.equal({
          accounts: [
            {id: 1, name: 'John'}
            {id: 2, name: 'Jack'}
          ]
        })
      })
  })
})

This is more of a "middle ground" and something you may want to consider when writing endpoint tests.

Object and property testing

This could be a blogpost by itself, but I want to briefly cover it because this is a scenario where you might need multiple asserts per test in order to verify the test case.

Imagine a simplified function that returns an object like:

const getUserPreferences = () => {
  // the below is hardcoded just for demonstration purposes
  return {
    preferences: {
      emailCommunication: false,
      phoneCommunication: false,
      smsCommunication: true
    }
  }
}

We can imagine this is some object related to notification preferences from an online service I signed up for.

The test for this could look like:

it('should return notification preferences', () => {
  const preferences = getUserPreferences()
  expect(preferences).to.deep.equal({
    preferences: {
      emailCommunication: false,
      phoneCommunication: false,
      smsCommunication: true
    }
  })
})

That test will pass, with only one assertion, but what if we want to add another property to the returned object later on? Like physicalMailCommunication for example.

We could update the test like so:

it('should return notification preferences', () => {
  const preferences = getUserPreferences()
  expect(preferences).to.deep.equal({
    preferences: {
      emailCommunication: false,
      phoneCommunication: false,
      smsCommunication: true,
      physicalMailCommunication: true
    }
  })
})

Or we could do something like this:

it('should return notification preferences', () => {
  const {emailCommunication, phoneCommunication, smsCommunication, physicalMailCommunication} = getUserPreferences()
  expect(emailCommunication).to.be.false
  expect(phoneCommunication).to.be.false
  expect(smsCommunication).to.be.true
  expect(physicalMailCommunication).to.be.true
})

This is a case where having multiple assertions per test is valid, because you are testing properties of the object. Now, you might not need to check each property like this, and instead could just do a deep object euality check like the first test. If the test fails Mocha (and most test runners) will show you the properties of the object that don't match the expected values. But testing each property one by one may be valid if you have a function that returns an object where you don't care about testing all properties and only want/need to test a few. And in that case asserting on deep object equality won't work.

Multiple assertions in an integration test

Lastly, with integration tests you're likely to run into scenarios where you need to have multiple assertions for a given test case in order to test it properly.

An example of this is in some tests I wrote for a Redis Streams library, you can see the test linked here and a slightly simplified version below:

it('should subscribe to a consumer group and process the event', async () => {
  const workerFunction = () => {};
  await streamClient.subscribe({
    groupName, streamName, readTimeout: 20, workerFunction,
  });

  const eventPayload = {
    test: 'this is a subscribe test',
  };
  const streamClient2 = new EventStreamClient();
  await streamClient2.publish({ streamName, eventMeta: defaultEventMeta, eventPayload });

  // we can't spy on private functions (workerFunction is private w/in the closure in subscribe
  // but testing there are items, and they've been acked (not on PEL) works just as well
  const eventFromStream = await getRange(streamName, 1);
  const formatted = getFormatted(eventFromStream);
  expect(formatted[0]).to.include(`${PAYLOAD_PREFIX}-test`, 'this is a subscribe test');

  const pending = await getPending(streamName, groupName);
  expect(pending[0]).to.equal(0);
});

The subscribe() function above subscribes to events from a stream, processes the received event and - if processing happens successfully - finally ACK's (acknowledges) it. This test case is testing a single concept - that it processes the event (and in order to do that, subscribes to the event stream) - but has two assertions.

This is because we need to first check the event, as part of test setup, is published to the stream. If that part fails, then anything related to the subscribe() function will fail and we want that to happen early because we need to fix that. This first assertion is also needed, because as noted in the code comment we can't spy on the workerFunction passed in to make sure it was called. So instead of having a separate assertion for that, like expect(workerFunction)to.have.been.called, we can make sure those events are there and then they are no longer in pending status, because only successfully processed items - the workerFunction does the processing - are ACK'd and removed from the pending status.

And that's what the second assertion checks.

As you work from unit to integration to end-to-end tests, this can become more common. That's because generally with integration tests you're testing a larger scope, and with end-to-end tests you're testing a larger scope than that. "Scope" meaning number of touchpoints. So it follows that you'll likely need more assertions to verify the behavior.

Summary

Hopefully you have a more nuanced understanding of the "multiple assertions rule" now, and can make a more informed decision on how you should write any given test for your application. I didn't want to provide anything prescriptive, because each application and context is different and rules/best practices become dogmas when this is not taken into account. Every single production application you've ever worked on for a company has done things slightly (if not totally) differently. That doesn't mean that just because it's different that its correct and you can ignore bad practices, but it's important to take into consideration.

The point is not that you have to or should use multiple assertions per test - you may not have any. But moreso that it's not a hard rule you should strictly adhere to. Context is everything.

If you find yourself having multiple assertions per test, consider the following:

Does having to add more assertions mean that the code being tested is doing too much? Does it need to be refactored to better handle a single responsibility?
Are you focusing on what the "output" of the unit under test is? Or are you testing extraneous and irrelevant things? This especially applies to "pure functions" that don't call a database, API, etc.
Are there multiple assertions because you are testing integration or end-to-end tests, that would be difficult to only assert on one output or action being performed?

If you consider the above and decide that multiple assertions makes sense, then buck the "rule" and go for it. And if you want to learn how to better structure your tests using Mocha, check out this other post on doing that.

Better test structuring using Mocha's context()

Corey Cleary — Thu, 03 Nov 2022 00:00:00 +0000

Sometimes when working on adding or modifying unit tests within a test suite, you may not be sure what the best way to structure the tests within that suite is.

The tests may be getting cluttered, out of control, and you may have repetition.

But better organization will lead to making writing tests less painful (and even enjoyable) because they're easier to maintain.
And since you're usually working on code within a team, it will make it easier for everyone on that team to maintain, understand what the tests are testing, and to add new tests as well.

If you're using for your tests, one way to do this is by using the context() function. You're almost definitely already using describe() to group your test cases, but context() will provide for more explicit grouping of tests within your suite.

Example

Take a look at the example below, that is only using describe() for grouping tests:

describe('Array', () => {
  describe('indexOf()', () => {
    it('should not throw an error when the array item is not present', () => {
      (() => {
        [1, 2, 3].indexOf(4)
      }.should.not.throw())
    })

    it('should return -1 when the array item is not present', () => {
      [1, 2, 3].indexOf(4).should.equal(-1)
    })

    it('should return the index where the element first appears in the array', () => {
      [1, 2, 3].indexOf(3).should.equal(2)
    })
  })
})

You'll notice that the phrase "when the array item is not present" is repeated twice, in the 1st and 2nd tests.

It was kind of annoying to type that twice, and from reading these tests you don't immediately pick up on the fact that those tests are related. If you have business requirements for a feature you're building, you might even have requirements like "when X is not present, do Y" and "when X is not present and A, do B".

Instead it would be great if we could group these together. Now, we could use another describe() to do this, like so:

  describe('indexOf()', () => {
    describe('when not present', () => {
      it('should not throw an error when the array item is not present', () => {
        (() => {
          [1, 2, 3].indexOf(4)
        }.should.not.throw())
      })

      it('should return -1 when the array item is not present', () => {
        [1, 2, 3].indexOf(4).should.equal(-1)
      })
    })

    it('should return the index where the element first appears in the array', () => {
      [1, 2, 3].indexOf(3).should.equal(2)
    })
  })

But then it sort of begs the question... what is the difference between the nested describe() block and the other individual it() blocks within the same "parent" describe('indexOf()', () => {})?

This is where the context() function comes in. We can instead use it to group the contextually similar test cases, and have them still be within our "parent" unit test describe() block.

Take a look at the example above, rewritten below using it:

describe('Array', () => {
  describe('indexOf()', () => {
    context('when not present', () => {
      it('should not throw an error', () => {
        (() => {
          [1, 2, 3].indexOf(4)
        }.should.not.throw())
      })

      it('should return -1', () => {
        [1, 2, 3].indexOf(4).should.equal(-1)
      })
    })

    context('when present', () => {
      it('should return the index where the element first appears in the array', () => {
        [1, 2, 3].indexOf(3).should.equal(2)
      })
    })
  })
})

context() is just an alias for describe(), so it works exactly the same and is the same function.
From a test runner perspective this doesn't do anything - it doesn't change the way the tests are run, the order they're run in, etc.

But BDD-style tests are designed to be readable to even non-developers (like QA and business analysts/product managers).
So having an "extra" function we can use to make it clearer how the tests are grouped and what a given "test block" is testing is a huge boost for readability.

When to use context?

A good heuristic for when to use context() over describe() is when you repeat yourself across tests.
That's a key indicator that what you're dealing with is, in fact, a contextual set of tests, and that they can likely be grouped together.

Not everything needs to be grouped using it, I certainly don't do that. But when you have the following:

you're repeating the same language in your test cases
you're starting to have many test cases within a single "parent" unit, and that starts getting out of control, and can group some into contextually similar sub-groups

...context() can come in very handy.

It will make your test suites easier to read, your teammates will understand what the intent of the tests are, and it can help the tests better serve as documentation of requirements/functionality.

What is the difference between a job queue and worker threads?

Corey Cleary — Thu, 20 Oct 2022 00:00:00 +0000

There are several different architectural patterns and solutions for handling asynchronous code. Queues, promises, worker threads, child processes, the cluster module, scaling horizontally with multiple containers, etc.
Whether you're dealing with scaling up or computationally-heavy tasks, certain patterns/solutions will be the right choice over others.

Two of these patterns/methods that might be confusing as to when you would use which one due to perceived similarities are job/task queues and worker threads.
Both are options if you want to handle computationally heavy work and not block the event loop.
With both you are "outsourcing" work to be executed elsewhere besides the event loop.

Getting a better grasp on these will help you know when to use which solution, and help you implement your designs faster and in a more robust way.
With architecture choices I've found that once you decide on it, it can be more difficult to change compared to changes at the "code-level", so best to pick the right solution for your problem/use case off the bat.

So what are the differences and when would you use which one?

Job queue

Compared to worker threads, a "job queue" (or "task queue") is more of an architectural design pattern.
It's not a part of the NodeJS API, but is a pattern you implement (or a third party library implements for you),
with a set of common features.

Job queues generally provide features like scheduling, coordinating, and retrying jobs, which helps with scalability as well as "fault-tolerance".
Instead, if you handle all of that in application memory then it will be ephemeral, and less able to recover upon failure.
If the main process dies, it will take down all threads with it.

With a queue, the jobs are usually run in a separate component/service from the application producing the jobs,
and can run in parallel with it. The service that produces/schedules/creates the jobs is the producer,
and the service(s) that do the work are consumers.

Worker threads

"Worker threads" are more of a primitive as opposed to an architectural or design pattern.
This primitive is a building block used to constructu a larger abstraction, and you could even use worker threads to implement the job queue pattern.
Of course, you would need a lot of other code and features to implement that pattern - simply "dropping in" worker threads
doesn't automatically mean you have a job queue - but it can be done.

Each worker thread has its own V8 instance and event loop, compared to single-threaded NodeJS where there is one event loop that all code uses.
This is an important distinction, because having multiple event loops is a big part of what allows worker threads to work as semi-isolated.
I say "semi-isolated" because they still run as part of the same process as the "main" NodeJS thread.

To use queue terminology, in this case worker threads are generally the "consumers", and the "main" NodeJS thread is the "producer".

In contrast to a job queue (via the library you're likely using), you need to implement retry logic, scheduling, and priority levels for the worker threads yourself.
There might be some libraries out there that do this that I'm not aware of, but the general idea is that retry-ability
is not native to worker threads. And since the "main" Node thread is the orchestrator of the worker threads, it would be something you'd have to implement at that level, including
tracking of which ones failed, initiating retries, etc.

Instead, if you have a CPU-intensive task that only has a single step - where failure may mean less than if you have a multistep process, in which case you'd likely have to rollback certain things at certain steps -
and you can afford to let it fail, a worker thread might be the right choice for your use case.

Another important thing to note - an uncaught exception will kill the whole process, taking all threads including the "main" Node thread down with it. If that happens in a job consumer/processor, you still have all the others running
assuming you have more than one consumer. Obviously you can mitigate this by having a "global" error even listener and handle the exception there,
but even in case of a single worker thread failure another worker thread would have to be spun up and would not automatically recover.
That's logic you would have to implement yourself in the "main" thread.

Questions for use case

Some things to consider:

Do you already have queue infrastructure setup, a library selected, pattern for calling it, etc.?
- If you do, and you're on the fence about which solution to choose given your use case, it may make sense to use queues if you already have them setup.
Do you need schedulability? (then use a job queue)
- It would be very difficult to build this feature using a purely worker threads solution - the main Node thread would have to act as an orchestrator polling some database where you have the schedules stored, and use some lookup to determine what code to run for the job, etc. What you would end up with would likely resemble a job queue anyways.
Do you need retryability? (then probably use a job queue)
- You can build this feature with worker threads but what if the main node thread goes down or the container is restarted? If you were tracking retries in memory you would lose the number of attempts and when the process/container restarts it wouldn't be retried. Because threads are more "ephemeral" you have to factor in fault tolerance.
How many threads do you have available vs. how many "nodes" can you distribute for workers, and what is the processing power of each of those?
- Depending on how many threads you have available in your service, and how many individual job consumer processes you can sping up will likely factor into your decision. If it's too expensive to be running that many separate job consumers, which will likely be a server/container per consumer, and you've determined you need that many consumers given your workload, it may make sense to look at if worker threads would be more cost-effective. Of course, if you have such a high workload you need lots of consumers then you might need lots of threads too, which may come back around to being too expensive, but it's something to consider.

Examples of when you would use which

At a certain point if you have enough requirements like being able to set job priority, scheduling, etc. it may make more sense to have a separate architectural piece for this, via a job queue.

A job queue is useful if you have lots of things in your "order of operations" to be done rather than a single thing
As briefly mentioned before infrastructure/setup cost is a factor you'll have to consider in setting up a queue.

But looking at some more specific examples...

Lots of sensors / "Internet of Things" (IoT)

In this use case you have many sensors communicating frequently over the network with your server and database, likely calling multiple services and inserting/updating several different tables with sensor data.
It's a more involved, multi-step process to process the data sent in by the sensor. It's not just a single task that happens.
And you need to return a successful response from the server to your sensors as quickly as possible since sensors usually can't wait a long time for a response, and the server has lots to process.

So, you recognize you need to use an asynchronous pattern for this.

Worker threads are good for CPU-intensive work but don't help much with I/O-intensive work,
and in an IoT use case where you have potentially hundreds of thousands of devices sending data every X seconds
its very I/O-intensive, at the network and database layers as opposed to CPU-intensive processes like machine learning or image processing
(generally speaking).

It's also possible that you have too many requests from sensors at once and don't have enough threads to delegate all that work in a timely fashion to the worker threads.

This is a use case where a queue is almost always going to be the better option.

Image processing

Imagine a use case where you have an auction site and a user can upload images of the items they're selling.
These images may need to be scaled down to a reasonable resolution so as to not eat a ton of storage space,
as well as be able to be edited by the user (rotated, brightened, etc).

While this can happen in the browser, generally image processing, resizing, etc is computationally itensive and so happens on the server using something like ImageMagick.

This "task" is not complex in that there are multiple steps that need to happen. Instead it's a single step but is compute-heavy.

But it's also a task that if it fails, likely isn't a big deal. You can show an error message and the user can simply retry.
Could you setup a queue for this? Sure, but the infrastructure setup if you're solely using it for this use case may not be worth the extra setup overhead and cost.

Because you probably don't need retries, you definitely don't need schedulability, and this is a one-off task, using worker threads is a good solution for this use case.

Summary

Hopefully this examination of differences and tradeoffs between worker threads and a job/task queue gives you a better idea
of which one to choose for you given use case. There are lots of asynchronous patterns in NodeJS and software architecture in general,
so having a clearer understanding of some of these patterns will make choosing the right solution from the start easier, and remove headaches later on.

Love JavaScript but still getting tripped up by local dev, architecture, testing, etc? I publish articles on JavaScript and Node every 1-2 weeks, so if you want to receive all new articles directly to your inbox, here's that link again to subscribe to my newsletter!

Using a task queue vs. just not waiting for Promise to resolve

Corey Cleary — Thu, 04 Aug 2022 00:00:00 +0000

When working with Node and JavaScript one of the benefits is that we can make code asynchronous, whether via callbacks or Promises. Instead of having to wait for a line of code to finish executing we can continue on if we don't await or .then() the Promise, or don't nest the callbacks if using those.

You are also likely aware of task queues, where instead of executing the code in your "main" service you create a job/task in a queue and a consumer watches the queue and it does the work rather than the "main" service. Rather than being a native asynchronous Node/JS thing, this is an asynchronous pattern at the architecture level.

Usually a task queue is used when you want to offload a longer running block of code and you don't need the results of that code in the rest of your code.
But if we can skip waiting for asynchronous JavaScript code to finish, and keep the code "fast" that way, doesn't that accomplish the same thing?
Why would you need a queue to begin with?

This is an important concept to understand especially as you become more "senior" and are making architecture decisions. So let's explore both and understand what the difference is / why you would want to use one option over the other.

Code processing

When you don't wait for the Promise to resolve, the most important thing to remember is that Node is still processing that Promise from the event loop. It's not like it disappeared, or was sent to some magic factory that does the work for free.
So even if you don't wait for resolution, your server is still executing that code. This is important to point out because you may have a scenario where that execution is computationally expensive (using lots of CPU and/or Memory).
So even if you don't wait for it to complete, server performance will be something you need to factor in.

Imagine you have a computationally intensive task like image processing, where when that is executed in the Node event loop it bogs down your server.
This is a prime candidate for something that should be pushed to a task queue. You're offloading that computationally expensive somewher else, again you can't avoid it. But that work is no longer in the main service bogging it down, and instead you can more immediately return the response to the user. And you can now scale up or down consumers (the "services" executing the code) to essentially load balance the work.

Error handling when not waiting for Promise resolution

This is probably a good time to discuss another important consideration when not waiting for Promise resolution.
If the Promise rejects, you still need to catch it. If you don't you'll get an Unhandled promise rejection error.

The most "local" way to do that is to use .catch(), like so:

async function test() {
  // artificial rejection just to demonstrate
  return Promise.reject('this is a rejection')
}

// notice, NO .then() or await
test().catch((err) => {
  // handle Promise rejection here
  console.error(err)
})

Note that you can't use try/catch here like so:

try {
  test()
} catch (err) {
  console.error(err)
}

In a try/catch even without await it will result in an uncaught Promise error. There's not another way of doing this with try/catch that I'm aware of.

You could also use a "top-level" as opposed to "local" error handler, something like:

process.on('unhandledRejection', (reason, promise) => {
  console.log('Unhandled Rejection at:', promise, 'reason:', reason)
  // Application specific logging, throwing an error, or other logic here
})

But regardless, it needs to be handled. Especially if you're using newer version of Node. Depending on the version, newer versions won't just throw a warning, they will kill the server. And if you go the "top-level" route you may lose out on supplementing the error with other variables or information that are within the function's scope.

Retrying failed Promises

Another thing to consider if you are thinking about not waiting for Promise resolution is that if it does fail/reject, you need to add code to handle retrying the Promise (if you in fact want to retry it). Something like:

const retry = (fn, ms) => new Promise(resolve => { 
  fn()
    .then(resolve)
    .catch(() => {
      setTimeout(() => {
        console.log('retrying...')
        retry(fn, ms).then(resolve)
      }, ms)
    })
})

retry(someFnThatReturnsPromise, 2000)

Of course if you don't care about the function/Promise rejecting, and can live with that, then you don't have to do this. But usually you're probably going to want that code to execute successfully.

The code above gets us Promise function retries, but what if the someFnThatReturnsPromise above keeps failing? Maybe there is a logic error or TypeError somewhere within the function definition. No number of retries are going to get it to successfully complete.

We can implement a maxNumberRetries in the retry() function, and that will stop the retries after X number of times. But we're still back to the issue that the code isn't completing successfully.
And those retries that happen are still in the event loop, using server processing power (back to point #1). What if you absolutely need those functions to complete and it's mission critical to your app?

Retrying those "permanent" failures becomes more difficult.

Also, in order to monitor these failures, we have to instrument the code to log out retries, number of attempts, etc. Again, that's doable, but it means more code to implement.
And unless you have something custom setup like a custom counter using statsd, Splunk, etc. to instrument and monitor the failures in some dashboard, you're probably going to just be logging the failures. And that means coming through logs to find the failures, or maybe setting up a CloudWatch query to watch for these failures.

Maybe a queue would make some of this simpler though? With less custom work you have to do on your end?

Depending on which queue solution you use, you usually get the following out of the box:

configurable retries
Dead letter queue ("DLQ")
queue monitoring/observability

Instead of adding custom retry code you usually get configurable "automatic" retries out of the box with a task queue solution.
In a scenario in which you get continual failures, that task can be automatically moved to a DLQ, where it will sit until you act on it. But will help you avoid an infinite retry loop.

Imagine you have some asynchronous code where a user signs up to your app, your code sends a welcome email out, creates credentials for them, and kicks off some marketing sequence. Maybe not super processing-intensive, but something you decide you don't wait to wait for (maybe your email provider is a bit slow, for example).
What if you pushed some bad processing code (i.e. your email-send code had a bug in it)? With a queue solutoin, you could make a fix, and then retry all these with the fixed code using the items from the DLQ.

And you'll also get observability into not just the DLQ - you want to know when code just won't successfully execute - but generally your others tasks too. Things like how many are currently in the queue, how many are processing, completed, etc.

The main point here is that you get these things out of the box (again most solutions should have these features but always make sure to check).

Infrastructure setup required for queue if not already setup

If you don't have the infrastructure already setup for a task queue, that is "overhead" work you or someone on your team will have to take care of. And obviously with more infrastructure comes more cost, so that's something to factor when you're looking at pricing/billing.

If you're building out a MVP, or can live with some code execution failures and less observability into the execution of that code, maybe the infrastructure setup is not worth it for you.
If you go with just not waiting for Promise resolution, the good thing is that solution is just application code. No queue setup, worker setup, etc.

A note on Lambdas

It's worth pointing out that if you're using AWS Lambdas and you don't await or .then() the Promise, you run the risk of that code "hijacking" and finishing its resolution within another Lambda request. I'm not an expert on Lambdas but I've personally seen this happen. A single Lambda was executing two different requests, with the part of one request that wasn't await'ed finishing in that Lambda run.
So the above discussion on Promises needs to be weighed against Lambda nuances.

Summary

I've gone through every consideration I can think of when determining if you should use a task queue or just skip Promise resolution and continue code execution.
But to end with a pseudo decision matrix for when you'd likely use which:

If processing (like image processing) is going to take several seconds or minutes, you should probably use a queue. It's likely too processing intensive for the server and you might end up with ancillary performance issues even though you're skipping resolution and continuing to the next bit of code.
If the task is not mission-critical and not processing intensive, and you can deal with some failures here and there, not waiting for Promise resolution is probably someFnThatReturnsPromise
- The same goes for if you can live with continual failures (in the case of a programming bug related to the task)
If the task is mission-critical, even if it's not processing intensive, you should probably use a queue so you get observability, retries, and a DLQ (which again is really useful in case you had a programming bug)
If infrastructure setup is too much work for you, even given the above considerations, just don't wait for Promise resolution and don't use a queue
- This might seem obvious but if you either can't setup the queue infrastructure or it's too much work, you're not going to have a queue anyways so can't use that solution.
- If given your non-functional requirements and technical considerations you determine a task queue is right for your application though, I'd recommend biting the bullet and setting up the infrastructure.

The ability to work with asynchronous code in Node and JavaScript is great and obviously a core part of the language, but it can bring up some confusions too. Hopefully this discussion and explanation of the differences give you more of a nuanced understanding of the differences between the two approaches and helps you decide when to use which.

ExpressJS antipattern: making everything middleware

Corey Cleary — Wed, 20 Jul 2022 00:00:00 +0000

Something I see in many ExpressJS APIs is the overuse / incorrect use of middleware. Sometimes to the extent that almost everything is middleware.

What this usually ends up looking like is this:

const router = express.Router()

const getCustomerData = async (req, res, next) => {
  try {
    const customerId = req.body.customerId

    const customerDetails = await fetchUserDetails(customerId)

    res.locals.transactionHistory = await fetchCustomerTransactionHistory(customerDetails)

    next()

    return
  } catch (err) {
    next(error)

    return
  }
}

const processOrder = async (req, res, next) => {
  try {
    const customerDiscount = await calculateDiscountFromCustomerTransactionHistory(res.locals.transactionHistory)

    let recalculatedOrderTotal

    if (customerDiscount) {
      recalculatedOrderTotal = req.body.orderTotal - (req.body.orderTotal * customerDiscount)
    }

    const itemsAreInStock = await doubleCheckStock(req.body.orderItems)

    if (!itemsAreInStock) {
      return res.send('Item(s) out of stock')
    }

    await insertOrder(recalculatedOrderTotal)
    await chargeCustomerPayment(recalculatedOrderTotal || orderTotal, req.body.paymentDetails)

    next()

    return
  } catch (err) {
    next(error)

    return
  }
}

const sendConfirmationEmailToCustomer = async (req, res, next) => {
  try {
    await dispatchConfirmationEmailJob(req.body.customerId, req.body.orderItems)

    res.send('Order complete')

    return
  } catch (err) {
    return
  }
}

router.post('/order', getCustomerData, processOrder, sendConfirmationEmailToCustomer)

The "middleware" here is anything that depends on the ExpressJS req/res/next context. You can see that they are also chained where the route is defined:

router.post('/order', getCustomerData, processOrder, sendConfirmationEmailToCustomer)

Note: your controller will also usually depend on the Express context but it won't act like middleware in that it's chained from one call to the next in the route definition.
The controller will usually have a single entrypoint - so one controller function per route. This isn't a hard-set rule but is generally a best practice.

You usually see middleware in ExpressJS using app.use(someMiddleware) to register the chain of middleware in order. And while this is not an example of that, I'd argue it's still coded essentially as middleware because of the hard dependency on the ExpressJS context.
It's just in a different place in the code - in the route definition instead of the index.js or app.js part of your code where you see the app.use(someMiddleware) setup.

What is this code doing? A few things:

getCustomerData()
- fetches user details (likely from a database)
- fetches the customer transaction history (also likely from a database)
processOrder()
- calculates any potential discount for the user
- checks the item(s) are in stock
- inserts the item order into the database
- charges the customer's credit card or other form of payment
sendConfirmationEmailToCustomer()
- send the user a confirmation email with their order details

What makes this a problem?

The problem is not really what the code is doing but how, for the following reasons:

These three functions now depend on the request context. If you want to reuse them / use them in multiple places, every function that calls this must have req, res, and next (the Express "context").
- You also have to assume sequence of calls and next(), so even though they may be individual functions, they aren't reusable.
If you have to pass one value from one middleware function to the next, you have to use res.locals to (when we could just return it and pass it via a function argument).
It makes writing automated tests more difficult.

Request context dependency

One of the biggest issues in my opinion is that these functions are not reusable. Because the function definition is now coupled via its arguments to req, res, and next, and those are coupled to ExpressJS, you can't call them anywhere else in your code.
Unless it's somewhere you have the ExpressJS context (more on this a bit further down).

If these were just "regular" functions, the context would not matter. That is, if you could just pass in "agnostic" values/objects/arrays etc, then you could reuse them elsewhere in your code.
Sure, the expected types and expected arguments matter but you can reuse a functions in ways that make sense for your application.
You can call your utility functions in your service layer code or your database code, for example.
And obviously the business logic still matters, i.e. you're not going to arbitrarily call functions.
Similarly, you're not going to call controller functions from within another controller either.

But by not being totally coupled to the core Express objects/functions this gets us a long ways towards reusability. We should always strive for loose coupling when designing our software.

You might be able to "reuse" that middleware elsewhere but only as middleware and even then it might not be reusable.
Consider a function that is supposed to end the request by calling res.send(response). You can't really reuse that (without changing the function definition), because it ends the request so you couldn't call it in the middle of your chain.
And if you need to pass value(s) from one middleware function to the next, this pesudo-middleware reusability becomes even more difficult, as explained in the next section.

Passing values from one function to the next

In our code above, getCustomerData() calls fetchCustomerTransactionHistory() and then needs to pass it to the next middleware function, processOrder(). Because these functions get called in a chain, we need some way of passing that value to processOrder(), since we have no intermediary variable to store the result in.

You can do that thru res.locals.transactionHistory = transactionHistory or by attaching a new property to the res object arbitrarily, like res.transactionHistory = transactionHistory.
Any property added to res.locals is only available for the lifecycle of the request, so when the request completes you can't access it again.

This is much messier than if we could just call getCustomerData(), store the result in a variable customerData or whatever and then pass that to processOrder().

Also, this further reinforces that the order of middleware function calls matters when going about it this way. Because one function willy rely on previous res.locals being set, the order of calls must stay the same.
And if you want to change the value that gets passed, you inevitably have to change the implementation of more than one function, you can't just change the one function.

While res.locals is supported by ExpressJS, and you can of course set new properties on objects if you go the custom property on res route, I don't recommend this unless it's something you absoluetly need to do as it can make troubleshooting mroe difficult.
But anyways, it's best to avoid this altogether and have your utility/business/DB logic in non-middleware code.

Makes writing automated tests more difficult

In order to write tests for this type of code, we now either need to stub req and res or we need to test this end-to-end using something like supertest.
Endpoint/end-to-end tests are good to have, but these functions we want to test are individual/modular (or at least, should be modular/resuable) and should be able to be tested more as units.
We shouldn't have to test them by spinning up a mock server, or by manually stubbing req and res - that's unnecessary complexity and work.
And stubs for request and response objects can require more maintenance, tight coupling, etc.
Not that stubs are bad - quite the opposite - and in the case of the functions above we'd likely want to stub some of the database and async calls.
But in this case we don't want to have to write them for req/res. They would have to be more like mocks, where we define the next() function and make assertions that it was called, stub the res.send() function, which is an implementation we don't care about, etc.

Instead if we could just break these pesudo-middlewares into resuable funtions without the ExpressJS context, we could test them by passing expected parameters to the functions which makes the test setup much easier.

What middleware is really for

This topic could be a few blogposts by itself, but to get the general idea across middleware should be used for things that are common to all HTTP requests but don't contain business logic, and that need to be processed before everything else.

Things like:

Authorization/authentication
Caching
Session data
CORS
HTTP request logging (like morgan)

All of the above are their own category of API concern, separate conceptually from code that is concerned with fetching data from the database, sending out a user registration email, etc.
Authorization and authentication need to happen before a user or client application access a service. That's something that is common to every (or most) requests.
Caching, that is generally common to most requests, and is a utility that is a separate concern from business or view logic.
Same with session data, same with CORS, same with request logging.

While there are always exceptions to any rule, middleware almost always should not contain the core of your code that handles business logic, that handles code specific to your REST API, that is "further down" the chain of function calls.

I like to think of business logic as the more "pure" form of logic. It is logic that shouldn't care about validating the request or handling anything framework-specific. It just handles algorithms/rules for processing data, storing of data, fetching data, formatting that data, etc. These rules are usually determined by business requirements.

For example, if you had an API that returned how many users had been registered on your platform within the last X amount of days, the business logic here would be querying the database and doing any formatting of that data before it returned it to the controller, which returns the HTTP response.
That logic won't handle caching or auth or session data. The middleware takes care of that.

How to fix it

If we make these "normal" functions rather than "middleware" functions coupled to ExpressJS, this is what they could look like. Of course you could refactor it further but this is the general idea:

const getCustomerData = async (customerId) => {
  const customerDetails = await fetchUserDetails(customerId)

  return fetchCustomerTransactionHistory(customerDetails)
}

const processOrder = async (orderTotal, orderItems, paymentDetails, transactionHistory) => {
  const customerDiscount = await calculateDiscountFromCustomerTransactionHistory(transactionHistory)

  let recalculatedOrderTotal

  if (customerDiscount) {
    recalculatedOrderTotal = orderTotal - (orderTotal * customerDiscount)
  }

  const itemsAreInStock = await doubleCheckStock(orderItems)

  if (!itemsAreInStock) {
    return null
  }

  await insertOrder(orderTotal, orderItems)
  return chargeCustomerPayment(recalculatedOrderTotal || orderTotal, paymentDetails)
}

const sendConfirmationEmailToCustomer = (customerId, orderItems) => {
  return dispatchConfirmationEmailJob(customerId, orderItems)
}

Note: sendConfirmationEmailToCustomer() is just a wrapper function basically. We could just call dispatchConfirmationEmailJob() directly now, but I'm leaving it in to demonstrate the before and after.

Now we have functions that are more reusable, not coupled to ExpressJS, and require less test setup to write tests for.

You might call these functions in your controller like so:

// Controller
const createOrder = async (req, res, next) => {
  const {customerId, orderTotal, orderItems, paymentDetails} = req.body

  try {
    const customerData = await getCustomerData(customerId)
    await processOrder(orderTotal, orderItems, paymentDetails, customerData)
    await sendConfirmationEmailToCustomer(customerId, orderItems)

    res.sendStatus(201)

    return
  } catch (err) {
    res.sendStatus(500) // or however you want to handle it

    return
  }
}

// Route
router.post('/order', createOrder)

You could of course use these individual functions elsewhere in your code though, now that they are reusable!

Simplifying local dev setup with Docker Compose

Corey Cleary — Tue, 23 Jun 2020 00:00:00 +0000

If you've ever had to deal with setting up a Node.js project in which you had to install a bunch of things - like MySQL/Postgres, Redis, etc. - and then run some setup scripts just to be able to get the project running locally on your machine...

...then you've likely experienced the pain of losing half a day - at least - to solely just getting set up.

This is especially frustrating and anxiety-inducing if you're new to the team and want to start contributing right away, not waste time in the maze of steps you have to run, or waste time having to ask the team every 5 minutes how to get over the next install hurdle.

What's worse is, as the project evolves, you might need to install more things, you might have more complex setup scripts, and (worst of all IMO) documentation for that setup might become out of date.

Rather than having to install a bunch of things - or figure out what you need to install in the first place in the case of bad documentation - there's a much easier way that can get you up and running in as little as one or two commands.

Enter Docker Compose

Docker Compose gives us the ability to define install dependencies - like databases and other software - and run them within containers that your "main" code can interact with.

In order to best explain how to use Compose - and how to convert an existing project with local install steps, scripts, etc - I'll use an example of a demo repo I wrote awhile back (that accompanied this post on designing reliable queues).

When I originally built that project, it was using "the old way", without Compose.

But I recently re-wrote it to use Compose for creating Redis and Postgres containers, and to be able to run the tests against those containers (using Compose is also really good for having local test databases).

New world and old world

First, let's look at how the project was setup using "the old way":

first install Homebrew
then install Postgres
then create a "root" database
then define the schema
then run a script to install Redis
then run a script to start Postgres
then run a script to start Redis

That's a lot of steps...

Now, let's take a look at the steps involved using Docker Compose: docker-compose up

...and that's it.

How were we able to accomplish this?

Let's look at how I converted this project over to using Compose.

Postgres

Instead of having to install Postgres (and Homebrew, if you didn't already have it installed), and then define our database and schema, using Compose that becomes:

version: '3.7'
services:
  db_queue:
    image: postgres:9.6.17
    container_name: db_queue
    environment:
      POSTGRES_DB: library
      POSTGRES_USER: root
      POSTGRES_PASSWORD: password
    volumes:
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
      - db-data:/var/lib/postgresql/data
    ports:
      - 5432:5432

volumes:
  db-data:

Note that the above is contained in the docker-compose.yml file in the root of our project.

Second note: you'll need to have Docker installed on your machine in order to use Docker and Docker Compose.

We define our "install dependencies" within the services section, in this case, Postgres.

Then we define the basic environment variables that Postgres needs to startup the database. In the old world, where we were creating the database from the command line via psql, here we just define it under POSTGRES_DB.

The service's volumes section uses an initialize script (more on this in a second) and defines a database volume that gets "mounted" alongside the container. And we define that volume name using the "root" volumes section, in this case using the name db-data.

The reason we do that is so that if we bring down the "stack" using docker-compose down, it won't clear the schema definitions + data stored in the database. Note, if we want to delete that information and bring it totally down, we can use the command docker-compose down -v, using the -v flag for "volume".

The init.sql (used to create the table schema as the container boots up) still needs to be created, but instead of you having to manually define the schema, the SQL script just gets leveraged by Compose instead. In other words, its automatic rather than manual, and removes a step for us.

And here's what that init.sql script looks like:

CREATE TABLE books (book_number int, isbn text)

Lastly, we map the container port to the host machine port (the host machine being your machine itself), so that you can access the container from your machine. That's done in the service's ports section.

Redis

For Redis, it's even simpler. In that same services section, we do:

redis_queue:
  image: redis:5.0.6
  container_name: redis_queue
  ports:
    - 6379:6379

Define the Docker Redis image to use, give the container a name, and map the ports. Simple.

Compared to the old world, where we had to run a script to wget to install Redis and build that code using make, then start Redis using a separate script, the Compose way is much easier.

Leveraging the Compose containers

Real quick, here's the entire docker-compose.yml file in its entirety:

version: '3.7'
services:
  redis_queue:
    image: redis:5.0.6
    container_name: redis_queue
    ports:
      - 6379:6379
  db_queue:
    image: postgres:9.6.17
    container_name: db_queue
    environment:
      POSTGRES_DB: library
      POSTGRES_USER: root
      POSTGRES_PASSWORD: password
    volumes:
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
      - db-data:/var/lib/postgresql/data
    ports:
      - 5432:5432

volumes:
  db-data:

Like I mentioned before, all we need to do to start the "stack" is to run docker-compose up, and Docker will use the Compose file and services defined therein to spin up the containers.

Because we have the container ports mapped to the local machine, we can run the unit/integration tests using npm test - nothing different we need to do.

You can also run the code against the containers, not just the tests. Simple.

Wrapping up

If you're continuously bumping up against problems running your project locally, strongly consider using Docker Compose for this instead.

It makes defining a local "stack" for local development a lot simpler and more headache-free then installing a bunch of stuff on your machine. And in this post we've really only scratched the surface of what you can do. It can make your developer life SO much easier.