DEV Community: Nate Anderson

Designing a Scalable Backend in Node.js

Nate Anderson — Wed, 14 Jun 2023 11:27:54 +0000

As a developer supporting a growing business, you’ll likely spend a lot of time thinking about scaling. We live in a wonderful time to be solving scale problems. Computers are so fast and capable that many of the development decisions you make when writing backend systems won’t noticeably impact performance (until your app reaches an absolutely massive scale). If this weren’t the case, we’d all be writing web servers in C.

This article will explore three scaling challenges in a fictional Node.js backend system, written as user stories. While this system is fictional, these issues closely mirror ones I have seen in my career, and they may also be familiar to you.

We’ll discuss useful tools for diagnosing the root causes of scaling issues, as well as techniques for alleviating bottlenecks and eliminating points of failure.

Let's get started!

About Our System With a Node.js Backend

For my examples, I’ll discuss a system that classifies user transactions to help with personal budgeting. Customers link a payment card, and their bank or credit issuer informs our system of their purchases. Our system is a web server with an API endpoint called by the participating banks’ systems (a “webhook”), and a set of API endpoints called by consumer frontends when they wish to view their categorized transactions. The banks send between 1 and 100 transactions per request, which we parse and process.

Our server calls a third-party web API to look up merchant details which help us classify transactions into categories like “groceries”, “dining out”, and “entertainment”. These classified transactions are then written to a highly normalized relational database. Users can see updated data within 24 hours of each transaction.

Our backend is a Node.js HTTP server running on a virtual host behind an Nginx reverse proxy. The server process is configured to automatically restart if it terminates.

Without further ado, let's explore our first challenge.

Challenge 1: Issues When Logging In As a User

Scenario: As a user, I attempt to log into the mobile app between 3 and 4 pm on weekdays. I expect to see my transactions quickly, but logging in takes a very long time and often fails.

Intermittent failure is never fun to deal with; in this case, we benefit from a consistent, recurring time window.

One approach might be to jump straight to our production logs to see what the application is logging during this time window, but a common problem with timeouts is that we’ve included log statements that execute on completion of the problematic instruction. It can’t hurt to check logs, but when requests are very slow and frequently time out, it’s likely a resource is overloaded. This is why it’s a good idea to monitor key resources like the application server’s CPU and memory consumption, and key database metrics like CPU, memory, and IOPs (I/O operations per second).

In this case, we are greeted with a moderate increase in host CPU and memory consumption during the user’s reported window of 3 to 4 pm, and a large spike in write IOPs and CPU on our database instance, as well as a small spike in memory consumption on the database host.

Side-note: Check out the Host Monitoring section of our post Monitor the Health of Your Node.js Application to see how you can use AppSignal to track some of these host metrics, including memory and CPU usage.

It's fairly obvious our application's compute resources are not under enough load to cause these time outs and slow responses. However, the spiking database write IOPs and CPU usage point to a high volume of inserts or updates slowing down the database. The fact that read IOPs remain low confirms this. There is also some evidence that the high amount of write activity is slightly slowing down reads elsewhere in the app.

Where's the Point of Failure?

So what happens when a user logs in, and how could this spike in write activity interfere with that?

First, the user's record is looked up using the email they signed in with. This should be a read based on a unique index, and there is little evidence that reads are affected significantly.

Second, the user's password is hashed and compared to the database record, using something like bcrypt. This is certainly CPU intensive, but we know the application host is not under strain here; again, this is unlikely to cause login to fail, even when the database server is under load.

Third, a session token is created that will be included in requests from the app. This is stored in a session table in the same relational database. The user's last-seen timestamp is also updated in the users table. The steps are suspect points of failure for these possible reasons:

The large number of database write IOPs are slowing down all writes to the database. This is plausible, but on its own, unlikely to cause such slow queries that logins fail.
The queries spiking the write IOPs and CPU are locking the resources used by the login, or even causing deadlocks. The somewhat inconsistent nature of the user's reported issue indicates this is a likely cause; the same queries may not always cause deadlocks or lock resources used by any given user. It largely depends on the content of the transactions themselves.

Investigating Further

To confirm or invalidate our theory, we should investigate the cause of the write IOPs and database CPU spike between 3 and 4 pm. Again, we could go to our logs for this, as ideally we are logging basic information about every request to our server.

But since we already know the database is under load at this time, and we are going to end up evaluating queries anyway, let's hope we have slow query monitoring enabled either natively in our RDBMS or inside our application performance monitoring tool and check there first.

In this case, our APM allows us to sort queries by impact, and the three slowest queries are:

Query	Impact
A batch insert into the `transaction` table which occurs when the payment card provider calls our webhook with new transaction data	78.1%
An update that touches the last transaction timestamp in the `user` table and occurs after one or more transactions are inserted for a user	8.6%
The update which touches the last login timestamp in the `user` table during login	8.1%

Interestingly, the first and second most impactful queries occur within a single transaction. When the webhook is called, the transaction data content is bulk-inserted, and the last-transaction timestamp in the user table is updated. This transaction takes a long time to complete and holds the row in the user table hostage until it completes.

If the webhook is called several times for a single user, we may experience a deadlock. This explains our login delays and failures: the user's row is tied up!

But why are these inserts spiking between 3 and 4 pm on weekdays? It's likely that one of our partner banks is simply calling our webhook via a cronjob that runs at 3 pm on weekdays.

Handling Bursty Workloads

In this case, the spikes in database load originate outside our system. We could try asking the partner nicely to spread out their webhook invocations (check your contract, it might be in there!). But if you're a small startup, asking a bank to change its systems and help spread its load is not likely to bear fruit.

Instead, we can use a queue to smooth out spikes and control throughput. When properly resourced, message queues like Amazon SQS, ZeroMQ, RabbitMQ, and others can hold millions of messages in flight. The message queues will let your database work at a more reasonable pace by delaying or spacing out messages. Here's a tutorial for one approach using SQS.

If we imagine the webhooks as a hose we cannot control and our application server as a water wheel, think of the queue as an extremely large bucket with an adjustable spout. Rather than aiming the hose directly at the wheel and overloading it, we aim the hose into the bucket and control the rate at which data reaches our application. This way, we can control the size of batches we bulk insert and the number of those batches we insert in a given time.

Here's an example architecture:

Lessons Learned

Observability tools like database monitoring and APM can often help us spot production issues faster than simply reading logs, but good logging habits are key to tying patterns in APM to code points.
Not everything has to be synchronous (some might argue most things shouldn't be!). Using highly available tools like queues and streams can help smooth out load over time and prevent unpredictable bouts of unavailability.

Challenge 2: Missing Transactions in Our Node.js App

Scenario: As a user, when I log into the app, I expect to see all of my transactions from more than 24 hours ago, but some of my recent (and older) transactions are missing.

Missing data is a significant problem for our app. If we fail to report spending, users may incorrectly believe they are within their budget and spend more than they should.

Let's say a user reports a specific transaction at 3:15 on 2023-01-12 for -€12.00 that appears on their statement, and we can confirm by querying our database that we have no record of it. Because all of our transaction data comes through our bank partners, those partners are the source of truth. If it turns out the partner missed this transaction, it's a bit of a mixed bag — there isn't necessarily any work for us to do to fix the issue, but we also don't have a solution to offer the customer.

This particular customer's bank has a synchronous HTTP API we can call to fetch the user's transactions on demand. Calling the API reveals that the bank does, in fact, know about the transaction. We should confirm they sent us a webhook by looking at our request logs.

The example below uses AWS CloudWatch syntax to query logs generated by logging middleware in our application code:

fields @timestamp, @message
| filter path = "/hook/transaction"
| filter method = "POST"
| filter account.number = 1940117
| filter amount = -12.00
| filter @timestamp > "2023-01-12 03:00:00" AND @timestamp < "2023-01-13 04:00:00"
| sort @timestamp desc
| limit 20

This query yields zero results. We have a 30-minute message delivery SLA with this partner, so we might conclude they just didn't send the message at all.

But before we cast any blame and make an incident out of this, we should take a step back and see what our system was doing at this time. If we remove the request-specific query filters and just look at all the logs, we may notice trends like an upstream error, dropped database connection, or other internal errors.

Using this broader query, we find quite a few messages to sift through:

fields @timestamp, @message
| filter @timestamp > "2023-01-12 03:00:00" AND @timestamp < "2023-01-13 04:00:00"
| sort @timestamp desc
| limit 20

But two stand out in particular. The first is a stack trace indicating the server crashed due to an unhandled exception at 3:17, after which there are no logs for 20 seconds while the server restarts, then the server's startup message log. This means the webhook messages were dropped outright.

For more information on stack traces and how you can monitor them in AppSignal, see An Introduction to Async Stack Traces in Node.js.

Evaluating our logs further, we receive dozens of webhook requests per second at certain points during this window (remember, this is when the provider sends the bulk of their webhooks). Each request contains as many as 100 transactions, meaning that a 20-second outage could have caused us to drop hundreds, even thousands, of transaction records.

Our Nginx reverse proxy is configured for basic request logging and confirms partner requests were dropped during the outage. Luckily, we're able to call the banks' APIs to recover missing data, but we need better uptime and resiliency to maintain our guarantees to our customers.

Restarting: Not Always Enough

Even once we've fixed the culprit uncaught exception that caused the app to crash (and even if we properly implement error handling middleware to prevent similar issues), we can never be 100% sure our process won't crash — we wrote the program, after all!

This is why it's important to have redundancy, as well as automatic restarts. Process managers often support running multiple instances of a program (like PM2's "cluster" mode), and in certain cases, that may be all you need. But the kind of load bursts our application undergoes merits something more sophisticated. This is a great time to evaluate containerization and clustering with a tool like Kubernetes or AWS Elastic Container Service. A container manager will manage multiple instances of our program with dedicated resources, automatic restarts, and a load balancer.

This way, even if one of our processes crashes and takes a long time to restart, the load balancer will simply direct traffic to the healthy nodes until the crashed node is replaced. Even in the unexpected event of a process exit, we will only experience true downtime if all three instances crash.

The Benefits of Queueing

This is also a great time to reiterate the benefits of queueing. Imagine if, instead of consuming data directly from webhooks, we simply read the request data and enqueue it on a highly reliable queue. It's relatively easy to make software like this very reliable; it isn't doing anything complex, is easy to write tests for and to load tests in a deployed environment, and it can use serverless technologies to scale quickly.

Once the messages are on our queue, they will stay there until "acked" (short for acknowledged). If our database writing process crashes in the middle of handling some messages, those messages aren't lost to the ether — they remain in the queue until someone tells the queue to release them with an ack. And if non-fatal errors prevent the messages from being processed, we can place them on a dead-letter queue to be replayed later (quickly remedying the data missing from our application without coding up a new solution to read from the banks' APIs). Queue replayability is a fantastic failsafe when dealing with stream-like data and is relatively simple to implement — see this example with AWS SQS.

Lessons Learned

We shouldn't find out about downtime by noticing missing data. Configure an APM like AppSignal to monitor your application's uptime and address downtime before it becomes a bigger problem.
Restarting after a crash often means downtime unless you have redundancy. Clustering is a great way to achieve reliable redundancy.
Replayable queues can help you go back in time to catch streamed data your application missed the first time.

Challenge 3: Transactions Taking A Long Time to Load

Scenario: As a user, I log into the app and expect my transactions to display right away, but they take a long time to load. Basic profile information loads quickly, but transactions do not.

This one isn't surprising, right? The scale of transaction data in our system makes it a prime candidate for bottlenecking requests — look at the size of our tables:

Table	Rows	Size on Disk
`user`	129K	258MB
`session`	241K	120MB
`transaction`	39M	46.8GB

Transactions are both the largest and most plentiful records in our relational database — averaging 302 transactions per user at 1200 bytes per transaction, we end up with a pretty sizeable table.

Naturally, the first thing we should do when a query runs slowly is to ensure we take proper advantage of indexing. It is likely that multiple queries on this table can be optimized, but let's just look at the one that runs when a user initially views their transactions in the mobile app (again, our APM can help us identify this by monitoring for slow RDBMS queries).

This query loads up to 50 settled transactions selected in the last 30 days for the logged-in user. It might look something like this:

SELECT
    user_id,
    card_id,
    transaction_id,
    amount,
    merchant_name
FROM transaction
WHERE
    user_id = 155
    AND card_id IN (210, 340)
    AND settled_ts > (NOW() - INTERVAL '30 day')
LIMIT 50;

This query runs a lot, so we should have an index on the table covering the clauses in the filter expression — user_id and settled_ts. This is easy enough to verify:

Index Name	Index Type	Columns
`transaction_pkey`	BTREE	transaction_id
`user_card_ts_idx`	BTREE	user_id, card_id, settled_ts
`fraud_alert_idx`	BTREE	merchant_name, amount

We can use an EXPLAIN or EXPLAIN ANALYZE query to confirm that the index is being used. With PostgreSQL, we would look for a line similar to Index Scan using user_ts_idx on transaction. Index Only Scan and Bitmap Index Scan are also good news in this output.

When a query that uses indices correctly continues to be slow, it can be for several reasons:

The index columns may have insufficient cardinality relative to the table size.
The query may result in a lot of inefficient table access.
Your index could be too large to fit into memory.

For the first two scenarios, you can try a handful of diagnoses and solutions.

But in our case, the third issue is the most likely. Our table is rather large, our index is composite, and our database server is not particularly beefy. With only 1GB of memory and lots of concurrent usage, the database server likely can't keep the entire user_card_ts_idx in memory at all times and must page the disk to traverse the index.

Disclaimer: I am not an expert in relational database indexing and query optimizations, so I will avoid going into too much detail on ways to internally reduce the cost of your queries. I can say that BTREE indices are great when used appropriately, but large tables often demand more than just a new or modified index when queries start to slow down. When indices do meet their match, you still have options to restore performance.

Table Partitioning

When your index becomes so large that it:

Slows down inserts
Can't fit into memory without paging to disk
Isn't selective enough to provide the needed performance, relative to the number of tuples in your table

Then you can partition your table (effectively, divide it into smaller tables) to gain a couple of benefits:

Smaller tables to query. An effectively partitioned table can chop down the size of the table you are actually querying by orders of magnitude. When your access patterns are well-defined, and you partition your table along boundaries that queries are unlikely to cross, you can significantly reduce query times.
Smaller indices speed up writes. Smaller tables mean smaller indices, and when your writes follow the same patterns as your reads, you may see similar (or even greater) performance boosts to queries that write to the table. Faster writes may also reduce your risk of deadlocks.

Upgrading your Database Server

I realize the example of an index overwhelming a database server with 1GB of memory is a bit comical. It's pretty rare, in my experience, for production systems with 6-digit user counts to rely on a database host this small for many reasons, not just index sizes. But I have seen much larger database boxes slowed down by much larger indices.

I don't recommend jumping to this solution straight away, because it's sort of a cop-out. However, if you have a large table and index that you know will grow at a rate of 10% per year, and can double the memory available to your database host, you can often buy yourself a few years of performance for a few dollars per month. However, if your table is growing rapidly and you're trying to keep costs low, it's better to save this option for last.

`INCLUDE` Key Columns in your Index

It may be the case that you only need one or two columns not in your index. In this case, just INCLUDE those columns in your index to prevent traversing the
potentially long linked lists of nodes in your B-Tree.

Consider A Different Data Store

Just like upgrading your database server, this can be a cop-out solution — often an even more expensive one due to the required development time. However, if your slow query is on a table that will continue to grow rapidly and your access pattern demands a fat index, you might consider a NoSQL database. Horizontal scaling offers a more linear cost schedule, and the high levels of abstraction and consistency offered by relational databases often come at the cost of performance. If we are willing to move the abstraction to our application code, we can reap significant benefits.

Also, if your business's needs change rapidly or often, you may find it difficult to re-index the database and adjust your queries fast enough to keep up. Consider adding a search engine like Elasticsearch as a secondary datastore to increase lookup speed, then look up the actual data in your RDBMS by primary key only for efficiency.

Wrapping Up

In this post, we explored three scaling challenges and their solutions in a Node.js application.

Every business scales differently, and every codebase demands its own responses to that change in scale. If there were a one-size-fits-all approach to building scalable backends, we might all be out of a job.

It's important to remember that scaling is a challenge and a blessing — too many users is a good problem to have!

The best solution to your problem depends on your specific circumstances. Hopefully this post has helped you to consider and identify the best solutions to different scaling problems.

Happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Node.js Architecture Pitfalls to Avoid

Nate Anderson — Wed, 30 Nov 2022 08:42:30 +0000

Architecting a maintainable, readable, and reliable codebase is important not only for the people who will work on the code after you, but also for your own sanity.

I’ve worked on dozens of production Node.js backend services and will share my experience of architecture pitfalls to avoid when building yours.

The examples in this article are tailored to JavaScript and TypeScript, and some of the problems (like dependency management) require more attention in Node.js projects than in other languages. However, many of these principles apply to other languages too.

Let's get going!

Why Worry About Node.js Architecture?

When I started my first mid-level developer job, I had left a company that used frameworks for every project and joined a department that wrote every backend service from scratch on Node.js.

I think many devs have experienced this moment. Juniors often work in established codebases or on smaller projects where the size and simplicity keep things under control.

But a day may come when you'll need to start a new project; the decisions you make at the beginning might affect your brand's reliability, revenue, and customer experiences for years to come. Thoughtful coding and architecture in a project's early days often impact the development experience more than specific library or framework choices.

Globals in JavaScript: You Were Warned for a Reason!

If you've ever taken a formal computer science course, your professor likely warned you to avoid using global variables but may not have explained why.

Globals are dangerous for several reasons. Let's say you import a function that depends on a global variable from one module to another, for example. That global variable is like an invisible argument to your function that the caller has no knowledge of or control over.

Worse yet, if a program mutates that global variable, you must consider its state and order of operations whenever you access the variable. I find it is almost always safer to pass a variable to a function or class constructor directly than to depend on a global.

The Problem with Globals: An Example

Arguments are more explicit and self-documenting than globals. Callers of a function will better understand the data's importance in a variable because it's right in the method signature with a relevant name.

Consider a function that alerts a system administrator to some event:

const adminEmail = "admin@example.com";

// notify an admin about a low-priority issue
export function notifyAdmin(subject, message) {
  emailClient.send(
    { subject, message, recipient: adminEmail },
    (error, result) => {
      if (error) {
        return console.error(error);
      }
      console.log("Email sent successfully!");
    }
  );
}

// ...

// critical error, needs attention now. wake up the owner!
export function wakeOwnerUp(subject, message) {
  pagerClient.wakeUp(
    {
      user: adminEmail,
      priority: "high",
      respectQuietHours: false,
    },
    (error, result) => console.log({ error, result })
  );
}

This is, admittedly, a dramatic and cheesy example. But notice how any code that calls notifyAdmin has no control over who is contacted.

Sure, it's easy enough to change the value of adminEmail and redeploy the code, but because that global is used in at least one other function, this change can have cascading effects. What if the owner and the admin are no longer the same person? We don't want to be the ones responsible for waking our boss up at 3 AM when it's some poor developer-on-duty's job to have a look.

Furthermore, what if we need to send an email to more than one address? We can't simply change adminEmail to an array and expect everything to work.

The Fix: Using Arguments

If we make notifyAdmin into a more generic notifyEmail function, we can call it many times with as many different addresses as we like.

And wakeOwnerUp and notifyEmail are no longer tightly coupled — they can interact with different people. This change also makes it easier to adjust our notification settings without redeploying the entire project; the admin and owner emails can come from a database instead of a hard-coded variable.

There are some more nuanced benefits to eschewing globals in favor of arguments. Global values are often runtime configuration parameters that change depending on the environment. So you may need to test behavior when these values change to test your code thoroughly.

Let's say we're writing tests for the example above. We've mocked the emailClient and pagerClient and want to assert that the email address these clients will use matches the expected value in our test setup.

That's currently impossible without opening the source code file and copying the email address to a new variable in the test.

Set Up A Test That Goes to One Email

Wouldn't it be easier to test that the code just emails the address we ask it to? We have two options here:

Export the adminEmail variable from its module and allow tests (and other code) to change its value at runtime.
Make the email address an argument to the function calls.

As previously mentioned, mutable globals require extreme care to avoid causing unpredictable and unwanted results.
This is true in tests too — especially if your tests run in parallel, as many modern test runners will encourage.

Imagine test A, a unit test, changes the value of adminEmail to assert that an email is sent to the correct address. But test B, an integration test of a function that calls wakeOwnerUp, asserts that a different user receives a pager call. If your tests run concurrently, you're likely to experience a flap, a test that passes inconsistently.

Instead, I would encourage you to go with option 2:

Define your expected outcome in the test setup
Pass a value into the function being tested
Assert that the result matches the expected value by inspecting the return value or mocking one of its dependencies

Another benefit of passing data as an argument is to reduce possible data races, like those that may occur when your program starts up.

For example, suppose your app connects to a database
using a global connection object variable, and your program sets up this connection using callbacks or asynchronous
code. Your app may reach a "ready" state and start accepting requests before it even attempts to connect to its database. For example:

import { connect } from "myDatabase";
import { router } from "pronto";

var DB;

function main() {
  const app = router();

  app.get("/", (request, response) => {
    DB.query("SELECT * FROM users", (error, rows) => {
      if (error) {
        response.error(error);
      }
      return response.json(rows);
    });
  });

  connect("localhost:3305", (error, connection) => {
    if (error) {
      console.log(`Failed to connect to DB: error ${error}`);
      process.exit(1);
    }
    DB = connection;
  });

  app.listen();
}

If the process of connecting to your database is slow (which isn't all that unlikely), the asynchronous nature of JavaScript means your app.get and app.listen calls may be reached well before your DB variable is assigned. Your service is online but not usable.

Environment Variables in JavaScript: Socially Acceptable Globals

Environment variables are often an excellent way to configure your application. They're nearly universal, supported by every major OS and programming language.

They're flexible, too; you can:

configure them system-wide or per-user in the shell
specify them at program launch by prefixing the command
configure them easily in dockerfiles, bash scripts, and .env files

But environment variables can be dangerous if you aren't careful about how you access them. If you think about it from your program's perspective, an environment variable is just a global variable owned by the operating system instead of the program.

Just like the first global variable example above, relying on an environment variable at the wrong level of abstraction makes your code difficult to test and reason about.

export function notifyAdmin(subject, message) {
  const address = process.env.ADMIN_EMAIL;
  emailClient.send(
    { subject, message, recipient: address },
    (error, result) => {
      if (error) {
        return console.error(error);
      }
      console.log("Email sent successfully!");
    }
  );
}

While the ways we can configure ADMIN_EMAIL as an environment variable are infinitely more flexible than a
hard-coded constant in the code, we are still susceptible to some familiar problems:

Reliance on global state makes the function's behavior unclear.
Changing environment variables affects all the code points that read that variable.
It's difficult to concurrently test that the code responds appropriately to different values of the environment variable.

But, unlike hard-coded globals, the moral of this section is not "don't use environment variables!". Instead, it's "be careful where you read environment variables."

Instead of reading from process.env throughout your code, define a configuration object that contains your environment variables under named keys. If you're using TypeScript, this even gives you some compile-time guarantees for code accessing your config variables.

Plus, if you ever need to read configuration values from a source other than your environment, none of the downstream code changes. Only the code that actually reads the values into memory changes.

A neat way to gather up all your configuration variables at program startup is to use an object and null coalescing operators to handle missing configuration values.

const config = {
  dbHost: process.env.DB_HOST ?? "localhost",
  dbPort: process.env.DB_PORT ?? 3306,
  dbUser: process.env.DB_USER ?? "web",
  dbPass: process.env.DB_PASS ?? "",
};

The details of the code that reads the configuration aren't super important. Obviously, there are cases where environment variables will change at runtime, requiring a more involved solution than the one presented here. The critical thing is to use this method to read configuration everywhere you can and avoid reading directly from the environment at runtime.

Make Your JavaScript Code Easy to Test with Environment Variables: An Example

Here's an example of how this approach makes your code more testable. Say we are testing a database migration tool and want to run tests in parallel against different databases. A function that relies directly on environment variables cannot do this.

import { connect } from "myDatabase";

async function migrate(migrations) {
  const { DB_HOST, DB_USER, DB_PORT, DB_PASS } = process.env;
  const db = connect(`${DB_USER}:${DB_PASS}@${DB_HOST}:${DB_PORT}`);
  for (const migration in migrations) {
    await db.exec(migration);
  }
}

Simply rewriting this function to take a configuration object as an argument instead of reading from the process.env global makes concurrent testing possible.

import { connect } from "myDatabase";

async function migrate(migrations, config) {
  const db = connect(
    `${config.DB_USER}:${config.DB_PASS}@${config.DB_HOST}:${config.DB_PORT}`
  );
  for (const migration in migrations) {
    await db.exec(migration);
  }
}

We can call this function as many times, in as many concurrent invocations, as we want, and each invocation will use the exact database connection details we provide. The tests can get these values anywhere they please — coupling to a specific environment is removed.

Request-scoped Dependencies: Not Global Enough?

In almost the opposite vein of the previous two pitfalls, a variable's scope can be too small. Certain resources are meant to be long-lived and shared between requests:

Database connections are often pooled by default and should be limited to one pool per process.
The number of files a process may have open at once can be limited by the OS. By leaving files open, you might run out of available file descriptors.

Let's look at a common example of this problem. Many backend web development tutorials either demonstrate assigning your database connection to a global variable or instantiating a connection on each request. I have discouraged both practices in this piece, so let's look at an alternative.

A request-scoped database connection might look something like this:

import { connect } from "myDatabase";
import { router } from "pronto";

const app = router();

const dbCreds = "user:pass@localhost:3305";

app.get("/users", async (request, response) => {
  const db = await connect(dbCreds);
  const users = db.query("SELECT id, first_name, last_name, email FROM user");
  res.json({ users });
});

This will likely work fine when you test it on your own machine, with only one active user. But what happens when the site goes live? If the connect function here returns a connection pool instead of a single connection, each request will result in multiple connections to the database, even though only one is being used.

It's worth noting that not all database connection libraries pool by default, so pay attention to the docs for the one you're using.

Suppose we define our routes inside a class or function instead. We can pass the database connection in once we're sure it's ready, and use a single database connection throughout our app, referencing it as needed.

This technique is called dependency injection, and I've written a rundown of using dependency injection in JavaScript and TypeScript. Here's an example:

function routes(app, db) {
  app.get("/users", async (request, response) => {
    const users = db.query("SELECT id, first_name, last_name, email FROM user");
    res.json({ users });
  });
}

async function main() {
  const db = await connect(dbCreds);
  const app = router();

  routes(app, db);
}

Encapsulating our routing logic into functions makes it easy to write routes without thinking about dependencies until it's time for the code to actually run. It also makes test setup easier, allowing you to test your routes with a mock router or database if you choose.

It's best to limit this pattern to resources designed to be long-lived. There are many times when request-scoped variables are completely appropriate. When deciding how to scope your dependency variables, you should consult library documentation for things like:

database connections
caching server connections
any other external stateful dependency

Node.js Modules: A Slippery Slope

The Node.js ecosystem has a reputation for overusing third-party modules, and it’s not entirely undeserved.

Simple, popular example projects that rank highly on Google often rely on Node modules with hundreds of megabytes spread across dozens of packages. Newcomers to the ecosystem are encouraged to npm install early and often.

Repo	Forks	Stars	node_modules Size	Vulnerabilities	Last Commit
Example Express app	1.5k	3.3k	123mb	45 high, 14 critical	5 years ago
Node.js Integration Tests Best Practices	120	2.3k	288mb	16 high, 2 critical	6 months ago
Bulletproof Node.js API	1.1k	4.8k	201mb	8 high, 5 critical	1 year ago

I don't mean to call out these repos; the authors generously provided examples and information that have helped thousands of people get started coding in Node.js.

And these are not unusual examples. I've seen Node.js projects with gigabytes of third-party dependencies. I do think these are good examples to point out, though — specifically "Bulletproof Node.js". Notice here, NPM identified over a dozen high and critical priority vulnerabilities in what is meant to be an example of best security practices. This repo has only been untouched for one year!

The Issue with Third-party Libraries for Node.js

So what's wrong with depending on lots of third-party libraries? I pointed out the disk space used by these example projects, but storage these days is one of the cheapest available resources. It's certainly cheaper than developer time. So am I encouraging you to use only the Node.js standard library and write every other piece of code you need by yourself?

Definitely not. There will always be areas of expertise where we need to borrow from other developers, or areas of our business mission that are critical enough to merit relying on highly-tested and well-known libraries.

What we should be aware of, however, are the types of dependencies we bring in. I'd like to propose a few metrics to examine before including a dependency in your project, and then will dispel a few metrics that I think get too much attention.

Metrics to Assess Third-party Libraries

Some good metrics to evaluate third-party packages include:

Test coverage: does the package publish a test coverage figure? Is that number similar to, or optimally greater than, that of your own project? Note that branch coverage and statement coverage are very different numbers. Be sure to compare like to like.
Subdependencies: does this package import the world to get the job done? Will you have to monitor for vulnerabilities in code you don't even consume?
Usage: does the package have many downloads on NPM or stars on Github? If not, it may not have been used thoroughly enough to identify edge cases and bugs that could affect your code. Note that this risk is mitigated somewhat if the package is very well-tested.
Usability: does this library solve multiple problems you will need to address?

For example, say you need to remove items from an array.

While you could write this code yourself, you could also
import a third-party library to do it. There's an NPM package called remove-array-items that will do the trick.

If we install a single package to solve this problem, and another for the next problem we encounter, our dependencies will quickly spiral out of control. Instead, we could turn to a utility library like lodash, which will solve hundreds of little problems like this, and add only one dependency to our package.json.

Metrics to Avoid

Some problematic metrics for evaluating third-party packages:

Date of last release: while it's tempting to assume that a package that hasn't been updated in a year is outdated or abandoned, this may not always be the case.

Express, one of the most downloaded backend libraries on NPM, frequently goes six months to a year without a release. In fact, Express 5.0 has been in alpha for eight years. But a library that has been in use as long (and by so many people) as Express is unlikely to surprise you.

Shininess: the JavaScript ecosystem has a reputation for producing an exciting new framework or backbone library every month. I highly recommend allowing these freshly-baked treats to cool off before you take a bite. While new tech is always exciting, you don't want your production app to be the reason a major bug is discovered.

I present these do and don't metrics not as a definitive rule set, but as a gentle guide towards producing more reliable software. Every decision has trade-offs: a larger library may have features you don't use and take up more disk space than a selection of single-purpose tools. There are many facets to consider.

I have seen many Node.js projects, even those under active development, fall prey to dozens of vulnerabilities in the worst case, and have their dependencies go badly out of date in the best case.

Letting your out-of-date dependencies pile up can keep you from responding quickly to changing business needs, or prevent you from upgrading your Node.js version quickly if your release is approaching end-of-life.

If you think of the code you write as your child, the code you install is your pet. It doesn't require the same degree of care, and it can make your life easy and fun at times, but you still have to clean up its messes.

Wrapping Up: Work Smarter, Not Harder

In this post, we touched on:

Why it's important to worry about Node.js architecture
Why arguments are preferable to globals
How environment variables can be useful

We also looked at issues around request-scoped dependencies, before sharing some metrics to assess third-party Node.js libraries.

If I were to draw a single thread that runs through this article, it would be "code thoughtfully". It's easy to get excited and rush into a new project. The thrill of getting something to work for the first time often outweighs the fear of its future cost. And we're developers — we're told to move fast and break things.

I simply argue that a few moments of thought at the start of a project can save days (or weeks) of pain once it's in motion.

Consider the testability, maintainability, and security of your code where it matters. These decisions will pay dividends down the road.

Happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Node.js 18 Release: What's New

Nate Anderson — Wed, 04 May 2022 11:41:04 +0000

The Node.js team announced the release of version 18 on April 19. Node.js 18 has some significant new features that Node developers should be aware of, including:

The upgrade of the V8 engine to version 10.1.
An experimental test runner module.
Most excitingly, experimental support for browser APIs like fetch and Streams.

In this article, we'll look at some of the major highlights from this release.

Long-term Support Release Status for Node.js 18

Node.js 18 is a long-term support (LTS) release, so it will continue to receive updates for 30 months. Because this is a new release, it is considered the current release, meaning development work on this version is ongoing, and we should expect regular updates.

After six months, in October 2022, it will become the "active LTS" release, because version 19 will be released. The active LTS release will continue to receive updates, which may be either bug fixes, security patches, or new features backported from the current release.

Once an LTS release switches to maintenance mode, typically at least two years after release, it will receive security patches and bugfixes almost exclusively.

V8 Engine Upgrade in Node.js 18

The V8 engine is the JavaScript runtime developed by Google for Chrome, and used by Node.js to execute JavaScript. Any changes to how we use the JavaScript language inside Node.js must come through the V8 engine.

V8's upgrade to 10.1 means that the Array methods findLast and findLastIndex are now available in Node.js. These methods return the last item or index of an item in an array that passes a filter function.

Array.findLast works just like Array.find, except it begins at the end of the array and traverses it in reverse order. Array.findLastIndex simply returns the item's index, rather than the item itself.

These methods both take a higher-order function as an argument and call this function on each element in the list, returning from the method call when the function returns true.

The 10.1 release of the V8 engine also includes minor performance improvements, which you can read about in the release notes on the V8 project blog.

Node.js 18 Has A Built-in Test Runner

Node.js has included the assert library since very early days, making it fairly easy to write simple tests without a third-party library. But good test suites grow quickly, and most projects today depend on a test runner library like Jest, Mocha, Jasmine, or Ava.

These libraries make it easier to organize tests into logical groups, visualize test outcomes, collect coverage data, and simplify setup and teardown.

The node:test module included in Node.js version 18 adds a lot of functionality for which you would previously need one of these modules.

The built-in test runner supports subtests, test skipping, limited test runs, and callback tests. Here are a few examples of these features in action:

import assert from "assert";
import test from "node:test";

Notice the "node:" prefix, distinguishing this module from any user-created packages named "test".

test("Concatenate user's full name", (t) => {
  const user = new User("John", "Doe");
  assert.strictEqual(user.fullName(), "John Doe");
});

The test import is a function used to define tests. The first argument is a string describing the test, and the second is the function containing the test logic. This API is roughly the same as Ava's test, and takes the place of both it and describe from Jest.

It covers the same ground as Jest's describe because it can also define test groups, with individual tests (or even test subgroups) defined by calling t.test within the callback function.

For example, we can create a test group for serialization of a class and individual tests for each type of supported serialization.

test("User class serialization", (t) => {
    const user = new User("John", "Doe");

    await t.test("Serialize to string", (t) => {
        const str = user.toString();
        assert.ok(str.includes("John"));
        assert.ok(str.includes("Doe"));
        ...
    });

    await t.test("Serialize to JSON", (t) => {
        const userJson = user.toJSON();
        assert.doesNotThrow(() => JSON.parse(userJson));
        ...
    });
});

Now, the string passed to test describes a test group, and each call to t.test defines an individual test. Note that t.test returns a promise, hence the awaits in the example.

While it's exciting to see these testing features embraced by the Node.js standard library, I think it's reasonable to expect most projects to use a third-party testing module still.

Many Node devs already have a favorite test library, and it's likely to be a while before the node:test core module supports all the features that third-party libraries offer.

New Browser-compatible APIs in Node.js 18

Commonly supported browser APIs like fetch and Streams are among the most significant additions to Node.js 18.

In the browser, fetch replaced the cumbersome XMLHttpRequest with a flexible, terse HTTP client. The Streams API allows us to perform incremental processing of data that's received from or sent to a network.

It's important to note that these APIs are still considered experimental in Node.js 18. It may be wise to avoid their use in mission-critical applications for the moment, as breaking API updates are not out of the question.

Node.js Now Supports Fetch API

Adding support for fetch to Node.js allows us to make HTTP requests concisely without:

Third-party libraries like axios or node-fetch (which exposes an API roughly equivalent to the new built-in global)
Relying on the more complicated http and https packages to make requests from Node projects.

The existing http and https packages are very flexible and support advanced functionality. However, the fetch global function is much more concise; writing Node.js code will feel more natural to developers used to writing JavaScript for the browser. Additionally, because Node.js and modern browsers now share this API, it is easier to write code to run in both environments.

Here's how we would write an HTTPS request in Node.js using
only first-party modules before the addition of the fetch API:

import https from "https";

const data = {
  nameFirst: "John",
  nameLast: "Doe",
};

let responseData = "";

const req = https.request(
  "https://example.org/user",
  { method: "POST" },
  (res) => {
    res.on("data", (data) => {
      responseData += data;
    });

    res.on("error", (error) => console.error(error));
  }
);

req.end(JSON.stringify(data), "utf-8", () => {
  console.log(`Response data: ${responseData}`);
});

Notice the heavy reliance on callbacks and the fact that the https module does not support promises. We have to register handlers for response data and errors, and be careful not to use the response data variable before the request completes.

While this is a somewhat awkward example, it highlights the
difficulty of using the https and http modules in Node, and explains why most developers choose to use a third-party module like Axios, request, or node-fetch.

If we were to make the same request in Node 18 using the fetch API, it would look something like this:

const data = {
  nameFirst: "John",
  nameLast: "Doe",
};

try {
  const response = await fetch("https://example.org/user", {
    method: "POST",
    body: JSON.stringify(data),
  });

  const responseJson = await response.json();
  console.log(`Response data: ${responseJson}`);
} catch (error) {
  console.error(error);
}

This version is decidedly more terse and familiar to JavaScript devs used to modern language features. I/O operations return promises, errors are thrown and caught, and the code reads synchronously from top to bottom.

While there will certainly still be use cases for the http and https packages and their lower-level capabilities, I think most devs will prefer the new fetch global function for daily use.

Support for Streams API

The Streams API broadly describes a set of interfaces such as ReadableStream and WritableStream, which allow us to process data incrementally before a whole object loads in memory. The Streams API is actually a prerequisite for fetch compatibility because the body property of a fetch response is a ReadableStream.

A common use case for streams is that we want to use a large amount of data, but don't need the entirety of the data to operate on: think multimedia applications or other real-time apps.

While Node.js has always supported these use cases, the availability of the same high-level APIs in both the client and server environments may make developing these applications easier.

Wrap Up: Ecosystem Implications and Node.js 18 Release Notes

One of the selling points of Node.js, and more broadly, JavaScript-based backends, is that full-stack engineers can work at all levels of a web application's stack without switching languages.

It's clear that the Node team is keen on playing to this advantage through Node.js 18's new features, continuing to narrow the gap between JavaScript in the browser and JavaScript on the server. When we consider the increased prevalence of code that needs to run in both environments, such as server-side rendered websites, it's encouraging to see that gap continue to close.

There have been several smaller changes in Node.js 18, such as changes to internationalization APIs, formalization of the Blob and BroadcastChannel APIs, and more.

If you want to dig into the full set of changes in this version, you should take a look at the official Node.js 18 release notes.

I hope you enjoy trying out the new features in this exciting new LTS release of Node.js. Happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Dependency Injection in JavaScript: Write Testable Code Easily

Nate Anderson — Wed, 02 Mar 2022 13:23:31 +0000

I struggled with two aspects of software development as a junior engineer: structuring large codebases and writing testable code. Test-driven development is such a common technique that is often taken for granted, but it's not always clear how code can be made fully testable.

I remember reading examples where an author would cleanly unit test a function, and in principle, it made sense. But real code doesn't look like those examples. No matter how thoughtfully it is written, real code has some level of complexity.

Ultimately, a lot of that complexity comes down to managing dependencies. This is arguably one of the chief challenges of software engineering; to quote the famous poem, "no man is an island entire of itself."

This article shares a few powerful tools to help you write testable code that grows into neat, manageable code bases.

But first, we need to ask: what are dependencies?

What Is A Dependency?

A dependency is any external resource a program needs to work. These can be external libraries the code literally depends on or services the program functionally needs, like internet APIs and databases.

The tools we use to manage these dependencies are different, but the problems are ultimately the same. A unit of code depends on other units of code, which themselves often have dependencies. For the program to work, all dependencies must be resolved recursively.

If you're not familiar with how package managers work, you might be surprised at the complexity of this problem. However, if you've written and attempted to test a webserver that relies on a database, you're probably familiar with another version of the same problem. Luckily for us, this is a well-studied problem.

Let's take a quick look at how you can use SOLID principles to improve the maintainability and stability of your code.

SOLID Principles

Robert Martin's SOLID principles are excellent guidelines for writing object-oriented code. I argue that two of these principles — the Single Responsibility principle and Dependency Inversion principle — can be critically important outside of OO design, as well.

Single Responsibility Principle

The Single Responsibility principle states that a class or function should have one — and only one — purpose, and thus only one reason to change. This resembles the UNIX philosophy — in essence, do one thing, and do it well. Keep your units simple and reliable, and achieve complex solutions by composing simple pieces.

For example, an Express handler function might sanitize and validate a request, perform some business logic, and store the result in a database. This function performs many jobs. Suppose we redesign it to follow the Single Responsibility principle. In that case, we move input validation, business logic, and database interactions into three separate functions that can be composed to handle a request. The handler itself does only what its name implies: handle an HTTP request.

Dependency Inversion Principle

The Dependency Inversion principle encourages us to depend on abstractions instead of concretions. This, too, has to do with separation of concerns.

To return to our Express handler example, if the handler function directly depends on a database connection, this introduces a host of potential problems. Say we notice our site is underperforming and decide to add caching; now we'll need to manage two different database connections in our handler function, potentially repeating cache checking logic over and over throughout the codebase and increasing the likelihood of bugs.

What's more, the business logic in the handler typically won't care about the details of the cache solution; all it needs is the data. If we instead depend on an abstraction of our database, we can keep changes in persistence logic contained and reduce the risk that a small change will force us to rewrite a ton of code.

The problem I've found with these principles is often in their presentation; it's difficult to present them on a general level without a fair bit of hand-waving.

I want to explain them concretely. Let's look at how to break a large, difficult-to-test handler function into small, testable units using these two principles.

Example: An Overwhelmed Express Handler for Node.js

Our example is an Express handler function that accepts a POST request and creates a listing on a job board for Node.js developers. It validates the input and stores the listing. If the user is an approved employer, the post is made public immediately, otherwise, it is marked for moderation.

const app = express();

app.use(express.json());

let db: Connection;

const title = { min: 10, max: 100 };
const description = { min: 250, max: 10000 };
const salary = { min: 30000, max: 500000 };
const workTypes = ["remote", "on-site"];

app.post("/", async (req, res) => {
  // validate input
  const input = req.body?.input;
  try {
    const errors: Record<string, string> = {};
    if (
      input.jobTitle.length < title.min ||
      input.jobTitle.length > title.max
    ) {
      errors.jobTitle = `must be between ${title.min} and ${title.max} characters`;
    }

    if (
      input.description.length < description.min ||
      input.jobTitle.length > description.max
    ) {
      errors.description = `must be between ${description.min} and ${description.max} characters`;
    }

    if (Number(input.salary) === NaN) {
      errors.salary = `salary must be a number`;
    } else if (input.salary < salary.min || input.salary > salary.max) {
      errors.salary = `salary must be between ${salary.min} and ${salary.max}`;
    }

    if (!workTypes.includes(input.workType.toLowerCase())) {
      errors.workType = `must be one of ${workTypes.join("|")}`;
    }

    if (Object.keys(errors).length > 0) {
      res.status(400);
      return res.json(errors);
    }
  } catch (error) {
    res.status(400);
    return res.json({ error });
  }

  const userId = req.get("user-id");

  try {
    // retrieve the posting user and check privileges
    const [[user]]: any = await db.query(
      "SELECT id, username, is_approved FROM user WHERE id = ?",
      [userId]
    );

    const postApprovedAt = Boolean(user.is_approved) ? new Date() : null;

    const [result]: any = await db.query(
      "INSERT INTO post (job_title, description, poster_id, salary, work_type, approved_at) VALUES (?, ?, ?, ?, ?, ?)",
      [
        input.jobTitle,
        input.description,
        user.id,
        input.salary,
        input.workType,
        postApprovedAt,
      ]
    );

    res.status(200);
    res.json({
      ok: true,
      postId: result.insertId,
    });
  } catch (error) {
    res.status(500);
    res.json({ error });
  }
});

This function has a lot of problems:

1. It does too many jobs to be practically testable.

We can't test that validation works without being connected to a functioning database, and we can't test storing and retrieving posts from the database without building fully-fledged HTTP requests.

2. It depends on a global variable.

Maybe we don't want tests polluting our development database. How can we instruct the function to use a different database connection (or even a mock) when the database connection is hard-coded as global?

3. It's repetitive.

Any other handler that needs to retrieve a user from their ID will essentially duplicate code from this handler.

Layered Architecture for Separation of Concerns in JavaScript

Suppose each function or class performs only one action. In that case, a function needs to handle the user interaction, another needs to perform the desired business logic, and another needs to interact with the database.

A common visual metaphor for this that you're likely familiar with is a layered architecture. A layered architecture is often depicted as four layers stacked on top of one another, with the database at the bottom and the API interface at the top.

When thinking about injecting dependencies, though, I find it more useful to think of these layers like the layers of an onion. Each layer must contain all of its dependencies to function, and only the layer that immediately touches another layer may interact with it directly:

The presentation layer, for example, should not interact directly with the persistence layer; the business logic should be in the business layer, which may then call the persistence layer.

It may not be immediately clear why this is beneficial — it certainly can sound like we are just making rules for ourselves to make things harder. And it actually may take longer to write code this way, but we are investing time in making the code readable, maintainable, and testable down the road.

Separation of Concerns: An Example

Here's what actually happens when we start separating concerns. We'll start with classes to manage the data stored in the database (part of the persistence layer):

// Class for managing users stored in the database
class UserStore {
  private db: Connection;
  constructor(db: Connection) {
    this.db = db;
  }

  async findById(id: number): Promise<User> {
    const [[user]]: any = await this.db.query(
      "SELECT id, username, is_approved FROM user WHERE id = ?",
      [id]
    );
    return user;
  }
}

// Class for managing job listings stored in the database
class PostStore {
  private db: Connection;
  constructor(db: Connection) {
    this.db = db;
  }

  async store(
    jobTitle: string,
    description: string,
    salary: number,
    workType: WorkType,
    posterId: number,
    approvedAt?: Date
  ): Promise<Post> {
    const [result]: any = await this.db.query(
      "INSERT INTO post (job_title, description, poster_id, salary, work_type, approved_at) VALUES (?, ?, ?, ?, ?, ?)",
      [jobTitle, description, posterId, salary, workType, approvedAt]
    );

    return {
      id: result.insertId,
      jobTitle,
      description,
      salary,
      workType,
      posterId,
    };
  }
}

Notice these classes are incredibly simple — in fact, they're simple enough to not need to be classes at all. You could write a function returning plain-old JavaScript objects or even "function factories" to inject dependencies into your functional units. Personally, I like to use classes, as they make it very easy to associate a set of methods with their dependencies in a logical unit.

But JavaScript was not born as an object-oriented language, and many JS and TS developers prefer a more functional or procedural style. Easy! Let's use a function that returns a plain object to achieve the same goal:

// Service object for managing business logic surrounding posts
export function PostService(userStore: UserStore, postStore: PostStore) {
  return {
    store: async (
      jobTitle: string,
      description: string,
      salary: number,
      workType: WorkType,
      posterId: number
    ) => {
      const user = await userStore.findById(posterId);

      // if posting user is trusted, make the job available immediately
      const approvedAt = user.approved ? new Date() : undefined;

      const post = await postStore.store(
        jobTitle,
        description,
        salary,
        workType,
        posterId,
        approvedAt
      );
      return post;
    },
  };
}

One disadvantage of this approach is that there isn't a well-defined type for the service object that's returned. We need to explicitly write one and mark it as the return type of the function, or use TypeScript utility classes elsewhere to derive the type.

We're already starting to see the benefits of separation of concerns here. Our business logic now depends on the abstractions of the persistence layer rather than the concrete database connection. We can assume the persistence layer will work as expected from inside the post service. The only job of the business layer is to enforce business logic, then pass persistence duty off to the store classes.

Before testing the new code, we can rewrite our handler function with injected dependencies using a very simple function factory pattern. Now, this function's only job is to validate an incoming request and pass it off to the application's business logic layer. I'll spare you the boredom of the input validation since we should be using a well-tested third-party library for this anyway.

export const StorePostHandlerFactory =
  (postService: ReturnType<typeof PostService>) =>
  async (req: Request, res: Response) => {
    const input = req.body.input;

    // validate input fields ...

    try {
      const post = await postService.store(
        input.jobTitle,
        input.description,
        input.salary,
        input.workType,
        Number(req.headers.userId)
      );

      res.status(200);
      res.json(post);
    } catch (error) {
      res.status(error.httpStatus);
      res.json({ error });
    }
  };

This function returns an Express handler function with all contained dependencies. We call the factory with the required dependencies and register it with Express, just like our previous inline solution.

app.post("/", StorePostHandlerFactory(postService));

I feel pretty comfortable saying the structure of this code is more logical now. We have atomic units, be they classes or functions, that can be tested independently and re-used when needed. But have we measurably improved the testability of the code? Let's try writing some tests and find out.

Testing Our New Units

Observing the Single Responsibility principle means that we only unit test the one purpose a unit of code fulfills.

An ideal unit test for our persistence layer does not need to check that primary keys increment correctly. We can take the behavior of lower layers for granted or even replace them entirely with hard-coded implementations. In theory, if all our units behave correctly on their own, they will behave correctly when they compose (though this is obviously not always true — it's the reason we write integration tests.)

Another goal we mentioned is that unit tests shouldn't have side effects.

For persistence layer unit tests, this means that our development database is not affected by the unit tests we run. We can accomplish this by mocking the database, but I would argue that containers and virtualization are so cheap today that we may as well just use a real, but different, database for testing.

In our original example, this would be impossible without altering the app's global configuration or mutating a global connection variable in each test. Now that we're injecting dependencies, though, it's actually really easy:

describe("PostStore", () => {
  let testDb: Connection;
  const testUserId: number = 1;

  beforeAll(async () => {
    testDb = await createConnection("mysql://test_database_url");
  });

  it("should store a post", async () => {
    const post = await postStore.store(
      "Senior Node.js Engineer",
      "Lorem ipsum dolet...",
      78500,
      WorkType.REMOTE,
      testUserId,
      undefined
    );
    expect(post.id).toBeDefined();
    expect(post.approvedAt).toBeFalsy();
    expect(post.jobTitle).toEqual("Senior Node.js Engineer");
    expect(post.salary).toEqual(78500);
  });
});

With only five lines of setup code, we're now able to test our persistence code against a separate, isolated test database.

Mocking on the Fly with Jest

But what if we want to test a unit in a "higher" layer, such as a business layer class? Let's look at the following scenario:

Given the job listing data from a user who is not pre-approved for immediate publishing, the post
service should store a post with a null approved_at timestamp.

Because we're only testing business logic, we don't need to test the process of storing or pre-approving an application user. We don't even need to test that the job posting is actually stored in an on-disk database.

Thanks to the magic of runtime reflection and the underlying dynamic nature of JavaScript, our testing framework will likely let us replace those components with hard-coded "mocks" on the fly. Jest, a popular JavaScript testing library, comes with this functionality baked in, and many other libraries provide it as well (such as SinonJS).

Let's write a test for this scenario, isolating it from any actual persistence or database logic using some simple mocks.

describe("PostService", () => {
  let service: ReturnType<typeof PostService>;
  let postStore: PostStore;
  let userStore: UserStore;
  const testUserId = 1;

  beforeAll(async () => {
    const db = await createConnection("mysql://test_database_url");
    postStore = new PostStore(db);
    userStore = new UserStore(db);
    service = PostService(userStore, postStore);
  });

  it("should require moderation for new posts from unapproved users", async () => {
    // for this test case, the user store should return an unapproved user
    jest
      .spyOn(userStore, "findById")
      .mockImplementationOnce(async (id: number) => ({
        id,
        username: "test-user",
        approved: false,
      }));

    // mocking the post store allows us to validate the data being stored, without actually storing it
    jest
      .spyOn(postStore, "store")
      .mockImplementationOnce(
        async (
          jobTitle: string,
          description: string,
          salary: number,
          workType: WorkType,
          posterId: number,
          approvedAt?: Date | undefined
        ) => {
          expect(approvedAt).toBeUndefined();
          return {
            id: 1,
            jobTitle,
            description,
            salary,
            workType,
            posterId,
            approvedAt,
          };
        }
      );

    const post = await service.store(
      "Junior Node.js Developer",
      "Lorem ipsum dolet...",
      47000,
      WorkType.REMOTE,
      testUserId
    );

    expect(post.id).toEqual(1);
    expect(post.posterId).toEqual(testUserId);
  });
});

Benefits of Mocking

Mocking, here, is simply temporarily replacing functions or class methods with predictable replacements (that have no external dependencies), inside which we can:

Test the data that higher layers pass in.
Fully control the behavior of layers of code lower than the layer we are currently testing.

That last part is incredibly powerful. It allows us to do things like test whether specific types of errors return accurate HTTP status codes, without actually having to break things to create those errors.

We don't need to disconnect from the test database to test if a connection refused error from the database results in a 500 Internal Server Error in the HTTP response. We can simply mock the persistence code that calls the database and throw the same exception we would see in that scenario. Isolating our tests and testing small units allows us to test much more thoroughly, so we can be sure that the behavior depended on by higher layers is correctly specified.

In well-isolated unit tests, we can mock any dependency. We can replace third-party web APIs with mock HTTP clients that are faster, cheaper, and safer than the real thing. If you want to ensure your application behaves correctly when an external API has an outage, you can replace it with a dependency that always returns a 503 for a subset of tests.

I know I'm really selling mocking here, but understanding the power of mock dependencies in small, focused unit tests was a kind of revelation for me. I'd heard the expression "don't test the framework" dozens of times, but it was only when mocking that I finally understood how it was possible to only test the behavior you're responsible for as a developer. It made my life much easier, and I hope this information can make yours easier, too.

A Note on Test Frameworks When Mocking Dependencies

I used Jest in the above example. However, a more universal (and in some ways superior) way of mocking dependencies in object-oriented code is through polymorphism and inheritance.

You can extend dependency classes with mock method implementations or define your dependencies as interfaces and write entirely isolated classes that fulfill those interfaces for testing purposes. Jest is just more convenient because it lets you easily mock a method once without defining new types.

Dependency Injection Libraries for TypeScript and JavaScript

Now that we're starting to think about dependencies as a sort of directed graph, you might notice how quickly the process of instantiating and injecting dependencies might become tiresome.

Several libraries are available for TypeScript and JavaScript to automatically resolve your dependency graph. These require you to manually list the dependencies of a class or use a combination of runtime reflection and decorators to infer the shape of your graph.

Nest.js is a notable framework that uses dependency injection, with a combination of decorators and explicit dependency declaration.

For existing projects, or if you don't want the weight of an opinionated framework like Nest, libraries like TypeDI and TSyringe can help.

Summing Up

In this post, we've taken a concrete example of an overwhelmed function and replaced it with a composition of smaller, testable units of code. Even if we accomplish identical lines-of-code test coverage for both versions, we can know exactly what broke and why when tests fail in the new version.

Before, we only generally knew that something broke, and we'd likely find ourselves digging through error messages and stack traces to figure out what input led to an exception, what the breaking change was, etc.

I hope this concrete example has helped to explain the two critical SOLID principles of single responsibility and dependency inversion.

It's worth noting that this is not the hammer for every nail. Our end goals are maintainability and reliability, and simple code is easier to maintain. Inversion of control is a great tool for managing complexity, but it is not a reason to introduce undue complexity to a simple program.

Until next time, happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

TypeScript Compiler API: Improve API Integrations Using Code Generation

Nate Anderson — Wed, 25 Aug 2021 13:39:46 +0000

Developing against third-party or unfamiliar web APIs can be painful and slow compared to using native libraries.

For example, making requests with an HTTP client is simple enough but offers no compile-time sanity checks and no way for code suggestion tools like Intellisense to help you navigate. Worst of all, if the API you are consuming introduces breaking changes, you won't find out until runtime.

If you prefer the safety and convenience of strong types when working with web APIs, code generation can be a convenient solution. If the API provides a structured service definition, code generation can ease the development of client code, reduce runtime errors, and alert you to breaking changes before your code is deployed.

For TypeScript and JavaScript apps, the TypeScript Compiler API provides everything you need to build your own code generation tooling.

Examples of Existing Code Generation Tools

There is no sense in building a code generation tool if there's already one available that meets your needs. Several mature code generation tools for web APIs exist already; there's a good chance you've even used one.

The following are fantastic tools for adding safety and sanity to your networked Node.js apps:

Protobufjs generates input and output types and function signatures for gRPC clients by reading a .protobuf file.
openapi-generator generates well-defined JavaScript clients for REST APIs based on an OpenAPI spec document.

The above can add a layer of safety to your deployments, especially if the API definition file you're consuming is available over the web and can be fetched as part of your CI process.

However, if you're dealing with a less common API definition format (like a SOAP WSDL file or a custom JSON definition), these tools won't help you at all.

But you're not totally out of luck! Creating code generation tools like these does not require experience writing compilers or even an intimate knowledge of the internals of TypeScript and JavaScript.

Build the Codegen Tooling Yourself

In the many cases when a well-defined API doesn't have existing codegen tooling, such as:

a home-grown JSON-over-TCP API
a message queue with specific input and output formats defined in a YAML file
something dated (and dreaded) like a SOAP API

Then we can take advantage of the TypeScript compiler API to generate types or even complete clients. It simply becomes a problem of coercing one form of structured data into another — and if you're a developer, that's probably your bread and butter.

Abstract Syntax Trees with the TypeScript Compiler API

Part of the TypeScript compiler's job is to construct abstract syntax trees (ASTs) of the input source code when transpiling TypeScript to JavaScript. These trees are structured representations of the source code that are fairly easy to understand, even if you're not used to working with intermediate representations of code (after all, the compiler is emitting plain old JavaScript).

The methods used to construct ASTs from a source code file are exported from the typescript library, allowing us to generate ASTs from scratch. It's easier to get started with a structured API definition.

Example: Generating Input and Return Types for a SOAP API

SOAP APIs are a good example of a messaging protocol with a strongly-typed schema and relatively poor tooling in the JavaScript ecosystem.

If you're unfortunate enough to have to work with a SOAP API in a Node.js project (as I have been several times), you need all the help you can get. So let's work through an example.

A common library for consuming SOAP APIs in JavaScript is soap. This will enable you to make API calls with JavaScript objects as inputs and outputs, but these objects are not type-checked at compile time. All client methods are typed any, and Node won't even know if methods in your code are callable until runtime.

Let's look at an example API definition, how we might currently consume it with the soap package, and some easy-to-miss bugs. Here's a (partial) definition of a stock quote SOAP service:

<definitions
    name="StockQuoteService"
    targetNamespace="http://www.example.com/wsdl/StockQuoteService.wsdl"
    xmlns="http://schemas.xmlsoap.org/wsdl/"
    xmlns:soap = "http://schemas.xmlsoap.org/wsdl/soap/"
    xmlns:qs = "http://www.example.com/wsdl/StockQuoteService.wsdl"
    xmlns:xsd = "http://www.w3.org/2001/XMLSchema"
>
    <message name="FetchQuoteRequest">
        <part name="tickerSymbol" type="xsd:string"/>
        <part name="dateRangeStart" type="xsd:dateTime"/>
        <!-- Optional, defaults to current date -->
        <part name="dateRangeEnd" type="xsd:dateTime" minOccurs="0"/>
    </message>

    <message name="Quote">
        <part name="tickerSymbol" type="xsd:string"/>
        <part name="current" type="xsd:int"/>
        <part name="dateRangeHigh" type="xsd:int"/>
        <part name="dateRangeLow" type="xsd:int"/>
        <part name="dateRangeDelta" type="xsd:string"/>
        <part name="volume" type="xsd:int"/>
    </message>

    <portType name="FetchQuotePortType">
        <operation name="FetchQuote">
            <input message="qs:FetchQuoteRequest"/>
            <output message="qs:Quote"/>
        </operation>
    </portType>

    <binding>
        ...
    </binding>
</definitions>

This WSDL defines a single RPC operation, FetchQuote, and its input and output types, FetchQuoteRequest and Quote. In our client code, these would correspond to a method definition and two object types.

We could jump right in and consume this API with the soap package, with an implementation that's something like this:

import soap from "soap";

// format integer cents as a currency amount string
const formatCents = (cents: number, currencySymbol: string = "$"): string =>
  `${currencySymbol}${(cents / 100).toFixed(2)}`;

// format number as fixed-width percentage string
const formatPercentage = (value: number): string => `${value.toFixed(2)}%`;

async function main() {
  const client = await soap.createClientAsync(
    "http://www.example.com/wsdl/StockQuoteService.wsdl"
  );

  const req = {
    tickerSymbol: "FOO",
    dateRangeStart: "2021-03-10T09:00:00",
    dateRangeEnd: "2021-05-10T18:00:00",
  };

  const res = await client.FetchQuote(req);
  console.log(`Stock Quote for ${res.tickerSymbol}:
    Current price: ${formatCents(res.current)}
    Range: ${req.dateRangeStart} - ${req.dateRangeEnd}
    Range high: ${formatCents(res.dateRangeHigh)}
    Range low: ${formatCents(res.dateRangeLow)}
    Change in range: ${formatPercentage(res.dateRangeDelta)}
    Volume: ${res.volume}
  `);
}

Even with the advantage of a strongly-typed WSDL file, this implementation is open to all kinds of type errors.

There's already a bug in this code that would not be caught until runtime. Notice the WSDL defines Quote.dateRangeDelta as a string, but this code expects a number. The .toFixed() method call will throw a runtime type error, but even using TypeScript as we are in this example will not help us catch this error during compilation.

Accessing the Compiler API

To generate a TypeScript AST, you need the ubiquitous typescript package.

npm install typescript

To create the initial AST, import the compiler and instantiate an in-memory representation of a TypeScript source file:

import ts from "typescript";

const sourceFile = ts.createSourceFile(
  "soap-types.ts", // the output file name
  "", // the text of the source code, not needed for our purposes
  ts.ScriptTarget.Latest, // the target language version for the output file
  false,
  ts.ScriptKind.TS // output script kind. options include JS, TS, JSX, TSX and others
);

Reading the API Definition

We need a structured, programmatically accessible representation of the API schema. In JavaScript and TypeScript, this will usually be an object. Depending on your API definition format, this could be as simple as const schema = require("api.json").

In our case, we'll need a simple library to read the WSDL XML file into something a bit easier to work with. We'll use xml2js.

Since part of our goal is to achieve better type safety, we'll create some types for the API definition as well. For most APIs, we'll have some version of an input and output object, and a named function with the input type as an argument, returning the output type.

At this stage, it's helpful to focus on modeling the schema of the API in an implementation-agnostic way. Don't worry about things like client method signatures or dependency injection if you can help it.

import { Parser } from "xml2js";
import { readFileSync } from "fs";

// Primitive datatypes defined by SOAP (there are more)
type SoapPrimitive =
  | "xsd:boolean"
  | "xsd:double"
  | "xsd:float"
  | "xsd:int"
  | "xsd:short"
  | "xsd:string"
  | "xsd:dateTime";

// SOAP message type
interface IMessage {
  meta: {
    name: string;
  };
  part: {
    meta: {
      name: string;
      type: SoapPrimitive;
      minOccurs?: string; // default = 1, making the field required. 0 notes an optional field
    };
  }[];
}

interface IOperation {
  meta: {
    name: string;
  };
  input: {
    meta: {
      name: string;
    };
  };
  output: {
    meta: {
      name: string;
    };
  };
}

// Top-level WSDL object structure
interface IServiceDefinition {
  definitions: {
    message: IMessage[];
    portType: {
      operation: IOperation[];
    }[];
  };
}

We can then write a function to generate instances of these types. It should read the file into an object and coerce it into the IServiceDefinition interface.

async function readWSDL(filePath: string): Promise<IServiceDefinition> {
  const wsdlData = readFileSync(filePath, { encoding: "utf-8" });

  const xmlParser = new Parser({
    attrkey: "meta", // instructs the parser to pack XML node attributes into a sub-object titled "meta"
  });

  const serviceDefinition = await xmlParser.parseStringPromise(wsdlData);

  // I would recommend a more explicit conversion, but for the sake of brevity, this example just casts the object to the interface type.
  return serviceDefinition as IServiceDefinition;
}

Generating the Input and Output Types

The process of converting a structured API definition into a Javascript object will vary widely based on use case and implementation. It's important to get well-organized information about the types you want to generate. Here, the critical bits are the name, data type, and optionality of the field.

To create TypeScript interfaces to represent the WSDL message types, we'll employ the TypeScript compiler's createInterfaceDeclaration factory (if you prefer class definitions, createClassDeclaration works similarly). We need to provide it with the same information we would provide an interface declaration if we were writing it by hand: attributes, modifiers, and obviously, a name. If you've ever used a compiler API like LLVM, this may feel familiar. In
more specialized use cases, you can even provide generic type parameters and decorators.

For our example, we just need simple interfaces with strongly-typed fields. This will require mapping from WSDL-defined types to TypeScript types.

WSDL defines a lot of types, and your implementation may not need all of them. For this example, I'm excluding some lesser-used ones like hexBinary and unsignedByte and default unrecognized types to string:

const dataTypes: Record<SoapPrimitive, string> = {
  "xsd:boolean": "Boolean",
  "xsd:double": "Number",
  "xsd:float": "Number",
  "xsd:int": "Number",
  "xsd:string": "String",
  "xsd:short": "Number",
  "xsd:dateTime": "Date",
};

// Convert from SOAP primitive type to a Typescript type reference, defaulting to String
function typeFromSOAP(soapType: SoapPrimitive): ts.TypeReferenceNode {
  const typeName = dataTypes[soapType] ?? "String";
  return ts.factory.createTypeReferenceNode(typeName, []);
}

Some input fields may be optional, so we will need a reference to the relevant syntax (a question mark in TypeScript, ignored when importing distributable code into a JavaScript file). To create simple tokens like this (and things like comparison operators, math and logical operators) we can use the createToken factory method.

// used for marking attributes as optional
const optionalFieldMarker = ts.factory.createToken(ts.SyntaxKind.QuestionToken);

It's important we can use these types elsewhere, so we'll need to add the export modifier to the declared types. The ts.ModifierFlags object contains an assortment of modifiers, like class member modifiers (public, protected, static), and the async keyword.

// used for adding `export` directive to generated type
const exportModifier = ts.factory.createModifiersFromModifierFlags(
  ts.ModifierFlags.Export
);

With the basic tools defined, we can write a function to generate the actual AST nodes that will be written to our generated code. The goal is to translate from our familiar structure to TypeScript AST using the factories provided by the compiler to create nodes, then assemble those nodes into interface declarations.

We need an identifier (essentially, the code symbol that names the interface), and a list of properties, each with its own identifier, type, and possibly optionality token (?).

function generateMessageInterface(message: IMessage): ts.InterfaceDeclaration {
  // name of the interface
  const interfaceSymbol = ts.factory.createIdentifier(message.meta.name);

  const interfaceMembers = message.part.map((part) => {
    const propertySymbol = ts.factory.createIdentifier(part.meta.name);

    // WSDL treats fields with no minOccurs as required, and fields with minOccurs > 0 as required
    const required =
      !Boolean(part.meta.minOccurs) || Number(part.meta.minOccurs) > 0;

    // representation of the interface property syntax
    const member = ts.factory.createPropertySignature(
      undefined, // modifiers (not allowed for interfaces)
      propertySymbol, // name of the property
      required ? undefined : optionalFieldMarker,
      typeFromSOAP(part.meta.type) // data type
    );
    return member;
  });

  const messageType = ts.factory.createInterfaceDeclaration(
    undefined, // no decorators
    exportModifier, // modifiers
    interfaceSymbol, // interface name
    undefined, // no generic type parameters
    undefined, // no heritage clauses (extends, implements)
    interfaceMembers // interface attributes
  );

  return messageType;
}

Then to generate all the needed types, it's as simple as mapping over our IServiceDefinition object's message array to this function, placing the resulting interface declaration nodes in a NodeArray, and writing the AST to a TypeScript file.

// read service definition into object
const serviceDefinition = await readWSDL("quote_service.wsdl");

const messageInterfaces = serviceDefinition.message.map(
  generateMessageInterface
);

// add nodes to a node array
const nodeArr = ts.factory.createNodeArray(messageInterfaces);

To write the source tree to disk, we need an instance of the ts.Printer class. Printing the node list to a file is as simple as specifying the file format (usually MultiLine as it is the most readable) and a destination file path.

const printer = ts.createPrinter({ newLine: ts.NewLineKind.LineFeed });
const result = printer.printList(ts.ListFormat.MultiLine, nodeArr, sourceFile);
writeFileSync("soap-types.ts", result, { encoding: "utf-8" });

This will write the following code to a file called soap-types.ts:

export interface FetchQuoteRequest {
  tickerSymbol: String;
  dateRangeStart: Date;
  dateRangeEnd?: Date;
}
export interface Quote {
  tickerSymbol: String;
  current: Number;
  dateRangeHigh: Number;
  dateRangeLow: Number;
  dateRangeDelta: String;
  volume: Number;
}

That's it! Correctly typed input and output that's easy-to-read (for both people and machines). Using these types in place of the Plain Old JavaScript Objects we used in the first example will give us a bit of compile-time safety.

Even standard JavaScript projects can benefit from improved IDE code completion through simple changes to the transpilation target.

We could go further down this path and generate a full type-safe client (likely just a wrapper around a soap.Client), with methods defined for each WSDL operation, using the generated types as inputs and outputs. Generating such a function might look something like this:

// used for adding `async` and `export` modifier to generated function
const asyncExportModifier = ts.factory.createModifiersFromModifierFlags(
  ts.ModifierFlags.Export | ts.ModifierFlags.Async
);

function generateOperationFunction(
  methodName: string,
  inputType: ts.InterfaceDeclaration,
  outputType: ts.InterfaceDeclaration
): ts.FunctionDeclaration {
  const functionIdentifier = ts.factory.createIdentifier(methodName);

  const inputType = ts.factory.createTypeReferenceNode(inputType.name);
  const outputType = ts.factory.createTypeReferenceNode(outputType.name);
  const outputTypePromise = ts.factory.createTypeReferenceNode("Promise", [
    outputType,
  ]);

  const inputParameter = ts.factory.createParameterDeclaration(
    undefined, // decorators,
    undefined, // modifiers,
    undefined, // spread operator
    "input", // name
    undefined, // optional marker,
    inputType, // type
    undefined // initializer
  );

  const fn = ts.factory.createFunctionDeclaration(
    undefined, // decorators
    asyncExportModifier,
    undefined, // asterisk token
    functionIdentifier,
    undefined, // generic type parameters
    [inputParameter], // arguments
    outputTypePromise, // return type
    undefined // function body
  );

  return fn;
}

const operationFunctions =
  serviceDefinition.definitions.portType[0].operation.map((op) =>
    generateOperationFunction(
      op.meta.name,
      messageInterfaces[0],
      messageInterfaces[1]
    )
  );

// add nodes to a node array
const nodeArr = ts.factory.createNodeArray(operationFunctions);

Using this function with the input and output types we just generated, we create the following function stub in soap-types.ts:

export async function FetchQuote(input: FetchQuoteRequest): Promise<Quote>;

To make this really useful, you'd want to inject calls to methods on a soap.Client within these generated functions - but that's beyond the scope of this post.

TypeScript Compiler: A Powerful Codegen Tool

The TypeScript compiler is a powerful code generation tool — any code that you can manually write, you can write programmatically using this API.

It's worth noting that, as in the case of many specialized APIs, the TypeScript compiler API's documentation is incomplete and not particularly friendly to newcomers. This crash-course README is, to my knowledge, the best place to get started, but I feel the best way to get familiar with the API is to play around with it yourself and dive into the code.

There are tons of ways custom code generation can benefit your project beyond generating API types, including instrumenting code at compile time (think performance monitoring, consistent logging, etc.) and generating skeleton code when porting from another similar language.

I hope this article has given you ideas of how to improve your project's reliability or developer experience using code generation. Cheers!

Check out the code from this article in this repo.

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Nate Anderson is a software engineer working on new products at CARD.com, a tech company in the United States providing personalized payment solutions like premium banking. He believes the key to great technology is a close relationship between product and engineering, and is excited about tech like Go, TypeScript, GraphQL, and serverless computing.

DEV Community: Nate Anderson

Designing a Scalable Backend in Node.js

About Our System With a Node.js Backend

Challenge 1: Issues When Logging In As a User

Where's the Point of Failure?

Investigating Further

Handling Bursty Workloads

Lessons Learned

Challenge 2: Missing Transactions in Our Node.js App

Restarting: Not Always Enough

The Benefits of Queueing

Lessons Learned

Challenge 3: Transactions Taking A Long Time to Load

Table Partitioning

Upgrading your Database Server

INCLUDE Key Columns in your Index

Consider A Different Data Store

Wrapping Up

Node.js Architecture Pitfalls to Avoid

Why Worry About Node.js Architecture?

Globals in JavaScript: You Were Warned for a Reason!

The Problem with Globals: An Example

The Fix: Using Arguments

Set Up A Test That Goes to One Email

Environment Variables in JavaScript: Socially Acceptable Globals

Make Your JavaScript Code Easy to Test with Environment Variables: An Example

Request-scoped Dependencies: Not Global Enough?

Node.js Modules: A Slippery Slope

The Issue with Third-party Libraries for Node.js

Metrics to Assess Third-party Libraries

Metrics to Avoid

Wrapping Up: Work Smarter, Not Harder

Node.js 18 Release: What's New

Long-term Support Release Status for Node.js 18

V8 Engine Upgrade in Node.js 18

Node.js 18 Has A Built-in Test Runner

New Browser-compatible APIs in Node.js 18

Node.js Now Supports Fetch API

Support for Streams API

Wrap Up: Ecosystem Implications and Node.js 18 Release Notes

Dependency Injection in JavaScript: Write Testable Code Easily

What Is A Dependency?

SOLID Principles

Single Responsibility Principle

Dependency Inversion Principle

Example: An Overwhelmed Express Handler for Node.js

Layered Architecture for Separation of Concerns in JavaScript

Separation of Concerns: An Example

Testing Our New Units

Mocking on the Fly with Jest

Benefits of Mocking

A Note on Test Frameworks When Mocking Dependencies

Dependency Injection Libraries for TypeScript and JavaScript

Summing Up

TypeScript Compiler API: Improve API Integrations Using Code Generation

Examples of Existing Code Generation Tools

Build the Codegen Tooling Yourself

Abstract Syntax Trees with the TypeScript Compiler API

Example: Generating Input and Return Types for a SOAP API

Accessing the Compiler API

Reading the API Definition

Generating the Input and Output Types

TypeScript Compiler: A Powerful Codegen Tool

`INCLUDE` Key Columns in your Index