DEV Community: Matt Krick

Replacing Express with uWebSockets

Matt Krick — Mon, 06 Jan 2020 15:48:28 +0000

One of the best parts of running an enterprise SaaS is that our traffic takes a nosedive at the end of the year while clients universally take vacation. The low traffic is a great excuse for larger refactors and with our crazy growth this year, we've been considering scaling our server horizontally. Before we do, I figured it'd be smart to squeak out as much performance as possible. So, after 4 years, we ditched Express for something faster: uWebSockets.

uWebSockets is lighting fast because it is so low level. Saying it is like Express without the training wheels is an understatement. It's more like taking off the training wheels, reflectors, mud guards, hand grips, seat, and then stripping the paint because, well, speed. While I appreciate the speed and low memory footprint, I also don't want to run the risk of my sloppy code crashing the server, so my goal is to make a couple reasonable performance sacrifices to make it as safe as an Express framework. In other words, I'll take the bike-- just give me a darn helmet.

Practically, that means I don't want to worry about a call to Redis somehow failing, which throws an uncaught promise rejection, thus hanging the response and in turn hanging the server. To save myself from myself, I came up with a few reasonable patterns to avoid both rewriting my sloppy code and a crash. Hopefully, you find them useful, too.

Response Handling

At all costs, we must close the HttpResponse or it will hang and bring the server to its knees. There are 2 ways the response can close: calling a terminating method (end, tryEnd or close) or being hung up on by the client (onAborted fires). Unfortunately, once the response has been closed, you cannot attempt to close it again. That restriction creates a race condition. Imagine the scenario where a request comes in to read a record from the DB:

const handler = async (res) => {
  const doc = await db.get('user')
  res.end(doc)
}

In a perfect world, the doc from the DB returns and gets sent as the response. In the real world, the client disconnects just after the call to the DB is made. When that happens, the socket is closed, onAborted fires, and by the time res.end is called, the response has already been invalidated, which produces an error.

To tackle this problem, I need to guarantee 3 things:

A terminating method is not called after onAborted fires
A terminating method is not called after a terminating method was already called
There is only 1 onAborted handler for each response

To accomplish #1 & #2 without rewriting all my route handlers, I decided to monkeypatch the response with some safety checks. For example, I put a done one-way flag on the response and if a terminating method is called after the response is already done, it is ignored:

  res._end = res.end
  res.end = (body) => {
    if (res.done) {
      console.log(`uWS DEBUG: Called end after done`)
      return res
    }
    res.done = true
    return res._end(body)
  }

Guaranteeing a single onAborted handler was necessary because there were some cases where the thing I was trying to clean up (e.g. a ReadStream or setInterval id) was created after onAborted was already made. To keep my code modular, I again monkeypatched onAborted to support multiple handlers:

  res.onAborted(() => {
    res.done = true
    if (res.abortEvents) {
      res.abortEvents.forEach((f) => f())
    }
  })

  res.onAborted = (handler) => {
    res.abortEvents = res.abortEvents || []
    res.abortEvents.push(handler)
    return res
  }

Async Handler Wrapping

With uWebSockets, async http handlers also require extra care. Aside from having to cork response methods to achieve maximum performance, errors can creep in from various sources:

Attempting to close an already closed response (as discussed above)
An unplanned error (uncaught exception, unhandled promise rejection)
Returning without closing the response

Since I'm not sure where these errors may live, the safest bet is to apply the guards as soon as possible, at the beginning of the handler. To keep my code DRY, I wrapped each async handler in a higher order function that catches the 3 error types:

const uWSAsyncHandler = (handler: uWSHandler) => async (res: HttpResponse, req: HttpRequest) => {
  monkeyPatchRes(res)
  try {
    await handler(res, req)
    if (!res.done) {
      throw new Error('Async handler did not respond')
    }
  } catch (e) {
    res.writeStatus('500').end()
    sendToReportingService(e)
  }
}

It's a pretty simple function. First, it monkeypatches the res so we get free done tracking (Type #1). Then, it tries to execute the handler. If the handler throws an error (Type #2), or it returns without closing the response (Type #3), it gracefully closes the connection and reports the error to our monitoring service. With very little computational overhead, I can keep on writing sloppy code and not worry about crashing the server. Success! 🎉

Body Parsing

The code example in the uWebSockets repo does a great job of showing how to parse an incoming body. Written as a promise, it can be quite elegant:

const parseBody = (res: HttpResponse) => {
  return new Promise<string>((resolve) => {
    let buffer: Buffer
    res.onData((chunk, isLast) => {
      const curBuf = Buffer.from(chunk)
      buffer = buffer ? Buffer.concat([buffer, curBuf]) : 
               isLast ? curBuf : Buffer.concat([curBuf])
      if (isLast) {
        resolve(buffer.toString())
      }
    })
  })
}

The funny buffer ternary is necessary because onData reuses the same memory allocation for the following chunk. That means we'll need to clone the buffer by calling concat or toString before yielding. I like to return the stringified JSON instead of parsed JSON because sometimes I need the string itself (e.g. SAML response processing or verifying a Stripe webhook payload).

It's worth noting that uWebSocket's onData handler doesn't play well with breakpoints when using Node's built-in debugger: Issue #191. To mitigate that issue, you can simply clone the chunk and resolve inside a setImmediate call. Since that has a nontrivial amount of overhead, I only do it when Node is in debuging mode (process.execArgv.join().includes('inspect')).

Serve Static

Almost all of our assets are served from our CDN in production. However, there are a few exceptions: index.html, serviceWorker.js, and everything in development mode. So, I needed something like Express' serve-static that did the following:

Serve whitelisted items from memory to reduce disk reads
Serve those whitelisted items in a compressed format, if supported
Support webpack-dev-middleware by serving webpack assets in development

While the first two were rather simple to implement (actual code here), supporting webpack-dev-middleware is a bit more interesting. Since performance in development isn't an issue and I wasn't trying to rewrite webpack-dev-middleware from scratch, I decided to simply pass it something that looked like an Express handler:

const makeExpressHandlers = (res: HttpResponse, req: HttpRequest) => {
  res.setHeader = (key: string, value: unknown) => {
    res.writeHeader(key.toLowerCase(), String(value))
  }
  const headers = {}
  req.forEach((key, value) => {
    headers[key] = value
  })
  return {
    req: {
      url: req.getUrl(),
      method: req.getMethod().toUpperCase(),
      headers
    },
    res,
    next: () => {}
  }
}

Since the middleware thinks it's getting a standard Express response, checking the result is as easy as checking the res.statusCode as seen here.

WebSocket Performance

The biggest benefit to moving to uWebSockets is, of course, the fast & memory efficient WebSocket server. While most WebSocket messages are tiny, frequent messages like pongs and peer updates, some initial payloads could get rather large, up to 45KB in our case. Assuming an MTU of 1500 bytes, that's 30 packets! Since WebSockets are built on top of TCP, which guarantees that packets arrive in order, this meant users with less-than-great connectivity could experience significant lag. Combatting this was easy: reduce the number of packets via compression. Using uWebSocket's SHARED_COMPRESSOR and monitoring packet size with WireShark, I could reduce the 45KB payload down to 6KB with no additional memory overhead, but I was left wondering if I could still do better. Deduplicating JSON objects and using msgpack only yielded savings of an extra 100 bytes each, which was hardly worth the extra computational overhead. So, I decided to look deeper.

First, WebSocket extensions only support the DEFLATE compression algorithm, which yields results about 30% bigger than Brotli compression. Second, there's no way to selectively compress messages, which means CPU cycles were being wasted compressing messages from the browser as well as single-packet messages from the server. So, I brought compression to the application layer. Since most browser messages to the server were very small, it made no sense compressing them, which means the client only needed a decompressor. I wrapped a Brotli decompressor written in Rust into a WASM package. I chose WASM over JS because in my tests (using Chrome 79), it was over 10x faster at decompression. On the server, I only compressed messages larger than 1400 bytes (100 bytes smaller than the MTU limit to account for headers) to guarantee compression would result in at least 1 less packet. The end result is best-in-class compression where you need it, and no compression where you don't. Best of both worlds! The only drawback is the size: the WASM decompressor compiles to about 300KB. To get around this, I compress it and persist it with a service worker to make sure it doesn't affect returning users. This works for us because we only use WebSockets for users who are logged in, however your business logic may be different and it's very likely that the added complexity of custom compression may not be worth the marginal savings. The only way to know is to measure, so I'll be testing that over the coming months.

Conclusion

Overall, I'm thrilled with uWebSockets. Not simply for the performance boost, but because it forced me to refactor a server that grew a little unwieldy as we've grown from nothing to a seed-funded startup with over 16,000 users. If this stuff sounds like fun to you, get paid to work on it! We're a remote team, our codebase is open source, and if you're reading articles like this one, chances are we already like you. Reach out to me directly or apply at https://www.parabol.co/join.

Fixing Memory Leaks in Node Apps

Matt Krick — Thu, 12 Dec 2019 16:12:48 +0000

A few months back, our web server crashed. It only lasted a minute before restarting, but as the tech guy in a small startup, it was a pretty stressful minute. I never set up a service to restart when memory got low, but we did have some reporting tools connected, so after the crash, I dug into our logs.

Yep, that's a memory leak alright! But how could I track it down?

Just like LEGOs

When debugging, I like to think about memory like LEGOs. Every object created is a brick. Every object type, a different color. The heap is a living room floor and I (the Garbage Collector) clean up the bricks no one is playing with because if I don't, the floor would be a minefield of painful foot hazards. The trick is figuring out which ones aren't being used.

Debugging

When it comes to triaging memory leaks in Node, there are 2 strategies: snapshots and profiles.

A snapshot (AKA heap dump) records everything on the heap at that moment.
It's like taking a photograph of your living room floor, LEGOs and all. If you take 2 snapshots, then it's like a Highlights magazine: find the differences between the 2 pictures and you've found the bug. Easy!

For this reason, snapshots are the gold standard when it comes to finding memory leaks. Unfortunately, taking a snapshot can last up to a minute. During that time, the server will be completely unresponsive, which means you'll want to do it when no one is visiting your site. Since we're an enterprise SaaS, that means Saturday at 3AM. If you don't have that luxury, you'll need to have your reverse proxy redirect to a backup server while you dump.

A sampling allocation profile is the lightweight alternative, taking less than a second. Just as the name implies, it takes a sample of all the objects getting allocated. While this produces a very easy-on-the-eyes flamechart akin to a CPU profile, it doesn't tell you what's being garbage collected.

It's like looking at the LEGOs being played with, but not looking at which ones are being put down. If you see 100 red bricks and 5 blue bricks, there's a good chance the red bricks could be the culprit. Then again, it's equally likely all 100 red bricks are being garbage collected and it's just the 5 blues that are sticking around. In other words, you'll need both a profile and deep knowledge of your app to find the leak.

The Implementation

In my case, I did both. To set up the profiler, I ran it every hour & if the actual memory used had increased by 50MB, it wrote a snapshot.

import * as heapProfile from 'heap-profile'

let highWaterMark = 0
heapProfile.start()
  setInterval(() => {
    const memoryUsage = process.memoryUsage()
    const {rss} = memoryUsage
    const MB = 2 ** 20
    const usedMB = Math.floor(rss / MB)
    if (usedMB > highWaterMark + 50) {
      highWaterMark = usedMB
      const fileName = `sample_${Date.now()}_${usedMB}.heapprofile`
      heapProfile.write(fileName)
    }
  }, 1000 * 60 * 60)

The snapshot was a little more interesting. While a normal approach is to send a SIGUSR2 signal to the node process using kill, I don't like that because you know what else can send a SIGUSR2? Anything. You may have a package in your dependencies right now (or in the future) that emits that same signal and if it does, then your site is going down until the process completes. Too risky, plus a pain to use. Instead, I created a GraphQL mutation for it. I put it on our "Private" (superuser only) schema and can call it using GraphiQL.

The code behind the endpoint is dead simple:

import profiler from 'v8-profiler-next'

const snap = profiler.takeSnapshot()
const transform = snap.export()
const now = new Date().toJSON()
const fileName = `Dumpy_${now}.heapsnapshot`
transform.pipe(fs.createWriteStream(fileName))
return new Promise((resolve, reject) => {
  transform.on('finish', () => {
    snap.delete()
    resolve(fileName)
  })
})

We take a snapshot, pipe it to a file, delete the snap, and return the file name. Easy enough! Then, we just upload it to Chrome DevTools Memory Tab and away we go.

Reading the Dump

While the profile wasn't very helpful, the heap dump got me exactly what I needed. Let's take a look at a leak called ServerEnvironment.

In our app, we do some light server side rendering (SSR) for generating emails. Since our app is powered by Relay (a great GraphQL client cache like Apollo), we use what I named a ServerEnvironment to fetch the data, populate the components, and then go away. So why are there 39 instances? Who's still playing with those LEGOs?!

The answer rests in the Retainers section. In plain English, I read the table like this, "ServerEnvironment can't be garbage collected because it is item 56 in a Map, which can't be garbage collected because it is used by object requestCachesByEnvironment. Additionally, it's being used by environment, which is used by _fetchOptions, which is used by queryFetcher which is used by" ...you get it. So requestCachesByEnvironment and requestCache are the culprits.

If I look for the first one, I find the offender in just a couple lines of code (edited for brevity, original file here):

const requestCachesByEnvironment = new Map();

function getRequestCache(environment) {
  const cached = requestCachesByEnvironment.get(environment)
  if (!cached) {
    const requestCache = new Map()
    requestCachesByEnvironment.set(environment, requestCache)
  }
  return requestCachesByEnvironment.get(environment)
}

This is your classic memory leak. It's an object at the outermost closure of a file that's being written to by a function in an inner closure & no delete call to be found. As a general rule of thumb, writing to variables in outer closures are fine because there's a limit, but writing to objects often leads to problems like this since the potential is unbounded. Since the object isn't exported, we know we have to patch this file. To fix, we could write a cleanup function, or we can ask ourselves 2 questions:
1) Is that Map being iterated over? No
2) If the Map item is removed from the rest of the app does it need to exist in the Map? No

Since the answer to both questions is No, it's an easy fix! Just turn Map into WeakMap and we're set! WeakMaps are like Maps, except they let their keys get garbage collected. Pretty useful!

The second retainer can be tracked down to requestCache. Instead of a Map, this is a plain old JavaScript object, again kept in the outermost closure (notice a pattern here? it's a bad pattern). While it'd be great to achieve this in a single closure, that'd require a big rewrite. A shorter, elegant solution is to wipe it if it's not running in the browser, seen here.

With those 2 fixes, our ServerEnvironment is free to be garbage collected and the memory leak is gone! All that's left to do is make the fixes upstream and use the new version. Unfortunately, that can take weeks/months/never happen. For immediate gratification, I like to use the FANTASTIC gitpkg CLI that publishes a piece of a monorepo to a specific git tag of your fork. I never see folks write about it, but it has saved me so much time forking packages I had to share.

Memory leaks happen to everyone. Please note that I'm not picking on code written by Facebook to be rude, insult, or take some weird political stance against their company ethics. It's simply because 1) These are memory leaks I found in my app 2) they are textbook examples of the most common kind of leaks and 3) Facebook is kind enough to open source their tooling for all to improve.

Speaking of open source, if you'd like to spend your time writing open source code from anywhere in the world (👋 from Costa Rica) come join us! We're a bunch of ex-corporate folks on a mission to end pointless meetings & make work meaningful. Check us out at https://www.parabol.co/join or message me directly.

Distributed State 101 - Why I forked Facebook's Relay

Matt Krick — Fri, 12 Jul 2019 14:09:37 +0000

Just over a year ago, I forked Facebook's Relay to fix a bug that caused an incorrect state based on network latency (yikes!). While the concepts of publish queues and distributed state are pretty complex, the bug itself is darn simple and a great foray into distributed systems, which is why I'm using it here to illustrate the fundamentals (and gotchas!) of building a simple client cache. This isn't a slam against Facebook developers; bugs happen & the shackles of legacy code at a mega corp are real. Rather, if it's something that professional developers at Facebook can goof up on, it can happen to anyone, so let's learn from it!

State vs. Transforms

The year is 1999 and I have a counter showing how many people are currently on my fresh new site. If I want that number to update in real time, My server could send 1 of 2 messages:

State: "Hey, the new value is 8."
Transform: "Hey, add 1 to whatever your counter is currently".

State works great for small things like a counter (8), whereas transforms work better for large things like a Google Doc (at position 5, insert "A"). With document stores like Relay, it may seem like a state update (replace old JSON with new JSON), but the server is just sending down a patch that Relay merges into a much larger document tree using a default transform. It then executes any extra transforms in the mutation updater function. The appearance of state makes it simple, the workings of a transform make it powerful. The perfect combo!

Updates and Lamport's Happened-Before

In all client caches, there are 3 kinds of updates: Local, Optimistic, and Server. A local update originates from the client & stays on the client, so it only contains state for that session. An optimistic update originates from the client & simulates the result of a server update so actions feel snappy, regardless of latency. A server update originates from a server and replaces the optimistic update, if available.

In all 3 cases, there's just one rule to follow: apply updates in the order they occurred. If I call an optimistic update, followed by a local update, the optimistic updater should run first, then pass its result to the local updater. This concept was cutting edge stuff when Leslie Lamport published it in 1978! Unfortunately, it's what Relay got wrong.

Instead of processing updates in the order they occurred, Relay processes server updates, then local updates, then optimistic updates. That means even though an optimistic update occurred first, Relay applies it after the local update. That's the crux of the bug.

Let's use that logic in a simple component like a volume slider that goes from 1 to 10. Say the volume is 3 and I optimistically add 1 to it. Then, I locally set the volume to 10. What's the result? If you guessed 10, you've correctly applied Lamport's relation. If you guessed 11, then you've got a broken app and a bright future at Facebook (Kidding. I'm totally kidding. 😉).

A Better Approach

If the current approach isn't mathematically sound, what's the alternative? The answer is pretty easy. Let's take a look at an example publish queue with 4 events:

The above publish queue starts with 4 events: A local update, a server update, an optimistic update, and another local update. It doesn't matter what each update does because as long as they're applied in the order they occurred, we'll end up with the right answer.

In Row 1, we know A & B are deterministic (the opposite of optimistic) so we can commit those, meaning we'll never have to "undo" what they did. However, C is optimistic. If the C from the server is divergent from the optimistic C, then everything following could be different as well. For example, what if D were to multiply the result of C by 2? So, we apply those updates to create a current state, but keep them around in case we have to replay them.

In Row 2, we've got a save point that is the state after A and B have been applied. We've also kept all the events starting with the first optimistic event because they're all dependent on the result coming back from the server. As we wait for that server response, new events like E trickle in. We apply them so that the state is current but also hold onto them.

In Row 3, the server event for C comes back! We remove the optimistic event and replace it with the server event. Starting from the save point, we commit every event until there's another optimistic event. Since there are no more optimistic events, the queue is empty and we're done! It's really that simple. Now, why does C from the server get to jump in the queue? That's because C occurred at the time of the optimistic update, but because of latency, it wasn't received until after E. If you grok that, you grok distributed data types. If you'd like to see what that looks like in code, the package is here: relay-linear-publish-queue. Note that it depends on Relay merging this tiny PR.

With such a simple publish queue, it's possible to compare server events to optimistic events when they come in. If the server event just confirms what the optimistic event suspected, then we can flush the queue without performing a recalculation because we know it's correct. Performance gains to come!

Real World Application

Theory is boring. Now that we understand it, we can get to the fun stuff! With a functioning publish queue, I built an online sprint retrospective for folks like me who don't like conference rooms. If you're not familiar with a retrospective, it's a meeting where teams anonymously write what could've gone better last sprint, group them by theme, and then discuss the important issues. It's a great engineering habit that is slowly making its way into sales, marketing, and executive teams. While building the grouping phase, I didn't want to lose the ability for everyone to participate simultaneously. That meant building a system that could reliably share when someone else picked up and dragged a card:

If you'd like to play around with the demo you can check it out here (no signup necessary) or even view the source code.

Conclusion

I hope this clears up the purpose of a publish queue! If distributed systems sound interesting, this is only the beginning. From here, you can dive into data types such as Operational Transformations (what Google Docs uses) or serverless CRDTs, such as Automerge. If you'd like to get paid to learn about these things while avoiding pants and mega corps, we're hiring a few more remote devs. Reach out.