DEV Community: Chris Opperwall

How to more or less contribute to open source

Chris Opperwall — Mon, 29 Jun 2020 01:01:57 +0000

First off, why?

I just want to start off by saying that contributing to projects can be super rewarding. Maybe there’s this weird bug in a node module that you use at work that you’ve built around a fix for in your own application. If you can find a way to move that fix into the module, not only are you being a good open source citizen by not just consuming that module (at no cost to you or your company, but with all of the utility) but giving back time and energy to a tool or library that has saved your company both the time and resources not having to build themselves. You also can feel good about yourself knowing that every other person who uses that module in the future will be using your contribution! And maybe in 2150 when all software on the internet relies on that module (you know, like left-pad or something) you will truly have reached immortality and cheated death, living on in the electrical signals and wires of the networked world, like some 90s idea of some cyber god/entity. Anyways, where were we? Oh yeah, making some PRs on GitHub. There was a really amazing podcast on Maintainer’s Anonymous with Maggie Appleton discussing open source as a gift economy that discusses how open source can be viewed outside of a market economy or just a water faucet of endless free packages. It resonated with me as motivation for why contributing is one good way to reciprocate the benefit you (or your company) has received from open source software. Having your company sponsor maintainers of open source projects your product depends on is an even better form of reciprocation (maintainers gotta eat), but that’s for a different blog post.

The Journey

What should I know before I get started?

If you take one thing away from this, remember to treat other projects as you would a house that you’re a guest in. If some rando from the internet came to your house, was rude to you, and demanded you let them make changes to your house it’d be totally reasonable to kick them out.

“But I’m totally a chill person, I would never come across that way.”

And maybe you are, but to expand upon the guest in someone’s house theme imagine you visit a friend’s house and upon entering walk straight past an area full of shoes and straight into their living room. Your host is a little annoyed and asks that you take off your shoes before walking around their house. But you wear shoes in your house all the time, so not a problem right? Problem is you’re not in your house, you’re in their house and so far you’ve already started your visit on the wrong foot or shoe or sock or… whatever. Just because you think you’re being a good guest doesn’t mean you are by your host’s standards. What constitutes “not being a jerk” to you might not be the same for them. If only there was a way your host could’ve conveyed how they expected you to behave while you’re in their house…

Code of Conduct

Good news! There’s this thing called a Code of Conduct that a lot of repos have now. They usually have a layout of behavior they expect you to exhibit, as well as negative behavior that they as a project won’t tolerate from you. GitHub even has this nice feature where they provide you a link to one if it exists whenever you make a pull request for the first time on a project. This isn’t a EULA to blindly click through like when you installed iTunes. This is a guide for how you should (and should not) act when participating in this project. The maintainer or maintainers took the time to specify the behavior they expect, so take the time to read it and follow it.

For example, the Octokit organization has Code of Conduct markdown files in each of their repos.

“But this has nothing to do with writing code!”

In my experience, writing software (in both a job setting and an open source setting) involves equal parts, if not more, the ability to treat others with empathy and respect as it does writing the actual software.

How do I even get started?

This is probably the hardest part. How do I know what needs working on? A lot of repos use the standard Good First Issue and Help Wanted, which are used to annotate issues that are a good starting point for new contributors. These are a great starting point, because the maintainer has kept these around to introduce new people to contributing to the project. In larger projects these can be picked up pretty quickly by other people, so sometimes they can be a little hard to find unassigned. Don’t worry though, just because an issue doesn’t have one of these labels means that you can’t work on it, it just might require a little more of your time.

Commenting on the issue with your intent to work on it is also a good start. Someone may already be working on it or the issue might not even be valid anymore, so it’s good to check in with the maintainer before you spend time on a change.

Documentation!

As a new potential contributor, you are totally the right person to find places in the documentation that don’t quite make sense, or could have better examples, or could be more consistent. It can be easy for someone who knows all of the ins and outs of the project to have trouble explaining it to someone new. This is an example of the Curse of Knowledge, which basically means that once you know a lot about a certain subject, you can lose the ability to teach it to someone else. In terms of effort, there’s less upfront time investment than having to set up an entire environment for that codebase when updating documentation. A great way to improve documentation is to find a project that you actually use at work and think of a time you had to spend time figuring out why some aspect of it wasn’t working the way you expected it to work. Couldn’t figure out all of the options a function takes from the documentation or the project’s GitHub README? Had to go searching for why some feature wasn’t quite working the way you thought it would? If you had trouble, chances are other people spent time trying to figure that out as well. Adding extra clarification to a documentation site or a README can mean the difference between a module working as you expect it to and an hour-long (or more) trek through search results and help forums and source code.

Other considerations (style, linting, tests, CI/CD)

When making a code change, try to keep it relevant to the issue you’re fixing. If the project uses semicolons, but you have personally vowed to never use them again in your own projects, don’t remove them from every file you touch. Ideally the project has a formatter like prettier set up to run on pre-commit or in CI, but if not, don’t go rearranging someone’s project just because you happen to like things that way. You’ll likely be asked to revert those changes because the changes will be inconsistent with the rest of the codebase.

If the changes you make break existing unit tests make sure to fix them. The test suite is there to help automate testing each change for any regressions, so it’s imperative to keep it running. If you fix a bug and no there were no tests to catch that bug, you can go above and beyond and add some test cases for that fix to make sure it doesn’t break again!

Make sure to pay attention to the automated checks that run on your pull request. A lot of repos have some continuous integration pipeline through TravisCI or GitHub Actions or whatever that automatically runs your changes through a pass of lint checking, formatting, and unit tests (and sometimes even functional tests!). The time of the person maintaining the repo is just as valuable as yours, so those checks act as a baseline for them to spend time reviewing and giving you feedback. Ignoring those is a pretty surefire way to not get a response to your changes, or at the very least a delayed one.

The Review

After you’ve opened a pull request and made sure you don’t have any outstanding linting, formatting, or test failures, you’ll hopefully get a review from either the maintainer or some other contributors. It’s important to explain what your change does and why the change is being made. If your change is visual it definitely helps to provide some screenshots showing the before and after versions of the software with your fix.

Make sure to follow the Code of Conduct when responding to reviews and criticisms. It’s more than likely the maintainer will ask you to make sure changes to the pull request you originally provide. Don’t beat yourself up over not being perfect; as a new contributor you most likely don’t have the context and historical knowledge that a maintainer would have after working with this project for a prolonged amount of time.

From here there’s kind of two ways things can go. I’ll highlight the happy path first.

After a some back and forth comments and code changes your pull request is merged and you’re a contributor now! Yay!

Another path is that you make a pull request and don’t hear anything for a few days. That’s okay, maintainers are real people with jobs and families and hobbies and lives. Just like how you are volunteering to make some changes or fix some bugs, maintainers are volunteering to maintain the project! If you don’t hear back in a week post a follow-up comment in the pull request and hopefully they’ll see it and you can get some comments and your changes merged. If they don’t respond they might not have the bandwidth to review and merge your changes right now. DON’T spam them on twitter asking for a code review, DON’T email them; you don’t know their life, you don’t know what they’re going through personally at the moment. At the end of the day it’s just software, and software you’re volunteering for at that.

At the same time, if interacting with people who maintain the project leaves you feeling uncomfortable or you don’t feel like you have the time or fortitude to continue working on something at this time, it’s totally fine to walk away. Your mental and emotional health matters and you are absolutely under no obligation to to volunteer/give away work for free if you don’t feel able to or don’t want to. Remember to take care of yourself!

Where do I go from here?

I’d really recommend looking for issues labeled Help Wanted or Good First Issue in open source projects that you use in side projects or in projects at work. You’ll be more likely to get a response from a smaller project that is still somewhat active than trying to make your first PR on the React repo. A good resource for how to contribute and where to contribute to is https://opensourcefriday.com/#participate. Also if you feel comfortable doing so, ask your employer if you can set aside some time periodically to work on open source.

If you have any questions, or if you’ve had different experiences and think I could expand upon this in any way, feel free to reach out on twitter at @copperwall.

Staycation

Chris Opperwall — Tue, 07 Apr 2020 04:53:55 +0000

It's been a weird month. Two months ago if you'd ask me what I'd be doing in late March/early April, I'd have told you that I'd be in Japan for a trip that was in the making for the better part of a year. Times change though, and it became pretty obvious last month that traveling to a foreign country was not going to happen. We kept our hopes up that the situation would level out, that things would stabilize magically in time for us to not have to make any sacrifies and continue living life as usual, but here we are.

Considering the burnout I was starting to feel at work and the PTO that I had already cleared at work for late March/early April, I decided to take half of the vacation I allotted to stay at home and catch up on some personal things that had fallen by the wayside over the last few months. The goals were lofty; finish n books, go running every day, start and finish (like that ever happens) a new and interesting software project. It's always easy to set the bar high when you have multiple days ahead of you (five days is so long, the possibilities are endless!) and feel a little melancholy at the end when you obviously fail to meet every single goal. Looking back though, it was a chill and busy week at the same time.

Cycling

I biked around 50 miles over the week around places like Redwood City and Woodside. The crowning achievement was on Wednesday when I did a 22-mile round-trip from my apartment in downtown Redwood City to Skyline Blvd in Woodside, at the top of one of the ridges between 280 and the coast. I really want to do a ride from 101 to the coast (maybe San Gregorio or something?), and that ride is the farthest I've gone so far. It's not quite halfway to the coast, but it's progress! That was the first ride I tried out my new clipless shoes and I'll have you know I only fell over two times. It was probably more embarrassing than painful, considering both times I was at a busy four-way intersection where drivers in each direction got to witness the majesty of me slowly realizing my feet wouldn't unclip and I helplessly toppled onto the asphault (while still shackled to my bike). In addition to that shining achievement, I went on a couple social distance bike rides up some mean hills in Redwood City by 280.

Books

Books! I haven't really had time to just sink half or an entire day into a book, but I made sure to do that almost every day in the last week. I started Farenheight 451 for the third or fourth time and made sure to finally finish it this time. It was a particularly relevant book to read when I was feeling like I hadn't devoted time to a good book in a while. I also enjoyed that it pivoted from the expected "books are great" narrative into a more general narrative about discource and discussion between people. I tried to balance out between fiction and non-fiction, so I finished It Doesn't Have To Be Crazy at Work by the Basecamp founders. As far as I wasn't reading it for work purposes, I definitely have a few takeaways I can keep in mind for when I go back to work. It was really refreshing to read after working in a start-up environment for the past few months. Most of the takeaways can just be reduced to "things don't have to be crazy", so nice job on the book title.

In a weird fiction/work-appropriate mix of a book, I finally finished The Phoenix Project, which describes the story of a middle-manager promoted to VP of IT in a dying company and bringing modern work practices like agile and continuous delivery to save what seems like a completely sunken ship.

I won't have time to finish it before I go back to work (unless I stay up all night), but I started How Emotions Are Made, by Lisa Feldman Barrett. I'll have to write a little more once I make more progress on that.

Open Source

I love love love contributing to open source projects. Part of me was a little worried that doing work-adjacent things like open source contributions would hurt the burnout feeling more than help, and part of me thinks that I probably should've taken it a little easier than I did, but to be honest it makes me happy, so I did it anyways :P.

I made two pull requests on gatsbyjs/gatsby, one for a build issue in gatsby-source-graphql and a doc improvment for their documentation site. I accidentally spent some of my Friday trying to fix the gatsby-source-graphql after trying to use it on my personal site and realizing that it wasn't quite working. At the end of the day, they opted to take an alternative approach to fixing it than my PR did, but everyone was super friendly and I got to learn a little more about how that particular plugin works. My documentation PR did get merged though, and they gave me a voucher for some free Gatsby socks, so I'll chalk that up as a win. I'm definitely going to be more active in that community. I made a PR today using the information I learned from my gatsby-source-graphql PR experience to make a PR on the gatsby-starter-github-api starter example. Maybe they'll merge it, maybe they won't, but I feel pretty happy about it.

In another vein, I got a PR approved for a hacklang mocking library called hammock that Quizlet maintains, and I made some error-handling features to an Octokit repo that helps out when generating tokens for a GitHub OAuth application. I got a small front-end change in today as well for the documentation site for octokit/rest.js. The maintainer for that org is awesome and makes contributing a pleasure (shout-out to @gr2m).

Also I've been working through this cool blog post about writing a bittorrent client in Go (a language that I have no experience in). I've also been working my way through tour.golang.org to feel a little more comfortable with the language. After spending most of my time in JavaScript for that last two years it's pretty refreshing to try out different ones like hack and go.

Anyways...

Yeah it's kind of a bummer that I didn't get to have the vacation that I was planning on having. Sometimes you've just got to roll with it though. I'd also just like to put out there that in times like these that I feel extremly grateful to have a job that I can continue to do from home and to have a job that I can take a week off of and go back to afterwards.

I'd also like to thank my friends for almost always being available for shenanigans on discord even if I fall off the face of the earth sometimes to do stuff like I was mentioning above.

Also it wouldn't be a quarantine vacation without some homemade sourdough and Animal Crossing.

Shitty Weekend Project: Spongebob Mock Text

Chris Opperwall — Mon, 25 Nov 2019 18:06:06 +0000

Every once in a while it's nice to build something that's dumb or not useful or downright questionable.

This weekend I had an urge to make something that takes some text and returns the text from that one SpongeBob meme where he repeats something said to him back in a mocking way.

You know, this one.

I started off with a new CodeSandbox (which is a great way to mess around or experiment with ideas without having to create and initialize a new project in VSCode or whatever editor you use. It's especially helpful if you're trying out something with React and you don't want to go through the motions of create-react-app.

The first attempt was two functions, mock and toggleCase. toggleCase accepts a single-length string, checks if it's alphabetical (using regex), and toggles the case if depending on if it's already uppercase or not. This could have a little more error handling like checking if the value is a string type, but it generally follows the principle that if you give it invalid input, it'll give you that input right back. I think that's the "garbage in garbage out" principle, but I'm not sure that's the best name for this use case, because some inputs like whitespace and numbers can't be toggled, so they should be returned as is. I'm not sure if that's technically "garbage". If you give the function a string that isn't one character, it returns that string as is, and if you pass a non-alphabetical character it returns it back untouched.

function toggleCase(str) {
  if (str.length !== 1) return str;

  if (str.match(/^[A-z]$/)) {
    if (str.toUpperCase() === str) {
      return str.toLowerCase();
    } else {
      return str.toUpperCase();
    }
  }

  return str;
}

The mock function takes a string value and returns a "mocked" version of that string. So if you pass "Don't use that weird spongebob meme", you'd get back "doN'T UsE ThAt wEiRd sPoNgEbOb mEmE".

It works by iterating over each character in the string and toggling the case if the character's position in the string is even. Since strings in JavaScript are immutable, instead of modifying each character in the string in place, the function splits the string on the empty string, which gives us a list of all characters in the string. With a list of characters we can use Array.prototype.map to iterate over the characters and conditionally apply the toggleCase function to each character based on its position. At the end of the function we join the list of characters back into a single string and return that.

function mock(str) {
  return str
    .split("")
    .map((char, index) => {
      if (index % 2 === 0) return toggleCase(char);
      return char;
    })
    .join("");
}

With these two functions you can take a string of text and return a mocked version of it. So if you wanted to make a certain paragraph tag's contents into a mocked version of the original text you could do

// mock, toggleCase definitions above

// For element <p id="contact">We're happy to hear from you</p>
let contact = document.querySelector("p#contact");
contact.innerText = mock(contact.innerText);

// Element is now <p id="contact">we'rE HaPpY To hEaR FrOm yOu</p>

This is cool and all, but if that <p> has anything like an <a href="/contact">click here</a> within it or any other HTML tags, you'll end up removing the html tags when you run the above code. Ideally you only want to run the mock function on text nodes. By modifying all descendent text nodes, you'll only be modifying the text of each descendent tag.

One possible way to do this correctly is by starting with the root element that you'd like to mock, and recursively all text nodes in its children and its childrens' children and so on. This is totally doable considering the root node is a root of a tree of Nodes, and that can totally be left as an exercise to the reader.

While dealing with this I also found an API called document.createTreeWalker. This API lets you create a TreeWalker, which is effectively an iterator over Nodes in the DOM tree. That tree has a root of whichever Node you pass in as the first argument to document.createTreeWalker, so you could pass in document.querySelector('p#contact') or even document.body. The main method you probably want to use is nextNode, which advances the iterator to the next node and returns that node. The value of the node returned from nextNode is also readable by the currentNode property. The TreeWalker instance starts with the root node as the currentNode property, so it's helpful to call nextNode to get the first Node that matches the NodeFilter.

Okay, back to document.createTreeWalker. The arguments it takes are

root - a Node such as document.body or document.getQuerySelector('p.coolClass')
whatToShow - A constant to determine which type of nodes to include in the TreeWalker instance. Some include NodeFilter.SHOW_ALL, NodeFilter.SHOW_ELEMENT, and NodeFilter.SHOW_TEXT.
NodeFilter - An object that has a method named acceptNode, which consumes a Node and returns a constant value to determine whether or not to include this Node in the iteration. Acceptable return values from acceptNode are NodeFilter.FILTER_ACCEPT, NodeFilter.FILTER_REJECT, and NodeFilter.FILTER_SKIP. The difference between FILTER_REJECT and FILTER_SKIP are that a rejected Node will not have its children traversed, while a skipped Node will.

So if you wanted to have a TreeWalker that returned all Text nodes in the document, you could do document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT). If you wanted to take that further and only receive text nodes with word characters (a-z, 0-9, _), you could do

document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT, {
    acceptNode(node) {
        if (!node.data.match(/\d+/))
            return NodeFilter.FILTER_REJECT;
        return NodeFilter.FILTER_ACCEPT;
    }
});

There's a lot you can do with this API if you have a need to walk over the document tree and don't want to implement you're own recursive logic to do so.

To take this all the way back to the SpongeBob mock example, we can make all text on a website to be the mocked version with this much code.

// mock and toggleCase definitions above

function mockDocument(document, root = document.body) {
  let walker = document.createTreeWalker(root, NodeFilter.SHOW_TEXT);

  while (walker.nextNode()) {
    let text = walker.currentNode;
    text.replaceData(0, text.length, mock(text.data));
  }
}

So what's the point?

Basically I'm trying to convey that even if you're building something that doesn't seem that important, you can find some useful piece of knowledge that you can add to your toolbelt as a developer.

Also I published this to npm under mock-text-node if you want to use it in your own projects.

Bonus

I tried this out on the Gatsby docs I think this worked out pretty well.

Additional Resources

While reading about TreeWalker, I also learned about an alternative traversal method called NodeIterator. A NodeIterator can be created by calling document.createNodeIterator, which has almost the same interface as document.createTreeWalker, but without an optional fourth argument, entityReferenceExpansion. The methods on a NodeIterator instance only include nextNode and prevNode, while TreeWalker include additional methods to access the current Node's parent, children, and siblings.

I wasn't able to find any documentation on MDN explaining the main different between the two of them, but in the W3C spec on Document Object Model Traversal explains in its Overview section.

NodeIterators and TreeWalkers are two different ways of representing the nodes of a document subtree and a position within the nodes they present. A NodeIterator presents a flattened view of the subtree as an ordered sequence of nodes, presented in document order. Because this view is presented without respect to hierarchy, iterators have methods to move forward and backward, but not to move up and down. Conversely, a TreeWalker maintains the hierarchical relationships of the subtree, allowing navigation of this hierarchy. In general, TreeWalkers are better for tasks in which the structure of the document around selected nodes will be manipulated, while NodeIterators are better for tasks that focus on the content of each selected node.

So for our example we could actually use a NodeIterator instead, considering we don't need to access the hierarchical structure of the document as we iterate through text nodes. We only need access to the next node in the traversal.

I hope you learned something helpful from this. Feel free to tweet at me at @copperwall if you have any comments or questions.

Introduction to ES2015 Proxy

Chris Opperwall — Mon, 19 Aug 2019 16:23:42 +0000

I was playing around with some Object.defineProperty stuff at work today and I thought I'd give Proxys another look since I was knee deep in setting getters and setters and enumerable and writeable fields. It put me in the mood for some JavaScript stuff.

The Proxy object in ES2015 lets you create a new object by combining a target object (the original object) and a handler spec. A handler spec is an object with methods that are called when certain actions are taken on the returned object from new Proxy(target, handler). When I say object I don't just mean something like { key: value }, functions are also objects, as well as things like new String("hello"). MDN, as always, has a wonderful reference of all of the possible handler methods you can add. Some pretty cool ones that stand out to are handler.apply(), handler.construct(), handler.set(), and handler.get().

One silly example to start off with is creating a loud object that console.logs any time a value is set or accessed from it. Our handler spec includes get and set methods, which are called whenever those particular actions happen to our object. This example prints out what happens to the object and outputs the stack trace to see where exactly the operation happened. This is kind of a fun trick if you want some more information about how an object is being used in your code.

It's important to note that the target argument passed to your Proxy handlers is the original object passed to the Proxy constructor. It's not a copy, it's the same reference. Any mutations or alterations you do on that object will affect the original object, which may introduce bugs if you're not careful.

const myObject = {};
const loudObject = new Proxy({}, {
  get(target, p) {
    console.log(`Accessing key ${String(p)} at ${(new Error()).stack}`);
    return target[p];
  },

  set(target, p, value) {
    console.log(`Setting key ${String(p)} to ${String(value)} at ${(new Error()).stack}`);
    target[p] = value;
    return true;
  }
});

// "Accessing key hello at Error
//    at Object.get (/Users/user/projects/proxy/index.js:21:62)
//    ..."
loudObject.hello;
// "Setting key hello to woop at Error
//    at Object.get (/Users/user/projects/proxy/index.js:21:62)
//    ..."
loudObject.hello = 'woop';

myObject.hello // 'woop'

One interesting application for proxies is creating mock or spy functions for testing. If you've ever used Jest, Sinon, or another test framework that adds support for spies and mocks (Mockito is another one outside of JavaScript land), you've probably created a mock function in order to assert what that function was called with when used in your application code.

The example below shows how to make a test.spy method that takes a function and returns a proxied function which keeps track of the list of arguments and return value for each function call to the proxied function. It's important to note that the return value of test.spy acts the same as the original function passed in, but it has a special handler that runs when the function is called that records metadata about that call. You can use Proxy to instrument your functions without the need for your calling code to be aware of that.

const test = {
  spy: function(fn) {
    const calls = [];
    return new Proxy(stuff, {
      apply(target, thisArg, args) {
        const result = target.apply(thisArg, args);
        calls.push([args, result]);
      },

      get(target, p){
        if (p === 'calls') {
          return calls;
        }

        return target[p];
      }
    });
  }
};

function stuff(arg1, arg2) {
  return `${arg1} ${arg2} cool`;
}

function doSomeStuff(fn) {
  return stuff(1, 2);
}

const spyStuff = test.spy(stuff);

spyStuff("hello", "cool");

doSomeStuff(spyStuff);

const calls = spyStuff.calls;
assert(calls[0][0][0] === "hello");
assert(calls[1][1] === "hello cool cool");

Proxy can be a pretty powerful abstraction, but I haven't seen it used in code too much, so I don't have a good idea it's useful and when it might actually be overcomplicating your program. Adding it for debugging and testing use cases sounds pretty novel, but I don't know what the performance cost is for using that abstraction, so I'm not sure if it's the best to use in production code.

If you've come across any cool use cases for Proxy I'd be really interested in hearing about them! I'm going to try to make a tool that uses Proxy to persist a JS object in S3 through a Proxy with get and set handlers, which will probably make it's way into my next post.

Make shitty versions of software libraries you use

Chris Opperwall — Tue, 13 Aug 2019 14:15:04 +0000

Read through software projects you use

I've been spending some time over the last few weeks picking some software projects that I use and attempting to write small simple versions of them. It's been a pretty neat way to learn how or especially why certain things work the way they do.

The biggest example of a software project that I've used professionally for years is express. I've used express for practically any web project I've built with node, and chances are you've used it too if you're a backend node dev. Maybe sometimes you've questioned how the whole middleware pattern even works, or why you have to call a callback function instead of returning a Promise to advance to the next middleware function. Maybe you've been stepping through your web application and you find yourself in node_modules/express/lib/router/index.js and you can see the matrix and somehow you now know kung fu. Ending up in a node_modules file might be kind of intimidating, but if you look a little further you might realize that that scary module code is easy enough or small enough for you to grasp if you take a little time out of your day to read it.

For example, express only has 12 files for all of the request, response, routing, view rendering, nesting, and middleware chaining functionality it has. It is totally possible to read through express in a weekend. You probably won't understand all of it right off the bat, but you'll learn some neat implementation details and you can always take notes of parts that you don't quite understand to go back to later. The first time I was reading through the router logic, I came across this kind of weird detail where if you register a middleware function that has more than three arguments, express just silently skips it and moves on to the next middleware function. They don't write a debug message if you're in development mode, or even call the function knowing that the fourth argument will just have to be undefined (which sounds like a more JavaScript-y thing to do to me). Would you have done that differently if you were writing an application library? Up until that point I was reading through the source and looking at it like "yeah I guess that makes sense" or "oo cool I wouldn't have thought of that", but that was the first instance where I thought that I'd rather have done something differently. Which leads me to my next point...

Rewrite software projects you use (AKA make shitty versions of them)

I believe that reading through software projects and libraries that you use is helpful to have a better understanding of how the whole application works, but I think the next step in that journey is to pick some small libraries and make really simple or shitty versions of them. Continuing on with the express example, you can make an express clone that only implements app.use. Doing just that gives you an end product where you can handle all of the routing and rendering outside of the framework. In order to get to that point you need to be able to create an application object that can have middleware registered, you have to decide how you want to store middleware, how to chain each of the functions together and pass in a next function that will eventually call the next middleware. You'll also need to figure out how to run an HTTP server to get the Request and Response objects/streams that you'll pass to your registered middleware functions. As a next step maybe you can add some default middleware functions like express does to handle query string parsing and body parsing. Do you want to build in functionality for automatically parsing JSON request bodies or do you leave that up to the user? It's totally your call. Maybe you can parse the query string and let the user access it as SearchParams instance instead of a plain old object. Something you can aim for is to replicate the library's interface and then try dropping it into a project that uses the reference implementation. You can get a lot of insight from drop-in testing it and see which parts break or aren't implemented yet. I've been trying this out with Rapid, an express clone. Is it going to replace express? Hell no. Am I going to use it instead of express for all of my personal projects? Probably not. It's been fun so far and there's plenty of more functionality to add or recreate. Also if you feel like making a PR on Rapid they're more than welcome.

Don't just rewrite the projects verbatim, though. Copy the interface but try the implementation out for yourself, maybe you'll get the opportunity to arrive at the same decision crossroads the original implementors did. You could end up making a different choice or at least appreciating the different ways you can tackle the problem. You can always go check your work against the reference afterwards and improve it if the reference has a better solution. It probably will if it's a large project like express, but maybe you'll come up with a better solution! Big projects can be encumbered with complexity from maintaining backwards compatability for features that you might not need or care about.

Another cool project to recreate is DataLoader. DataLoader is a tool to batch load and cache resources. You make an instance of it by constructing it with a batch function that takes some keys and returns a Promise that resolves to the values that those keys relate to. The batch function could run a SQL query or hit a REST API or some other data source. The user interacts with the instance by calling a load function with a single key and that returns a Promise which will resolve to the key's value when the batch function successfully loads the data. The cool part is that DataLoader schedules the batch function to run after the current frame of execution by using some Node asynchronous primitives and caches the results. If you ask for the same resource multiple times throughout your web request lifecycle, DataLoader will only load it once. Also you can ask for individual resources throughout the request lifecycle and DataLoader will batch load them at the end of each frame of execution.

DataLoader's implementation is only one file (excluding tests) and there's even a YouTube video from one of the creators that covers the entire source. Writing your own implementation of that means you'll have to cover concepts like creating promises, caching, using things like process.nextTick for asynchronous operations and deciding how to handle errors from invalid user entries like bad batch functions.

Hopefully this left you with some projects in mind that you use often, but don't really understand how they work under the hood. If you have any other project recommendations that would be interesting to read through or rewrite feel free to tweet me at @copperwall.

My Experience at Facebook's F8 2019

Chris Opperwall — Sat, 15 Jun 2019 20:54:34 +0000

I (semi-)recently went to Facebook's Developer Conference, F8, in San Jose, CA. It was the first conference I've gone to since I went to GitHub Universe 2017 on behalf of iFixit, and the first time I've gone to a conference on behalf of PayPal. Or at least I guess I sort of went on behalf of PayPal. I asked my manager if I could go because some of the talks on React and GraphQL sounded pretty cool. React is definitely the present and future of front-end development at PayPal (and the general web?) and we have a pretty strong group of people at the company advocating for going in on GraphQL. So I thought learning more about those at the conference would be fun for me, and beneficial for my job.

Anyways, I totally meant to write a recap of this after I went, but life happens, and work things go crazy sometimes. I'm camped out in a hip coffee/donut shop in beautiful San Luis Obispo, so I thought I'd start a pomodoro timer and see what I can bust out in that time.

Going to a conference alone

So this was the first time I've gone to a conference without friends or coworkers. I didn't know of anyone else who was going, so I was completely on my own. In one sense this was great; I could go to whatever talks or sessions I pleased without having to worry about what the rest of a group was going to do. On the other hand, it was pretty odd going to lunch or happy hour and not knowing a soul. It looked like people had already either arrived in groups or had somehow found people that they jived with, but I found it kind of intimidating. So I didn't really go to too many of the happy hours or social events after the conference. I also live like 15 blocks from the convention center, so it was kind of easy to just walk home if I didn't feel like being there.

Next time I go to a conference alone I definitely need to try and reach out to people a little more. I'm not really sure how to do that, but I've started going to meetups in the city more often, so I'm trying to put myself in situations where I don't know people and have to find something in common or something interesting to talk about. This is especially true if I end up actually travelling for a conference. If I go to some event in San Jose or even San Francisco it's pretty easy to just go home afterwards. But if I end up going to some React or GraphQL conference out of state or out of the bay area (which I totally want to do after going to F8) it'd really be worth my while to meet other developers who travelled to go to that event as well.

Different parts of F8

The Keynote

The keynote was fine. It sounds like they're working on a bunch of cool things. They mentioned privacy in basically every other sentence. Based on their history who knows if that's genuine. I don't really use Facebook that much so I wasn't really that taken by the keynote. It was kind of interesting seeing Mark Zuckerberg on stage in the flesh.

Workshops

They had a pretty cool workshop area set up where you could sit at long tables with your laptop and follow along people giving demos of Facebook tools like PyTorch, Docusaurus, React Hooks, GraphQL, etc. The Docusaurus and React Hooks workshops were really nice to attend. I had no idea that Docusaurus was used by so many different places like Jest or even Smash.gg. The React Hooks intro workshop went over some hooks basics and how to go about making custom hooks, which I had never tried before.

Team Meetups

A big portion of the conference area was set aside for the Facebook Open Source tools, where you could just walk up to someone who works on React or someone who works on WebXR and just ask them questions or talk to them. They also had slightly more structured times to meet with teams specifically. I went to multiple events where members from the React core team would just stand in a circle with everyone who attended and make jokes and took questions from people. To my surprise I had the courage to ask them some hooks questions and React Native Web questions in front of everyone and they were really candid and thorough about answering them.

I can't really express how nice they were. I had a few minute conversation with some of the team members on the first day, and when I came back to the open source area on the second day they remembered me and let me ask even more questions! I got to ask a core team member about data loading with suspense and they talked about some issues they were currently having internally and some things I could try out if I wanted to use some not quite stable APIs. I honestly left more questions than when I started, but it so great getting to pick the brains of people who work on a tool that I use basically every day. They were so friendly and happy to help as well.

This was obviously the highlight if you couldn't tell by how stoked I am about how it went.

Would I go back?

I'm not really sure if I'd go back to F8 next year. The react meetup was an outstanding experience, but I wonder if it'd just be better to go to React-specific conferences like React Conf or React Rally or any of the other myriad of React conferences.

The talks that were given at the conferences were really informative, but I could have probably gotten the same benefit by watching them on YouTube.

If I went back I'd definitely want to bring a coworker or friend to feel a little more socially comfortable about talking to other people. Or I should probably just make it a priority to meet new people and see what they're working on at their respective companies and network a little.

Reading through a project: Formy

Chris Opperwall — Tue, 09 Apr 2019 16:11:10 +0000

Sometimes it helps to take a software project and just read through the source code. If the documentation is good enough or the interface is simple enough you can probably getting away with not knowing how most of the project works, but sometimes it’s kind of nice to look a little deeper.
I’ve used a React form library called Formy for a few projects at iFixit. Formy lets you configure a form using JavaScript objects and apply that configuration to a view using React components. The documentation has a lot of examples, which makes it really easy to get up and running with it, but to be honest I don’t really understand how most of it works. Here’s my attempt at learning a little bit more about it.

Where to start

It’s probably not a bad idea to start looking in the entrypoint of the module. In the package.json file that’s specified by the main field of the JSON document. For Formy, it’s dist/Formy/Form.js, but that file doesn’t show up in GitHub. The dist directory is the result of a build step which converts each file in the src directory to an ES5 target, so it’s safe to say that we can treat src/Formy/Form.js as the entrypoint. The src/example and src/index.js directories and files are only used for documentation and development, so those can be ignored.

Exports

Form.js is responsible for exporting functions and data that users of the library can access. The file specifies a default export named Form, which is an object which contains named functions. It doesn’t look like Form has any state or prototype (apart from the Object prototype), so the functions it holds can be viewed as static functions and can be looked at individually.

Form.Component

Form.Component = ({ id, name, onSubmit, children }) => (
   <form
      id={id}
      name={name}
      onSubmit={onSubmit}
   >
      {children}
   </form>
);

Form.Component.propTypes = {
   id: PropTypes.string,
   name: PropTypes.string,
   onSubmit: PropTypes.func,
};

Component is a functional React component that takes id, name, onSubmit, and children as props. The return value of that functional component is a form with those props applied to it. Any child components that are included within Form.Component are passed through to the form component. That’s probably used for including form inputs or submit buttons as children to a form.

Component seems like a kind of general name for a React component. Maybe it would be better to name it Form, because it wraps an actual form JSX tag.

Form.Field

Form.Field is defined in a separate file, so I’m not totally sure what that means yet. Why is FormField in a different file, but not Form.Component? That might make things seems a little more consistent. We can revisit this later after going through Form.js.

Form.customValidityFactory

Form.customValidityFactory = (constraint, validationMessage = 'Invalid') => (...args) => (
   constraint(...args) ?  '' : validationMessage
);

A function that takes a constraint and validation message and returns a function that takes a variadic number of arguments and applies its arguments to the constraint function provided in the first function and returns either empty string if truthy or validationMessage if not. Maybe it’d be cool if the custom validity factory let the validity constraint function return its own error message, and not just empty string vs validationMessage. Would that allow multiple validation messages?

The end result of the customValidityFactory is to call setCustomValidity on the form input with the string result from calling the constraint function on the arguments passed to the resulting function. However, this happens in the component library and not in Formy itself. Formy assumes that passing a customValidity property to an input component will handle that properly, so that’s important to know if you want to include your own component library to use with Formy.

Form.fields

Form.fields = (globalProps = {}, fields) => Object.assign({},
   ...Object.entries(fields).map(([fieldKey, field]) => ({
      [fieldKey]: {
         ...Form.Field.defaultProps,
         ...{ name: fieldKey },
         ...globalProps,
         ...field,
      },
   })),
);

Function that takes globalProps and an object of field definitions. Global props are useful for when you want to use the same onChange handler. The global field props will be applied to any field component, unless overridden by the individual field itself. Setting a global onChange prop to update state whenever any form field is changed is a good example of a global prop. The return value of this function is an object with form input name keys that map to an object with properties for that form input.

Form.getData

Form.getData = form => Object.assign({},
   ...Object.entries(Form.getProps(form).fields)
   .filter(([fieldKey, field]) => !field.disabled)
   .filter(([fieldKey, field]) =>
      !['checkbox', 'radio'].includes(field.type) || field.checked
   )
   .map(([fieldKey, field]) => ({ [fieldKey]: field.value })),
);

Wow, this function is kind of dense. The gist looks like it returns an object with data from the value of each form field, but does not include unchecked radio or checkbox fields or disabled fields. The shape of the returned object is field name keys that map to the value of that field. This is particularly helpful if you want to get input data out of the form for submitting.

Form.getProps

Form.getProps = form => Object.assign({},
   ...Object.entries(form)
   .filter(([formPropKey, formProp]) => formPropKey !== 'fields')
   .map(([formPropKey, formProp]) => ({
      [formPropKey]: formProp instanceof Function ? formProp(form) : formProp,
   })),
   {
      fields: Object.assign({}, ...Object.entries(form.fields).map(([fieldKey, field]) => ({
         [fieldKey]: Object.assign({}, ...Object.entries(field).map(([fieldPropKey, fieldProp]) => ({
            [fieldPropKey]: fieldProp instanceof Function ? fieldProp(form, fieldKey) : fieldProp,
         }))),
      }))),
   },
);

form.getProps goes over all non “fields” fields and if the value is a function, calls it with the form. An example of a prop like this is the return value from Form.onSubmitFactory, which expects a form and returns an event handler that goes on the actual form. The “fields” field maps each form field name, and for each form field prop, if it’s a function it passes the form and the fieldName to the function value. A good example of this is Form.customValidityFactory, which takes a constraint function and returns a function that takes a form and fieldKey, which is called by Form.getProps.

For all the ES6+ magic going on here, we’re basically mapping an object full of form level props and transforming properties that are functions by applying them with the form object and a fieldKey (if it’s a form field property).

Wow there’s a lot going on here. From examples it looks like this returns a list of props that can be passed to Form.Component and Form.Field in user component’s render method.

This function (and Form.getData) makes pretty heavy use of Object.assign. What does Object.assign actually do?

Object.assign is like an object spread operator. The first argument is the target object and all other arguments are sources to copy fields from into the target object. Later source properties override earlier ones. It looks like most of its uses use an empty target object and a list of sources from global to more specific properties. Object.assign can also take a source that is an array of objects and it will merge those together and then copy those into the target object.

The project’s babelrc specifies using the transform-object-rest-spread plugin, so maybe those *Object.assign*s can be converted to use the object spread operator.

Form.onChangeFactory

Form.onChangeFactory = fn => (form, fieldKey) => updatedProps => fn({
   ...form,
   fields: {
      ...form.fields,
      [fieldKey]: {
         ...form.fields[fieldKey],
         ...updatedProps,
      },
   },
});

A function that takes a handler function fn, which returns a function that takes a form and fieldKey, which returns a function that takes an updatedProps object, which applies the handler function to a merged object with form as a base, an overridden fields key with the keys from form.fields with the fieldKey key overridden by the updatedProps object.

The example handler function receives a new form object with the updated fields and calls setState with that new form state. Kind of interesting that you have to specify that in order for the form to work. Maybe it could be a nice default.

Form.onSubmitFactory

Form.onSubmitFactory = fn => form => ev => {
   ev.preventDefault();
   fn(Form.getData(form));
};

A function that takes a handler function fn, which returns a function that takes the form object, which returns a function that takes an event, which I would assume is the submit event. That function prevents the default behavior of the submit event, calls the handler function of the result of calling getData on the form object. This is useful for specifying what to do when the form is submitted, such as sending off an AJAX request or creating some action with the form data.

The resulting function from calling Form.onSubmitFactory is used as the value for the onSubmit key in the form state. The Form.Component component needs a onSubmit function that takes an event. In order to convert the onSubmit function in the form state into the onSubmit function prop, call From.getProps on the form state. This will supply the form state to the onSubmit function in the state, which takes a form and returns a function that takes an event. The result from calling that function will.

FormField.js

import React from 'react';
import FormFieldPropTypes from './FormFieldPropTypes';
import FormDefaultComponentLibrary from './FormDefaultComponentLibrary';

const FormField = ({componentLibrary, ...props}) => {
   const Component = componentLibrary[props.type];
   return <Component {...props} />;
}

FormField.defaultProps = {
   checked: false,
   componentLibrary: FormDefaultComponentLibrary,
   type: 'text',
   value: '',
};

FormField.propTypes = FormFieldPropTypes;

export default FormField;

So FormField isn’t actually that complicated. FormField is functional React component that accepts componentLibrary and type props along with additional props. The type prop given is used as the key in the componentLibrary object to grab the component from, the return value is the JSX of that component with the props given to FormField.

FormField specifies some defaultProps such as checked, componentLibrary, type, and value. Checked is false by default, componentLibrary is Toolbox by default, type is text by default, and value is empty string by default. Not too weird for defaults.

FormField’s propTypes are imported from the FormFieldPropTypes.js file. Maybe that’s something that would be better specified by the component library? I’m not sure.

Implementing Promise.all

Chris Opperwall — Tue, 19 Mar 2019 04:59:17 +0000

Promise.all is a static method on the Promise object that takes a list of items and returns a promise that resolves with a list containing the values of all resolved values in the input list. If any of the values are rejected promises, the returned promise will also be rejected with the rejection message of the promise that is rejected first. This is particularly helpful for when you want to run multiple promises concurrently, but wait until all of them have been fulfilled before continuing.

If you're using promises directly in your code you may write something like this to make multiple concurrent requests to different API endpoints, and wait until all have completed to operate on the responses.

Promise.all([
    fetch('/api/a'),
    fetch('/api/b'),
    fetch('/api/c')
]).then([responseA, responseB, responseC] => {
    // Use the responses from all three async requests.
});
~~~{% endraw %}

You can also use Promise.all in async functions if you want multiple async function calls to operate concurrently instead of running them sequentially. In the following code snippet two network requests are made, but the second isn't initiated until the first one completes. This means that if the first request takes five seconds and the second request takes five seconds, the section after the requests have completed will have to wait for the sum of the request times to run.{% raw %}

~~~js
// This waits for the result of the first fetch
// before starting the second fetch.
async function doSomeThings() {
    const result1 = await fetch('/api/a');
    const result2 = await fetch('/api/b');

    return {
        ...result1,
        ...result2
    };
}
~~~

In this example both network requests are initiated at the same time. If both requests take five seconds, the section after the requests have completed will have to wait for the maxiumum of the request times to run.

~~~js
// Both fetches are initiated concurrently, but the function waits for
// both to complete before continuing.
async function doSomeThings() {
    // Because we don't await the fetch,
    // we get the promise, not the resolved value.
    const resultPromise1 = fetch('/api/a');
    const resultPromise2 = fetch('/api/b');

    // We can await the resulting promise from the
    // two fetches before continuing.
    try {
        const [ result1, result2 ] = await Promise.all([resultPromise1, resultPromise2]);    
    } catch (e) {
        // If either request fails, we can catch the error.
        debug('There was an error', e.message);
    }

    return {
        ...result1,
        ...result2
    };
}
~~~{% endraw %}

It's important to remember that if any of the values in the list passed to Promise.all settle to rejected promises, the entire result will be a rejected Promise.

I thought it'd be neat to go over some possible implementations for Promise.all.

## Recursive Solution

Promise.all can be implemented using a recursive solution. The base case is when Promise.all is called with an empty array, in which case it returns a promise that resolves to an empty array. Otherwise it take the resolved value of the first item in the list and calls Promise.all on the rest of the elements in the list.{% raw %}

~~~js
Promise.all = function promiseAllRecursive(values) {
    // Base case.
    if (values.length === 0) {
        return Promise.resolve([]);
    }

    const [first, ...rest] = values;

    // Calling Promise.resolve on the first value because it could
    // be either a Promise or an actual value.
    return Promise.resolve(first).then(firstResult => {
        return promiseAllRecursive(rest).then(restResults => {
            return [firstResult, ...restResults];
        });
    });
}
~~~{% endraw %}

## Iterative Solution

For the iterative solution, you'll want to return a new promise that only resolves once each of the values of the array provided has resolved, and rejects if any of the promises reject.

The executor function given to your function can keep track of the results as each promise resolves and keep track of the number of promises that have resolved. You can use a for loop or a forEach to iterate over the list of values and call the _then_ method on each of them, adding the result to the results list as they resolve. It's important to remember that Promise.all maintains the order of the results from the promises provided as input, so you can't just append to the results list whenever a promise resolves. You'll need to know the index of the promise that is resolving in order to know where to place it in the results list. In the example I'm doing this by taking the _index_ argument to the _forEach_ callback.{% raw %}

~~~js
Promise.all = function promiseAllIterative(values) {
    return new Promise((resolve, reject) => {
       let results = [];
       let completed = 0;

       values.forEach((value, index) => {
            Promise.resolve(value).then(result => {
                results[index] = result;
                completed += 1;

                if (completed == values.length) {
                    resolve(results);
                }
            }).catch(err => reject(err));
       });
    });
}
~~~

## Reducer Solution

Yet another way to implement Promise.all is to use a reduce function. The initial value for the reduce function will be a Promise that resolves to an empty list, in a similar fashion to the base case to the recursive solution. Our reducer function will take an accumulator, which will be a promise that will resolve to all of the results of the resolved values so far, and a value argument, which is the current value in the iteration on the list of values (promise or not) to Promise.all. The reducer function should return a new promise that will resolve to the list of results that the accumulator will resolve to, as well as the result that the current value will resolve to. As the reducer iterates over the list of values, each return value will be a promise that resolves to a larger subset of the results of the values passed to Promise.all.

We don't need to explicitly handle catching promise rejecting because the promise we return will be implicitly rejected.{% raw %}

~~~js
Promise.all = function promiseAllReduce(values) {
    return values.reduce((accumulator, value) => {
        return accumulator.then(results => {
            return Promise.resolve(value).then(result => {
                return [...results, result];
            });
        });
    }, Promise.resolve([]));
}
~~~{% endraw %}

## Implementations from Promise libraries

### Bluebird Implementation

[Bluebird](http://bluebirdjs.com/docs/getting-started.html) is a pretty common Promise library to use outside of the Promise implementation provided by most JS environments. Their [design principles](http://bluebirdjs.com/docs/why-bluebird.html) include taking the pragmatic or performance-oriented approach over elegant solutions, so it should be interesting to see how they implement Promise.all.

Bluebird's implementation of Promise.all works by creating a [_PromiseArray_](https://github.com/petkaantonov/bluebird/blob/master/src/promise_array.js) instance and returning a promise from that, so it looks like most of the implementation details will be involved with initializing a PromiseArray from a list of values.

#### [PromiseArray](https://github.com/petkaantonov/bluebird/blob/master/src/promise_array.js#L19-L30)

The PromiseArray constructor takes a _values_ parameter, which can either be an Iterator of any value (this includes both promises and concrete values), or a Promise that resolves to an Iterator. The constructor sets up the following instance fields
* _promise (a new promise that'll be used to return from PromiseArray.prototype.promise())
* _values (the values passed to Promise.all)
* _length (initialized to 0, set to the length of values later on)
* _totalResolved (initialized to 0, incremented on promise fulfillment)

After initializing these fields, the constructor calls the __init_ method.

#### [_init](https://github.com/petkaantonov/bluebird/blob/master/src/promise_array.js#L41-L86)

The __init_ method does some setup and error checking on the _values_ input. It checks if _values_ is a promise and sets up the promise to call __init_ when the promise resolves. It will also check if the promise is already rejected or fulfilled, so that it can either immediately reject the return promise or immediately set the _values field to the fulfilled value of the input promise.

If the _values_ argument isn't a promise, Bluebird tries to convert it to an array. If the conversion fails, the return promise is immediately rejected. If the list is empty, the return promise is immediately resolved with an empty list.

If the _values_ argument is a list with more than zero elements, the list is passed to the __iterate_ method.

#### [_iterate](https://github.com/petkaantonov/bluebird/blob/master/src/promise_array.js#L88-L127)

The __iterate_ method does a lot of the heavy lifting for PromiseArray. Each item in the _values_ argument is passed to _tryConvertToPromise_. If the result is a promise, a local bitField variable is set to the bitField of the promise, otherwise the bitField is set to null. The bitField is later used to determine the resolution status of the promise (i.e. whether it's pending, fulfilled, or rejected).

Following that logic, there are cases to handle what to do based on the promise's resolved status. If the promise is already fulfilled, Bluebird calls __promiseFulfilled_ with the fulfilled value. Similarly if the promise is already rejected, Bluebird calls __promiseRejected_ with the rejection reason. __promiseFulfilled_ and __promiseRejected_ are similar in that they both increment the __totalResolved_ field from earlier, but they differ in that __promiseRejected_ immediately rejects the resulting promise, while __promiseFulfilled_ only resolves the promise if the number of resolved promises is greater than or equal to the length of values given to _PromiseArray_.

Something that tripped me up a little was not seeing how promises that weren't yet resolved were handled. There's a small case for *IS\_PENDING\_AND\_WAITING* promises, which just calls __proxy_ on that promise and places the promise in the values list as a temporary placeholder. __proxy_ takes the _PromiseArray_ and an _index_ as arguments sets up the _PromiseArray_ as a receiver on the pending promise. When the pending promise settles, Bluebird checks to see if it has a proxyable receiver and calls __promiseFulfilled_ or __promiseRejected_ on the receiver object. Assuming all pending promises eventually are fulfilled, the promise returned from PromiseArray will resolve when the last pending promise resolves. When any of the pending promises are rejected, the promise returned from PromiseArray will reject as well.

Hopefully that was kind of interesting. The solution implemented by Bluebird obviously isn't as "elegant" or "simple" as some of the recursive or reducer solutions we introduced earlier, but I thought it was interesting to see some of the optimizations they chose to take. The idea of proxying promise resolutions of each of the individual promises back to the returned promise seemed like a pretty simple way of handling a list of promise resolutions. I suppose you'd have to build in the idea of a receiver promise into your promise implementation, so I'd be interested if Bluebird has other uses for that functionality outside of Promise.all.

"Why Transducers Are Cool"

Chris Opperwall — Tue, 19 Mar 2019 04:48:19 +0000

I mostly got the inspiration for writing this post after reading the Appendix A of Functional-Light JavaScript, which explains the concepts of transducers. This is the third of fourth time I've gone through that appendix and I feel like I'm actually starting to grasp the concept, so I thought I'd write about it to grasp it a little better.

This post is going to include some functional programming concepts like function composition and currying, as well as some common list operations like map, filter, and reduce. I'll try to include a little background, but this mostly assumes you're familiar with those concepts.

Credit

This is mostly a rehashing of Appendix A of Functional-Light JavaScript, which is a super great book on practical functional programming in JavaScript. I definitely recommend getting that book if this is interesting to you. Also if this post doesn't totally make sense, the book should be a really good resource and will probably explain these concepts better.

Function Composition

Function composition is the idea of creating new functions by composing, combining, or chaining multiple other functions together. You can think of it like using multiple functions as building blocks or LEGOs and creating a new structure by combining those smaller pieces together. Composition is also idea used frequently in shell scripting, in which multiple commands with specific purposes are able to be easily combined to make pretty powerful programs.

In this post we'll look at compose, a higher order function which takes a variadic list of functions are arguments and returns a new function such that the input to that return function is passed to the last function in the argument list and that function's return value is passed as input to the second to last function in the argument and so on.

Here's a small example to illustrate.

import { compose } from 'lodash/fp'
const addOne = x => x + 1
const isEven = x => x % 2 == 0

const isPlusOneEven = x => isEven(addOne(x))
const composed = compose(isEven, addOne)
~~~{% endraw %}

In the above example, _isPlusOneEven_ and _composed_ are equivalent functions which both take a number, add one to it, and then return a boolean if that result is even or not. When using _compose_, the functions are executed from right to left. It's helpful to visualize the order they're shown in the _isPlusOneEven_ implementation.

## Composing mapping functions

One interesting property of composed functions is that they can be used to consolidate multiple calls to _map_ on an array.{% raw %}

~~~js
const resourceIds = [1,2,3,4,5];

const buildResourceURL = id => `https://api.example.com/${id}`
const fetchResource = url => fetch(url)

let responses = resourceIds.map(buildResourceUrl).map(fetchResource)

// This an intermediate list returned from the first map can be avoided by composing the two mappers
// The result should be the same as before (barring any side effects)
const getResourceFromId = compose(fetchResource, buildResourceUrl)
responses = resourceIds.map(getResourceFromId)
~~~{% endraw %}

By composing the two mappers, _resourceIds_ can be iterated over only one time instead of two, and any intermediate lists created by separating mapping steps are avoided. So anytime there's multiple chained map calls on an array, you can condense that to a single map call with all of the mapping functions composed together.

Mapping functions can be composed because they have the same "shape". A mapper function takes a single argument and returns a single new value. The return value from a mapping function can be easily accepted as an argument for another mapping function.

## Trying to compose filter and reduce

Some other common list operations include _filter_ and _reduce_. _filter_ takes a predicate function (a function that returns true or false) and returns a new list that only includes elements that caused the predicate function to return true when that value was passed to it. _reduce_ takes a reducer function and an optional initial value. The reducer function takes a accumulator parameter and a current element parameter and is called on each member of the array that reduce is called on. _reduce_ is special in that it takes the result from the previous call to the reducer function and passes that return value as the _accumulator_ argument when calling the reducer function on the next item in the array. Some common uses for _reduce_ include taking a list of items and "reducing" them into a single result.{% raw %}

~~~js
// Filter example

const isGreaterThanTen = x => x > 10
const isEven = x => x % 2 == 0

[5,12,2,13,10].filter(isGreaterThanTen) // [12,13]
[5,12,2,13,10].filter(isGreaterThanTen).filter(isEven) // [12]
~~~

~~~js
// Reduce Example

// Sum doesn't need an initial value, because the first element in the array can be used as the initial value.
const sum = [1,2,3,4].reduce((accumulator, current) => accumulator + current); // 10

// The flattened reduce call uses an empty list as the initial value, because the accumulator needs to always be an array
// and the first element doesn't match that type. Calling accumulator.concat when accumulator is 0 would cause an error.
const flattened = [0, [1,2], [3,4]].reduce((accumulator, current) => accumulator.concat(current), []) // [0,1,2,3,4]
~~~{% endraw %}

If we can compose mapper functions, maybe we can compose adjacent filters and reduces.{% raw %}

~~~js
// Compose filters example

const isGreaterThanTen = x => x > 10
const isEven = x => x % 2 == 0

const isGreaterThanTenAndEven = compose(isEven, isGreaterThanTen)

// Uh oh, doesn't work :(
[5,12,2,13,10].filter(isGreaterThanTenAndEven)
~~~{% endraw %}

In the above example, the composed function _isGreaterThanTenAndEven_ doensn't have the same shape as the two functions it's composed of. _isEven_ and _isGreaterThanTen_ both expect a number as input and return a boolean. When trying to compose them, a problem happens when the result of _isGreaterThanTen_ is passed as the input to _isEven_. At this point _isEven_ expects a number, but is given a boolean, and it loses the context of the element it's supposed to run on. There's a similar issue with _reduce_, where the reducer function returns a single value, but takes two arguments as input, so the output of one reducer function can't be simply passed to another reducer function.

The remainder of this post is about how to compose maps, filters, and reduces, such that you can take multiple maps, filters, and reduces and consolidate them into a single reduce.

## Enter Transducers

### Using reduce for map and filter

An interesting property of _reduce_ is that _map_ and _filter_ can be expressed using _reduce_.{% raw %}

~~~js
function reduceMap(fn) {
    return function reducer(accumulator, current) {
        accumulator.push(fn(current))
        return accumulator
    }
}

[1,2,3].reduce(reduceMap(x => x + 1)) // [2,3,4]

function reduceFilter(predicate) {
    return function reducer(accumulator, current) {
        if (predicate(current)) accumulator.push(current)
        return accumulator
    }
}

[1,2,3,4].reduce(reduceFilter(x => x > 2)) // [3,4]

[1,2,3,4]
.reduce(reduceFilter(x => x > 2))
.reduce(reduceMap(x => x + 1))
// [4,5]
~~~

Now that we can express maps and filters and multiple chained reduce calls, maybe there's something we can do to compose those reducer functions.

Something we can do in the meantime is to abstract out the functionality that combines results in _reduceMap_ and _reduceFilter_. In these examples we're always appending to a list and returning the list, we could pass the accumulated value and new value to combine and return the results.

~~~js
function reduceFilter(predicate, combiner) {
    return function reducer(accumulator, current) {
        if (predicate(current)) return combiner(accumulator, current)
        return accumulator
    }
}

function reduceMap(fn, combiner) {
    return function reducer(accumulator, current) {
        return combiner(accumulator, fn(current))
    }
}

function listCombine(list, value) {
    list.push(value)
    return list
}

function sumCombine(sum, number) {
    return sum + number
}

[1,2,3,4].reduce(reduceFilter(x => x > 2, listCombine), []) // [3,4]
[1,2,3,4].reduce(reduceMap(x => x + 1, sumCombine), 0) // 14
~~~

In the example above, we defined a _listCombine_ and a _sumCombine_. Both of these can be used as a combiner function because they accept _an accumulator and an item and return a new value_. If you look at those functions, they have the same shape as reducer functions. Maybe we can find a way to compose the combine functions with our map reducers and filter reducers! If we think of the second argument to _reduceFilter_ and _reduceMap_ and the next reducer, maybe we can chain those functions together.

~~~js
const addOne = x => x + 1
const isEven = x => x % 2 == 0
const sumCombine = (sum, number) => sum + number

const chainedReducer = reduceFilter(isEven,
                            reduceMap(addOne,
                                sumCombine))

[1,2,3].reduce(chainedReducer);
~~~

We created a function called _chainedReducer_, which creates a filter reducer which checks if the value given to it is even. If the value is even, it passes the accumulator and value to the _combiner_ given to _reduceFilter_, which is this case is the addOne reducer returned by _reduceMap_. The _addOne_ reducer then passes the result of calling _addOne_ to the value and passing the accumulator and new value to its _combiner_ argument, which is _sumCombine_.

The way we're taking the output of one reduceMap function and placing it as the input of a reduceFilter is similar to how composed functions look in code.{% raw %}

~~~js
const chainedReducer = reduceFilter(isEven,
                            reduceMap(addOne,
                                sumCombine))
const chainedFunctions = arg => x(y(z(arg)))
~~~{% endraw %}

The only issue is that _reduceFilter_ and _reduceMap_ take two arguments, which makes them more difficult to compose than unary functions, or functions that take a single argument. We can fix this by making _reduceFilter_ and _reduceMap_ curried functions, such that we can pass our mapper and predicate functions and then compose the partially applied functions. This is where everything starts to come together.{% raw %}

~~~js
import { curry, compose } from 'lodash/fp'
const transduceMap = curry(reduceMap)
const transduceFilter = curry(reduce)

const addOne = transduceMap(function addOne(x) { return x + 1 })
const isEven = transduceFilter(function isEven(x) { return x % 2 == 0 })

let transducer = combiner => isEven(addOne(combiner))
// OR
transducer = compose(
    isEven,
    addOne
)

[1,2,3,4].reduce(transducer(listCombine), []) // [3,5]
[1,2,3,4].reduce(transducer(sumCombine)) // 8
~~~{% endraw %}

In the above example, _transducer_ is the composition of our _reduceFilters_ and _reduceMaps_ and it takes a _combiner_ as a single argument.

Something to note is that data flows through the composed reducer functions from left to right, as opposed to right to left when usually using compose. So each value will pass through _isEven_ first, and if it passes the predicate, will then be passed to _addOne_, and eventually to _listCombine_ or _sumCombine_.

### Why is it reversed?

isEven is a partially applied function that takes a combiner function as an argument. The result of calling it is a reducer function which has the predicate function and combiner function available in lexical scope. By calling the composed function with a combiner argument, the values that are being piped from right to left are reducer functions which are being passed as an argument to the next leftmost function.

So when we call _transducer(listCombine)_, _listCombine_ is passed to _addOne_ first, which is a _reduceMap_ call which has already been partially applied with a mapper function. Since _listCombine_ fulfills the last argument for the curried/partially applied function, the original _reduceMap_ function is called and returns a reducer function which accepts accumulator and current arguments, but has the mapper function and combiner functions in lexical scope. The returned reducer function is then passed into _isEven_ as _isEven_'s _combiner_ argument, which results in _isEven_ returning a reducer function that takes accumulator and current arguments, and has the predicate function and reducer from _addOne_ as its combiner in scope. The resulting value of the transducer is a reducer function that can be plugged in to any reduce function, be it _Array.prototype.reduce_, or a _reduce_ from any other functional programming library.

### Stepping through it
If you want to get a better idea of how it works, I really recommend putting some debug breakpoints in an editor like VS Code or Atom and stepping through the reducer call.

I placed breakpoints in the returned reducer functions from reduceFilter and reduceMap and stepped through to see in which order they were called and what the values of the predicate and mapper functions were, as well as the combiner functions. In the first image, the reduceFilter breakpoint is triggered first, and the predicate function value is the named _isEven_ function passed to _transduceFilter_ and the combiner function is the anonymous reducer function passed by the _addOne_ function. The first value passed to it is __1__, which doesn't pass the predicate, so the combiner function isn't called.

![debugging reduceFilter](https://s3-us-west-2.amazonaws.com/copperwall-blog-images/copperwall-dev/debug1.png)

The filter reducer is called again with __2__, which calls the combiner, which triggers the breakpoint in _reduceMap_. At this point the mapper function is the named _addOne_ function passed to _transduceMap_ and the combiner in this case is the _listCombine_ function.

![debugging reduceMap](https://s3-us-west-2.amazonaws.com/copperwall-blog-images/copperwall-dev/debug2.png)

## Libraries

If you want to try using this in your code, there's a [transducers-js](https://github.com/cognitect-labs/transducers-js) library by the people who make Clojure and ClojureScript. The docs are pretty awesome too, I really recommend taking a look at it.

## Thanks for reading!

Writing this was as much for my understanding of the concepts as it was for yours, so if anything is unclear, or if anything could be better explained, or if anything is wrong, please please let me know. I'm [@copperwall](https://twitter.com/copperwall) on Twitter.