Decaffeinating 2020: Migrating our CoffeeScript app to TypeScript

#typescript #graphql #django #coffeescript

With the start of the new decade, the engineering team at Fellow.app decided to take on a new year’s resolution to do something that was holding us back: we decided to convert our CoffeeScript web app to TypeScript. This is the story of how we pulled through as a team to overcome the issues in our way, and how completing this massive migration improved our product.

(Illustration by Matt Emond, modifications to it below by me!)

From the start, we’ve highly valued the ability to quickly iterate on features based on feedback we get from the teams that use Fellow, and CoffeeScript was part of what enabled that speed. CoffeeScript is a language that compiles to JavaScript and was meant to reduce the space code takes up and improve readability, while also adding new functionality that JavaScript would later adopt (like a clean way of defining classes). With all the developers on the team already knowing Python, CoffeeScript was a natural choice because of the similarities between the two (like a lack of braces and use of whitespace for logical separation) which made context switching from one language to the other easy. Spending less time mentally shifting between two distinct syntaxes and having to write fewer braces and parentheses made us more efficient at building features, and we were able to keep up with our value of fast iteration.

But suddenly, we didn’t need to keep iterating on features quite so quickly anymore: while we still wanted to be able to ship new features often, we had built a product that we were proud of, and the user base had started to grow at a rapid pace. Our priorities shifted, and we needed to start providing the most stable experience that we could while building new features more methodically. CoffeeScript started to get in the way: the lack of parentheses made code hard to understand, the non-existent debugging tools made fixing bugs a lengthy process, and developers had to learn a whole new syntax when they joined the team. We were also running into completely avoidable bugs centered around types: maybe a function would incorrectly receive a null variable, or a number would be passed as a string and not an integer -- and these would all cause runtime errors, only detectable when people would use the app. It was creating a latte extra work.

// The following is valid CoffeeScript. Can you easily understand the operation?
square = (x) -> x * x
add = (x, y=1) -> x + y
console.log(add square add 2, add 1) // 17

We were starting to lose time dealing with these issues, when we had sought to use CoffeeScript as a way to speed up development.

On December 13th 2019, exactly 10 years after CoffeeScript came out, we sat down and started to plan our full transition plan off of the language, after having slowly converted some files here and there starting in the summer. We wanted to start using a language that had a strong community, effective debugging tools, IDE support, and would enable us to avoid as many errors as possible. TypeScript fit the bill: type annotations would allow us to reduce runtime errors while acting as mini documentation for our code, the available tooling and editor integrations are endless, and it’s something that we were excited to write code in instead of dreading. Our transition plan? Sit together as a team for an afternoon for “The Decaffeination” and convert tens of thousands of lines of CoffeeScript to TypeScript in parallel before going on break for the holidays, and eventually releasing the conversion to production in one go. And we did it.

We let the CoffeeScript removal plan brew in our minds for a few days, and then poured our ideas into our shared meeting notes for everyone to see. We compiled the list of files that needed to be converted and added them as action items in the same note, assigning each one to a member of the team so we’d be more or less equal in the task ahead. We decided to create a git branch dedicated to the task, decaffeinate, and all pushed our code there. A week later, on December 20th, we got to work.

But even though we were ultimately successful, the task didn’t come without issues! Sitting all together spread out over couches and cozy chairs drinking tea (anything but coffee) enabled us to work really well together: any issues we ran into were shouted out and everyone would help, and every successfully converted file was celebrated. Some of the hardest parts we had to tackle were a result of our tech stack (we use React/Relay), creating errors that had no results on StackOverflow and new bugs that were difficult to trace.

We learned a lot by tackling this conversion, and found that there were limited resources on the web about our particular stack -- and so we’d like to contribute back to the community and share our struggles. If you’re looking to convert a project to TypeScript, I highly recommend reading Dominik Kundel’s detailed blog post about how to get started.

Here are some of the top issues that we encountered during this process:

String interpolation

We have a lot of strings in the codebase where we insert variables, which looks like this in CoffeeScript:

"You have  #{num_1on1s} upcoming 1-on-1s today."

But in TypeScript, the same string interpolation would look like this:

`You have ${num_1on1s} upcoming 1-on-1s today.`

Since TypeScript detects string interpolation through back-ticks and not quotation marks, we had to be extra careful to check our converted code for strings that would need to be converted manually, because TypeScript would literally output "You have #{num_1on1s} upcoming 1-on-1s today." if we didn’t update the format to back-ticks and dollar signs.

Incorrect types for 3rd party libraries

A lot of 3rd party libraries have typed versions, which we believed would decrease the errors we saw in the code even more. Unfortunately, a number of the type interfaces for the libraries we were using weren’t being updated in conjunction with the non-typed versions that we had been using before the translation. This led to a number of extra errors, and we found that sometimes it was better to simply set the accepted types to any and be extra careful about what we were passing around instead of trusting an incorrect version.

Typing Higher Order Components

In React, a Higher Order Component is a function that takes a component and returns a new component, the goal being to reuse code that modifies components in a similar way (for example, a HOC might accept a component and return a loading spinner instead of the component if some API call hasn’t finished yet, which is a pattern you may want to use for many different areas).
Higher Order Components are notoriously difficult to type because of the range of component types that might be passed to them, and all the different props those components have. In our codebase we have one particularly large HOC withNoteStream, which we use to control a lot of the data and rendering logic for the parts of the project that have streams of notes. And we have a lot of notes! Personal notes, meeting notes, 1-on-1 notes, goal notes, … all of these sections have their own components which use withNoteStream, and each has their own unique set of properties that needs to be passed through. We haven’t yet found a great way to work around this issue without casting the types to a generic component, and it’s an ongoing discussion in the React/TypeScript community.

GraphQL types are undefined by default

We use GraphQL to pass data between our React frontend and Django backend, so when we started converting to TypeScript and found out that GraphQL types are undefined by default, we had to either write all of our TypeScript code to check for the existence of data before using it (if this.props.dataWeKnowExists && this.props.dataWeKnowExists.value...), or we had to find another solution.
We use Graphene to expose our Django models to GraphQL, and we partially solved the undefined problem by wrapping the list fields in graphene.NonNull() and setting required=True in the node fields.
This solution did not work for DjangoConnectionFields though, which is a field type that is used to create a paginated list of related objects. Fields with this type were still being considered as possibly undefined in TypeScript, even if we knew with certainty that they would not be. To fix this issue, we created the following class to inherit from, which would set all of the nodes to required=True and all of the edges to graphene.NonNull() so that we could have peace of mind when writing our frontend code:

class NonNullConnection(graphene.relay.Connection, abstract=True):  # type: ignore
   @classmethod
   def __init_subclass_with_meta__(cls, node, edge_name=None, **kwargs):
       if not hasattr(cls, 'Edge'):
           _node = node

           class EdgeBase(graphene.ObjectType):
               class Meta:
                   name = edge_name or f'{node.__name__}Edge'

               cursor = graphene.String(required=True)
               node = graphene.Field(_node, required=True)

           cls.Edge = EdgeBase

       if not hasattr(cls, 'edges'):
           cls.edges = graphene.List(graphene.NonNull(cls.Edge), required=True)

       super(NonNullConnection, cls).__init_subclass_with_meta__(node=_node, **kwargs)

Typing anything that is passed inexplicit props

Writing out {...props} in a component is a nice and easy way to pass props down to a component that you’ve extended, without needing to be explicit about what those props are (maybe because the component you’re writing doesn’t care what they are, and you don’t want to maintain them in multiple spots). In TypeScript though, this caused issues for us, because for every component we did this for, we needed to find all the permutations of the properties that were being passed through and correctly type them. Even though this was a lengthy process, it ended up being well worth the effort because it also provided some refactoring! Refactoring is typically a risky idea when converting code because of the possibility of introducing bugs from both the refactor and the translation (with little to indicate which caused the issue), but we were able to remove props that we found we were no longer using at all, which also reduced the number of types we needed to resolve.

A few weeks later when we finished converting the last few lines and tested out every last little bit of the product, we merged our decaffeinate pull request into our main branch and waited excitedly to see our error count go down and code readability go up 🎉

We’re now writing all new features in TypeScript, and even though prototyping features may not be as fast as it was with CoffeeScript, the benefits we’re getting out of using it at our stage of team growth are undeniable:

New hires and junior developers have an easier time adapting to the codebase because the code is more readable, and we’re using a more common language.
Type-related errors are being caught as we’re writing code.
Argument definitions are explicit, which helps the code self-document.
Developers are happier because of the increase in tooling and ease of debugging.

Overall, our code is more maintainable, which is enabling us to create a reliable product and keep developers happy. We’re excited that we hit our goal as a team, and looking forward to our coffee-free 2020!