DEV Community: Akshay Nathan

Building a Typescript CLI

Akshay Nathan — Thu, 07 Nov 2019 16:51:52 +0000

Building a CLI with Typescript

At walrus.ai, we're building a platform for end-to-end testing via a single API call. Our users give us a url and plain English instructions, and we use a human assisted trained model to verify each test-case. While one can use the walrus.ai API using curl or any http library of their favorite language, we recently decided to build a command line tool to make it easier to submit walrus.ai tests, and plug them into existing CI/CD pipelines.

This blog post will go over building this CLI in Typescript. First, the finished product:

Setting up

Let's create a new directory and initialize npm.



$ mkdir cli
$ cd cli
$ npm init -y

We will need to install Typescript, the types for node, as well as ts-node which will enable us to run Typescript files directly without compiling.



$ npm install -D typescript @types/node ts-node

Notice how we're installing all Typescript related packages as dev dependencies? This is because our published package will only need the compiled Javascript. More on that later.

For now, let's create a basic tsconfig.json for the Typescript compiler:



{
  "compilerOptions": {
    "baseUrl": ".",
    "target": "ES2017",
    "module": "commonjs",
    "strict": true,
    "esModuleInterop": true,
    "rootDir": "src",
    "outDir": "dist"
  }
}

And now our first Typescript file:



// src/index.ts

console.log('Hello World');

Now, we can compile and run this file:



$ npx tsc
$ node dist/index.js
Hello World

Remember ts-node, which we installed earlier? We can use it to run our program more easily while developing. Let's create an npm script using ts-node.



// package.json
...
'scripts': {
  'dev': 'ts-node src/index.ts'
}
...



$ npm run dev

> npx ts-node src/index.ts

Hello World

Accepting input

Almost all command line tools follow similar flows — they accept input via arguments or stdin, they do something, and then they output results to stdout and errors to stderr.

In node, a program's arguments are stored in an array inside process.argv. You can access these arguments directly, or you can use an option parsing library to simplify access and create a better user experience. Some node options include yargs, commander, and argparse. All three libraries have similar APIs, but we chose to go with yargs.

The walrus.ai API functionally takes in 3 required parameters. An API key to identify the user, the url of the application we want to test against, and a list of instructions to execute and verify. Let's install yargs and parse these arguments.



npm i yargs
npm i -D @types/yargs



// src/index.ts

import yargs from 'yargs';

const args = yargs.options({
  'api-key': { type: 'string', demandOption: true, alias: 'a' },
  'url': { type: 'string', demandOption: true, alias: 'u' },
  'instructions': { type: 'array', demandOption: true, alias: 'i' },
}).argv;

console.log(args);

We can use the demandOption parameter to require a program argument. If we try to rerun our script now, our program will complain about the missing arguments:



$ npm run dev

> npx ts-node src/index.ts

Options:
  --help              Show help                                        [boolean]
  --version           Show version number                              [boolean]
  --api-key, -a                                              [string] [required]
  --url, -u                                                  [string] [required]
  --instructions, -i                                          [array] [required]

Missing required arguments: api-key, url, instructions

When we supply them, we can see that yargs has parsed our arguments into a strongly-typed map.



$ npm run dev -- -a 'key' -u 'url' -i 'instruction'

> ts-node src/index.ts "-a" "key" "-u" "url" "-i" "instruction"

{
  _: [],
  a: 'key',
  'api-key': 'key',
  apiKey: 'key',
  u: 'url',
  url: 'url',
  i: [ 'instruction' ],
  instructions: [ 'instruction' ],
  '$0': 'src/index.ts'
}

Doing something

Now that our CLI is accepting input, the next step is to do something.

In the case of the walrus.ai CLI, we want to call the API with our parsed arguments. Again, there are many libraries we can use to make HTTP requests including superagent, axios, and request. In our case, we chose axios.



npm i axios



// src/index.ts

import yargs from 'yargs';
import axios from 'axios';

const args = yargs.options({
  'api-key': { type: 'string', demandOption: true, alias: 'a' },
  'url': { type: 'string', demandOption: true, alias: 'u' },
  'instructions': { type: 'array', demandOption: true, alias: 'i' },
}).argv;

axios
  .post(
    'https://api.walrus.ai',
    { url: args['url'], instructions: args['instructions'] },
    { headers: { 'X-Walrus-Token': args['api-key'] }, },
  )
  .then(
    (response) => {
      console.log(JSON.stringify(response.data, null, 2));
    },
    (reason) => {
      console.error(JSON.stringify(reason.response.data, null, 2));
    },
  );

Note that we're handling both branches of the Promise returned by axios.post. Maintaining convention, we print successful results to stdout and error messages to stderr. Now when we run our program, it will silently wait while the test is completed, and then print out the results.



$ npm run dev -- -a fake-key -u https://google.com -i 'Search for something'

> cli@1.0.0 dev /Users/akshaynathan/dev/blog/cli
> ts-node src/index.ts "-a" "fake-key" "-u" "https://google.com" "-i" "Search for something"

{
  "error": "Authentication required. Please sign in at https://app.walrus.ai/login."
}

Displaying progress

We can improve on our CLI by making it slightly more interactive. On the web, long-running operations are often handled in the UI by displaying some sort of loading state. There are a few node libraries that can help us bring these UI paradigms to the command line.

Loading bars are useful when the long-running task takes a relatively static amount of time, or if we have a discrete intuition about 'progress'. node-progress or cli-progress are both good libraries for this solution.

In our case, however, while all walrus.ai results are returned under 5 minutes, we don't have a discrete notion of progress. A test is either pending, or it has been completed. Spinners are a better fit for our CLI, and ora is a popular node spinner library.

We can create our spinner before making our request, and clear our spinner once the Promise resolves or rejects.



// src/index.ts

import yargs from 'yargs';
import axios from 'axios';
import ora from 'ora';

const args = yargs.options({
  'api-key': { type: 'string', demandOption: true, alias: 'a' },
  'url': { type: 'string', demandOption: true, alias: 'u' },
  'instructions': { type: 'array', demandOption: true, alias: 'i' },
}).argv;

const spinner = ora(`Running test on ${args['url']}`).start();

axios
  .post(
    'https://api.walrus.ai',
    { url: args['url'], instructions: args['instructions'] },
    { headers: { 'X-Walrus-Token': args['api-key'] }, },
  )
  .then(
    (response) => {
      spinner.stop();
      console.log(JSON.stringify(response.data, null, 2));
    },
    (reason) => {
      spinner.stop();
      console.error(JSON.stringify(reason.response.data, null, 2));
    },
  );

Now when we run our program, we will see the spinner from the GIF above!

Exiting

The last thing our CLI program has to do is to exit, and exit correctly. When programs exit, they can specify an integer exit code to indicate success of failure. Generally, any non-zero exit code indicates failure.

For the walrus.ai CLI, correctly specifying an exit code is imperative. Our users call our CLI from CI/CD pipelines. When a test fails, we have to exit with a non-zero exit code so that the next step in the pipeline, usually the deploy to production, doesn't run.

You may be tempted to use node's process.exit API:



// src/index.ts

...
(response) => {
      spinner.stop();
      console.log(JSON.stringify(response.data, null, 2));
      process.exit(0);
    },
    (reason) => {
      spinner.stop();
      console.error(JSON.stringify(reason.response.data, null, 2));
      process.exit(1);
    },
...

However, process.exit will exit the program synchronously, even if there are operations waiting to be run or caches that need to be flushed. The most common problem here is output. In the above code, depending on how our output is buffered, our program may exit before our success or error messages are printed to the screen.

We can solve this by simply setting the exit code, and letting the node script automatically exit upon completion.



// src/index.ts

import yargs from 'yargs';
import axios from 'axios';
import ora from 'ora';

const args = yargs.options({
  'api-key': { type: 'string', demandOption: true, alias: 'a' },
  'url': { type: 'string', demandOption: true, alias: 'u' },
  'instructions': { type: 'array', demandOption: true, alias: 'i' },
}).argv;

const spinner = ora(`Running test on ${args['url']}`).start();

axios
  .post(
    'https://api.walrus.ai',
    { url: args['url'], instructions: args['instructions'] },
    { headers: { 'X-Walrus-Token': args['api-key'] }, },
  )
  .then(
    (response) => {
      spinner.stop();
      console.log(JSON.stringify(response.data, null, 2));
    },
    (reason) => {
      spinner.stop();
      console.error(JSON.stringify(reason.response.data, null, 2));
      process.exitCode = 1;
    },
  );

Now when we run our script, it will fail with a non-zero exit code:



$ npm run dev -- -a fake-key -u https://google.com -i 'Search for something'

> ts-node src/index.ts "-a" "fake-key" "-u" "https://google.com" "-i" "Search for something"

{
  "error": "Authentication required. Please sign in at https://app.walrus.ai/login."
}
$ echo $?
1

Publishing

Now that we've built our CLI, we need to publish it so our users can use it.

There are many options we have here. Most simply, we can distribute the package and the CLI via npm. Alternatively, we could use a library like pkg or oclif to bundle node itself into our binary. This way, users will not need to have npm or node installed to run our tool.

Since walrus.ai is a tool for running browser end-to-end tests, and our users are probably already familiar with npm and node, we decided to go with the simple option. First, we can edit our package.json to specify a binary, in this case walrus.



{
  "name": "@walrusai/cli",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "dev": "ts-node src/index.ts"
  },
  "bin": {
    "walrus": "dist/index.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@types/node": "^12.12.6",
    "@types/yargs": "^13.0.3",
    "ts-node": "^8.4.1",
    "typescript": "^3.7.2"
  },
  "dependencies": {
    "axios": "^0.19.0",
    "ora": "^4.0.2",
    "yargs": "^14.2.0"
  }
}

Next, let's make our index.ts runnable by telling the shell how to run it:



// src/index.ts

#!/usr/bin/env node
...

Now we can use npm link, to effectively link our node script into our path, as if we installed the binary.



$ npx tsc
$ npm link

Now we can run our binary directly.



$ walrus -a fake-key -u https://google.com -i 'Search for something'
{
  "error": "Authentication required. Please sign in at https://app.walrus.ai/login."
}

npm link is useful for development, but we want our users to be able to install our CLI more easily. For that, we can publish to npm.

First, we should create a unique name for our package — @walrusai/cli in our case.

Next, we will need to create an account on npm, authenticate in our command line, and then run:



$ npx tsc
$ npm publish

Now, our users can install our cli more easily:



$ npm install -g @walrusai/cli

Conclusion

In this blog post, we've built a Typescript CLI that accepts user input, makes an api call, outputs results, and exits correctly. You can check out the final open-source implementation of the walrus.ai CLI here.

Are you an engineer tired of building and maintaining flaky browser tests? Try walrus.ai now, supply instructions in plain english, and receive results in under 5 minutes.

Towards a fully programmable inbox

Akshay Nathan — Wed, 09 Oct 2019 18:20:57 +0000

At Monolist, we're building the command center for engineers. We integrate with all the tools engineers use (code hosting, project management, alerting), and aggregate all their tasks in one place.

While we're constantly shipping new integrations, we understand that engineers may have specific workflows or internal tools that we can't natively integrate with. In this post, we'll go over using the new Monolist API to build a personalized workflow and get one step closer to automating your inbox.

The problem

Let's say we have an app with mobile and web clients. Sometimes, because of bugs, these clients end up in request loops that flood our app servers and cause downtime. Now, obviously we could solve this problem at various points in the stack -- we could introduce throttling at the client level, at the load balancer level, or at the application level.

For now, let's assume that we introduced some throttling at the application level using something like this library for Rails, or this one for Node.

Once we've shipped these changes we will want to monitor their behavior. How many users are we throttling? What is the false positive rate? How does the added limiting improve service availability?

Obviously, we could integrate our throttling directly into a more sophisticated monitoring system. We could (and should) build out dashboards that measure number of throttled users, and alerting for when excessive throttling takes place. However, this would require additional effort, and we may want to add this only after we've smoke tested the system. Is there a simpler solution?

Enter Monolist

Perhaps we simply want to be notified when a user is being throttled. We could send ourselves an email, or text message, but this notification is likely to be lost in the flood of emails/texts. On the flip side, we don't (yet) want to send an alert using something like PagerDuty, because we're not sure how frequently this will be happening, or what the correct severity should be.

Instead, let's use the Monolist API to add a task to our Monolist every time a user is throttled.

Note: The code samples in this blog post use Ruby and the HTTP library, but the Monolist API is accessible through any language.

def push_to_monolist(user)
  HTTP.post(
    "https://api.monolist.co/v1/action_items/new",
    json: {
      api_key: "API_KEY",
      action_item: {
        content: "Investigate throttled user #{user.id}",
        description: <<~DESCRIPTION
          Links
            * [Mixpanel](https://mixpanel.com/users?user_id=#{user.id})
            * [Logdna](https://logdna.com/logs?time=#{Time.now})
        DESCRIPTION
      }
    }
  )
end

Now we will be notified in Monolist along with our other tasks. We can prioritize and schedule this task the same as any other Monolist item. Because Monolist descriptions support Markdown, we have nicely formatted links that we can use to diagnose the issue.

One-click actions

At this point, you may be thinking, "couldn't I just have sent myself an email?"

The difference between Monolist and an email/text or any other type of notification, is that with the Monolist API, any task you create can be resolved directly in-line with fully customizable actions.

Let's say that when we investigate a throttled user, we want to notify the ops on-call rotation for any users that are not false positives. This way, they will be able to have context on any downtime implications.

With the Monolist API, we can add custom actions directly to our custom items. When a user takes a custom action, we can specify a destination url that Monolist will POST a payload to.

Let's use the PagerDuty API to directly create an incident:

def push_to_monolist(user)
  HTTP.post(
    "https://api.monolist.co/v1/action_items/new",
    json: {
      api_key: "API_KEY",
      action_item: {
        content: "Investigate throttled user #{user.id}",
        description: <<~DESCRIPTION
          Links
            * [Mixpanel](https://mixpanel.com/users?user_id=#{user.id})
            * [Logdna](https://logdna.com/logs?time=#{Time.now})
        DESCRIPTION
      },
      actions: [
        {
          name: "Escalate to Ops",
          url: "https://api.pagerduty.com/incidents",
          headers: [{ "Authorization" => "Token token=#{PAGERDUTY_TOKEN}" }],
          payload: {
            incident: {
              type: "incident",
              title: "User #{user.id} throttled",
              service: { id: PAGERDUTY_OPS_SERVICE_ID },
            }
          }
        }
      ]
    }
  )
end

When we create this new item, Monolist will encrypt and store the payload and headers objects. In the UI, the user will be able to take an action "Escalate to Ops"

When the user takes that action, Monolist will POST the payload to the PagerDuty API, automatically creating an incident for the Ops team.

Conclusion

In this post, we used the Monolist API to push a custom item with custom one-click actions into an user's Monolist.

Interested in building your own integrations? Check out the docs here.

Keep it simple stupid: Building full text search across all your tools with Postgres

Akshay Nathan — Thu, 03 Oct 2019 17:49:30 +0000

Introduction

In the beginning there were tools, oh so many tools. There were code hosting tools, project management tools, messaging tools, alerting tools – an infinite set of tools. And for developers, all these tools contained tasks: code they had to review, bugs they needed to look at, questions to respond to, incidents to triage. Around a year ago, we asked "what if there was one tool where we could manage all our tasks from all our other tools", so we built it.

As our customers started using Monolist, we found that they were often asking a common question.

"I know that Sarah made a comment about the marketing page designs, but was that in the Jira task, or in the spec Google Doc, or on my pull request where I merged the designs, or on Slack?"

Of course, each one of these tools have a search feature, but remembering where it is, or what syntax it supports is a massive pain. Monolist is already integrated with all these tools, and they all support search over their API, so we built a global search bar in Monolist. It was pretty cool! It would debounce the user's input, search any items already in Monolist, call out to every API for every integrated tool in parallel, merge and deduplicate the results, and present them to the user.

But it was slow... It was so slow, it took between 500ms and 5 seconds to deliver any results, which is unacceptable for a page load, let alone an autocomplete bar. You could already be typing "sea", and the autocomplete would only then finish the request for "s". What's more, it was so complex it was painful to debug, and painful to improve. We used a job system, Sidekiq, to make the API calls, Redis to store the result stream, and websockets to deliver them asynchronously to the user. The code was cool, but it was overburdened with complexity, which led to a slow, buggy, and ultimately poor user experience.

In this blog post, we'll walkthrough the foundation of our second attempt at search. We tried to follow one guiding principle from our first try, the age old cliche: KISS or Keep it simple, stupid. While we're not big on the "stupid" part (KIS), we've learned time and time again that complexity is the enemy, and that premature optimization, unnecessary abstractions, and cool technology for cool technology's sake are toxic to the end user experience. We'll use this principle throughout our journey, but first, let's look at the other objectives we had for the new experience.

Objectives

It needs to be fast.

For an autocomplete experience to be effective, the results need to render quickly. We need the entire request, response, and render flow to take < 100ms, which is the largest amount of time an action can take while still being perceived as "instantaneous" to the user.

The results need to be relevant and (mostly) comprehensive.

We talked to our users and found that while they expected breadth from the search on a frequent basis (they wanted results across all their tools), they didn't necessarily need depth as often (they weren't frequently searching for terms in the body of an email, content of Google Doc, or description of an Asana Task).

And, of course, KIS.

We wanted to stay vigilant against complexity, which leads us to our next section.

What about ElasticSearch?

ElasticSearch is great. It's made for this purpose. It's unbelievably flexible. There are hosted solutions out there. But while it may have been easier to use ElasticSearch, and definitely easier to use a hosted search service like Algolia, simplicity != easiness.

For one, it would introduce another system dependency to our architecture. In a self-hosted world, we would need to stand up, monitor, and maintain an ElasticSearch cluster. In a hosted world, this would be a little easier, but we'd still need to build in failsafes and retry semantics to get our data in and out of the search service.

But even if we used a hosted service, we'd still need to think about data privacy. Our users trust us with sensitive business data, and we would have to guard against unauthorized access. We also take account deletion very seriously, and if they deleted their accounts, this introduces another place we would have to ensure was properly wiped clean.

What if there was a service that we had already setup to be highly available, already had sufficient monitoring in place, and already connected with our application and authorization system...

Just use the database (Postgres)

Luckily, Postgres natively supports full text search and is plenty flexible for our use case. (Note: The code samples in this section are taken from our real Ruby (on Rails) code, but should be easily adapted to any language)

Let's start with what we we're searching for. Monolist search supports multiple result types, including Google Drive files, contacts, and various types of tasks. For the sake of simplicity though, let's assume we only have two domain models, contacts and pull requests.



class Contact < Model
  attribute :user_id
  attribute :email
  attribute :display_name
  attribute :last_sent_at
end

class PullRequest < Model
  attribute :user_id
  attribute :title
  attribute :description
  attribute :repository_name
  attribute :created_at
end

These are some (adapted) examples from our own models. The fields are mostly self-explanatory. Contact#last_sent_at is a denormalized timestamp of the last time this contact was the recepient or sender of an email for our user. Now let's make these objects searchable.

There are two main database functions we will use for full text search in Postgres. First, to_tsvector parses and tokenizes a string into lexemes, which are basically words.



monolist=> SELECT to_tsvector('simple', 'A quick brown fox jumped over a lazy dog');
                               to_tsvector
--------------------------------------------------------------------------
 'a':1,7 'brown':3 'dog':9 'fox':4 'jumped':5 'lazy':8 'over':6 'quick':2
(1 row)

To query a tsvector, we can use the @@ operand and a tsquery.



monolist=> SELECT to_tsquery('simple', 'red') @@ to_tsvector('simple', 'A quick brown fox jumped over a lazy dog');
 ?column?
----------
 f
(1 row)

monolist=> SELECT to_tsquery('simple', 'fox') @@ to_tsvector('simple', 'A quick brown fox jumped over a lazy dog');
 ?column?
----------
 t
(1 row)

Note, the first argument to both functions is a Postgres Full Text Search "config", which you can read more about here, but we won't be using for now.

Simple right? We can just query for results directly on the attributes, something like SELECT * FROM contacts WHERE to_tsquery('simple', 'QUERY') @@ to_tsvector('simple', 'display_name'). However, this only works for one table. If we have multiple tables (pull_requests), we'll have to outer join them as well. Let's add a new denormalized table.



class TypeaheadResult < Model
  attribute :user
  attribute :searchable
  attribute :result_type
  attribute :object_id # contact or pull request id
  attribute :object_type # contact or pull request
end

Here, searchable refers to the indexed attribute (title for pull requests, or display_name for contacts).

But wait, another table? Aren't we storing unnecessary data? KIS: We don't have a storage problem yet, let's not prematurely optimize.

One thing we can (maturely) optimize is based on a learning from our initial search. These tables are quite large (10 million + contacts, many more action items or google drive files). Even an indexed join takes up a significant portion of our 100ms allocation. Let's denormalize further and bring our whole base models into the TypeaheadResult. We can make the searchable column json instead of text, and then write a service to populate it for a user.



class PopulateTypeaheadResultsForUser
  def call(user:)
    user.contacts.map do |contact|
      TypeheadResult.create!({
        user: user,
        result_type: :contact,
        searchable: { email: contact.email, display_name: contact.display_name },
      })
    end
    user.pull_requests.map do |pr|
      TypeheadResult.create!({
        user: user,
        result_type: :pr,
        searchable: { title: pr.title, description: pr.description, repo: pr.repo },
      })
    end
  end
end

Hold on, but what about our to_tsvector function. We can't use that on JSON can we? Of course we can, it's Postgres.



monolist=> SELECT to_tsquery('simple', 'cat') @@ to_tsvector('simple', '{ "name": "fox", "type": "canine" }'::jsonb);
 ?column?
----------
 f
(1 row)

monolist=> SELECT to_tsquery('simple', 'fox') @@ to_tsvector('simple', '{ "name": "fox", "type": "canine" }'::jsonb);
 ?column?
----------
 t
(1 row)

Let's write a service to actually retrieve the results for a user:



class GetTypeaheadResultsForUser
  def call(user:, query:)
    TypeaheadResult.find_by_sql([
      <<SQL,
        SELECT * FROM typeahead_results
          WHERE user_id = ? AND to_tsquery('simple', ?) @@ to_tsvector('simple', searchable),
      SQL
      user_id,
      query,
    ])
  end
end

And finally, we can precompute the tsvectors and add an index to this column to make this query much, much faster.



ALTER TABLE typeahead_results ADD COLUMN searchable_vector tsvector;
UPDATE typeahead_results SET searchable_vector = to_tsvector('simple', searchable);
CREATE INDEX ON typeahead_results USING GIN (user_id, searchable_vector);

Great! We have a comprehensive and simple solution. However, we still have no notion of relevancy. If a user types in "a", the first result may be "alice@monolist.co", or it may be "analytics-marketing-campaign@marketing.google.com'.

Postgres actually has many solutions for relevancy, mainly around the handy ts_rank function. We could define a relevancy metric based on which indexed field we matched (repository name should be less relevant than title for a pull request), the length of the match, and the type of the result...

Or, we can KIS. From talking more to our users, we decided that (last_interacted_with || updated_at || created_at) was a good enough proxy for relevancy. In other words, take the items the user has interacted with most recently, or have been updated or created most recently first. For contacts, we can use last_sent_at, and for pull requests, we can use created at. After adding a new column to typeahead_results our new services look like this:



class PopulateTypeaheadResultsForUser
  def call(user:)
    user.contacts.map do |contact|
      TypeheadResult.create!({
        user: user,
        result_type: :contact,
        searchable: { email: contact.email, display_name: contact.display_name },
        last_referenced_at: contact.last_sent_at,
      })
    end
    user.pull_requests.map do |pr|
      TypeheadResult.create!({
        user: user,
        result_type: :pr,
        searchable: { title: pr.title, description: pr.description, repo: pr.repo },
        last_referenced_at: pr.created_at,
      })
    end
  end
end

class GetTypeaheadResultsForUser
  def call(user:, query:)
    TypeaheadResult.find_by_sql([
      <<SQL,
        SELECT * FROM typeahead_results
          WHERE user_id = ? AND to_tsquery('simple', ?) @@ searchable_vector
          ORDER BY last_referenced_at DESC
      SQL
      user_id,
      query,
    ])
  end
end

There's still one more problem. Now that we're ordering results, the most frequent results will push out the others. For the query, "a", there may be tons of pull requests in the "analytics" repository that flood out the "alice@monolist.co" contact who sent us an email last week. We could implement some sort of intelligent scheme, where depending on query length we priortize one result type or another, or...

KIS We could also just limit results by type:



class GetTypeaheadResultsForUser
  def call(user:, query:)
    TypeaheadResult.find_by_sql([
      <<SQL,
        SELECT * FROM typeahead_results
          WHERE user_id = ? AND to_tsquery('simple', ?) @@ searchable_vector
          ORDER BY last_referenced_at DESC
      SQL
      user_id,
      query,
    ]).group_by(&:result_type)
      .map { |k, v| [k, v.take(5)]}
      .values
      .flatten
  end
end

And now, the moment you (and we) are all waiting for. How does this relatively simple solution perform?

IT FLIES Even with millions and millions of typeahead_results, the indexed query with no joins gives us results around 100ms. With some clever React work (which we'll talk about in a subsequent post), we were able to get the entire request->response->render median time to just under 100ms (97ms).

Tradeoffs and future

Obviously, there are tradeoffs in this solution. For one, the denormalized table requires us to be very careful about updates and deletions. Luckily, Postgres supports indices on nested JSON keys, so we can index contact.email for example, and then just delete and reinsert all results when a specific contact is updated.

Another large tradeoff is flexibility. ElasticSearch is obviously more powerful than Postgres, and from a scaling perspective has a better story. We can horizontally scale ES instances separately from our database, since our search index may grow relatively independently of live data volume.

Lastly, our search may be faster, but is also more shallow. We're no longer searching email or drive file content, but we haven't found this to be problematic for our users, since the "See all results" link still takes the user to a (slower) deep search experience when necessary.

All in all, the response to our search improvements has been unanimously positive, and we think these tradeoffs were necessary decisions in our pursuit of a fast, simple, and maintainable solution.

Want to see for yourself? Tired of going to 5 different tools to search for the "Product Spec"? Try Monolist for free here.

Enjoyed this post? Join our mailing list here.

Ruby 2.7 Experimental Features in Production: Pattern matching and numbered block args

Akshay Nathan — Tue, 01 Oct 2019 18:12:55 +0000

At Monolist, we're building the command center for engineers. We integrate with all the tools engineers use (code hosting, project management, alerting),
and aggregate all their tasks in one place. If you've read our previous
blog posts, you know that most of our backend services are built using Ruby.

The creator of Ruby, Matz, once said that "the goal of ruby is to make programmers happy." We share this goal at Monolist. Ruby enables us to iterate quickly to deliver new integrations
and features to our customers, programmers, faster. We are especially excited about Ruby 2.7, and we're already running some of our less critical services on the preview build released earlier this summer.

In this blog post, we'll talk about our favorite new (experimental) features of Ruby 2.7 in the few months we've been running it in production.

Pattern Matching

Pattern Matching is one of Ruby 2.7's marquee features. Here's a simple example:

item = { integration: :github, title: "Fix onboarding bug" }

case item
in integration: :github, title:
  puts "Found github pull_request: #{title}"
in integration: :asana, title:
  puts "Found asana task: #{title}"
else
  puts "Found other item"
end


# => Found github pull request: Fix onboarding bug

Pattern matching in Ruby is a little bit different than in other languages. In statically typed languages like Rust or Haskell, pattern matching constructs enable the compiler to guarantee that they're exhaustive. In other words, code that doesn't handle every possible case won't compile. In Ruby, if the pattern is not exhaustive, you'll instead get an error at runtime, NoMatchingPattern. This isn't as useful, but still better than a potentially hidden bug were we to use a conditional.

We've found instead that pattern matching's most useful quality in Ruby is for destructuring deeply nested hashes. Consider the following (real but simplified) code we use to get a pull request's last updated timestamp. If the pull request doesn't have any completed build statuses, we use the time the pull request was created. Otherwise, we use the timestamp from the build status.

def pull_request_updated_at(pr)
  status = pr[:statuses]&.first

  if status && (status[:status] == "success" || status[:status] == "failure")
    return status[:created_at]
  end

  pr[:created_at]
end

Now let's rewrite this code using pattern matching:

def pull_request_updated_at(pr)
  case pr
  in { statuses: [status: { status: "success" | "failure", created_at: updated_at }] }
    updated_at
  else
    pr[:created_at]
  end
end

Notice that with pattern matching, we can eliminate some of the nested conditionals and safe navigation from our first attempt. Additionally, the pattern matching version is easier to read, and easier to reason about. The patterns explicitly connote our assumptions about the shape of our data, assumptions that are otherwise hidden behind if statements.

Since Monolist relies heavily on third party API's, we find ourselves often dealing with complex, nested data structures. Pattern matching has helped us be more explicit and deliberate in our handling of this data, reducing bugs, and making it easier for the next person to add functionality.

Numbered block arguments

At Monolist, we're heavy users of blocks/procs and lambdas. We especially love Ruby's fluent api with Enumerables, which enables us to write clean and readable code like follows:

# Get relevant pull requests for user by repository
pull_requests
  .select { |s| s.assignees.include?(user) }
  .select { |s| s.merged_at.nil? || s.closed_at.nil? }
  .reject { |s| s.author == user }
  .group_by { |s| s.repository.id }

It's easy to tell what this code is doing. We're filtering pull requests by assignee, then selecting only the open ones, removing any that the user created, and finally grouping by repository id.

However, one annoyance we faced before was the constant need for arbitrary block param names in simple blocks. It's obvious that the s in each block is referring to a pull request, but we still have to repeat our name every time. This pattern littered our codebase:

➜  monolist ag --stats "{ \|s\|" | ag " matches"
219 matches
133 files contained matches

This has been solved in Ruby 2.7:

# Get relevant pull requests for user by repository
pull_requests
  .select { @1.assignees.include?(user) }
  .select { @1.merged_at.nil? || @1.closed_at.nil? }
  .reject { @1.author == user }
  .group_by { @1.repository.id }

Instead of using random characters (s, t, p) all over our codebase, we can reliably refer to block params by number. We think that this improves readability, and the general developer experience.

Monolist ❤️ Ruby

At Monolist, we're incredibly excited for the stable release of Ruby 2.7 later this year, Ruby 3 in the coming years, and beyond. For more Ruby content, check out our first foray into adding types with Sorbet here, or the patterns we use to scale our API integrations here.

Are you a Ruby engineer that sometimes spends more time scanning tools for tasks and pull requests, than actually writing code? Monolist can help! Sign up for free here.

Liked this post? Join our mailing list for more content here.

Delightful Goodbyes: Fully Deleting User Data

Akshay Nathan — Mon, 30 Sep 2019 21:05:37 +0000

At Monolist, we’re building the command center for engineers. We integrate with all the tools an engineer uses, aggregate their tasks, and provide intelligence to make sure that nothing slips through the cracks.

Unfortunately, like any company, we sometimes have to say goodbye to one of our users. Creating a delightful experience throughout the product is core to our company culture, and we believe that account deletion should be no exception. While it may seem counterproductive to focus on this user story, we think it’s a dark pattern to make deletion difficult. It frustrates me when it’s hard to delete my account from certain services, and we don’t want our users to feel the same way.

However, because Monolist is built on integrations with different tools, we rely on a complex data model that makes deleting all of a user’s data, fully and efficiently, rather difficult.

In this post we’ll focus on ensuring that a user’s data is fully deleted, which is important for compliance with GDPR and other regulations, but is also just the right thing to do.

Starting Simple

In the beginning, our user deletion service was as simple as possible. (Note: The code snippets in this blog post are written in Rails and ActiveRecord, but the behavior and message should be the same regardless of language/framework/ORM).

class DeleteUser
  def call(user)
    user.destroy!
  end
end

The destroy! method in ActiveRecord deletes a record from the database and executes all relevant callbacks. Rails provides an easy way to handle associations as follows.

class User < ApplicationRecord
  has_many :action_items, { dependent: :destroy }
end

This ensures that when a user is destroyed, their action items will be destroyed as well.

The problem with this approach, however, is that it requires us to be very careful about adding “dependent: :destroy” to our associations. Let’s take a look at the action item model.

class ActionItem < ApplicationRecord
  belongs_to :user

  has_one :properties, { dependent: :destroy }
  has_many :notifications
end

Do you see the problem here? When we call user.destroy! now, we will correctly destroy all action items and their properties, but we won’t destroy notifications! This means that we’re still storing user data, even after they’ve deleted their account.

Adding Testing

An obvious solution to this problem is to use foreign keys. Foreign keys enable referential integrity, and ensure that an associated database record (Action Item), cannot be deleted while it’s associations (Notifications and Properties) are still present.

However, note that foreign keys still depend on us to actually create them! If we forget to create a foreign key, and forget to add { dependent: :destroy } to an association, we’re back at square one.

We quickly realized that we needed a comprehensive way to test deletion. We needed our test to be self-updating -- that is, the efficacy of the test shouldn’t rely on the engineer adding an association to update it. Here’s a simplified version of what we came up with:

describe "DeleteUser" do
  let(:user) { User.create! }
  let(:exclusions) { [:blacklisted_ips, :marketing_email_types] }

  before(:each) do
    action_item = ActionItem.create!({ user: user })

    Notification.create({ action_item: action_item })
    Properties.create({ action_item: action_item })
  end

  it "should exhaustively delete user" do
    tables = ActiveRecord::Base.connections.tables.reject { |s| exclusions.include?(s.to_sym) }

    tables.each do |table|
      expect(ActiveRecord::Base.connection.execute("SELECT count(*) from #{table}").first["count"])
        .to(be > 0, "#{table} is empty")
    end

    subject.call(user)

    tables.each do |table|
      expect(ActiveRecord::Base.connection.execute("SELECT count(*) from #{table}").first["count"])
        .to(eq(0), "#{table} is not empty")
    end
  end
end

Let’s walk through this test line by line.

tables = ActiveRecord::Base.connections.tables.reject { |s| exclusions.include?(s.to_sym) }

First, we retrieve the full list of tables in our database. We exclude some tables from our test. Exclusions are tables that aren’t associated with a specific user. For example, we have a dynamic list of IP addresses that we blacklist, which will not be mutated when a user deletes their account.

tables.each do |table|
  expect(ActiveRecord::Base.connection.execute("SELECT count(*) from #{table}").first["count"])
    .to(be > 0, "#{table} is empty")
end

Next we verify that our test is set up properly. We make sure that any table we expect to have data, has data. This step is extremely important. Consider an engineer building a new feature. If they add a new model, the tests will fail in CI, since their new table will have no entries. Only then are they forced to remember to update the DeleteUser test, and update the association to be properly destroyed.

Next, we call our service: DeleteUser.new.call(user)

tables.each do |table|
  expect(ActiveRecord::Base.connection.execute("SELECT count(*) from #{table}").first["count"])
    .to(eq(0), "#{table} is not empty")
end

Finally, we assert that all our tables are empty, which implies that our test user has been properly deleted.

Conclusion

While this test may seem simple, we’ve found that it’s an effective way for us to guarantee that we’re fully deleting our user’s data upon their request. When we run into false positives, we add tables to the exclusions list. This opt-in vs opt-out approach forces our engineering team to consider the implications of every database association we add, while still iterating quickly.

However, this is only one facet of building a delightful deletion experience. In our next post in the series, we’ll talk about our learnings in deletion performance, and how we optimized our user deletion interaction from minutes to seconds.

Tired of refreshing Github and Jira for notifications? Try Monolist and aggregate all your tasks in one place!

Getting Started with the Sorbet Type Checker in Rails

Akshay Nathan — Wed, 25 Sep 2019 17:13:57 +0000

At Monolist, we’re building a command center for engineers. We integrate with the APIs of all the tools engineers commonly use to aggregate their tasks in one place. For example, our customers trust us to pull in their Jira issues, Github pull requests, and Pagerduty alerts in real-time, so that they can triage, prioritize, and complete their work all from within Monolist.

Unfortunately, this is easier said than done. The APIs of these SaaS tools vary wildly, and assumptions about responses and return values propagate through our Ruby on Rails backend. When these assumptions are incorrect, because of Ruby’s dynamic nature, they manifest as runtime errors that cause issues for our users, and are difficult to debug. We realized quickly that we could benefit from types. While our frontend and mobile clients are written in Typescript, we didn’t have a good solution for our large rails API.

Enter Sorbet. Sorbet is a ruby typechecker written by Stripe. Like Typescript, Sorbet allows for gradual typing, which enables us to slowly introduce types to our codebase. In other words, we can reap the safety benefits of adding types file by file.

In this blog series, we’ll detail our experiences with adding Sorbet to a large-ish (> 100k LoC) Rails codebase. In this first post, we’ll add sorbet to the project, and add types to our first file.

Setup

We’ll start by adding Sorbet and the Sorbet runtime to our Gemfile. Sorbet provides both static analysis, and runtime checks. While the runtime checks will run in production, we’ll only need the static analysis in development, so we can separate the gems by environment.

gem "sorbet", group: :development
gem "sorbet-runtime"

According to the Sorbet documentation, at this point we can run:

bundle exec srb init

Unfortunately, here’s where we hit our first hurdle – the command never completed. The "srb init" command requires every .rb file in the repository, and uses runtime reflection to detect missing constants and generate RBI type definition files for your existing code. If you have a large number of vendored dependencies in vendor/bundle/ or node_modules/, this can be an extremely long (and sometimes infinite) process.

There are a couple open Github issues about this problem. Luckily, the help output from srb init gives us an easy way to solve this problem. We can simply add # typed: ignore to all the ruby files in our vendor/ and node_modules/ directories, and Sorbet will skip them.

for rb in $(find vendor node_modules -type f -name '*.rb');
  do sed -i '1i\# typed: ignore' "$rb"
done

Now when we run srb init, the command completes. We can now run:

➜  bundle exec srb tc
No errors! Great job.

Great!

Turning on the type-checker

The good news is we have no errors. The bad news is that we’re not actually type-checking most of our code.
This is because when srb init cannot typecheck an entire file due to missing method type signatures,
it will add # typed: false or # typed: ignore to the file.
This means that srb tc will not typecheck the types and methods defined in these files or their callers.

Let’s take a look at one of our files:

# typed: false
class CreateStripeBillingCustomer
  def call(user:, stripe_token: nil)
    return if user.email.match?(/test\+\d*@monolist.co/)

    stripe_customer = Stripe::Customer.create({ email: user.email, source: stripe_token })

    BillingCustomer.create!({ user: user, stripeid: stripe_customer.id  })
  end
end

This service is responsible for initializing a user with Stripe, our billing platform.
For a non-test user, it calls the Stripe API to initialize a customer, and stores the customer_id in our database.

Billing code is especially sensitive – you don’t want to double charge a customer because of an unexpected null object.
We figured that our billing code would be a good place to start adding Sorbet types, and this service is an especially good candidate because of its simplicity.

Let’s change # typed: false to # typed: true.
In # typed: true mode, Sorbet only checks types for methods that it knows about. That is, Sorbet
only checks methods that are explicitly annotated, either via hand-written type signatures or type definition RBI files.
For any other methods, Sorbet assumes their return values are T.untyped, which is a special type that matches all types.
While # typed: true isn’t the strictest static type checking level, we can start with it to enable us to gradually type our codebase.

Once we’ve changed the annotation, we can rerun srb tc.

➜  bundle exec srb tc
app/services/billing/create_stripe_billing_customer.rb:6: Method create does not exist on T.class_of(Stripe::Customer) https://srb.help/7003
     6 |    stripe_customer = Stripe::Customer.create({
     7 |      name: user.display_name,
     8 |      email: user.email,
     9 |      source: stripe_token,
    10 |    })
  Autocorrect: Use `-a` to autocorrect
    app/services/billing/create_stripe_billing_customer.rb:6: Replace with freeze
     6 |    stripe_customer = Stripe::Customer.create({
                                               ^^^^^^

Errors: 1

Hm, it appears that Sorbet doesn’t seem to know about the Stripe::Customer.create method.
In Sorbet, all methods must either include an inline type signature via a “sig”, which we’ll see later, or must include type signatures in a corresponding “.rbi” file.

Adding type signatures

Sorbet ships with .rbi files for the Ruby standard library. All other type definition files must either be hand-written, or pulled down from sorbet/sorbet-typed, a central repository of types for common gems.

When we run srb init, sorbet will automatically pull these type definitions for any gems installed in our gem file. Let’s see what it did:

➜  ls sorbet/rbi/sorbet-typed/lib
actionmailer  activemodel   activesupport railties      sidekiq
actionpack    activerecord  bundler       rainbow       stripe
actionview    activestorage minitest      ruby

At first, it looks like we pulled down all the definitions for the “stripe” gem. However, upon closer inspection:

➜  cat sorbet/rbi/sorbet-typed/lib/stripe/all/stripe.rbi
# This file is autogenerated. Do not edit it by hand. Regenerate it with:
#   srb rbi sorbet-typed
#
# If you would like to make changes to this file, great! Please upstream any changes you make here:
#
#   https://github.com/sorbet/sorbet-typed/edit/master/lib/stripe/all/stripe.rbi
#
# typed: strong

class Stripe::Card
  sig { returns(String) }
  def brand; end

  sig { params(other: String).returns(String) }
  def brand=(other); end

  sig { returns(Integer) }
  def exp_month; end

  sig { params(other: Integer).returns(Integer) }
  def exp_month=(other); end

  sig { returns(Integer) }
  def exp_year; end

  sig { params(other: Integer).returns(Integer) }
  def exp_year=(other); end

  sig { returns(String) }
  def last4; end

  sig { params(other: String).returns(String) }
  def last4=(other); end
end

It looks like the “sorbet-typed” definitions for the stripe gem are really sparse. This was new to us after learning to lean so heavily on DefinitelyTyped, which is the Typescript equivalent of a shared type library.

However, this is easily solvable. Let’s add a .rbi file of our own at "sorbet/rbi/lib/stripe/customer.rb".

class Stripe::Customer
  sig do
    params({
      email: T.nilable(String),
      source: T.nilable(String),
    }).returns(Stripe::Customer)
  end
  def self.create(email:, source:); end
end

Here we define the create method in accordance to the Stripe API documentation.
Our sig states that the method takes in an optional String email, and an optional String source, and returns an Stripe::Customer.
Let’s run srb tc again.

➜  bundle exec srb tc
app/services/billing/create_stripe_billing_customer.rb:8: Method id does not exist on Stripe::Customer https://srb.help/7003
     8 |    BillingCustomer.create!({ user: user, stripeid: stripe_customer.id })
                                                            ^^^^^^^^^^^^^^^^^^
Errors: 1

Let’s add the id method to our Stripe::Customer type.

sig do
  returns(String)
end
def id; end

And now…

➜  bundle exec srb tc
No errors! Great job.

Stronger Guarantees

Remember when we said that # typed: true isn’t the strictest type-checking mode? Now that we’ve resolved the errors with that level of strictness, let’s see if we can go further with the aptly named # typed: strict.

Once we changed the annotation, we can rerun srb tc.

➜  bundle exec srb tc
app/services/billing/create_stripe_billing_customer.rb:3: This function does not have a `sig` https://srb.help/7017
     3 |  def call(user:, stripe_token: nil)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Autocorrect: Use `-a` to autocorrect
    app/services/billing/create_stripe_billing_customer.rb:3: Insert sig {params(user: T.untyped, stripe_token: T.untyped).returns(T.untyped)}

     3 |  def call(user:, stripe_token: nil)
          ^
  Autocorrect: Use `-a` to autocorrect
    app/services/billing/create_stripe_billing_customer.rb:3: Insert   extend T::Sig

     3 |  def call(user:, stripe_token: nil)

In strict mode, Sorbet mandates that all methods must have a signature. Let’s add one:

sig do
  params({
    user: User,
    stripe_token: T.nilable(String)
  }).returns(T.nilable(BillingCustomer))
end
def call(user:, stripe_token: nil)

Now when we rerun srb tc:

➜  bundle exec srb tc
app/services/billing/create_stripe_billing_customer.rb:12: Method email does not exist on User https://srb.help/7003
    12 |    return if user.email.match?(/demo\+\d*@monolist.co/)
                      ^^^^^^^^^^
  Autocorrect: Use `-a` to autocorrect
    app/services/billing/create_stripe_billing_customer.rb:12: Replace with eval
    12 |    return if user.email.match?(/demo\+\d*@monolist.co/)
                           ^^^^^

app/services/billing/create_stripe_billing_customer.rb:14: Method email does not exist on User https://srb.help/7003
    14 |    stripe_customer = Stripe::Customer.create({ email: user.email, source: stripe_token })
                                                               ^^^^^^^^^^
  Autocorrect: Use `-a` to autocorrect
    app/services/billing/create_stripe_billing_customer.rb:14: Replace with eval
    14 |    stripe_customer = Stripe::Customer.create({ email: user.email, source: stripe_token })
                                                                    ^^^^^

Errors: 2

The user.email method is dynamically generated by ActiveRecord from the database schema. Unfortunately, Sorbet does not know about dynamically generated methods, and adding .rbi definitions for every column among our 100+ models would be incredibly tedious.
Luckily, the open-source project "sorbet-rails" allows us to autogenerate .rbi files for our models.

Once we install it with gem "sorbet-rails", we can run bundle exec rake rails_rbi:models to generate .rbi files for every model in our project. Let’s rerun srb tc:

➜  bundle exec srb tc
app/services/billing/create_stripe_billing_customer.rb:12: Method match? does not exist on NilClass component of T.nilable(String) https://srb.help/7003
    12 |    return if user.email.match?(/demo\+\d*@monolist.co/)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Autocorrect: Use `-a` to autocorrect
    app/services/billing/create_stripe_billing_customer.rb:12: Replace with T.must(user.email)
    12 |    return if user.email.match?(/demo\+\d*@monolist.co/)
                      ^^^^^^^^^^
Errors: 1

Whoa! Our first real type error! Let’s look into the generated signature in "sorbet/rails-rbi/models/user.rbi"

sig { returns(T.nilable(String)) }
def email; end

Looks like sorbet-rails assigned T.nilable to the return type of User#email. This is because the email column is nullable in our database. This is a legacy remnant from when email address was not required to sign up for Monolist. For these legacy users, we should simply ignore them and not attempt to create a customer on Stripe.

Let’s implement this:

def call(user:, stripe_token: nil)
  return unless (email = user.email)
  return if email.match?(/demo\+\d*@monolist.co/)

  stripe_customer = Stripe::Customer.create({ email: email, source: stripe_token })

  BillingCustomer.create!({ user: user, stripeid: stripe_customer.id })
end

Now, when we rerun srb tc:

➜  bundle exec srb tc
No errors! Great job.

Here, we can see the value of Sorbet’s “flow-sensitivity”. This means that Sorbet tracks types through program control flow. Although user.email starts as T.nilable(String), the conditional check on line 1 unwraps email to String, fixing the type error from before.

Conclusion

In this blog post, we setup our Rails API project with Sorbet, added type definitions for our models and some of our external dependencies, and fully typed our first file. While Sorbet still has some rough edges (inflexible initialization process, small type library), we were still able to relatively easily enable it’s static type-checking and discover our first real type error.

At Monolist, we believe that adding types is a great way to ensure the stability of the API integrations that our customers depend on. In the next posts in this series, we’ll integrate Sorbet into our CI system, enable runtime checking, and start adding types to more complicated code paths.

Want to see how Monolist can enable engineers like you to be more productive? Sign up for free here.

Liked this post? Join our mailing list for more content here.

Fail Fast and Fail Often: Handling API Errors at Scale

Akshay Nathan — Wed, 18 Sep 2019 18:19:04 +0000

At Monolist, we’re building the software engineer’s ideal inbox. Our users depend on us to surface all relevant and actionable tasks and context across all the tools and services they use. For a typical engineer, this includes emails, outstanding Slack messages, Github pull requests, and Jira issues or Asana tasks.

To do this, we make millions of API requests each day, spread over a dozen different services. These requests are made via ruby code running on multiple Sidekiq job processing containers across many hosts. (The following post uses Ruby and Sidekiq as examples, but the learnings are relevant to any language and job processing framework.)

As these requests succeed, much more processing has to occur to decide which data is relevant to the user, and how to present it to them. In the process of building Monolist, we’ve learned a lot about how to do this scalably, which is probably a good topic for another blog post. In this post, however, I want to share our learnings about what to do with the 100,000 requests that fail each day.

Retry early but retry correctly

When working with APIs, we generally see two classes of errors: Ephemeral errors are errors that may not occur when repeating the same request some time in the future. In contrast, persistent errors will consistently happen on each subsequent request. A connection timeout, for example, is an ephemeral error which will likely be resolved on retry. A 401 error, however, is probably indicative that a user is not authenticated and will not get automatically resolved.

Generally, persistent errors require us to change our business logic to make successful requests, while ephemeral errors will succeed on the next retry with no intervention.

At Monolist, we rely on Sidekiq’s default retry semantics for both classes of errors. For persistent errors, we deploy code fixes and let the next retry succeed. We see around 10 of these errors daily, and the failure rate decreases as our new integrations become more stable. The rest of our daily 100,000 errors are all ephemeral. While these errors usually resolve within 1 or 2 retries, we’ve had to carefully architect our systems to handle this failure volume successfully.

We believe in failing fast and often, rather than letting failed state propagate down our code paths with unintended consequences. As such, we raise exceptions for unexpected api responses, and avoid catching them in our workers.

Below is a simplified snippet of real code that we use to synchronize our users’ inboxes with Gitplace. Given a user and an api client, we poll Gitplace for the user’s pull requests, and create Monolist action items for each one we find.

def poll_gitplace(user, client)
  client.get_pull_requests.each do |pull_request|
    comments = client.get_pull_request_comments(pull_request)

    create_action_item(user, pull_request, comments)
  end
end

Note that if the api call at line 3 fails, Sidekiq will automatically retry the entire job, and hopefully it will succeed the next time. But is there anything wrong with this code?

The answer lies on line 5. Let’s say the job made it through 4 pull requests the first time, before the comments api call failed. Because Sidekiq will naively retry the entire worker, we’ll duplicate all 4 of the already created action items the next retry!

Obviously, this is not preferable. When relying on retries to solve ephemeral errors, it’s imperative that our jobs are idempotent. An idempotent job has no additional side effects when successfully rerun. In other words, we should be able to run our jobs N times, and expect the same results regardless of N. Let’s fix that.

def poll_gitplace(user, client)
  client.get_pull_requests.each do |pull_request|
    comments = client.get_pull_request_comments(pull_request)

    unless user.action_items.find { |s| s.gitplace_id == pull_request.id }
      create_action_item(user, pull_request, comments)
    end
  end
end

Now when our job is rerun, we will only create action items that haven’t already been created. This means Sidekiq can run our job however many times it wants, and the end result for our users will still be correct.

Always make progress

While we’ve fixed one bug, there’s another more silent and dangerous issue with the code above. Let’s say we have a user with 1000’s of pull requests. If we hit an ephemeral error, say a network timeout, while retrieving the comments for the 999th pull request, when the job retries we will start all over and make another 1000 api calls.

Even more problematic, even if our ephemeral error rate is low, an increased number of api calls increases our chance of hitting an ephemeral error. Thus, the jobs that are slowest and take the most resources, are not only the most likely to be retried again, but they’re the least likely to succeed on each subsequent retry! Even with Sidekiq’s automatic exponential backoff, this can debilitate our queue’s when we have hundreds or even thousands of expensive jobs failing and retrying together.

The solution to this problem is usually API specific, but follows a common principle: always track and make progress in your expensive jobs.

def poll_gitplace(user, client)
  time = user.gitplace_last_sync

  client.get_pull_requests({ created_after: time }).each do |pull_request|
    comments = client.get_pull_request_comments(pull_request)

    unless user.action_items.find { |s| s.github_id == pull_request.id }
      create_action_item(user, pull_request, comments)
    end

    time = pull_request.created_at
  end
ensure
  user.update!({ gitplace_last_sync: time })
end

In the above code sample, we store the last synced pull request created_at field. When polling Gitplace, we only retrieve pull requests created_after the last synced request. We then make sure that stored progress tracker gets updated regardless of whether there is an exception. That way, even if our job retries, we’ll only make the necessary api requests to continue.

Note: This approach does have concurrency implications that are outside the scope of this post.

Track your failures

While our code can now handle all the ephemeral errors we can throw at it, our monitoring systems are a different story. At Monolist, we use Sentry for exception tracking and alerting, but the following strategies are applicable regardless of how your team handles exceptions.

Obviously, we can’t have 100,000 irrelevant exceptions flooding our Sentry daily. However, we can’t catch the exceptions in code because we still need them to propagate up to Sidekiq so that the jobs are retried. We also don’t want to blindly ignore these exceptions -- while each incremental ephemeral error is usually just noise, we want to know if the rate of ephemeral errors changes significantly, to alert us to some larger problem with our integration, network, or service.

def track_exception(exception)
  redis.hincrby("tracked_exceptions", exception.class.to_s, 1)
  raise Monolist::TrackedException.new(exception)
end

def poll_gitplace(user, client)
  time = user.gitplace_last_sync

  client.get_pull_requests({ created_after: time }).each do |pull_request|
    comments = client.get_pull_request_comments(pull_request)

    unless user.action_items.find { |s| s.github_id == pull_request.id }
      create_action_item(user, pull_request, comments)
    end

    time = pull_request.created_at
  end
rescue Gitplace::ConnectionTimeout => e
  track_exception(e)
ensure
  user.update!({ gitplace_last_sync: time })
end

In the end, we came up with a simple solution where we explicitly track each ephemeral exception that we’re aware of. We wrap the exceptions in a Monolist::TrackedException wrapper which we’ve added to our Sentry blacklist so we don’t see them in Sentry. Since Monolist::TrackedException is still an exception, Sidekiq will still retry the job as usual. At the same time, we increment a counter in Redis to keep track of the number of exceptions we’ve seen.

In our monitoring systems, we’ve surfaced the “tracked_exceptions” key in Redis to our Prometheus instance. This allows us to add graphs like the one above to our dashboards, and alerting when the rate of exceptions changes significantly.

Conclusion

While the APIs we interface with at Monolist vary wildly, we’ve consistently found that the quality of our integrations depends heavily on how resilient we are to api errors, and how much of that complexity we can hide from the end user.

By abstracting away retry behavior, ensuring that jobs are idempotent, and making sure that we’re always getting closer to success, our end users are fully oblivious to the errors, and can focus on staying productive, writing code, and being the best they can be at their jobs.

Don’t believe me? Try out Monolist for yourself.