Building with the GitHub API

I have a long history of building things with the GitHub API.

• Many years ago when running the Docker open source project, where we needed tools for GitHub tasks automation (icecrime/poule) and project health metrics (icecrime/vossibility-stack).
• Today with Echoes, which integrates with GitHub (among others) to help engineering leaders create high performing organizations.

We will look in this post at an unordered list of things that I've learned along the way.

GitHub Apps or OAuth Apps

There are two ways to integrate with GitHub: as a GitHub App or as an OAuth App. While the difference is explained in GitHub documentation, it's worth reinforcing how neat GitHub Apps are.

• GitHub App use their own identity, not that of a user. In other words, actions performed by the app will appear as originating from the app itself, not "whoever lended a token".
• GitHub Apps have fine grained permissions. Considering the sensitivity of GitHub data, this is crucial in order to build integrations which follow the principle of least privilege. In the example of Echoes, this is how we drastically reduce our security requirements by not requesting access to the content of the git repository.
• GitHub Apps provide an excellent installation experience, which includes: installing on all or selected repositories, reviewing permissions and permissions scope updates.

If you're building in Go, bradleyfalzon/ghinstallation is an implementation of http.RoundTripper which authenticates as a GitHub App, for example to use in combination with google/go-github.

GitHub v3 or GitHub v4

GitHub has two active versions of its API: a v3 REST API, and a v4 GraphQL API. Which one you should build against is partly a matter of personal preference, but chances are high that you will end up using both.

The entire API surface is not available in both versions. For example, the Webhook Deliveries API referenced later in this post is only available in v3.

Another, trickier, aspect is that while GitHub Apps permissions naturally map to v3 endpoints, things are not as clear when using the v4. A GraphQL query requires the union of all permissions of its individual fields to succeed, or will result in a confusing “resource not accessible by integration” error. It is not always obvious which part of the query causes the error, especially when a seemingly equivalent call using the v3 REST API succeeds under the same permissions.

In our particular codebase, the interactions with the GitHub API appear to converge to using v4 GraphQL API for data retrieval, and to using v3 REST API for mutations (e.g., adding a label to an issue) or situations where v4 didn't behave the way we expected.

Defending against GitHub-side issues

GitHub, like any other complex system, is imperfect. Two issues we have seen are minor API downtimes (with 502 responses), and occasional authentication issues (with 401 responses) using a freshly-issued access token.

Both cases can be mitigated with automated retries. We went with hashicorp/go-retryablehttp as Echoes is built in Go. The library's default retry policy rightly doesn't consider a 401 response as a recoverable error, but this can be changed using the CheckRetry hook.

Defending against integration-side issues

Chances are that your integration won't be perfect either. What will happen to GitHub events when your app is unresponsive or fails to handle a message?

GitHub has a UI for inspecting recent deliveries and manually triggering new deliveries. This is immensely useful for development and debugging, but cannot help if you need to recover from any significant number of missed messages.

At Echoes, like we did at Docker, we're using our own durable queues to handle message processing. However, GitHub has since introduced the Webhook Deliveries API which allows listing past deliveries and triggering redeliveries.

Learn about secondary rate limits

GitHub API has two rate limiting mechanisms: one based on a number of requests per hour (5,000 in the case of GitHub Apps for github.com), and one based on the nature of the interactions themselves. The former is quite typical and easy to account for by examining GitHub API’s response headers. Complying with the latter, however, requires to be thoughtful and review how the system as a whole interacts with the GitHub API, including all services, tasks, and jobs in combination with various retry-policies.

Two of the guidelines to avoid hitting the secondary rate limits deserve particular attention:

• "Do not make requests for a single user or client ID concurrently."
• "If you're making a large number of POST, PATCH, PUT, or DELETE requests for a single user or client ID, wait at least one second between each request."

As a general rule of thumb, we would recommend against multiplying systems that will have to share the API budget.

Conclusion

GitHub is the place where most development work happens, and building tools for it can be extremely rewarding. GitHub puts a lot of effort in its platform, as seen by the introduction of GitHub Apps, and how the API continues to evolve.

We hope this post will help you with some of the less-obvious lessons of interacting with the GitHub API. If you curious to learn more about what we’ve built with it and how we believe it can help every engineering organization out there have more focus, more clarity, and more success, check us out at https://echoeshq.com!