DEV Community: Mark Michon

Six security risks of user input in ruby code

Mark Michon — Mon, 10 Apr 2023 23:02:18 +0000

We all know that user input should be treated extra cautiously. Any input field, GET request, or API endpoint is a risky entry point. With that in mind, here's a rundown of some of some especially dangerous ways you'll want to avoid using user input.

Unsafe deserialization

Deserializing untrusted data can be pretty sketch. Attackers can transfer payloads or malicious code disguised as serialized data, which you'll in turn unfurl. This can take the shape of image uploads for profile pics, specialized file formats, and more. Protect required formats like image uploads by validating them, and don't use any output from them in things like error messages.

In other instances, avoid it if you can. That wouldn't be much of a tip though, so if you have to consume serialized data, use a data-only and language-agnostic format like JSON—or XML if you must. These formats lessen the chance of any custom logic existing in the data. You can learn more in the OWASP cheatsheet on the subject.

Don't use Eval or system with user input

Eval might not come up often, but if it does you definitely don't want to use it on anything supplied by a user. Since eval will build new ruby code based on strings, attackers can use it to

Ruby's system, exec, spawn, and command are equally dangerous if the user input it making up the whole or partial command. Fortunately, as advised by the Rails docs, you can use system with a second argument to safely pass parameters. For example system(command, parameters.

Don't assemble paths or URLs

There are many instances where taking user selections or input, and forming a URL or path may be beneficial. Search is the most common instance, but you may also use it for uploading images, adding links, redirecting pages, etc.

Generally, you'll want to avoid directly building paths based on user input. The most common reason is to avoid forms of server-side request forgery. This type of attack can expose details about your own services, or be used to trick users into thinking a redirected path is coming from your application instead of the attacker's.

You can instead restrict selections to a subset of safe options, and then use those options to build the URL.

In instances where you can't possibly know all the safe values, you can match safe patterns—such as a list of approved domains for URL forwarding or file sources. The same goes for file types. When in doubt, it's better to be overly conservative when restricting the type of user input you use to build URLs or file paths.

Sessions

It's useful to start by making sure your sessions use a database as a backend rather than cookie based session storage.

In addition, you should avoid allowing user input as the session key. You want a system that has no outside influence, even though all session data sent from the client to the server should be considered untrusted. This allows you to properly validate session keys and IDs without the variability of any user-defined values.

unsanitized renders

With all the focus on sanitizing user input, it can be easy to neglect sanitizing renders. Ideally the user input shouldn't make it this far, but an additional step will prevent any mistakes on the receiving end.

If you're using render html you can wrap items in strip_tags to ensure code isn't generated in the HTML output. This helps prevent cross-site scripting (XSS) attacks in which content is injected into your page.

If you're using an external view engine, or a javascript framework like react in addition to your ruby backend, you can rely on similar sanitization methods like the DOMPurify library.

Stop runaway regex

We think of regular expressions as a common means of validating input, but some may find it valuable to allow users to build their own conditions. Allowing user input use in a regular expression isn't dangerous on it's own, but it can cause an issue when the regular expression requires excessive CPU resources.

To prevent this, specify a timeout when working with regular expressions—particularly those that rely on use input.

Regexp.new(/a|b/, timeout: 3)

This will help in stoping regular expression denial of service (ReDoS) attacks.

Keeping up with top ruby security tips

It can be challenging to keep up with security best practices. In addition to watching for vulnerability reports, you can also run regular scans on your codebase with a SAST tool like Bearer CLI. It's a free and secure way to get practical security feedback on your ruby code. Check it out on GitHub at bearer/bearer.

Let’s scan DEV’s forem project with Bearer and analyze the results

Mark Michon — Wed, 29 Mar 2023 18:13:59 +0000

Using open-source tools to test open-source projects feels like a great match. It wasn't until the other day that I remembered that the team behind DEV had open-sourced the bones of the site as Forem. To make it an even better match, the stack matches up nicely with the currently supported languages included in Bearer's new free and open-source security application security testing (SAST) tool. Unlike many security tools, this one is really focused on helping devs make sense of security concerns in an actionable way.

In this article, we'll install the Bearer CLI, run it against forem, and analyze the results to see what issues arise and how we might tackle them. Spoiler alert: we'll find a bunch of results—some easy to fix, some we can ignore, and some that might be larger judgment calls.

Install bearer

If you'd like to follow along, follow the install and scan instructions. Otherwise, you can skip down to the analyze the results section below.

The quickest way to install Bearer is with homebrew or curl.

Homebrew:

brew install Bearer/tap/bearer

Curl:

curl -sfL https://raw.githubusercontent.com/Bearer/bearer/main/contrib/install.sh | sh

You can then confirm that everything is installed and working by running the following:

bearer version

You should see the version number and sha value. At the time of this writing, the version is 1.1.0.

Clone forem and run a security scan

If you haven't already, go ahead and clone forem locally.

git clone https://github.com/forem/forem.git

It's a pretty big project, so it might take a few minutes. Once complete, navigate into the project—likely with cd forem.

We'll start by running Bearer with all of its default settings. This will give us a text-based security report.

bearer scan .

Note: The . in the line above just means "run this command on the current directory".

Depending on your machine, this could take a few minutes. For reference, the scan took 1m26s for me, with the rule analysis taking another 1s.

When the scan completes, you'll see the security report.

Analyzing the results

Here's a gist with the full output. The scan detected 0 critical failures, 44 high failures, 2 medium failures, and 36 warnings. That's 82 items that have the potential for improvement.

These failures and warnings are from Bearer's built-in rules. Rules are checks against best practices. Most are derived from the OWASP Top 10 and the CWE, and some are common best practices for languages and frameworks. At the time of this scan, there were 107 default rules.

Back to the results, 46 failures and 36 warnings seem like a lot, but if we break down the results we can manage them more easily. For one, let's ignore the warnings for now.

You can tell the scan to only show certain severity levels:

bearer scan . --severity critical,high,medium

Don't worry, the scan itself is cached, so this won't take the full amount of time again.

Of the 46 remaining failures, we can start to see that there are only a handful of rules that make up these failures. This is because Bearer shows you each location within your code where the failure occurred. These rules are:

Javascript: open redirect
React: dangerouslySetInnerHTML
Assorted ruby user input rules
Ruby: weak encryption
Ruby: data sent to Honeybadger

Remediating the failures

With our nice to-do list in place, let's start at the highest severity level—in this case, there were no critical items so we'll start at "high".

Javascript: open redirect

This rule, javascript_lang_open_redirect, is one I actually hadn't heard of before. If we look at the docs link provided in the output, we have some advice on fixing the problem and a handful of resources. We can see this relates to OWASP A01:2021 and CWE-601. If we review the OWASP open redirect guidance, it basically says:

Unvalidated redirects and forwards are possible when a web application accepts untrusted input that could cause the web application to redirect the request to a URL contained within untrusted input. By modifying untrusted URL input to a malicious site, an attacker may successfully launch a phishing scam and steal user credentials.

Still a bit unsure? Let’s look at the file and line output:

HIGH: Open redirect detected. [CWE-601]
https://docs.bearer.com/reference/rules/javascript_lang_open_redirect
To skip this rule, use the flag --skip-rule=javascript_lang_open_redirect

File: app/javascript/runtimeBanner/RuntimeBanner.jsx:63

 63     window.location.href = targetLink;

Looking at the RuntimeBanner component and heading down to line 63, we can then make a judgment on whether this should change. In this case, if we go a bit earlier, we'll see this:

...
const urlParams = new URLSearchParams(window.location.search);
const targetPath = urlParams.get('deep_link');
if (currentOS() === 'iOS') {
// The install now must target Apple's AppStore
installNowButton.href = FOREM_APP_STORE_URL;
// We try to deep link directly by launching a custom scheme and populate
// the retry button in case the user will need it
const targetLink = `${APP_LAUNCH_SCHEME}://${window.location.host}${targetPath}`;

retryButton.href = targetLink;

window.location.href = targetLink;
...

I don't know enough about their codebase to make a valid judgment here. It may be safe since it's only used for appstore links, but it also accepts get params as input for the redirect—which you normally want to avoid without properly validating them. Have a thought? Leave a comment below.

React: dangerouslySetInnerHTML

If you've done any React work, you know all about dangerouslySetInnerHTML. It's a necessary evil sometimes, but there is in fact a way to make it safer. That's why the rule javascript_react_dangerously_set_inner_html exists. Checking the doc link, we see the following advice:

Sanitize data when using dangerouslySetInnerHTML

<div dangerouslySetInnerHTML={{__html: sanitize(data)}}  />

Along with a link to OWASP's advice for further context. Great! We can include the DOMPurify library and it’s an easy fix.

Assorted ruby user input rules

I'm grouping many of the triggered ruby rules together here because they all have one thing in common: avoid doing things directly with user input. The following rules trigger in various places throughout the code:

ruby_lang_http_url_using_user_input
ruby_lang_reflection_using_user_input
ruby_rails_redirect_to

While different in their implementation, they all share the same risk and can more or less be resolved in the same way. If we look at the remediation docs for ruby_lang_reflection_using_user_input, the advice is to compare the input to pre-set values, then pass those values to the method instead of directly passing the input. Like this:

method_name =
    case params[:action]
    when "option1"
        "method1"
    when "option2"
        "method2"
    end
method(method_name)

There may be some cases where the scan picks up perfectly safe redirects and reflections due to the nature of params, so it’s good to evaluate these incidents on a case-by-base basis.

Ruby: weak encryption

The weak encryption rule, ruby_lang_weak_encryption, is one of my personal favorites. Many of us come to assume that most encryption libraries and algorithms are good enough, but it turns out that many of the old standards are showing their age. OWASP identifies algorithms it considers weak, so let's avoid using them to encrypt sensitive data.

MEDIUM: Weak encryption library usage detected. [CWE-331, CWE-326]
https://docs.bearer.com/reference/rules/ruby_lang_weak_encryption
To skip this rule, use the flag --skip-rule=ruby_lang_weak_encryption

File: app/services/mailchimp/bot.rb:153

 153       Digest::MD5.hexdigest(email.downcase)

In this case, it looks like the mailchimp service is only using MD5 to encrypt email addresses. We can replace it with something like BCrypt instead.

Ruby: data sent to Honeybadger

Finally, the last rule to trigger on forem's code, ruby_third_parties_honeybadger, is the following:

MEDIUM: Sensitive data sent to Honeybadger detected. [CWE-201]
https://docs.bearer.com/reference/rules/ruby_third_parties_honeybadger
To skip this rule, use the flag --skip-rule=ruby_third_parties_honeybadger

File: app/controllers/omniauth_callbacks_controller.rb:77

76 Honeybadger.context({
77 username: @user.username,
78 user_id: @user.id,
79 auth_data: request.env["omniauth.auth"],
80 auth_error: request.env["omniauth.error"].inspect,
...
82 })

You may wonder why this is a problem. In the case of this code, we're sending the user's username to a third-party service. While username isn't inherently sensitive data, it certainly has to potential to be and should be treated as such. It's better to use IDs that can't identify the user if the third party—in this case, honeybadger—is breached. You can see the full list of supported data types, sorted by category, on the docs.

What else can we do

On top of all the rules mentioned so far, there were also a handful of warnings—mostly around setting application-wide security. This is good advice, but depending on the use case might not be as vital. The great thing is, you can customize the scan to your needs by adjusting the flags, and even add comments to your code—much like you would with a linter—to ignore parts that are intentional choices. You can even write your own custom rules to check for any code patterns or security best practices that aren't included in the default ruleset.

The SAST security report is the core of the Bearer CLI app, but there are also privacy reports, secret detection, and a deeper dataflow report. You can even run it as a GitHub action each time new code gets pushed. Give it a try on your own apps, improve the code on some of your favorite open-source projects, and help make it better.

Let me know in the comments what other guides you'd like to see on using the CLI. And, if you work on forem, let us know if we can improve the results!

How to scan your ruby or JS project for security improvements, for free.

Mark Michon — Tue, 28 Mar 2023 00:35:22 +0000

Security tools are intimidating. They’re made for security teams that already know the jargon and the details like CWE identifiers. But what about developers? We have tools that check for vulnerable dependencies and tools that check for leaked secrets, but we’re missing easy—actionable—advice on making our code more secure.

Good news! There’s a free open-source tool that can scan your code, check for known risks, and give you a list of things that need fixing. All are sorted by how risky the code is—based on things like how sensitive the data is and how damaging a breach or leak would be. It’s called Bearer.

Here's why it's pretty rad:

Quick scans: most projects take under a minute, with big ones like forem, gitlab, etc taking between 3 and 10 minutes.
Your data never leaves your computer. The open source scanner reads your code, but doesn't send it or metadata to any servers.
Practical advice: Each triggered rule shows you where in your code the problem is, and links out to documentation on how to improve it.

TLDR: The workflow ends up looking like this:

bearer scan /your-project

You can run that locally, or as part of CI/CD, and each time you’ll receive a summarized report. Let’s get into it.

Installing Bearer

There is a full list of ways to install Bearer in the docs, but the most common are using Brew or curl.

Homebrew:

brew install Bearer/tap/bearer

Curl:

curl -sfL https://raw.githubusercontent.com/Bearer/bearer/main/contrib/install.sh | sh

Run your first scan

Now navigate to the project you’d like to scan, and run the scan command.

bearer scan .

The app scans your project in a few stages. It starts by detecting and classifying sensitive data types, then feeds that data into whichever report type you use. The default is the Security report, which shows all the security risks found in your codebase by checking against a set of “Rules”.

You get a summary that looks a bit like this:

107 checks, 12 failures, 6 warnings

CRITICAL: 0
HIGH: 1 (CWE-798)
MEDIUM: 11 (CWE-201, CWE-209, CWE-313, CWE-315, CWE-319, CWE-326, CWE-331, CWE-532, CWE-539)
LOW: 0
WARNING: 6 (CWE-312)

Plus each failure and warning shows you where it happened and has a link to docs on how to fix the problem.

MEDIUM: Sensitive data in a JWT detected. [CWE-315]
https://docs.bearer.com/reference/rules/ruby_lang_jwt
To skip this rule, use the flag --skip-rule=ruby_lang_jwt

File: lib/jwt.rb:6

 3     JWT.encode(
 4       {
 5         id: user.id,
 6         email: user.email,
 7         class: user.class,
 8       },
 9       nil,
    ...
 11     )

These rules come from the OWASP top 10, popular CWEs, and some general best practices from the appsec community. It’s a quick way to get a second pair of eyes on your code—especially if you aren’t a security expert.

Check it out

It's a big ask to put something in your pipeline or test flow, but I really love just using it as a one-off scan as I'm building something new. Kind of like linting, but for security. Right now the main security scan supports ruby and JS/TS codebases. Give it a try—you can use our test repo if you like. Let us know what you think and if there's something you'd like to see added open an issue on GitHub.

Measuring Performance in Node.js with Performance Hooks

Mark Michon — Fri, 11 Dec 2020 15:21:09 +0000

📣 This post originally appeared as Measuring Performance in Node.js with Performance Hooks on The Bearer Blog.

Measuring performance in Node.js applications can sometimes be a challenge. Due to the nature of the event loop and asynchronous code, determining the actual time a piece of code takes to execute requires tools built into the platform. First added in Node.js v8.5, as stableas of v12, the Performance Measurement APIs are stable and allow much more accurate monitoring than earlier implementations. In this article we'll look at the basics of Node.js performance hooks and how to use them to time the execution of functions.

Why are Performance Measurement APIs important?

Sometimes called Perf Hooks, in part because they are imported from perf_hooks in Node applications, these APIs allow developers to set various markers that make measuring the run-time of an application easier. Node's implementation is an adaptation of the W3C's Web Performance APIs, but with changes that make more sense for Node apps rather than browser javascript.

With these APIs, you can measure the time it takes individual dependencies to load, how long your app takes to initially start, and even how long individual web service API calls take. This allows you to make more informed decisions on the efficiency of specific algorithms, the effects of API choices on application performance, and establish baselines for "normal" operation to help identify anomalies when they happen.

In the past, this may have been done using Date.now() and some basic operations to find the duration. There are some flaws in this technique, as occasionally you can end up with a zero value or negative numbers. A slightly more accurate approach is to use process.hrtime(), but it still has limitations and needs to be manually set everywhere you end to use it.

To better understand how these newer APIs work, let's look at an example.

Using perf hooks

Imagine we have an asynchronous function called someAction, and we want to know how long it takes to run.

await someAction()

To track it's performance, first we need to:

Import the perf_hooks module from Node.js
Establish and observer to watch for performance events
Initialize the observer
Mark the appropriate areas with start/stop markers, and measure the difference.

Let's start by importing the module and setting up the observer—steps 1-3.

const { performance, PerformanceObserver } = require("perf_hooks")

const perfObserver = new PerformanceObserver((items) => {
  items.getEntries().forEach((entry) => {
    console.log(entry)
  })
})

perfObserver.observe({ entryTypes: ["measure"], buffer: true })

The observer code can look intimidating at first if you haven't used a similar API (like IntersectionObserver, for instance). In the code above we establish a new PerformanceObserver and pass it a callback function. Each time one of our performance events fires (more on that shortly), the entry is added to a list of Performance Entries (items). items.getEntries() is a bit of work that needs to happen to get the entries into an iterable format, which we then loop over with forEach and isolate each entry into the entry argument in the callback function.

Finally, perfObserver.observe tells our new observer what to look for and how to act. In this case, we want to watch for measure events (more on this shortly), and we set the buffer to true. This buffer setting just means that the observer will wait until all events are finished before running the PerformanceObserver callback. If it were set to false, items would always have a single entry and the callback would run every time a matching entryType occurred.

This boilerplate handles our setup, so let's actually measure the example function we spoke of earlier.

// ...setup code from previous example

performance.mark("example-start")
await someAction()
performance.mark("example-end")

performance.measure("example", "example-start", "example-end")

In the code above, we are using performance.mark and performance.measure. The mark method is used to place a performance timestamp in our code. The name can be anything, but using some form of start/end or similar suffix can help avoid user error. The measure method takes three arguments. A label for the measurement, the starting marker, and the ending marker.

With this done, our observer from earlier will pick up the measure type, add it to the callback argument, and when our code finishes we'll see the entry logged to the console. It will look something like this:

PerformanceEntry {
    name: "example",
    entryType: "measure",
    startTime: 3869.689664,
    duration: 122.123131
}

The startTime can be useful for organizing the data in logs, but we mostly care about duration as it indicates how long the operation took.

Monitoring function performance

In our last example, we set markers for the start and end of a code snippet. We did this in part because our function was asynchronous, and we were using the await keyword. If you're measuring the performance of a synchronous function, there is a helper available to handle the markers for you. Instead of setting a start and end with performance.mark, you can wrap the function in performance.timerify and change the observe code to watch for function entries.

// ...perfObserver = new PerformanceObserver...
// ...

perfObserver.observe({ entryTypes: ["function"] })

const perfWrapper = performance.timerify(syncFunction)

perfWrapper()

By changing the entryTypes to function and wrapping the synchronous function in timerify, we can avoid the need to set start and end marks.

Practical application for API calls

Let's imagine a scenario where we have calls to third-party APIs and we want to keep track of how long each call takes. We can use this data to create a baseline, track performance, etc.

Using the technique from our first example, we can start logging the performance. Here's what that looks like with the full setup code, and the addition of an external call using the Axios library.

const { performance, PerformanceObserver } = require("perf_hooks")
const axios = require('axios')
const customLogger = require('our-custom-logging-solution')

const perfObserver = new PerformanceObserver((items) => {
  items.getEntries().forEach((entry) => {
    customLogger(entry) // fake call to our custom logging solution
  })
})

perfObserver.observe({ entryTypes: ["measure"], buffer: true })


try {
    performance.mark('swapi-start')
    await axios.get('https://swapi.dev/api/people/1/')
} catch(err) {
    console.error(err)
} finally() {
    performance.mark('swapi-end')
    performance.measure('https://swapi.dev/api/people/1/', 'swapi-start', 'swapi-end')
}

In a scenario like this, we can imagine a codebase that manipulates and formats the performance data, and sends it off in batches to a monitoring tool or logging solution. Then, over time, establishes the normal behavior of an API in order to detect when anomalies occur—kind of like what Bearer does.

Using performance measurement to your advantage

Measuring and parsing the individual performance of all API calls can be cumbersome. That's part of the reason we built a tool at Bearer to monitor APIs using techniques similar to those mentioned here, along with the extra niceties of a full SaaS solution.

A similar addition to Node.js to keep an eye on is the Async Hooks API. It's still experimental, but can allow you to apply the features of the performance measurements API to asynchronous functions more easily. There's still room for improvement in our examples above and in similar methods. We also need to take into account any lag or pauses in the event loop.

We'll dive into async hooks and ways to use the performance monitoring API to monitor event loop lag in the future so subscribe for updates on The Bearer Blog.

The Importance of Request Timeouts

Mark Michon — Mon, 30 Nov 2020 16:27:48 +0000

📣 This post originally appeared as The Importance of Request Timeouts on The Bearer Blog.

Networks are unreliable. Third-party dependencies, like APIs, are unreliable. This is why we build resiliency into our applications and services. Most prep work is focused around HTTP requests, but an aspect of them that is often overlooked is timeouts.

What are timeouts?

Timeouts set a max wait time on a request. We can think about timeouts in two contexts: the client and the server. Client timeouts set a limit on how long they are willing to wait for a response to come back—and in some cases be completed. Server timeouts set a limit on how long the connection should remain open. For example, if a client is taking an exceptionally long time to receive a response, the server may choose to close the connection. Be careful not to limit your thinking of client as browser and server as backend. The client is the application making the request.

You can also consider timeouts to be split into three types: a connect timeout, a write timeout, and a read timeout. Connect timeouts affect how long the client will wait for a connection to establish. Write timeouts affect how long the connection will wait while the client tries to send data, like a POST request. Read timeouts cover the amount of time it takes to actually receive the response back from the server. Some HTTP clients allow you to set each separately, and other's only allow you to set a total combined value.

Timeouts are all about "waiting." When your application makes a request, it has to wait for the response. Even in asynchronous code, eventually that action needs to be handled and processed. Without a timeout established, your code relies on the assumption that it will either receive a successful response or an error. In reality, this isn't always the case. As we said at the start, networks are unreliable. If the request is delayed along the way, and never receives an official error response, your application will hang indefinitely. This is why timeouts are important. They stop the indefinite hang scenario.

Libraries can't be trusted

We make assumptions about "battle-tested" libraries and packages. Their heavy use adds clout, but sometimes the "good default" for a large audience isn't always the best default for your needs. For a few examples:

Requests, the leading Python library for making HTTP requests, does not have a default value for timeouts. They encourage you to set timeout values, but if you forget or assume a sensible default exists, your code will be susceptible to hanging connections.
Go's net/http module also has no default timeout. You can, and should, set your own timeout value.
The default browser implementations lack default timeouts, and in some cases lack easy ways to add timeouts. Both XHR and Fetch have issues, but most packages on NPM do incorporate ways to set timeouts. Axios, for example has a default of 0, which it interprets as infinite.
Ruby's net/http has 60s defaults for all timeout types. This is pretty high for most use cases, but is at least a default value.

As we can see, even if you're using the language's built-in request module or one of the most popular packages available, you will likely hit problems with insufficient timeouts.

Choosing a timeout value

Determining your ideal timeout involves many variables. Connection timeouts should be kept short, while read and write timeouts are more dependant on the needs of the service. There are some considerations to make when determining the ideal timeout duration for each API call.

One way is to monitor the response time from the APIs you connect to using a tool like Bearer. This works well for helping estimate the average time a successful request takes, and how long a successful error takes. Another approach is to focus on the user experience. If a request is connected to the UI—directly or otherwise—your timeouts should be tied to how long you're willing to delay the interaction. For background-style tasks, higher timeouts are appropriate. For more immediate tasks, shorter timeouts are a better option. Combined with optimistic updates and a retry strategy, timeouts can directly improve your application's UX.

Another variable to consider is serverless functions. If your code is making a request from a serverless endpoint, make sure the timeout is no greater than the max invocation time of the function. The function's max invocation time is often a hard limit. Combined with a little buffer for the time it takes the request and response to move between your application and the remote server, these values can better inform your timeout value. For example: timeout = maxInvocationTime + averageResponseTime, where the average response time is the time it normally takes. This isn't a perfect formula, but it's a great place to start.

Finally, don't punish users for their connection speeds. This primarily applies to browsers, but a user on a 3G connection may require a significantly longer timeout when uploading an image than a user on a faster connection. Keep this in mind for any user-facing connections.

So what should you do?

Ideally, set specific timeouts for each API call your application makes. Use the needs of the interaction, or your services, to determine what the maximum timeout should be. If you're using an HTTP library or module that doesn't surface an easy configuration for this, consider switching to one that does.

Setting custom timeouts for every request can be tedious, especially when you don't have the data necessary to make an informed decision. One alternative approach is to use a solution like Bearer's active remediations to define max timeout values for specific types of requests. For example, you can set all third-party APIs that you know are further away to a higher timeout than those with local servers. This allows you to manage these limits separate from your codebase, which makes experimentation easier. It also makes adapting to unexpected network conditions easier.

Whatever you do, don't neglect timeouts. Don't assume the defaults are safe, and don't assume the network is reliable. Keep your applications resilient against problems that third-party APIs may experience and your users will never know when major issues happen. If you want to learn more about Bearer and our active remediation features, give our in-app agent's a try today.

API Security Best Practices

Mark Michon — Wed, 07 Oct 2020 09:38:59 +0000

By nature, APIs are meant to be used. Even if all of your users are internal, security problems can still arise. To help with this, we've assembled a list of best practices to keep in mind when securing and locking-down an API or web service.

Use HTTPS

The web has moved past standard HTTP. With browser vendors flagging URLs that don't use a secure layer, it's time to do the same for your API. HTTPS uses Transport Layer Security (TLS) to encrypt traffic. This means communication between the client and server is encrypted. For your API, this means the content sent from your API is secured from third-parties, but more importantly it means that the access credentials are secured.

Authenticate

Speaking of access credentials, the clearest way to avoid unexpected use of your API is to ensure proper authentication. Authentication is how you allow or prevent access to the API. Even publicly available APIs that are free to use should consider an authentication strategy. This allows you to limit or remove users that abuse the API, and also protect your users by giving them the ability to reset their credentials if needed. You can learn more about types of API authentication in our article on the three most common authentication methods.

Authorize

Authentication's sibling is authorization. Where authentication is concerned with who the user of your API is, authorization focuses on what they have access to. For example, free plan users may only be authorized to access a subset of your full API. When you think about integrations like social login, the user authorizes your application to read their profile data from the social platform.

Secure endpoints AND resources (Object level authorization)

It's common to secure an endpoint or set of endpoints through authorization. A more future-proof approach is to secure the individual resources as well. This prevents misconfigured, endpoint-level authorization from reaching your data. It also means that the endpoints themselves aren't restricted by user type, but instead the resource controls who can and cannot view it.

Rate limit

When we think of security, we often think of inappropriate access. It can also be useful to think of security as managing resources. Rate-limiting is a technique for throttling the usage of an API. It can protect your resources financially, but also ensure that your servers aren't overloaded by a flood of requests at one time. Many rate-limiting approaches are time-based. They can be set for a billing period to handle overall usage, as well as use a "burst" approach to limit large influxes of requests. If you've ever seen the 429 HTTP status code, you've experienced rate-limiting.

Validate and sanitize input

One of the oldest attack points for web applications is input fields. The way we access data may have changed, but the need to validate any user input hasn't. Client-side validation is helpful for preventing mistakes and improving the user experience, but your API needs to also validate and sanitize all input before acting on it. Sanitization strips malicious or invalid code from requests. Validation ensures that the data meets the necessary criteria that your resources expect. This could be type and shape, or even factors like password structure.

Expose only what is needed

It can be tempting to take a shortcut and directly map data models to endpoints. Not only can this provide overly-verbose responses and increase bandwidth usage, but it can also expose data that users don't need access to. For example, consider a /user endpoint that returns the user's profile. It may need some basic information about the user, but it doesn't need the user's password or access levels.

Configure error messages

In addition to sanitizing data that goes into your API, you'll want to sanitize the information that comes out of it. Error messages play a key role in helping users understand that a problem occurred, but make sure not to leak any sensitive data. Providing end-users with details about the structure of your internal code can open up areas for attackers to focus on. Make sure to configure error messages to provide enough information to help users debug and enough for them to report problems, but not enough to expose the inner workings of your application or sensitive data.

Don't expose sensitive info

To build on protecting sensitive data, make sure not to expose details in JSON web tokens (JWTs) or cookies. The body of a JWT have the illusion of being secure, but can easily be decoded. Because of this, you should avoid including user information that could be used to access your application. The same advice goes for URLs as well. Make sure query strings aren't exposing sensitive details.

Assess your dependencies

We no longer develop in a silo. A good portion of every codebase now contains libraries, middleware, and a variety of dependencies from outside sources. While it is generally safe to assume that popular packages are "battle-tested", that doesn't mean they aren't completely safe from vulnerabilities. Make sure to assess your dependencies. Are they well maintained? Are you using the latest version? Is there a history of problems? Perhaps more importantly: Do you really need a library for what you're doing?

Allow users to track and reset authentication keys

One additional way to improve the security of your own API is to allow users to reset their credentials and monitor their own usage. A common mistake is not allowing users to reset their API keys. If a consumer of your API exposes their key accidentally, or it is taken maliciously, the problem now directly affects your API. Instead, create a means for them to manage access.

Standardize auth across your services

We've seen the big players in API like Google, Microsoft, and Amazon standardize the authorization and authentication process for their APIs. Consider a centralized means of doing this, like using an API gateway or a dedicated point of entry for handling authentication requests.

Follow standard guidelines from OWASP

In addition to these best practices, consider adopting recommendations from The Open Web Application Security Project (OWASP). They offer platform-specific guides as well as an upcoming API-specific guide, The API Security Top 10. Beyond securing your API on a code-level, you'll also want to ensure that your servers and infrastructure are configured properly to avoid unauthorized access.

Protecting your API from other APIs

If your API relies on third-party APIs, consider implementing a solution like Bearer. Integrating the Bearer Agent will allow you to track, observe, react, and receive alerts when an API isn't performing as expected. Try out Bearer today, and connect with us @BearerSH.

Top Tools to Make Debugging APIs Easier

Mark Michon — Tue, 06 Oct 2020 15:09:20 +0000

Integrating with APIs can be challenging. Each with its own documentation format, SDK patterns, and authentication quirks. Once you get the integration up and running, the harder step is testing and debugging any problems. This part can be exploratory—before you finish building—or as part of problem resolution.

While you can certainly debug within your codebase, by testing methods and reviewing logs, it can be beneficial to isolate the API calls by using one of the many API testing tools available.

Rather than list the pros and cons of each, let's focus on what makes each tool unique and worth trying.

Postman

Cost: Free + Paid Plans

Postman is an institution in this space. It's been around for years, and went from a small app that made API calls into the full ecosystem that it is today. While Postman's primary use case is building and managing your own APIs, their emphasis on encouraging third-parties to publish collections has made it a great tool for exploring and debugging web services. These collections are what make Postman a fantastic tool for debugging external API calls.

Some providers, like Zendesk, link directly to their Postman collections in their documentation. Others can be found on Postman's Explore page, and then opened within the Postman app. Once installed, you can explore endpoints and resources with minimal setup.

Learn more about Postman.

Hoppscotch

Cost: Free

Hoppscotch started as a faster, browser-first alternative to Postman, called Postwoman. It has since been rebranded and continues to add features. Its main benefits are speed and availability. It is also open source, and can even run as a browser extension or progressive web app (PWA).

Learn more about Hoppscotch.

Paw

Cost: Free Trial, then $49.99

Moving on to platform-specific applications, Paw is a paid Mac application for interacting with APIs. Like many of the others, it's core focus is on providing a tool to build, test, and document your own APIs. Perhaps Paw's greatest selling point is the UI. Compared to many of the other application's on this list, it really feels like quality Mac-first software. It also offers Team functionality, and an interesting feature called Pawprint that lets you share request/response pairs via a URL, even if the receiver doesn't use Paw.

Learn more about Paw.

Insomnia

Cost: Free + Paid plans.

Insomnia is another API client that lets you design, debug, and test APIs. Unlike some of the other "all-inclusive" applications, Insomnia splits the API designer functionality into a seperate app. This makes the UI much more approachable, and makes it easier to quickly try out a new request without excess setup and configuration.

One additional tool that you can find in their documentation is a CLI version called Inso. It allows you to use the functionality from both their core Insomnia client as well as the designer app, right from the command line, and even from within CI/CD environments.

Learn more about Insomnia.

Fiddler

Cost: Free plan + additional Paid plans

While the rest of the applications in this list are centered around allowing you to directly interface with APIs, Fiddler takes a different approach. It is more of a general purpose web debugging tool. It acts as a proxy to capture local requests between your device and the internet. This means you can run your application locally, and inspect any outgoing requests and incoming responses from the Fiddler application.

This approach can be useful for live-debugging, but it also allows you to mock requests and responses that match specific rules. Combined with features around creating collections of requests and responses, and the ability to collaborate with a team, this approach could really help debugging more complex API problems.

Learn more about Fiddler

Finding the right tool

Choosing the right tool, not just for the task, but also for your workflow can be a challenge. Fortunately, all of the options we've looked at here are free or offer free trials to get you started. They can all help make exploring and debugging APIs easier. When it comes to debugging API problems in your live site, Bearer has a solution. We can detect problems automatically, help you track API performance, and you can enable detailed rules to watch for edge cases. Check out Bearer today, and take control of your API usage.

Sort, Filter, and Remap API Data in Python

Mark Michon — Wed, 30 Sep 2020 13:30:17 +0000

Are you taking data from an API in the format the web services gives it to you? You should not dictate the structure of data inside your application based on how an API provider structures their data. Instead, you can take advantage of the power of Python's list manipulation techniques to sort, filter, and reorganize data in ways that best suit your needs.

In this article, we'll explore a few methods of using native Python features to sort, filter, and remap data from an external REST API that you can then use within your application, or pass down to a client as part of your own API. Looking for techniques to do this in Javascript? We wrote about that in a previous article.

To follow along, you'll need to have Python3 installed and be familiar with running Python code.

Retrieving data from an API

To get started, we'll need some data to manipulate. For the examples in this article, we can use GitHub's v3 REST API and its search/repositories endpoint. The endpoint takes a few parameters, but we'll use q to pass in the Bearer search query, and per_page to limit the results to 10. This makes it easier to follow along if you are printing results to the console, but feel free to increase the returned results per page if you want a larger dataset to work with. You can also change the search query to anything you like, as it won't affect the manipulations that we'll perform.

You can use any HTTP client, but our examples will be in Requests. Begin by installing requests and importing it into your project.

To install it, use pip (or pip3, depending on your local setup):

pip install requests

Next, import requests and set up the API call.

import requests

def get_repos():
  res = requests.get('https://api.github.com/search/repositories?q=bearer&per_page=10')
  return res.json()['items'] # Returns the value of the 'items' key from the JSON response body

The code above defines a function, get_repos that will make the API call and return only the part of the response that we are interested in. In this case, the items list on the JSON response. From this point forward, whenever we need the items data, we can assign it to a variable with x = get_repos(). This will leave us with a list of dict, or dictionary, data types.

Now with the setup out of the way, we can begin manipulating the data from the API.

Sorting results in Python

While many APIs allow you to sort the results in some way, rarely do you have access to sort by any property. GitHub, for example, allows sorting by stars, forks, help-wanted-issues, and how recently a repo has been updated. If you want a different sorting method, you'll need to handle this on your own. One thing to keep in mind is that we are only going to sort the results we retrieve. This means that if you capture the first ten results, sort them, then capture the next 10 results, they will not all be in the right order. If working with large datasets, it's best to capture all the data before sorting to avoid repeating the work on the same set of data multiple times.

Python offers the sorted built-in function to make sorting lists easier. sorted takes the list to you want to sort, and a key representing the value to use when sorting the results. Let's take the following example: Sort the repositories by open issue count, from fewest issues to most issues.

# This line can be used in each example, but you only need it once in your file
repos = get_repos()

fewest_issues_sort = sorted(repos, key=lambda repo: repo['open_issues_count'])

The code above passes repos in to the sorted function, and then uses a lambda function for the key. Lambda functions in Python are anonymous functions that you can use in-line without defining them elsewhere. They are great for use-cases like ours. This lambda is saying: "pass each iteration of repos to this function as repo, then return its open_issues_count." The sorted function uses the open issues count as the key. The result is a new list of dictionaries sorted by fewest issues to most issues.

What if you want to reverse the order to show repos with the highest issue count first? You can achieve this with the reverse argument in sorted. By setting reverse to True the same logic will apply, but in reverse.

most_issues_sort = sorted(repos, key=lambda repo: repo['open_issues_count'], reverse=True)

You can use the same technique to sort by any value.

Filtering list data with Python

Filters pair well with sorting. They allow you to reduce a list down to only the entries that matter for your needs. There are a variety of ways to filter a list, but they all use the same concept of building a new list from a subset of the original list's entries.

One of the most effective ways to do this is using list comprehension. List comprehension is a method for creating lists, but it can also simplify complex loops into shorter expressions. Let's take an example where we only want repositories in our list that have a description filled out.

# Fetch the repos if you haven't already
repos = get_repos()

repos_with_descriptions = [repo for repo in repos if repo['description'] is not None]

The code above does a lot with a single line, so let's break it down. It assigns repos_with_descriptions to a new list, [ ]. The first repo in repo for represents what we want to add to the new list. The next part repo in repos should look familiar from any for-in loop. Finally, the end is our condition: if repo['description'] is not None. When a description is not set in the GitHub API, it returns null to the JSON response. Python interprets this as None, so we check to see if the description is not set to None since we only want repositories with a description.

What if we want two conditions? For example, only repositories that have a description and a homepage URL set.

repos_with_desc_and_home = [repo for repo in repos if repo['description'] is not None and repo['homepage'] is not None]

As with our previous example, this code says: "Make a new list made up of each repo in repos, but only include those where the description is not None and the homepage is not None."

In both examples we've used the is not operator. Let's look at a more conventional condition and only include repositories with 100 or more stars.

popular_repos = [repo for repo in repos if repo['stargazers_count'] >= 100]

The code again uses list comprehension, but this time includes a more traditional greater-than-or-equal condition to compare values. You could also implement any of these examples with your preferred looping and list-generation methods.

Remapping or normalizing data

Sometimes an API returns more information than you need. GraphQL solves this by allowing the client to specify exactly the data it needs, but for now most APIs are still REST-like. If you need to reduce the data down to only a few properties before you interact with it or send it on to other parts of your app, you can do so by iterating over the list and building a new one with only the parts you want.

Loops or list comprehension can do this, but we will use Python's map function. map applies a function to every item in an iterable. Let's look at an example where we take the original repos list and simplify it into fewer key/value pairs with map.

def simplify(repo):
    return {
        'url': repo['html_url'],
        'name': repo['name'],
        'owner': repo['owner']['login'],
        'description': repo['description'],
        'stars': repo['stargazers_count']
    }

new_repos = map(simplify, repos)

Here, map receives the simplify function which will run over each item in repos. You can use this same method to manipulate the data further, normalize values, and even strip sensitive information.

Beyond the basics

The techniques mentioned in this article are a great starting point to handle the majority of your use cases. API's mostly return JSON, which Python can convert into dictionaries. In our examples, we captured the items list from the response dictionary and used it alongside many built-in Python functions to iterate over the data.

Whether the data coming from an API is the core of your product offering, or only a part of the value you provide to your users, you need to be able to manipulate it to best suit your needs. You also need to know that the source of this data is reliable.

How vital is this API data to the success of your app? What happens when the APIs experience performance problems or outages? There are safeguards you can put in place to avoid and manage downtimes and protect your integrations from failure. At Bearer we're building a solution that automatically monitors API performance, notifies you of problems, and can even intelligently react when a problem happens. Learn more about Bearer and give it a try today.

API Monitoring: What should you measure?

Mark Michon — Wed, 23 Sep 2020 13:02:54 +0000

When it comes to monitoring third-party APIs and web services, what you monitor is as important as how you monitor. Data is useful, but actionable data is where the true value is. Below we've listed the most common and valuable metrics to monitor when relying on third-party API integrations and web services. Accurate monitoring and alerting can provide your business with the data it needs to make decisions about which APIs to use, how to build resilient applications, and where to focus your engineering efforts.

Here are the metrics that we recommend when you start to monitor an API or web services:

Latency
Response Time
Availability
Consumption
Failure rate
Status Codes

Latency

Latency is the time the message spends “on the wire”. Here, the shorter the number, the better. Latency can be caused by the connection between your server and the API server. It can also be caused by delays that occur between your server and the API server. This may be the result of network traffic or resource overload—where throttling the requests might accommodate the heavy load.

To monitor latency, the web service needs to track timestamps for the outgoing and incoming requests and compare them to past and future requests over a given time. This can still be tricky, as the responses from the server will also be affected by response time. If available, pinging an endpoint or calling a health-check endpoint can be the best way to receive an accurate latency estimate.

This evaluation can be useful when positioning servers geographically. By determining the lowest latency your business can make decisions as to which provider to select. You can also select specific regional provider services if it is determined that the latency is the true cause for delayed responses, or select different providers if the response time of their resources is the problem. In actual practice, latency and response time will often be combined as a single value.

Response Time

Where latency takes into account the delays in the network itself, response time is the time it takes a service to respond to a request. This can be harder to track with third-party APIs and web services, as the latency to send and receive data is part of the response time. You can estimate response time by comparing the response time across multiple resources on a given API. From this, you can estimate the shared latency between the API's servers and your servers, and decide what the true value is.

The response time has a direct effect on your application's performance. Delays in the response of an API will result in slower interactions for your users. You can avoid this by ensuring your chosen API providers have response time guarantees, or by implementing a solution that uses a fallback API or cached resources when spikes are detected.

Availability

The availability of an API can be described as either downtime or uptime. Both are based on the same data, but can tell a different story depending on the context.

Availability is perhaps the easiest metric to keep track of. Downtime errors are recognizable and sometimes expected as an API provider will announce scheduled outages. However, even the most reliable APIs experience unforeseen downtimes. Downtime can be presented as individual events, or as an overall average across a given period. While downtime quotas and assurances such as "99.999% uptime" can be valuable when assessing an API provider, even the smallest downtime can have large impacts on your application.

Many APIs rely on external providers such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Services. As a result, the downtimes of an individual web service provider are also now dependent on a third-party that your application does not directly do business with. Even if the API provider's services are running as expected, the third-party may not be. As a result, when large downtimes occur, you will want to have a fallback in place that does not rely on the same underlying provider as the original API.

While similar to downtime in the way it is measured, the uptime of an API can provide insight into business decisions. If you know that one API has better uptime during key business hours for your customers, you can use this metric to move between providers.

Where some stakeholders may respond to downtime when selecting which API provider to drop, others may be more likely to respond to uptime when considering which to select. These values are linked, however, they can tell a different data story.

Consumption

It can be easy to forget usage, or consumption, when monitoring APIs. Internal APIs may not require a usage metric, but telemetry into third-party API consumption can aid in making business decisions. Estimating costs when consuming a web service can be difficult without the appropriate data. Consumption can be evaluated as a whole, or in bursts. Some API providers bill on a monthly scale, but some may have rate limits on their pricing tiers that also watch for usage over a smaller time window.

By keeping track of consumption and setting alerts for high usage, you can avoid unnecessary costs. Additionally, recognizing when APIs are not being used can also be beneficial. A lack of consumption is a sign that an API is still part of your codebase, but may not be vital to your application. In this case, you can adjust feature priority and gain insight into the usage of your application.

Consumption is best viewed as a running value, and filterable by a time window. This allows dashboards to provide an overview, as well as granular details about when an API is being used

Failure rate

There are a variety of reasons a request will fail. When a request to a third-party API or web service fails, it may be from user error, API downtime, rate limiting, or a variety of network-related issues. While API failures can sometimes be caused by your application, when it comes to tracking third-party APIs you want to focus primarily on failure rates out of your control.

Tracking failures and determining failure rates can aid in:

Reporting problems to the API provider
Deciding between multiple API providers
Making informed decisions related to fallback scenarios
Building resiliency around certain resources

Some errors may come from invalid requests. These can tell you that your application needs to adjust internal validation before making a request. Errors that come from server-related issues, like status codes in the 400 and 500 ranges, are a sign that the problem is likely with the API or web service provider.

Status Codes

Tracking HTTP responses can give you granular details about an individual API, but tracking specific status codes can give you better insight into the type of problems. For example, some API providers will respond with a 200 OK status, even when an error occurred. This false metric may lead you to believe that everything is working as expected, but users may experience problems and your internal logging may tell a different story.

Comparing status code metrics from API providers with internal error logs can provide additional insight into the true error rates of the third-party web services your application relies on.

Wrapping up

With these metrics in mind, your applications can better handle the unavoidable issues that will arise when relying on third-party integrations.

Measuring all these metrics may sound like a daunting task. Fortunately, some developer tools, like Bearer, can aid in both monitoring many of these metrics and reacting to problems that arise automatically.

Use Javascript's Array Methods to Handle API Data

Mark Michon — Mon, 21 Sep 2020 13:03:27 +0000

📣 This post originally appeared as Use Javascript's Array Methods to Handle API Data on The Bearer Blog.

Manipulating data is a core skill for any developer. In an API-driven environment, so much of the data you receive is formatted in a way that doesn't directly match the way that your application or UI needs it. Each web service and third-party API is different. This is where the ability to sort, normalize, filter, and manipulate the shape of data comes in.

In this article, we'll explore some common ways to work with data in Javascript. To follow along, you'll need to be working with code in either Node.js or the Browser.

Retrieving data from an API

Before we start, you need some data. For the rest of the examples in the article, we'll be using the data returned from a search of GitHub's v3 REST API. We'll use the search/repositories endpoint to make a query for repositories matching a search term (the q parameter, in this case set to bearer). We're also going to limit the number of results to 10 per page, and only one page. This makes it more manageable for our examples.

Start by using Fetch to connect to the API, and wrap it in a function with some basic error handling. You can reuse the function later in each of our examples.

const apiCall = () => fetch('https://api.github.com/search/repositories?q=bearer&per_page=10').then(res => {
  if (res.ok) {
    return res.json()
  }
  throw new Error(res)
})
.catch(console.err)

If you're using Node.js, you can use the node-fetch package to add Fetch support. Install it to your project with npm install -S node-fetch. Then, require it at the top of your project file.

const fetch = require('node-fetch')

We'll also make use of async/await. If your platform (like some versions of Node.js) doesn't support top-level async/await, you'll need to wrap the code in an async function. For example:

async function example() {
 // Code here

    let results = await apiCall()

// More code

}

With the setup out of the way, let's get started handling the response data. The results from the API call provide you with an object that contains some general metadata, as well as an array of repositories with the key of items. This lets you use a variety of techniques to iterate over the array and act upon the results. Let's look at some example use cases.

Sorting results

Many APIs, including GitHub's, allow you to sort the results by specific criteria. Rarely do you have full control over this. For example, GitHub's repository search only allows ordering by stars, forks, help-wanted-issues, and how recently an item was updated. If you need results in a different order, you'll have to build your own sorting function. Let's say you want to sort results by the number of open issues the repository has. This means the repository with the fewest issues should show first, and the repository with the most should show last.

You can achieve this by using Array.sort along with a custom sort function.

// Sort by open issues
const sortByOpenIssues = repos => repos.sort((a,b) => a.open_issues_count - b.open_issues_count)

// Run the apiCall function and assign the result to results
let results = await apiCall()

// Call sort on the items key in results
console.log(sortByOpenIssues(results.items))

To understand what's happening, lets look at how sort works. The method expects a specific return value:

A value less than 0 means the first value is greater than the second, and should come before it in the order.
A value of 0 means both values are equal.
A value greater than 0 means the second value is greater than the first, and should come before it in the order.

The easiest way to work with these conditions is to subtract the second value from the first. So in our code above, you subtract b.open_issues_count from a.open_issues_count. If the number of issues for "a" is greater, the result will be greater than 0. If they are equal, it will be zero. Finally, if b is greater the result will be a negative number.

The sort method handles all of the movement of items around for you, and returns a brand new array. In the example above, two values are compared, but you can use any calculation that results in the criteria mentioned above to sort the results of an array.

Filtering results

Sorting changed the order of our data, but filtering narrows the data down based on specific criteria. Think of it as removing all of a certain color of candy from a bowl. You can use Javascript's built-in filter method on arrays to handle this. Similar to sort, the filter method will iterate over each item and return a new array. Any Let's look at a few filter scenarios.

In the first, we'll create a filter that only shows repositories that contain a description.

// Filter only repos that have descriptions
const descriptionsOnly = (repos) => repos.filter((repo) => repo.description)

let results = await apiCall()
console.log(descriptionsOnly(results.items))

In this case ,we're returning the truthiness of repo.description to represent whether the API returned a value or null. If the current iteration in the loop returns true, that iteration's item is pushed to the new array.

What if we want only repositories that have both a description and homepage URL? You can modify the previous example to achieve this.

// Filter only repos with URL and description
const homeAndDescription = repos => repos.filter(repo => repo.homepage && repo.description)

let results = await apiCall()
console.log(homeAndDescription(results.items))

Using Javascript's AND operator (&&), you can check that both the description and URL exist. If they do, the whole expression returns true and the item in the array is added to the new array. If either are false, the whole expression is false and the iteration will not be added to the new array.

What about something a little more complex? Lets say you want all repositories that have been updated after a certain date. You can do this by setting a threshold and comparing it to the updated_at value on each repository.

// Set a threshold

let date_threshold = Date.parse('2020-08-01')

// Filter over results and compare the updated date with the cutoff date
const filterByDate = (repos, cutoff_date) => repos.filter(repo => Date.parse(repo.updated_at) > date_threshold)

let results = await apiCall()

console.log(filterByDate(results.items, date_threshold))

Just as in the previous example, the truthiness of the returned value in the function passed to filter determines if the item is added to the new array.

Changing the shape of data and format of data

Sometimes the data you receive isn't what you need for your use case. It can either include too much, or it can be in the wrong format. One way to get around this is by normalizing the data. Data normalization is the process of structuring data to fit a set of criteria. For example, imagine these API interactions are happening on the server, but the client needs a subset of the data. You can re-shape the data before passing it down to the client.

const normalizeData = repos => repos.map(repo => ({
    url: repo.html_url,
    name: repo.name,
    owner: repo.owner.login,
    description: repo.description,
    stars: repo.stargazers_count
})

let results = await apiCall()

console.log(normalizeData(results.items))

In the above code, the map array method is used to iterate over the results. It returns a new array made up of the values you return. In this instance, the data from the repos is simplified to include only a few key/value pairs, and the names of the keys have been made more digestible.

You can even use this time to modify any data. For example, you could wrap repo.stargazers_count in Number() to ensure the count was always a number and never a string.

Wrapping up

Managing the data you receive from an API is a critical part of any API integration. Every API will return a different shape of data, in their own format. The exception being GraphQL APIs that give you more control over the shape, and sometimes the sort order and filtering options.

Whether you are using the data as part of a larger data-processing effort, or using it to improve the usefulness of your application to your users, you will need to perform some actions to make the data digestible for your app.

These API integrations are integral to your application, but what happens when they fail? We've written here before about some of the actions you can take to protect your integrations from failure. At Bearer, we're also building a complete solution to monitor the performance, notify you of problems, and automatically remediate problems with the APIs your app relies on. Learn more about Bearer, and give it a try today.

Machine Learning APIs for Web Developers

Mark Michon — Wed, 16 Sep 2020 14:55:13 +0000

Machine learning (ML) used to be a tool limited to specialized developers and dedicated teams. Now, thanks to many web service providers and approachable tooling, your applications can use pre-build learning models and machine learning techniques the same way you would use any web service API.

This is a quick way to test out and benefit from machine learning without having to invest in artificial intelligence, building your own learning models, or shaping your application around ML. In this article, we'll focus on:

The big players in ML APIs.
Smaller companies and startups offering API services.

What can you do with a machine learning API?

When we talk about artificial intelligence (AI) and machine learning (ML) in these contexts, they will sometimes be used interchangeably.

At a basic level, machine learning is the process of training a computer to understand the data it receives on a level deep enough to make decisions. Artificial intelligence is the decisions it makes and as a result, the decisions that your application uses.

All the big web services players are now involved. Microsoft Azure, Google Cloud, Amazon Web Services, and the IBM Cloud all provide similar products with a few unique exceptions. The types of services offered revolve around the types of learning and prediction that your app needs. They can help you do things like:

Focus on a specific part of an image the user uploads. Twitter, for example, crops image previews to center on text.
Provide recommendations based on purchase or browsing history.
Predict outcomes based on a set of inputs (forecasting).
Handle language processing to translate text and autocomplete sentences.
Provide conversation-style chat bots that can use sentiment analysis to better understand the customer.
Detect anomalies in your data by looking at past trends.

Many providers allow you a way to build upon existing machine learning models of specific types. This allows developers to feed the machine information, without the need to deeply understand how everything works.

For example, most APIs that provide computer vision or image recognition use a pre-built model that already knows what to look for in an image. They can detect common items like buildings, household objects, and animals.

Your team's developers then provide images to the machine learning prediction API in the programming language you prefer, and it uses the pre-built model to make a prediction. Similar techniques can be used for tracking movement, facial recognition, and text analysis.

The big players in ML APIs

When we talk about machine learning APIs, we are really talking about machine learning as a service (MLaaS). Rather than run the full stack of ML tooling yourself you rely on a provider to handle the processing and, to some extent, the training of the machine.

Many providers now offer similar services at competitive price points. For the most part, you can stick with your preferred provider, but there are a few instances where some offer a better solution.

Nearly every provider offers services like computer vision, facial recognition, text translation, and text-to-speech. Let's look at a brief overview of each provider, with a few of their services highlighted.

Amazon

Amazon has expanded Amazon Web Services (AWS) to include a variety of ML and ML-adjacent services. The Amazon machine learning APIs offer nice hooks into the AWS ecosystem.

Key offerings:

Recommendations: Their Amazon Personalize service gives you the power to use a recommendation system similar to Amazon's to offer "related content" style recommendations to your users. It uses your own dataset, and a simplified interface to creating and training models.
Transcription: One of the first to offer transcription services, Amazon Transcribe will let you transcribe recorded videos or even streams in near real-time.
Code review: Amazon CodeGuru is a newer offering that analyses your codebase and uses machine learning to alert you of expensive code and potentially problematic code. They use their own internal codebase and over 10,000 open source GitHub projects to train the machine learning models used by this API.

Amazon's pricing differs with each machine learning API and can be found on each APIs product page. You can see the full list of AWS' ML and AI-related offerings on the Machine Learning on AWS page.

In addition to their machine learning APIs, Amazon also offers solutions for teams looking to host and deploy their own learning processing and ML applications.

Google

Google's machine learning offerings are heavily focused on being a platform for hosting, training, and managing ML models. That said, they have begun to offer many more machine learning APIs for developers to take advantage of.

Some of the more interesting APIs are centered around Google's AI Building Blocks. These are similar to drop-in tools that can add AI and ML features to existing apps without the need for in-house expertise.

Key offerings:

Language Processing: Translation is the most obvious machine learning API offered, but they also offer text-to-speech, speech-to-text, sentiment analysis, and dialog flow for conversational interfaces.
Vision: Machine learning developers can build upon existing models that Google offers, but also take advantage of pre-built features like emotion detection, classification, text OCR, and more.

Google offers a generous free tier for the majority of its machine learning and AI products. Developers that need more from the Google cloud can view their pricing information on an API by API basis.

Microsoft

Microsoft's cloud services offering, Microsoft Azure, groups its machine learning APIs under the title of Cognitive Services. With similar offerings to the other providers, services can be broken down into the key categories:

Vision: Focused around APIs related to object detection, classification, and recognition. These services can do things like detect faces, recognize documents, and train custom models to identify images.
Language: Just like most providers, Azure also offers translation, sentiment analysis, and natural language processing. Beyond this, they offer a few interesting APIs such as Immersive Reader, which adds additional information to text for learners.
Speech: Developers can access the Speech Service API to generate synthetic speech. A new offering that stands out amongst the other providers is Speaker Recognition. It allows developers to identify and verify speakers using existing data.
Decision: The decision APIs offer tools like recommendations, similar to Amazon's offering, but also provide Content Moderation and a new Anomaly Detection service. Anomaly detection analyses data to look for outliers over time.
Search: Utilizing the foundation of Microsoft's Bing search engine, this API falls under the cognitive services umbrella, even though many developers may not classify it as a machine learning API.

Like all of our providers in this list, Azure's pricing is on a usage basis. All services are managed from within the Azure Portal. See the breakdown for each API on their pricing page.

IBM

Often forgotten next to AWS, Google Cloud, and Azure, IBM's web service offerings are vast. All of their AI/ML APIs are grouped under the Watson brand. While many of their offerings are similar to the other providers, here are a few interesting differentiators:

DevOps insights: Like Azure's anomaly detection, IBM offers Watson AIOps to help developers resolve and detect incidents.
Automated Assistant: Watson's big sell is its Watson Assistant. It is an AI-powered chat interface for responding to customers. It works both as a text-chat on sites, but also on call systems to let users ask questions in natural language.

While the audience for IBM services tends to skew toward the enterprise, they offer generous free tiers for teams and developers looking to try out their services. See pricing for individual APIs on their pricing page.

ML and AI Startups offering APIs

The landscape of startups providing ML and AI APIs is a bit more volatile, with some going out of business after a few years. The big players dominate the space, but there are smaller startups and open source APIs that offer interesting takes on common features. These range from traditional web service APIs, to full software suites. Both traditional developers and learning developers alike will find value in services like language processing, image recognition, and more. Here are a few players in the space:

Kairos: Kairos offers a variety of services centered around facial recognition. Some of their features include face detection, verification, identification, tracking, and insights into facial features like age and gender.
TextRazor: A Natural Language Processing (NLP) API, TextRazor does more than attempt to understand the text. It works to provide additional meaning by pulling useful information from the text, like business names and places, to provide more information about the intent and meaning behind the text.
RxNLP: Another API that aims to provide value to text content, RxNLP has a few APIs that do things like comparing the similarity of text and group similar types of text.
Imagga: Imagga offers an array of image recognition services ranging from categorizing, automated cropping of key items, color extraction and modification, and even content moderation of inappropriate images.

More companies continue to spawn as the accessibility of ML and AI tooling increases. Expect to see startups targeted at specific industries to offer unique ML APIs in the future.

How should you choose a machine learning API?

As we look over the available APIs from the providers above, we can see that they all offer similar services. Unless you need one of the more edge-case offerings, it is often best to stick with the platform that you already use. This allows billing, management, and credentials to all exist in one place.

It is also important to consider maturity and performance. While large platform providers are more stable, they are still susceptible to downtime.

Always make sure to use a monitoring and remediation tool when utilizing third-party APIs.

At Bearer, we offer a tool to automatically detect anomalies, notify your team when problems happen, and even remediate API issues for your application. Check it out and get started for free.

Making API Requests with Python

Mark Michon — Wed, 29 Jul 2020 14:00:18 +0000

Python is in the midst of a resurgence. It never went away, but usage now grows like never before. With machine learning developers and data scientists relying on Python, much of the web development ecosystem around the language continues to grow.

One aspect that affects all three of these specializations is the powerful benefits of APIs. Pulling in data, and connecting to external services, is an essential part of any language. In this article, we'll look at the primary libraries for making HTTP requests, along with some common use cases that allow you to connect to an API in Python. Before that, we should ask an important question.

Is Python good for APIs?

It seems like a strange question, but given the large web presence of Node.js and Ruby, you may think that Python isn't good for making API calls. This isn't true. In fact, Python has had a long and dedicated presence on the web, specifically with it's Flask and Django libraries.

As Python is a powerful, accessible way to manipulate data, it makes sense to also use it for acquiring the data sources. This is where API calls come in. Let's start with the most popular Python HTTP library used for making API calls. Requests.

Requests

Requests is the accessible, leading library that developers use for making API requests in Python. It offers an interface to make HTTP requests synchronously. Let's get right into some common types of requests you can make with Requests. The following examples will all assume that your project includes Requests. You can follow their installation instructions, but the gist is:

Install it via pip or pipenv:

pip install requests

Then, make sure to import requests into your project

import requests

Common types of API calls with Requests

The most simple GET request is intuitive.

response = requests.get('https://example.com')

As we can see with the get method above, Requests offers shortcut methods for the HTTP verbs, including POST, PUT, DELETE, HEAD, and OPTIONS.

The previous request is pretty simple. Let's look at more complex requests. Often, an APIs documentation will require that you pass query parameters to a specific endpoint. To pass query parameters, we can pass them into get as the second argument.

response = requests.get('https://example.com', params={'name': 'Bearer'})

The response variable contains the data returned by the API in our examples. There are three primary ways to access the data.

As text, with response.text
As bites with response.content
As JSON with response.json()
Or as the raw response with response.raw

In addition to the body of the response, we can also access the status code with response.status_code, the headers with response.headers, and so on. You can find a full list of properties and methods available on Response in the requests.Response documentation.

As we saw with the params argument, we can also pass headers to the request.

response = requests.get('https://example.com, headers={'example-header': 'Bearer'})

Here, we pass the headers argument with a python dictionary of headers.

The last common API call type we'll make is a full-featured POST, with authentication. This will combine the previous headers technique with the use of the data argument.

url = 'https://example.com'
headers = {'Authorization': 'Bearer example-auth-code'}
payload = {'name':'Mark', email: 'mark@bearer.sh'}

response = requests.post(url, headers=headers, data=payload)

This sends the payload as form-encoded data. For most modern APIs, we often need to send data as JSON. In this next example, we use the built in json helper from requests.

url = 'https://example.com'
headers = {'Authorization': 'Bearer example-auth-code'}
payload = {'name':'Mark', email: 'mark@bearer.sh'}

response = requests.post(url, headers=headers, json=payload)

This will encode the payload as JSON, as well as automatically change the Content-Type header to application/json.

Requests is excellent for synchronous API calls, but sometimes your app may depend on asynchronous requests. For this, we can use an asynchronous HTTP library like aiohttp.

aiohttp

When making asynchronous HTTP requests, you'll need to take advantage of some newer features in Python 3. While the requests library does have variations and plugins to handle asynchronous programming, one of the more popular libraries for async is aiohttp. Used together with the asyncio, we can use aiohttp to make requests in an async way. The code is a little more complex, but provides all the additional freedom that async calls provide.

To get started, we'll need to install aiohttp.

Install aiohttp.

pip install aiohttp

Common types of API calls with aiohttp

We will begin with the same GET request we saw earlier. To start, import both libraries, and define an async function.

import asyncio #  [1]
import aiohttp

async def main(): # [2]
    async with aiohttp.ClientSession() as session: # [3]
    async with session.get('http://example.com') as resp: # [4]
        response = await resp.read() # [5]
        print(response)

asyncio.run(main()) # [6]

In the code above, we perform the following:

We import the required libraries.
Define main as an async function.
We set up a ClientSession from aiohttp.
We use the session to perform an HTTP GET request.
Next, we await the response and print it.
Finally, we use the run method of Python's asyncio to call the asynchronous function.

If you haven't worked with async in Python before, this may look strange and complicated compared to the earlier examples. The makers of aiohttp recommend setting a single session per application and opening/closing connections on the single session. To make our examples self-contained, I have left the examples in the less efficient format.

Next, let's look at a full-featured POST with auth headers, like in the requests example.


# ...

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.post('http://example.com',
            headers={'Authorization':'Bearer 123456', 'Content-Type':'application/json'},
            json={'title':'Try Bearer'}) as resp: # [1]

            response = await resp.json() # [2]
            print(response)

asyncio.run(main())

There are a few differences between this example and the previous:

The session uses the post method, and passes in headers and json dictionaries in addition to the URL.
We use the library's built-in json method from the response to parse the returned json.

With these two snippets, we're able to perform the majority of common API-related tasks. For additional features like file uploading and form data, take a look at aiohttp's developer documentation.

Additional libraries to try

While Requests is the most popular, you may find value in some of these additional libraries for more unique use cases.

httpx: httpx offers both sync and async support. It also uses a requests-compatible API that will make moving between the two much easier. Currently the library is in a beta state with a 1.0 expected min-summer 2020, but it is worth keeping an eye on as it matures.
httpcore: Keeping with the pre-1.0 trend, httpcore is an interesting option if you are building a library. It is low-level, so you can build your own abstractions on top of it. They explicitly recommend not to use it unless you need a low-level library.
urllib3: We shouldurllib3, if only because it is the the underlying library that Requests and many others (including pip) are built on top of. While less user-friendly than some of the high-level libraries, urllib3 is powerful and battle-tested. If for some reason you need something with fewer abstractions than requests, it is a good option.

Making the most of API calls

With Python's power in data processing and recent resurgence, thanks in part of the ML and data science communities, it is a great option for interacting with APIs. It is important to remember that even the most battle-tested and popular third-party APIs and services still suffer problems and outages. At Bearer, we're building tools to help manage these problems and better monitor your third-party APIs. Give Bearer a try today, and let us know what you think!