A fully realized OAuth2 Flow in Bash

In this article, I describe how to implement a fully realized OAuth2 flow in a bash script. The goal is to minimize external dependencies while still maintaining full support. Features include:

• Open the browser to initiate the third party authentication flow.
• Automatically retrieve the token from the oauth2 callback.
• Store the refresh token for future script invocations.

The dependencies are minimal and available by default on most modern Linux distros:

• nc or netcat - used to listen for the HTTP callback from the oauth flow
• xgd-open or equivalent - opens the browser
• jq - parse the JSON responses
• grep and cut - extract values from strings
• curl or wget - make HTTP requests to the OAuth2 backend

For the purposes of this article I will demonstrate using the Google Data API OAuth2 flow but this technique should be adaptable to any standard OAuth2 flow.

Preamble

To start we will need an OAuth2 client ID and to register for API access through the API console. This process will obviously differ greatly for each API and every context you use an API in.

To follow along here you can get access to the Google Data APIs through the Google Cloud Developer Console. You'll need to create a new project (or select an existing one), access APIs and Services, go to Credentials, and create a new OAuth 2.0 client ID. The application type will be "Desktop". You will have to create a new consent screen if you do not have one already, the default values are fine here. This client ID will be for a test app with limited access, which is fine for testing. Google provides an explanation of OAuth2 and their specific flow here.

Once you have an OAuth2 client ID and client secret, we have everything we need to start. For this demo, we will store these in environment variables in the script:

CLIENT_ID="<...>.apps.googleusercontent.com"
CLIENT_SECRET="<...>"

I have elided my client id and secret here as they are secret values which provide access to your specific app for purposes of security, auditing, and billing. How do distribute these keys with an app is an open question I do not have a satisfactory answer to. A few options I've seen so far.

1. Encrypt the tokens within the app: this is of limited value, while it avoid the tokens appearing as plain text, the encryption key is inside a self contained app, ergo, it can still be reverse engineered.

2. Store the tokens securely on a server and provide access with a web service: this provides good security (or at least as good as the security you provide on your web service) but now makes what could be a self contained app dependent on your web service being online.

3. Require the end user to provide their own app credentials: this is obviously highly inaccessible to all but the most motivated and technically capable end users.

For a bash script, option 3 might be possible, and certainly this is not an issue if you are building a tool for your own use or within your own team or organization, but I will not address a specific solution in this article.

Invoke the browser

Now that we have the OAuth2 client secret and id, we can generate the OAuth2 authentication URL and open the browser at that location.

AUTH_URL="https://accounts.google.com/o/oauth2/auth\
?client_id=${CLIENT_ID}\
&redirect_uri=http://localhost:8000\
&scope=https://www.googleapis.com/auth/youtube.readonly\
&response_type=code\
&access_type=offline"

xdg-open $AUTH_URL 2>/dev/null

Here we create our authentication URL. It takes several parameters:

• CLIENT_ID and CLIENT_SECRET: for the developer app we created earlier.
• redirect_uri: where to pass the eventual authentication code back to.
• scope: the specific pieces of data which we are requesting permission to access.
• response_type: the specific authentication token we want returned. In this case we are asking for an authentication code which we can exchange for our access and refresh tokens later.
• access_type: the type of access we request, in this we request offline in order to get long-lived tokens that can be stored between invocations.

Finally we use xdg-open to invoke the URL. This will request that the OS open the URL in the default application (hopefully a web browser) and redirect any warnings to /dev/null just to make the script interface cleaner.

Retrieve the Authentication Code

In the OAuth2 flow, we invoke a browser to load the third party service's authentication page. In this case, we open a Google website, where the user will see a sign in page hosted by Google. They will then be presented with the permissions page to approve the request for data that we embedded in our scope parameter. Once the user approves the permissions, the Google page must pass the authentication code back to our script. This is where the redirect_uri parameter from earlier comes into play. The Google site will send a redirect to the URI that we specify but with the addition of the authentication code and other information added as query parameters in the URL.

So we need to do something to capture this callback request. The first thing we do is specify a redirect_uri of http://localhost:8000 meaning the request will come back (encrypted) to the machine we are running on at a port which is usually available (an enhancement to this script might be to verify the port is available before running the script and selecting a new port at random if the port is in use--this would increase reliability, but would increase dependencies).

Next we must listen for this request. A typical way to do this is to spin up a http server to process the request. This is rather expensive though and would be a significant dependency. But we don't actually need a full HTTP server. We don't even need to process an HTTP request exactly. We just need to extract the provided authentication code from the URL and then pass back a canned response. We can do this with nc or netcat.

RESPONSE=$(\
  echo -e "HTTP/1.1 200 OK\r\n\r\n\
    Authorization code received. You can close this page now." \
  | nc -N -l -p 8000)

AUTH_CODE=$(echo "$RESPONSE" \
  | grep -o "code=.*" \
  | cut -d'=' -f2 \
  | cut -d'&' -f1)

Here we run netcat or nc with the command nc -N -l -p 8000 to open port 8000 and to listen on that port using -l. We also tell nc to close the port after receiving an EOF on the input with -N. This ensures that nc terminates on it's own after the HTTP request is received and the response is sent.

Next, we pipe the HTTP response we wish to send back to the user into nc using the echo -e "HTTP/1.1 200 OK\r\n\r\nAuthorization code received. You can close this page now.". In this case, we're sending a simple plain text HTTP response starting with the HTTP status code 200 and OK message, with a simple plain text message instructing the user to close the browser window.

Finally, we capture the output of the nc command into the RESPONSE variable. That content will consist of the HTTP request that the browser made to the redirect_uri. This text will include the url that the browser loaded, which includes the query parameter ?code=<...>, we extract that query parameter using grep and cut, but it would also be possible to use other tools like sed or awk for this.

Now we have the authentication code.

Exchange the Authentication Code for Tokens

Once we have the authentication code, we must exchange it for the access and refresh tokens. The access token is the token we will use to make actual API requests, while the refresh token is the long lived token that we can store use to get a new access token as needed.

TOKEN_RESPONSE=$(curl -s --request POST \
  --url "https://accounts.google.com/o/oauth2/token" \
  --header 'content-type: application/x-www-form-urlencoded' \
  --data-urlencode "code=$AUTH_CODE" \
  --data-urlencode "client_id=$CLIENT_ID" \
  --data-urlencode "client_secret=$CLIENT_SECRET" \
  --data-urlencode "redirect_uri=http://localhost:8000" \
  --data-urlencode "grant_type=authorization_code")

ACCESS_TOKEN=$(echo "${TOKEN_RESPONSE}" \
  | jq -r '.access_token')

REFRESH_TOKEN=$(echo "${TOKEN_RESPONSE}" \
  | jq -r '.refresh_token')

Here we are using curl to make an HTTP request to the google endpoint which exchanges the authentication code for the access and refresh tokens. In this case, we use a POST request to end the endpoint url-encoded form data. We pass the authentication code, client id, client secret, and tell the endpoint we are using an authorization_code. The redirect_uri is required, but not used in this case.

The response to this request is captured and the we use jq to parse the JSON response and extract the relevant fields.

Finally we have the access token that we need to make API requests to the Google Data APIs. We can pass the token to curl to authenticate requests to the API either through an explicit header:

RESPONSE=$(curl -s --request GET \
    --url "https://www.googleapis.com/youtube/v3/playlists?..." \
    --header "Authorization: Bearer $ACCESS_TOKEN")

Or through the --oauth2-bearer parameter:

response=$(curl -s --request GET \
    --url "https://www.googleapis.com/youtube/v3/playlists?..." \
    --oauth2-bearer="$ACCESS_TOKEN")

Store and Reuse the Refresh Token

While we have an access token that we can use to make API requests to Google, that token will expire quickly. Thus, this authentication flow must be repeated every time the script is invoked. This is inconvenient for the user.

To improve on this, we will do is store the refresh token in a config file so that we can re-authenticate the script in subsequent invocations without needing to go through the OAuth2 authentication flow. First, lets set some paths.

TOKENS_DIR="$HOME/.config/<myscript>"
TOKENS_FILE="$TOKENS_DIR/tokens"

Here we just use the standard config directory on any Linux system, specifically in the users $HOME we store in a hidden directory .config and under that directory, we store in a directory named for our script. Our config file is simply named tokens.

Next, we store our refresh token in that directory.

mkdir -p "$TOKENS_DIR"
echo "REFRESH_TOKEN=$REFRESH_TOKEN" > "$TOKENS_FILE"

First we ensure the directory exists with mkdir -p, then we echo a string to set a variable and pipe it into our file.

Finally, as the first thing we do in our script, we can source this file and then check if the REFRESH_TOKEN variable is set. If it is, we use it to get a new access token, otherwise we do the full authentication flow.

source "$TOKENS_FILE"

if [[ -n $REFRESH_TOKEN ]]; then
  RESPONSE=$(curl -s --request POST \
    --url="https://oauth2.googleapis.com/token" \
    --header 'content-type: application/x-www-form-urlencoded' \
    --data-urlencode="client_id=$CLIENT_ID" \
    --data-urlencode="client_secret=$CLIENT_SECRET" \
    --data-urlencode="refresh_token=$REFRESH_TOKEN" \
    --data-urlencode="grant_type=refresh_token" )

  ACCESS_TOKEN=$(echo "$RESPONSE" \
    | jq -r '.access_token')
fi

if [[ -z $ACCESS_TOKEN || "$ACCESS_TOKEN" == "null" ]]; then
  #Full authentication flow
fi

Here we again use curl to send an HTTP POST request to the Google tokens endpoint with all the url-encoded form fields in the request. And again we use jq to parse the JSON response for the relevant field. Finally, we check if we have a valid access token and if we do not, we invoke the full OAuth2 authentication flow. (A quirk of jq means that if the access_token field is not present in the JSON response--perhaps because of an error or authentication expiry--the ACCESS_TOKEN field will receive the string value "null" which we must separately check for).

Each of these snippets can be combined together into a full script that implements the fully ergonomic OAuth2 flow a user would expect, but some judicious use of bash functions would improve re-usability and readability.

I hope you find it useful.