jq + httpie = 💓

Setting up

You'll need to install:

• jq
• httpie

For this demo, we'll connect to the Spotify api. Create a new Spotify application here

Client credentials flow

There is a great little plugin for httpie which makes this very easy. It needed a little update. So, for now I'll link my fork.

To install just clone the repo and run sudo python setup.py install

OAuth2 flow

Spotify uses OAuth2. In order to get your access token to make requests relevant to your spotify account, you'll need to acquire an access token. I used this cli tool written in go to accomplish this. You can find that here

oauth2-cli \
  -id $CLIENT_ID \
  -secret $CLIENT_SECRET \
  -auth https://accounts.spotify.com/authorize \
  -token https://accounts.spotify.com/api/token \
  -scope "user-read-private user-read-email user-library-read"

Making some requests

Downloading artist images from a search and then opening them

First, we'll make some requests that only require the client credentials flow.

#!bin/bash
AUTH_URL="https://accounts.spotify.com/api"
BASE_URL_API="https://api.spotify.com/v1"
CLIENT_ID=$(cat client_id)
CLIENT_SECRET=$(cat client_secret)

http --print="b" --auth-type oauth2 \
              --auth $CLIENT_ID:$CLIENT_SECRET \
              --issuer-uri "$AUTH_URL/token" \
              --scope "user-read-private" \
              GET $BASE_URL_API/search q=="Sophie Hutchings" type=="artist"

You should see a big chunk of JSON as output. Couple things to note here:

• CLIENT_ID=$(cat client_id): This is command substitution. There is a file in the directory I'm running this script that contains my client id.
• --print="b" is telling httpie to print the body of the request only (we'll pipe the body into jq later). Other arguments to --print are:
  • H: Request headers
  • h Response headers
  • B request body

Let's pipe this into jq and just get a list of images from each artist.

# Beginning lines omitted

## Simple search
ARTISTS=$(http --print="b" --auth-type oauth2 \
     --auth $CLIENT_ID:$CLIENT_SECRET \
     --issuer-uri "$AUTH_URL/token" \
     --scope "user-read-private" \
     GET $BASE_URL_API/search q=="Sophie Hutchings" type=="artist")

# List of artist images
echo $ARTISTS | jq '.artists.items | map(.images | map(.url)) | flatten(1)' 

This chunk begins with more command substitution to store the response from before into the ARTISTS variable. The contents of the variable is piped into jq to print a flat list of all the images from the artist we searched on. Let's cover some of the jq functions we used:

The simplest filter we used is the "Object Identifier-Index" .artists.items. This simply accesses a property within a JSON object.

{
  "artists": {
    "href": "https://api.spotify.com/v1/search?query=Bad&type=artist&offset=0&limit=20",
      "items": [...] // <--- .artists.items is grabbing this array
  }
}

The items array is piped into the map function. The map function iterates over each item returning a new array of image urls. Let's look at the JSON at this point:

[
  [
    "https://i.scdn.co/image/23009960c33ef08d5973440cca17985a6c70a515",
    "https://i.scdn.co/image/8ff3e392402169f239bce72b3d80c701b75150b8",
    "https://i.scdn.co/image/624dd15f5bdcc1bae5fa47739601f3e0be62ebda",
  ],
  [
    "https://i.scdn.co/image/23009960c33ef08d5973440cca17985a6c70a515",
    "https://i.scdn.co/image/8ff3e392402169f239bce72b3d80c701b75150b8",
    "https://i.scdn.co/image/624dd15f5bdcc1bae5fa47739601f3e0be62ebda",
  ]
]

An array of arrays isn't useful for what we're doing next. So we use flatten to return a flat array of all the image URLs.

"https://i.scdn.co/image/23009960c33ef08d5973440cca17985a6c70a515",
"https://i.scdn.co/image/8ff3e392402169f239bce72b3d80c701b75150b8",
"https://i.scdn.co/image/624dd15f5bdcc1bae5fa47739601f3e0be62ebda",
"https://i.scdn.co/image/23009960c33ef08d5973440cca17985a6c70a515",
"https://i.scdn.co/image/8ff3e392402169f239bce72b3d80c701b75150b8",
"https://i.scdn.co/image/624dd15f5bdcc1bae5fa47739601f3e0be62ebda",

Nice! Now, let's try to download these and then open them.

# Previous lines omitted

IMAGE_URLS=$(echo $ARTISTS | jq -r '.artists.items | map(.images | map(.url)) | flatten(1) | @csv')

# Download and open all the images from a search on artists
echo $IMAGE_URLS | tr ',' '\n' | \
   xargs -I % sh -c 'wget % && xdg-open %' \

We modified a couple things in our jq command.

• -r: raw output. In this example it omits "
• @csv: Omits csv for further processing. Only works on arrays.

The csv output by jq is piped into tr. The tr command replaces , with new lines. Each line is piped into xargs. Let's dissect the xargs command:

--I: the replace-str command. The argument is the string that will be replaced by the value from standard input in the final command. So % will turn into our image url.

• bash -c 'wget % && xdg-open %': Starts a new sub shell (each % is replaced by the image url)
• wget: Downloads files from the web
• xdg-open: Opens a file or URL in the user's preferred application

The result of running this should be a bunch of images being downloaded to your computer and then opened by whatever program is preferred by your user.

Inspecting your saved albums and printing all the tracks for each album

#!bin/bash
AUTH_URL="https://accounts.spotify.com/api"
BASE_URL_API="https://api.spotify.com/v1"
CLIENT_ID=$(cat client_id)
CLIENT_SECRET=$(cat client_secret)

# You'll have to run this first. Copy the access token to `auth_token` in the same directory as this script
# oauth2-cli \
#   -id $CLIENT_ID \
#   -secret $CLIENT_SECRET \
#   -auth https://accounts.spotify.com/authorize \
#   -token https://accounts.spotify.com/api/token \
#   -scope "user-library-read"

# Get a list of albums from your library
ALBUMS=$(http GET $BASE_URL_API/me/albums \
  Authorization:"Bearer $(cat auth_token)")

# List saved albums from your spotify profile and their artists
echo $ALBUMS | jq '.items | map({ "name": .album.name, "artists": .album.artists | map(.name)})'

# Get tracks from your albums
ALBUM_IDS=$(echo $ALBUMS | jq -r '.items | map(.album.id) | @tsv' )

## Separtes each album by a newline, for further processing and then make another request for each album to get a list of tracks
echo $ALBUM_IDS | tr ' ' '\n' | xargs -I % sh -c 'http --print="b" GET https://api.spotify.com/v1/albums/%/tracks \
  Authorization:"Bearer $(cat auth_token)" | jq ".items | map({ "name": .name, "artists": .artists | map(.name)})"'

Everything we covered in the previous section is used in here too! One difference is that we use map to return a list of objects instead.

Final thoughts

Using jq with httpie is powerful once you get the hang of using some of the linux utilities and idiosyncrasies of shell scripting. I'm still learning new stuff everyday. If you have any thoughts on how to improve this workflow, please share!

Comments

[Francisco Quintero 🇨🇴]

Great article and a very interesting way to use jq with HTTPie :D

thanks for sharing!

[Paul]

Thank you! I was definitely trying to illustrate the flexibility rather than the practicality of using jq and httpie to pull data from an API. Tapping into multiple utilities that may have their own domain specific language to learn can take a bit to wrap your head around, but can usually result in some compact (mostly) easy to understand programs.

It's a neat way to discover an API or just process a lot of data. I'm on the fence if this approach has any advantages over just using a general purpose programming language like Python, besides it being a bit of fun. 😆

[Francisco Quintero 🇨🇴]

Yeah. Sometimes it's good to get our hands dirty with UNIX-way tools. I think in the long term a programming language might be superior to just bare tools but sometimes just bare tools are really enough.

[Francisco Quintero 🇨🇴]

BTW, I also sometime ago built an script with just two utilities: aws and jq. Here it is gist.github.com/cesc1989/495642524...