Automated morning music with Pipedream, Huginn, and Mailbrew

Get a great music album delivered to the inbox every morning, with a Spotify link so that you don't have to find the album yourself. Sounds good? Let's make it happen.

We will break this down into these parts:

1. Get music albums
2. Finding out if said music album is any good
3. Get the Spotify link to the album
4. Deliver this link every morning. By email or chat.

One of my finds after using this for a while

We can solve the first problem in many ways. We can use the /r/music subreddit on Reddit or even Pitchfork Medias "best new music". What I settled on is Metacritic. They have a simple feed of albums that is easy to develop for. There is also little risk ending up with Led Zeppelin IV in your inbox like you would on Reddit (although that would be good too).

To do this we are going to use Pipedream. It is a more developer oriented Zapier alternative for automating workflows. One of the benefits compared to Zapier is that you can run any Node.js code you like as a step. I've written about Huginn in the past as a self-hosted Zapier alternative. You can do everything in Huginn if you wish but I wanted an excuse to give Pipedream a try.

The pitch for Pipedream

So what you need to do is sign up for Pipedream first. They have a very generous free tier which will be more than enough for this project. Once you've that you go to the sources tab and create a new source. The easiest way to get data out of Metacritic is to use their official RSS feeds, so create a new source and select RSS.

All you have to do now is name your feed and put https://www.metacritic.com/rss/music in the Feed URL. Pipedream will now emit events for every new item in the feed.

Step 1 is now complete. We have a way to get music albums. Step 2 will be a bit more complicated. The album review score is not available in the RSS feed and Metacritic does not make this available through any official means. You can solve this problem by scraping the Metacritic site, but there is a better way. Some awesome person has wrapped a GraphQL API around Metacritic. The thing is available for free over at https://mcgqlapi.com/api.

Using GraphQL we can now get album metadata using a query like so:

query {
  album(input: { album: "Ignorance", artist: "The Weather Station" }) {
    url
    productImage
    criticScore
  }
}

Fantastic! Now we can get critic scores. This works for games, movies, and tv shows as well so you can use this for all kinds of projects. There is a problem though. The RSS feed does not make a distinction between Album and Artist, and we need both to be able to use the API. To solve this we need to scrape Metacritic.

To create a scraper we will once again make use of Pipedream. Select the workflow tab and create a new workflow. Select "USE ONE OF YOUR EXISTING SOURCES" and choose the RSS source we created earlier. Then click the "+"-icon to add an action and select "Run Node.js code" to create a new custom action.

There are many ways to fetch the needed data. I chose to use Cheerio. Add the following code:

const axios = require("axios")
const cheerio = require("cheerio")

async function fetchHTML(url) {
  const { data } = await axios.get(url)
  return cheerio.load(data)
}
const $ = await fetchHTML(event.link)

const album = $(".product_title:first").text().trim()
const artist = $(".band_name:first").text().trim()

return {album, artist}

That should do the trick. The return value will be usable in future steps in Pipedream.

Now, you might have noticed that we could grab the score as well using Cheerio. We don't have to use the GraphQL API for this, so why do it? I did it to learn how to do GraphQL with Pipedream. Feel free to change the Cheerio part above to also include score and then skip the next step. There is also more data available in the GraphQL so have a look around if you want to add something else.

We now have an album. Time to figure out if it's any good! Making a GraphQL request with Pipedream is easy. Choose "Run Node.js code" again and use the following snippet:

const fetch = require("isomorphic-fetch");

const response = await fetch('https://mcgqlapi.com/api', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ query: `
query {
  album(input: { album: "${steps.metacritic_data.$return_value.album}", artist: "${steps.metacritic_data.$return_value.artist}" }) {
    url
    criticScore
    album
    artist
    productImage
  }
}` 
  }),
})

const jsonData = await response.json()
if (!jsonData.data.album) {
  $end()
}
const {url, criticScore, album, artist, productImage} = jsonData.data.album 

if (criticScore < 75){
  $end()
}

return {url, criticScore, album, artist, productImage}

You can access the data from previous steps through the ${steps.metacritic_data.$return_value.album} variables. I decided that I only want albums with a Metacritic score of 75 or above. $end()will end the workflow in Pipedream and it will make sure that we do not proceed to the next step.

Next up: I only want albums that are available on Spotify, because I intend to listen to the album that morning. This is another one of those cases where Pipedream is great since it has built-in support for the Spotify API. Simply select Spotify for the next step and search for "Search". You are looking for the GET /search endpoint. You have to authorize with Spotify to use this.

This step takes the parameter Q. Add the following: {{steps.metacritic_score.$return_value.album}} {{steps.metacritic_score.$return_value.artist}}. Also, make sure you add the "Type" optional parameter as well and put "album" in the field.

Right! Now we have a good album with an artist name, album name as well as a Spotify URL and a nice image of the album. What we want to do now is have this delivered to us every morning. There are so many ways to do this. You can send it by email or you can add it to Trello/Slack/Asana/Monday/Twitter. I chose to have it delivered in my morning newsletter. I like to read that while having my morning coffee so that is perfect.

To do this I use an app called Mailbrew. It can create custom newsletters for you from Twitter data, Reddit subreddits, RSS feeds, and much more. I find it handy for setting up summaries of subreddits so that I don't have to refresh Reddit all day. If you want to try Mailbrew and want to support this blog then feel free to use my referral code to sign up. I would really appreciate it! Now, of course you don't have to use Mailbrew for this. This example will assume that we are using Mailbrew or something similar.

This is unfortunately where we run into an issue with Pipedream. Since we need to output the data as an RSS feed to use Mailbrew then we need to turn the album list into RSS. As far as I know you cannot do that with Pipedream since there is no way to create an "API endpoint" of sorts. Luckily, I got just the thing for this: Huginn!

The first thing you'll need is a Webhook agent to accept incoming albums from Pipedream. Create one and then create another step in Pipedream to post the albums to Huginn. It will be another Node.js step an will look something like this:

Once you've set this up you have completed the Pipedream part.

Of course, we don't want all the albums at once. We want one album every day. So use a Delay agent in Huginn. Set "max_emitted_events" to 1 and it will emit a single event every time it runs.

Finally, add an output agent to add an RSS feed. The full Huginn scenario is available here: https://gist.github.com/mskog/4504fbc4e52df04f2221103bb8306c88

Now you can add this RSS feed to Mailbrew. It looks like this for me:

There you! A lot of effort to save 1 minute every morning. But it's cool is it not? This is what automation is for me! You learn something and you get a little treat every morning in your inbox. Of course this is just the beginning.

Now there are ways to improve this of course. Here are a couple of ideas:

• Make sure it doesn't throw an error if the Album is not available on Spotify.
• Use the GraphQL API to extract genres for the albums and then only select the genres you like.
• Count the number of songs in the album (the data is available in the API response from Spotify) and then don't include albums with only one song
• Add a deduplicate agent in Huginn to remove duplicate albums. Just in case something goes wrong and Pipedream sends the same album twice

So this was just a single example of what you can do with automation tools. Since then I've also done something similar with Movies and TV Shows. I get a newsletter every week with all this so that I don't have to check Metacritic. I find it very relaxing to receive emails instead and read them on my terms.