Cloud Ops Auto Remediation: A Holiday Allegory

Auto remediations are tools that respond to events with automations able to fix, or remediate, the underlying condition. Building a demo that features auto remediation fix is hard, because generally modern infrastructure is resilient so keeping it in an error state is difficult. In order to feature an auto-remediation example, we’re going to get a little creative.

A Holiday Auto Remediation

I am writing this post in the first week of December, and let’s face it — holiday music is pretty inescapable this time of year. To describe the auto remediation that we will fix, let’s turn to one of the greatest fiction writers of our time:

As Mr. King has clearly presented — everyone has a least favorite holiday song, and the playing of this song can . During the holiday season, it is pretty common to use playlists generated by other users on Spotify. Wouldn’t it be great if we could auto remediate the playing of our disliked holiday songs automatically — without any intervention?

This is a great example of building auto remediation — an xRunBook in unSkript that identifies the song playing, and if there is an “error state” (a song on the blocklist), the RunBook will correct this issue without human intervention.

Auto Remediation in Spotify

In this post, we will build an auto remediation RunBook for your Spotify account.

1. We will create a Spotify app that connects with unSkript.
2. Curate a block list of songs.
3. We will build a xRunBook that checks to see if Spotify is playing.
4. If Spotify is playing a song on the block list, unSkript will automatically remediate the issue by skipping to the next track.
5. We can then place this xRunBook on a schedule to run every minute- ensuring that we’ll only hear a few seconds of our least favourite songs (at least when listening on *our* Spotify account). Note: This last step is only possible in the Open Source Docker build, or in a SAAS install of unSkript. This won’t work in the sandbox

Let’s get started:

Build a Spotify app

In order to interface with Spotify, we create an app at Spotify.

When you create an app, you’ll need to submit a redirect url. I used unskript.com in my example. Once you generate the app, you’ll get a clientId and clientSecret that you will need later to interface with your xRunBook. You can also add your account email as an authorized user of the application:

The name and description of my Spotify application gives away *my* reasoning for this app.

Building the xRunBook

NOTE: For full automation, install unSkript locally using Docker. If you just want to follow along the tutorial, you can use our free Sandbox to run the xRunBook manually. (If you are using the Sandbox, complete the tutorial.)

Now we are ready to create our xRunBook.

• In Sandbox, Click xRunBook, and then “+Create”, and connect your RunBook to the proxy you created in the tutorial.
• In Docker, there are instructions to create a new xRunBook in the README file.

xRunBooks are based on Jupyter Notebooks. Each action is an independent Python application. When the RunBook is executed, each Action is run in order. I have pasted the code for each action below. You’ll need to create a new Action for each code snippet. Do this by Clicking the + Add button at the top, and choosing “Action.”

Action 1: Install Spotipy

We’re going to use a Python library to interact with Spotify. In your first action:

!pip install spotipy - quiet

Action 2: Add our ClientIds.

In the OSS, and Sandbox, there is no built in Secret vault. so we’ll put them here. Make sure the redirect url matches what you placed in your Spotify application/

client_id = "<client ID from Spotify>"
client_secret="<secret from Spotify"
client_redirect="https://unskript.com"

Action 3: Connect to Redis

Note: not needed for Docker, and will not work for Sandbox, but this will work in unSkript SAAS:

import redis
redis = redis.Redis(host='<redis-host>', port=6379, db=0)
redis.set('foo', 'bar')
True
redis.get('foo')
b'bar'
print(redis)

Spotipy stores the authentication in a local cache — which is fine in the unSkript Docker instance.

The Sandbox does not have a local cache, nor a Redis instance, so this xRunBook cannot be run on a schedule as a result.

In our SAAS version, a Redis database can be attached to the xRunBook to ensure that the credentials are stored locally, and can be reused.

Action 4: Find SongIDs

Let’s do some searches in the Spotify database to extract the songIds we want to add to our blocklist. Here we are searching for songs sung by Mariah Carey:

import spotipy
from spotipy.oauth2 import SpotifyClientCredentials

scope = "user-library-read user-modify-playback-state user-read-playback-state"

sp = spotipy.Spotify(auth_manager=SpotifyClientCredentials(client_id=client_id,
                                               client_secret=client_secret))

results = sp.search(q='mariah carey', limit=5)
#print(results)
for idx, track in enumerate(results['tracks']['items']):
    print(idx, track['artists'][0]['name'], " ", track['name'], " " ,track['id'])

The results for this Action are:

0 Mariah Carey All I Want for Christmas Is You 0bYg9bo50gSsH3LtXe2SQn
1 Mariah Carey Fantasy 6xkryXuiZU360Lngd4sx13
2 Mariah Carey Christmas (Baby Please Come Home) 3PIDciSFdrQxSQSihim3hN
3 Mariah Carey We Belong Together 3LmvfNUQtglbTrydsdIqFU
4 Mariah Carey Fantasy (feat. O.D.B.) 2itAOPLerxnnc8KXHMqPWu

We want to (ok, I want to) block the song with ID: 0bYg9bo50gSsH3LtXe2SQn.

Action 5: Build a block list

This Action defines the array of songs we want to block.

songList = ["0bYg9bo50gSsH3LtXe2SQn", 
            "4iHNK0tOyZPYnBU7nGAgpQ",
            "0SorhWEyl6wkQ6vYAQt2D0"]

Action 6: Authenticate the user

This requests access for a user at Spotify. We need to read the playback state (is Spotify playing?), and we need permission to modify the state (skip the song!).

We have commented out the Redis cache_handler. If you are using the SAAS version of unSkript — remove the comment.

import spotipy
from spotipy.oauth2 import SpotifyOAuth
import json

scope = "user-library-read user-modify-playback-state user-read-playback-state user-read-recently-played"

spUser = spotipy.Spotify(auth_manager=SpotifyOAuth(client_id=client_id,
                                               client_secret=client_secret,
                                               redirect_uri=client_redirect,
                                              # cache_handler=spotipy.cache_handler.RedisCacheHandler(redis),
                                               scope=scope,
                                               open_browser=False))

Action 7: Skip the track

This step will be interactive. When this action is run, you’ll be asked to visit a url (that does a redirect) and then paste in the redirected url. This url has your authentication token in it. Once this is done, a token is stored locally that is valid for one hour. Every time the xRunBook is run, the token is re-authenticated for another hour.

#get current track
currentTrack = spUser.current_user_playing_track()

## only test the playback if there is currently a song playing
if currentTrack is not None:
    #
    track = currentTrack["item"]["uri"]
    print(track)
    #remove 'spotify:track:' from front of string to get the ID
    track = track[14:]
    print(track)

    # all i want for christmas is you. spotify:track:0bYg9bo50gSsH3LtXe2SQn
    songs_i_hate = songList

    for song in songs_i_hate:
        print("song", song)
        if track == song:
          print("ahhh save us")
          spUser.next_track()
          break
        else:
          print("its all good")
else:
    print("the music is off")

This completes the xRunBook creation. Save the RunBook (by closing it, and then selecting Save)

Scheduling the RunBook

In the SAAS (and Sandbox), it is possible to schedule your xRunbooks. By scheduling this xRunBook to run every minute — you can ensuyre that you’ll never have to listen to more than 59 seconds of the songs you dislike.

A Holiday Allegory

An Allegory is a metaphor that symbolizes an idea or message. Many stories told at Christmas time are allegories (like Dicken’s a Christmas Carol). In this post, we have used Spotify playlists as an allegory to a Cloud System in distress. By auto-skipping songs, our unSkript RunBook is automatically solving a situation without involving a human (potentially paging them out of a “long winder’s nap.”)

By building a “Skip the horrible track” auto remediation, we show the power of unSkript, and also potentially save the family Christmas by avoiding arguments over the Christmas playlist.

Interested in learning more about how unSkript can help you build internal auto remediation tooling for your team? Check out our free trial, star our GitHub repository of xRunBooks and Actions, or join our Slack Community!