DEV Community: Laura Vuorenoja

My First Time at GopherCon

Laura Vuorenoja — Wed, 30 Aug 2023 08:20:23 +0000

At the beginning of this year, I set myself a target to speak at a Go programming language conference. There were several reasons to do so. Go has been one of my favorite tools for years, and I have longed for an excuse to join a Go event. Giving a speech was perfect for that purpose. Plus, I have
multiple excellent topics to share with the community as we do open-source development in our project, and I can share our learnings along with our code more freely. Furthermore, I want to do my part in having more diverse speakers at tech conferences.

As Go released 1.20, it inspired me to experiment with the new feature to gather coverage data for binaries built with Go tooling. I refactored our application testing pipelines and thought this would be an excellent topic to share with the community. I was lucky to get my speech accepted at GopherCon UK in London, so it was finally time to join my first Go conference.

The Brewery hosted the event. Surprisingly for London, weather was excellent during the whole conference.

The conference was held in the Brewery, a relaxed event venue in London City. The word on the conference halls was that the number of event sponsors had decreased from the previous year, and therefore, it had been challenging to organize the event. Luckily, the organizers were able to pull things still together.

Apart from the recession, the times are now interesting for Gophers. Many good things are happening in the Go world. As Cameron Balahan pointed out in his talk "State of the Go Nation," Go is more popular than ever. More and more programmers have been adding Go to their tool pack in recent years, pushing language developers to add new and better features. Moreover, Go is not only a programming language anymore; it is a whole ecosystem with support for many kinds of tooling. Newcomers have a way easier job to start with development than, for example, I had seven years ago. Balahan stated that improving the onboarding of new developers is still one of the Go team's top priorities. He mentioned that they are working on the libraries, documentation, and error messages to help newcomers and all Go developers be more productive.

Cameron Balahan is the product lead for Go.

Automated Testing and Test Coverage

The topic of my talk was "Boosting Test Coverage for Microservices." I described in the presentation how vital automated testing has become for our small team. Automated testing is usually the part you skip when time is running out, but I tried to convince the audience that this might not be the best approach – missing tests may bite you back in the end.

On the stage. Photo by Tapan Avasthi

Furthermore, I discussed test coverage in the presentation along with how one can measure test coverage for unit tests and now even - with the Go new tooling – for application tests, i.e., tests you run with the compiled application binary instead of unit testing tooling.

The audience received the talk well, and I got many interesting questions. People are struggling with similar issues when it comes to testing. It is tough to decide which functionality to simulate in the CI pipeline. Also, we discussed problems when moving on to automated testing with a legacy code base. The Go's new coverage feature was unknown to most people, and some were eager to try it out instantly after my session.

All participants were given adorable Gopher mascots.

Unfortunately, when you are a speaker at a conference, you cannot concentrate fully on the conference program because one needs to prepare for the talk. However, I was lucky enough to join some other sessions as well. There were mainly three themes that I gained valuable insights from.

Logging and tracing

Generating better logs and traces for applications seems to be a hot topic – and no wonder why. Services with high loads can generate countless amounts of data, and for the developers to use the logs for fixing issues efficiently, they must be able to filter and search them. The ability to debug each request separately is essential.

Jonathan Amsterdam from Google gave an inspiring presentation on the slog package, the newest addition to the standard library regarding logging. Go's default logging capabilities have always lacked features. The missing log levels have been the greatest pain point in my daily developer life. More importantly, the ability to send structured data to analysis tools is crucial for production systems. Until now, teams have had to use different kinds of 3rd party libraries for this to happen.

Now, the slog package fixes these shortcomings, with the ability to handle and merge the data from existing structured logging tools. The presentation revealed how the team refined the requirements for the new package together with the developer community. Also, it was fascinating to hear which kind of memory management tricks the team used, as the performance requirements for logging are demanding.

Another exciting presentation handled also the capability of solving problems quickly, but instead of logs, the emphasis was on tracing. Tracing provides a more detailed view of the program's data flow than logs and is especially useful when detecting performance bottlenecks. Konstantin Ostrovsky described how their team is using OpenTelemetry to add traceability to incoming requests. Using this approach, they do not need other logs in their codebase (excluding errors).

Konstantin Ostrovsky presenting OpenTelemetry usage with Go.

OpenTelemetry tracing uses the concept of spans in the code. One can utilize the spans to store the request data parameters and call relationships. Different analysis tools can then visualize this data for a single request. According to Konstantin, these visualizations help developers solve problems faster than searching and filtering ordinary logs. However, in the presentation Q&A, he reminded us that one should use the spans sparingly for performance reasons.

Service Weaver

Service Weaver is an open-source project that another Google developer, Robert Grandl, presented at the conference. The tool allows one to develop a system as a monolith, as a single binary, but the underlying framework splits the components into microservices in deployment time. Therefore, development is easier when you do not have to worry about setting up the full microservices architecture on your local machine. In addition, the deployment might be more straightforward when you can work at a higher level.

I participated in a workshop that allowed participants to try the Service Weaver hands-on. The target was to build a full-blown web application with a web UI and a backend service from which the UI could request data. Other sections described testing the weaver components, routing from one service to another, and even calling external 3rd party services.

The workshop suited me well; I could learn more than just listening to a presentation. Furthermore, the topic interested me, and I will dig into it more in the coming days to better understand which kind of projects would benefit from this kind of development model the most. The workshop organizer promises that Google will not stop investing in the product. They are searching for collaborators to get more feedback to develop the product further.

UI development with Go

Another topic that got my interest was a discussion group for UI development with Go. Andrew Williams hosted this discussion and presented a project called Fyne that allows Gophers to write applications with graphical user interfaces for several platforms. UI development is not my favorite thing to spend my time on; therefore, I am always curious to find better, more fun ways to implement the mandatory user-clickable parts. Using Go would undoubtedly click the fun box. So, I added another technology experiment to my TODO list.

In addition to these three themes, one session that handled JWT security was also memorable. Patrycja Wegrzynowicz hacked the audience live with the help of a small sample application she had built for this purpose. It demonstrated which kind of vulnerabilities we Go developers might have in our JWT implementations.

Patrycja hacking the audience with JWTs.

The presentation was informative and entertaining with the live hacking, and the audience understood the problems well due to the hands-on examples. The session proved that no well-known battle-tested documentation exists on handling the JWTs. We have (too) many different libraries with different qualities, and it is easy to make mistakes with the token generation and validation. No wonder the audience asked for a book on the subject from Patrycja – we need better resources for a topic as important as this.

See you in Florence

Overall, the event was well-organized, had a great atmosphere, and was fun to visit. Meeting fellow Gophers, especially the Women Who Go members, was a unique treat. Let's set up our chapter in Finland soon. (If you are a Finland-based woman who writes Go code, please reach out!) I also got to spend some free time in London and share the World Cup final atmosphere with the English supporters cheering for their team.

Public viewing event for the World Cup final.

Bye til the next event; I hope we meet in Florence in November! In the meantime, check out videos of the GopherCon UK 2023 sessions once they are published - I will do the same for the ones I missed live!

Listing Organisation Contributors

Laura Vuorenoja — Tue, 23 May 2023 13:53:30 +0000

What I built

GitHub Action for easily generating a markdown file that lists all organisation contributors. The file can be linked e.g. to GitHub organisation profile page.

Category Submission

Maintainer Must-Haves

App Link

https://github.com/lauravuo/fetch-contributors-action

Screenshots

Result markdown file

The contributors file can be linked to organisation profile

Scheduled action updates the data regularly

Description

The action utilizes GitHub API for fetching the contributor data for all organisation repositories. It then generates a markdown file for the data and stores the result file to the repository.

The organisation maintainers can integrate this action to organisation's .github-repository so that the contributors data can be linked or even listed directly in the organisation's profile page.

Link to Source Code

https://github.com/lauravuo/fetch-contributors-action

Permissive License

Apache 2.0

Background

I have wanted to learn more about GitHub API for a while. In one open-source project, there became a need to list the organization's contributors to determine who has been making code contributions. The contributors can differ from the organization members, so the organization member listing does not give this information directly. This need was a perfect excuse to learn about the GitHub API more.

How I built it

I built a Javascript GitHub Action and learned a lot about crafting GitHub Actions using Typescript, and how to test them. I also know now more about the GitHub API in general.

Getting Started with SSI Service Agent Development

Laura Vuorenoja — Wed, 08 Feb 2023 11:47:26 +0000

Self-sovereign identity sounds like an exciting concept for most, but starting with the development may seem overwhelming. I have gathered simple samples that get you full speed towards integrating the SSI functionality into your application.

The text was originally published in the Findy Agency project blog.

In the new SSI world, we craft digital services according to the self-sovereign identity model. We will have applications that issue credentials for the data they possess and applications that can verify these credentials. The central entity is the digital wallet owner that can hold these credentials and present them when needed.

"Ok, sounds great!" you may think. "I want to utilize credentials also in my web application. But where to start?"

Developing decentralized applications is tricky as it usually requires setting up multiple applications on your local computer or acquiring access to services set up by others. Using Findy Agency tackles this hurdle. It is an SSI solution that offers a complete set of tools for managing your digital wallet and agent via a user interface or an API.

Findy Agency provides tools for playing each role in the trust triangle: CLI and API clients have the complete tool pack, and the web wallet user can currently hold and prove credentials.

The Agency tooling provides you with a web wallet and a CLI tool that you can use to test your service's issuing and verifying features. You can easily setup the whole Findy Agency software to your local computer using Docker containers and a simulated ledger. Or, if you have an agency cloud installation available, you can utilize it for your service agent development without using any extra proxies or network tools.

"So, I have the agency up and running. What next?"

TL;DR Take a look at the examples found in the sample repository!

The sample repository provides simple yet comprehensive examples to start issuing and verifying using the CLI tool or with the agency API. The easiest path is to start with the CLI.

Run the CLI

"findy-agent-cli" is a command-line tool that provides all the required agent manipulation functionality. It provides means to quickly test out
the issuing and verifying before writing any code.

The sample script is a good starting point. It shows how to allocate an agent in the cloud and issue and verify credentials using a simple chatbot. You can run it by cloning the repository and following the instructions in the README.

CLI Script Initialization Phase

The sample script initialization phase allocates a new agent from the agency (1) and authenticates the CLI user (2-3). The authentication returns a JWT token exposed to the script environment so that further CLI calls can utilize it automatically.

For the agent to issue credentials, an applicable schema needs to exist. The schema describes the contents of a credential, i.e., which attributes the credential contains. The sample script creates a schema "foobar" with a single attribute "foo" (4-5).

There needs to be more than the mere schema for the issuing process; the agent needs also to create and publish its credential definition (6-7) attached to the created schema so that it can issue credentials and verifiers can verify the proof presentations against the published credential definition.

We assume that the holder operates a web wallet and has taken it into use. Note that you can play the holder role also with the CLI tool.

CLI Script Issuing Credential

The next task is to create a pairwise connection between the agent operated by the CLI user and the web wallet user. The pairwise connection is an encrypted pipe between the two agents that they can use to exchange data securely. The CLI script creates an invitation (1-2) and prints it out (3) as a QR code that the web wallet user can read (5).

Once the new connection ID is known, the CLI script starts a chatbot (4) for the new connection. The bot logic follows the rules for changing the bot states in the YAML file. Therefore, the bot handles the rest of the issuing process (6-7).

Once the issuer bot notices that credential issuing succeeded, it stops the bot (10-11), and the sample script moves on to verifying the same credential.

CLI Script Verifying Credential

Steps 1-6 proceed similarly to the issuing: first, the agents form a new pairwise connection. However, the process continues with a proof request sent by the verifier bot (7). The proof request
contains the attributes the bot wishes the holder to present. The web wallet user sees the requested data once they receive the message (8), and they can either accept or reject the request.

After the proof is accepted (9), the agency verifies it cryptographically. If the verification succeeds, the agency notifies the verifier bot with the proof values (10). It can reject the proof if the values are not acceptable by the business logic. The sample bot accepts all attribute values, so the verifying process is continued without extra validation (11). The bot exits when the proof is completed (12-13).

CLI as a Test Tool

Note that you can also utilize the CLI for testing. It is an excellent tool to simulate the functionality on the other end.

For instance, let’s say you are developing an issuer service. You can use the CLI tool to act as the holder client and to receive the credential. Or you can use the web wallet to hold the credential and create another client with the CLI tool to verify the issued data.

Feedback

The CLI sample script presented above demonstrates all the core features of verified data flow. It should make you well-equipped to play around with the CLI tool yourself!

Your tool pack will extend even more with our next blog posts. They will describe how to use the agency API programmatically and dive deep into crafting verified data supporting chatbot state machines.

Let us know if you have any feedback regarding
the Findy Agency functionality or documentation. You can reach us, for example creating an issue or starting a discussion in GitHub.

You can also reach me via these SoMe channels: GitHub, LinkedIn, Mastodon, and Twitter.

Good luck on your journey into the SSI world!

GitHub Actions Reporting My ❤️ Music

Laura Vuorenoja — Wed, 08 Dec 2021 13:50:27 +0000

This project is a PoC for utilizing GitHub Action workflows for static website generation.

The idea is to create static websites with low or zero coding. The project generates a website that lists my monthly top tracks from Spotify.

My Workflow

lauravuo / my-heart-music

PoC project for utilizing static website generation through GitHub actions workflow.

my-❤️-music

This project is a PoC for utilizing GitHub Action workflows for static website generation.

The idea is to create static websites with low or zero coding:

Fetch data from an API (in the PoC case, Spotify top tracks) and export the data to markdown
Append the generated markdown to a Hugo static website project
Build the Hugo site and deploy it to GitHub pages.

The result site displays my top tracks from Spotify with Hugo Terminal theme. The page is updated with a scheduled GitHub action, once per week.

A similar model could be easily used to generate different kind of sites, by variating the data sources and Hugo themes.

Action for Fetching Data

The inspiration to use the Spotify API for this project came to me when browsing Spotify related actions in GitHub Actions Marketplace. Spotify Box is an action that fetches the user's…

View on GitHub

The repository includes the static website skeletons. It utilizes different GitHub actions to generate the website.

Fetch data from an API (in the PoC case, Spotify top tracks) and export the data to markdown
Append the generated markdown to a Hugo static website project
Build the Hugo site and deploy it to GitHub pages.

The result site displays my top tracks from Spotify with Hugo Terminal theme. The page is updated with a scheduled GitHub action, once per week.

A similar model could be easily used to generate different kind of sites, by variating the data sources and Hugo themes.

Submission Category:

Wacky Wildcards

Yaml File or Link to Code

Action for Fetching Data

The inspiration to use the Spotify API for this project came to me when browsing Spotify related actions in GitHub Actions Marketplace. Spotify Box is an action that fetches the user's top tracks and updates those to a Gist. I took spotify-box as the basis and modified my fork according to my needs:

Instead of exporting the result to a Gist, I export the API result to json (tracks.json) and markdown (tracks.md).
I removed unneeded dependencies and added node_modules to version control, so that also external projects (other than the action repository) can easily take the action in use.

The use of the action is easy, the trickiest part is to acquire the needed Spotify keys and tokens. Luckily, spotify-box authors have described this process very detailed. The keys and tokens need to be added as secrets to the repository using the action, so that they are not exposed accidentally when the action is run.

After the API action is run, the result data files are saved and pushed to this repository. Markdown file is copied to target path hugo-site/content/index.md. It will be the source markdown for the site's landing page.

  updateTopTracks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: lauravuo/spotify-box@main
        env:
          SPOTIFY_CLIENT_ID: ${{ secrets.SPOTIFY_CLIENT_ID }}
          SPOTIFY_CLIENT_SECRET: ${{ secrets.SPOTIFY_CLIENT_SECRET }}
          SPOTIFY_REFRESH_TOKEN: ${{ secrets.SPOTIFY_REFRESH_TOKEN }}
      - run: |
          git config --global user.email "spotify-bot"
          git config --global user.name "spotify-bot"
          cp tracks.md hugo-site/content/index.md
          git add tracks.*
          git commit -a -m "Add latest tracks."
          git push

Deploying the Hugo Site

If the data fetch from previous step succeeds, the workflow continues by building the static website with Hugo. Hugo is setup using action peaceiris/actions-hugo. When the files are ready, the result is published to GitHub pages, using another GitHub action, peaceiris/actions-gh-pages.

 deploy:
    needs: updateTopTracks
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v2
        with:
          ref: 'main'
          submodules: recursive
          fetch-depth: 1

      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v2
        with:
          hugo-version: 'latest'
          extended: true

      - name: Build
        run: cd hugo-site && hugo --minify

      - name: Deploy
        uses: peaceiris/actions-gh-pages@v3
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./hugo-site/public

Additional Resources / Info

Used tools:

Hugo static site generator
Hugo Terminal theme
spotify-box GitHub action
Hugo setup Github action
Github pages deploy action

MacGyvering the OS X Menu Bar

Laura Vuorenoja — Sun, 06 Dec 2020 09:40:41 +0000

What I love about today's desktop tools such as Slack, Telegram, etc. is that they enable me to execute tasks in the context I am already in without opening yet another program. Besides, I can run my apps through them and there is no need to build a separate graphical or command-line user interface. However, when I am the only one in need of the tool, building a slackbot or similar app that can be accessed from the web is quite an overkill.

Fortunately, a while back a colleague of mine from work reminded me that there exists a handy framework for adding custom functionality to the OS X menu bar, BitBar. BitBar works by running scripts or programs and presenting their output in the menu bar. So actually one can add a self-written program to the menu bar without the need to write any Objective-C code or knowledge for OS X programming.

I thought that this kind of custom menu bar button would be quite handy to execute some of my chores, like refreshing the content of my Christmas radio I coded recently. Wouldn't it be just cool to trigger my Christmas radio update flow from the menu?

So I decided to try out BitBar. Here are the steps in case you would like to try something similar.

BitBar installation via brew cask

Homebrew is an OS X package manager and its cask extension is intended specifically for GUI application management.

Open new Terminal window and run:


brew install --cask bitbar

Output should look like something similar to this:


Updating Homebrew...
...
==> Downloading https://github.com/matryer/bitbar/releases/download/v1.9.2/BitBar-v1.9.2.zip
==> Downloading from https://github-production-release-asset-2e65be.s3.amazonaws.com/14376285/807c40
######################################################################## 100.0%
==> Installing Cask bitbar
==> Moving App 'BitBar.app' to '/Applications/BitBar.app'.

Create a folder for your BitBar plugins. I decided to create the folder in my Christmas radio repository kaneli/bitbar.
Open Finder and launch application from Applications folder
Answer "Open" to the confirmation dialog
Browse to the plugin folder created in step 2 and tap "Use as Plugins Directory"
Create the script for the menu bar

Create a new script file in the plugins folder. The script file name format should be defined as the following: {name}.{time}.{ext} The {time}part in the file name defines the menu item refresh interval.

Why we need a refresh interval? For example, if you would display in the item a clock with second precision, you would want to define the time interval as 1s. Bitbar would call your script once per second, and the clock in the menu bar would update correctly.

In my case, however, the menu content will be unchanged so I defined a long refresh interval and named my script askaneli.9999d.sh according to the app name.

After creating the script it needs to be given execution rights:
```
chmod a+x kaneli.9999d.sh
```
Write the menu bar script

The menu bar script should echo the UI controls definitions to the standard output. Bitbar parses the output and constructs the menu based on that.

If I would like to implement the clock example from the previous step, the script would be really simple:
```
  #!/bin/bash
  date
```
It would just output the current time.

However, my goal was to have a button in the menu bar that would show a menu with two items when clicking it. The first menu item would open my Christmas radio playlist in Spotify and the second one would fetch and update new songs to it by using the kaneli app.
```
#!/bin/bash

echo :christmas_tree:
echo "---"
echo "Open radio | href=https://open.spotify.com/playlist/5x5mdsVit4ngNyvglqkO8f"
echo "Update kaneli | bash=~/work/github.com/lauravuo/kaneli/run.sh terminal=false"
```
The first row defines the item that is shown in the menu bar. In this case, I use a Christmas tree emoji. The second row defines a separator line between the bar button and the menu items.

Options for the menu items are defined after the pipe | character. The first menu item has the text Open radio and a href option with which you can define a hyperlink that is opened when clicking the item.

The second item has the playlist update functionality. It will call another script in my file system that will call kaneli with the needed parameters. The option terminal can be used to define if a Terminal window is opened when the command is run.
Try it!

In the BitBar menu, there is Preferences/Refresh all functionality that will reload your menu script if you make any changes.

That's it. Try it out!

Check BitBar documentation for more instructions. There is also a bunch of plugins written by the community that you can just download and use.

Cover photo by Hunter Haley on Unsplash

DIY Christmas Radio

Laura Vuorenoja — Sun, 29 Nov 2020 17:32:30 +0000

Many people who are starting their programming journey often wonder where to get the ideas for their side projects. My advice is that work with a topic you are passionate about. For example, if you love cooking, make yourself a cookbook service or perhaps a digital egg timer. Or if you like to wander in nature, code an app that tracks your routes or helps you to identify which bird is singing. You will have an extra motivational boost when solving the problem will benefit you also otherwise.

"The thing" for me has always been music, so most of my side projects have been related to music in a way or another. One of my favorite apps is Spotify and luckily for me, they have published excellent APIs that one can utilize versatilely in their projects.

So it is no surprise that my latest project is also built on Spotify API. As Christmas is getting closer, my daily listening queue is filled with Christmas tunes. However, I tend to listen to the same songs every Christmas. This year I decided to find some new favorites.

We have an excellent internet radio for Christmas songs in Finland called Jouluradio. The only problem when listening to Jouluradio is if you like some new song you hear, saving it for later listening is quite cumbersome. You need to find the track information from the Jouluradio web page and manually search the track from Spotify.

Fortunately, Jouluradio publishes its playlist of the last 20 songs on the service web page. I decided to make "a radio app" that grabs this information. If the tracks are found in Spotify, they are added to my personal Christmas radio playlist and when playing it I can easily save the ones I like for later listening.

My radio app, named Kaneli (cinnamon in Finnish) is a command-line program written in Golang. The trickiest part of the project was to implement the user OAuth 2.0 authentication to Spotify that I wrote about in my last week's post. The user authentication and authorization part is required for the app to have permission to add tracks to the user-owned Spotify playlist.

Otherwise, the program is mainly about fetching and posting JSON to various endpoints.

API providers and app interaction flow

When the songs' information is acquired from Jouluradio, the app searches from Spotify if the track is found there. The Spotify search API endpoint provides handy tools for this.


...

for _, track := range response.PageLastPlayed.Content.RecentlyPlayed.Songs {
  query := url.QueryEscape(fmt.Sprintf("artist:%s track:%s", track.Artist, track.Song))
  // search for song by artist and title
  trackResponse, trackErr := doGetRequest(fmt.Sprintf("https://api.spotify.com/v1/search?type=track&q=%s", query), bearerHeader)
  if trackErr != nil {
    fmt.Printf("Unable to fetch track data %s\n", err.Error())
    continue
  }

  trackData := SpotifyResponse{}
  err = json.Unmarshal(trackResponse, &trackData)
  if err != nil {
    fmt.Printf("Unable to parse track data %s\n", err.Error())
    continue
  }

  // just pick the first found track
  if len(trackData.Tracks.Items) > 0 {
    item := trackData.Tracks.Items[0]
    fmt.Printf("Add track %s: %s\n", track.Artist, track.Song)
    songIds = append(songIds, item.URI)
    removeIds = append(removeIds, &SpotifyRemoveItem{URI: item.URI})
  }
}

...

The search query is done for each track using the artist and song title information. If the search returns results, the first found result id is saved for later use.

When the looping is done, it's time to add the results to the user's playlist. For this Spotify provides a playlist API endpoint.

In this phase, I had one problem though: how to prevent the program from adding duplicate tracks. First I tried to find an API that would allow me to search if the playlist contains the specific track already. But luckily, there didn't exist this kind of API. Instead, I figured out a more efficient way of avoiding the duplicates: before adding the items, the program would make a delete request for the tracks that are about to be added.


...

apiPath := fmt.Sprintf("https://api.spotify.com/v1/playlists/%s/tracks", playlistID)

// first delete all tracks with similar id to avoid duplicates
_, err = doJSONRequest(apiPath, http.MethodDelete, &SpotifyPlaylistDelete{Tracks: removeIds}, bearerHeader)
if err != nil {
  return err
}

// then add all tracks to the start of the list
_, err = doJSONRequest(apiPath, http.MethodPost, &SpotifyPlaylistModify{URIs: songIds, Position: 0}, bearerHeader)

...

Both the delete and the add request are done by utilizing batches, so it is more efficient than removing or adding the tracks one-by-one.

The codes can be found in GitHub if you want to take a closer look. To run the app, you need to register your application in the Spotify developer dashboard and check the further instructions in the repository README.

The current version of the app loops through a couple of different radios dedicated to specific genres and adds the tracks to the user-defined playlist. So far I have been running the program manually at random times but I guess it would be possible to create a cron job for it to run e.g. once per hour. That is if one would like a really extensive radio list 😄

You can find my Christmas radio on Spotify.

Cover photo by Markus Spiske on Unsplash

How to OAuth from the Command Line

Laura Vuorenoja — Sun, 22 Nov 2020 10:54:44 +0000

This week I came up with an idea about a personal Christmas radio. I am an eager listener of Christmas tunes and every year I try to find some new favorite songs. I decided to code me a little helper that would ease the discovery process and automatically add new Christmas songs to my chosen Spotify playlist.

However, I encountered an authentication-related problem when implementing the functionality. Accessing the Spotify user APIs (such as modifying the playlists on behalf of the user) requires permission both from Spotify and the user. The user's permission is acquired through browser interaction (user logs in to Spotify and authorizes the app in the web UI), but my app was designed to work from the command line. So the problem was how to make the command line program interact with the browser flow?

The Spotify API authentication is implemented according to the popular OAuth 2.0 specification. I decided to use the authorization code flow that would suit best for my purposes. The flow has two parts: first, the client application (my radio app) directs the user to an authorization server that handles the user authentication. Then the authorization server directs the request back to the client application (to a predefined redirect URI). With this redirect is delivered a code that the client application can use for requesting the actual API access token from another endpoint.

Although this protocol may sound a bit complex at first, it provides important security benefits: the user's credentials are never shared with the client application. And on the other hand, the final access token is delivered directly to the client application. This way it's not being exposed to the browser or the user and actually, only the client application can receive the token.

So the solution was to use a temporary localhost webserver. The server lives only as long as the redirect endpoint is called and the authorization code is received. The following figure describes the steps:

The user launches the app from the command line
The client program starts the temporary server
The client program launches the browser to the API authentication page.
The user authenticates in the browser and authorizes the client application to access the API on her behalf.
The authorization server redirects the request to the predefined redirect URL (localhost).
The client program parses the redirect request and receives the authorization code.
The client program exchanges the authorization code to the API access token calling the authorization server endpoint.
The client program receives the API access token and can make API requests on behalf of the user.

I used golang to implement the app and the sample code is attached here:



package main

import (
    "context"
    "encoding/base64"
    "encoding/json"
    "fmt"
    "log"
    "math/rand"
    "net/http"
    "net/url"
    "os"

    "github.com/pkg/browser"
)

type AuthResponse struct {
    AccessToken string `json:"access_token"`
}

func fetchUserToken() string {
    const (
        redirectURL     = "http://localhost:4321"
        spotifyLoginURL = "https://accounts.spotify.com/authorize?client_id=%s&response_type=code&redirect_uri=%s&scope=%s&state=%s"
    )

    var (
        clientID     = os.Getenv("SPOTIFY_CLIENT_ID")
        clientSecret = os.Getenv("SPOTIFY_CLIENT_SECRET")
        authHeader   = fmt.Sprintf("Basic %s", base64.StdEncoding.EncodeToString([]byte(clientID+":"+clientSecret)))
    )

    if clientID == "" && clientSecret == "" {
        panic(fmt.Errorf("spotify client ID and secret missing"))
    }

    // authorization code - received in callback
    code := ""
    // local state parameter for cross-site request forgery prevention
    state := fmt.Sprint(rand.Int())
    // scope of the access: we want to modify user's playlists
    scope := "playlist-modify-public&playlist-modify-private"
    // loginURL
    path := fmt.Sprintf(spotifyLoginURL, clientID, redirectURL, scope, state)

    // channel for signaling that server shutdown can be done
    messages := make(chan bool)

    // callback handler, redirect from authentication is handled here
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        // check that the state parameter matches
        if s, ok := r.URL.Query()["state"]; ok && s[0] == state {
            // code is received as query parameter
            if codes, ok := r.URL.Query()["code"]; ok && len(codes) == 1 {
                // save code and signal shutdown
                code = codes[0]
                messages <- true
            }
        }
        // redirect user's browser to spotify home page
        http.Redirect(w, r, "https://www.spotify.com/", http.StatusSeeOther)
    })

    // open user's browser to login page
    if err := browser.OpenURL(path); err != nil {
        panic(fmt.Errorf("failed to open browser for authentication %s", err.Error()))
    }

    server := &http.Server{Addr: ":4321"}
    // go routine for shutting down the server
    go func() {
        okToClose := <-messages
        if okToClose {
            if err := server.Shutdown(context.Background()); err != nil {
                log.Println("Failed to shutdown server", err)
            }
        }
    }()
    // start listening for callback - we don't continue until server is shut down
    log.Println(server.ListenAndServe())

    // authentication complete - fetch the access token
    params := url.Values{}
    params.Add("grant_type", "authorization_code")
    params.Add("code", code)
    params.Add("redirect_uri", redirectURL)
    data, err := doPostRequest(
        "https://accounts.spotify.com/api/token",
        params,
        authHeader,
    )
    if err == nil {
        response := AuthResponse{}
        if err = json.Unmarshal(data, &response); err == nil {
            // happy end: token parsed successfully
            return response.AccessToken
        }
    }
    panic(fmt.Errorf("unable to acquire Spotify user token"))
}

Go provides quite nice tools for implementing this kind of concurrency handling that is needed here: the localhost server is shut down using goroutines and channels. I encourage you to check them out if Go is something new for you.

So, now I have the access token. Now I need just to make the functionality for adding the songs 🙂

P.S. If you want to mess around with the Spotify API, remember first to register your application to get the client id and secret.

Photo by Steve Halama on Unsplash

Hacking the Orb

Laura Vuorenoja — Sun, 15 Nov 2020 20:03:57 +0000

I remember times when setting up CI meant long hours of server setup, Jenkins studies, XML configuration pain, and Java memory issues. No wonder continuous testing and integration was not the first thing that came to mind when starting a new project.

Luckily, nowadays things are different. Services like GitHub Actions, CircleCI or Travis (just to name a few) are cloud services that handle the CI flows swiftly for you. Pipeline configurations can be stored neatly next to your code in the version control repository and the changes to them are picked up automatically whenever new commits are introduced. Furthermore, containerization technologies have simplified setting up external dependencies, so even complex end-2-end test environments can be configured only with a couple of lines of YAML.

And the best part is that these services are usually free for open source projects.

However, one thing that has puzzled me over the years when using the aforementioned services is how to avoid DRY a.k.a. Don't Repeat Yourself principle. Many projects have similar CI configurations especially if they are related or implemented using the same technology. In the past, I have found myself copy-pasting configurations between projects or repeating similar steps over and over again and this is something I would rather always avoid.

Let's take an example from the world of Node.js. Unit testing CI process for a node project consists typically of the following steps:

Select the base OS and OS version for the job
Select (and install) Node.js version
Clone the project sources
(If a cache is found) load cached dependencies
Install dependencies
Run tests
Save dependencies to cache

Steps 1-2 may be combined if you use a system that allows the jobs to be run in a container: the base container can have all the global bells and whistles preinstalled that your project needs. But the rest of the steps you usually cannot avoid defining and actually only one, step number 6, interests me as the application or service developer. After figuring out and defining this process for the first time, I would like to be able to reuse the configuration in multiple projects.

Fortunately, the CI services are constantly introducing new features and most of them have enabled the sharing of common modules in a way or another.

Especially before GitHub Actions was made available for all GitHub users, my standard choice of CI was CircleCI. During last Hacktoberfest I happened to notice that CircleCI had an extra challenge for crafting orbs. I got intrigued and it turned out that the orbs were just the thing I had been missing a few years back: "A reusable package of YAML configuration that condenses repeated pieces of config into a single line of code."

So I decided to give the orbs a go and replaced the unit test job in one of my personal Node.js projects with a job from an orb. The CI configuration was trimmed exactly as I wanted: I got rid of the lines related to cloning, dependencies installation, and cache handling, and all that was left was the test command. The CI configuration was trimmed by 15 lines and everything worked as before the change.

Example of orb usage

In this case, the common functionality was authored by CircleCI. Node orb that I utilized in my personal project above provides Node.js related basic workflows related to dependency management and testing. But how about creating an own orb? The Hacktoberfest challenge was just about that, i.e. providing some meaningful CI functionality as an orb module that could be used by multiple repositories.

As it happened, I had just come across an interesting documentation tool, alexjs, that lints documentation files for insensitive and inconsiderate writing.

Alex helps to find "gender favouring, polarising, race related, religion inconsiderate, or other unequal phrasing". I thought that Alex would be a good addition to many of my projects, especially as I am a non-native English speaker and take gladly all the help I can get when authoring the project documentation. Also, the use of alex requires the npm toolchain as it is implemented with Node.js. So the additional motivation to implement an orb for alex was to use it in projects based on other technologies than Node.js.

CircleCI hosts a registry for all available orbs so before you can use your self-authored orb in a CI workflow, you need to publish it to the registry. The easiest way to launch your orb development is to install circleci cli and use an orb development kit that sets up a project template for you to get started with the development. Make sure to configure circleci cli first if you try to use the development kit at home.

The orb project template may seem overwhelming at first.

In the source folder there are placeholders for the following sections:

concept	description
commands	steps and their parameters and mapping of those to shell scripts
examples	use-case examples, displayed in registry documentation
executors	environment in which the steps of a job will be run
jobs	full jobs definitions that may combine steps even from other orbs
scripts	shell scripts that implement the actual command functionality
tests	test scripts for the shell scripts

For me it took a while to get my head around these concepts in order to achieve the end result I wanted: provide a preconfigured job that would do the code checkout and linting with alex. The target was that the orb user could just import the orb and call the job in the workflow just as with Node.js testing example before.

The other thing that was a bit challenging to get working at first was the orb CI pipeline. The template project is preconfigured to use CircleCI for orb linting and testing. It even packs and publishes the development version of the orb to the registry and executes the integration tests with an actually deployed version of the orb before publishing the final production version. (I'd say nicely designed flow 😊)

CI execution log after successful PR merge and deployment

However, with some documentation browsing, detective work and just trying out things, I finally got the things working as I had planned and as a result I was able to see my orb in the registry. One key thing was to learn how to setup the CircleCI credentials to the organization context so that CI was able to publish the orb on my behalf successfully.

Using the orb is now straightforward:

version: 2.1

orbs:
  alexjs-orb: lauravuo/alexjs-orb@0.0.1

workflows:
  lint:
    jobs:
      alexjs-orb/lint

And if all goes well, the job succeeds and logs can be observed from the CI output:

...
CHANGELOG.md: no issues found
README.md: no issues found
src/README.md: no issues found
src/commands/README.md: no issues found
src/examples/README.md: no issues found
src/executors/README.md: no issues found
src/jobs/README.md: no issues found
src/scripts/README.md: no issues found
src/tests/README.md: no issues found
src/tests/test.md: no issues found
CircleCI received exit code 0

The sources for my orb can be found in GitHub in case you want to take a closer look. In fact, currently all orbs published to the orb registry are open source. There is a definite need for private orbs as well, so it is interesting to see what happens in this front in the future.

What about you? Which CI system do you use and does it have similar support for shared functionality?

Cover image by Patrick McManaman on Unsplash

The Best Python Resources for Beginners

Laura Vuorenoja — Thu, 05 Nov 2020 19:07:14 +0000

My friend told me the other day that she would like to learn Python coding for data manipulation and visualization. Even though I know that there are tons of Python tutorials out there, I had no clue where to point her to. In particular, because I haven't ever learned Python properly, I might not be the best judge for evaluating Python tutorials 😄.

So I decided to phone-a-friend and ask wiser people for some recommendations. As my friend has no previous background in programming, the material needed to be for absolute beginners. Here's a list of the suggestions I got:

Resources in English

Resources in Finnish

Is one of your favourite tutorials or learning resources missing? Please suggest more!

Great sneak photo 🐍 by Tamara Gore on Unsplash

Hello dev.to, here’s one more dev too 👋

Laura Vuorenoja — Mon, 02 Nov 2020 16:23:29 +0000

I joined the dev.to community already in the summer. Although I have been thinking about writing my experiences and learnings in tech for a while, I haven’t had the chance to start my author's career at dev until now. I got the final motivational boost to get posts out from the inspirational members of the LevelUp Koodarit community.

If you are a woman or member of the gender minorities and seeking a Finnish coding community, I highly recommend you to join LUK 😊. It is an excellent place to share information, receive, and provide help.

This literacy journey starts by introducing me. My name is Laura, and I am living in Tampere (work partially in Helsinki), Finland. I started coding when I was in my twenties. At that time I began to study at the university and chose computer studies for my major even though I had not messed around with computers much before. I was not sure what I would want to be “when growing up” and IT seemed to be a suitable choice since it was easy to get in. And of course, we had this company called Nokia that employed all available coders in its years of success.

The start of my studies was bumpy: I just was not able to get my head round C++. It was the language they used for teaching all newbies back then, at the start of the millennium. Fortunately, I did not give up, and when Java and a teacher with a more pedagogic approach came along, I did not only get the hang of it, but fell in love with solving problems through code.

I have made my living in coding for 15 years now. It has been an exciting journey filled with a bunch of different technologies, projects, teams, and companies. I have written code for embedded systems as well as extensive web services, from C code to Javascript. And I do like being a rolling stone in this sense, learning new is one of the best things in this job.

Roughly 1,5 years ago I landed my dream job: I am currently researching emerging technologies in the largest bank and insurance company in Finland. One of my focus areas is self-sovereign identity and technologies enabling trust-over-IP. It is super exciting to design and build better ways to construct the digital society of the future.

I try to be technology agnostic, have a pragmatic approach, and choose the best tool for the job. Thus I do not have strict favorites when it comes to programming languages or platforms. That is also why I tend to acquire skills as I need them: depending on the project phase and requirements, I may be writing code related to infra scripts, database queries, API interfaces, or user interfaces. Recently I have been playing around mostly with web services built with Golang, AWS, and React.

Coding is part of my free time as well. I often get goofy ideas of little apps or services that ease my other hobbies or everyday life. Hopefully, I can present those projects in my future posts in more detail. Sometimes I also study new programming languages or other tech related topics. For example, this fall I have been trying to learn a little bit of Haskell.

Otherwise, my free time fills up with music and nature. I love to sing: together with my choir, with my friends in karaoke bars, and of course, alone in the sauna. I also love wandering in nature, and I try to do it as often as I can. My yearly highlight is a regular summer vacation trip to the Finnish Lapland to ride with my mountain bike.

I am glad if you had the time to read this far. I hope I can share more of my experiences and thoughts of my projects in the future with you. It is great to be part of this community!

P.S. The cover image is from my laptop. What do you think, is the laptop stickers a window to the developer's soul? 😀