DEV Community: Marc Dougherty

Handy Yaml Tricks!

Marc Dougherty — Thu, 09 Feb 2023 02:38:11 +0000

In the past few years, YAML (http://yaml.org) has become an essential part of software, particularly for infrastructure-as-code tools. Yaml at the heart of kubernetes configuration, kubernetes-inspired APIs like Google's config connector, and a number of workflow systems like Google Cloud Workflows and Github Actions.

In its simplest forms, Yaml is quite human-readable, but over time many of these configurations become more complex, and the documentation of these formats is not always as complete or searchable as we might like. Included below are some small tips that may help you make the most of your Yaml configurations.

IDE assistance with JSON Schemas!

Do you struggle to remember the names of fields in your Yaml objects? I sure do!

Most modern editors and IDEs support the Language Server Protocol, which powers the code completion, validation, and tooltip features. Combined with a Yaml Language Server, we can get rich completion for Yaml files!

Installation varies depending on the editor, but I found installation with VSCode to be pretty straightforward.

For many common yaml files, the correct schema can be inferred from the file name, and looked up automatically with SchemaStore. SchemaStore hosts a wide collection of JSON Schemas, which can be used to validate Yaml. SchemaStore is backed by the SchemaStore Github Repository, and contributions of additional JSON Schemas are welcome!

Schema detection can also be configured manually in your editor if auto-detection is inaccurate.

Setting up the Yaml Language Server takes a bit of work, but if you regularly work with complex Yaml objects, you'll be glad you took the time to set it up!

Better Multiline Strings with `>` and `|`

Sometimes, with workflow configuration like cloudbuild.yaml and Github Actions, we write long commandlines that are not very readable in an IDE.

The Yaml spec has what they call literal style (|) and folded style (>) to help with this.

Literal Style preserves newlines in the string, so it can be used to run multiple commands as a single step. For example, installing python dependencies and running the relevant tests can be done like this:

      - name: Run tests
        run: |
          pip install -r requirements.txt
          pytest -v .

Note: In some situations, this may mask failures in the earlier commands if the last command exits successfully.

Folding Style "condenses" whitespace in the string, replacing spaces and/or newlines with a single space as describe in the spec section on line folding. In short, it lets us insert newlines in a string to make it more readable.

Consider this very long line from a github actions workflow that calls the github API with curl:

     - name: Get current repo settings
       run: curl -o repository.json https://api.github.com/repos/${{ github.repository}} -H "Authorization: Bearer ${{ github.token }}"

Rewritten with Folding Style, it looks like this instead:

     - name: Get current repo settings
       run: >
          curl -o repository.json
          https://api.github.com/repos/${{ github.repository}}
          -H "Authorization: Bearer ${{ github.token }}"

For long lines, this can make them easier to read, and easier to edit and review.

These tips have helped me write more maintainable Yaml files, and I hope they help you, too!

Using the Github cli with Multiple Repos

Marc Dougherty — Thu, 24 Feb 2022 04:25:40 +0000

I have been using the Github CLI for a while now, and
I've found it to be really great when working in the context of a single
repository. But what happens when you're responsible for work in dozens of
repos? Keeping tabs on many repos can be done, thanks to the gh api
subcommand, and aliases!

gh api allows you to call any github api endpoint, and for my particular
needs, the search/issues endpoint is the most useful, as it searches both
issues and PRs.

There are two different ways to format responses, either with the --jq flag,
which uses jq syntax or --template
which uses Go's text/template syntax. I
found prototyping with the jq syntax to be nice, but the color and formatting
choices in the --template flag won me over.

Note that the --template args in the below examples are all the same, and they
produce output like this (but with some tasteful terminal colors, which do not
appear in markdown):

[  98] chore(deps): update dependency maven to v3.6 (15 days ago)
  https://github.com/GoogleCloudPlatform/cloud-run-samples/pull/98

More information about formatting output can be found in the gh formatting
help.

My issues/PRs that need followup

In my job, my team is responsible for the care and feeding of many different
public github repositories, which requires a lot of issue triage, and pr
followups. Switching to each of these contexts to run gh issues and
gh status is pretty tiresome, so I made a command that uses gh api to search
for open issues and PRs in any repo that are assigned to me:

gh api -XGET search/issues -f q='assignee:@me is:open  ' \
  --template '{{range .items -}}[{{.number | printf "%5.f" |autocolor "blue" }}] {{.title}} ({{ timeago .updated_at | autocolor "yellow+d" }}) {{printf "\n  " }}{{.html_url}} {{print "\n" -}}{{end}}'

This command is not much fun to type, so I've created an alias within the gh tool (not to be confused with shell aliases). However, using the gh alias set command would require escaping the quotation marks, I chose to edit the config file directly - it lives at ~/.config/gh/config.yaml on my linux machine. Adding the following block to the aliases list will create the alias:

    mine: |
        api -XGET search/issues -f q="assignee:@me is:open"
        --template '{{range .items -}}
        [{{.number | printf "%4.f" |autocolor "blue" }}] {{.title}} ({{timeago .updated_at | autocolor "yellow+d" }})
        {{print "\n  " }}{{.html_url}} {{print "\n" -}}
        {{end}}'"

(Note that the yaml heredoc syntax (|) is used to allow a multi-line string.
Ensure the indentation is the same for all lines, or there may be problems
parsing your config!)

Now I can see all my assigned issues and PRs by running gh mine!

Unclaimed team reviews

Tracking issues and PRs that are waiting on the team's review (that nobody has
claimed yet) can be done with a similar approach. Note that you'll need to update
the search query to use your own Github Organization and Team in the query below.

gh api -XGET search/issues -f q='team-review-requested:yourorg/yourteam is:open -review:approved no:assignee' \
--template '{{range .items -}}[{{.number | printf "%4.f" |autocolor "blue" }}] {{.title}} ({{timeago .updated_at | autocolor "yellow+d" }}) {{printf "\n  " }}{{.html_url}} {{print "\n" -}}{{end}}'

or ready for your config.yml:

    teamreviews: |
        api -XGET search/issues -f q="team-review-requested:yourorg/yourteam is:open -review:approved no:assignee"
        --template '{{range .items -}}
        [{{.number | printf "%4.f" |autocolor "blue" }}] {{.title}} ({{timeago .updated_at | autocolor "yellow+d" }})
        {{printf "\n  " }}{{.html_url}} {{print "\n" -}}
        {{end}}'"

Future directions

I would like to have a bit of extra information about these PRs, like whether or
not their required checks are passing, all in the same output. To achieve this,
I'd probably have to write a
gh-extension.
To avoid replicating existing gh functionality, there's a go-gh
library.

Do you use other github searches or gh tricks to keep tabs on your work? Tell
me about it in the comments!

Elgato Stream Deck with Linux

Marc Dougherty — Fri, 21 Jan 2022 22:42:31 +0000

With all the time I'm spending on video chat these days, I was looking for an
easier way to manage my video setup. I was intrigued by the Elgato Stream Deck,
with its small footprint, and fancy LCD screen buttons -- but I run linux, and
(as usual) that requires a bit more effort.

Fortunately, there is plenty of effort that's already been put in by others in
the community. For those not looking to build something custom, there's the
streamdeck ui which allows
the configuration of different types of built-in actions, all configurable
graphically.

While this is a great way to get started with the Stream Deck, I found myself
wanting more than the actions available with this tool. So I kept looking.

I soon found a blog post from Lorna
Mitchell, and the associated
streamdeck-tricks repo. Thanks
to their thorough explanation and ample references, I started tinkering with the
same underlying library they suggest
(http://github.com/magicmonkey/go-streamdeck).

I've since learned of similar library that interfaces with the Stream
Deck (https://github.com/dh1tw/streamdeck) - while it looks equally compelling,
I had already started my work when I found this library. If you're starting
fresh, this library appears to be more actively developed.

Mental Model

Before I get into the app, let me introduce the way I think about the stream
deck, as this impacts what I built, as well as how the components relate to each
other.

The largest level of organization is the Stream Deck itself. This layer handles
the physical communication with the device.

When it comes to creating Buttons on the device, I expect them to share some
resources - for example, Lorna mentions several buttons that all interact with
their audio mixer. I chose to group this type of complexity into what I called a
plugin. Specific plugins are discussed below.

The lowest level of organization is a button. Buttons have an appearance,
and generally take some action when pressed.

Plugins

Google Meet

I spend a lot of time in Google Meet, and I sometimes forget the keystrokes for
muting audio and video, so the first plugin I built has dedicated buttons for
these.

Prior to the streamdeck, I had used some keyboard shortcuts that called
xdotool, and the plugin basically just runs those. xdotool is a bit arcane,
but it can be very powerful:

It can find windows based on their title or window class: xdotool search --name "Meet - "
It can also send key presses to those windows without stealing your mouse and keyboard focus. Even if the window is on a different virtual desktop! This is done by combining the windowfocus and key commands of xdotool.
It can move your mouse and keyboard focus to the chosen window, if desired, using the windowactivate command instead of windowfocus.

So, to mute my audio in Google Meet, i can run this command:

xdotool search --name "Meet -" windowfocus key ctrl+d

The streamdeck plugin just runs these commands for me, at the push of a
dedicated button.

Lights

The next plugin I created was to control my video chat lighting, so I could turn
it off when not in use (to decrease eye strain), and adjust brightness as
needed. The lighting needs in my office change considerably depending on the
weather and time of day.

To achieve this, I invested in an Elgato Key Light Air. They are not the most
affordable light, but the build quality is excellent, and after using my
previous (cheap) option for a year, I deserved the upgrade :)

The Elgato Key lights are interesting, because they are operated by making calls
to an onboard api server in the light itself, over HTTP. This means that not
only was a golang client readily available, but i started thinking differently
about the streamdeck - as a way to not just run commands on my local machine,
but to interact with APIs and other connected devices directly!

The initial setup of the Elgato light was frustrating, as my dual-band wifi
network was not visible in the setup app, and required a fussy
workaround.

New challenge: managing button state

Now that I could control the light, I realized that i wanted the buttons to
reflect whether the light was connected, so I would not press them if the light
was, for example, unplugged.

I first tried this out with the go-streamdeck library's Decorator objects,
which allow you to apply visual effects over the normal button appearance. I
found that that this caused some corruption in the button appearance. In
hindsight, this was probably just me calling button update handlers too often,
from multiple goroutines.

But before I figured that out, I removed the use of Decorators, and created my
own ImageButton class, which implements the Button interface. It also has a
pre-set "disabled" appearance, which greys out the button, so i can tell at a
glance that my light is not available.

OBS: Open Broadcaster Software

With all my current video chat needs addressed, it was time to tackle something
new: OBS Studio. OBS Studio is popular with streamers,
who use it to customize and combine video and audio sources. In addition to
Recording and Streaming, OBS supports a Virtual Camera, which can be used
transparently by most video chat software.

I know OBS has a lot of capabilities, but my current desire is to have a
"intermission"-style image replace my camera feed, while I step away briefly to
grab a snack, or a drink, before a meeting gets started.

To allow streamdeck control of OBS, i'm using the
obs-websockets
plugin, and a go library for
obs. The only action I've
implemented so far is to switch Scenes. If you're a more advanced user of OBS,
you may want to use a library that is more actively maintained (like this
one), to get more recent features, like
the ability to start/stop the Virtual Camera.

The scene switching buttons use the same state-tracking built for the Keylight,
and grey out when obs is not connected. A goroutine watches the OBS connection
in the background, and will reconnect automatically.

Conclusions

So far, the streamdeck has improved a few routine tasks for me, and provided a
fun project. As I get more familiar with OBS, I expect I'll create more
elaborate configurations to switch between.

While I currently use the streamdeck plugged into my laptop, I also see
opportunity for a "headless" streamdeck, attached to something like a Raspberry
Pi Zero, which can serve as a physical interface to any connected API (e.g.
smart home lights).

Do you have a streamdeck? How do you use it? Leave a comment to let me know!

Elgato keylight wifi setup workaround

Marc Dougherty — Fri, 21 Jan 2022 22:28:42 +0000

I recently purchased an Elgato Keylight Air, and had some trouble with the
setup. This article documents the process I went through to get it set up.

The problem

The problem I had was during setup, when my wifi network did not appear in
Elgato's Android app (though i believe this problem also applies to other setup methods).

My wifi network is dual-band, meaning it is broadcasing on both the 2.4Ghz
spectrum (which the keylight supports), and the 5Ghz spectrum (which the
keylight does not support). It does this using the same network information on
both networks, and steers clients toward the best frequency for their needs.

It seems that Elgato's app is removing from the setup list any network that
has a 5Ghz component, so I could not select my wifi network from the dropdown in
the app, even though it also uses the 2.4Ghz spectrum.

The workaround

The workaround was a simple bait-and-switch.

First, I powered off my main wifi network.
Using my phone's hotspot function, I created a new wifi network using the
same network name and password as my main wifi network.
Using the Elgato app at this point, I was able to configure the keylight with
the network name and password, and it successfully connected to the phone
wifi hotspot.
Power off the keylight, disable the phone hotspot, and restart the main wifi
access point.

At this point, the keylight is able to join the main wifi network, because it is
configured with the correct network name and password.

This little workaround is rather annoying, and took me quite a while to figure
out, but now that it is configured on the network, the keylight is working well
for me!

Continuous Delivery of dev.to articles with Github Actions

Marc Dougherty — Tue, 29 Dec 2020 01:05:56 +0000

I'm a big proponent of Continuous Delivery: delivering value should be routine, fast, and easy. I've recently applied that mindset to my writing as well, thanks to Github Actions, and the dev.to API.

But Why?

My typical workflow for writing is to start brainstorming in a text file (usually with vim or vscode), and gradually shape that into a coherent piece of writing. I prefer to write in markdown, since the format is simple enough to do by hand, and is easy to render as html, or just paste into Google Docs.

The publishing part is usually where I'd get bogged down. I'd debate with myself what the right platform was, and then have to reformat to make sure links worked. What if I needed to make a correction? Repeat the process.
Often I'd just stop working on a piece, because the process was annoying. This felt exactly like a problem Continuous Delivery could solve.

A writer's CI/CD with Github Actions

My normal writing workflow starts with markdown and text files, so there was no change needed for me to keep my writing in a git repo. In fact, I had a mostly-empty repo already laying around, with some old writing already in it!

Creating the Github Action

I will admit up front that creating my own Github Action for publishing was unnecessary, and I only did it as a learning exercise. There was already an action to publish to dev.to, but I am not much good at javascript, and wanted to understand Actions better, so I built my own: muncus/devto-publish-action.

The Golang code to call the Dev.to API is fairly straightforward, and I spent more time with the Dockerfile and the Github Actions YAML files.

The binary basically takes 3 inputs: a directory full of markdown files, a dev.to api key, and a json file in which to store article ids (which allows us to re-publish the same file when there are changes).

These arguments need to be declared in the inputs section of action.yaml, and referenced in the args section.

As an extra bonus, I added a workflow to build and test the go code for the action as .github/workflows/build-go.yml. This workflow is taken from an Actions tutorial (but I forgot the source, sorry!).

Now that our action repo is created and published, it is time to use it in our writing repo, by creating another workflow.

Calling the Action from a Workflow

The full workflow for publishing an article has 3 steps:

Check out my private repo containing my posts
Publish the articles for dev.to
Update ids.json with newly published article IDs, so I can edit posts after initial publishing.

Setting up some Secrets

Before we can set up this workflow, we need a safe place to store two key pieces of secret information.
Github Secrets is perfect for this.

Github credentials to read my private repository
- I created a Personal Access Token, and named it MY_GH_PAT.
Dev.to credentials to post on my behalf.
- These are created in the dev.to account settings under "DEV API Keys".
- This secret is named DEVTO_API_TOKEN

With these secrets created, we can create a workflow that references them, without exposing our precious credentials! Let's look at each step individually, and the full example is below.

Step 1: Check out private repo

    - name: Check out repo
      uses: actions/checkout@v2
      with:
        token: "${{ secrets.MY_GH_PAT }}"

This uses the built-in action to check out a repo, which accepts a token input. Because I keep my writing in a private repo, we must specify an access token.

Step 2: Publish with our custom action

    - name: Publish to Dev.to
      uses: muncus/devto-publish-action@release/v1
      with:
        directory: "$GITHUB_WORKSPACE/dev.to/"
        api-key: "${{ secrets.DEVTO_API_TOKEN }}"
        state-file: "$GITHUB_WORKSPACE/ids.json"

This references my custom action, and provides the inputs it needs. Note that $GITHUB_WORKSPACE is a built-in environment variable that is the base directory of the repo we checked out in the previous step.

Step 3: Make a PR with local changes

When my publishing action runs, it updates a "state file" which maps filenames to dev.to article IDs. This means after a new article is added, there will be local changes. To preserve those changes, we'll need to create a Pull Request from those local changes.

    - name: Create PR to update state file
      uses: peter-evans/create-pull-request@v3
      with:
        title: "[CI] Update state file with published article ids"
        reviewers: ${{ github.actor }}
        token: "${{ secrets.MY_GH_PAT }}"
        base: master
        branch: "ci/state-file-update"

There are many options for the peter-evans/create-pull-request action, but lets run through the ones I'm using here:

title: entirely cosmetic, but this helps me approve PRs quickly when I know they're from my CI system.
reviewers: to clarify that whoever pushed the new article is responsible for approving the PR, they are listed as the reviewer.
token: this is the github access token created above. it uses my credentials in this private repo, so all these PRs look like they were made by me. Its not ideal for a shared repo, but fine for my needs.
base: the branch against which to make the PR. We check out master to publish, so that's what we're comparing to.
branch: using a stable PR branch name keeps merges simpler, and helps "stack up" multiple changes to the state file, in case I forget to approve a previous PR before publishing a new article.

Conclusions

In my experience, the hardest part of the Action development process was the trial-and-error needed to get the YAML "just right". There's still room for improvement in the Dockerfile, and I could use a pre-built image to save another 30 seconds on each article publication. But it works well enough to get us back to writing prose instead of code.

I hope this inspires you to use Github Actions in a non-traditional way!

Appendix: full workflow yaml

name: Publish
# Only trigger this workflow on master, and a feature branch specifically for testing the workflow.
on:
  push:
    branches: [ master, ci-action ]

jobs:
  build:
    name: Publish
    runs-on: ubuntu-latest
    steps:
    # First, check out the private repo.
    - name: Check out repo
      uses: actions/checkout@v2
      with:
        token: "${{ secrets.MY_GH_PAT }}"
    # Then publish articles.
    - name: Publish to Dev.to
      uses: muncus/devto-publish-action@release/v1
      with:
        directory: "$GITHUB_WORKSPACE/dev.to/"
        api-key: "${{ secrets.DEVTO_API_TOKEN }}"
        state-file: "$GITHUB_WORKSPACE/ids.json"
    # Last, update the state file by making a PR.
    - name: Create PR to update state file
      uses: peter-evans/create-pull-request@v3
      with:
        title: "[CI] Update state file with published article ids"
        reviewers: ${{ github.actor }}
        token: "${{ secrets.MY_GH_PAT }}"
        base: master
        branch: "ci/state-file-update"

My Top 5 Books for DevOps/SRE

Marc Dougherty — Sun, 29 Nov 2020 21:13:55 +0000

Disclaimer: The opinions expressed here are my own, and do not represent those of my employer.

I am the type of person that learns well from self-study like reading books, or watching instructional videos. I recognize that this method does not work for everyone, but if it works for you, I'd like to recommend a few books that have helped me grow in my career as a Site Reliability Engineer.

Note: these are not affiliate links, and I have not been compensated in any way to recommend these items.

1. Crucial Conversations

You may have heard the old Operations adage "when things are going well, everyone forgets you exist". The corrolary is that we're usually involved when other teams are having a Bad Day, and tempers can flare on both sides.

Crucial Conversations has helped me build the skills to better understand the needs of partner teams, and leadership, when they may not be communicating very clearly (because things are a 🗑🔥).

This book has helped me identify my own defensive reactions, and build habits to help avoid them. This leads to the perception that I am keeping a cooler head during outages, even if i'm still freaking out!

This is a book that I try to at least skim every year or two, because the lessons and practices that speak to me are different each time I read it.

2. Thanks for the Feedback

This book is all about asking for feedback, and really understanding it, (regardless of how poorly it might be delivered).

I'm not going to lie, this might be the hardest book i've ever read. The book illustrates difficult situations, and I often found myself identifying with the more antagonistic party.

Because of that, this book had a real effect on how I treat feedback, and I certainly feel better about receiving difficult feedback than before reading this book.

3. Accelerate: The science of lean software and devops

Finally, we get to the books with technical content!

As an SRE, I spend a lot of my time making the case for change, whether in business requirements, or in engineering time. This book provides the data to make these cases more easily, and point you at the right measurements to show the impact of the change.

If you're not familiar with this book, it is based on the last 4(?) years of DORA State of DevOps Report survey data, making it the largest dataset you could hope to find on the topic.

The book (and to some extent the site above) lays out a set of 24 Key Capabilities, and the relationships between them. The relationships between these capabilities are sometimes obvious (e.g. Continuous Delivery drives Software Delivery Performance), and sometimes more surprising (e.g. Continuous Delivery also drives Job Satisfaction).

My favorite parts are "intuitive" correlations that do not hold up in this large of a data set, e.g. having a separate QA team that owns tests does not correlate to fewer failed changes.

If you're looking to make change in your organization, this book is a must-have.

4. Site Reliability Workbook

The sequel to the first SRE Book, The SRE Workbook improves on the original by using more thorough non-Google examples, and tools and techniques available to those outside of Google. Many of the success stories come from Google customers, but since they are using industry-standard technologies, the examples are more relateable.

I especially appreciate that SLOs are called out as the first step toward reliability. Reliable systems are inherently more expensive, and high-availability systems often extremely so. Getting a shared understanding of reliability requirements between Engineering, Operations (or SRE), and Product (representing the Customer) records the expectations of all parties, and can make it clear that for some systems, 90% uptime (or lower) can be the Right Thing.

There are several chapters about how Google has structured SRE, and how it structures SRE Engagements. I'd caution readers to be skeptical here, as these decisions have consequences on how your SRE org is able to engage with partner teams and other parts of your organization, and what worked for Google may not work for you.

5. Everybody Writes

For most of us, our written communication has the largest audience. Unless you're doing a lot of public speaking, the emails and documents that we write reach many more people than our verbal communications.

The author of this book works in Marketing, but the first 3/4 of this book is full of lessons that anyone can benefit from. Perhaps most importantly, it teaches us that writing well is a learned skill, not an innate ability. The best way for us to improve our written communication is to do more of it!

I was surprised to see a discussion of "content limits", basically maximum sizes for specific types of content (e.g. a blog post should be at most 1500 words). This was a new concept for me, and these guidelines help me make content (from blog posts to emails) that are more accessible to the reader. (even if it does take me longer to write them!).

Do you have a favorite book that's helped you grow personally or professionally?
Tell me about it in the comments!

DEV Community: Marc Dougherty

Handy Yaml Tricks!

IDE assistance with JSON Schemas!

Better Multiline Strings with > and |

Using the Github cli with Multiple Repos

My issues/PRs that need followup

Unclaimed team reviews

Other useful searches

Future directions

Elgato Stream Deck with Linux

Mental Model

Plugins

Google Meet

Lights

New challenge: managing button state

OBS: Open Broadcaster Software

Conclusions

Elgato keylight wifi setup workaround

The problem

The workaround

Continuous Delivery of dev.to articles with Github Actions

But Why?

A writer's CI/CD with Github Actions

Creating the Github Action

Calling the Action from a Workflow

Setting up some Secrets

Step 1: Check out private repo

Step 2: Publish with our custom action

Step 3: Make a PR with local changes

Conclusions

Appendix: full workflow yaml

My Top 5 Books for DevOps/SRE

1. Crucial Conversations

2. Thanks for the Feedback

3. Accelerate: The science of lean software and devops

4. Site Reliability Workbook

5. Everybody Writes

Better Multiline Strings with `>` and `|`