GitHub Actions add useful automation to your development workflows, but too often your YAML files become a copy-and-pasted labrynth of scripts that lack any real structure. Treat your workflow YAML files as a code project in their own right or you'll find your workflows get in the way of what they're trying to achieve.
Get your .github folders back on track by thinking through these five ideas for reforming your approach to workflow development.
1. Refactor your workflows
Copying an existing workflow is often the easiest way to start a new test, but take a look through your YAML files to see if there is now too much repetition.
Composite Actions
Let's say you're running tests on each of four cloud providers. Each YAML file will have their own cloud setup (e.g. for AWS, or Google Cloud) but the rest of the YAML files may be similar.
Fetch those common steps into a new file called .github/actions/cloud-test.yaml:
name: 'Run Cloud Test'
description: 'Deploy app to the cloud then test it'
inputs:
vendor:
description: 'Which cloud vendor to use'
runs:
using: "composite"
steps:
- name: Install App
shell: bash
run: |
./app/install ${{ inputs.vendor }}
- name: Test App
...
You can then reference it from each of the original workflows, cutting out the repeated steps in the middle. For example, in .github/workflows/aws.yaml:
...
jobs:
deploy:
name: Deployment
runs-on: ubuntu-latest
steps:
- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: $AWS_ROLE_ARN
aws-region: us-east-1
- uses: ./.github/actions/cloud-test
with:
vendor: 'aws'
Above is an example of a Composite Action. There are also such things as Reusable Workflows. Here is a great explanation of the difference - either way, cut out repetition where you can!
Library of Actions
Especially if you need code reuse across multiple repositories, or if you need to do something that needs a real scripting language to achieve, you should go a step further and create repositories to contain fully-fledged actions to be used by your team. Encourage your developers to share useful actions at that level rather than hiding them within the repos where they first came to light.
I have two favourite guides to getting started writing an Action: Creating GitHub Actions in Python and Your First JavaScript GitHub Action.
Before that, the very first step can be as simple as outlining the possibilities to your developers, then they can dive in once they find a candidate action that should be shared more widely.
It is also possible to add workflows to these actions repos that will automatically run tests on the standalone actions themselves, making maintainance a whole lot easier!
2. Knowing how to debug
How often do you see the commit logs pictured in the cover image above? Just another quick commit to get the workflows to run to see if your YAML changes work... This will often run every workflow on the branch!
It is important your developers understand that, however tempting it is, it's very inefficient. As soon as you find yourself falling into this pattern, even for something that seemed like a simple fix to a YAML workflow, break out of it as soon as you can.
Workflows aren't easy to debug, mainly because they're not easy to run.
Create a simple proof of concept repo
The cleanest solution can be to create a new throwaway repo to replicate the essential parts of the problem you're trying to fix.
Maybe you're just trying to understand the interaction between the ${{ github.base_ref }} variable and the bash shell into which you pass it... There is no need to run all workflows to achieve this.
Debug live GitHub runners
You can opt to see extra debug logging output by your workflows. The easiest way is to Re-run jobs with debug logging after they've failed, or you can turn on debug logs in advance.
It's a bit dramatic, but it's possible to connect directly into the GitHub runner if you need to explore the machine. To do this, insert the Debugging with tmate action where you want the 'breakpoint' to occur, then the logs will show you a connection URL. Without this, you can be developing blind if you're not sure which paths are available on your runner, for example.
Run GitHub Actions locally
Sometimes you just want to be able to step through your YAML workflows on your local development machine. It can't replicate everything, but act is a very useful project that aims to run your workflow files using Docker containers on your own computer. This significantly improves the feedback cycle for the developer!
3. Opt out of workflows on certain runs
Do you really need to run all workflows on every commit? If you can cut down on the number of workflows that are run in certain circumstances, that will speed up the workflow run, bringing feedback to the developer quicker. You might be able to prevent alerting the developer to a workflow failure that wasn't relevant to them in the first place.
One obvious place to start is to consider if on: push is right for each workflow. Use paths and paths-ignore to control which push events will actually trigger a particular workflow.
Maybe you shouldn't use on: push at all, or only on main by using on: push: branches: - main. Individual developers might push to their own development branch simply to save their work, so many workflows may be irrelevant - consider using on: pull_request instead, to target completed work.
Do you also really need to test on every matrix combination of Python 3.5 through 3.11 and all versions of some other package? Again, maybe that's overkill until release time.
The point here is not to bother a developer with red failures for workflows that don't really need attention yet.
4. Speed up workflow runs
Also in the spirit of not leaving the developer waiting for feedback after commiting, consider optimizing your workflow runs themselves. Your usual software development skills should help here.
If you see repeated build or preparation steps that do not change when your codebase changes, look into caching the results. Here is a straightforward guide to caching, but also be aware caching is built into a lot of marketplace actions anyway, e.g. actions/setup-node can cache npm dependencies.
Use larger GitHub runners or self-host runners on your own cloud which can also save costs.
5. Streamline use of notifications
There are two extremes that teams can inhabit when it comes to monitoring their workflows: either pinging urgently every time a workflow completes (perhaps through a Slack notification action as the last step); or barely monitoring at all - perhaps just noticing failures via email or whenever they encounter them.
My main complaint about Slack notification actions is that they make a lot of unnecessary noise that can interrupt developers throughout the day. But a further problem comes when developers are working on the workflows themselves. Each time they push new YAML code they are conscious that their teammates will keep being disturbed, so they end up temporarily commenting out the notification action.
Reporting on overall workflow status becomes complicated for workflows with multiple jobs, and it is onerous to ask developers to maintain this notification code in each workflow YAML file. It's incredibly frustrating when slack-notify is itself the cause of a failure!
I built a native Slack app called Endid that monitors your workflows automatically, and only alerts you intelligently - when failures first occur, and then only on the first following success once fixed. You are welcome to install it on your own repos. It is entirely free for public open source repositories!
Please let me know your own tips for teams to work on GitHub Actions more effectively.
Top comments (0)