Automated Markdown Testing: Two Options

Recently I've been writing a lot of tech tutorials at work. These usually comprise of a Markdown file mixed in with code snippets. A user follows the instructions and executes the code snippets at the correct time.

For example:

kind create cluster --config=config.yaml
kubectl get namespaces

As I started scaling these tutorials, it became apparently that they could be flaky if / when the underlying technology changed or was updated. Obviously, the problem gets worse the more complexity and piece of tech that are involved in the tutorial.

Option 1: codedown

I need a way to take that Markdown and automatically run the snippets in a "unit test".

One solution is a brilliant, simple project I discovered called codedown which extracts the snippet from Markdown.

I can then pipe that to a new file and execute that file as a single e2e test:

cat index.md | codedown shell > e2e.sh
chmod +x e2e.sh
./e2e.sh

Option 2: mkdocs snippets

I write and host the tutorials as Markdown using the mkdocs material theme.

An alternative that I'm experimenting with is to keep the entire e2e instructions in a single file, annotating that file in a particular syntax and using the mkdocs snippets extension to include parts of the code at relevant places.

For example, imagine the docs are hosted in docs and this is the e2e shell script docs/e2e/script.sh:

#!/bin/sh
kind create cluster --config=config.yaml
kubectl create namespace foo

kubectl apply -f ingress.yaml

kubectl create namespace bar

kubectl apply -f app.yaml

I could use the snippets extension my mkdocs.yaml file:

site_name: An Example Tutorial
nav:
  - Homepage: index.md
theme:
  name: material
markdown_extensions:
  - pymdownx.snippets:
      base_path: ["docs"]

Then adjust docs/e2e/script.sh - adding annotations to split into logical sections that I want to include:

#!/bin/sh
--8<-- [start:createCluster]
kind create cluster --config=config.yaml
--8<-- [end:createCluster]

--8<-- [start:createFirstNamespace]
kubectl create namespace foo
--8<-- [end: createFirstNamespace]

--8<-- [start:installIngress]
kubectl apply -f ingress.yaml
--8<-- [end:installIngress]

--8<-- [start:createSecondNamespace]
kubectl create namespace bar
--8<-- [end:createSecondNamespace]

--8<-- [start:onboardApp]
kubectl apply -f app.yaml
--8<-- [end:onboardApp]

In the documentation, simply refer to the snippet names wherever you want to include them:

    ## Time to Spin up the cluster

    First, create the cluster:

    ```


    --8<-- "docs/e2e/script.sh:createCluster"


    ```

    ## Create First Namespace

    Create the first namespace and install the ingress.

    ```


    --8<-- "docs/e2e/script.sh:createFirstNamespace"
    --8<-- "docs/e2e/script.sh:installIngress"


    ```

With this method, the e2e.sh script is already ready to run as a single unit.

Summary

I'm still not sure which method I'll end up using, whether I'll find a different way or which is the most scalable (and understandable for other contributors).

I also need to build out the infrastructure to test this (for example when a PR is merged).

Watch this space!