DEV Community: David McKay

Kubernetes SDKs from the Pulumiverse

David McKay — Mon, 31 Jan 2022 17:15:54 +0000

Pulumi provides an amazingly rich interface for developers and operators to define their Kubernetes workloads, providing typed access to recourses from the Kubernetes API and allowing our IDEs to provide code completion and refactoring opportunities through the native language plugins.

As great as that is, it's always gotten a little cumbersome when it comes to Custom Resource Definitions (CRDs), as the first option is to leverage the CustomResource escape-hatch that allows you to define any Kubernetes object you wish; however this does mean we lose the rich interface we've become accustomed to.

new k8s.apiextensions.CustomResource('marvin-the-martian', {
    apiVersion: 'looneytunes.com/v1990',
    kind: 'LooneyTune',
    metadata: {
        name: 'Marvin the Martian',
        planet: 'mars'
    },
    spec: {
        armor: {
            style: 'hoplite',
            colors: ['green', 'red'],
        },
        phrases: [
            'Wheres the kaboom',
            'Oh, drat these computers. They’re so naughty and so complex. I could pinch them',
            'I do so enjoy observing the flora and fauna of that tiny planet',
            'Please, sir. Do not interrupt my chain of thought. I am a busy Martian',
            'Brace yourself for immediate disintegration'
        ]
    },
});

While this is a huge unblocker for working with Kubernetes, we can do better.

crd2pulumi

Back in August, 2020, we published }}">crd2pulumi. crd2pulumi allows you to generate a Pulumi SDK for any Kubernetes Custom Resource. Now, when you want to start using cert-manager, Ambassador, Linkerd, or any other project within the Kubernetes and Cloud Native space; you can download the CRD YAML and run crd2pulumi, which will generate the SDK for whatever supported language you wish. Neat?

I've been using this approach for the past year and it's the easiest way to provide that rich interface to developers, but the repetition can become a little frustrating. For every new project that I was working on, I'd need to build out the same automation and there's also the maintenance burden of ensuring the SDK is up to date with the latest version deployed to the cluster.

Further more, these SDKs never change; if you generate a SDK for cert-manager and someone else does, supporting the same version - there's no difference in the code that is generated. Do we really need everyone to do this themselves?

Perhaps some further tooling is required.

Kubernetes SDKs at Pulumiverse

I like solving problems like this. It's not technically challenging, the hard work is already done by the team supporting crd2pulumi. The problem we need to solve here is a quality of life / developer experience issue. How can we make this easier for developers to consume?

For us to make this easier, we have two issues:

How do we automate the generation of the SDKs, ensuring that the SDKs are always up to date?
Where do we publish them?

Automation

GitHub Actions plays such a large role in our developer world these days, especially for open source projects. So it was an easy choice to make that any automation I build out for this will be built on top of GitHub Actions. Knowing that we want the SDKs to be up to date, we also know that this automation must run on a regular cadence. As such, the beginnnig of our GitHub Action can take some shape:

name: Build SDKs

on:
  schedule:
    - cron: "10 3,15 * * *"

Here, we have our GitHub Action scheduled to run every day at 3:10am and 3:10pm.

Next, we need to know what to build. We don't want to build EVERY SDK that the project is aware of, so we need some code to look up GitHub Releases for some new tags. For this, we can drop to Python to do some quick lookups.

from atoma import parse_atom_bytes
from datetime import datetime, timedelta, timezone
from json import dumps
from requests import get as httpget
from typing import List, Mapping, Optional
from yaml import safe_load, YAMLError
import sys


def has_been_updated(updated_at: Optional[datetime] = None) -> bool:
    if updated_at == None:
        return False
    hours_since: int = int(sys.argv[1]) if len(sys.argv) > 1 else 12

    return (datetime.now(timezone.utc) - updated_at) < timedelta(hours=hours_since)


def get_new_tags(repository: str) -> List[str]:
    response = httpget(f'https://{repository}/tags.atom')
    feed = parse_atom_bytes(response.content)

    return list(map(lambda entry: entry.title.value, filter(lambda entry: has_been_updated(entry.updated), feed.entries)))


with open("./sdks.yaml", "r") as stream:
    try:
        sdks: Mapping[str, str] = safe_load(stream)
    except YAMLError as exc:
        print(f"Error parsing SDK Yaml: {exc}")

sdks_to_build: List[str] = []
for name, repository in sdks.items():
    for tag in get_new_tags(repository):
        sdks_to_build.append(
            f'{name}|{repository}|{tag}')

print(dumps(sdks_to_build))

This Python will loop through our configured SDKs, loaded from a YAML document, and reach out to the atom feed that GitHub provides for every public repository. This allows us to quickly loop over the tags/releases and find any published within the last window. We don't need to do anything else besides print this to the terminal and allow GitHub Actions to store the output for use in our build matrix.

jobs:
  find_new_tags:
    runs-on: ubuntu-latest
    steps:
      - ... random boring steps

      - id: find_new_tags
        run: echo "::set-output name=sdk_versions::$(poetry run python ./bin/find-new-tags.py ${{ github.event.inputs.since_hours }})"

    outputs:
      sdk_versions: ${{ steps.find_new_tags.outputs.sdk_versions }}

build_nodejs_sdk:
    needs: find_new_tags
    runs-on: ubuntu-latest
    strategy:
      matrix:
        version: ${{ fromJson(needs.find_new_tags.outputs.sdk_versions) }}

Once our Python script has run and the output stored, we're using a helper function from GitHub called fromJson to use the output as a dynamic input to their matrix build system. We'll get exactly one job for every new tag that we've found.

The job to generate the SDK and publish it to npm looks like this:

- name: crd2pulumi-generate-sdk
  run: |
    IFS='|' read -r -a build_parts <<< "${{ matrix.version }}"
    mkdir -p ./_output
    curl -fsSL -o crd.yaml https://doc.crds.dev/raw/${build_parts[1]}@${build_parts[2]}
    crd2pulumi --nodejsName ${build_parts[0]} --nodejsPath ./_output ./crd.yaml --force
    # Fix Package Name
    sed -ie "s#@pulumi/${build_parts[0]}#@pulumiverse/${build_parts[0]}#g" ./_output/package.json
    # Fix Package Version
    sed -ie "s#\"version\": \"\"#\"version\": \"${build_parts[2]}\"#g" ./_output/package.json
- run: npm publish --access=public
  working-directory: ./_output
  env:
    NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Like I said earlier, the hard work was already done; we just needed to glue some automation together to improve the DX.

Publishing

Now that we have this system in place to regularly check and build new SDKs, we needed a home for them to live. Recently, some amazing Pulumi community members, worked together on a new GitHub initiative, with Pulumi support, called Pulumiverse. Pulumiverse aims to be a community lead and governed initiative to provide new abstractions and libraries for consumers of Pulumi to use and make their lives easier. This is the perfect place for these SDKs to live.

You can checkout the repository today.

SDKs

Currently, the automation publishes NodeJS SDKs (JavaScript and TypeScript) to npm, but plans to add Python will be executed very soon. We're also looking to publish packages for dotNet, but research is still underway.

I don't expect to publish packages for Go ... sadly, because Go modules use GitHub for fetching source code - we'd need to publish generated code to the repository and I'm not entirely keen on that approach. I could possibly be persuaded, but I'd love some suggestions for alternate ways to resolve this.

We currently provide SDKs for:

argocd
cert-managaer
crossplane
knative
redpanda

Need more? No problem! Open a pull request adding a SINGLE LINE to sdks.yaml and the SDK will be available shortly.

Nice.

Prior Art

While writing this article, I discovered an old repository within the Pulumi organization that actually did something similar to this last year 😮 It's not automated like this new initiative, but it was great to find that previous attempts to improve the developer experience for Kubernetes teams using Pulumi. Mad props to Paul Stack and Albert Zhong.

So that's it! Thanks for reading. We hope you find this useful and we encourage you to join us and help by adding your favourite Kubernetes SDKs to the automation.

We'll see you in 2022, have a great New Year. 🎉

Introduction to OpenEBS

David McKay — Sat, 24 Oct 2020 09:56:20 +0000

OpenEBS builds on Kubernetes to enable Stateful applications to easily access Dynamic Local PVs or Replicated PVs. By using the Container Attached Storage pattern users report lower costs, easier management, and more control for their teams.

OpenEBS is a 100% Open Source CNCF project made with heart by MayaData & the community. Prominent users include Arista, Optoro, Orange, Comcast and the CNCF itself.

🕰 Timeline

00:00 - Holding Screen
02:10 - Introductions
03:35 - Slides - Introduction to OpenEBS
16:15 - What is DPDK (Data Plane Development Kit) and SPDK (Storage Performance Development Kit)
22:30 - My summary of what we've covered
31:00 - Why Huge Pages / Enabling Huge Pages on Linux
38:40 - Deploying OpenEBS with Mayastor
44:00 - Fixing my unhealthy cluster
47:30 - Adding the nvme kernel modules
52:30 - Configuring Mayastor
59:30 - Requesting a PersistentVolumeClaim
1:11:30 - Deploying fio to run some benchmarks
1:18:00 - Closing thoughts

🌎 Resources

Kiran Mova - https://twitter.com/kiranmova
Paul Burt - https://twitter.com/idvoretskyi
Jeffry Molanus - https://twitter.com/JeffryMolanus
OpenEBS - https://openebs.io/

Introduction to GitOps with GitOps Toolkit / Flux v2

David McKay — Sat, 24 Oct 2020 09:53:00 +0000

GitOps is a way to do Kubernetes cluster management and application delivery. It works by using Git as a single source of truth for declarative infrastructure and applications. With GitOps, the use of software agents can alert on any divergence between Git with what's running in a cluster, and if there's a difference, Kubernetes reconcilers automatically update or rollback the cluster depending on the case. With Git at the center of your delivery pipelines, developers use familiar tools to make pull requests to accelerate and simplify both application deployments and operations tasks to Kubernetes.

The GitOps Toolkit is a set of composable APIs and specialised tools that can be used to build a Continuous Delivery platform on top of Kubernetes.

These tools are build with Kubernetes controller-runtime libraries, and they can be dynamically configured with Kubernetes custom resources either by cluster admins or by other automated tools. The GitOps Toolkit components interact with each other via Kubernetes events and are responsible for the reconciliation of their designated API objects.

🕰 Timeline

00:00 - Holding Screen
01:25 - Introductions
02:00 - What is GitOps / GitOps Toolkit?
05:00 - Should I use Flux v1 or GitOps Toolkit?
07:45 - Bootstrapping GitOps Toolkit
15:00 - What are the GitOps Toolkit components?
17:40 - GitOps Toolkit CRDs
21:00 - Suspending reconciliation
23:30 - Deploying our first workload
27:10 - Questions
34:30 - Add another GitRepository
43:30 - Dependencies and health-checks
59:20 - Deploying Helm charts
1:09:00 - Final questions

🌎 Resources

Stefan Prodan - https://twitter.com/stefanprodan
GitOps Toolkit Soruce - https://github.com/fluxcd/toolkit
GitOps Toolkit Docs - https://toolkit.fluxcd.io
Walkthrough - https://gist.github.com/stefanprodan/1f5e0b31303a95885221e5c7733fc639

QuestDB on Kubernetes

David McKay — Mon, 24 Aug 2020 00:00:00 +0000

I was first introduced to QuestDB by my former colleague, David Simmons, who now runs their Developer Relations team. As I'm a rather curious individual that is interested in databases, especially time-series, I wanted to have a play with it.

I don't, unless absolutely need to, have a JVM installed on my laptop. QuestDB is written in Java ... so I had two options: Docker or Kubernetes.

I opted for Kubernetes, as there was no Helm chart currently available; this would give me a great opportunity to contribute to QuestDB.

If you'd prefer to watch a video than read this blog, fear not! You can watch David and I run through this process in the following video, which also includes configuring Telegraf to write data to QuestDB.

Installing QuestDB on Kubernetes with Helm

In order to install QuestDB to Kubernetes, we need to clone the Helm chart locally. This step is currently required because a Helm chart repository isn't currently being published by QuestDB yet, though this is something I expect to assist them with over the coming weeks with some further contributions.

git clone https://github.com/questdb/questdb-kubernetes

The chart ships with a values.yaml that should be familiar to anyone that's deployed a Kubernetes resource before. I cover each block in a little detail in the next section, should you be interested in that kinda thing.

For those of you that don't care, lets get to the fun bit. The following command assumes we've run the git clone above in the current working directory. If not, update the path at the end to wherever you cloned.

helm upgrade --install questdb ./questdb-kubernetes/charts/questdb

This command takes only a second, or two, and will print something like so to the terminal.

Release "questdb" does not exist. Installing it now.
NAME: questdb
LAST DEPLOYED: Mon Aug 24 11:15:15 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=questdb,app.kubernetes.io/instance=questdb" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:9000 to use your application"
  kubectl --namespace default port-forward $POD_NAME 9000:9000

Confirming the Install

The first thing we'll want to do is ensure that the pods are scheduled and running. You may see the STATUS as ContainerCreating for a few moments, this means that the container image is still being pulled into your cluster.

$ kubectl get pods

NAME READY STATUS RESTARTS AGE
questdb-0 1/1 Running 0 3s

💥 Perfect.

Now that QuestDB is running, we'll want to port-forward to the service in-order to checkout the web UI.

kubectl port-forward svc/questdb 8080:80

This setups a port-forward from our local machine into the Kubernetes cluster. We're specifying that the local port 8080 should be forwarded to port 80 our QuestDB service, svc/questdb, inside the Kubernetes cluster.

As if by magic, you'll now be able to browse to http://localhost:8080 and see the QuestDB UI.

Don't believe me? Try it 🤪

Writing Data

One of the really cool features of QuestDB, besides its ridiculous performance with large data sets, is the support for InfluxDB line protocol. That means we can use Telegraf to write data to QuestDB within a matter of minutes.

Assuming we have this telegraf.conf, which is configured with input plugins to gather metrics from our host; notable the cpu, disk, and mem plugins.

[agent]
  interval = "5s"

[[inputs.cpu]]
[[inputs.disk]]
[[inputs.mem]]

We can add Telegraf's generic socket_writer output plugin to write our data.

[[outputs.socket_writer]]
  address = "tcp://questdb:9009"
  data_format = "influx"

This output configuration specifies that we want Telegraf to use the line protocol data format and we want to write to a TCP socket on port 9009. This does require enabling one parameter within the values.yaml file.

First, lets take a look at our QuestDB service.

$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
questdb ClusterIP 10.105.188.157 <none> 80/TCP 52m

You can see that under PORT(S) that it only lists port 80, which is the web UI. So we need to upgrade our QuestDB deployment with a change that enables the InfluxDB line protocol support.

cat <<EOF >> values.yaml
service:
  type: ClusterIP
  port: 80
  expose:
    influx:
      enabled: true
      port: 9009
EOF

helm upgrade -f values.yaml questdb ./questdb-kubernetes/charts/questdb

We've used -f to specify our own values file, which is augmented / merged with the default one provided by the chart. Our values.yaml adds the configuration we need to enable the InfluxDB support; which exposes our port on the QuestDB service. Cool, huh? Now if we check out the service again ...

$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
questdb ClusterIP 10.105.188.157 <none> 80/TCP,9009/TCP 56m

We have port 9009 😀

I'm not going to cover running Telegraf inside our Kubernetes cluster, at-least not in this article. I also maintain the chart that makes that easy-easy too; which you can find here.

Easy Peasy

That is all it takes to get QuestDB running on a Kubernetes cluster using the official Helm chart and write some data to it with Telegraf.

Of course, there's a variety of things to consider when making the leap to production; and I'll explore that more as I continue my journey with QuestDB. Until then, go check it out for yourself.

Below is the deeper dive into the configuration available, if you're really keen.

Have a great day!

Helm Chart Values Explanations

Firstly, we have the option to override which image and tag to use for our QuestDB deployment. By default, it's configured to use questdb/questdb, which is officially maintained by the QuestDB team. I would always recommend using officially maintained images, but if you're in an air gapped environment or have compliance teams to appease, you can override as needed. This may require using imagePullSecrets to handle authentication for your private registry, which can also be provided. At the time of writing, the most recent version of QuestDB is 5.0.1.

image:
  repository: questdb/questdb
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  #tag: "5.0.1"
imagePullSecrets: []

Next, we have nameOverride and fullnameOverride. Pretty much every chart offers these and they're strictly superficial. They allow configuring the name of your resources that are created as a result of the deployment. We're not going to use them today.

nameOverride: ""
fullnameOverride: ""

Now we get a little more interesting. podAnnotations allow you to inject arbitrary annotations onto the pods. This can be useful for gathering metrics from your QuestDB deployment, primarily with Prometheus; which can use annotations for endpoint discovery.

podAnnotations: {}

Now for some meaty security stuff. We can configure the user that our process within the containers run as, as well as the group ownership of the filesystems we need for storing our data. We also have the ability to be rather explicit about the capabilities our containers have at the kernel level. This is a HUGE topic that needs to be handled gracefully, like a swan. HONK. There was a great TGIK with Duffie Cooley and Ian Coldwater recently that's worth watching, if Kubernetes security is something that tickles your fancy.

podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000

OK. Now we get to configuring QuestDB itself. I'll be honest, I don't know much about this, yet. However, it is documented.

questdb:
  config: #{}
    enabled: true
    options:
      shared.worker.count: 2

Running QuestDB on Kubernetes is great ... but we also want to be able to consume it. We can use the service block to enable other pods within our cluster to access QuestDB. QuestDB runs a web server, to provide its API and UI, on port 9000. However, it's quite common to expose that on the HTTP native port, 80, within Kubernetes. This means we can, from within the cluster, run curl http://questdb and get what we expect.

QuestDB also offers a few other ports, which allow for support of InfluxDB line protocol and PostgreSQL; on ports 9009 and 8812 respectfully. I actually forgot to add this to the Helm chart and it was contributed by Niclas Mietz; so thank you! 😃

service:
  type: ClusterIP
  port: 80
  expose:
    postgresql:
      enabled: false
      port: 8812
    influx:
      enabled: false
      port: 9009

You know what's more fun than accessing QuestDB internally within your cluster? Making it public ... ooooooh yeah.

The Helm chart allows you to leverage Kubernetes Ingress controllers to expose your services to the big bad world. In fact, QuestDB being rather bold ... actually runs their own public QuestDB that allows you to query 2.6 billion records ... that's wild. I don't think anyone's broken it ... yet 😂

ingress:
enabled: false
annotations: {}
# kubernetes.io/ingress.class: nginx # kubernetes.io/tls-acme: "true"
# hosts:
# - host: chart-example.local
# paths: []
# tls: []
# # - secretName: chart-example-tls
# # hosts:
# # - chart-example.local

You know what good a database is without state / persistence? NOTHING.

We can configure our QuestDB deployment to use persistence through Kubernetes primitives, PVC/PV's. Resizing these isn't as easy as one might hope, but it does continually get easier (especially since Kubernetes 1.17). So think carefully about how much space to provision.

persistence:
  enabled: true
  #storageClass: "-"
  accessMode: ReadWriteOnce
  size: 50Gi

QuestDB does some magic stuff with the JVM to disable GC, which means that it can run on limited resources. You can use the resources block to limit this explicitly. There's no guessing these numbers, I'm afraid. You'll need to run without these constraints and profile / monitor your pods to see what "normal" usage patterns look like. Then add a safety ceiling of (10|20|30)% and go with that. Best of luck ... resource constraints are tricky business.

resources: {}
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi

Finally, the last block. Like every Helm chart before it and every Helm chart that will come after it, we provide the ability to encourage the scheduling of your QuestDB pods to the nodes that you wish, whether it be for storage class support or locality to your other producing or consuming pods; you can do whatever you need with the following three keys.

nodeSelector: {}
tolerations: []
affinity: {}

Packet Metadata for SaltStack

David McKay — Mon, 03 Aug 2020 00:00:00 +0000

In my last article, we spun up some bare metal compute on Packet with Pulumi and installed SaltStack.

In order to use SaltStack to provision our workloads on our servers, we need a way to identify which machines should be provisioned with what workload. SaltStack uses Grains to do this ... and there's a metadata grain that can read metadata from cloud providers; unfortunately, it doesn't support Packet.

Drats ☹️

Happy news though, SaltStack is rather extensible; as long as you don't mind getting your hands a little dirty with Python.

Writing a Custom Grain

Writing a SaltStack grain module is SUPER easy. Lets take a look at the simplest implementation I can put together.

def test():
    return dict(test={
        "name": "David",
        "age": "18",
    })

Yeah, yeah. I know I'm not 18 anymore. Shush.

Grain modules are Python functions that return key value pairs. This code above returns a grain named "test" with the key/value pairs name = David and age = 18. This means we can run salt minion-1 grains.item test and we'll see:

minion-1:
    ----------
    test:
        ----------
        name:
            David
        age:
            18

Of course, we don't want to return hared coded key values! We want to return information about our servers from Packet's metadata API.

The code to handle this isn't particularly complicated. In-fact, performing a HTTP request in Python is really simple 😀

Lets take a look.

import json
import logging
import salt.utils.http as http

# Setup logging
log = logging.getLogger( __name__ )

# metadata server information
HOST = "https://metadata.packet.net/metadata"

def packet_metadata():
    response = http.query(HOST)
    metadata = json.loads(response["body"])

    log.error(metadata)

    grains = {}
    grains["id"] = metadata["id"]
    grains["iqn"] = metadata["iqn"]
    grains["plan"] = metadata["plan"]
    grains["class"] = metadata["class"]
    grains["facility"] = metadata["facility"]

    grains["tags"] = metadata["tags"]

    return dict(packet_metadata=grains)

The important lines here are these three:

HOST = "https://metadata.packet.net/metadata"

response = http.query(HOST)
metadata = json.loads(response["body"])

We first query the metadata API endpoint, defined by the variable HOST. We then decode the body of the response into a Python dict, using json.loads.

This gives us access to every bit of metadata returned by the Packet metadata API. That looks like:

{
  "id": "c5ce85c5-1eef-4581-90b6-88a91e47e207",
  "hostname": "master-1",
  "iqn": "iqn.2020-08.net.packet:device.c5ce85c5",
  "operating_system": {
    "slug": "debian_9",
    "distro": "debian",
    "version": "9",
    "license_activation": {
      "state": "unlicensed"
    },
    "image_tag": "b32a1f31b127ef631d6ae31af9c6d8b69dcaa9e9"
  },
  "plan": "c2.medium.x86",
  "class": "c2.medium.x86",
  "facility": "ams1",
  "private_subnets": ["10.0.0.0/8"],
  "tags": ["role/salt-master"],
  "ssh_keys": [
    "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGf0w9b+lPcZhsNHU8Sw5hJPBhpNICTNkjlBz9jxtLbWNGvHTE1lBeXU5VA2/7cuYw48apHmMURHFtK5AZx3srg="
  ],
  "storage": {
    "disks": [
      {
        "device": "/dev/sdd",
        "wipeTable": true,
        "partitions": [
          {
            "label": "BIOS",
            "number": 1,
            "size": "512M"
          },
          {
            "label": "SWAP",
            "number": 2,
            "size": "3993600"
          },
          {
            "label": "ROOT",
            "number": 3,
            "size": 0
          }
        ]
      }
    ],
    "filesystems": [
      {
        "mount": {
          "device": "/dev/sdd1",
          "format": "vfat",
          "point": "/boot/efi",
          "create": {
            "options": ["32", "-n", "EFI"]
          }
        }
      },
      {
        "mount": {
          "device": "/dev/sdd3",
          "format": "ext4",
          "point": "/",
          "create": {
            "options": ["-L", "ROOT"]
          }
        }
      },
      {
        "mount": {
          "device": "/dev/sdd2",
          "format": "swap",
          "point": "none",
          "create": {
            "options": ["-L", "SWAP"]
          }
        }
      }
    ]
  },
  "network": {
    "bonding": {
      "mode": 4,
      "link_aggregation": "bonded",
      "mac": "50:6b:4b:b4:a9:3a"
    },
    "interfaces": [
      {
        "name": "eth0",
        "mac": "50:6b:4b:b4:a9:3a",
        "bond": "bond0"
      },
      {
        "name": "eth1",
        "mac": "50:6b:4b:b4:a9:3b",
        "bond": "bond0"
      }
    ],
    "addresses": [
      {
        "id": "5d28837b-29c5-4505-bb05-930fd3760bac",
        "address_family": 4,
        "netmask": "255.255.255.252",
        "created_at": "2020-08-03T14:07:50Z",
        "public": true,
        "cidr": 30,
        "management": true,
        "enabled": true,
        "network": "147.75.84.128",
        "address": "147.75.84.130",
        "gateway": "147.75.84.129",
        "parent_block": {
          "network": "147.75.84.128",
          "netmask": "255.255.255.252",
          "cidr": 30,
          "href": "/ips/7a30c2bf-f0e5-402c-b0c0-b8ab03359e63"
        }
      },
      {
        "id": "937552c6-cf1a-474d-9866-9fb1e0525503",
        "address_family": 4,
        "netmask": "255.255.255.254",
        "created_at": "2020-08-03T14:07:49Z",
        "public": false,
        "cidr": 31,
        "management": true,
        "enabled": true,
        "network": "10.80.76.4",
        "address": "10.80.76.5",
        "gateway": "10.80.76.4",
        "parent_block": {
          "network": "10.80.76.0",
          "netmask": "255.255.255.128",
          "cidr": 25,
          "href": "/ips/8f8cd919-165a-4e62-b461-af7c15a25ec4"
        }
      }
    ]
  },
  "customdata": {},
  "specs": {
    "cpus": [
      {
        "count": 1,
        "type": "AMD EPYC 7401P 24-Core Processor @ 2.0GHz"
      }
    ],
    "memory": {
      "total": "64GB"
    },
    "drives": [
      {
        "count": 2,
        "size": "120GB",
        "type": "SSD",
        "category": "boot"
      },
      {
        "count": 2,
        "size": "480GB",
        "type": "SSD",
        "category": "storage"
      }
    ],
    "nics": [
      {
        "count": 2,
        "type": "10Gbps"
      }
    ],
    "features": {}
  },
  "switch_short_id": "f8dd5e3f",
  "volumes": [],
  "api_url": "https://metadata.packet.net",
  "phone_home_url": "http://tinkerbell.ams1.packet.net/phone-home",
  "user_state_url": "http://tinkerbell.ams1.packet.net/events"
}

I decided not to make all of this available within the grains system, as only a few data points make sense for scheduling workloads. Hence, I cherry pick out the attributes I want for the next demo. You can pick and choose whatever you want too.

grains = {}
grains["id"] = metadata["id"]
grains["iqn"] = metadata["iqn"]
grains["plan"] = metadata["plan"]
grains["class"] = metadata["class"]
grains["facility"] = metadata["facility"]

grains["tags"] = metadata["tags"]

return dict(packet_metadata=grains)

Provisioning the Custom Grain

Now that we have a custom grain, we need to update our Pulumi code to install this on our Salt master.

NB : We only need to make this grain available on our Salt master, as the Salt master takes responsibility for syncing custom grains to the minions.

I've updated our user-data.sh to create the directory we need and added the mustache template syntax that allows us to inject the Python script. We use & before the variable name to request that mustache doesn't escape our quotes to HTML entities ... I only learnt that today 😂

mkdir -p /srv/salt/_grains

cat <<EOF >/srv/salt/_grains/packet_metadata.py
{{ &PACKET_METADATA_PY }}
EOF

Next up, we provide the Python script at render time and provide some tags for our servers when we create them.

const pythonPacketMetadataGrain = fs
  .readFileSync(path.join(__dirname, "..", "salt", "packet_metadata.py"))
  .toString();

const saltMaster = new Device(`master-${name}`, {
  // ... code omitted for brevity
  userData: mustache.render(bootstrapString, {
    PACKET_METADATA_PY: pythonPacketMetadataGrain,
  }),
  // Add tags to this server
  tags: ["role/salt-master"],
});

Syncing Custom Grains

Finally, we need to tell our Salt master to sync the grains to our minions.

salt "*" saltutil.grains_sync

You can now confirm the custom grain is working with:

root@master-1:~# salt "*" grains.item packet_metadata
minion-1:
    ----------
    packet_metadata:
        ----------
        class:
            c2.medium.x86
        facility:
            ams1
        id:
            ab0bc2ba-557b-4d99-a1eb-0beec02adff2
        iqn:
            iqn.2020-08.net.packet:device.ab0bc2ba
        plan:
            c2.medium.x86
        tags:
            - role/salt-minion
master-1:
    ----------
    packet_metadata:
        ----------
        class:
            c2.medium.x86
        facility:
            ams1
        id:
            97ce9196-077d-4ce9-82a5-d58bf59d0dbc
        iqn:
            iqn.2020-08.net.packet:device.97ce9196
        plan:
            c2.medium.x86
        tags:
            - role/salt-master

That's it! Next time we'll take a look at using our tags to provision and schedule our workloads.

See you then.

SaltStack on Packet with Pulumi

David McKay — Thu, 30 Jul 2020 00:00:00 +0000

WIP: You probably don't want to read this yet 😂

I joined Packet on July 27th (4 days ago, at the time of writing) as a Senior Tech Evangelist. This is exciting! I get to work for a Cloud company that specialises in bare metal compute. So what should I do first?

I'm going to launch my favourite configuration management software with my favourite infrastructure as code software on my new favourite cloud provider. Sweet, right? 🍭🍬

If you don't want to read the walk through, the code is available here.

Why SaltStack?

Reasons

Why Pulumi?

Reasons

Step 1. Pulumi

Pulumi is a Infrastructure as Code (IaaC) tool that allows you to describe your infrastructure, much like Terraform. Unlike Terraform, Pulumi doesn't impose a specific DSL, HCL, on you; and instead, you can use your programmaing language of choice ... provided it's supported.

At the time of writing, Pulumi supports:

C#
F#
Go
JavaScript
Python
TypeScript
VisualBasic

I know, I know. I'm disappinted Rust isn't there either. Maybe one day.

Creating the Stack

So to create our stack, we need to generate a new Pulumi project. For this example, we'll use the Pulumi "local" login, which stores the statefile on our local disk. The statefile is very similar to Terraform state. It is needed to build an execution plan for our apply commands. Using local will suffice today, but you should investigate using alternative options for production deployments.

pulumi login --local

We're going to use the TypeScript template to get started. Unfortunately, there isn't a template for all supported languages, but templates do exist for the following.

packet-go # A minimal Packet.net Go Pulumi program
packet-javascript # A minimal Packet.net JavaScript Pulumi program
packet-python # A minimal Packet.net Python Pulumi program
packet-typescript # A minimal Packet.net TypeScript Pulumi program

If you want to use one of the dotNet languages, you can use a generic template; then add the Packet provider manually. Generic templates for dotNet are called.

csharp # A minimal C# Pulumi program
fsharp # A minimal F# Pulumi program
visualbasic # A minimal VB.NET Pulumi program

To create our project from the TypeScript template, let's run:

pulumi new packet-typescript

You'll be walked through a couple of questions to create your stack, after which you'll have a directory that looks like:

drwxr-xr-x - rawkode 30 Jul 18:07 node_modules
.rw------- 286 rawkode 30 Jul 18:06 index.ts
.rw-r--r-- 28k rawkode 30 Jul 18:07 package-lock.json
.rw------- 201 rawkode 30 Jul 18:06 package.json
.rw-r--r-- 85 rawkode 30 Jul 18:07 Pulumi.dev.yaml
.rw------- 100 rawkode 30 Jul 18:06 Pulumi.yaml
.rw------- 438 rawkode 30 Jul 18:06 tsconfig.json

If we take a look inside of index.ts, we'll see:

import * as pulumi from "@pulumi/pulumi";
import * as packet from "@pulumi/packet";

// Create a Packet resource (Project)
const project = new packet.Project("my-test-project", {
  name: "My Test Project",
});

// Export the name of the project
export const projectName = project.name;

This TypeScript uses the Pulumi SDKs to provide a nice wrapper around the Packet API. Hopefully it's pretty self-explanitory; you can see that it creates a new project and exports it by name.

Exports are similar to Terraform outputs. We use an export when we want to make some attribute from our stack available outside of Pulumi. Pulumi provides the pulumi stack output command, which displays these exports.

We'll use these later.

Cleaning Up the Stack

This step is completely subjective, but I don't like my code just chilling in the top level directory with all the other stuff. What is this, Go? 😏

Fortunately, we can append main: src/index.ts to the Pulumi.yaml file, which tells Pulumi our entyrypoint for this stack lives somewhere else. I'm going to use a src directory.

mkdir src
mv index.ts src/
echo "main: src/index.ts" >> Pulumi.yaml

Creating a "Platform"

I like to create a Platform object / type / class that can be used to pass around the Pulumi configuration and some other common types that my Pulumi projects often need. This saves my function signatures getting too gnarly as we add new components to our stacks.

The Platform object I'm using for this is pretty trivial. It loads the Pulumi configuration and stores our Packet Project, which means we can pass around Platform to other functions and it's a single argument, rather than many.

// ./src/platform.ts
import { Config } from "@pulumi/pulumi";
import { Project } from "@pulumi/packet";

export type Platform = {
  project: Project;
  config: Config;
};

export const getPlatform = (project: Project): Platform => {
  return {
    project,
    config: new Config(),
  };
};

Now we can update our ./src/index.ts to look like so:

import * as packet from "@pulumi/packet";
import { getPlatform } from "./platform";

const project = new packet.Project("pulumi-saltstack-example", {
  name: "pulumi-saltstack-example",
});

const platform = getPlatform(project);

export const projectName = platform.project.name;

Creating the SaltMaster

Now we want to create the Salt Master server. For this, I create a new directory with an index.ts that exports a function called createSaltMaster; which I can consume in our ./src/index.ts, much like we did with our Platform.

Here's the complete file, but I'll run through each part seperately too; don't worry! I'm not going to explain the imports, I'll do that as we go.

// ./src/salt-master/index.ts
import {
  Device,
  IpAddressTypes,
  OperatingSystems,
  Plans,
  Facilities,
  BillingCycles,
} from "@pulumi/packet";
import { Platform } from "../platform";
import * as fs from "fs";
import * as path from "path";
import * as mustache from "mustache";

export type SaltMaster = {
  device: Device;
};

export const createSaltMaster = (
  platform: Platform,
  name: string
): SaltMaster => {
  // While we're not interpolating anything in this script atm,
  // might as well leave this code in for the time being; as
  // we probably will shortly.
  const bootstrapString = fs
    .readFileSync(path.join(__dirname, "./user-data.sh"))
    .toString();

  const bootstrapScript = mustache.render(bootstrapString, {});

  const saltMaster = new Device(`master-${name}`, {
    hostname: name,
    plan: Plans.C1LargeARM,
    facilities: [Facilities.AMS1],
    operatingSystem: OperatingSystems.Debian9,
    billingCycle: BillingCycles.Hourly,
    ipAddresses: [
      { type: IpAddressTypes.PrivateIPv4, cidr: 31 },
      {
        type: IpAddressTypes.PublicIPv4,
      },
    ],
    projectId: platform.project.id,
    userData: bootstrapScript,
  });

  return {
    device: saltMaster,
  };
};

SaltMaster Return Type

export type SaltMaster = {
  device: Device;
};

Because this is TypeScript, we want to be very explicit about the return types within our code. This allows us to catch errors before we ever run our Pulumi stack. As we're using createSaltMaster function to create our SaltMaster, we want that function to return a type with the resources we create.

While our function only returns a Device, it's still nice to encapsulate that in a named type that allows for our function to evolve over time.

import { Platform } from "../platform";

export const createSaltMaster = (
  platform: Platform,
  name: string
): SaltMaster =>

This is the function signature for createSaltMaster. You can see our return type is the type we just created, SaltMaster.

Our function also takes a couple of parameters, namely platform and name. The platform is our Platform object with our Pulumi configuration and the Packet Project, so we also need to import it. The name allows us to give our SaltMaster a name when we create the device on Packet. We could hardcode this inside the function as salt-master, but then we can't use createSaltMaster more than once for a highly available set up in a later tutorial.

import * as fs from "fs";
import * as path from "path";
import * as mustache from "mustache";

const bootstrapString = fs
  .readFileSync(path.join(__dirname, "./user-data.sh"))
  .toString();

const bootstrapScript = mustache.render(bootstrapString, {});

I know what you're thinking ... but trust me. As Pulumi allows us to use a programming language to describe our infrastructure, we also have access to that programming languages entire eco-system of libraries. As such, if I want to template some user data for a server ... say, to provision and install SaltStack ... I can use a popular templating tool, such as mustache, from npm 😉

My user data for the Salt Master looks like so:

# ./src/salt-master/user-data.sh
#!/usr/bin/env sh
apt update
DEBIAN_FRONTEND=noninteractive apt install -y python-zmq python-systemd python-tornado salt-common salt-master

LOCAL_IPv4=$(ip addr | grep -E -o '10\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}')

cat <<EOF >/etc/salt/master.d/listen-interface.conf
interface: ${LOCAL_IPv4}
EOF

systemctl restart salt-master

Lastly, we need to create the device with the Packet API. We use the Pulumi SDK to do so. I'm explictly importing the required types that I need to use as much of the type system as I can.

import {
  Device,
  IpAddressTypes,
  OperatingSystems,
  Plans,
  Facilities,
  BillingCycles,
} from "@pulumi/packet";

const saltMaster = new Device(`master-${name}`, {
  hostname: name,
  plan: Plans.C1LargeARM,
  facilities: [Facilities.AMS1],
  operatingSystem: OperatingSystems.Debian9,
  billingCycle: BillingCycles.Hourly,
  ipAddresses: [
    { type: IpAddressTypes.PrivateIPv4, cidr: 31 },
    {
      type: IpAddressTypes.PublicIPv4,
    },
  ],
  projectId: platform.project.id,
  userData: bootstrapScript,
});

Next up, we can call our createSaltMaster function from ./src/index.ts and we'll have a server with the correct user data for running our salt-master.

const saltMaster = createSaltMaster(platform, "master-1");
export const saltMasterPublicIp = saltMaster.device.accessPublicIpv4;

We're going to export it's public IPv4 address; so that we can access it easily later and SSH into the machine later.

Automate Google Calendar Invitations with Slack and Pipedream

David McKay — Wed, 15 Jul 2020 15:01:39 +0000

When I was working at InfluxData, I put together a programme that aimed to solve a common business challenge.

How do we encourage and enable cross-functional collaboration within a fast moving organisation?

Having spent many years working and contributing to open source communities, I've seen many governance and collaboration models used across various successful open source projects; but the one that continues to impress me to this day is the Special Interest Groups (SIGs) model used by Kubernetes.

I was curious what would happen if we adopted SIGs internally to provide guidance around our organisations involvement, support, and enable of our technology within external communities. That is to say that InfluxData had people across multiple disciplines that wished to collaborate on the use of our product within OpenTracing, Kubernetes, Machine Learning, and many other communities and ecosystems; but until then had no structured way to drive that collaboration forward.

I'm not writing this article to talk about the progress of that programme, perhaps I'll do that later. What I wanted to write about was one of the challenges we had and how we solved it with a fantastic piece of open source software: Pipedream.

The Problem

When I put this programme together, one of my goals was for each SIG to be inclusive. This meant that I was keen to avoid any one person controlling the calendar invitations, deciding whom they seen fit to join each SIG. Instead, I wanted each SIG to have a doors open policy that invited anyone and everyone to join their initiatives.

Google Calendar provides "Team Calendars" to help achieve this. The concept is simple, but sadly the implementation is painful. While I was able to successfully create the shared calendar and pass around a link, the link didn't seem to work for everyone and was cumbersome, at best, to consume.

More importantly, there's no discovery of these events, you need to be told to look for them.

This led to the first few meetings having next to no participation, other than the core people that expressed an interest in the groups upfront.

The Solution

So I got my thinking hat on. One of the mandates of the SIG programme was that every SIG MUST have a Slack channel, where the chair will share updates after each meeting.

What if we could consider "membership" of the Slack channel as an indicator that they probably want to be aware of the events; hell, even auto-invite them? I kept my thinking hat on 🎩

The goal is now this:

Whenever anyone joins one of our #sig-channels on Slack, we automatically update the Google Calendar event (which still lives in the team calendar) with their email address.

Lets do this!

The Implementation

I like to avoid writing code or deploying anything as much as possible. NoCode and LowCode solutions are very exciting to me; because they're enablers for everyone, no matter what your constraints are. My biggest constraint is time, not technical knowledge; but NoCode and LowCode are great enablers, or gateways, for non-technical folk. I love these solutions.

So I decided to utilise one of these solutions to provide the "plumbing" of this auto-inviter, hopefully allowing me to deliver a solution to my problem without writing or deploying any code or containers.

Pipedream

You can read more about Pipedream at their website, but I'll share the first paragraph from their docs:

Pipedream is an integration platform for developers to build and run workflows that integrate apps, data, and APIs — no servers or infrastructure to manage!

Develop any workflow, based on any trigger.

Workflows are code, which you can run for free.

No server or cloud resources to manage.

It ticked all the boxes I had:

Support Slack and Google Calendar as sources / sinks
OpenSource
NoCode / LowCode
SaaS (I didn't want to deploy anything)

It took me 30 minutes to implement this, let me show you how.

Step 1. Connect Your Accounts

I'm not going to cover this really, but Pipedream makes it very easy to connect your Pipedream account with your other services.

For this tutorial, you need to connect Twitter and Slack using the built-in OAuth prompts.

Step 2. Prepare the Webhook

I decided that I wanted to use Slack's webhooks / events API to consume events from Slack. It's the easiest way to get started and it works really well in a consume and emit / event driven workflow.

Pipedream provides HTTP endpoints that can receive arbitrary payloads and you can build your workflow around these with ease.

So I created a webhook using the Pipedream UI and I got a URL like https://randomID.m.pipedream.net.

You can see how to do this yourself, on their docs.

Pro Tip

Pipedream allows you to see the payloads that have hit your endpoint. I suggest hooking up your source ASAP and building up a history of payloads to see what you need to handle.

Pipedream also provides autocomplete based on previous payloads when working with future steps in the workflow.

Ridiculous, right?

🥰🥰🥰

Step 3. Slack Challenge

When you add a new receiver for Slack events, it first sends a challenge. You need to be able to respond to this correctly with their challenge string.

Pipedream allows us to add arbitrary JavaScript to handle payloads through a "NodeJS Step". The code I used was really simple.

if (event.body && event.body.challenge) {
  $respond({
    status: 200,
    body: event.body.challenge,
  });
}

We check if the payload contains a challenge parameter and we respond with it.

Next!

Step 4. Slack Channel Lookup

I configured the Slack events integration to only send the join channel events. Part of this payload is a channel identifier, but not the channel name. So we need to query the Slack API to get the actual name of the channel.

We use the NodeJS Step again to build up a config object to send to Slack with axios.

Pipedream provides the authentication we need through the auths object that is available after we complete the OAuth connection from Step 1.

We need to configure a param for this step, which we can do through the GUI. You add a param called channel which using the shiny autocomplete dropdown we can set to event.body.event.channel.

event.body is the payload we receive from Slack, which contains event.channel.

// See the API docs here: https://api.slack.com/methods/channels.info
var include_locale = params.include_locale || false;
const config = {
  url: `https://slack.com/api/conversations.info?channel=${params.channel}&include_locale=${include_locale}`,
  headers: {
    Authorization: `Bearer ${auths.slack.oauth_access_token}`,
    "Content-Type": "application/x-www-form-urlencoded",
  },
};
const channel = await require("@pipedreamhq/platform").axios(this, config);

if (channel.ok != true) {
  $end("Couldn't fetch Channel information");
}

this.channelName = channel.channel.name;

return;

Step 4. User Lookup

Much like the channel lookup, we also need to lookup the user information. We can't add the Slack ID to a Google Calendar invite, we need their email address.

This time we configure a param called user, which comes from event.body.event.user.

//See the API docs here: https://api.slack.com/methods/users.info
const data = {
  user: params.user,
  include_locale: params.include_locale || false,
};
const config = {
  url: `https://slack.com/api/users.info`,
  headers: {
    Authorization: `Bearer ${auths.slack.oauth_access_token}`,
    "Content-Type": "application/x-www-form-urlencoded",
  },
  params: data,
};
const user = await require("@pipedreamhq/platform").axios(this, config);

if (user.ok !== true) {
  $end("Failed to get user information");
}

this.userName = user.user.real_name;
this.userEmail = user.user.profile.email;

return;

Step 5. Add to Google Calendar Event

Finally, we want to add them to the invite! Unfortunately you need to hardcode some values here, one identifier for each event. I'm sure there's a way to do this programmatically with the Google Calendar API, but I've not worked that out yet.

If the channel isn't one of the expected channels we have an event for; we exit early with $end().

For channels we do understand, we add the email address to the event. This is idempotent, so we don't need to check if the user already exists on the invite.

Pipedream allows us to fetch variables from the previous steps, which we use to get the users name and email address. Neat, huh?

This looks like steps.slack_get_user_info.userEmail, where slack_get_user_info is the name of the previous step, and userEmail is the variable I "exposed" with the this.userEmail = syntax.

const axios = require("axios");

switch (steps.slack_get_channel_info.channelName) {
  case "sig-kubernetes":
    eventId = "EventID from Google Calendar";
    break;

  case "sig-opentelemetry":
    eventId = "EventID from Google Calendar";
    break;

  case "sig-ml":
    eventId = "EventID from Google Calendar";
    break;

  default:
    $end("Not a SIG channel.");
    return;
}

calendarId = params.calendarId;

event = await require("@pipedreamhq/platform").axios(this, {
  url: `https://www.googleapis.com/calendar/v3/calendars/${calendarId}/events/${eventId}`,
  headers: {
    Authorization: `Bearer ${auths.google_calendar.oauth_access_token}`,
  },
  method: "GET",
});

newAttendee = {
  email: steps.slack_get_user_info.userEmail,
  name: steps.slack_get_user_info.userName,
};

if (event.attendees) event.attendees.push(newAttendee);
else event.attendees = [newAttendee];
data = event;

return await require("@pipedreamhq/platform").axios(this, {
  url: `https://www.googleapis.com/calendar/v3/calendars/${calendarId}/events/${eventId}`,
  headers: {
    Authorization: `Bearer ${auths.google_calendar.oauth_access_token}`,
  },
  method: "PUT",
  data: data,
});

Contributing to nixpkgs

David McKay — Wed, 15 Jul 2020 10:49:26 +0000

I've been using NixOS, on and off, for around 2 years now. It's got its challenges, which
usually means I switch back to Arch after a few weeks; but not this time ... I'm not switching back.

As I stumble my way through the trials and tribulations of Nix, NixOS, and nixpkgs; I'll document
my path so that others can learn from my misery. Starting right now ...

What is Nix / NixOS?

This is from the NixOS website:

Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. Share
your development and build environments across different machines.

NixOS is a Linux distribution with a unique approach to package and configuration management. Built on top of the Nix package
manager, it is completely declarative, makes upgrading systems reliable, and has many other advantages.

Contributing a GNOME Extension to nixpkgs

In this article, I am going to walk you through the steps I've taken to contribute a GNOME Extension
to the nixpkgs repository. The extension is one that I just CAN'T live without, the
emoji selector 😹🌟🥰

This wasn't my first contribution to nixpkgs, I've
sent 16 pull requests to nixpkgs since August 2018 ... and I still have no idea what I'm doing 😂

OK. That's a lie, I know enough to be dangerous, but not enough to be smart.

Lets try this, one step at a time.

Step 1. Clone the Repository

Not much needs said about this, right?

git clone https://github.com/NixOS/nixpkgs

Step 2. Find Something Similiar

I'm not a huge fan of reinventing the wheel. I'm certainly not going to type a bunch of Nix code
that shares about 95% of it's logic with many other packages already contributed to the nixpkgs repository.

Fortunately for us, this repository is segmented really well.

First, all packages live under ./pkgs.

drwxr-xr-x - rawkode 11 Jul 13:34 -- applications
drwxr-xr-x - rawkode 11 Jul 13:34 -- build-support
drwxr-xr-x - rawkode 11 Jul 13:34 -- common-updater
drwxr-xr-x - rawkode 11 Jul 13:34 -- data
drwxr-xr-x - rawkode 11 Jul 13:34 -- desktops
drwxr-xr-x - rawkode 11 Jul 13:34 -- development
drwxr-xr-x - rawkode 11 Jul 13:34 -- games
drwxr-xr-x - rawkode 11 Jul 13:34 -- misc
drwxr-xr-x - rawkode 11 Jul 13:34 -- os-specific
drwxr-xr-x - rawkode 11 Jul 13:34 -- servers
drwxr-xr-x - rawkode 11 Jul 13:34 -- shells
drwxr-xr-x - rawkode 11 Jul 13:34 -- stdenv
drwxr-xr-x - rawkode 11 Jul 13:34 -- test
drwxr-xr-x - rawkode 11 Jul 13:34 -- tools
drwxr-xr-x - rawkode 14 Jul 23:40 -- top-level

As you can see, there's directories for desktop applications, development stuff, games, and a few other categories.

Inside of desktops, we can see:

drwxr-xr-x - rawkode 11 Jul 13:34 -- cdesktopenv
drwxr-xr-x - rawkode 11 Jul 13:34 -- cinnamon
drwxr-xr-x - rawkode 11 Jul 13:34 -- deepin
drwxr-xr-x - rawkode 11 Jul 13:34 -- enlightenment
drwxr-xr-x - rawkode 11 Jul 13:34 -- gnome-2
drwxr-xr-x - rawkode 11 Jul 13:34 -- gnome-3
drwxr-xr-x - rawkode 11 Jul 13:34 -- gnustep
drwxr-xr-x - rawkode 11 Jul 13:34 -- lumina
drwxr-xr-x - rawkode 11 Jul 13:34 -- lxde
drwxr-xr-x - rawkode 11 Jul 13:34 -- lxqt
drwxr-xr-x - rawkode 11 Jul 13:34 -- mate
drwxr-xr-x - rawkode 11 Jul 13:34 -- pantheon
drwxr-xr-x - rawkode 11 Jul 13:34 -- plasma-5
drwxr-xr-x - rawkode 11 Jul 13:34 -- rox
drwxr-xr-x - rawkode 11 Jul 13:34 -- surf-display
drwxr-xr-x - rawkode 11 Jul 13:34 -- xfce

Pretty much every desktop environment there is ... and if you're shouting in your head
"WHAT ABOUT MY SHITTY ESOTERIC TILING WINDOW MANAGER?" ... then I've got you; it's under
./applications/window-managers. I love i3, but it's also quite nice being able to change
the volume or pair bluetooth headphones without having to put my beer down and grep some shell
history.

Notice how I delicately said "shell history" and not zsh, bash, fish, or nu ... I can't be
bothered getting into another shell debate; 2020's been shit enough.

Sorry, I've digressed.

Lets move forward. I've found some similiar packages. I've redacted some of the extensions below
... mostly at random; but you can see that we're in the ./pkgs/desktops/gnome-3/extensions directory
and we have a fair number of example packages to use as a base for our new one.

Perfect.

pwd

/home/rawkode/Code/src/github.com/NixOS/nixpkgs/pkgs/desktops/gnome-3/extensions

ll

drwxr-xr-x - rawkode 11 Jul 13:34 -- appindicator
drwxr-xr-x - rawkode 11 Jul 13:34 -- arc-menu
drwxr-xr-x - rawkode 11 Jul 13:34 -- caffeine
drwxr-xr-x - rawkode 11 Jul 13:34 -- dash-to-dock
drwxr-xr-x - rawkode 11 Jul 21:39 -- dash-to-panel
drwxr-xr-x - rawkode 11 Jul 13:34 -- paperwm
drwxr-xr-x - rawkode 11 Jul 13:34 -- sound-output-device-chooser
drwxr-xr-x - rawkode 11 Jul 13:34 -- topicons-plus

Step 3. Creating Our Package

I need to create a new directory with a default.nix and copy over my
example Nix from the sample extension. I started with dash-to-panel,
as I like that extension. The code for that looks like so:

{ stdenv, fetchFromGitHub, glib, gettext }:

stdenv.mkDerivation rec {
  pname = "gnome-shell-dash-to-panel";
  version = "31";

  src = fetchFromGitHub {
    owner = "home-sweet-gnome";
    repo = "dash-to-panel";
    rev = "v${version}";
    sha256 = "0vh36mdncjvfp1jbinifznj5dw3ahsswwm3m9sjw5gydsbx6vh83";
  };

  buildInputs = [
    glib gettext
  ];

  makeFlags = [ "INSTALLBASE=$(out)/share/gnome-shell/extensions" ];

  uuid = "dash-to-panel@jderose9.github.com";

  meta = with stdenv.lib; {
    description = "An icon taskbar for Gnome Shell";
    license = licenses.gpl2;
    maintainers = with maintainers; [ mounium ];
    homepage = "https://github.com/jderose9/dash-to-panel";
  };
}

Unfortunately, there was a problem. This extension uses a Makefile to document its
build steps (which I recommend for ALL repositories), but unfortunately; emoji-selector
doesn't ship with a Makefile 😢

So I'm going to copy the "core" sections and make up the rest from another example shortly.

What I copied looked like so:

{ stdenv, fetchFromGitHub, glib, gettext }:

stdenv.mkDerivation rec {
  pname = "gnome-shell-emoji-selector";
  version = "19";

  src = fetchFromGitHub {
    owner = "maoschanz";
    repo = "emoji-selector-for-gnome";
    rev = "${version}";
    sha256 = "0x60pg5nl5d73av494dg29hyfml7fbf2d03wm053vx1q8a3pxbyb";
  };

  buildInputs = [ glib ];

  meta = with stdenv.lib; {
    description =
      "This GNOME shell extension provides a searchable popup menu displaying most emojis";
    license = licenses.gpl3;
    maintainers = with maintainers; [ rawkode ];
    homepage = "https://github.com/maoschanz/emoji-selector-for-gnome";
  };
}

Lets break this down.

{ stdenv, fetchFromGitHub, glib, gettext }:

This first line of Nix is in almost every Nix file you'll work with. It's the
imports / dependencies that our Nix script needs from the environment / runtime.

I like to think of it as similiar to JavaScript's destructing syntax; selecting
only the values we need from the global list. If that's a terrible way to think
about it, I'm sure someone from HackerNews will be along shortly.

stdenv.mkDerivation rec {
  ...
}

Next up, we need to create a derivation. That's fancy functional talk for a set of values
that allow for some output to be derived.

The set that we need to build our GNOME Extension contain some obvious facts:

GitHub Repository
Version / Branch
Dependencies (Inputs)
Meta Description

  pname = "gnome-shell-emoji-selector";
  version = "19";

We define the package name and the version. The version is a tag / branch name that you can
get from the Git repository or the GitHub UI.

  src = fetchFromGitHub {
    owner = "maoschanz";
    repo = "emoji-selector-for-gnome";
    rev = "${version}";
    sha256 = "0x60pg5nl5d73av494dg29hyfml7fbf2d03wm053vx1q8a3pxbyb";
  };

fetchFromGitHub is a function that grabs some source code from GitHub. The
owner, repo, and rev are hopefully self-explanitory. However, we also have sha256.

Nix requires builds to be idempotent. That means it uses the sha to verify that
the downloaded and extracted code is what we expected. If it's different, it'll let
us know and we can decide what we want to do.

Warning

Word of warning ... if the sha that you use already exists in the Nix store (meaning you copied
it from another example that you had installed), then Nix bypasses the download step and reuses
the local files. This means that I ... you, will install some random extension instead of getting
the actual download you expected.

How Do We Get the Sha?

The simplest way is to use lots of 0's. When you try to build
this derivation, Nix will complain that your sha256 doesn't match the repositories download. You can
then copy the sha and update in your derivation.

Another approach is to calculate the actual sha yourself, using nix-prefetch-url.

Go to GitHub and grab the URL for the tar.gz artefact on the releases page for the revision
or version that you want to add to the repository. Now you can run:

nix-prefetch-url --unpack https://github.com/maoschanz/emoji-selector-for-gnome/archive/19.tar.gz

unpacking...
[1.0 MiB DL]
path is '/nix/store/694kcbsz38rni0lykffv89ndivgcccks-19.tar.gz'
0x60pg5nl5d73av494dg29hyfml7fbf2d03wm053vx1q8a3pxbyb

Dependencies

Understanding the dependencies you need to build a new Nix package can be a little intimidating.

Fortunately for GNOME Extensions, these don't vary too much and our example actually had this listed.

buildInputs = [ glib ];

If you need to work this out for another package and you don't know where to start? Follow these steps:

Clone Code
Enter directory
Create a Nix Shell (nix-shell -p depenendency1 depenendency2 ...)
Run Build Command

Repeat this, adding whatever you need to nix-shell -p until the build step works.

If you don't know what your dependency you need is called, search on the Nix Package page.

Example, if I need to build some code that needs Rust, Make, and bash - I'd run: nix-shell -p bash cargo gnumake rustc

To use our repository as an example, if I ran nix-shell with no dependencies and then ran ./install.sh - then
our install would have failed with glib-compile-schemas command not found; which is provided by glib.

Skipped Makefile

So I said that we had a problem, the problem was that our example extension used make and our
current extension doesn't. We know this because our example Nix contained:

makeFlags = [ "INSTALLBASE=$(out)/share/gnome-shell/extensions" ];

and our extension repository doesn't have a Makefile, it only has ./install.sh.

Drats.

Fortunately, I looked at another example (caffeine) and came across the following code.
It turns out that we can manually configure the build steps ourself. Sweet! 🥞

  uuid = "caffeine@patapon.info";

  nativeBuildInputs = [
    glib gettext
  ];

  buildPhase = ''
    ${bash}/bin/bash ./update-locale.sh
    glib-compile-schemas --strict --targetdir=caffeine@patapon.info/schemas/ caffeine@patapon.info/schemas
  '';

  installPhase = ''
    mkdir -p $out/share/gnome-shell/extensions
    cp -r ${uuid} $out/share/gnome-shell/extensions
  '';

Customizing the Build

In the code above, we've removed the makeFlags configuration that we found in the dash-to-panel Nix package,
because emoji-selector doesn't have a Makefile. We've instead provided buildPhase and installPhase configuration. These
two different "examples" also used slightly different inputs: buildInputs and nativeBuildInputs

Argh. What?

First, buildInputs vs nativeBuildInputs. This was tricky to track down, it's not that well documented. However,
I did find the following:

nativeBuildInputs
A list of dependencies whose host platform is the new derivation's build platform, and target platform is the new derivation's
host platform. This means a -1 host offset and 0 target offset from the new derivation's platforms. These are programs and libraries
used at build-time that, if they are a compiler or similar tool, produce code to run at run-time—i.e. tools used to build the new
derivation. If the dependency doesn't care about the target platform (i.e. isn't a compiler or similar tool), put it here, rather
than in depsBuildBuild or depsBuildTarget. This could be called depsBuildHost but nativeBuildInputs is used for historical continuity.

That's a mouthful. As I understand it, we use nativeBuildInputs when we expect the inputs to be build for the platform on our local machine;
and can use buildInputs for when we don't have such a constraint. I definitely need to understand this more and I'll write more on this
parameter soon; once I've done some more digging 😃

Next, those phases!

From the documentation; we can see that there's many phases in a build:

$prePhases unpackPhase patchPhase $preConfigurePhases configurePhase $preBuildPhases buildPhase checkPhase $preInstallPhases
installPhase fixupPhase installCheckPhase $preDistPhases distPhase $postPhases.

Each of these are called in this order, unless specifically overridden by the Nix package by providing phases = [].

We need to override the buildPhase because by default it runs make.

The default buildPhase simply calls make if a file named Makefile, makefile or GNUmakefile exists in the current directory (or the makefile is explicitly set); otherwise it does nothing.

This is also true for installPhase, only it tries to run the install target.

The default installPhase creates the directory \$out and calls make install.

It's a little less magic when you break it down; right? 🧙

Testing Our Package

Using the commands inside of ./install.sh, I was able to piece together the config
we needed for our extension to be successfully build and installed by nix.

The only "gotcha" here is that Nix provides $out variable that provides the directory
our package should install things into.

{ stdenv, fetchFromGitHub, glib, gettext }:

stdenv.mkDerivation rec {
  pname = "gnome-shell-emoji-selector";
  version = "19";

  src = fetchFromGitHub {
    owner = "maoschanz";
    repo = "emoji-selector-for-gnome";
    rev = "${version}";
    sha256 = "0x60pg5nl5d73av494dg29hyfml7fbf2d03wm053vx1q8a3pxbyb";
  };

  uuid = "emoji-selector@maestroschan.fr";

  nativeBuildInputs = [ glib ];

  buildPhase = ''
    runHooks preBuild
    glib-compile-schemas ./${uuid}/schemas
    runHooks postBuild
  '';

  installPhase = ''
    runHook preInstall
    mkdir -p $out/share/gnome-shell/extensions
    cp -r ${uuid} $out/share/gnome-shell/extensions
    runHook postInstall
  '';

  meta = with stdenv.lib; {
    description =
      "This GNOME shell extension provides a searchable popup menu displaying most emojis";
    license = licenses.gpl3Plus;
    maintainers = with maintainers; [ rawkode ];
    homepage = "https://github.com/maoschanz/emoji-selector-for-gnome";
  };
}

So how do we test this to make sure it works?

We can install it 😁

To do that, we need to cd into the nixpkgs directory; then
install our package using nix-env and the package name we declared above.

cd ~/Code/src/github.com/NixOS/nixpkgs
nix-env -f $(pwd) -i gnome-shell-emoji-selector

Assuming all goes to plan, you'll see something like:

installing 'gnome-shell-emoji-selector-19'
nix-env -f $(pwd) -i gnome-shell-emoji-selector  5.30s user 0.43s system 97% cpu 5.873 total

🎉🎉🎉 We did it! 🎉🎉🎉

Our package installed. Well done. Fire open GitHub and submit a PR.

What's Next?

There's still a fair amount to cover. When you submit a PR, you'll be presented with this checklist:

- [ ] Tested using sandboxing ([nix.useSandbox](https://nixos.org/nixos/manual/options.html#opt-nix.useSandbox) on NixOS, or option `sandbox` in [`nix.conf`](https://nixos.org/nix/manual/#sec-conf-file) on non-NixOS linux)
- Built on platform(s)
   - [x] NixOS
   - [ ] macOS
   - [ ] other Linux distributions
- [ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests))
- [ ] Tested compilation of all pkgs that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review wip"`
- [x] Tested execution of all binary files (usually in `./result/bin/`)
- [ ] Determined the impact on package closure size (by running `nix path-info -S` before and after)
- [ ] Ensured that relevant documentation is up to date
- [ ] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/.github/CONTRIBUTING.md).

We've only covered 2 measely steps. In the coming articles, we'll look at:

Sandbox
macOS Builds
Nixpkgs on Arch Linux

Until next time

Introduction to WebAssembly on Kubernetes with Krustlet

David McKay — Thu, 02 Jul 2020 22:31:16 +0000

Welcome! 👋

This article will aim to introduce you to something wild and wonderful - running WebAssembly binaries on Kubernetes, with Krustlet.

Why?

WebAssembly is bytecode that can be executed in your browser, regardless of operating system or architecture; this has opened up a, potentially,
whole new era of web programming; one that doesn't need transpiled to JavaScript.

Thus far, a few languages have trialed and tested support for compiling their code to WASM for usage within the browser. You can learn about some
of these efforts by checking out these links.

There are more, but I think that's enough to pique your interest.

ANY language, in the browser ... that's a pretty interesting idea, right? 😉

One thing we don't often talk about when discussing the browser ... it's pretty darn secure. Running JavaScript in your browser can't
actually do much damage to your host system. This is because the JavaScript engines, such as v8 or SpiderMonkey, run the JavaScript
code in a sandbox; much like modern Linux container systems, like containerd.

So what if we want to leverage WASM, in our many languages, and benefit from the sandbox out the browser ... but without the browser?

Enter, WASI and wasmtime.

WebAssembly System Interface (WASI)

WASI provides an interface for running WASM code as a process on your operating system, rather than
browsers; which allows us to interact with all the things the browser doesn't: files and filesystems, Berkeley sockets, clocks, and random numbers.

WASI does this in a fashion that still gives us similiar security boundaries that we get within the browser, but through a capabilities model defined
by the runtime. WASI is not a runtime, but the interface that runtimes must adhere too. This is handled by a capability-based security model.

How do capabilities work? Lets use an example. Most applications open a file by some reference, which is usually a file path on the filesystem. Any application
that has that reference (file path) can interact with that file, provided the Ambient Authority allows.

What does that mean? It means that if user rawkode executes a binary, and rawkode has permissions to read a file; then that binary can read that file.

That's not ideal.

WASI uses a Capability model, rather than ambient authority. When you execute a WASI binary, it can ONLY access the files that you have given that binary
the permission to.

What does that mean? Think about container images when executed, they can only access the files within the container image, or explictly bind mounted in. WASI is
the same. WASI binaries can only access files if the binary is given permission to access them, usually by providing a directory root.

wasmtime

wasmtime is an implemention of WASI. You can think of wasmtime much like node. In the same way that node allowed us to execute JavaScript outside of the browser,
wasmtime does that for WASM.

Krustlet

Krustlet is a Kubelet implementation for Kubernetes that supports running WASM bytecode, with the wasmtime runtime, instead of
container images with a container runtime.

Getting Started

Getting a Kubernetes cluster

For today's tutorial, we're going to use minikube. minikube
gives us a full Kubernetes cluster, in a VM, on our local machine. With minikube installed, create a
cluster with the following command.

minikube start

Building Krustlet

As this is rather experimental / early stages, you'll need to compile the Krustlet from source. Depending on your
machine, this can take anywhere from 3 minutes to 10 minutes; so you make want to go make a coffee or something ⏰

# Clone the Krustlet source code to a local directory and enter it
git clone https://github.com/deislabs/krustlet
cd krustlet

# Using Rust's build tool, Cargo, to build the application
cargo build

Bootstrapping

Now that we have the binaries needed to run our Krustlet, we need to perform some initial bootstrapping.

Bootstrapping is important, because Kubernetes restricts access to the control plane. We need to provide enough
configuration for our Krustlet to register itself with the control plane, so it can become a "Node" within the
cluster.

You can create this bootstrap configuration with the following command. PLEASE ensure your KUBECONFIG context is pointing to
your development cluster, and not something in production.

./docs/howto/assets/bootstrap.sh

You'll see some output, like so:

secret/bootstrap-token-x4jygy created
Switched to context "minikube".
Context "minikube" renamed to "tls-bootstrap-token-user@kubernetes".
User "tls-bootstrap-token-user" set.
Context "tls-bootstrap-token-user@kubernetes" modified.
Context "tls-bootstrap-token-user@kubernetes" modified.

What does this all mean? 🤓

First, this bootstrap script creates a secret inside of our cluster; using our default KUBECONFIG.
This will be setup for you when you spin up minikube.

This secret is very similiar to a Kubelets bootstrap secret. It's all very 😴

auth-extra-groups:                         # system:bootstrappers:kubeadm:default-node-token
expiration: MjAyMC0wNy0wMlQxMzozOTo0OVo=   # 2020-07-02T13:39:49Z (1 hour from bootstrap)
token-id: eDRqeWd5                         # x4jygy
token-secret: OWR1enFtamdvM3BlaXQ3YQ==     # 9duzqmjgo3peit7a
usage-bootstrap-authentication: dHJ1ZQ==   # true
usage-bootstrap-signing: dHJ1ZQ==          # true

Next, the bootstrap script grabs your existing KUBECONFIG, copies it, crates a new user within your cluster,
modifies the context to use that new user, and stores it at ~/.krustlet/config/bootstrap.conf. This bootstrap.conf
can now be used to provide authentication for our Krustlet to the control plane.

Running the Krustlet

KUBECONFIG=~/.krustlet/config/bootstrap.conf krustlet-wasi --node-ip $(minikube ip) --port 3000 --bootstrap-file ~/.krustlet/config/bootstrap.conf

The first time this runs, it will generate the required TLS certificates. You'll need to approve the CSR in another terminal tab / window.

kubectl certificate approve $(uname -n)-tls

Lets confirm everything looks good, we should see 2 "nodes" now.

NAME       STATUS   ROLES    AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE               KERNEL-VERSION   CONTAINER-RUNTIME
arch       Ready    agent    3m15s   0.3.0     192.168.39.37   <none>        <unknown>              <unknown>        mvp
minikube   Ready    master   57m     v1.18.3   192.168.39.37   <none>        Buildroot 2019.02.10   4.19.107         docker://19.3.8

Congratulations, you've got a Krustlet running on minikube

Building a WASI Application

Next, we're going to throw together a quick "Hello World" style application. Unfortunately, WASI hasn't defined what networking / Berkeley sockets will
look like yet within WASI-land; so we can't run a web server or anything more interesting (yet).

All in good time though 😁

TL;DR

If you don't want to follow the DIY steps below, you can clone my repository and jump straight to the Building section.

git clone https://gitlab.com/rawkode/wasi-hello-world

DIY

First, create a Rust project. This command generates a skeleton Rust project.

cargo new --bin wasi-hello-world

You'll want to update main.rs with the following code. This code prints a message,
and sleeps for 5 seconds; it then loops this process indefinitely.

// Code from
// https://github.com/deislabs/krustlet/blob/master/docs/intro/tutorial01.md
use std::thread::sleep;
use std::time::Duration;

fn main() {
    loop {
        println!("Hello, from WASI and Krustlet!");
        sleep(Duration::from_secs(5));
    }
}

Building

We need to enable the WASI target, which we'll do with rustup. We also need to build our application, targetting WASI as a runtime.

# Enable  wasm32-wasi as a compilation targeht
rustup target add wasm32-wasi

# Build WASI bytecode
cargo build --target wasm32-wasi

Running & Publishing

When we build a WASI compatible binary, what we get is a wasm file.

❯ ls target/wasm32-wasi/debug
... wasi-hello-world.wasm ...

In order to run WASI bytecode, we use a WASI runtime; like wasmtime.

❯ wasmtime ./target/wasm32-wasi/debug/wasi-hello-world.wasm
Hello, from WASI and Krustlet!
^C

Oooh, shiny! 🌠

However, that's not what we want to do. We want to run this on Kubernetes using our Krustlet.

In-order to do so, we need to "publish" our image to "somewhere".

Fortunately, there's wasm-to-oci which allows publishing wasm binaries to OCI compatible registries.

Note: Docker Hub and Quay do not support these artifacts at the time of writing, but GitLab, Google Cloud, and Azure Cloud do.

Sadly, we do need to build this ourselves too; fortunately, it's I've prepared the commands 😃

git clone https://github.com/engineerd/wasm-to-oci
cd wasm-to-oci
make
sudo cp ./bin/wasm-to-oci /usr/local/bin/wasm-to-oci

Now, lets go back to our wasi-hello-world directory and push an image with our WASI binary.

# You can use this image if you want, it's public
wasm-to-oci push ./target/wasm32-wasi/debug/wasi-hello-world.wasm registry.gitlab.com:rawkode/wasi-hello-world:latest

INFO[0008] Pushed: registry.gitlab.com/rawkode/wasi-hello-world:latest
INFO[0008] Size: 1808571
INFO[0008] Digest: sha256:a9617d29b859994cc4b6f5ea73f7d68096921f7225ed31e645f0dda7b27163ed

The Last Kilometer

OK. We're almost there. Thus far, we've:

☑ Created a Kubernetes cluster with minikube
☑ Cloned, compiled, and ran a Krustlet from source
☑ Created a Rust application and compiled it to WASI
☑ Converted our WASI binary to an OCI compatible image and pushed it to the GitLab Container Registry
❌ Run our Rust WASI targetting binary on Kubernetes

OK, so now we need to schedule our WASI workload on the Krustlet. Like ALL Kubernetes resources, we need YAML for this.

Save the following to pod.yaml.

apiVersion: v1
kind: Pod
metadata:
  name: wasi-hello-world
spec:
  containers:
    - name: wasi-hello-world
      image: registry.gitlab.com/rawkode/wasi-hello-world:latest
      imagePullPolicy: IfNotPresent
  tolerations:
    - key: "krustlet/arch"
      operator: "Equal"
      value: "wasm32-wasi"
      effect: "NoExecute"

Apply this YAML file to your Kubernetes cluster.

kubectl apply -f pod.yaml

💥 DONE 💥

We've now deployed a WASI binary, via Krustlet, to our Kubernetes cluster.

Confirm the Awesome

To confirm your WASI binary is running, you can use the Kubernetes tooling to check a few things.

First, is the pod scheduled and running and do we have log output? Run the following commands.

kubectl get pods
kubectl logs -f wasi-hello-world

Awesome! I hope this helps you on your path to understanding, getting excited, and adopting WASM, WASI, and Krustlet.

If you have any questions, grab me on Twitter.

Until next time 🤘

Tip 6: Bat, Rip, & Skim

David McKay — Wed, 08 Jan 2020 23:20:10 +0000

Consider this the "Embrace Rust" tip post. If an app can be rewritten in Rust, then someone's probably done it already.

Here are my top 3 (Besides exa, as I covered that yesterday)

Bat

Bat is a cat rewrite, in Rust. It has colours. Lots of colours.

Ripgrep

Ripgrep, as one can imagine, is a grep rewrite in Rust. Now, it's not got as many colours (Well, why would it?), but it's crazy fast.

Skim

Finally, one without a rusted name. Skim is a fzf alternative. It's good for presenting lists of content in a filterable fashion. Lets take a look at kill -9 followed by Tab

Thanks for tuning in,
Until next time 🤘

Tip 5: Replacing ls with exa

David McKay — Tue, 07 Jan 2020 13:25:53 +0000

Rust ... slowly but surely, all applications are being rewritten in Rust.

My next few tips will cover a few of these :)

Exa

Exa is an ls replacement written in Rust. As they say in their README:

with more features and better defaults

What does it look like?

Pretty shiny, eh?

Things worth noting:

Colours that distinguish directories, files, and executable files
Colours for permissions, owners, and size of file
Git status of each file in long output

Trees Everywhere

When working with code, it can be pretty nice to get a tree style look at the repository. Exa surpasses expectations here too.

First, here are a few aliases I have configured:

# ls
TREE_IGNORE="cache|log|logs|node_modules|vendor"

alias ls=' exa --group-directories-first'
alias la=' ls -a'
alias ll=' ls --git -l'
alias lt=' ls --tree -D -L 2 -I ${TREE_IGNORE}'
alias ltt=' ls --tree -D -L 3 -I ${TREE_IGNORE}'
alias lttt=' ls --tree -D -L 4 -I ${TREE_IGNORE}'
alias ltttt=' ls --tree -D -L 5 -I ${TREE_IGNORE}'

Let me explain:

--tree - Enables Tree display
-D - Show directories only (Useful for large trees)
-L n - How many levels deep to display
-I ${TREE_IGNORE} - This omits/hides common vendor directories, as defined in the variable above the aliases

Here's a short asciinema of me using the lt, ltt, and lttt aliases:

Tip 4: Git Commit Templates & Conventional Commits

David McKay — Mon, 06 Jan 2020 16:18:32 +0000

Git commit messages are important. We all know that, yet ... in the heat of the moment, we're prone to this non-sense:

git commit -m "stuff"

git commit -m "fix"

git commit -m "it'll work this time!"

Any of them look familiar? You bet your ass they do.

Adding a Git Commit Template

This is nice and easy. First, create a text file for your template inside your ~/.config/git directory (~/.git/ on macOS I think).

# ~/.config/git/commit.txt
My Commit Template

Then update your Git configuration. You can do this one of 2 ways:

git config --global commit.template ~/.config/git/commit.txt

Or you can update your configuration file:

[commit]
template=~/.config/git/commit.txt

Conventional Commits

Now that you've got a simple template. What should you put there?

Let me introduce you to Conventional Commits.

You can find my template here.

The following is the summary and some examples from the website linked above.

Conventional Commits

The Conventional Commits specification is a lightweight convention on top of commit messages. It provides an easy set of rules for creating an explicit commit history; which makes it easier to write automated tools on top of. This convention dovetails with SemVer, by describing the features, fixes, and breaking changes made in commit messages.

The commit message should be structured as follows:

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

Examples

Commit message with description and breaking change footer

feat: allow provided config object to extend other configs

BREAKING CHANGE: `extends` key in config file is now used for extending other config files

Commit message with `!` to draw attention to breaking change

refactor!: drop support for Node 6

Commit message with both `!` and BREAKING CHANGE footer

refactor!: drop support for Node 6

BREAKING CHANGE: refactor to use JavaScript features not available in Node 6.

Commit message with no body

docs: correct spelling of CHANGELOG

Commit message with scope

feat(lang): add polish language

Commit message with multi-paragraph body and multiple footers

fix: correct minor typos in code

see the issue for details

on typos fixed.

Reviewed-by: Z
Refs #133