DEV Community: Matt Thompson

The Tacit Knowledge Series

Matt Thompson — Wed, 15 Sep 2021 08:04:23 +0000

From a personal and professional standpoint I'm interested in what is required to keep engineers moving from being experienced and competent to being expert.

Many views point to tacit knowledge. But by its very nature tacit knowledge is hard to define and even harder to share.

Shawn Wang describes the problem perfectly in his excellent book The Coding Career Handbook...

Much of your learning from Junior to Senior involves gaining tacit knowledge. You can read all the programming books in the world, but, by definition, you are still limited to things that people can write down. That is explicit knowledge, and it is only the tip of the iceberg.

We see tacit knowledge in action every day - when a doctor is able to make a prognosis during a crazy short appointment, or, hopefully more commonly, when an expert engineer looks at a design for all of 5 seconds and can immediately tell you her concerns or questions.

In trying to understand how they reached these conclusions so quickly, explanations might start with some principles, then move on to the exceptions and caveats to these principles, and may often ends with something like "it just feels right". It's these natural judgements that are made that can't be simply explained so that others could reach the exact same outcome.

For an excellent breakdown of this, with detailed examples in software engineering and other knowledge fields, I highly recommend Cedric Chin's Commonplace article - Tacit Knowledge is a Real Thing.

So I want to tap into the tacit knowledge of those around me. I know I can't replace gaining the experience that will build this knowledge myself, but I hope that by being in tune with it I can see the situations, ask the questions, and hopefully understand the thought processes behind these judgements.

My plan is to pick up on scenarios where experienced engineers might have taken a path through a situation or problem that wasn't obvious to me, pin them down to walk through how they approached it; what experience, principles and ideas they drew on, and then write it down and share it here...

The Tacit Knowledge Series so far

Originally posted on https://logical-progression.matttea.com

Adding a Custom Domain to a GitLab Pages Site

Matt Thompson — Mon, 31 Aug 2020 11:48:20 +0000

My logical-progression blog site is built using the Hugo static site generator and is hosted on GitLab Pages. With GitLab Pages custom domains are possible, and I was keen to use the logical-progression alias.

Initially, however, I couldn't remember who my DNS host was! A problem, as resource records need to be added there to enable the alias. The problem was based on the fact that...

The matttea.com domain was purchased via Google Domains
My webhosting was with Interserver

My thinking was that I needed to add the relevant CNAME and TXT resource records to the Google Domains dashboard (as it was the clearest dashboard to work with), and by doing so I was prompted to use the Google domain nameservers. Doing this broke all my other sites, highlighting that Google obviously wasn't my DNS host!

Realising that Interserver therefore was my DNS host I created the CNAME and TXT records in the correct place, and reset the nameservers (so that my other sites worked again)

The CNAME or 'canonical name' record maps the alias name to the true or canonical domain name.
The TXT record is required to verify ownership of the domain, giving a gitlab verification code.

The place to add these records on the Interserver site is via the cPanel > Domains > Zone Editor, and then by clicking the MANAGE option for the domain you want to add a subdomain alias to. In this case matttea.com.

As DNS changes can take anything from a few minutes to a few days to propagate, it can be useful to use sites such as whatsmydns.net to check in on whether your changes have worked.

GitLab Pages also offers free certificate management with Let's Encrypt, allowing you to enable HTTPS for alias names. This can be switched on in the Pages settings. This can be flaky though and I received a few Something went wrong while obtaining the Let's Encrypt certificate error messages. Without this, browsers will give 'not secure' messages to visitors, so it's worth getting it working. The troubleshooting guide suggests the following...

Go to your project's Settings > Pages.
Click Edit on your domain.
Click Retry.

This didn't work for me, so the guide continues with...

Make sure you have properly set only one CNAME or A DNS record for your domain. ✔️
Make sure your domain doesn't have an AAAA DNS record. ✔️
If you have a CAA DNS record for your domain or any higher level domains, make sure it includes letsencrypt.org. ✔️
Make sure your domain is verified. ✔️

Turns out I just needed to retry on the first part quite a good few times.

The final step to set up the new alias is to change the baseUrl in the Hugo config to the new domain, this enable CSS on the page amongst other things.

(Originally posted on https://logical-progression.matttea.com)

The Joy of People Management

Matt Thompson — Tue, 25 Aug 2020 19:28:05 +0000

We can certainly be forgiven for fearing the worst and focusing on the negatives right now - a global pandemic that shows no signs of disappearing, Brexit just around the corner, and strategic reviews all around us at work - but being, or becoming a people manager shouldn't be one of them.

So, with all the change going on right now I'd like to share a few, maybe less obvious reasons why people management has been an overwhelmingly positive experience throughout my career.

Time saving

This may sound completely counter-intuitive, but being a manager saves me time.

Being a people manager, even to only a handful of people, immediately gives you access to a network that extends into each of their product areas and domains, and gives you an extremely candid insight into the personalities, the decisions, the challenges and most importantly the work in each of those areas.

For the last 18 months I've been fortunate enough to manage anywhere between 16 and 22 engineers across almost as many product teams, and far from this taking up all my time, it's provided me with so much insight into what's going on around our business that I rarely have to do any digging to determine who to speak to, or what the current goals or priorities are for a specific area.

A great recent example of this was when my own product team was trying to align priorities and roadmaps with others in our wider 'Findability' product domain, those of us in my People Management team who work in this domain were quickly able to get together and openly share our thoughts (and dare I say, our secrets!) and get to a great level of understanding far easier than we'd otherwise have been able to do.

This has proved even more valuable during lockdown and this 'new (ab)normal', because I now no longer have the luxury of walking around an area to catch conversations or people.

Learning

I make no joke when I say I have learnt infinitely more from the people I've managed than they have from me.

Those whom I've managed have always been and will always be far better developers, analysts and engineers than I'll ever dream of being, but through working with, and coaching people to be even better I've been able to more clearly focus on areas for my own improvement. When working with other skilled professionals and working through their objectives and targets, you quickly become very attuned to what good looks like, and this is a huge benefit for your own career development.

At an individual skill level, you're surrounded by and regularly meeting the people in your team who are used to showing you what they can do and showing off their skills. It's far from a one way street, and I am equally coached on my problems and challenges by my team. There is no better example here than my own continuing journey in backend web development.

A safe place to try things

The teams I've worked with have been wonderful at letting me try things out, they've not only been a safe space to get things wrong, they've also been amazingly honest at giving feedback and ideas - in many respects you represent your team, so they don't want you to look bad!

My team have been on the receiving end of draft versions of pretty much everything I've delivered at Practice meetings or external conferences, and have always been completely on point with the feedback they've given - thank you!

People Management and individual team contribution are not mutually exclusive

The last point I'd like to share is that I've never felt that being a people manager and being a contributor in a team have ever been mutually exclusive parts of my job. I have many examples of where people management conversations have helped me unblock stuff in my team assignments and vice versa. My delivery team (hopefully!) benefit from having me as a person, the combined engineer and people manager that I am, and from me bringing all my skills to the team.

(Originally posted on https://logical-progression.matttea.com)

Containerising and Deploying with k8s

Matt Thompson — Sat, 25 Jul 2020 15:55:18 +0000

This started less an article or blog, more a journal of my many failed attempts at containerising and deploying a small app. I was keen to practice with the technologies I use every day, but without the guardrails of a fantastic platform team with all the tools and support they provide.

The following describes the (eventual!) successful steps to...

Package a Kotlin web application with environment secrets in a Docker image
Upload the image to a container registry, and run both manually and as part of a CI build pipeline
Create a Kubernetes (k8s) cluster and deploy the app, both manually and as part of the CI build pipeline
Expose the application to the internet

The code for the Tempo-Valence application used can be seen here

1. Package app in a docker image

Google Kubernetes Engine (GKE) accepts Docker images as the application deployment format.
Using gcr.io container registry (after failing with other registry providers - see out takes below!)
- Enable Cloud Registry API in Google Cloud Platform (GCP) console
- Set up Service Account in GCP Service Accounts - IAM & Admin so that this can be used to access the Google container registry (gcr). This service account will need to have a Storage Admin role
- Download key from created GCP service account and add it to a gitlab-ci variable - called GOOGLE_CREDENTIALS in my example, which is accessed in the gitlab-ci.yml file below
- (The key can also be saved to a local, non-committed keyfile somewhere if you wish to run the jib docker image locally to check)
- Update build step of gitlab-ci.yml to...

build:
    <<: *gradle-image
    stage: build
    variables:
        CLIENT_KEY: $CLIENT_KEY
        GOOGLE_CREDENTIALS: $GOOGLE_CREDENTIALS
    before_script:
        - echo "$GOOGLE_CREDENTIALS" > keyfile.json
    script:
        - export CLIENT_KEY="$CLIENT_KEY"
        - gradle jib -Djib.console=plain
        - gradle clean build
    only:
        - master

Add jib config in build.gradle.kts of the project as follows...

jib {
    container {
        mainClass = application.mainClassName
        environment = mapOf(Pair("CLIENT_KEY", System.getenv("CLIENT_KEY")))
        ports = listOf("8000")
    }
    from {
        image = "openjdk:11"
    }
    to {
        image = "gcr.io/<gcp-project-name>/tempo-valence:latest"
        auth {
            username = "_json_key"
            password = file("keyfile.json").readText()
        }
    }
}

(The CLIENT_KEY environment variable above refers to a secret env var required to access an external api from the application. The value of this env var is stored as a Variable in Gitlab CI so the pipeline can access to run the app.)

2. Upload docker image to a container registry

See previous step
To run locally...

docker run --rm -p 9000:9000 gcr.io/<gcp-project-name>/tempo-valence:latest

# in new terminal tab
curl http://localhost:9090
# Welcome to TempoValence!

3. Create a k8s container cluster

Following manual commands create cluster, deploy application and expose on an external IP

gcloud auth list

gcloud config set account m.........@gmail.com

gcloud config set project <gcp-project-name>

gcloud config set compute/zone europe-west2-a

gcloud container clusters create tempo-valence-cluster

gcloud compute instances list

kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest

kubectl get deployment

kubectl get pods

kubectl scale deployment tempo-valence --replicas=2

kubectl autoscale deployment tempo-valence --cpu-percent=80 --min=1 --max=5

kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
# the ports here matter - think it has something to do with how we've set up the application in the image

kubectl get service

# and use the EXTERNAL_IP of tempo-valence-service to access, e.g...
http://35.234.150.122:9000/

4. Deploy

See step 3 for manual deployment
In order to deploy from the gitlab-ci pipeline, the gitlab-ci.yml looked like this for the deploy stage...

deploy:
stage: deploy
variables:
    CI_SERVICE_ACCOUNT: $CI_SERVICE_ACCOUNT
image:
    name: kiwigrid/gcloud-kubectl-helm
before_script:
    - echo "$CI_SERVICE_ACCOUNT" > key.json
    - gcloud auth activate-service-account --key-file=key.json
    - gcloud config set project <gcp-project-name>
    - gcloud config set container/cluster tempo-valence-cluster
    - gcloud config set compute/zone europe-west2-a
    - gcloud container clusters get-credentials tempo-valence-cluster --zone europe-west2-a
script:
    - kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest
    - kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
    # Wait for the deployment to be applied
    - kubectl rollout status deployment tempo-valence --watch

Few things to note here!

Set up another service account in GCP, to be used by gitlab ci to authenticate against both the container registry and k8s cluster (may be possible just to amend the roles of previous service account, but I added a new one to keep it separate)
- Give the service account 2 project roles -> Kubernetes Engine Developer and Storage Object Admin
- Download the key associated to the account and add in to a new CI variable in gitlab
The image is the image used to run the deployment - bit of trial and error here, as this image needed to have both kubectl and gcloud cli tools loaded (and I couldn't be bothered to create one and store it somewhere that didn't need authentication!) The image included above is open and does the job.
For this to work, the tempo-valence-cluster of course needs to be up and running

Steps to run

As this will not be continually running here are the steps needed before this will be available on the internet

To setup

gcloud container clusters create tempo-valence-cluster

# Run ci pipeline, or...
kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest

kubectl scale deployment tempo-valence --replicas=2

kubectl autoscale deployment tempo-valence --cpu-percent=80 --min=1 --max=5

kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000

To teardown

kubectl delete service tempo-valence-service

gcloud container clusters delete tempo-valence-cluster

Links and article credits

A lot of ideas from this Google k8s tutorial
And this GCP tutorial
One of our own from Matt Dowds on Kubernetes deployment form Gitlab CI

Outtakes - my many failed attempts at step 1!

Package app in a docker image

Google Kubernetes Engine (GKE) accepts Docker images as the application deployment format. To build a Docker image, you need to have an application and a Dockerfile (in that application)
Add Dockerfile to root of project. 1st attempt looked like this...

# use base image
FROM openjdk:11

# set working directory
WORKDIR /

# copy the runlocal.sh file (contains env vars and gradle run command)
COPY runlocal.sh .

# Inform Docker that the container is listening on the specified port at runtime
EXPOSE 8080

#Run the specified command within the container
RUN ./runlocal.sh

# Copy the rest of your app source code from your host to your image filesystem
COPY . .

Run the following command to build and test docker image locally...

docker image build -t tempo-valence:1.0 .

Sadly this exited in error...

Step 5/6 : RUN ./runlocal.sh
---> Running in 7d1ddc474a6f
./runlocal.sh: line 5: ./gradlew: No such file or directory
The command '/bin/sh -c ./runlocal.sh' returned a non-zero code: 127

So next I tried working with a gradle base image...

FROM gradle:5.4.1-jdk11 AS build

Gradle started, but sadly a config error was returned...

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project ''.

Then looked in to --env-file and using other ways of getting the relevant client key into the image without exposing it. Finally added the env_var to a .env file and accessed in Dockerfile as follows...

FROM openjdk:11

WORKDIR /

EXPOSE 8080

ENV CLIENT_KEY .env

COPY . .

This successfully builds and tags the image, but my concern is now that this .env file is only available locally, as it is .gitignore'd

Running the image as a container using

docker container run --publish 8000:8080 --detach --name tv tempo-valence:1.0

# --publish 8000:8080 asks Docker to forward traffic incoming on the host’s port 8000, to the container’s port 8080. Containers have their own private set of ports, so if you want to reach one from the network, you have to forward traffic to it in this way. Otherwise, firewall rules will prevent all network traffic from reaching your container, as default

# --detach asks Docker to run this container in the background

# --name specifies a name with which you can refer to your container in subsequent commands, in this case tv

didn't run the app, as we have no current run command in the Dockerfile

So, added a couple of bits back in the Dockerfile...

FROM gradle:5.4.1-jdk11

WORKDIR /

EXPOSE 8080

ENV CLIENT_KEY $CLIENT_KEY

CMD [ "gradle", "run" ]

COPY . .

This successfully runs the container, but I cannot access the app via the browser on http://localhost:8000 as expected

Some links being used so far...

Some useful commands...

docker exec tv env (lists env vars set in the container)

New attempts... (13.03.20)

Tried using jib to containerise the app
Added the following into build.gradle file...

plugins {
    id 'com.google.cloud.tools.jib' version '2.1.0'
}

jib {
    container {
        mainClass = application.mainClassName
        environment = [CLIENT_KEY:System.getenv("CLIENT_KEY")]
        ports = ['8000']
    }
    from {
        image = "openjdk:11"
    }
    to {
        image = "matttea/matttea-images/tempo-valence:latest"
    }
}

Run the following commands to build...

export CLIENT_KEY=<client_key_value>

gradle jib

Returned error related to authentication...

Containerizing application to matttea/matttea-images/tempo-valence...
Base image 'openjdk:11' does not use a specific image digest - build may not be reproducible
The credential helper (docker-credential-desktop) has nothing for server URL: registry-1.docker.io

Got output:

credentials not found in native keychain

...

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Build image failed, perhaps you should make sure your credentials for 'registry-1.docker.io/matttea/matttea-images/tempo-valence' are set up correctly. See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-should-i-do-when-the-registry-responds-with-unauthorized for help

Using this link
See also this journal entry on docker credentials helper

New attempts... (25.05.2020)

tried changing job image to detail to image = "https://index.docker.io/v1//matttea/matttea-images/tempo-valence:latest" based on docker logout command url response
didn't work...

> Task :jib FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jib'.
> Invalid image reference https://index.docker.io/v1//matttea/matttea-images/tempo-valence:latest, perhaps you should check that the reference is formatted correctly according to https://docs.docker.com/engine/reference/commandline/tag/#extended-description
For example, slash-separated name components cannot have uppercase letters

(Originally posted on https://logical-progression.matttea.com)