DEV Community: rhazn

Choose names for Google, not people

rhazn — Sat, 22 Jan 2022 13:17:16 +0000

"There are only two hard things in Computer Science: cache invalidation and naming things."

- Phil Karlton

AngularJS was released in 2010 and revolutionized modern frontend development. In 2016, the AngularJS team published a new framework, written in TypeScript and incompatible with AngularJS. They also made the baffling choice to call this framework Angular 2.

Mayhem ensued, confused developers talked to each other about incompatible frameworks. The excellent community documentation AngularJS had built in the form of blogs and StackOverflow answers became an active hindrance for people trying to learn Angular. Undeterred by this, the Angular team doubled down and kept releasing incompatible framework versions, only distinguished by a number.

Five years later, the team announced that AngularJS would be discontinued. Immediately they needed to clarify that they would not discontinue Angular but only AngularJS.

Even with the best documentation, users are going to google for information. They are going to read blogs, ask on StackOverflow and copy answers. So to make it easy on them, please:

Use unique names for major versions ("Android KitKat" instead of "Android 4.4"). They are easier to include in searches.
Use error codes (like "Error 34") or slugs (like "ERROR_SOME_NAME") in addition to localized, readable messages. Unique strings enable users with different languages than English to contribute.
As a user, use Software in English. It makes it easier to google for errors ;).

This post was originally posted on my website, https://heltweg.org if you like it you might want to look at what else I write :)

Who wrote this shit?

rhazn — Mon, 10 Jan 2022 12:38:23 +0000

It is a beautiful right of passage for a bright-eyed junior developer to join a team, take some tasks full of enthusiasm, and have the life and joy sucked out of them one sprint at a time. Soon enough, they sit in planning meetings, miserably complaining, accusingly asking who wrote that shit.
Their transformation to a full team member is complete. They have become one of us.

I was once that bright-eyed junior developer. Many encounters with PHP4 later could no longer look forward to creating new features. Instead, I looked back and complained about old ones. Trashing legacy software is fun. It creates camaraderie. Who wrote this? What were they thinking? Looking at the commit log showed names that meant nothing to me. Ghosts that moved on long ago, leaving nothing but their shit code.

One day, I was pair programming with my friend Torben. Torben was a senior developer who introduced me to the team months earlier. When I found some weird code, I followed the tradition. Who was dumb enough to write this? I opened the commit log, and for the first time, I recognized a name: "Torben."

Since then, I remind myself that legacy software is written by people like me. Torben had constraints I knew nothing about. He worried about deadlines long past. Torben had learned a lot since then. Development workflows had evolved, infrastructure had gotten better.

And to all the junior developers who have complained to my face about my trash code: I forgive you, I understand.

This post was originally posted on my website, https://heltweg.org if you like it you might want to look at what else I write :)

You can reach me online at https://heltweg.org.

Wisdom from the Internet, content that influenced how I think about software (and life).

rhazn — Thu, 03 Dec 2020 14:57:55 +0000

This post was originally posted on my website, https://heltweg.org if you like it you might want to look at what else I write :)

Finding needles

There has never been as much information easily accessible to anyone as right now. The ease of publishing your own writing leads to new problems: The question is no longer where do you find content about a topic but what content is good and worth your time?

To help you find the needle in the biggest haystack ever, here is my personal list of good content that influenced me.

The list

1. "Things You Should Never Do" by Joel Spolsky

One of the main things I have learned when going from a young software engineer to working professionally for years is the value of iteration. I've personally been burned by completely rewriting projects and ever since keep this article around to remind myself of that. Also of the pain of living through the javascript framework frontend wars.

Things You Should Never Do, Part 1

2. "Choose Boring Technology" by Dan McKinley

Years ago I read a blog post titled "Why I write Java" that basically boiled down to the author writing Java and using established technologies to focus on the product and not the tech. The blog's first comment was "So you write java because you are an old, boring guy". Sadly, that gem of internet discussion seems to be lost to time. This article presents the same ideas of cautioning against new technology (but not being dogmatic about never using it) and as a fellow old, boring person now I can see myself in it.

Choose Boring Technology

3. "The Product-Minded Software Engineer" by Gergely Orosz

Job identities are hard. Am I a software engineer that wants to build an amazing algorithm? Am I a "creator" that wants to build a product? Am I a manager that wants to build a team? If you struggle to pin down what you actually want to do, this article introduces yet another category to fit your life into.

The Product-Minded Software Engineer

4. "Give it five minutes" by Jason Fried

Jason Fried and anything Basecamp are an interesting source for thoughtful content on business, software engineering and as it turns out even life advice ;). This is a really short blogpost that reminds me to keep an open mind and be humble.

Give it five minutes

5. "Are Passions Serendipitously Discovered or Painstakingly Constructed?" by Cal Newport

A short blog for people being lost in life. If you are unsure about your goal in life or why you have no passions, maybe this is something to give you ideas. Also it has a headline that literally is "Short Case Study #2: The Bored Programmer". If that does not fit the audience of this article, nothing will.

Are Passions Serendipitously Discovered or Painstakingly Constructed?

6. "Function + Feeling" by Haraldur Thorleifsson

This is a talk and not a blog but it fits the list. In a world focused on monetizing and optimizing everything, talking about predatory business models and the death of privacy this was a nice reminder that our work can have real, positive impact on people and it's worth fighting for that.

Function + Feeling

7. "Salary Negotiation: Make More Money, Be More Valued" and "Don't Call Yourself A Programmer, And Other Career Advice" by Patrick McKenzie

For software engineers just starting a career I think these blogs are an absolute must read to navigate the business side (not only of coding but for anything). Since reading these my opinions on the approach to working itself has changed quite a bit but I still consider them a solid into.

Salary Negotiation: Make More Money, Be More Valued

Don't Call Yourself A Programmer, And Other Career Advice

About Me

This post was originally posted on my website, https://heltweg.org if you like it you might want to look at what else I write :)

I am a full stack developer and digital product enthusiast, I am available for freelance work and always looking for the next exciting project :).

You can reach me online at https://heltweg.org.

Machine learning basics: Machine Learning by Stanford University (Coursera) review and notes

rhazn — Mon, 16 Mar 2020 13:46:06 +0000

This post was originally posted on my website.

The popularity of machine learning, data science and related disciplines is exploding and with it the amount of courses, books, block posts etc you are exposed to. I recently finished the relatively old but highly rated course Machine Learning by Stanford University on Coursera and wanted to take the chance to offer my review and notes I took.

The course

Although the course is old enough to be referred to as "classic" by quite a few descriptions I have read it is timeless in the sense that most good introductions are. It covers concepts from tools in ML like supervised- or unsupervised learning, tips for their application with algorithm rating and debugging and much more. Most topics are covered very timeless with the theory and math behind them being the focus.

Coursera offers video lectures, lecture notes (as pdf) and a short text summary for most of the 11 weeks of the course. Grading is done using short quizzes at the end of each topic as well as programming exercises in Octave or Mathlab. Forums are available to help with any questions and mentors are good at answering anything that comes up.

The good

The video lectures are really well done and the content seems excellent and very "grounded". It is definitely not a course trying to cash in on the hype surrounding ML.

The programming exercises are solid and introduced me well to a different side of software development that is more focussed on solving complex mathematical problems than displaying a centered div.

The forums and additional resources prepared by the mentors are very good, in every case where I had a problem it was already solved by someone else asking it in the forums.

If you don't want to get a certificate the course is completely free.

The bad

There is a noticeable drop in quality in the later weeks. While the content stays excellent the written summaries disappear, sometimes obvious double takes that should have been edited are left in videos and programming exercises have a lot of pre-written code in their descriptions.

Solving programming assignments in Octave: Probably due to the age of the course the setup of Octave is not friction free - in the version I used the "pause" function was broken which made all programming assignments hang at the first stop. It was relatively easy for me to debug and fix by overwriting the internal pause function but I've seen quite a few questions about Octave issues in the forum. In addition Octave seems to not be the primary choice in the ML community but since the course teaches concepts over concrete implementation that is fine by me.

Notes

I took a lot of notes during the course (more than 120 pages to be exact). They are mostly very close to the course material provided but sometimes rephrased and annotated with additional comments from me. In case they help anyone you can:
Download them here

About Me

I am a full stack developer and digital product enthusiast, I am available for freelance work and always looking for the next exciting project :).

You can reach me online at https://heltweg.org.

Analyzing twitter: Import tweets with NodeJS and the twitter API

rhazn — Thu, 31 Oct 2019 13:48:32 +0000

A tweet in the database is worth two in the API

Working with tweets from the twitter API probably means importing data into your own database - the standard API does not provide historical data (only the last seven days) and has various rate limits.

So regardless of the final goal in this blog we'll explore importing tweets from the API into a database for future use. All done with NodeJS, written in Typescript and utilizing MongoDB as data store.

Big numbers, big problems

Once you authenticate with the API and pull in the first tweets (for example using the twitter module on npm) you will notice tweets contain ids as numbers and "id_str" which is the same id just as string:

{
 "created_at": "Wed Oct 10 20:19:24 +0000 2018",
 "id": 1050118621198921728,
 "id_str": "1050118621198921728",
 "text": "To make room for more expression, we will now count all emojis as equal—including those with gender‍‍‍ ‍‍and skin t… https://t.co/MkGjXf9aXm",
 "user": {},  
 "entities": {}
}

The reason for this is that some languages (Javascript being one of them) can not work with big numbers. For example JS numbers are internally 64-bit floats and only use the first 53 bits for the integer value. Javascript provides the static property Number.MAX_SAFE_INTEGER as 9007199254740991 which is smaller than the id in the example tweet already.

To work with tweet ids we need a way to handle bigger numbers and use the "id_str". big.js provides that functionality and is used in all following code examples.

Saving tweets

Saving tweets in MongoDB is easy. Since we are using typescript we can rely on the excellent (Typegoose library)[https://github.com/typegoose/typegoose] to create models for tweets and interact with MongoDB:

import { prop, Typegoose, index } from "@hasezoey/typegoose";

@index({ "entities.user_mentions.screen_name": 1 })
export class TwitterStatus extends Typegoose {
    @prop({ required: true, unique: true, index: true })
    id_str!: string;

    @prop({ required: true })
    full_text!: string;

    @prop({ required: true })
    entities!: { user_mentions: { screen_name: string }[] }

    @prop({ required: true })
    created_at!: string;
}

export const TwitterStatusModel = new TwitterStatus().getModelForClass(TwitterStatus, { schemaOptions: { strict: false } });

Notice I only defined some properties I wanted to use in this model & the index is also related to my use case. You might need to change those depending on the project.

If schemaOptions define strict as false (see the last line) typegoose saves the whole JSON of the tweet in MongoDB, not just defined fields.

Import logic

To optimize the amount of tweets you can crawl from the API in the limits twitter provides an excellent resource on using the since_id and max_id parameters correctly here: https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines.

In summary this means:

set the since_id to the highest tweet id your application has already imported defining a lower bound for the imported tweets
set the max_id to the max_id from the last import and subtract 1 defining the upper bound
import tweets while setting max_id to the lowest id in the returned list until no new ones are returned, moving the upper bound closer to the lower bound
once no new tweets are returned set max_id to undefined to remove the upper bound for future imports

If you want to crawl all mentions for an account you can keep track of your crawl status with this model:

import { prop, Typegoose } from "@hasezoey/typegoose";

export class TwitterCrawlStatus extends Typegoose {
    @prop({ required: true, unique: true, lowercase: true, trim: true })
    account!: string;

    @prop({ trim: true })
    sinceId?: string;

    @prop({ trim: true })
    maxId?: string;

    @prop({ trim: true })
    overallMaxId?: string;
}

export const TwitterCrawlStatusModel = new TwitterCrawlStatus().getModelForClas(TwitterCrawlStatus);

A basic algorithm without any safeguards against failing that uses that logic and imports all mentions for a specific account follows:

    while(true) {
        const twitterCrawlStatus = await TwitterCrawlStatusModel.findOne({ account: account };

        if (!twitterCrawlStatus) {
            twitterCrawlStatus = await TwitterCrawlStatusModel.create({ account: account });
            await twitterCrawlStatus.save();
        }

        const tweets = await twitterService.getMentions(
            account,
            twitterCrawlStatus.sinceId ? Big(twitterCrawlStatus.sinceId) : undefined,
            twitterCrawlStatus.maxId ? Big(twitterCrawlStatus.maxId).minus(1) : undefined,
        );

        if (tweets.length > 0) {
            await TwitterStatusModel.bulkWrite(tweets.map(tweet => {
                return {
                    updateOne: {
                        filter: { id_str: tweet.id_str },
                        update: { $set: tweet },
                        upsert: true
                    }
                }
            }));

            const lowestId = (getLowestId(tweets) as Big);
            const highestId = (getHighestId(tweets) as Big);

            twitterCrawlStatus.maxId = lowestId.toFixed();

            if (!twitterCrawlStatus.overallMaxId || Big(twitterCrawlStatus.overallMaxId).lt(highestId)) {
                twitterCrawlStatus.overallMaxId = highestId.toFixed();
            }
        } else {
            twitterCrawlStatus.sinceId = twitterCrawlStatus.overallMaxId;
            twitterCrawlStatus.maxId = undefined;
        }

        await twitterCrawlStatus.save();

        if (tweets.length === 0) {
            break;
        }
    }

The twitter service

The twitter service itself is just a minimalist wrapper around the twitter npm module:

import * as Twitter from "twitter";
import { Status } from "twitter-d";
import Big from "big.js";

export class TwitterService {
    private client: Twitter;

    constructor(
        consumerKey: string,
        consumerSecret: string,
        bearerToken: string
    ) {
        this.client = new Twitter({
            consumer_key: consumerKey,
            consumer_secret: consumerSecret,
            bearer_token: bearerToken
        });
    }

    public async getMentions(
        account: string,
        sinceId?: Big | undefined,
        maxId?: Big | undefined
    ): Promise<Status[]> {
        return await this.client.get("search/tweets", {
            q: `@${account} -filter:retweets`,
            result_type: "recent",
            count: 100,
            include_entities: true,
            tweet_mode: "extended",
            since_id: sinceId ? sinceId.toFixed(0) : undefined,
            max_id: maxId ? maxId.toFixed(0) : undefined
        }).then(response => {
            return response.statuses;
        });
    }
}

About Me

I am a full stack developer and digital product enthusiast, I am available for freelance work and always looking for the next exciting project :).

You can reach me online at https://heltweg.org.

Deploy an app into your personal cloud

rhazn — Tue, 17 Sep 2019 11:39:47 +0000

As the final step in this long series of blogposts we are going to deploy a simple webapp in a docker container to my personal cloud. For context here is the personal cloud setup with Traefik/Let's Encrypt (Run a personal cloud with Traefik, Let's encrypt and Zookeeper).

In previous blogposts I also described how I built the app (Build a PWA in docker).

App deployment

The deployment runs the docker container. If you follow my personal cloud setup make sure to use more than one replica as the nature of preemtive VMs means one replica might randomly get shut down.

Note that the image will be set and updated by gitlab ci as described here: (Deploy to google kubernetes engine using gitlab ci).

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: ???-app
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: ???-app
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: ???-app
        image: "eu.gcr.io/???/app:latest"

The service and traefik ingress

apiVersion: v1
kind: Service
metadata:
name: ???-app
spec:
selector:
    app: ???-app
ports:
- name: web
    port: 80
    targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ???-app
annotations:
    kubernetes.io/ingress.class: traefik
    traefik.frontend.passHostHeader: "false"
    traefik.frontend.priority: "1"
spec:
rules:
- host: app.???.com
    http:
    paths:
    - path: /
        backend:
        serviceName: ???-app
        servicePort: web

Updating the traefik config

Updating the traefik config is important so traefik requests a new HTTPS certificate for the app from Let's encrypt. You will need to add this line to the traefik toml file that is described here (Run a personal cloud with Traefik, Let's encrypt and Zookeeper):

[[acme.domains]]
    main = "app.???.com"

DNS

Now you can just point the A record of your domain to the traefik external IP and the rest will automatically be handled by your personal cloud :).

About Me

I am a full stack developer and digital product enthusiast, I am available for freelance work and always looking for the next exciting project :).

You can reach me online at https://heltweg.org.

Deploy to google kubernetes engine using gitlab ci

rhazn — Wed, 11 Sep 2019 14:43:05 +0000

In a previous blogpost I showed how I build and publish docker images on gitlab ci (Build a docker image on gitlab ci)

Make sure to read that post first for an overview and permission setup.

Update the kubernetes service with the new docker image

You can easily set up a deploy step using google's own cloud SDK docker images. Note the service account with the permissions to change the kubernetes setup is saved as "GCLOUD_K8S_KEY" variable here.

This job changes the image of my deployment for the app. You will need to change the last line in the script to whatever change you want to make to your kubernetes setup on deploy.

deploy:
  stage: deploy
  image: google/cloud-sdk:257.0.0
  script:
    - echo $GCLOUD_K8S_KEY | base64 -d > ${HOME}/gcloud-k8s-key.json
    - gcloud auth activate-service-account --key-file ${HOME}/gcloud-k8s-key.json
    - gcloud config set project personal-cloud-project-id
    - gcloud config set compute/zone your-compute-zone
    - gcloud container clusters get-credentials production
    - kubectl set image deployment/???-app ???-app=eu.gcr.io/docker-project-id/app:${CI_COMMIT_SHA}
  only:
    - master
  when: manual

About Me

I am a full stack developer and digital product enthusiast, I am available for freelance work and always looking for the next exciting project :).

You can reach me online at https://heltweg.org.

Build a docker image on gitlab ci and publish it to google container registry

rhazn — Fri, 06 Sep 2019 11:13:29 +0000

In previous blogposts I explained my concept of a personal cloud for my own projects (Kubernetes for Sideprojects) and how I set it up (Run a personal cloud with Traefik, Let's encrypt and Zookeeper). I also showed how I packaged a PWA project with docker (Build a PWA in docker).

With all those ingredients ready to go the last hurdle to solve is building the docker image automatically as well as publishing it to a private container registry so I can deploy it to my cloud from there.

Overview

The goal of the setup is to:

build a docker image from any git hash using gitlab ci
push that docker image to a google container registry tagged with the git hash
update the kubernetes service description in the personal cloud project to pull the new docker image

Permissions

To enable gitlab to do these actions on our behalf we need to set up service accounts. We need

one account that allows gitlab to upload a new docker image to the container registry
an account that allows it to change our kubernetes setup in the personal cloud registry

You can create these service accounts in the "IAM & admin" -> "Service accounts" section of google cloud. Make sure to download and save the generated json file.

We will also need to allow the personal cloud project to pull the docker image from the container registry that is in a different project. For that I followed this excellent blogpost by Alexey Timanovskiy (Using single Docker repository with multiple GKE projects).

Publish a docker image with gitlab ci

To allow gitlab ci to use your service account you need to save the content of the json files as a base64 encoded variable in the backend. You can find the setting under "Settings" -> "CI /CD" -> "Variables". Be careful with this data since it is security relevant. The variables here will be available as environment variables during your jobs.

I use the following gitlab ci stage to build and publish a project. Note that it only runs manually and for master. In this case it uses the service account saved in GCLOUD_SERVICE_KEY:

publish:
  stage: publish
  image: docker:19.03.1
  services:
    - docker:dind
  variables:
    DOCKER_DRIVER: overlay
  script:
    - echo $GCLOUD_SERVICE_KEY | base64 -d > ${HOME}/gcloud-service-key.json
    - docker login -u _json_key --password-stdin https://eu.gcr.io < ${HOME}/gcloud-service-key.json
    - docker build -t eu.gcr.io/projectid/app:${CI_COMMIT_SHA} .
    - docker push "eu.gcr.io/projectid/app:${CI_COMMIT_SHA}"
  only:
    - master
  when: manual

As a last step you can update the kubernetes description of the service to deploy it. I wrote about that process here: https://heltweg.org/posts/deploy-to-google-kubernetes-engine-using-gitlab-ci/

About Me

I am a full stack developer and digital product enthusiast, I am available for freelance work and always looking for the next exciting project :).

You can reach me online at https://heltweg.org.

Build a progressive web app in docker with nginx to deploy to kubernetes or docker swarm

rhazn — Tue, 27 Aug 2019 11:22:16 +0000

An updated version of this article with current examples is available at https://www.heltweg.org/posts/build-a-progressive-web-app-in-docker-with-nginx-to-deploy-to-kubernetes-or-docker-swarm/.

The old, outdated post follows below:

With my personal cloud setup based on kubernetes done (you can read about it here: https://www.heltweg.org/posts/run-a-personal-cloud-with-traefik-lets-encrypt-and-zookeeper/) it is time to actually deploy the first project into it.

The easiest application to deploy is a pure client side single page application, packaged in a docker container with a webserver like nginx to deliver the files. Packaging the application into it's own container allows us to build a standardized container that can be run locally for testing or deployed to docker swarm and kubernetes.

Setting up and configuring our own HTTP server also allows for fine tuning of caching to achieve good lighthouse scores:

Building in docker

For this setup we build the app using docker. That way the app is always built with the same node version and can be consistently reproduced, regardless of installed software on the local computer.

The project here is a react application based on create-react-app but it works similarly with any frontend framework:

FROM node:12.6.0 AS build

WORKDIR /

COPY package.json package-lock.json tsconfig.json ./

RUN npm ci

COPY ./src ./src
COPY ./public ./public

RUN npm run build --prod

Configuring nginx

For the nginx config I placed a config file into the project and checked it in. This config file is later on copied into the container that serves the SPA. To achieve good performance we

enable gzip for HTML/CSS and JS files
set up caching for any file for one year (because create-react-app builds new file names with each production build that invalidates the cache on deploy)
disable the cache for the actual index.html file (since we need to make the browser request the newest files)
Redirect any request to index.html so the SPA router can handle them

You can see the complete config file here:

server {
    listen 80;
    server_name _;

    gzip on;
    gzip_types text/html text/css application/javascript;

    root /var/www/;
    index index.html;

    # Force all paths to load either itself (js files) or go through index.html.
    location /index.html {
        try_files $uri /index.html;

        add_header Cache-Control "no-store, no-cache, must-revalidate";    
    }

    location / {
        try_files $uri /index.html;

        expires 1y;
        add_header Cache-Control "public";
    }
}

Building the final container

The end result will be a combination of a) building the SPA in docker in the "build" step and then setting up a container from the nginx image and copying the JS from the build step as well as the checked in nginx config described above.

Finally we expose port 80 and start nginx to serve the files.

FROM node:12.6.0 AS build

WORKDIR /

COPY package.json package-lock.json tsconfig.json ./

RUN npm ci

COPY ./src ./src
COPY ./public ./public

RUN npm run build --prod

FROM nginx:1.16.1

COPY --from=build /build /var/www/
COPY ./k8s/config/nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

About Me

I am a full stack developer and digital product enthusiast, I am available for freelance work and always looking for the next exciting project :).

You can reach me online at https://heltweg.org.

Run a personal Cloud with Traefik, Let's encrypt and Zookeeper

rhazn — Sat, 17 Aug 2019 10:51:05 +0000

Kubernetes ingress with Traefik

As mentioned in my last blog post I want to focus on a provider neutral setup for my own cloud, using technology that is not bound to any cloud offering whenever possible.

While google cloud offers load balanced HTTP ingress by default it is apparently very expensive in comparison to running small nodes and I have heard only good things about using Traefik for kubernetes ingress.

For setting up Traefik I followed Manuel's excellent guide with minor modifications (you can find the final files at the end of the article.)

HTTPs and Let's encrypt

Traefik has built-in support for automatically getting and renewing HTTPS certificates with Let's Encrypt. As HTTPS is good practice and a requirement for HTTP2 and PWAs anyway I set it up using example configurations from the Traefik docs.

Because I was using just one node for Traefik I chose to go with the easy setup of a local acme.json file that stores the certificate while the node is running.

GKE Preemptible nodes, your own chaos monkey

To save costs I chose to use "Preemtible VMs" as nodes to power my kubernetes cluster on GKE. According to google's docs: "Preemptible VMs are Google Compute Engine VM instances that last a maximum of 24 hours and provide no availability guarantees." This means the nodes in my kubernetes cluster randomly go down and are never up more than 24h. While this obviously is not a smart decision for a production setup I have chosen to embrace it and consider the nodes going down my own "chaos monkey" that forces me to write resilient code.

A concrete example I ran into: The Let's encrypt production API has a rate limit of requesting 5 certificates for the same URL in a week. Because my initial naive setup did not save the certificate anywhere it got lost whenever my Traefik node was terminated. While Traefik regenerates the certificate without any issue on startup... after five startups I hit my rate limit and was greeted by an insecure warning without certificate.

Shared K/V store for Traefik with Zookeeper

Enter a shared Key/Value store for Traefik. Using one is required if you want to run Traefik in cluster mode anyway (and I like to think my setup is easily scalable). It also means I can store my generated certificate in the K/V store where it will no longer just disappear when Traefik restarts.

Since I have previous experience with Zookeeper and the setup was relatively painless I went with it.

All Kubernetes yaml files for the setup

Finally the meat of the blog post, my complete setup as yaml files you can directly deploy into your GKE cluster:

Set up Zookeeper first

From this excellent ressource: https://github.com/kow3ns/kubernetes-zookeeper/blob/master/manifests/README.md

apiVersion: v1
kind: Service
metadata:
  name: zk-hs
  labels:
    app: zk
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  clusterIP: None
  selector:
    app: zk
---
apiVersion: v1
kind: Service
metadata:
  name: zk-cs
  labels:
    app: zk
spec:
  ports:
  - port: 2181
    name: client
  selector:
    app: zk
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: zk
spec:
  serviceName: zk-hs
  replicas: 1
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: zk
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - zk
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: kubernetes-zookeeper
        imagePullPolicy: Always
        image: "gcr.io/google_containers/kubernetes-zookeeper:1.0-3.4.10"
        resources:
          requests:
            memory: "200M"
            cpu: "0.3"
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        command:
        - sh
        - -c
        - "start-zookeeper \
          --servers=1 \
          --data_dir=/var/lib/zookeeper/data \
          --data_log_dir=/var/lib/zookeeper/data/log \
          --conf_dir=/opt/zookeeper/conf \
          --client_port=2181 \
          --election_port=3888 \
          --server_port=2888 \
          --tick_time=2000 \
          --init_limit=10 \
          --sync_limit=5 \
          --heap=512M \
          --max_client_cnxns=60 \
          --snap_retain_count=3 \
          --purge_interval=12 \
          --max_session_timeout=40000 \
          --min_session_timeout=4000 \
          --log_level=INFO"
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        volumeMounts:
        - name: datadir
          mountPath: /var/lib/zookeeper
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

Permissions for Traefik

# create Traefik cluster role
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
 name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
# create Traefik service account
kind: ServiceAccount
apiVersion: v1
metadata:
  name: traefik-ingress-controller
  namespace: default
---
# bind role with service account
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: default

Traefik config

Note the configuration of zookeeper using the service address for the "client service" (cs) as well as the Let's encrypt config here.

# define Traefik configuration
kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-config
data:
  traefik.toml: |
    # traefik.toml
    defaultEntryPoints = ["http", "https"]
    [entryPoints]
      [entryPoints.http]
        address = ":80"
        [entryPoints.http.redirect]
          entryPoint = "https"
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]

      [zookeeper]
        endpoint = "zk-cs.default.svc.cluster.local:2181"
        watch = true
        prefix = "traefik"

      [acme]
      email = "your@email.com"
      storage = "traefik/acme/account"
      onHostRule = true
      caServer = "https://acme-v02.api.letsencrypt.org/directory"
      acmeLogging = true
      entryPoint = "https"
        [acme.httpChallenge]
        entryPoint = "http"

      [[acme.domains]]
        main = "your.domain.com"

Deployment for Traefik

I run just one replica in here to save costs in my dev setup but I've also scaled it up to three to test if it would stay up 100% of the time even with random nodes going down and everything works fine :).

# declare Traefik deployment
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: traefik-ingress-controller
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      volumes:
        - name: config
          configMap:
            name: traefik-config
      containers:
      - name: traefik
        image: "traefik:1.7.14"
        volumeMounts:
          - mountPath: "/etc/traefik/config"
            name: config
        args:
        - --configfile=/etc/traefik/config/traefik.toml
        - --kubernetes
        - --logLevel=INFO

Traefik service

# Declare Traefik ingress service
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-controller
spec:
  selector:
    app: traefik-ingress-controller
  ports:
    - port: 80
      name: http
    - port: 443
      name: tls
  type: LoadBalancer

Final result

The final workloads with traefik and zookeeper

And the kubernetes ingresses (ignore the app I used as demo for this)

About Me

I am a full stack developer and digital product enthusiast, I am available for freelance work and always looking for the next exciting project :).

You can reach me online at https://heltweg.org.

Kubernetes for sideprojects: Hardware is dead

rhazn — Sat, 17 Aug 2019 10:25:39 +0000

Why invest in a personal cloud

It has never been easier to host your personal side projects. Tools like surge.sh or Heroku make it painless to run your code. And if all else fails the old reliable "drag and drop files to a ftp" is always there - so why invest time into setting up your own personal cloud with kubernetes?

My goal for technology is typically to find a setup that gets boring to work with because I know it well and can focus on delivering new functionality. For that a setup needs to be future proof (so it continues to work for a long time), generic (so I can use it for a wide range of applications and do not need to switch for every project) and not bound to any company or product.

Docker & kubernetes

Docker and Kubernetes check most of these boxes. Kubernetes has de-facto won the orchestration war for containerized applications and managed kubernetes offerings from all major cloud providers means there is no provider lock-in. As an open source project that is not bound to individual company or setup you retain flexibility. Lastly learning more about it is useful in any case - if you stop developing your side projects the devops knowledge you gained still looks good on a CV.

A further important point that separates kubernetes from similar offerings is that you can 100% forget about hardware while still being free to use standard technology. That means you do not need to maintain a set of servers if you use a managed kubernetes offering (like you would need to with docker swarm) but you are still not locked into any provider (like you would be by using AWS Beanstalk, Firebase or Heroku).

Why not...?

My own personal journey has made me try out the following alternatives in the order they are written down here and I have abandoned them all.

Own servers
- You need to maintain and update your own servers. If you do it without automation you will forget what you did.
- The complexity and learning curve is close to kubernetes anyway. Instead of learning about deployments and stateful sets you will need to know about processes or package managers.
- Scaling is harder.
Services (e.g. Firebase)
- Very easy to set up and use, probably advisable if you work on one or a few projects
- Using vendor specific tools (like firebase realtime database) locks you and the logic of your app into that vendor. What if they change their offering (price? functionality?)
- Services are the hammer that makes very problem look like a nail. Instead of picking the best technology for a job you will start trying to shoehorn your problems to be solved by what is available.
Docker Swarm
- I really liked it, very close to kubernetes but considerably easier
- Sadly still forces you to manage your own servers to set up the swarm cluster, I could not find a "managed docker swarm" solution (if you know one, let me know!)

The value of infrastructure as code

By choosing kubernetes you also commit to keeping your infrastructure in code which has a plethora of benefits:

it automatically and correctly documents your infrastructure and any changes you made by reading git logs
you can easily redeploy it on new providers or e.g. locally for development
all your infrastructure is in one place so you don't need to think about how each project solved a problem

Organizing your projects

For my personal cloud I chose:

A Google Cloud Kubernetes Engine Cluster as cloud
- Traefik as ingress router to forward requests to my projects
- Zookeeper for shared state between Traefik nodes
- Let's Encrypt to automatically set up HTTPS
Every project has it's own Google Cloud Container Registry as a private docker repository
Service accounts allow the GKE cloud to pull the docker images of individual projects from the respective repository
Infrastructure descriptions for the generic GKE setup and services is in one place, infrastructure descriptions for the individual projects is in their respective code
Deployment is handled with gitlab

About Me

I am a full stack developer and digital product enthusiast, I am available for freelance work and always looking for the next exciting project :).

You can reach me online at https://heltweg.org.

Entity-Systems for typescript based games

rhazn — Fri, 16 Aug 2019 09:23:04 +0000

Entity-Systems for typescript based games

This post is also available on my blog where I plan to write more about game development with typescript if you are interested :).

For my latest game project Frozzen I want to explore how an external UI, build with Angular, would work for a browser based game. Since Angular is written in Typescript that means ideally the game should also use the same.

I have used Artemis ODB as framework for a Java based game in the past and liked it a lot. Entity-Systems are much better introduced by any of the huge amount of articles out there (for example the classic on T=Machine but I feel they are especially well suited to Javascript/Typescript development.

If you work with a strict separation of logic into systems and data only into components there is a very natural way to serialize components, JSON. Whole levels can be expressed as an array of JSON data that is used to set up components. That is why I prefer a very basic but strict implementation like artemis over similar frameworks like PhaserJS.

I started my development with artemists, a Typescript port of artemis by darkoverlordofdata. Unfortunately the code is a bit outdated and does not use import/export and can not be directly imported for newer Typescript versions (since it extends the built-in Array).

With darkoverlordofdata’s permission I did a quick update to the Typescript parts of the code only, adding import/export support and fixing the build for newer Typescript versions. You can find the updated version here. If you are looking for an example of that framework in action you can play an example level of Frozzen here.

About Me

I am a full stack developer and digital product enthusiast, I am available for freelance work and always looking for the next exciting project :).

You can reach me online at https://heltweg.org.

DEV Community: rhazn

Choose names for Google, not people

Who wrote this shit?

Wisdom from the Internet, content that influenced how I think about software (and life).

Finding needles

The list

1. "Things You Should Never Do" by Joel Spolsky

2. "Choose Boring Technology" by Dan McKinley

3. "The Product-Minded Software Engineer" by Gergely Orosz

4. "Give it five minutes" by Jason Fried

5. "Are Passions Serendipitously Discovered or Painstakingly Constructed?" by Cal Newport

6. "Function + Feeling" by Haraldur Thorleifsson

7. "Salary Negotiation: Make More Money, Be More Valued" and "Don't Call Yourself A Programmer, And Other Career Advice" by Patrick McKenzie

About Me

Machine learning basics: Machine Learning by Stanford University (Coursera) review and notes

The course

The good

The bad

Notes

About Me

Analyzing twitter: Import tweets with NodeJS and the twitter API

A tweet in the database is worth two in the API

Big numbers, big problems

Saving tweets

Import logic

The twitter service

About Me

Deploy an app into your personal cloud

App deployment

The service and traefik ingress

Updating the traefik config

DNS

About Me

Deploy to google kubernetes engine using gitlab ci

Update the kubernetes service with the new docker image

About Me

Build a docker image on gitlab ci and publish it to google container registry

Overview

Permissions

Publish a docker image with gitlab ci

About Me

Build a progressive web app in docker with nginx to deploy to kubernetes or docker swarm

Building in docker

Configuring nginx

Building the final container

About Me

Run a personal Cloud with Traefik, Let's encrypt and Zookeeper

Kubernetes ingress with Traefik

HTTPs and Let's encrypt

GKE Preemptible nodes, your own chaos monkey

Shared K/V store for Traefik with Zookeeper

All Kubernetes yaml files for the setup

Set up Zookeeper first

Permissions for Traefik

Traefik config

Deployment for Traefik

Traefik service

Final result

About Me

Kubernetes for sideprojects: Hardware is dead

Why invest in a personal cloud

Docker & kubernetes

Why not...?

The value of infrastructure as code

Organizing your projects

Further reading

About Me

Entity-Systems for typescript based games

Entity-Systems for typescript based games

About Me