DEV Community: Anthony Wales

Breaking News! Startup develops stronger architectural foundation by gathering requirements first!

Anthony Wales — Thu, 07 Nov 2019 21:50:22 +0000

The start of any project is exciting, and shirtctl is no different. Last month Blair and I announced we’re working a brand new project to bring continuous delivery to tech tees.

We set out with a goal to help tech brands create and ship cool t-shirts reliably to their fans at any time. 👕👚

From our experience, we know just how easy it is to get carried away with building technology solutions before we even understand our requirements.

This article is a summary of the principles we developed throughout first #shirtday hack, and how they informed certain architecture decisions. And since we’re building shirtctl in the open, expect to see more posts about how we implement these as our project develops.

Our Mission:

We make it super easy for companies to build their brand and connect with the community by harnessing the awesome power of t-shirts.

How: Our MVP: For shirtctl to work, we need to be able to join brands and fans together for the reliable transfer of merchandise.

With our MVP defined, we had everything we need to consider what high level platform architecture decisions we would need to make to build a strong technology foundation.

Requirement Principles:

    * [Flexibility](#Flexibility)
    * [Relentless Focus](#Relentless-Focus)
    * [Security in Depth](#Secure)
    * [Frugal by Design](#Frugal-Architecture)
    * [Easy to Consume](#Easy-to-Consume)

Flexibility

We are going to make mistakes. Our path, like any startup, is unclear. Recognising that we are only a small team, we have to live and breath the fail-fast approach to bring shirtctl to life as soon as possible.

If we need to change directions, we need to be able to do so quickly without having to rebuild from scratch.

Image Source

Adopting Architectures:
Hybrid Cloud Architecture provides:

Rapid deployment.
Horizontally scalable.
Heavily automated.
Standardisation.
Minimum commitment (some services are billed per 100ms).

Continuous Delivery Architecture (Continuous Integration with Continuous Delivery):

Short software deployment cycles releasing software with greater speed and frequency.
Incremental updates for patching and feature enhancement.
Software release governance.

Relentless Focus

We know our main goal is to connect tech brands and their fans, not to host and manage infrastructure. We don’t have time to do the Design, Build and Run of a new infrastructure and platform service. In addition, the management overhead of a single tenancy wouldn't be time or cost effective. We are trying to shift as much as possible to managed services so we can worry just about shirtctl and leave the rest to the to the infrastructure experts.

Image Source

Adopting Architectures:
Managed Cloud Architecture:

Responsibility abstraction (just worry about the code, 'Serverless').
Consumption based billing.
Pre-architected building blocks.

Serverless Architecture provides; outsourcing as much responsibility as possible so we can just worry about the code.

Security in Depth

With cyberattacks on the rise, and increasing consumer awareness of the risks of personal information being exposed, we need a modern approach to security that is simple and effective. We want to keep shirtctl free of traditional approaches to security that are prone to failure and costly to design, build and manage.

Image Source

Adopting Architectures:
Zero Trust Security Model:

An alternative security model to not trust anything remediating the rudimentary fault in traditional strategies to only protect from the outside. In this architecture, every pillar is verified.
All communication paths need to be encrypted especially for micro-services.
Validation and authentication are performed between all services.
Low cost with minimal special tooling required.

Frugal by Design

As a super interesting side-project, shirtctl has no funding to splurge on unnecessary expenses. Just like a 1990’s startup using the garage as an office, we’re using modern architectures for their opportunities to save big on cost and time. As a public cloud consumer, we’re capitalising on the opportunities that cloud presents to pay for only what we consume.

We don’t expect to ever have to invest big in infrastructure development projects to enable our data lake or deployment pipeline, and just like we see the biggest disruptors changing their markets, we’re hoping it works for tech tees too.

Image Source

Adopting Architectures:
Public Cloud Architectures:

Consumption based billing with some products such as serverless allowing you to only pay for the CPU/RAM cycles used to run the code not the OS and below.
Free tier pricing for certain services to help get you started allowing for experimentation with different technologies ensuring you pick the best one.
Startup credits to be used on any service allowing you to try services that have costs associated.

Easy to Consume

With our small team and non-existant budget, we need shirtctl to be usable by as many people as possible. So we’re starting out with a web app at https://shirtctl.com. We think you’ll like it when ready. A lot.

Image Source

Adopting Architectures:
Web App Architecture:

Availability to be accessed over the internet.
Compatibility with many different platforms and devices.
Very low time to get your product into the hands of your potential consumers.

Kubernetes (K8s) Private Cloud with Raspberry Pi 4s

Anthony Wales — Mon, 23 Sep 2019 23:43:32 +0000

Deploying a Kubernetes cluster on a public cloud provider is easy, but what if you want a private bare-metal deployment? This walk-through will go through the steps I took (and why we need to do them) in order to have a private Kubernetes cluster at home.

Kubernetes enables you to have the flexibility to move your workload where it is best suited. The most common reasons I hear are; latency, performance (special hardware requirements) and security (regulation or data governance). This compliments the hybrid cloud story and in my career it has become more apparent that my customers see this as well to help them resolve issues like; cost, availability and compliance. In parallel software vendors are starting to embrace containers as a standard deployment model leading to a recent increase in requests for container solutions.

As you can see in the workflow comparison below, there is greater room for error when deploying on-premises. Public clouds provide the automation and reduces the risk of error as less steps are required. But as mentioned above, private cloud provides you more options when you have unique requirements.

Workflow comparison for deploying Kubernetes on or off premises

To help people understand some of the pitfalls and challenges with on-premises Kubernetes/Container deployments, below is a guide to help you build your own.

"A cloud you can touch!"

What you will need

Compute:

4 x Raspberry Pi 4 - 2GB RAM [2GB+ RAM is recommended as we are deploying K8s not K3s]

Storage:

4 x 32GB High Speed Sand-disk Micro-SD Cards

Network:

4 x Raspberry Pi POE HAT [Optional if you don't want to provide USB power to the Raspberry Pi]
1 x Network Switch [POE Capability is only required if using Raspberry Pi POE HAT]
1 x Network Router
5 x Ethernet Cables
1 x Keyboard, HDMI, Mouse (for initial setup only)

Other:

DNS Server [Optional if you want to provide round robin infrastructure resiliency]

Walk Through

Initial Raspberry Pi Configuration

Flash Raspbian
Refer to the Raspbian Installation guide by Raspberry Pi Foundation
https://www.raspberrypi.org/documentation/installation/installing-images/README.md
In short:
- Download image from the link above
- Use balenaEtcha to flash image onto micro-SD card
Perform Initial Setup on Boot on startup screen (this will require keyboard, video and mouse to be connected)

Source: RaspberryPi.org

Choose Country, Language, Timezone
Define new password for user 'pi'
Connect to WiFi or skip if using ethernet
Skip update software (this caused my Raspberry Pi to hang, not sure if there's currently a bug. We will perform this activity manually later).
Choose restart later
- Configure Additional Settings Click the Raspberry Pi icon (top left of screen) > Preferences > Raspberry Pi Configuration
System
- Configure Hostname
- Boot: To CLI
Interfaces
- SSH: Enable
Choose restart later
- Configure Static Network Perform one of the following:
Define Static IP on Raspberry Pi: Right Click the arrow logo top right of screen and select 'Wireless & Wired Network Settings'
Define Static IP on DHCP Server: Configure your DHCP server to define a static IP on the Raspberry Pi Mac Address.
- Reboot and Test SSH
Username: pi
Password: Defined in step 2 above
On Terminal: ssh pi@[IP Address]

Repeat steps for all of the Raspberry Pis.

Kubernetes Cluster Preparation (via SSH)

Perform Updates
- apt-get update: Updates the package indexes
- apt-get upgrade: Performs the upgrades

  sudo apt-get update
  sudo apt-get upgrade
  sudo reboot

Configure Net.IP4.IP configuration Edit sudo vi /etc/sysctl.conf, uncomment net.ipv4.ip_forward = 1 and add net.ipv4.ip_nonlocal_bind=1.
- Note: This is required to allow for traffic forwarding, for example Node Ports from containers to/from non-cluster devices.

sudo reboot

Install Docker

curl -sSL get.docker.com | sh && sudo usermod pi -aG docker

Disable Swap
- You can verify this before/after reboot with the top command, on the top left corner next to MiB Swap should be 0.0.

sudo systemctl disable dphys-swapfile.service
sudo reboot

SSH Session Screen after `top` command

Install Kubernetes
- Currently forcing the previous version (1.15.3), ran into compatibility issues with the most recent version (1.16).
- There shouldn't be any errors, however during my installation the repos were down and I had to retry in a few hours.

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | \
sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | \
sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q

sudo apt-get install -qy kubelet=1.15.3-00 kubectl=1.15.3-00 kubeadm=1.15.3-00

Repeat steps for all of the Raspberry Pis.

Kubernetes Master Node Configuration

Note: You only need to do this for the master node (in this deployment I recommend only 1 master node). Each Raspberry Pi is a node.

Initiate Master Node

sudo kubeadm init

Enable Connections to Port 8080
- Without this Kubernetes services won't work

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Add Container Network Interface (CNI)
- I've chosen to use Weaver, however you can get others working such as Flannel (I've verified this works with this cluster)

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Get Join Command
- This will be used in the next section to join the worker nodes to the cluster. It will return something like: kubeadm join 192.168.0.101:6443 --token X.Y --discovery-token-ca-cert-hash sha256:XYZ

kubeadm token create --print-join-command

Kubernetes Worker Node Configuration

Note: You only need to do this for the worker nodes (in this deployment I recommend 3 worker node).

Join Cluster
- Use the join command provided at the end of the previous section

sudo kubeadm join 192.168.0.101:6443 --token X.Y \
--discovery-token-ca-cert-hash sha256:XYZ 
#Example Only

Verify Node Added Successfully (SSH on Master Node)
- Should have status ready after ~30 seconds

kubectl get nodes

First Deployment and Service

Note: We will perform the deployment via SSH on the Master Node. Below are two ways to deploy the deployment and service; using either YAML or single line commands. YAML allows for easier complex actions while single line commands can be used for simple actions but for this example it will be the same outcome.

Deploy NGINX (Option A: Simple)

kubectl create deployment nginx --image=nginx

Deploy NGINX (Option B: YAML)
- kubectl apply -f nginx.yaml

#nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx

Testing NGINX

kubectl get deployments

List of running deployments

Deploy Node Port Service (Option A: Simple)

kubectl create service nodeport nginx --tcp=80:80

Deploy Node Port Service (Option B: YAML)
- kubectl apply -f nginxservice.yaml

#nginxservice.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      nodePort: 32000

Testing Node Port Service

kubectl get services

List of running services

Testing Nginx (from your laptop or anything with connectivity to the cluster)

curl [IP address of any node (RPi)]:[Port by the Node Port]

Result of curl request.

DNS A Round Robin

As a form of load balancer, I chose to use DNS A Record Round Robbin.

Added entries for a shared hostname and pointed it to all the IP addresses of the nodes.
- Note: The CNI creates a VXLAN network that allows all hosts to redirect to the container which hosts the container via the defined Node Port.

A Few Lessons Learnt

Require sandpits for testing
- Without proper testing prior to deploying, updating any of the container components might fail. I saw this with the incompatibility of Kubernetes v1.16 with Weaver (hence the v1.15.3 install).
Auxiliary/Supporting Services requires additional effort
- Public clouds provide ready to go managed supporting services such as load balancing, DNS, container registries, user authentication etc.
Additional work arounds required
- Not just one click deployment, workarounds maybe required depending on the infrastructure.