DEV Community: Justin

Benchmarking & diagraming my home network

Justin — Sun, 02 Apr 2023 20:33:05 +0000

Every once and a while I have to call my ISP and renegotiate my internet speed.

Before this last call, I was at 300/300 Mbps, which I thought was quite fast and ridiculous.

I call or chat online when I start seeing offers for new customers well exceeding the performance or beneath the price I'm paying.

I've learned to expect they will not give you the same price as new customers, as they are introductory offers and only last 1 year or so. But using that as leverage can spark up some pretty good deals.

So I was paying $90 for 300/300, I saw new customers were offered $45 for 1G/1G.

Popped in a chat with my ISP and now I'm at 2G/2G for $90.

Now with 2G/2G, I'm going back and reevaluating my network and seeing if I need to address any issues.

Thus, this network diagram was born. It's still slightly a WIP, I want to better document the actual ISP provided equipment and connections, but I'm going out to see the new D&D movie with any luck, so this will have to do for now.

One thing I've found interesting is my M1 MBP is not performing as well as my desktop that is running in nearly the same configuration. I might need to check cables there. Happy to see the desktop is reaching nearly gig speeds at least.

Now to ponder about upgrading the internal network to support 2.5G (:sad-face:).

I didn't think I'd be here doing this. I thought 1 Gig Fiber was out of reach even but it seems things are changing quickly.

If you haven't reached out to your ISP lately, I'd highly suggest doing so.

🥇 The ultimate kubernetes homelab setup

Justin — Sun, 13 Nov 2022 15:28:05 +0000

Overview

Want an easy way to deploy and control kubernetes in a totally sealed homelab environment?

Let's get started with k3, gitlab, proxmox, ansible and terraform to automate our homelab infrastructure for repeatability and expandability.

Sections

Some of the sections we're going to cover for build out:

proxmox
terraform
ansible
k3
- dns
- test deploys
gitlab
sshportal
persistant volumes

Proxmox

The baseline for this project is going to be a proxmox server. We could run k3 bare metal, but proxmox gives us the ability to practice spinning up and down k8 nodes of different sizes/restraints via terraform. Plus if anything goes wrong, we just destroy a proxmox vm and start fresh, which happens in minutes rather then having to reinstall an entire OS on baremetal.

We're going to need/want to

Setup a cloud-init image to be our base image for terraform
Network a remote cluster via tailscale (vpn) [optional redundancy]

So with proxmox, there's not much to it other then to follow the installation instructions: https://www.proxmox.com/en/proxmox-ve/get-started

Download ISO image
Boot from USB or CD/DVD
Configure the host machine (make sure you can login web ui)

Once you have it so you can log into a web ui with something like



https://[your_ip_here]:8006/

You can decide if you want to setup the remote cluster, one day I'll document how to network the two with tailscale.

Cloud init

With proxmox setup, we need setup a base image to be our "golden image" for terraform to use to deploy new nodes etc.

For that, I'd suggest following a tutorial like this:
https://www.youtube.com/watch?v=shiIi38cJe4

https://docs.technotim.live/posts/cloud-init-cloud-image/

I followed the tutorial pretty closely and even used the suggested image id of 8000. So you'll see that being used in the next terraform section.

Cloud init gives us the ability to spin up a base image with predefined username/password, accepted ssh keys, and ip configuration. As well as gives us the ability to override those settings using the terraform provider.

So with all that, let's jump into the terraform section.

Terraform

So now that our local has it's own "cloud", i.e. we have proxmox instead of AWS or GCP for our "unlimited" VMs, we need a way to orchestrate them.

Part of the big switch to kubernetes is, it enables us to not care about the underlying infrastructure. For example, on AWS, if we have a kubernetes node that runs out of diskspace, we can expect aws to launch a new node and kubernetes move the current workload to the new node.

Terraform is what will enable us to recreate some of that underlying node scalability via proxmox.

Ultimately we'll have something like var.node_count = 3, let's say we want to deploy more pods but we've reached the max kubernetes will deploy to our current 3 cluster setup. We can change to var.node_count = 5 and let terraform handle rolling that out across proxmox and ensure everything is kept uniform.

Terraform gives the ability to easily create and manage a grouping of different size nodes for experimenting with changing workloads or setting certain node affinity on kubernetes.

Some of the groups I've made so far are like the following:



variable "prxmx_api_nodes" {
  default = 1
}

variable "prxmx_worker_nodes" {
  default = 3
}

variable "prxmx_worker_xl_nodes" {
  default = 1
}

variable "prxmx_worker_xxl_nodes" {
  default = 1
}
...

That enables the following types of workgroups that scale to certain resources:

name	cores	ram	hd space	hd type
worker_nodes	4c	4G	30G	hdd
worker_xl_nodes	12c	12G	80G	hdd
worker_ssd_xl_nodes	12c	12G	80G	ssd
worker_xxl_nodes	24c	24G	160G	hdd
worker_ssd_xxl_nodes	24c	24G	160G	ssd

In my setup, I have "xeon" 2 socket, 32 core proxmox server, 96gb ram, and 12 core intel i7, 16gb ram "river" server.

Terraform has to know to deploy these different VMs across both river and xeon proxmox servers, but once those nodes get added to kubernetes (k8s), the implementation is abstracted from k8s. K8s will deploy across any/all available nodes.

But having a good selection of different types of nodes really helps it feel like a full replacement for AWS. And even if you don't use all the different workgroups, having them available but scaled back definitely helps for organization and scalability once it's needed.

Most of my workload might run on worker_xl_nodes which that workgroup can exist across both my river and xeon servers, so if I need to scale down my xeon server, kubernetes can migrate everything to the river server for maintenance.

But some applications need lots of cores and ram that river doesn't have, so without having another large xeon like server, scaling down my worker_xxl_nodes is going to result in downtime.

Ansible

One of the other things we'll also do here is use ansible to setup each of our nodes to our liking via provisioner "local-exec"



resource "proxmox_vm_qemu" "api_nodes" {
  count       = var.prxmx_api_nodes
  name        = "k3-api-${count.index + 1}"
  desc        = "k3-api-${count.index + 1}"
  target_node = "xeon"

  clone = "ubuntu-cloud"

  os_type      = "cloud-init"
  ipconfig0    = "ip=10.0.4.${count.index + 1}/16,gw=10.0.2.1"
  nameserver   = "1.1.1.1"
  searchdomain = "1.1.1.1"

  cores        = 4
  memory       = "4096"

  disk {
    storage = var.disk
    type    = "scsi"
    size    = "30G"
  }

  lifecycle {
    ignore_changes = [
      ciuser,
      sshkeys,
      network,
    ]
  }

  provisioner "local-exec" {
    command = "sleep 30 && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -u knox -i '10.0.4.${count.index + 1},' playbook.yml"
  }
}

What this means is, after each successful setup of our proxmox vm, we'll run a ansible playbook to install the tool to our liking.

Whether that's k3 itself, or neovim and exporting the editor to use neovim on each VM. Ansible enables that.

K3

So with our hypervisor (proxmox) chosen, vm layout designed (terraform + ansible), we're ready to start on the kubernetes cluster, which is the thing that will handle our actual workloads/applications.

For this iteration, I'm using k3 for its total ease of use.

Using terraform, I deploy one api server (high-availability to come later) with terraform something like



# cat main.tf

terraform {
  required_providers {
    proxmox = {
      source  = "Telmate/proxmox"
      version = "2.9.11"
    }
  }
}

provider "proxmox" {
  pm_api_url      = "https://10.0.2.3:8006/api2/json"
  pm_tls_insecure = true
}

variable "prxmx_api_nodes" {
  default = 1
}

variable "disk" {
  default = "hdd-12tb"
}

resource "proxmox_vm_qemu" "api_nodes" {
  count       = var.prxmx_api_nodes
  name        = "k3-api-${count.index + 1}"
  desc        = "k3-api-${count.index + 1}"
  target_node = "xeon"

  clone = "ubuntu-cloud"

  os_type      = "cloud-init"
  ipconfig0    = "ip=10.0.4.${count.index + 1}/16,gw=10.0.2.1"
  nameserver   = "1.1.1.1"
  searchdomain = "1.1.1.1"

  cores        = 4
  memory       = "4096"

  disk {
    storage = var.disk
    type    = "scsi"
    size    = "30G"
  }

  lifecycle {
    ignore_changes = [
      ciuser,
      sshkeys,
      network,
    ]
  }

  provisioner "local-exec" {
    command = "sleep 30 && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -u knox -i '10.0.4.${count.index + 1},' playbook.yml"
  }
}

And a playbook like



# cat playbook.yml

---

- name: Install k3
  gather_facts: true
  hosts: all
  tasks:
    - name: Install k3
      shell: curl -sfL https://get.k3s.io | sh -

Once it's deployed, in my case it'll deploy it to a static ip 10.0.4.1, so I'll ssh to it, and grab the k3 token to add future nodes to the k3 cluster.



# cat /var/lib/rancher/k3s/server/token
K100.......

So now I spin up some worker_xl_nodes, once they're finished, I ssh in and run



curl -sfL https://get.k3s.io | K3S_URL=https://10.0.4.1:6443 K3S_TOKEN='K100.....' sh -

Once that's finished and I have the workers I need, can verify with



# kubectl get nodes

NAME              STATUS   ROLES                  AGE   VERSION
k3-worker-xxl-1   Ready    <none>                 12h   v1.25.3+k3s1
k3-worker-xl-1    Ready    <none>                 12h   v1.25.3+k3s1
k3-worker-3       Ready    <none>                 14h   v1.25.3+k3s1
k3-worker-2       Ready    <none>                 14h   v1.25.3+k3s1
k3-api-1          Ready    control-plane,master   14h   v1.25.3+k3s1
k3-worker-1       Ready    <none>                 14h   v1.25.3+k3s1

DNS

Now one other thing is, I setup a DNS A record to point



*.kube.reaz.io -> 10.0.4.1

Because I use tailscale for everything, I have no need for exposing the IP to the outside world.

Test deploys

One other thing I'll often use is this little deploy.sh helper script for quickly slinging out an docker image to an entire k8s deployment.



# cat deploy.sh

#!/bin/bash

image=$1
name=$2
port=$3

kubectl create deployment $name --image=$image
kubectl expose deployment $name --port=80 --target-port=$port --name $name --type=LoadBalancer
kubectl create ingress $name --rule="$name.kube.reaz.io/*=$name:80"

The usage is like



## Example: sh deploy.sh [image] [name] [port]

# sh deploy.sh linuxserver/nginx nginx 80

# curl nginx.kube.reaz.io
    <html>
        <head>
            <title>Welcome to our server</title>
...

Gitlab

With that in place, we're ready to deploy a test gitlab env. I like to use gitlab as a self-hosted/self-contained git for my local stuff that I don't want on the public web (mainly for things that aren't serious, just a nice playground).

But I do like keeping things in source control and using pipelines where possible.

So using the deploy script earlier, we'll just use the quick and easy docker image for right now and until we get persistent volumes setup, we'll just be careful to not put anything in there we aren't afraid to lose (in the event kubernetes reschedules the pods, without persist storage, everything in the gitlab container will be lost -- we'll setup storage later, for now lets just get to use more of our cluster).



# sh deploy.sh gitlab/gitlab-ee:latest gitlab 80

Once that finishes up after about 5-10 minutes, I'm able to access it at https://gitlab.kube.reaz.io but I'm greeted with a login prompt.

Here are the steps to track down the initial root password the image creates:



# kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
gitlab-54955459cb-52wr4      1/1     Running   0          12h

# kubectl exec -it gitlab-54955459cb-52wr4 -- bash

# cat /etc/gitlab/initial_root_password

With that in place, we can start to put some of our work into gitlab and eventually start to build some pipelines, just remember anything in there will be destroyed until we setup persistent storage.

There are lots of different ways to host gitlab or even use gitlab.com, but for this case, I really wanted to say I'm hosting everything in k8s.

The downside I've run into with that though is ssh isn't splittable the way http and ingress is. I.e. you can't just route ssh traffic from gitlab.kube.reaz.io using an ingress controller. Or at least I couldn't find a way with k3's traefik ingress controller for the 15mins I looked. You can get fancy with haproxy. I found a different way.

sshportal

sshportal (https://github.com/moul/sshportal) is in my mind, a ssh gateway. It's a little cumbersome to setup but it accomplishes routing ssh the way we'll want within a kubernetes cluster with replication.

I use the kubernetes cli tools as starting point for my yaml:



# Create a deployment a pod
# kubectl create deployment --image moul/sshportal sshportal

# Create a service
# kubectl expose deployment sshportal --port=2222 --target-port=2222 --name sshportal --type=LoadBalancer

So next, I'm going to rework my api node to replace it's normal sshd server with this new sshportal service we just deployed to k8s.

So these next couple modifications need to go on to live in our ansible playbook for the api-nodes, but here is the run down:



# On the api node, gonna move the sshd port off of 22

vi /etc/ssh/sshd_config

# Change


Port 22
---

Port 2233


# Next we're going to change the k3 server to allow us

# to set a nodePort for ssh

vi /etc/systemd/system/k3s.service

# Change



ExecStart=/usr/local/bin/k3s </span>

   server </span>
---


ExecStart=/usr/local/bin/k3s </span>

   server </span>

   --kube-apiserver-arg service-node-port-range=10-32767


# Then reload k3s

systemctl daemon-reload

systemctl restart k3s

# Then once k3 restarts,

# we're going to edit the sshportal service

kubectl edit svc sshportal

# And change the nodeport to be 22


- nodePort: 31840
---

- nodePort: 22


# From there you can grab the invite code from the logs

kubectl logs deployment/sshportal

# 2022/11/13 11:21:21 info: system migrated

# 2022/11/13 11:21:21 info 'admin' user created, use the user 'invite:DU6oTsAy8E1vQd9d' to associate a public key with this account

# 2022/11/13 11:21:21 info: SSH Server accepting connections on :2222, idle-timout=0s

# With that you should be able to ssh to the api-node 

# with the invite code and setup the 'admin' account

ssh 10.0.4.1 -l invite:DU6oTsAy8E1vQd9d

# With the admin account, you can start to create ssh jumps

ssh 10.0.4.1 -l admin

#

#    __________ _____           __       

#   / _/ _/ // / _ _  _/ /_ / /

#  _\ _\ \/ _  / _/ _ \/ _/ / _ '/ /

# //////_/   _//  _/_,//

#

#

# config> host ls

#   ID | NAME |           URL            |        KEY        |  GROUPS  |   UPDATED   |   CREATED   | COMMENT | HOP |  # LOGGING

# -----+------+--------------------------+-------------------+---------+-------------+-------------+---------+-----+-------------

#   1 | knox | ssh://knox@10.0.4.1:2233 | nervous_goldstine | default | 3 hours ago | 4 hours ago |         |     | everything

# Total: 1 hosts.

# Then for me, I can use

ssh knox@10.0.4.1

# Which will jump from sshportal to ssh://knox@10.0.4.1:2233

# Then I can add gitlab's ssh://git@gitlab

# And wahla! A single ssh endpoint that I can use for multiple different services.

# Just don't forget to go back and

# setup mysql replication and/or 

# occasionally backup the configs

ssh admin@10.0.4.1 config backup > ~/sshportal/backup.json

ssh admin@10.0.4.1 config restore ~/.sshportal/backup.json --confirm

Persistent Volumes

What good is all this kubernetes setup if nothing persists?

Ideally everything would be ephemeral as possible. But somethings like databases need file storage. So let's set that up.

[ coming next ]

🔧 Using git with a local ssh server

Justin — Sun, 13 Nov 2022 15:06:16 +0000

Let's say you're ripping on a project and maybe have two computers at home or you don't feel like sharing the project you're working on to github/gitlab just yet.

You can use git + ssh to keep those two projects in sync and its super easy.

This is also a good way to work projects across computers without having to rsync or scp files.

Assuming you have a git project and a ssh connection already, you are pretty much all set. Just follow these instructions:

# on the remote server copy down what git you do have _or_
# create a new folder/git to sync it to
ssh user@host "git init --bare /mnt/foo/bar/my-project.git"

# then add that remote server
git remote add origin ssh://user@host/mnt/foo/bar/my-project.git

# and finally, push to it!
git push origin master

Keep in mind, if you already have an origin, you can rename that to something like "local-server", then you have a "git push local-server" for local testing or "git push origin" for when its time to really deploy.

Happy coding!

📦 Set up minio w/ Laravel (Local S3)

Justin — Sun, 13 Nov 2022 15:02:57 +0000

Example docker-compose.yml for development

version: "3.9"
services:
  web:
    build:
      context: .
      dockerfile: Dockerfile.web
    volumes:
      - ./:/app
    ports:
      - "8000:8000"
  storage:
      hostname: minio
      image: quay.io/minio/minio
      command: server /data --console-address :9001
      environment:
          MINIO_ROOT_USER: minio
          MINIO_ROOT_PASSWORD: minio123
      healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
      volumes:
          - ./storage/data:/data
      ports:
          - "9005:9000"
          - "9006:9001"

Now with that in place, setup a .env

AWS_ACCESS_KEY_ID=minio
AWS_SECRET_ACCESS_KEY=minio123
AWS_DEFAULT_REGION=us-east-1
AWS_BUCKET=testbucket
AWS_USE_PATH_STYLE_ENDPOINT=true
AWS_ENDPOINT=http://minio:9000

And that is it! You are ready to rock. Just remember to set

FILESYSTEM_DRIVER=s3

And let it eat!

Route::get('/',function(){
    return Storage::files();
});

Stick around and later we'll learn how to upload files there!

Getting started with twirp 🦜

Justin — Mon, 18 Apr 2022 11:51:43 +0000

Twirp is a library that has all the strictly typed power of gRPC but retains the flexibility of HTTP/JSON for communication.

You define your routes, requests, and responses in one *.proto protobuf file.

Then all consumers of your API can import the protobuf file and generate their own client library in their language of choice.

Let's set up a quick and example.

First things first, you'll need to install protobuf, so on mac, start with

brew install protobuf

With that installed, you'll need to make sure you have golang setup as well as GOBIN defined; These are some reasonable defaults to add to your .zshrc:

GOBIN="~/go/bin"
PATH="$PATH:~/go/bin"

Next you'll need to install the protobuf binary plugins to generate the files for both twirp and golang;

go install github.com/twitchtv/twirp/protoc-gen-twirp@latest
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

I'm building a project called "ProxyMe" so I'll use it as an example; Let's start with a simple protobuf file to define the routes I'll start with;

syntax = "proto3";
package reaz.io.proxyme;
option go_package = "reaz.io/proxyme";

service ProxyMe {
  rpc ListProxies(ListReq) returns (ListRes);
}

message ListReq {
  string subject = 1;
}

message ListRes {
  string text = 1;
}

What's important to note is;

# We'll need this later, this is just a namespace for my projects, yours might be more like '{username}.com.{project_name}'
package reaz.io.proxyme;

Then the rest of the file goes onto describe a single ListProxies endpoint that accepts a ListReq (List request) and returns a ListReq List Response.

Now let's use twirp and protobuf to automatically scaffold some code for us;

protoc --go_out=. --twirp_out=. ProxyMe.proto

At this point, we'll have something like

├── ProxyMe.proto
└── reaz.io
    └── proxyme
        ├── ProxyMe.pb.go
        └── ProxyMe.twirp.go

2 directories, 3 files

The original file we wrote, ProxyMe.proto, and the namespace we used pointing to a .pb.go which is the go generated portion and *.twirp.go is the twirp magic that contains a HTTP service.

With that, let's make a server.go to actually implement the endpoint:

package main

import (
    "context"
    "log"
    "net/http"

    pb "proxyme/reaz.io/proxyme"
)

type ProxyMeServer struct{}

func (s *ProxyMeServer) ListProxies(ctx context.Context, req *pb.ListReq) (*pb.ListRes, error) {
    return &pb.ListRes{Text: "Hello " + req.Subject}, nil
}

const PORT = "8011"

// Run the implementation in a local server
func main() {
    twirpHandler := pb.NewProxyMeServer(&ProxyMeServer{})
    // You can use any mux you like - NewHelloWorldServer gives you an http.Handler.
    mux := http.NewServeMux()
    // The generated code includes a method, PathPrefix(), which
    // can be used to mount your service on a mux.
    mux.Handle(twirpHandler.PathPrefix(), twirpHandler)
    log.Println("Listening on http://0.0.0.0:" + PORT + twirpHandler.PathPrefix())
    err := http.ListenAndServe(":"+PORT, mux)
    if err != nil {
        log.Fatal(err)
        return
    }
}

That's it, we're mostly importing our generated code, adding a ListProxies that uses our ListReq, ListRes.

Now to run it with the following:

$ go run server.go
2022/04/18 07:07:16 Listening on http://0.0.0.0:8011/twirp/reaz.io.proxyme.ProxyMe/

With it running, we can test it using the following

curl --request POST \
  --url http://0.0.0.0:8011/twirp/reaz.io.proxyme.ProxyMe/ListProxies \
  --header 'Content-Type: application/json' \
  --data '{
    "subject": "mike"
}'

# Should see the following for a response
{
  "text": "Hello mike"
}

You can now just generate client code in other languages and use the new service!

Clients in other languages can also be generated by using the respective protoc plugins defined by their languages, for example --twirp_ruby_out.

And can use protobuf or json, all generated from one protobuf file!

Learn more about how to generate client code here.

🪖 Keycloak instance on docker

Justin — Sat, 18 Dec 2021 14:23:25 +0000

Stand up a keycloak instance w/ username: admin, password: pa55word. With postgres as a database. And SSL forwarding (for use with like cloudflare).

Create this docker-compose file i.e. docker-compose.yml



version: '3'

volumes:
  postgres_data:
      driver: local

services:
  postgres:
      image: postgres
      volumes:
        - postgres_data:/var/lib/postgresql/data
      environment:
        POSTGRES_DB: keycloak
        POSTGRES_USER: keycloak
        POSTGRES_PASSWORD: password
  keycloak:
      image: quay.io/keycloak/keycloak:latest
      environment:
        DB_VENDOR: POSTGRES
        DB_ADDR: postgres
        DB_DATABASE: keycloak
        DB_USER: keycloak
        DB_SCHEMA: public
        DB_PASSWORD: password
        KEYCLOAK_USER: admin
        KEYCLOAK_PASSWORD: Pa55w0rd
        PROXY_ADDRESS_FORWARDING: 'true'
        # Uncomment the line below if you want to specify JDBC parameters. The parameter below is just an example, and it shouldn't be used in production without knowledge. It is highly recommended that you read the PostgreSQL JDBC driver documentation in order to use it.
        #JDBC_PARAMS: "ssl=true"
      ports:
        - 8010:8080
      depends_on:
        - postgres

Then run docker-compose up -d.

With that in place, you just need to

Make a new realm
Make a new client
Add a valid redirect uri to the client (from the application you are wishing to secure)

Then you can add the following type of logic for authentication on your browser facing application



var keycloak;
function init() {

    keycloak = new Keycloak({
        url: 'https://[key-cloak-instance]/auth',
        realm: '[your-realm]',
        clientId: '[your-client]'
    });

    keycloak.init({
        checkLoginIframe: false
    }).success(async function (auth) {
        if (auth) {
            var name;
            if (keycloak.tokenParsed['family_name'] || keycloak.tokenParsed['given_name']) {
                name = keycloak.tokenParsed['given_name'] + ' ' + keycloak.tokenParsed['family_name']
            } else {
                name = keycloak.tokenParsed.preferred_username;
            }
            hide('login')

            // Set axios header for auth
            axios.defaults.headers.common['Authorization'] = `Bearer ${keycloak.token}`;
        } else {
            console.info('Not Authenticated');
            window.location = await keycloak.login();
        }
    })
}

function show(id) {
    document.getElementById(id).classList.remove('hide');
    document.getElementById(id).classList.add('show');
}

function hide(id) {
    document.getElementById(id).classList.add('hide');
    document.getElementById(id).classList.remove('show');
}

window.onhashchange = init;
window.onload = init;

Then on your server (nodejs/express in this case), add the following logic:



const jwt = require('express-jwt');
const { data: response } = await axios.get('https://[keycloak-instance]/auth/realms/[your-realm]');
const secret = `-----BEGIN PUBLIC KEY-----\r\n${response.public_key}\r\n-----END PUBLIC KEY-----`;
const jwtMiddleware = jwt({ secret, algorithms: ['RS256'] });

And then you can either add the middleware globally



app.use(jwtMiddleware)

Or add it to a specific route



app.post('/api/endpoint', jwtMiddleware, async (req, res) => {
  return 'something-secure';
});

Now good luck to those pesky people that were previously trying to storm your castle!

🐳 Debugging Docker Engine Connection Issues on MacOS

Justin — Mon, 13 Dec 2021 12:22:40 +0000

So I was working on adding an insecure local registry to push images to and I managed to bomb my local docker installation.

The Docker desktop app wouldn't pull up at all and I couldn't find any logs or even find docker running with something like

ps aux | grep docker

And I kept getting this error;

$ docker ps

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

So after enough googling, I found the place the logs were stored was around ~/Library/Containers/com.docker.docker/Data/log/, it was this log specifically that lead me to the right place:

cat ~/Library/Containers/com.docker.docker/Data/log/vm/docker.log

This showed:

...
2021-12-12T16:45:46Z docker time="2021-12-12T16:45:46Z" level=error msg="(5c94f517) 0ff224fd-DockerdPKG C<-S a2f49a7f-VMAPI GET /engine/daemon.json (9.946583ms): while parsing JSON from /Users/justin/.docker/daemon.json: invalid character '\"' after object key:value pair"
...

Ah! I made a typo when updated my daemon.json to allow an insecure repository! After patching that up, I needed a way to restart docker:

killall Docker && open /Applications/Docker.app

After that, it was all ready to go! Hopefully it helps you track down any issues for you!

🚰 Docker connection reset during builds

Justin — Sat, 27 Nov 2021 17:31:16 +0000

Recently while trying to deploy a test docker container on my homelab's docker instance, I kept running into an error similar to this one

Caused by: java.net.SocketException: Connection reset
 at java.net.SocketInputStream.read(SocketInputStream.java:210)
 at java.net.SocketInputStream.read(SocketInputStream.java:141)
 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)

After a bunch of digging, I ran into this medium article; https://medium.com/swlh/fix-a-random-network-connection-reset-issue-in-docker-kubernetes-5c57a11de170

It summarized it's findings pretty well and walked through the whole discovery process. Something I appreciate.

Ultimately for it, it boiled down to running these commands to backup, flush, and reset iptables

# Keep a backup of the current rules incase we have to go back
iptables-save > /root/firewall.rules

# Flush all the current rules
iptables -F
iptables -X
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT

# Confirm the rules are now empty
iptables -L

Then I had to restart docker with service docker restart for docker to reset it's own iptable rules. But from then on, no more connection reset issues.

In the event you need to roll back, you can use

iptables-resoter < /root/firewall.rules

Now get back to building something awesome!

🖥 Using vim as a laravel/phpstorm developer

Justin — Sat, 03 Jul 2021 13:45:02 +0000

These are my quick notes on how to start seriously using vim as a laravel developer. My goal is to replace phpstorm w/ vim and match the feature set. For what I'm doing now, microservices in laravel/php, phpstorm is struggling to keep up.

My apologies they're a bit of a mess as they're undergoing, but wanted to get them out there if anyone has some feedback or suggestions I'd love to hear it!

# Using vim as a laravel developer
Ok, using vim as a phpstorm replacment, this is prettyyyy close.

Here are the steps and the useful parts to know.

# First things first, install awesome-vim, this gets 90% of the stuff you need and its maintained
git clone --depth=1 https://github.com/amix/vimrc.git ~/.vim_runtime
sh ~/.vim_runtime/install_awesome_vimrc.sh

# Next I'd say you're going to want phpunit baked in and here is an awesome plugin for that:
git clone https://github.com/c9s/phpunit.vim.git ~/.vim_runtime/my_plugins/phpunit.vim

# Run inside vim: -- this will let you click and drag panes to resize or jump between (learning vim more and being able to jump panes
# makes this unecessary, but i'm not that good yet.)
:set mouse=a

# So now some sections on easy key commands/shortcuts

## Splitting VIM screen Horizontally and Vertically
To open a new VIM window next to the existing one, press <Ctrl>+<w> then press <v>.

## Move panes around vim (left/right or top/bottom)
Ctrl w + L - Move the current window to the "far right"
Ctrl w + H - Move the current window to the "far left"
Ctrl w + J - Move the current window to the "very bottom"
Ctrl w + K - Move the current window to the "very top"

## Copying everything into clipboard
gg"*yG

## Indenting all the code
# Still need to look into a more serious formatter like:
https://github.com/vim-autoformat/vim-autoformat
gg=G

# AwesomeVIM Leader Key Shortcut
You'll see vim plugins mention <leader>, that <leader> for awesome view is "," so whenever you see leader hit that key.

## phpunit
### Set the path of phpunit (most cases for me, vendor/bin/phpunit)
let g:phpunit_bin = 'phpunit'

### Shortcuts
<leader>ta - Run all test cases
<leader>ts - Switch between source & test file
<leader>tf - Run current test case class

# Folding
`zo` to open folding
`zc` to close folding

# NerdTREE
<leader>nn - Toggles NerdTREE
While inside NerdTREE hit "m" to do a number of modifications from renaming, deleting or adding files.

# Terminal
Typing ":term" will open a terminal pane similar to phpstorm.

Of course you can reference https://github.com/amix/vimrc for more tips and tricks, such as <leader>w to write a file or <Ctrl+f> to fuzzy search for a file.

I'd like to make a video to accompany this setup to better describe how to use vim in this way so let me know if that's anything you'd like to see and best of luck in the new vim setup!

Getting around Brew's "Error: Calling Non-checksummed download of..."

Justin — Thu, 08 Oct 2020 11:52:11 +0000

I ran into this error while trying to install curl w/ http3 support:

brew install --HEAD -s https://raw.githubusercontent.com/cloudflare/homebrew-cloudflare/master/curl.rb

Error: Calling Non-checksummed download of curl formula file from an arbitrary URL is disabled! Use 'brew extract' or 'brew create' and 'brew tap-new' to create a formula file in a tap on GitHub instead.

The suggested fixes, brew extract, brew create, or brew tap-new are totally foreign to me and I wasn't sure where to begin. Googling it lead me down a path of more brew related terminology I had never heard of before.

Easiest way to bypass it?

wget [the-url]

Then

brew install --HEAD -s [the-script]

So from my example above:

# Instead of
brew install --HEAD -s https://raw.githubusercontent.com/cloudflare/homebrew-cloudflare/master/curl.rb

# Do
wget https://raw.githubusercontent.com/cloudflare/homebrew-cloudflare/master/curl.rb

brew install --HEAD -s curl.rb

Of course make sure you only do this for trusted sources!

Happy hacking!

Installing curl with http3 on MacOS

Justin — Thu, 08 Oct 2020 11:34:03 +0000

When you are all said and done with this tutorial, you'll be able to double check http3 support via the curl command like this:

curl --http3 https://cloudflare-quic.com -I
HTTP/3 200
date: Fri, 24 Jul 2020 08:10:24 GMT
content-type: text/html
content-length: 106072
set-cookie: __cfduid=d8647a359d68a89c060bde4f373e18cc61595578224; expires=Sun, 23-Aug-20 08:10:24 GMT; path=/; domain=.cloudflare-quic.com; HttpOnly; SameSite=Lax; Secure
cf-request-id: 042178a64b0000188b7b1d9200000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 5b7c2a1d4eb5188b-MAN
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400

Unfortunately, this currently involves building curl from source, luckily for us, cloudflare makes this even easier.

Note: You must have https://brew.sh/ installed already

# Clean up any old version of curl you may have already tried to install
brew remove -f curl

# Download the curl ruby install script provided by cloudflare
wget https://raw.githubusercontent.com/cloudflare/homebrew-cloudflare/master/curl.rb

# Install curl via that script from the latest git repos
brew install --HEAD -s curl.rb

# Tell your cli to use the curl version just installed (if you're using zsh, othwerise you might need `~/.bashrc`)
echo 'export PATH="/usr/local/opt/curl/bin:$PATH"' >> ~/.zshrc

# Reload your config
source ~/.zshrc

# Double check it's using the right curl
which curl # Should output "/usr/local/opt/curl/bin/curl"

# Double check http3
$ curl --version | grep HTTP3
  Features: alt-svc AsynchDNS brotli HTTP2 HTTP3 IDN IPv6 Largefile libz MultiSSL NTLM NTLM_WB SSL UnixSockets zstd

# Try curl on any HTTP/3 enabled sites.
curl --http3 https://blog.cloudflare.com -I

Have fun upgrading all the servers!

Profiling PHP from your Mac

Justin — Thu, 04 Jun 2020 12:58:47 +0000

I'm going to use a few open source tools to accomplish this, but alternatively, blackfire.io is an awesome service that handles all this setup for you if you'd rather pay to skip the elaborate setup.

The tools we're going to use are xdebug which will log/profile each part of your application to a file and then qcachegrind will let you visually inspect that log to find where the slow downs are happening.

So on a mac, to get started run

# To install xdebug
pecl install xdebug

At the end of the xdebug install you'll see something like:

zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/xdebug.so

Where it's telling you to add that to your php.ini file, to find where your php.ini file is, run php -i | grep "Configuration File" and you should see something like:

$ php -i | grep "Configuration File"
Configuration File (php.ini) Path => /usr/local/etc/php
Loaded Configuration File => (none)

Now just append that zend_extension line from earlier:

echo 'zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/xdebug.so' >> /usr/local/etc/php/php.ini

Now run php -v and you should see:

root@api-local:/srv/app# php -v
PHP 7.4.5 (cli) (built: Apr 23 2020 16:44:34) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Xdebug v2.9.6, Copyright (c) 2002-2020, by Derick Rethans
    with Zend OPcache v7.4.5, Copyright (c), by Zend Technologies
root@api-local:/srv/app#

So now with xdebug installed, let's set it up to start profiling requests.

vi /usr/local/etc/php/php.ini

Make yours look something like this now:

zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/xdebug.so
xdebug.profiler_enable=1
xdebug.profiler_enable_trigger=0
xdebug.profiler_output_dir=/tmp

Notice how xdebug.profiler_enable=1 is on, and xdebug.profiler_enable_trigger=0 is off, that's cause we want to make sure this works and later we can add a chrome extension so we'll only profile when we need.

So with xdebug.profiler_enable=1 enabled, hit a page on your application and make sure you see something like this in your tmp directory:

root@api-local:/srv/app# ls /tmp/cachegrind*
/tmp/cachegrind.out.10812
root@api-local:/srv/app#

Now install qcachegrind so you can inspect the dumped profile:

brew install qcachegrind

With that installed, use your spotlight shortcut (cmd+spacebar) and type qcachegrind to find and open the visualizer tool, and use the tool to open the /tmp/cachegrind.out.10812 file that gets made:

Now with it knowingly working, you can edit your php.ini to disable profiler always and set it to enable trigger.

vi /usr/local/etc/php/php.ini

zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20190902/xdebug.so
xdebug.profiler_enable=0
xdebug.profiler_enable_trigger=1
xdebug.profiler_output_dir=/tmp

Then use a extension like xdebug helper for chrome to toggle it when needed.