DEV Community: Jack Moore

BGP and Cilium Kubernetes

Jack Moore — Tue, 20 Sep 2022 05:00:00 +0000

For the current setup I have with kubernetes I ended up using Cilium as the networking provider configured with BGP on the OPNSense Firewall I have. This setup was a bit different because Cilium didn’t support having Nodes on the same network as the pods.. which makes sense, however it should have allowed for better configuration of the pod CIDR. Below is how I configured Cilium and OPNSense:

I made an attempt to run clium configured with BGP, but needed to move the nodes to 172.16.0.0/24 network.

This was the process I used:

Create new vlan on switch for 172.16.0.0/12 network;
- Provisioning of nodes was done with Terraform;
Pluck off a space in the network for kubernetes;
Add Firewall interface for the new L2 network
Update Terraform to move everything to a 172.16.0.0/12 network;
Update Scripts
Try Cilium again;

Installing Helm and Cilium

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium --version 1.12.2 --namespace kube-system --set bgp.enabled=true --set bgp.announce.loadbalancerIP=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true #--set bgp.announce.podCIDR=true

After installing Clium I needed to add a config map for bgp pointing to opnsense (note this may have needed to have been done before installing cilium):if email_message.is_multipart():
for part in email_message.get_payload():
body = part.get_payload()
# more processing?
else:
body = email_message.get_payload()

Note the peer address is my firewall ip
Note the peer asn is the AS of the firewall
Note the current AS is the AS of the kubernetes cluster

apiVersion: v1
kind: ConfigMap
metadata:
  name: bgp-config
  namespace: kube-system
data:
  config.yaml: |
    peers:
      - peer-address: 172.16.0.254
        peer-asn: 64512
        my-asn: 64513
    address-pools:
      - name: default
        protocol: bgp
        addresses:
          - 172.16.1.0/24

OPNSense Side:

On the opnsense side I had to install the frr package which allowed for better routing. From there I had to configure BGP.

This meant:

enabling bgp
Setting the BGP AS Number to 64512
Adding all the k8s nodes into the neigbors of the bgp config with the correct peer asn (peer asns are 64513)

After everything has been configured LoadBalancer IPs can now be used on the 172.16.1.0/24 network.

- BGP Cilium 1.12.2

| BGP.html |

- Cilium in VMs

| Helm | Installing Helm |

A Bit of Nginx Load Balancing

Jack Moore — Wed, 19 Jan 2022 05:00:00 +0000

Similar to the last post, I wanted to look more into NGINX features including load balancing and geographically serving requests from servers within the same region. In this post I look into load balancing requests with GeoIP2, serving requests based on locations and IPs, and nginx upstream servers for load balancing. I also added the echo module to nginx, and explored other logging formats for nginx.

Containers and Docker Networking

After committing the image I created last post (nginx:geocity), this post will use that same image to create multiple containers to server requests. I did move the image to a different server so I didn’t accidentally delete any production containers. To copy the image, I used the following:

docker save -o /opt/nginx/nginx-geocity nginx:geocity
scp /opt/nginx/nginx-geocity jack@remoteserver:/opt/nginx/nginx-geocity
docker load -i /opt/nginx/nginx-geocity

From the new host, it all starts with a network to connect the docker containers. For this simple POC, I created all the containers on the same host and used docker networking to link them all together. Below is the creation of the containers and network:

# Create Geocity container to proxy all requests
docker run --name nginx_geocity_1 -p 8081:80 nginx:geocity

# Create network
docker network create --driver bridge nginx-net

# Connect already running container to network
docker network connect nginx-net nginx_geocity_1

# Create new container connected to network
docker run --name nginx_region_1 --network nginx-net -d nginx:geocity
docker run --name nginx_region_2 --network nginx-net -d nginx:geocity
docker run --name nginx_region_3 --network nginx-net -d nginx:geocity

At this point there are 4 containers running, we have one main nginx server to process incoming requests (nginx_geocity_1) and three “regional” servers to serve requests (nginx_region_[1,2,3]).

Because the default page simply displays “Welcome to Nginx”, I needed a way to distinguish which regional server I was connecting to. Again because this is a POC, I edited the html/index.html to say “Welcome to Region $regional_server_id” which meant when I loaded regional server 1 it displayed “Welcome to Region 1”, and for regional server 2 it displayed “Welcome to Region 2”. Simple, but necessary.

Nginx Configuration

After this was setup, I modified the nginx.conf file in the nginx_geocity_1 server to look like the following to server requests based on region:

http {
    geoip2 /opt/GeoLite2/GeoLite2-City.mmdb {
        auto_reload 60m;
        $geoip2_metadata_city_build metadata build_epoch;
        $geoip2_data_city_name city names en;
    }
    geoip2 /opt/GeoLite2/GeoLite2-Country.mmdb {
        auto_reload 60m;
        $geoip2_metadata_country_build metadata build_epoch;
        $geoip2_data_country_code country iso_code;
        $geoip2_data_country_name country names en;
        $geoip2_data_continent_code continent code;
    }

    map $geoip2_data_continent_code $nearest_server {
        default all;
        EU eu;
        NA na;
        AS as;
        AF af;
    }

    # geo $geo {
    # default all;
    # 172.17.0.0/24 eu;
    # 127.0.0.1/32 all;
    # 10.0.4.0/24 na;
    # }

Note this is very similar to the last post. Essentially not much has changed, however we have added a geo $geo block (for testing purposes used later). Basically continent codes get mapped to a $nearest_server variable now and also the $geo variable also maps to continent character codes. These variables are for when serving requests where to direct the requests to the next server (ie eu will go to Europe servers, na will go to North American servers, all is used for defaults).

Now the default.conf site has been modified to look like the following:

server {

    listen 80;
    listen [::]:80;
    server_name localhost;

    location /echo {
        echo $geo;
    }

    location / {
        proxy_pass http://$nearest_server;
    }
    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
        root /usr/share/nginx/html;
    }
}

upstream all {
        server nginx_region_1;
}

upstream eu {
        server nginx_region_2;
}

upstream na {
        server nginx_region_3;
}

Now any request that comes in is mapped a country code from the GeoIP2 module, and then is served data via proxy from the closest geographic server. (I could and really should have used mod_rewrite to reroute the request for better performance instead of proxying requests through the one server, but will look into this later.)

Where I ran into problems

The above looks great. Everything looks like it works and all is well except that this is all on one server. It is hard to test out geo codes from localhost and from an internal network because all the requests are going to end up at the same proxy_pass location.

I tried a few different things including adding the echo module, and setting real_ip_addr for the proxy, but these didn’t work. I even contemplated IP Spoofing, but stayed away to prevent tangent-ing off into networking issues. My first thought was to add more logging. I added more logging to track the upstream requests to the containers that were serving the requests.

I added the following to my nginx.conf configuration within the http block to keep the regular logging, as well as log the requests sent to the upstream servers:

    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    log_format upstreamlog '$server_name to: $upstream_addr {$request} '
                           'upstream_response_time $upstream_response_time'
                           ' request_time $request_time';

    access_log /var/log/nginx/access.log main;
    access_log /var/log/nginx/access.log upstreamlog;

Although this was helpful to have extra logging, it did not help me solve my issue, it just showed I was able to process requests fast.

Then I added the echo module to nginx, and this is why there is a location to serve /echo which will just return the continent_code on a request. (Note: I had to re-compile nginx with the echo module because it is not builtin to NGINX by default.)

This is where the NGINX builtin geo module also came into play. With a few quick modifications I was able to test connections from the network I’m currently on, the local machine, and the docker network to the geocity server to see where requests were being sent. Uncommenting the geo block that’s shown above in my configuration, I modified the configuration and instead of proxy_passing the $nearest_server to proxy_pass to $geo instead.

Now when I curled from my local machine I would get different requests back from the server showing which variables were being used:

# Curl request from my local machine to the server:
curl remote.ip.addr.here:8081/echo
# response: na

# Curl request from within the container
curl localhost/echo
# response: all

# Curl request from a different docker container on the same subnet
curl nginx_geocity_1/echo
# response: eu

Then after this was complete, I tested to see if the /index.html page returned “Welcome to Region $region_id” indicating I was infact getting a response from the correct upstream server. After this verification everything was looking good. I figured this was a good POC, and continent codes would match, but would likely need to be tested on multiple servers before being moved into production.

Adding more servers to the upstream

NGINX upstream allows for multiple servers and load balancing by simply adding more servers within the upstream blocks. It also allows for different methods of loadbalancing including round robin (appears to be the default if two servers are added to the upstream), least_conn (least connections), ip_hash, and generic hash.

I tested this out by adding all the servers to one upstream and using least_conn. The upstream block looked like the following:

upstream na {
        least_conn;
        server nginx_region_1;
        server nginx_region_2;
        server nginx_region_3;
}

Using my browser and refreshing a few times, I was able to see all three region servers were serving the request.

From Here

From here I want to look into IP Spoofing for testing including using the scrapy package, and iptables. I also want to look into docker overlay netowrks for networking across docker hosts (although I’d like to do this without docker swarm or k8s for right now.)

Also, as I noted above, this is just an ok solution for georouting. It is far from perfect. Realistically, for the best georouting it would start at DNS georouting. Then from there if someone from europe managed to up hit the US version of the site at “example.com” it may be better to use mod_rewrite to send them to the “.eu” version of the site on their first request to better serve them. I may look into these types of solutions in the future.

Clustered Storage with DRBD

Jack Moore — Fri, 04 Dec 2020 05:00:00 +0000

Logical Volumes

Create an LVM on each server for the NFS Mount

On both servers:

lvcreate -n nfspoint -V 550G pve/data
mkfs.ext4 /dev/pve/nfspoint
echo '/dev/pve/nfspoint /var/lib/nfspoint ext4 defaults 0 2' >> /etc/fstab
mkdir /var/lib/nfspoint
mount /dev/pve/nfspoint /var/lib/nfspoint/
lvs

vim /etc/exports

/var/lib/nfspoint 10.0.0.0/255.0.0.0(rw,no_root_squash,no_all_squash,sync)

DRBD

DRBD is a little tricky.. Because LVM was thin provisioned it looked like some metadata was written to teh device /dev/pve/nfspoint. This means I had to zero out the nfspoint.

DRBD Configuration:

global { usage-count no; }
common { syncer { rate 100M; } }
resource r0 {
        protocol C;
        device /dev/drbd0 minor 0;
        startup {
        wfc-timeout 120;
            degr-wfc-timeout 60;
            become-primary-on both;
        }
        net {
            cram-hmac-alg sha1;
            allow-two-primaries;
            shared-secret "secret";
        }
        on HLPMX1 {
            disk /dev/pve/nfspoint;
            address 10.0.0.2:7788;
            meta-disk internal;
        }
        on HLPMX2 {
            disk /dev/pve/nfspoint;
            address 10.0.0.3:7788;
            meta-disk internal;
        }
}

This wasn’t a problem because there was no data. This may be a problem when I expand the volume with LVM or need to make any kind of resource changes to the device.

dd if=/dev/zero of=/dev/pve/nfspoint bs=1M count=200


mkfs.ext4 –b 4096 /dev/drbd0


curl --output drbd9.15.tar.gz https://launchpad.net/ubuntu/+archive/primary/+sourcefiles/drbd-utils/9.15.0-1/drbd-utils_9.15.0.orig.tar.gz
tar -xf drbd9.15.tar.gz
cd drbd9.15
./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc
make all
make install

Node 2: (because gcc wasn’t installed)

apt install build-essential
apt install gcc
apt install flex


# Copy config to other server 
sudo drbdadm create-md r0
sudo systemctl start drbd.service
sudo drbdadm -- --overwrite-data-of-peer primary all
mkfs.ext4 /dev/drbd0
mkdir /srv/nfspoint
sudo mount /dev/drbd0 /srv/nfspoint

Splitbrain

On Split Brain Victim

drbdadm disconnect r0
drbdadm secondary r0
drbdadm connect --discard-my-data r0

On Split Brain Survivor

drbdadm primary r0
drbdadm connect r0

Inverting Resourcing

On Current Primary:

umount /srv/nfspoint
drbdadm secondary r0

On Secondary:

drbdadm primary r0
mount /dev/drbd0 /srv/nfspoint

DRDB Broken?

Reboot both servers

systemctl start drbd.service
drbdadm status

umount /dev/pve/nfspoint
mkfs.ext4 -b 4096 /dev/pve/nfspoint

dd if=/dev/zero of=/dev/drbd0 status=progress


mkfs -t ext4 /dev/drbd0

Troubleshooting DRBD Issues

Odd Mountpoint with Loop Device

What should appear off lsblk:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.7T 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part
└─sda3 8:3 0 1.7T 0 part
  ├─pve-swap 253:0 0 8G 0 lvm [SWAP]
  ├─pve-root 253:1 0 96G 0 lvm /
  ├─pve-data_tmeta 253:2 0 15.6G 0 lvm
  │ └─pve-data-tpool 253:4 0 1.5T 0 lvm
  │ ├─pve-data 253:5 0 1.5T 0 lvm
  │ ├─pve-vm--105--disk--0 253:6 0 20G 0 lvm
  │ └─pve-nfspoint 253:7 0 550G 0 lvm
  │ └─drbd0 147:0 0 550G 0 disk /srv/nfspoint
  └─pve-data_tdata 253:3 0 1.5T 0 lvm
    └─pve-data-tpool 253:4 0 1.5T 0 lvm
      ├─pve-data 253:5 0 1.5T 0 lvm
      ├─pve-vm--105--disk--0 253:6 0 20G 0 lvm
      └─pve-nfspoint 253:7 0 550G 0 lvm
        └─drbd0 147:0 0 550G 0 disk /srv/nfspoint

What was incorrectly appearing off lsblk:

loop0 7:0 0 200M 0 loop /srv/nfspoint
sda 8:0 0 1.7T 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part
└─sda3 8:3 0 1.7T 0 part
  ├─pve-swap 253:0 0 8G 0 lvm [SWAP]
  ├─pve-root 253:1 0 96G 0 lvm /
  ├─pve-data_tmeta 253:2 0 15.6G 0 lvm
  │ └─pve-data-tpool 253:4 0 1.5T 0 lvm
  │ ├─pve-data 253:5 0 1.5T 0 lvm
  │ ├─pve-vm--100--disk--0 253:6 0 30G 0 lvm
  │ ├─pve-vm--102--disk--0 253:7 0 100G 0 lvm
  │ ├─pve-vm--103--disk--0 253:8 0 20G 0 lvm
  │ ├─pve-vm--104--disk--0 253:9 0 20G 0 lvm
  │ ├─pve-vm--101--disk--0 253:10 0 20G 0 lvm
  │ └─pve-nfspoint 253:11 0 550G 0 lvm
  │ └─drbd0 147:0 0 550G 0 disk
  └─pve-data_tdata 253:3 0 1.5T 0 lvm
    └─pve-data-tpool 253:4 0 1.5T 0 lvm
      ├─pve-data 253:5 0 1.5T 0 lvm
      ├─pve-vm--100--disk--0 253:6 0 30G 0 lvm
      ├─pve-vm--102--disk--0 253:7 0 100G 0 lvm
      ├─pve-vm--103--disk--0 253:8 0 20G 0 lvm
      ├─pve-vm--104--disk--0 253:9 0 20G 0 lvm
      ├─pve-vm--101--disk--0 253:10 0 20G 0 lvm
      └─pve-nfspoint 253:11 0 550G 0 lvm
        └─drbd0 147:0 0 550G 0 disk

If the server is creating a loop device and mounting that as the mount point, try rebooting the server, restarting the drbd service, and clearing out the device on the server having the issue and resyncing it with the primary.

I still dont know if it was split brain or what, but these were the commands I ran:

umount /nfs/sharepoint # This was the location of the mountpoint
drbdadm disconnect
drbdadm connect --discard-my-data r0

I honestly think rebooting the server in question fixed this issue. I think it was something about the /dev/drbd0 device that wasn’t working properly or created properly.

Links 2

https://www.howtoforge.com/high\_availability\_nfs\_drbd\_heartbeat\_p2 https://pve.proxmox.com/wiki/Logical\_Volume\_Manager\_(LVM) https://linux.die.net/man/8/lvremove https://access.redhat.com/documentation/en-us/red\_hat\_enterprise\_linux/6/html/logical\_volume\_manager\_administration/lv\_remove https://serverfault.com/questions/266697/cant-remove-open-logical-volume

Bug Reports

Bug Report

Less useful links

The Blog Post about the Blog

Jack Moore — Tue, 04 Aug 2020 05:00:00 +0000

Without getting too meta, this is the post about my blog as it stands and where I see the future of it going.

Reasoning for the Meta

I was exploring my Matomo Instance recently and noticed over the past year I have had 800 page views on this site. While this is really quite a small number, I was starting to think about the future of this blog and starting to spread the word on it more for both feedback and to spread more information to those who want/need it.

I was thinking about mediums I can share these posts on and getting the word out on some of the current projects I am working on. I think it would be a great way to get more involved with the developer/sysadmin community and also spread information for anyone wanting to learn more.

Current Blog

With the current blog as it stands, I create posts whenever I want about whatever I want. Nothing about this blog is a chore and I for the most part enjoy maintining this blog and writing these posts. Everything with this blog is easy. Netlify hosts it for free (shoutout netlify), DNS is configured for basically forever, and SSL on the site renews automatically. All 3 basic components to a blog are there. It’s everything I need right now to be perfect reference material for myself when I run into pesky recurring issues.

Material

The material for the blog is about things I run into and the fixes/workarounds for the issues pertaining to infrasture, developing ruby/rails/python/whatever and also some around concepts I find interesting. I think this is partially why the material flows/ comes out easily for me. I am able to just dump fixes into posts and write about what I learned. As long as I continue to learn, the posts will continue to come. Whether good or bad, I am usually always documenting.

Writing Style

The writing in these posts is very: here’s the point, here’s why this works for me, if you want a more in depth explaination look up details on your own and also very i wrote one draft of this post and here it is

Unfortunately for the reader, I really don’t read these posts back to myself like a novel in any way. Everything is meant to be technical, but usable. Most of the posts don’t “flow” per se. They more or less just come out with whatever English makes sense to me at the time. Dare I even say, every post for the most part is slapped together in a cryptic way I am able to comprehend for when I run into the issue the next time it happens. if it’s happened once, it’s bound to happen again

This blog is mostly for my reference, but if you are able to take something away, then cheers.

Community

I think feedback would be nice and I am all for writing back. This is mostly why I am going to start sharing these to a wider audience.

I love comments and questions and will always do my best to answer them!

Vim Configuration

Jack Moore — Tue, 12 May 2020 05:00:00 +0000

Vim, my server editor of choice. (I prefer vscode for development on my workstation.)

Plugins

vim-rainbow
powerline
nerdtree

~/.vimrc

This .vimrc is pretty gross looking, but please just note the major configurations:

Line/Relative Line
Tabs are 4 spaces
Typing { automatically generates a closing \n}
NerdTree is enabled
The theme is Solarized
Parentheses are pretty..

"VIMRC Configuration

"Default configuration
"Relative Line Number
:set number
:set rnu

"Spacing
:set tabstop=4
:set shiftwidth=4
:set expandtab

"Remap..
inoremap {<cr> {<cr>}<esc>ko
inoremap ( ()<left>

"Powerline for VIM
set rtp+=/home/jack/.local/lib/python3.8/site-packages/powerline/bindings/vim
set laststatus=2
set t_Co=256

"Nerd Tree
autocmd StdinReadPre * let s:std_in=1
autocmd VimEnter * if argc() == 0 && !exists("s:std_in") | NERDTree | endif

autocmd StdinReadPre * let s:std_in=1
autocmd VimEnter * if argc() == 1 && isdirectory(argv()[0]) && !exists("s:std_in") | exe 'NERDTree' argv()[0] | wincmd p | ene | exe 'cd '.argv()[0] | endif

autocmd bufenter * if (winnr("$") == 1 && exists("b:NERDTree") && b:NERDTree.isTabTree()) | q | endif

map <C-n> :NERDTreeToggle<CR>

"Solarized Theme
syntax enable
set background=dark
colorscheme solarized
let g:solarized_termcolors=256

"Vim-Rainbow
let g:rainbow_active = 1

let g:rainbow_load_separately = [
    \ ['*' , [['(', ')'], ['\[', '\]'], ['{', '}']] ],
    \ ['*.tex' , [['(', ')'], ['\[', '\]']] ],
    \ ['*.cpp' , [['(', ')'], ['\[', '\]'], ['{', '}']] ],
    \ ['*.{html,htm}' , [['(', ')'], ['\[', '\]'], ['{', '}'], ['<\a[^>]*>', '</[^>]*>']] ],
    \ ]

let g:rainbow_guifgs = ['RoyalBlue3', 'DarkOrange3', 'DarkOrchid3', 'FireBrick']
let g:rainbow_ctermfgs = ['lightblue', 'lightgreen', 'yellow', 'red', 'magenta']

Bash Powerline

Install Powerline

pip install git+git://github.com/Lokaltog/powerline
wget https://github.com/powerline/powerline/raw/develop/font/PowerlineSymbols.otf
wget https://github.com/powerline/powerline/raw/develop/font/10-powerline-symbols.conf
mv PowerlineSymbols.otf /usr/share/fonts/
fc-cache -vf /usr/share/fonts/
mv 10-powerline-symbols.conf /etc/fonts/conf.d/

~/.bashrc

# Powerline
export PATH="$HOME/.local/bin:$PATH"
export POWERLINE_COMMAND=powerline
export POWERLINE_CONFIG_COMMAND=powerline-config
powerline-daemon -q
POWERLINE_BASH_CONTINUATION=1
POWERLINE_BASH_SELECT=1

POWERLINE_BASH_CONTINUATION=1
POWERLINE_BASH_SELECT=1
. /home/jack/.local/lib/python3.8/site-packages/powerline/bindings/bash/powerline.sh

Getting Glyphs to work when SSHing from Mac to Server

I used FIRA Code to get powerline working on the server.

In ITerm2 command + , to open preferences, profiles > text > Non ASCII Font Fira Code retinat font size 12.

Metaprogramming Rails Helper Module for Accessible Attributes

Jack Moore — Thu, 16 Apr 2020 05:00:00 +0000

Dynamically creating instance methods:

module RundeckHelper

    def self.attr_value(*args)
        # Singleton Very Important
        singleton_class = class << self; self; end
        args.each do |key|
            name = key
            key = key.to_s.upcase
            val = nil
            if !ENV[key].nil?
                if ENV[key] != ""
                    val = ENV[key]
                else
                    val = nil
                end
            elsif !Rails.application.credentials.rundeck.nil?
                key2 = key.downcase
                if !Rails.application.credentials.rundeck[key2].nil?
                    val = Rails.application.credentials.rundeck[key2]
                else
                    val = nil
                end
            elsif !RundeckConfigurationOption.find_by(name: key).nil?
                val = RundeckConfigurationOption.find_by(name: key).value
            else
                val = nil
            end
            # Send the method to the instance
            singleton_class.send(:define_method, name) do
                return val
            end
        end
    end

    attr_value :base_url, :create_job_id, :project_id, :create_environment_id, :create_instance_id
end

What is this code and how does it work?

Going line by line, this code exists in a Rails module. This is why I believe this was difficult. Creating custom attr_’s had to be hacked as everything in this module is called as an instance method. The module isn’t instantiated like a class would be. This module in particular has all its methods called as instance methods. The module basically provides helpers to the classes that require this module.

Breaking down the key lines:

def self.attr_value(*args)
singleton_class = class << self; self; end
singleton_class.send(:define_method, name) do
attr_value :base_url, :create_job_id, :project_id, :create_environment_id, :create_instance_id

These three lines are crucial to creating custom attributes for the variables passed into the attr_value function.

Starting from the top: def self.attr_value(*args)

This line defines the method and the method accepts multiple arguments.
It is an instance method meaning it can be called from within the module

Moving on: singleton_class = class << self; self; end

We define a singleton_class inside this method to accept from the current object which is the module

Sending Methods: singleton_class.send(:define_method, name) do

We take the singleton_class we created that is attached to the module and send it the method
the name variable, as seen above, is just the name of the symbol that was passed

Calling the Attribute: attr_value :base_url, :create_job_id, :project_id, :create_environment_id, :create_instance_id

basically just instantiates the values we need and allows us to use the methods RundeckHelper.base_url and self.base_urlreturning the value stored in the environment, in the secrets, or in the database.

E2E Modded Minecraft Server

Jack Moore — Sun, 15 Mar 2020 05:00:00 +0000

This is the second time I have had to run through install of a minecraft client and server. I decided I need documentation this go around for the next time I want to install a modded minecraft server.

I decided to use ansible to make deployments easy. My biggest struggle was upgrading from an older version of ansible using python 2.7 to ansible 2.9.6 using python 3.5.

Client

Getting the client setup wasn’t too difficult.

MultiMc

I am running the Minecraft Client from my XPS workstation which runs Ubuntu 18.04.

I knew I had run Minecraft from the workstation before, but when I went to find the .jar client I didn’t know where it was.I couldn’t find it anywhere. I looked recursively in /home/jack and in /opt. It was nowhere to be found.

I started to install the forge client on my workstaion when I stumbled upon MultiMC which supports linux.

I then stumbled upon MultiMC already installed on my workstation with some older clients installed.

Setting RAM

MultiMC allows to set the RAM. I have 16GB of RAM available. I allocated 8GB to the MC instance/ to JVM. I haven’t run into any issues thus far.

Installing the E2E ModPack

Installing the Modpack was relatively simple besides the fact that the modpack is reccomended to be installed using the Twitch client. I had to find the zip file of the client. The version I wanted was here. It was about 20MB.

The filename is a zip, Enigmatica2Expert-1.77.zip. In MultiMC add an instance and then choose a zip file modpack. Also make sure to have the correct Minecraft Version installed. This modpack uses MC 1.12.2.

Server

The client requires 5GB, but recommends 6GB to 8GB. I wanted chunk loading and a few other compute and ram intensive features of Minecraft to run on a server so my workstation isn’t completely bogged down with processes.

The server also allows for other players to join.

Ansible

I know I will want to spin up a MC server again and since I have done this before I have decided to create an ansible playbook for the service.

The playbook will need to be revised, but currently everything is in the common role. As I start to add more servers for games I will revise the roles for the different applications.

The directory service looks like the following:

.
| ____ group_vars
| | ____ all
| ____ site.yml
| ____ roles
| | ____ common
| | | ____ files
| | | | ____ forge-1.12.2-14.23.5.2847-installer.jar
| | | | ____ eula.txt
| | | | ____ mc13.service
| | | ____ tasks
| | | | ____ configure_modpack.yml
| | | | ____ main.yml
| | | | ____ install_java.yml
| | | | ____ install_forge.yml
| | | | ____ start_server.yml
| ____ hosts
| ____ host_vars

The site.yml in the playbook root directory calls the common role (role/common/tasks/main.yml).

The common role runs as root. Really this is a prelimenary playbook that runs everything as root. It is really a temporary server meant to be run for fun. A more permanent solution would run as a minecraft user and take more variables into account.

This example uses a server with 8gb of ram and 2 cpus.

The one note I will make on this ansible setup is that the roles/common/tasks/configure_modpack.yml file copies files from my user profile up to the server. (I had trouble pulling down zip server files..) Here is what the task looks like:

- name: (Configure Modpack) Copy Folder - Mods
  copy:
    src: /home/jack/.local/share/multimc/instances/Enigmatica2Expert-1.77/minecraft/mods
    dest: /root

- name: (Configure Modpack) Copy Folder - Scripts
  copy:
    src: /home/jack/.local/share/multimc/instances/Enigmatica2Expert-1.77/minecraft/scripts
    dest: /root

- name: (Configure Modpack) Copy Folder - Schematics
  copy:
    src: /home/jack/.local/share/multimc/instances/Enigmatica2Expert-1.77/minecraft/schematics
    dest: /root

- name: (Configure Modpack) Copy Folder - Resources
  copy:
    src: /home/jack/.local/share/multimc/instances/Enigmatica2Expert-1.77/minecraft/resources
    dest: /root

- name: (Configure Modpack) Copy Folder - Config
  copy:
    src: /home/jack/.local/share/multimc/instances/Enigmatica2Expert-1.77/minecraft/config
    dest: /root

- name: (Configure Modpack) Copy Folder - Manifest.json
  copy:
    src: /opt/minecraft/manifest.json
    dest: /root

- name: (Configure Modpack) Add Eula
  copy:
    src: eula.txt
    dest: /root

Forge Installer vs Forge Universal

Forge installer is required to install MC Server. The Forge Universal is required to run the server.

Turning the Minecraft Server into a Service

The service runs java from an absolute path specifying 8gb of ram allocated to the JVM pointing to a jar file at the root users directory. The nogui at the end specifys to forge not to open a GUI.

[Unit]
Description=Minecraft Server
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=never
RestartSec=1
User=root
WorkingDirectory=/root
ExecStart=/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms8G -jar /root/forge-1.12.2-14.23.5.2847-universal.jar nogui

[Install]
WantedBy=multi-user.target

Tacotron-2 - Text to Speech, My Speech - Part 1

Jack Moore — Tue, 19 Nov 2019 05:00:00 +0000

The gist of this post is that every day for the past few weeks I have gone home and talked to my computer.Yes, you read that correctly. I have gone home and talked to my computer.Also, don’t let the title fool you. The beef of this post is setting up an environment and a process to help me develop a custom text to speech program. I would only call this a mere glance into the Text-To-Speech Algorithms.

I need the future, now. This project is to build a text to speech system using my own voice as the training model! I am very excited to build this from the ground up with my own voice as the training data.

I will be using a handful of Artificial Intelligence libraries to ensure this process goes as smooth as possible. Some include Mozilla’s TTS, Gentle for speech mapping, SOX for data cutting, and of course Python…

Picking a Library

After looking at and evaluating the libraries that are out there, I decided to go with Mozilla’s TTS Library. I felt that their library was relatively easy to put together and felt I could easily reproduce my own voice from text with their library.

Building a Dataset

It appears having a good, no scratch that, perfect, dataset is the most important part of building any decent text to speech application.

I looked at the LJ Speech Dataset and decided about 24hrs, (close to ~13,100 utterances/audioclips) of my own time would be needed to record and collect data. I am sure this will take upwards of 40-50hrs to ensure the data is properly Extracted, Transformed and Loaded. I plan to use the aeneas library to match my speech to text. Confirming this is the correct way to build a model, I also looked at building other datasets as well. The other popular dataset is from the Blizzard challenge, and the M-AILABS Speech Dataset. I could have used these, but the LJ Speech Dataseet seemed easier to replicate.

Mozilla provides a great article here on how to build a custom voice Text to Speech Application. The article mentioned will be one of many I will be using to learn more about building my custom TTS program.

There’s really just no chance

There is not a fucking chance I sit down and read 13,000 one liners back to back. I need to find an already exiting text broken up and I need it then matched to the wav file and broken down into 5 second wav files. Enter Python. This isn’t too crazy, but my plan is to basically read a chapter a day until I am done with three books. This should get me to ~15,000 sentences which should be all that’s needed for the training model. I will feed the model more data if I feel it is necessary.

The process will look like the following: 1) Find a Full Plain Text Book Online 2) Parse Text Sentence by Sentence into a single file data (python..) 3) Read and Record the Single file to a single wav file 4) Use Python Library Aeneas to match text to speech (still in bigger file) 5) Use Python to break up the large wav file into a smaller wav file using ffmpeg 6) Aeneas to Create the LJ wav folder and .csv file’

So I thought…

Refined Process

The 6 step process above is nice, but almost a little unrealistic and way too time consuming for someone as lazy as myself. If it can be better I will make it better. Here is the new process: 1) Find Text/Plaintext Script (Movie Scripts are fun to read) () -> plaintext file 2) Record Text on Garageband (Im on Mac, I couldn’t get my MIC to work with audacity) and save it to Wav Format () -> wav file 3) Upload Text & Unbroken Large Wav File to Gentle for it () -> json file 4) Parse JSON returned from Gentle and break large file with sox into LJSpeech Dataset, () -> wavs folder and csv mapping to the file 5) Pass LJSpeech Dataset to TTS Model

Seems Easy Enough.

Dataset

Microsoft’s site says, “the data needs to be a collection (.zip) of audio files (.wav) as individual utterances. Each audio file should be 15 seconds or less in length, paired with a formatted transcript (.txt).” They are basically correct with the information they provide.

This dataset example for mozillas TTS is what the custom dataset example should look like. I found the link on the Mozilla Form here. There is a good forum post mostly here and here that goes over training a custom voice.

In the /custom-dataset-sample/ directory there exists a wavs directory and a metadata_sample.csv file. The wavs directory stores .wav files and the metadata_sample.csv is structured to map wavs/file1.wav to the text inside of the wav file.

Writing a Preprocessor

Because we will be using a simliar format to the LJ Dataset, I will need to make sure the preprocessor uses the correct data processor. This could really fuck my model otherwise.

Training the Model

Looking at this example of the tacotron example, it appears the LJ Speech Dataset went through 441k steps and the results sound decent. I will be using the Tacotron2 library.

Looking Forward

Currently I know the process I am going to follow to achieve this goal of having my voice used by a computer. My plan is to write part 2 of this series after I am done with all the data collection.

This will allow me to really dive deep into curve fitting and understand the specifics of how ML/AI works.I plan to have a demystified understanding of AI/ML when I return for the second post.

Resources

Resources Final?

Github - Mozilla - Dataset

Mounting S3 as a filesystem with S3FS

Jack Moore — Wed, 01 May 2019 05:00:00 +0000

S3FS

This month I spent time working on creating a seamless file transfer system between my development machines and the cloud using an AWS S3 bucket and S3FS. I have a Mac laptop and an Ubuntu based desktop that have important files that often get out of sync with one another. These files exist outside of version control and I was looking for a better way than Google Drive or emails with attachments to transfer them among machines. After doing some quick research, I found S3FS as a way to mount S3 as a filesystem and decided this would be the best tool to use.

Easy Install

The developers of S3FS make it pretty easy to install the tool across unix based platforms and I had no trouble getting it installed on my Mac using:

brew cask install osxfuse
brew install s3fs

and on Ubuntu with:

sudo apt-get install s3fs

Credentials & AWS

The most complicated part of getting the tool up and running is configuring the aws credentials and the S3 bucket.

Creating the bucket was pretty much a snap, but be sure to set the correct settings on the bucket and also give it a DNS compliant name. I named mine itsltns, as I knew it was DNS compliant and it would be easy to remember.

TO MAKE THIS CLEAR: the credentials they are looking for are the Access Key ID and Access Secret Key. These can be found by logging in to AWS and finding the My Security Credentials page. This pair of keys is most commonly used by developers for development of AWS Applications. Make sure these keys or your IAM user has the correct permissions for S3, I gave my IAM user full permissions to read and write.

After generating the key pair they need to go in a file anywhere where you can find them. It was reccommended to put them in ~/.passwd-s3fs using ACCESS_KEY_ID:ACCESS_KEY_SECRET as the format, but I put them with my AWS SSH keys in ~/.ssh/AWS/passwd-s3fs. This file needs to have the permissions of the owner having read and write and group and others having none. Using chmod 600 ~/.ssh/AWS/passwd-s3fs we are able to run the s3fs command and access the S3 Bucket.

Mountpoints

Keeping everything simple and the mountpoint single user owned, I opted to create a mountpoint on both machines within my home directory. I created this mountpoint using mkdir ~/itsltns-s3 and as a little foreshadow, I also ran the command chmod a+rwx ~/itsltns-s3 because I ran into read/write permission issues when reading and writing files from mac-to-ubuntu and ubuntu-to-mac.

With the mountpoint configured, I was up and running using the command:

s3fs itsltns ~/itsltns-s3 -o passwd_file=${HOME}/.ssh/AWS/passwd-s3fs

This allowed me to run a basic ls ~/itsltns-s3 and it returned with nothing in it. To test if the mount was working, I added a test.txt file and sure enough it was copied up to the cloud and the test.txt was in the itsltns s3 bucket.

Issues & Logging

If you run into any issues with mounting or s3fs at all, I would highly reccommend using the logging function with the -o dbglevel=info -f -o curldbg flag at the end. After adding the debug option, the command would look like:

s3fs itsltns ~/itsltns-s3 -o passwd_file=${HOME}/.ssh/AWS/passwd-s3fs -o dbglevel=info -f -o curldbg

MacOS - launchd - mount on user login

One of the features I wanted was automounting the S3 drive on login, so I opted to create a launchd service to run on login.

When working with launchd services don’t bother installing the LaunchControl that is advertised on the launchd website unless you plan on doing serious agent or service development. The application has minimal functionality and requires a license.

Setting up the launchd wasn’t terrible, but I did have to debug the service with launchctl list | grep itsltns a couple times to get the status code of the application.

Without getting into too much into launchd detail, I created the file ~/Library/LaunchAgents/local.itsltns-s3.plist as my launchd service with the contents:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Label</key>
        <string>local.itsltns-s3.plist</string>
        <key>Program</key>
        <string>/Users/Jack/Library/LaunchAgents/itsltns-s3.sh</string>
        <key>RunAtLoad</key>
        <true/>
    </dict>
</plist>

If you look in the .plist xml file, you will see the <key> tags surrounding the word “Program” and <string> tags surrounding the location of the shell script to be executed.

With this, I also created the file ~/Library/LaunchAgents/itsltns-s3.sh with the contents:

#!/bin/bash
/usr/local/bin/s3fs itsltns /Users/Jack/itsltns-s3/ -o passwd_file=${HOME}/.ssh/aws/passwd-s3fs -o volname="ITSLTNS - S3"

The itsltns-s3.sh script uses absolute file paths for the s3fs command, location of the mount drive, and the location of the s3fs password file. I didn’t bother adding the path to the script.

The one additional piece you may notice in the above script is the osxfuse option to rename the attached drive with the -o volname="" command which makes the volume look prettier in Finder (replace in-between the quotes with desired drive name).

With the additions and launchd service created, my Mac was setup and ready to go on boot. I got it configured and everything was up to par.

Ubuntu - /etc/fstab - mounting on boot

Configuring Ubuntu to mount the S3 bucket was a little more challenging and I ran into some issues along the way, but it really only took me about 20 minutes, a quick google search, and one reboot.

S3FS offers an example for automounting using the /etc/fstab file, and I ended up with a similiar configuration to their example file. My /etc/fstab file had s3fs#mybucket /path/to/mountpoint fuse _netdev,allow_other 0 0 added on the bottom of it.

I ran sudo mount -a and sure enough the S3FS couldn’t mount because it had no idea where my developer credentials were. The sudo mount -a command spit out a response something like “cannot find access and secret keys.”

/etc/fstab requires root permissions (modified with sudo) and therefore it looks for a different password file, so I used a symbolic link to fix that. Using ln -s /home/jack/Documents/.ssh/aws/passwd-s3fs /etc/passwd-s3fs was the bandaid that prevented me from doing something dumb.

I re-ran sudo mount -a and sure enough it mounted with ease.

I rebooted my machine and all seemed to be in proper order.

Pricing

I haven’t run this for months on end, actually I just got it set up, but it looks like it will be around $2/month with all the requests that are made and the storage pricing. This was based off 16gb/mo and 600,000 total requests.

If I have to, I will revert back to manually mounting when I need to, but for now it is nice to have S3 auto mounted on login (for mac) and boot (for linux).

See AWS Pricing Calculator for more on pricing.

Notes & Todo

MacOS attaches attributes and metadata to files and it kinda sucks. File attributes can be removed with xattr -c filename, but I need to make sure every file in the bucket has them stripped. The reason for this is that Ubuntu is unable to read these attributes (different filesystems) and they also bug me.
After looking at this post, I need to move keys on both machines to similiar locations. Not fun managing two computers where ~/Documents/.ssh/aws and ~/.ssh/aws have similiar files. They should really be one or the other.
I will probably have to revise the s3fs code to allow for multiple s3 mounts as I definitely see it as a way to manage personal and work files between machines and aws accounts. I also might look into autofs with some custom scripts to get this fixed.
Permissions among the mountpoints on different machines may need to be looked at, but for now I am the only person using my machines so there is no security threat. knocks on wood
I also want to find a place on my Mac for launchd scripts. The ~/Library/LaunchAgents seems like a place where files could easily be placed and lost. Appears to be a place that can get messy quick.

React and Rails with Webpacker in Production

Jack Moore — Thu, 07 Mar 2019 19:09:45 +0000

I have written a hand full of Rails applications and typically I use the asset pipeline and coffeescript with bootstrap and the likes when deploying. Most of the projects are somewhat barebones from a front end perspective, but that's not to say they aren't functional from the server side. I don't usually have a hard time moving projects from development to production, most of the work ends up being trivial configuration details.

For my most recent rails project, I was looking to add a react form walk-through on sign up. I found a few ruby gems that worked with react and I decided to pick out reactjs/react-rails. I had no problem getting the development environment up and running, but when I tried to move the project into a test environment and then production, webpacker was giving me a time and a half. I ended up scrapping the project and moving forward to something new.

Anyone have similar experiences with dumping side projects? How much work do you usually put into a problem before cutting the cord and moving forward?

Everything I Learned from an Interview that Never Happened

Jack Moore — Wed, 06 Mar 2019 04:12:53 +0000

Interview Background

I was asked to interview with a large tech company, but the interview never took place due to timing issues and them moving forward with another candidate before I had the chance to interview in person. Truthfully I was neither qualified nor prepared for the job, but I saw no harm or foul with dropping my CV in the bucket and hoping someone saw it.

For a little background, I am getting ready to graduate school with a Bachelor of Science in Business Administration with a specialization in Management Information Systems, but the role they were looking to hire for was a Software Engineer. I knew going into the application process my degree would only get me so far and I would be required to learn a lot of the CompSci fundamentals on my own time to prepare for the position.

An internet search later landed me with millions of results and things to do. I started to feel in over my head before I had even begun preparation. This is everything I did, and all the concepts I learned in an attempt to prepare for a technical interview.

Baseline Coding Ability and Concepts

Coming from somewhat of a nontraditional degree for a Software Engineering position, I knew my technical bounds and coding ability would be tested. I don't spend every waking minute writing Python, Java, or *. I spend most of it working on things I want to work on (which is mostly infrastructure related) and looking at memes.

I have the fundamentals of Python down and I've built a couple smaller projects with it, so I ended up picking it as my language of choice for the interview process. This may have come back to haunt me in the end as with Python you have to roll most of your own data structures. I'll come back to this later though. For now all you need to know is I didn't have a problem with the first coding questions.

I didn't have a problem with the preliminary weed out problems, but I did prepare for them as much as I could using Leetcode.

Does it Scale (Big-O)

Something not covered in my degree but is in a CompSci is Big-O and the concept of computational time and space usage. I picked up Cracking the Coding Interview, read the chapter on Big-O and then I watched as many Youtube videos as I could on the concept to the extent I can now explain it to someone with a basic knowledge of programming.

Truth be told, I never ran into the issue of refactoring code to fit it within time and space bounds because the interview never took place. This is for better or worse, but I could have used the experience with a practicing Software Engineer.

Dynamic Programming and Recursion

Dynamic Programming and Recursion were tricky concepts for me. I was originally exposed to these in a high school Computer Science class, but past that class, these two concepts always turned into an after thought for me.

Never in my flask/rails/scripting side projects did I ever think of running a recursive method because most of the time I stored everything in a database and the code followed a flow.

Luckily, Youtube - Stanford Cs106B, Data Structures and Algorithms helped me to learn more about recursion and most of the basic data structures. I only watched a hand full of the playlist, but it helped me substantially.

For dynamic programming, I watched a hand full of shorter videos and applied recursive techniques to most of the problems.

Baseline Infrastructure and Systems Concepts

Writing code that scales is important to most major tech companies, but what's also important is the infrastructure that scales with systems as users join and use these systems. I didn't make it to the systems design portion, but the concepts were in my head and I didn't want to be caught off guard with a question I wasn't prepared for.

I prepared for these with whiteboard sessions and reviewing engineering blogs of popular tech companies. I enjoyed reading about the tools and architectures these companies use to build highly scalable systems.

Conclusion

Having only 3 weeks to prepare, I prioritized my time towards the concepts I thought I would be the most relevant for the position as well as the ones I was the weakest in.

When it came down to the end, I truly wish I had applied earlier for the position. Applying so late put me at a bit of a disadvantage from the beginning, but I wasn't disappointed with the results as I never set my expectations in the beginning.

Recommended Resources

Like I said earlier, everyone is only a search away from the answers they are looking for, but one must have to know the questions first. Here are the resources I found most valuable, but I would highly recommend picking up a book on the subject as well.

Github - Coding Interview University I would recommend going through the Github coding interview resource initially to prepare for a technical interview. It helped me immensely in grasping some of the basic Software Engineering concepts.
Github - The System Design Primer Even as I walked through the System Design Primer repo preparing for an interview, I think this repo would prepare me for building any type of system. It also has the links to popular engineering blogs of some of the larger tech companies towards the bottom.
Github - Fizz Buzz Enterprise Edition Excerpt from the Readme.md on github, "This project is an example of how the popular FizzBuzz game might be built were it subject to the high quality standards of enterprise software." I just think this repo is funny.

My experience with Development Cycles

Jack Moore — Thu, 01 Nov 2018 05:00:00 +0000

The first release of any my dev cycle feels like garbage. Whether it’s integrating a new api or library to building on an idea that’s not all the way flushed out, I often find my first iteration of a project gets thrown out.

This post describes some of the reasons why my first attempt on a proof of concept application isn’t a masterpiece.

Unfamiliarity

Being unfamiliar with tools and new libraries has forced me to read more documentation and learn more about deploying multiple types of applications. This unfamiliarity has ultimately made me look at projects in a new way.

The unfamiliarity with libraries has made my applications delicate rather than hardened, and I often find myself searching deep in github issues on projects attempting to fix mangled together parts of an application. By the time I am in production, most of the tools are mangled together and on the edge of breaking.

Library Versions featuring Brokenness

I often stay very up to date on library versions and pull most source code from repositories right off of github. With all these dependencies on other applications being tossed around, I sometimes find myself and my applications in a broken state because of dependency issues.

The most recent time I ran into an issue was upgrading a ruby project and running into some backwards compatability problems with methods. Methods were deleted that my application used and I essentially downgraded back to the older version to get the application back on track.

Perfection

I have a high standard for production applications, and I prefer to have most of my application tested before tossing it into the wild. It also helps to have a web app with decent views, so I tend to spend a lot more time working on frontend pieces than I would prefer.

This often holds the processes of releases up as I like to ensure the application works before I push it out the door.

Pipelines

I was bad about pipelines in the past and didn’t truly value them for what they do and how much time they save compared to scp-ing code to a machine and praying a deploy works and the libraries are the same. Looking ahead to containres, I now see the value in containerized releases. I am able to push my application to a container and run it on any machine. (Containers of today are the Java of Yesterday - Write Once, Run Anywhere)

DEV Community: Jack Moore

BGP and Cilium Kubernetes

Installing Helm and Cilium

OPNSense Side:

Links

- BGP Cilium 1.12.2

- Cilium in VMs

A Bit of Nginx Load Balancing

Containers and Docker Networking

Nginx Configuration

Where I ran into problems

Adding more servers to the upstream

From Here

Links:

Clustered Storage with DRBD

Logical Volumes

DRBD

Splitbrain

Inverting Resourcing

DRDB Broken?

Troubleshooting DRBD Issues

Odd Mountpoint with Loop Device

Links

Links 2

Links

Bug Reports

Less useful links

The Blog Post about the Blog

Reasoning for the Meta

Current Blog

Material

Writing Style

Community

Vim Configuration

Plugins

~/.vimrc

Bash Powerline

Getting Glyphs to work when SSHing from Mac to Server

Links

Metaprogramming Rails Helper Module for Accessible Attributes

What is this code and how does it work?

Links

E2E Modded Minecraft Server

Client

MultiMc

Setting RAM

Installing the E2E ModPack

Server

Ansible

Forge Installer vs Forge Universal

Turning the Minecraft Server into a Service

Tacotron-2 - Text to Speech, My Speech - Part 1

Picking a Library

Building a Dataset

There’s really just no chance

Refined Process

Dataset

Writing a Preprocessor

Training the Model

Looking Forward

Resources

Resources Final?

Mounting S3 as a filesystem with S3FS

S3FS

Easy Install

Credentials & AWS

Mountpoints

Issues & Logging

MacOS - launchd - mount on user login

Ubuntu - /etc/fstab - mounting on boot

Pricing

Notes & Todo

React and Rails with Webpacker in Production

Everything I Learned from an Interview that Never Happened

Interview Background

Baseline Coding Ability and Concepts

Does it Scale (Big-O)

Dynamic Programming and Recursion

Baseline Infrastructure and Systems Concepts

Conclusion