DEV Community: Mitch Pomery (he/him)

My Textable Cat Printer

Mitch Pomery (he/him) — Wed, 11 Aug 2021 04:39:08 +0000

I was scrolling on twitter one afternoon and saw that xssfox had a Bluetooth Thermal Cat Printer. I have had an idea that required a thermal printer for a while now and this looked like a really good printer for what I wanted to do, so I ordered one.

// Detect dark theme var iframe = document.getElementById('tweet-1420254881613451272-210'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1420254881613451272&theme=dark" }

The original idea I had was to set up a thermal printer so you could message it a picture, have it print off and stick it in a guest book, like you might do with a polaroid at a wedding. But It doesn't look like I will be going to a wedding any time soon. So instead, to allow people to interact with it remotely, I decided to give it a phone number and let people send messages that would print out on my desk.

To contact it, message cat:[message to print] to one of the below numbers to have it printed.
Aus: +61 480 029 995
US: +1 520 521 0228

Giving The Cat Printer A Number

To be able to give the cat printer a number, first I had to get it to print. I pulled out a raspberry pi I had from the time I had my apartment lights controlled using amazon dash buttons. xssfox had some code that allowed you to send messages and images to be printed via HTTP requests. After getting that running on my pi and debugging some differences between how python handles bytes between Windows and Linux, I was able to send my first messages to print!

The python code I am using is available at https://gist.github.com/mpomery/6514e521d3d03abce697409609978ede

// Detect dark theme var iframe = document.getElementById('tweet-1420599852061237248-210'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1420599852061237248&theme=dark" }

Once I had it printing, I went into my Twilio console and started adding some functions to my textable light project. I created a new function that would look at the incoming message for the prefix cat: and redirect the message to the cat printer. Later I went back and made my light require the prefix light: and an error message if neither of the prefixes were present.

The function looks like this:



exports.handler = function(context, event, callback) {
  let twiml = new Twilio.twiml.MessagingResponse();

  if (event.Body.toLowerCase().startsWith("cat:"))
    twiml.redirect("/cat_printer");
  else if (event.Body.toLowerCase().startsWith("light:"))
    twiml.redirect("/textable_light");
  else
    twiml.message(`"light:[colour]" to add a colour to the light
"cat:[text]" to print on the cat printer`);

  return callback(null, twiml);
};

I then set up a second function cat_printer to format the message to be sent to the printer. It takes the incoming message, removes the cat: prefix, redacts part of the number it was received from and formats it with a timestamp for printing. Then it makes a call to the web server running on my raspberry pi (which was made available to the internet using ngrok) to print it. Once that request has been made, the response from the printer gets sent back to the number who texted it.



const axios = require('axios');

exports.handler = function(context, event, callback) {

  let twiml = new Twilio.twiml.MessagingResponse();

let incomingMessage = event.Body;

  if (incomingMessage.toLowerCase().startsWith("cat:"))

    incomingMessage = incomingMessage.substr(4).trim();

let from = event.From;

  from = from.substr(0,3) + "*".repeat(from.length-6) + from.substr(from.length-3);

let now = new Date();

let printMessage = now.toLocaleString('en-GB', { timeZone: 'Australia/Sydney' }) + "\r\nFrom: " + from + "\r\n" + incomingMessage;

console.log(printMessage)

axios.post(context.catURL + '?text=' + encodeURIComponent(printMessage))

    .then((response) => {

      twiml.message(</span><span class="p">${</span><span class="nx">response</span><span class="p">.</span><span class="nx">data</span><span class="p">}</span><span class="s2">);

      return callback(null, twiml);

    })

    .catch((error) => {

      twiml.message(You sent a message with special characters or the cat printer is offline.);

      console.error(error);

      return callback(null, twiml);

    });

};

Getting People To Message It

Once I had it working I started telling people about it. Some of my friends had already seen my textable light, so the idea of sending a text message to something on my desk wasn't new to them. I was initially cryptic about what I had made, but once people knew what it was they started messaging it!

// Detect dark theme var iframe = document.getElementById('tweet-1420892406476468228-812'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1420892406476468228&theme=dark" }

Some of my friends made me regret telling them about the cat printer:
// Detect dark theme var iframe = document.getElementById('tweet-1421019519485571072-645'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1421019519485571072&theme=dark" }

I streamed it briefly so people could see what they sent print in real time, including this really long message:

Then I started to see several people sending it messages internationally, so I purchased a second Twilio number so I could respond to them:
// Detect dark theme var iframe = document.getElementById('tweet-1424959545302937601-107'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1424959545302937601&theme=dark" }

And then the TwilioDev twitter account tweeted it, expediting this blog post:
// Detect dark theme var iframe = document.getElementById('tweet-1425255647017324546-752'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1425255647017324546&theme=dark" }

Now that I have it built and it has a semi-permanent position on my desk, I need to make a better way to keep all of my projects displayed on my desk. I would really like to make them all visible on the internet permanently. I have ideas on how I could achieve this, but don't have access to what I need to make it possible or the space to do it at the moment.

And if you are considering making something similar yourself using Twilio Programmable Messaging, you can get $10 when you upgrade using this link.

Updating My "Get On The Beers" Indicator To Track Vaccinations

Mitch Pomery (he/him) — Sun, 08 Aug 2021 23:45:00 +0000

Over 6 months have passed since I first made this indicator light, and while it was good during previous outbreaks, the increasing availability of vaccinations in Australia means runs of 0 cases is less important. More important is getting jabbed! So I updated my light to track daily jabs!

The New Colour Code:

No data yet: White
Under 25,000 jabs in NSW today: Red
Under 80,000 jabs: Blue
Under 80,000 jabs: Green
100,000 plus jabs: Party Mode!

Finding The Data

Just like the original build, finding a data source to use was critical to making it work. Data NSW didn't have vaccination data available that I could find and I didn't want to go down the path of searching for different API's hoping that one would have the data I wanted.

The COVID LIVE website has a page with NSW Daily vaccinations on it in table form! There's also a nice JSON file with all the data that it's creator Migga told me about when he came across my indicator light the first time. The Electric Imp agent has a 2048kB memory limit though, so I can't load the full JSON file in. The page with the table on it will have to do!

The source code of the website makes it easy to extract the number I want to make the light colour decision on. I need to find the last column in the first table row (not table header) and understand that number.

Making It Light Up

I used the code from the previous version that tracked daily cases of known local transmission as the starting point. It has code on the device to set the lights colour based on the number it received. And the agent has code to make a web request and massage data in the response to determine the number and send it oto the device.

I started with the device code (code that runs on the hardware), changing the numbers for each colour. Then I reversed the order of the logic. This meant it would check if it was going to be party mode first, then work back towards 0 for red, and finally treat any other response (i.e. negative numbers) as an indication to show white. The main driver behind reversing the order of the code was to make it so I only needed to disable party mode once, instead of disabling it for each colour. Once done this was the device code:

Once I had the device code updated (and had tested it by feeding it some hardcoded data) I moved on to writing the agent code (code that runs in the cloud).

I renamed variables and functions to represent what the code would now be doing, then started writing code to parse the HTML that I was extracting the data from. To extract the data I:

Found the first line of the table rows
Found the end of the opening tag for the net change column
Removed everything prior to that
Found the first closing tag (which could either be the a span or a td tag)
Removed it and everything after it
Checked if there was still an opening tag that needed removing
Removed it if there was
Checked if all I had left was a "-" character (indicating that there was no data for the day yet)
If there wasn't, removed all commas from the string and turned it into an integer

I tried thinking of other ways I could simplify the code for extracting this data, but there weren't any libraries or functions I could see that would help. This was the agent code once I was finished:

Closing

Now that I have it updated and have tested that it is working (and fixed the edge cases I didn't think of when I was first coding it) I have it sitting somewhere a bit more visible on my desk, waiting for the day it first goes multicoloured.

Personally, I'm hoping it does that on tomorrow because I am getting jabbed today (and that's when today's numbers will be shown).

"Get On The Beers" Indicator Using Electric Imp

Mitch Pomery (he/him) — Mon, 04 Jan 2021 00:06:03 +0000

While I was building my friend an On Air Light that turns on when streaming, I thought it was really cool and wanted to build my own indicator light. I don't stream so had to think of something different to use the light to indicate. I saw this tweet in my twitter feed and it gave me an idea!

// Detect dark theme var iframe = document.getElementById('tweet-1340944007652753411-690'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1340944007652753411&theme=dark" }

I could use a light to indicate if there was community transmission of COVID-19 and if it was safe to go to the pub! The colour of the light would indicate how long since the last case of community transmission:

Local transmission -> Red
First week of no local transmission -> Blue
Second week of no local transmission -> Green
Further weeks -> Multi-Coloured Rainbow

Finding The Data

Finding a reliable source of data for this project took a lot of research. The data I am looking for is community transmission in New South Wales, Australia. There are an abundance of COVID-19 APIs available, almost all using data from the John Hopkins University CSSE COVID-19 Git Repo. This dataset does contain NSW data but does not contain information about community transmission, so any API that only uses this as its data source is out of the question.

Data.NSW has a dataset published for COVID-19 Cases By Likely Source Of infection, which could be used to determine how many days since the last case of community transmission, but this requires processing to understand. They also have some other datasets with similar information, but I was unable to find one that gave days since last community transmission as a value.

Then there is COVID-19 in Australia Data with beautiful graphs and tables visualising COVID-19 data in a variety of different ways. One of their visualisations is how many days it has been between different milestones, split by state. If I could access the data used for the visaulisations I could use it for my visualisation.

They use Datawrapper to do the visualisations and by looking at what Datawrapper does in the browser, I was able to see that it loaded a CSV from their CDN with the data in it. I can directly access the CSV files, and after checking multiple days I could see that the filenames did not change!

The CSV format used is simple. It has a single header row, then the data rows. The data rows start with the state, followed by three integer fields for local, overseas and active cases. CSV is a nice file format because it can generally be easily processed without the need for a library to understand it.

,Days since a new local case,Days since a new overseas case,Days since any active cases
...
NSW,0,0,0
...

Coding the Electric Imp

Now that I know I have the data, I can start working on the code. For this project I'm using an Electric Imp impExplorer Developer kit, meaning I don't need to do any electronics. After performing a BlinkUp, getting familiar with the impCentral IDE and creating a new project I was ready to start writing some code.

The Electric Imp platform gives you both a device and an agent that you can run code on. This lets you do power/resource intensive processing on the agent and send instructions as a result to the device. It also means that as long as the device can communicate with the agent, the agent can reach out to the internet. Conversely from the internet you can talk to the device via the agent. For this project, I want to process the CSV on the agent, send the result to the device and have the device do something as a result.

Turning the LED on

The first step in building the light is to get the light on the impExplorer to turn on and log that it has started. To do this, we need to include the WS2812 library, set pin 1 to be a digital output on the ImpExplorer, then configure and use the library to turn the LED on.

Logging on the device can be achieved by using server.log, which will log to impCentral.

Device Code:

Device To Agent Communication

Now that the LED is turning on when the device starts, we want to use device to agent communication. Electric Imp lets us do this by registering functions that will process a message from the device the the agent and vice-versa.

On the agent you use device.on to register a function to handle messages from the device. On the device you use agent.send to send the message to the agent. Using these two functions lets us talk from the device to the agent.

Agent Code:

Device Code:

Agent To Device Communication

Now that we have the impExplorer asking the agent for an update, we want the agent to respond. Similar to device to agent communication, there is agent.on to register a function to handle a message on the device and device.send on the agent to send a message to the device.

Agent Code:

Device Code:

Colour Decisions

Once we have the device asking the agent for an update and the agent responding, we can start programming the device to have it respond to the messages sent from the agent. The function we wrote to handle the messages from the agent has to have a parameter for the data that the agent sends. This data could be any object. Since we are sending days since the last community transmission, we can send a single integer and make the colour decision on the device from that.

Device Code:

Polling For The Latest Data

Our indicator light is working! We can manually change the data we are sending to get the light to change colour to test out the different colours.

Something we haven't done is make the device poll for updates. It will get the colour at power on and not update until a restart. We don't want to have to restart the imp regularly, so we need to create a loop.

We could use a while(true) loop, but doing this would prevent the imp from going into low power modes and drain the battery quickly (if run on batteries). Instead we can use imp.wakeup to create a timer that will execute a function when the timer fires. If we put this in the requestUpdate function and have it call requestUpdate we create an infinite loop that allows the imp to go into low power modes.

Device Code:

Getting The Correct Data

Now that we have the device polling for the agent for updates and updating the colour of the light based on the response, we need to update the agent to get our CSV, process the data and then send the our response.

Agent Code:

Working Code

We're nearly done! We have the device polling the agent and updating the lights, and the agent will reach out and get updated information when asked.

Agent Code:

Device Code:

Adding Multicoloured Lights

The light going white isn't as flashy as we want for the big achievement of reaching two weeks without any cases of community transmission. For that we want the light to go change colours! We can do this by making a Rainbow Mode! When in rainbow mode we have the light change colour slightly, travelling around the colour wheel.

Device Code:

Housing And Mounting It

Now that the indicator light is working, we need a way to display it so that the information is understandable. Otherwise it's just a bare board and the light could be indicating anything. For this project I drew up a box that the imp could sit inside in CAD and put text on the front of it with the indicator light to explain the status.

Keeping with the Dan Andrews "Get On The Beers" theme, I went with the following:

Red - The Pub Is Shut
Blue - Case Last Week
Green - Can't Be Any Clearer
Multicoloured - Get On The Beers

Using the laser cutter at my local makerspace I was able to cut out the box and glue it together, making sure not to glue the top so that I could still access the imp inside.

Finding somewhere to put it around the house was difficult. I wanted it somewhere where I would see it every day, but not somewhere where it would be visible to me for most of the day. I try to make sure I head out for a walk at least once a day around lunch time, so decided to mount it by the front door, plugged into an extension cord so I don't have to worry about battery life.

Reflection

I had a lot of fun building this indicator light and writing about it and got to learn more about the laser cutter I get to use. The more I read about Electric Imps, the more I see how I could make improvements to the code I have written. I am going to need to order more of them so I can use them in more projects.

I have been thinking about what I could an indicator light for when COVID-19 community transmission is no longer a major concern in my area.

I could update it to be a visual clock that indicates morning, lunch, afternoon, when it is time to head to the pub, and bedtime with different colours instead of the actual time.
I could swap the front panel from the box and make a busy light for meetings like Christine Sunu has done.
I could send it to the office with my colleagues in another state and use it as a presence light so they can see when I am at my desk and available.

The possibilities are endless!

How I made a light that turns on when streaming

Mitch Pomery (he/him) — Thu, 24 Dec 2020 02:32:57 +0000

I asked my friend Ben what I could get him for Christmas this year and he said that I should surprise him and make something. When I was looking at my list of projects thinking what I could make him it hit me: I had an idea of making an indicator light for our apartment so my partner could see if I was in a meeting when getting home from work¹. I could do something similar for Ben and make him an "On Air" sign that turns on when he starts streaming!

Electronics

The first thing I needed to figure out was the electronics. I had a couple of NodeMCU boards that have been sitting in my box of random things for years that I had yet to play around with. I could use one to act as a webserver and turn LEDs on and off using some webhooks. I pulled one of them out of the box, opened it and started looking at the documentation for them.

A lot of what I did to get the board set up is outlined in the tutorial I followed. Getting started with the code was straight forward and got me really far really quickly. At the start I needed to:

Connect the board to my computer
Flash it with some default firmware
Connect to it using the ESPlorer software
Write some basic code to connect it to my Wifi
Write some code so it could handle web requests

That got me to this point with the code:

After I had it connected to my WiFi and acting as a webserver I started running into problems. I couldn't get the HTTP Server Module to load, so didn't have access to nice functions for understanding the requests and responding to them. I decided I would keep it simple and do basic string manipulation to understand the requests and some basic error handling.

I also tried to use Pulse Width Modulation to control the RGB LEDs and had some difficulty understanding the documentation. Once the LEDs were lighting up and controllable I noticed that they were flickering. I stopped using pulse width modulation to control them and used a binary on/off instead. I would have liked to have more control over the colour of the LEDs, but the strip I had bought was fairly dull and didn't look that great even at full brightness.

The end result is some functional code.

I don't think I will come back to this code (or my other NodeMCU boards) any time soon for other projects.

Getting OBS to trigger webhooks

I found getting OBS to trigger webhooks was easier than I initially expected. I searched for "OBS Trigger Webhook" and came across an OBS plugin for sending webhooks to discord that made me aware of OBS Scripts. I used Parik27's code to figure out the minimum code I needed.

This was a good starting point, but the script wasn't outputting anything or running. Time to look at the OBS Scripting documentation. Reading this I saw what was causing my problems listed as a note.

NOTE: On windows, currently only Python 3.6 is supported.

After installing Python 3.6 and setting the Python install directory to point to it, my script was working!

Now that I had it logging out on events, triggering webhooks was next. Adding in a urllib request on the events hardcoded to the IP Address of the NodeMCU board was going to be more than sufficient for this project.

I want to come back to this code in the future, parameterize it and add in more events that you can trigger webhooks from.

The Housing

A spaghetti of electronics isn't exactly a nice sign that you could proudly have sitting on your desk, so I needed to create a housing for it. Fortunately for me I have a Cricut that I can use to cut text out of vinyl with, and my local makerspace has a laser cutter I could use to make the housing.

Cutting the text on the Cricut was simple. My partner had already pulled it out of the cupboard so I just needed to plug it in, put the text into a project in Cricut Design Space, load my material and then press cut. The machine had been in the cupboard for almost a year, and if it was still there I probably would have painted on the text or engraved it instead.

It took me two attempts to cut the housing out with the laser cutter. The first attempt failed because I had the speed too high and the laser didn't cut all the way through, and I moved the origin so I couldn't do another pass². Attempt number two was better and I had a housing!

Future Improvements

It's always good to look back on a project and think about what could be done differently next time.

Things I have thought about:

Putting the source code into version control. It would make it harder for me to hard code values because they would be publicly visible. It would also make it easier to go back when I made a mistake and revert to code I knew worked, instead of needing to fix forward.
Using a different board. The NodeMCU Boards are really nice for playing around with, but I'm giving this to someone else. I don't want to have to reprogram the sign when he moves house so I can connect it to his Wifi.
Not needing direct network access. This would make it easier to set up and would allow me to make it go into a party mode remotely.
Better LEDs. The ones I used where the cheapest 5V LED strip I could find. They aren't individually addressable so you can't do cool patterns like rolling colours. Either the NodeMCU board or the LEDs themselves didn't like PWM, which limits the range of colours and patterns I could do. At full brightness you can also see the distinct red, green, and blue, and they aren't as bright as I would have liked.
A better housing. The housing I made works, but having to open it up to access the board is annoying. Plus having the board and excess wiring inside it gets in the way of the LEDs and casts shadows, though better LEDs could make this a non-issue. I also didn't have access to any frosted acrylic, so did what I could with some non-laser safe acrylic I had, cutting it to size on the bandsaw and leaving the plastic protecting sheets on the backside to act as an opaque layer.

I had a different development board arrive right after I finished this, which is a better platform to work from. I have made version two, which I will give to Ben and write a follow up guide if you want to make one yourself.

Delivery

Getting this to Ben and setting up OBS on his machine to use it wasn't as straight forward as I would have liked. I had to talk to his dad to get permission to add it to the network. His dad gave me permission to add it to their guest Wifi, but I need direct access to be able to call the webhooks. After talking to his dad some more and explaining what it is and how it works, he gave me the details for the main network so I could program it in. If I didn't need to upload the program to change the network it connected to, I could have given it to Ben without the details programmed in and gotten him to do that himself.

Delays caused by unexpected car trouble and an outbreak of Covid19 in Sydney mean that I didn't get to give this to Ben when we had initially planned to catch up, so instead he got the improved version two that I haven't written about yet³, and he loves it! Version two is easier to reprogram to connect it to wifi, but we encountered issues when doing that. The new board I started using doesn't like connecting to Channel 12 or 13 which it appears Ben's home network was on (it was set to auto). It took around 40 minutes of me and his dad going back and forth trying some different settings to get it to work. Ben also uses Streamlabs OBS, which is different to OBS Studio and doesn't have the scripts menu or something similar. Instead of automating it, I set it up so that he has a simple UI that he can use to control it.

I got my idea from Christine on the Twilio Remote Participants stream when she talked about the light to indicate of she is in a meeting she made. This stream has been something I look forward to while I haven't been able to travel and see friends and I still think Phil should consider the pull request I made to the video chat application he made on his stream and add the "CRIME" reaction to it. ↩
I found out on my next makerspace visit that the lens and the mirrors had a light coating of smoke and were absorbing some of the lasers power, making it cut less. I now know how to check and clean these to help maintain the laser cutter. ↩
I am planning on writing about the version two build as a guide that you could follow, with a different trigger for the lighting (I'm thinking something data driven). ↩

The dangers of choosing the wrong identifier for your users when federating logins

Mitch Pomery (he/him) — Mon, 27 Jul 2020 06:09:43 +0000

Preface: I was going to title this something along the lines of "Twitter's username clean up is dangerous", but I felt that made it sound too much like what twitter was doing was the dangerous part, and not that other applications failing to follow good practice puts their user data at risk, regardless of what twitter is doing. I also started writing this and then promptly forgot about it for 8 months.

If you haven't seen what twitter did, they removed inactive accounts to free up the usernames. On top of the the issues that others raised with this, freeing up twitter handles made it possible for someone to potentially get access to data that is not theirs.

There's a lot of maybes there. I don't think this has happened, but it is theoretically possible. It is not because of a mistake that twitter has made. It is possible because people changing usernames could uncover issues where a third party application is not linking accounts from twitter as an identity provider correctly.

Example

If you use twitter as an Identity Provider for your application, you have to use a piece of information twitter gives you to link between a twitter account and your application. If you use their twitter username (or email) as this key between your user will experience issues if they change it. If someone else starts using a previously used twitter handle they also gain access to the old users information in your application.

Instead of using a username or email address between you application and an identity provider, which could change, you should look at a field that can't change. If you are using OIDC, you should use the subject claim (and other claims that don't change).

If you think that users information doesn't change, between starting this blog post 8 months ago and finishing it, I lost access to data in one application because I changed the email address I use for Facebook.

My First Website (A look back at what I did at the start of the decade)

Mitch Pomery (he/him) — Wed, 27 Nov 2019 03:18:20 +0000

At the start of the year I was planning to move across the country and needed to get rid of things. One of those things was my old HP Microserver that I used as a NAS. I couldn't remember putting anything important on it that I wanted to keep, so sent it with my parents to my brother so that he could put it to use. Him being the amazing person he is, checked to see what was on it before formatting it and found some of my personal files from 2014. The other night we finally got all of those files across the country and back to me, and spent some time looking through them together.

Among them is the code from my first website. Unfortunately I didn't know how to use version control back then (though I had installed mercurial server and did have a Bitbucket account), so the revision history has been lost. Fortunately though, I did keep some screenshots!

Time for some archaeology!

I started playing with web hosting around 2009 while I was in high school. I had an old Dell desktop, that one of the computer technicians at school gave me, sitting in the corner of my bedroom that I used as my server. A network cable ran out of my bedroom door, across the doorway for the bathroom and plugged in to a switch at the family computer desk that I had bought.

I set it up a WAMP server (Windows, Apache, MySQL and PHP), configured our router to run DynDNS and started hosting a proxy on it, so I could get around the school firewall. The IT Department knew I was doing this, and never blocked it, but they did talk to me about it and tell me that I shouldn't be bypassing the school's proxy.

In 2010 I got bored of just running proxies and wanted to build my own website, so I started by looking at different websites that existed and drawing designs that I liked. I then wrote all the HTML and CSS for the layout in Notepad++, relying on search engines to help me learn the syntax. I didn't want to have to update HTML files every time so I built my own CMS system, including the login screen, myself. This was a major security mistake, as one of my friends was able to deface my website within a week.

In 2011 I realized I didn't want to have to manage my own web server, so I bought a domain and paid for some web hosting with my first ever debit card. And with that I decided to do a complete website redesign. And I have pictures of it!

I won't explain each image, but you can see the progression in the images below:

I also kept my CMS system and started updating it to use a database in the backend instead of the flat files I had been using. This was great for making it easier to add new pages, but as a result I lost all of the content from my website, because I never took a database backup.

My home server ended up becoming a minecraft server for me and my friends, and I ended up working on some other projects (including building a job management system). I will have to do a deep dive on some of those projects, and put the code into public source control so that others can learn from my mistakes.

Footnote: If any one of the ten other active CaL members from 2010-2012 reads this, hi!

My Graveyard of Abandoned Posts

Mitch Pomery (he/him) — Thu, 07 Nov 2019 00:27:35 +0000

I've started a heap of draft posts here. Some are just just post titles, others are starting to get fleshed out. Before I go and delete them to clean up my dashboard, I thought I would list them out here with mini descriptions for what it was supposed to be about.

My Journey So Far

A timeline starting when I started to get into technology and acquired my first PC (from dumpster diving). It would talk about the communities I participated in, the jobs I have had and the different skills I have developed. I think I will keep this in my dashboard and work on fleshing it out more.

101 ways to do passwords wrong

A listicle on everything to do with passwords that people (developers and users) do wrong. Number 5 will shock you!

Startups Guide On When To Use "X"

Basically a long form version of this tweet thread.

Chief Problem Solver Mitch

@mitchpommers

@meacod @evanderkoogh When you stop talking about needing something and start talking about the tradeoffs and benefits of using it instead of something else

00:39 AM - 18 Jun 2019

"My Door Is Always Open" - getting feedback

I dislike the phrase "my door is always open". It puts it onto another person to provide feedback, when you really should be going out and seeking feedback. For other people to really feel safe providing feedback, they need to see that feedback is listened to and something happens as a result of it. Basically, if you aren't getting feedback on something, go looking for it!

The "Pipeline" Problem

This is another post i want to work more on and finish, especially after how one of my friends put it the other week. Claiming that the reason you don't have diversity in your team or organisation because there aren't enough people interested is a hand-wavy way of saying it's not your problem. What about the people you already have, how do they feel, are you providing them the opportunities to learn and grow?

This is another post I will keep around to flesh out.

Managing Expectations

At the end of last year my graphics card died. No big deal, I just took it back to the shop I bought it from to get them to fix it under warranty. What followed was 3 months of me chasing them up to find out what was happening with it. If all they had said was "we expect an update in 4 weeks", I would have been a lot happier with the service.

In the day of the T-Shaped Engineer, the resume should be dead

I think I could confidently say I have had the responsibilities of over 15 distinct job titles during my 10 years working in tech. All at only 5 companies. It doesn't map nicely onto a 2 page resume. I have to be selective about what skills I do and don't include. LinkedIn helps a little, but there really needs to be a better way to convey this information.

At some point in time, you will need to trust your engineers

I've worked for several people who don't trust anyone who works for them. At some point in time they need to trust the people they hire otherwise no work will get done.

Who is in your team and what do they do outside work?

A question I like to ask in interviews. Do the people hiring really know the people in their team and what they do out of the office? I don't need to hear something deep, just some hobbies or holidays that their teammates have had. It helps convince me that you can bring your full self to work.

Who is after your data?

A discussion about different threat actors.

Use Simple Patterns Not Complex Code

I've seen some really complicated libraries that were supposed to make development easier for other teams. Instead of that you could define simple guardrails and guidelines and let the teams work within them, trusting that they will do the right thing.

Community Building

My experiences in gaming communities, student clubs and running events. The importance of getting the right people involved at the start so you grow organically with the right values. The importance of clearing the path and letting doers do and training them to lead, instead of forcing them to.

My past projects

Things I have built myself. I really want to write this, but all the files I want for screenshots and to jog my memory are stored on my brothers machine on the other side of the country and he hasn't sent them to me yet.

Security is everybody's job

Listicle of things that everyone is able to do to improve the security of the systems they are building.

The pit of success

Making it hard for people to fail. Alternatively, making them fail into the correct place.

How to be lazy and get paid for it

This is something I've been working on for over a year. It's a piece about reducing the amount of work you have to do through automation, simplification and just stopping doing it.

Minimum Viable Engineering

Engineering standards and how they relate to MVP's.
Liquid error: internal

Simplicity is a feature

I remember reading a blog post about a timetable app and how they asked users what features they wanted. With over 50 new feature requests, adding all of them would remove the one biggest feature of the app; It's simplicity.

Build versus Deploy

I've seen teams that treat these two topics as one. There are clear boundaries between the two but if you're not careful you can accidentally merge them into one, which makes it harder to do some things.

The Cost Of Volunteering

I've volunteered at a lot of events this year and wanted to write up what it cost me to volunteer, because it definitely was not free. I need to start tracking my volunteering expenses better, but this year it has cost me at least 3 tanks of fuel and a day of parking in the city.

How Computers Shuffle - Explained with Cards and Dice

Mitch Pomery (he/him) — Mon, 04 Nov 2019 09:20:36 +0000

Introduction

This is meant as a hands-on way of looking at different shuffling methods and the steps a computer takes to do them. It is meant as a way of introducing the ideas of algorithmic efficiency without talking about Big O notation. For more detail about the performance of these algorithms, or for further reading, I suggest Mike Bostock's blog post on the Fisher-Yates Shuffle.

Materials

6 different cards
Paper with 1 to 6 written on it
Dice
Paddle pop stick

You can see the materials I used in the header image for this post.

Note on Dice Rolls: At times we don't want to roll a number between 1 and 6. Instead we want a number between 1 and something less than 6. To do this, we can roll the dice and ignore results that we don't want and instead roll the dice again. Alternatively, you can do what I did and make 5, 4, 3, 2 and 1 sided dice by putting crosses on the excess sides.

How Do Humans Shuffle

There are several techniques that people use to shuffle. Two techniques that you may have seen or used are smooshing and the overhand shuffle.

Smooshing - spread all of the cards out on the table and move them around pushing them into each other. Keep mixing them for a minute. Gather them up into a single pile.

Overhand Shuffle - Hold all the cards in one hand, grab some and then drop them back into the deck. Keep doing this until you're ready.

Numberphile has a great video on different shuffling techniques and how long you need to shuffle for to ensure that the deck is in a random order.

How Can Computers Shuffle

Computers don't have hands that they can use, or physical cards to move around. So how can they shuffle?

Lazy Naive Shuffle

Process:

Start with a deck of cards laid out and an empty deck to put the shuffled cards into.
Roll a six sided dice
Check if the card has already been shuffled
If it hasn't, move it to the shuffled pile
Repeat until no cards are left to be shuffled

Lazy Naive Shuffle Questions:

How many times did you need to roll the dice?
What happens if you never roll a six?
What would happen if you were shuffling a deck of 52 cards? How many times would you roll a number that had already been shuffled?

Better Naive Shuffle

Process:

Start with a deck of cards laid out and an empty deck to put the shuffled cards into.
Roll an n-sided dice (n is how many cards are left to be shuffled)
Move the card in that position to the shuffled pile
Move cards left one at a time to fill in the now empty space
Repeat until all cards are shuffled

Better Naive Shuffle Questions:

How many times did you need to roll the dice? Why was this?
What would happen if you were shuffling a deck of 52 cards? How many cards would you need to move if you rolled a 1?
What are the advantages of this approach over the Lazy Naive Shuffle? What are the disadvantages?

Fisher-Yates Shuffle

Process:

Start with a deck of cards laid out with a marker at the end to show which cards have been shuffled.
Roll an n-sided dice (n is how many cards are left to be shuffled)
Swap the card in that position with the last unshuffled card
Move the shuffled marker forwards one space
Repeat until no cards are left to be shuffled

Fisher-Yates Shuffle Questions:

How many times did you need to roll the dice? Why was this?
What would happen if you were shuffling a deck of 52 cards? How many times would you need to move cards if you rolled a 1?
Is this still shuffling because you are swapping cards instead of moving them to a new deck?

Answers/Discussion Ideas

Lazy Naive Shuffle:

You likely needed to roll the dice more than six times, unless you got really lucky.
If you never roll a six, you will never shuffle all of the cards. The same happens if you only ever roll a six. These situations are unlikely, but possible.
After half the cards (26 cards) have been shuffled, there is a 50% chance of rolling a number that has already been shuffled. Shuffling more cards gives you more chances to roll the same number twice making shuffling slower the more cards you start with.

Better Naive Shuffle:

You only needed to roll the dice six times. You could do this because now instead of rolling to fill a space, you would move the cards to fill the space.
If you rolled a 1, you would need to move every single unshuffled card forward one space before rolling again. You end up needing to move most of the cards several times while shuffling. If you only ever rolled a 1, then you need to move the last card 52 times, the card before it 51, then 50 and so on.
This approach is better as you only need to roll once for each card. It is worse because now you need to move cards multiple times.

Fisher-Yates Shuffle:

You only needed to roll the dice six times. You could do this because you were always rolling top pick one of the unshuffled cards.
If you rolled a 1, you would only need to swap the first card with the last unshuffled card. This means that for a deck of 52 cards, you only swap/move cards 52 times.
This is still shuffling. Every time you roll the dice there is an equal chance that one of the unshuffled cards will be picked and moved. They do not need to stay in the order they started in and they do not need to start in order. See the below for an example of items being shuffled that have no clear order.

My First Experiences Deploying Things

Mitch Pomery (he/him) — Mon, 14 Oct 2019 23:37:25 +0000

This is my response to the following tweet, asking about developer's first experiences deploying something.

Hayley van Waas

@hayleyavw

Devs, do you remember your first experience deploying something? Share your experiences, I'd like to share it with my students :)

11:38 AM - 14 Oct 2019

Very First Personal Websites (2010)

What: Several PHP websites, built completely from scratch.

Why: I wanted to learn how to make dynamic websites.

Deployment Process:

Collect an old PC from school
Wipe it and install Windows 7 onto it
Set up WAMP (Windows, Apache, MySQL, PHP) Server
DynDNS set up on our home router
.tk domains hiding the DynDNS domain

Things that went wrong:

The network cable rang across the bathroom door which made my parents unhappy with the trip hazard
WAMP was hard to set up and run
OS Patching? What's that!

C# IRC Bot (2010/11)

What: An IRC bot that I wrote without any libraries that could sit in a server with me and around 5 other people and their bots. All it would do is respond to commands (and log messages).

Why: Everyone else in the server had one and I wanted to learn some C#.

Deployment Process:

Compile code on my machine
Copy it to the server from before
Run the executable

Things that went wrong:

It would occasionally crash and I would need to start it again
Still not OS patching

Stocktake System (2011/12)

What: A rewrite of my works stocktake system, with the aim of making it easier/faster.

Why: Stocktake would take an entire day because you had to print off a sheet, find all the items that we had and mark them off, and then go and select all the items that didn't exist on the web interface and select them. New system was to enter the IDs of every item we could find, and then press "complete stocktake" which would take all of the items we didn't find and mark them as lost.

Deployment Process:

Write code on the in-house developers machine
Copy it to the web server
Test it

Things that went wrong:

Accidentally screwed up the entire production database (Thank goodness I took a backup first)
No one else wanted to use the software I wrote even though it was faster (writing it, testing it and doing a stocktake only took me a Saturday morning)

Websites for myself and others (2012-2015)

What: Several websites (static and PHP)

Why: People wanted websites and I knew how to make them.

Deployment Process:

Set up an account with a hosting provider
Configure databases in cPanel if I needed them
FTP the code to the server
Check it all
Quickly fix anything that I needed to fix
Zip up the website locally and keep it as a backup

Things that went wrong:

One hosting provider I was required to use didn't stop you from being able to see others websites code (and therefore their database passwords) so I had to convert one website from wordpress to static. All 300+ pages of it
I would occasionally have links set wrong
To save money on hosting providers I was writing PHP to split all my domains to different folders on the hosting provider. And that would regularly have edge cases

How I do deployments now

Looking back all of my deployments were manual, poorly tested, and very low impact if something went wrong (with the exception of both 300+ static page sites). Last week I wrote about one deployment I did, and so much has improved in them. You can read about that below:

Halloween Story: The Friday Deploy

Mitch Pomery (he/him) ・ Oct 3 '19 ・ 5 min read

#spooktober #devops

Halloween Story: The Friday Deploy

Mitch Pomery (he/him) — Thu, 03 Oct 2019 23:35:37 +0000

Note: While it does draw inspiration from several real events this story is a work of fiction. After the story are some of my thoughts on Friday deploys.

It's Friday, 10AM. Standup has finished and I'm looking at the log files on our test servers. Hang on, our log files aren't rotating. Is this happening in production? I look. This is bad. The disk is going to be full before Monday unless we do something. I call over Bethany and Thomas and show them what I've found.

We call incident management to tell them about the issue. They tell us to do what we have to to fix it and end the call. We agree that we should aim to fix this before 11:30. We have a team lunch that we don't want to miss.

Thomas starts raising a change request. He needs to find someone from another team to approve it and finds Alexandra. She has reviewed our changes before and knows how our team works. He tells her that we will be sending her a change for approval around 11.

Meanwhile Bethany and I are investigating the issue. The config changed and so stopped rotating our log files. It will be a simple thing to fix. How long has this issue existed? Were we lucky to have not had issues before? We look back at our change history and find it. The issue has existed for 3 months.

10:30AM. All we need to do is change one line in a config file. I make the change. Our build system kicks off.

Tests run.
Package built.
Deployment initiated.

We watch the deployment to our Test environment. All tests pass. Test environment deployment completed. Now to our Stage environment.

I click "Deploy To Stage". Our deployment script raises a change request and The deployment steps kick off. All the tests pass. Stage environment deployment complete.

11AM. Thomas links the stage change to our production change and goes to submit it. "Change must be more than 10 minutes in the future". None of us have ever seen that message before. Thomas sets the change to start at 11:12 and submits it again. Bethany messages to let Alexandra know. The change gets approved almost immediately.

Incident management calls back.
"How is everything going?"
"Good, we're waiting for our change window to start"
"Why did you do that? You can get approval later."
"Our normal process is fast enough so we decided to do it."
"Oh, okay."
The call ends.

It's time for our change window. We hit the deploy button and wait.

The deployment kicks off.
We wait.
The tests run.
We wait.

Tests pass. Change complete! We head to the lifts to leave the office for our team lunch.

Incident Management calls me at 12PM. They came to see our team and couldn't find us and wanted to know how the incident was going. I let them know that we are out for a team lunch and the issue is fixed.

2PM. Team lunch was great. I had the lamb shank. Everyone headed straight home afterwards. I lived close to the restaurant so am home first. I log into work and close the change.

My Thoughts On Friday Deploys

This isn't a scary story. Everyone is happy with what is happening and nothing goes wrong. This is what changes should be like. Deployments shouldn't be scary and dangerous. They should be something that is normal and boring.

I have worked in a team where everyone worked hard to build confidence in our pipelines and processes. We were always looking for ways to build more confidence. We wanted to stay at the point where we could diagnose issues quickly and deploy fixes even faster.

We didn't start there.

We started with our deployments being scary and dangerous. We did them at 7PM in the middle of the week so that if any issues occurred they wouldn't cause a major impact. Our deployments would fail without warning. Doing the same thing again would work.

One change we did started at 1PM on Valentines Day and should have finished by 2PM. Every time we deployed, the new version would fall over. We would look at it, it would be a pipeline problem, so we would try again. We extended the change window 3 times and eventually had to call someone from the networks team for help. At 8PM we finally had a successful deployment. My valentines day plans disrupted, my workmates evening gone, and the network engineers picnic date disturbed.

After this, we worked to improve our deployments. Infrastructure was causing the failures, but the application was what we were changing. So we split our infrastructure from our application. This gave us an immediate improvement.

We slowly built more confidence.

Infrastructure deployments still failed too often, so we investigated. We found that the timeout was too low, which caused the deployment fail on a random step. So we looked at what the timeout should be and gave it a buffer. That made our infrastructure deployments more stable.

We looked back to our application deployments. We had integration tests, but they had to be manually run. So we automated them. We looked at what tests we had and saw places where we could add more. Everyone in the team worked to write these tests together to give us all more confidence.

We found holes in our monitoring and alerting. There were metrics that we cared about that we couldn't see. So we made them easier to see. And we added alerts on them to tell us when they weren't right. More confidence.

We looked at the entire process from beginning to end. We could make a change and deploy it to our stage environment the same day. Moving that change to production would then take a week due to change processes outside the control of our team.

We spoke to change management to find out what we could do to reduce the lead time. They came back with suggestions to template our changes and re-evaluate the risk classification. We looked at how we were classifying our changes and found we were erring on the side of caution. Our changes were safer than ever before. We standardized our steps and created a template. The change management team removed the lead time after we started using the template.

We talked to the people approving our changes and found out what they cared about. We were doing changes so often and they weren't failing. Having to approve our changes became a pain for the other teams. They wanted to be removed, and only added if we thought the change we were making required extra scrutiny, so we removed them. We had built enough confidence in our system that they were happy.

After a year of slowly building confidence, we were finally at the point where we could make a change and have it safely deployed to production the same day.

And then we did have an incident where we had to make a change and deploy it the same day. And we did leave as a team for team lunch as soon as it finished.

But we weren't done. There were always things we could find that would build our confidence. We investigated contract tests, performance tests and UI tests. We weighed up the effort to implement them against the confidence it would build. Where it made sense we added them to our pipelines. There is always room for more improvement, but we were at the point where we would happily deploy on a Friday.

Should you deploy on a Friday?

If you've got the confidence in your build and deploy pipelines, go for it.
If you don't, go build some confidence.

What would it take to increase the password max length limit?

Mitch Pomery (he/him) — Sun, 22 Sep 2019 22:26:38 +0000

In the last post we established that password maximum length limits are good if they are set at a reasonable limit. They enable testing to ensure that passwords can not be set to longer than your systems (infrastructure, libraries, hashing algorithms and your own code) allow. Lets look at the restrictions used by some Australian Banks and determine what steps to take to bring them in line with the OWASP recommendations for password length.

To start we need to know what the current rules and limits are. I found the length restriction for 8 banks and have summarized the information below.

Password Length Limits

Bank	Input Field Limit	Password Limit
ANZ	16	16
Bankwest	16	16
Bendigo Bank	8	8
BoQ	255	20
Commonwealth Bank	16	16
Macquarie	Unlimited	256
Suncorp	Unlimited	8
Westpac	6	6

Note: NABs password tips page flexes on the ability to set a 17 character password. As I was unable to find their password rules I have left them out from this list.

Grouping Them

We can split the banks into several groups:

Those that have a low max length and a limited input length:

Bankwest
Bendigo Bank
Commonwealth Bank
Westpac

Those with a low max length and a higher input length:

ANZ
BOQ
Suncorp

Macquarie Bank, which is already in a good position

Going from Low to Woah!

To increase the length limit we are going to need to make some changes. Our assumptions for doing this are:

Passwords are currently stored hashed
Your password is not sent to an ancient mainframe (with a low rate of change and a high cost for change)
There is no technical limitation preventing a higher password limit
There is only one system uses the password

Low max length and a limited input length

The first step for working towards a better state is increasing the length of the password that that can be entered on login. If we started with the password set and change screens instead, then users would be able to create passwords longer than they could log in with. This would cause issues. Increasing the login screen limit can also cause issues.

Take, for example, the password: "bankpassword123456" (18 characters)

Imagine that this is my internet banking password.

It is what I typed in when I registered and it is what I type in every time I go to log in.

But my bank only has a 16 character password limit.

When I first set my password, I typed in those 18 characters on my keyboard. The field length limit stopped the last two characters from being entered. Then only 16 characters out of my 18 character password were sent to the server. Their login screen also limits me to 16 characters. I type in 18 characters every time and log in fine though. To me, 18 characters is correct and 16 is not.

What I type: bankpassword123456
What I see: ****************
What the server sees: bankpassword1234

So how do you increase the password field length when users may be typing in characters that your server does not see? Passwords are not stored in plain text so you can not identify the affected users ahead of time. You will need to determine a plan to handle this case.

You might think that this is an edge case. I have seen a poorly handled removal of a password length restriction. It caused chaos. Users were unable to log in and they had no idea why. There were hundreds of calls to the support line from users trying to log in. In the end we had to reinstate the limit to restore some order.

Your options include:

Attempting to communicate with every customer
Spend time determining which customers will be impacted
prepare for an increase in calls from users who are locked out

Once you have chosen your option and spent time planning, then you can get to executing it.

This is a lot of work for no new features, enhancements that users will notice (unless they are somewhat technical) or significant security benefit (given what banks are already doing on top of passwords for security). Banks are revenue driven businesses. Getting changes made requires a legal obligation or benefits that outweigh the costs. Increasing the password length is neither at the present time, so these first group are unlikely to make changes.

Low max length and a higher input length

Any bank that passes the first barrier gets into easier territory with this next group. What users type in is what gets validated. Anyone with the password "bankpassword1234" already fails to log in if they type in "bankpassword123456". To make passwords longer you need to increase the limit on your password change, set and login fields. You also need to ensure that you do not move back in to the first group, where the max input length equals the max password length.

And that is it. You are done. You have made the change. Get to testing and prove that there is not a security appliance in the middle of yoour network that expects short passwords.

Conclusion

There is a lot to think about before making changes to your password rules. Making changes to them is not as easy as increasing the length of your username or name fields. You need to be aware that your users do not count how many characters appear in the password field. You need to know about every system that sees your users passwords. Every system that has it's own login screen will need to go through these changes together. Only once they are all ready can you increase the max length that you allow users to enter.

Next time will be about centralizing your password screens or about why password rules are so bad in the first place. I have not decided which one yet.

What are your favourite meetups?

Mitch Pomery (he/him) — Sun, 15 Sep 2019 22:32:56 +0000

What are your favourite meetups and what makes you enjoy them?

I enjoy Junior Dev because I get to see all the cool things people have been learning about and building. I want to get a Junior Dev meetup set up in Sydney now that I'm living there.

Perth Fenders is also great. While I'm not a professional front end developer, I love playing around with HTML, JS and CSS at home on my own projects. The amount of work they put in to making sites accessible, fast and work across browsers and devices is amazing.

And Identity And Security Meetup Sydney is one that I will always remember as it is the first meetup that I gave a presentation at, and now I want to give more.

I'm also keen to get along to the Sydney Technology Leaders meetup and the Microsoft Reactor Lunch And Learns to catch up with friends, learn and network.

Oh, and I'm not big on food/drinks at meetups. I normally grab a snack before I go and dinner on the way home.