DEV Community: Alex Hyett

Giving back as a developer

Alex Hyett — Sun, 24 Dec 2023 12:35:01 +0000

Christmas is a time for giving, and giving comes in many forms. It's not all about the presents and stuffing your face with festive food (although there will be a lot of that as well). It is also a time to be grateful for everything that you have in your life, no matter how small.

As developers, we have a lot to be grateful for. There are not many professions that are in constant demand, with high pay, that you can do from the comfort of your own home (if the company will let you of course).

I know this year in particular has been tough with many still being effected by lay-offs, especially towards the beginning of this year. Fortunately, at least those I know personally effected, have managed to find themselves in new positions.

When it comes to pay, especially here in the UK, software developers are very well compensated. Sure, it is nowhere near the high 6-figure salaries seen in Silicon Valley, but if you at least live outside of London, then the cost of living is considerably lower as well.

If you have a look at the average annual salary in the UK by age you can see that the majority of the country isn't earning anywhere near what developers typically earn.

Age	Average Annual Salary
18-21	£22,932
22-29	£30,316
30-39	£37,544
40-49	£40,040

When I graduated in 2010 at age 22, and got my first job as a software developer I was earning £25,000 ($32,000). Adjusted for inflation that would be £36,815 ($46,500) in today's money. I know many developers, especially in London earning a lot more than that straight out of university.

In fact, if you earn over £59,200 ($75,000), then congratulations, you are in the top 10% of earners in the UK.

With the cost of living crisis getting worse and bills due to increase again in January, I can imagine it is going to be a difficult Christmas financially for many this year.

As a developer, there are many things you can do to give back to the community, and it isn't all about money.

Share your knowledge and experiences

One of the best ways is to share your knowledge. After all, we only get paid so highly because of what we know and can do.

As the famous saying goes:

Give a man some code and you help him for a day. Teach him how to code and you help him for a lifetime.

At least I think that is how it goes, although it may have had something to do with fish.

It isn't just your knowledge but your experiences as well. Can you tell others about how you got into tech and the path you took? Many people are looking to get into the industry and just don't know where to start.

Throughout the last year creating content I have been surprised by the number of questions I get from my readers and YouTube viewers. Unfortunately I don't always have the time to answer them all but I try and help where I can.

When something comes easy for you, it can be just as easy to forget how difficult it can be for someone else.

You don't need to be an expert at something to be able to teach it. You just need to be one step ahead of the person you are helping.

Support projects that have helped you

If writing or creating other content isn't your thing then another way is to support the projects and people that have helped you or are helping others.

There are many developers, like me, who spend hours every week writing articles, producing videos and other content to help other hopeful developers get into tech and progress their careers.

If there are creators that have helped you in some way, then make sure that you reach out to them and say thank you. Even if it is just a comment or a like on their article or video it can really help give them the motivation to continue doing what they do.

I can tell you first hand that it takes a lot of discipline and effort to show up every week and produce content. I have been writing this newsletter every week for over a year now and have gained 1,300 subscribers (thank you for reading)!

However, I don't make any money from this newsletter, it really is a labour of love. I don't have anywhere near the same subscriber numbers as some of the other tech newsletters out there, but I can see from my stats that at least 500 of you enjoy reading my newsletter and that means a lot to me.

Many creators have donation buttons on their pages whether it be Ko-Fi, Buy Me a Coffee or Patreon. Even if they don't need the money, paying for the content you receive for free is one way to say thank you and help motivate the creators to keep producing content which helps others.

Create and contribute to open-source projects

There are so many ways to give back with open-source software. Open-source software not only helps those that can't afford to pay for software such as in developing countries, but it also helps developers learn how to code.

If there is a piece of open-source software that you enjoy using then consider contributing to it if you have the skills. If you don't have the skills you can help by raising detailed bug reports, sharing it on social media or even supporting it financially.

Of course, you always have the option to write your own open source projects as well.

Mentor younger developers

There is always someone who is on the same path as you but just a few steps behind that you can help. You don't need to be an expert to start mentoring a junior developer.

Even if you only have 1 year of experience you will likely have gained a lot of knowledge in that first year and will know first hand the difficulties someone starting out is facing.

There is probably someone you already know who you could mentor. Whether it be a junior developer at work, your next door neighbour or a family member. I am sure they will appreciate the help if you have the time.

👨‍💻 Latest from me

When I started this newsletter I didn't think I would be sending an issue out on Christmas Eve and another on New Year's Eve!

Thank you for taking the time to read my newsletter this year. I hope you have enjoyed reading it as much as I have enjoyed writing it.

Merry Christmas 🎄!

❤️ Picks of the Week

🛠 Tool - Go Funko Yourself. I have a few FunkoPops sitting in my office and my wife and kids like them too. There is a Microsoft Designer prompt going round where you can turn yourself into a FunkoPop. It is good fun. This is me if I were a FunkoPop and my hair wasn't receding!

🧩 Puzzle - GCHQ Christmas Puzzle. Every year GCHQ come out with a Christmas puzzle. It is actually aimed at secondary school children (11-18) but it is good fun. I must admit I struggled with questions 5 and 7 and ended up guessing the final answer from my other answers.

📝 Article - All code is technical debt. An interesting view on technical debt. Writing code comes with a cost, and we need to make sure that features we add are more valuable than the technical debt that they will inevitably create.

📝 Article - Advice for new software devs who've read all those other advice essays. Some useful advice for new developers that you may not have heard before. I particularly like numbers 7, 8 and 10.

📝 Article - Mickey, Disney, and the Public Domain: a 95-year Love Triangle. An insight into the fascinating history of Disney, Mickey Mouse and the Public Domain.

📝 Article - New Mind-Reading "BrainGPT" Turns Thoughts Into Text On Screen. With just a cap on your head it can turn your thoughts into text. Depending on what your thoughts are like this could be a little worrying, but I can see this being of great use for disabled users.

📝 Article - Biggest productivity killers in the engineering industry. I have been guilty of every single one of these productivity killers at some point. I am sure I am not the only one.

📝 Article - Jason Fried challenges your thinking on fundraising, goals, growth, and more. I really enjoyed this podcast with Jason Fried. The team at 37 Signals are starting Once. Subscription free software that you only pay for once that you can host yourself. I am not sure how this will work with open-source software, but I am interested to see how it will work.

💬 Quote of the Week

To acquire knowledge, one must study; but to acquire wisdom, one must observe.

Marilyn vos Savant

📨 Join my free weekly newsletter for advice, technology and more →

Overcoming personal expectations

Alex Hyett — Sun, 17 Dec 2023 12:35:01 +0000

I have been on my new job now for 3 weeks, and it is going great. The team is fantastic, the work is interesting, and I have already released several features into production.

The only problem I am currently having is overcoming my personal expectations of how I should be doing.

It has been 4 years since I was last starting at a new job and even then it was at a company I had worked at before. In reality, it is more like 7 years since I was last a new fresh-faced member of a team with no idea how everything works.

I can tell you, it is a humbling experience.

Generally after a few months I understand the architecture and code well enough to make changes at a reasonable speed. It is only really once I have been working on a project for 6 months or more that I truly come into my element and start delivering features at a breakneck speed.

It is easy to forget when you have been working somewhere for several years, what it is like when you first started. The fumbling around trying to get your development environment set up, trying to work out how the architecture links together and picking up the coding style used by the team.

I know part of me used to get frustrated when developers would take a week to finish a feature that I could have done in one morning. I am now one of those developers again, and I am getting frustrated with myself at how long things are taking.

A lot of this is to do with having to learn a new tech stack. C# is still used, but the whole architecture is on Microsoft Azure which I haven't touched in over 7 years.

On top of that I am learning:

React
Next.js
Tailwind CSS
PostgreSQL

I have touched on some of these in the past, but now that I will be using them everyday I need to take some time to learn them properly. I love learning new things, but it is definitely slowing me down a bit.

So if you on a new job or new to software development in general, hopefully it is comforting to know that even a developer with over 13 years of experience can take a while to ramp up to full speed again.

I am currently having to remind myself that it is OK that it is taking a bit longer to deliver features. I will get faster over time and I just need to be patient with myself.

Luckily, with Christmas just around the corner we aren't looking to ship any big features which gives me a bit longer to learn the ropes and get acquainted with the architecture and data model.

I hope you all get a chance to take your foot off the gas a bit this festive season and spend some time with your families.

❤️ Picks of the Week

📝 Article - Polish Hackers Repaired Trains the Manufacturer Artificially Bricked. Now The Train Company Is Threatening Them - Some companies take it too far when it comes to protecting their products. If the product breaks then you shouldn't be forced to use a certain company to fix it. That is like taking your car to KwikFit for its MOT and then no longer being able to drive it because it wasn't done by the dealer.

📝 Article - How Plane, an open-source alternative to Jira, got to the #1 in project management on GitHub in less than a year - I haven't heard of Plane before but well done to them, and it does look like a great alternative to Jira.

📝 Article - Zero downtime Postgres upgrades - as I mentioned in today's issue I have been using Postgres more recently. This week I was getting stuck into doing backups with pg_dump and pg_restore. I am sure I will learn a thing or two from this article.

📝 Article - What Will Enter the Public Domain in 2024? - Many people selling things on Etsy don't seem to care about copyright law. Legally however you can only use something once it has entered the public domain. There are a few interesting additions coming next year including Steamboat Mickey Mouse, A. A. Milne's "The House at Pooh Corner" and the works of J. R. R. Tolkien e.g. The Hobbit and Lord of the Rings. This does vary between countries though so check first.

🛠️ Tools - Actual Budget - I have started tracking my spending more. I have always kept a budget, but I am never that clear whether I am always spending within it. I have started using Actual which is an open source application similar to YNAB. You can link it with a free GoCardless API to pull down all the transactions automatically. I will do a post next year about how I manage my personal finances as a software developer.

💬 Quote of the Week

If you can dream it, you can do it.

— Walt Disney

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →

SEO 101 for Software Developers

Alex Hyett — Sun, 10 Dec 2023 12:35:05 +0000

A lot of developers get put off with writing blog posts because they don't think anyone is going to read them.

After all writing is a big time commitment. It takes me a good 2 - 3 hours to write this newsletter every week and I can't guarantee you are going to read it. I am sure many of you just delete it as soon as it comes in, before eventually unsubscribing.

That is just the nature of writing and sharing online. Not everyone is going to want to read everything that you write.

Imagine you are writing a blog post about a problem you solved at work or something new that you learnt. I can't imagine everyone is going to want to read that, but it will be very useful for the select few who are having the same problem or learning the same thing.

The problem is there is so much content out there that the chances of someone stumbling across your article is going to be very slim.

What is SEO?

This is where Search Engine Optimisation or SEO comes in.

Now SEO gets a bad reputation for trying to scam users into visiting websites full of adverts. Certainly in the early days of the web, it was possible to convince Google that your article was relevant by stuffing it full of keywords that people might be searching for.

Luckily those days are behind us and the main way to get high rankings on Google, is to write good useful content.

Unfortunately there are so many people writing content that no matter how good your article is, it is still incredibly difficult to get good rankings in the search results.

SEO aims to help with this and has 2 main purposes:

Creating a good experience for your website visitors.
Helping people find your content.

Those aren't bad goals to have if you plan on sharing your writing online.

Creating a good experience for your visitors

In order to keep people continuing to use Google, they need to make sure that users can get the answer to their queries as quickly as possible.

A big part of that is to make sure that your website is fast and accessible. This part of SEO is really the bread and butter for web developers.

We all want fast websites that are easy to use for everyone.

This includes:

Using valid HTML and best practices for website layout
Making sure your page loads quickly
Making sure your website colours and contrast make text easily read
Ensuring page elements are large enough to be tapped on mobiles
Responsive layouts

There are a bunch of free tools available to check how your website scores:

One thing that is worth remembering, that has caught me out, is that you need to test that your website is still passing on a regular basis. This is especially important if you have changed anything about your website design or content since you last tested.

Admittedly it has been a while since I tested my own website, and it seems embedding some of my YouTube videos in my pages is slowing them down.

For example, my post on Git Flow vs GitHub Flow has the following results:

Google PageSpeed: Passes on desktop with decent results but fails on Mobile.
GTMetrix: Grade A 99% performance, 95% structure.
Pingdom: Grade C due to the 2.1MB page size. 🤦🏻‍♂️

It's not terrible, but things have definitely slipped from when I first created the website. I plan on redesigning my website next year, so I will make sure I take all of this into account.

Helping people find your content

If you are a developer then you are probably pretty familiar with testing websites for performance and accessibility. Having a fast website however isn't enough to ensure people find your content.

For most people, their experience of sharing content online looks like this:

Spend several hours writing their article
Publish it online
Share it on social media
Article gets viewed by their own followers and maybe shared a couple of times
Views eventually die off to only a handful a month
Start pondering their life choices for wasting time writing an article

Usually the main reason for the above is that they didn't consider what the user would be searching for in order to find their content.

Take my article that I mentioned earlier on Git Flow vs GitHub Flow. I wrote this article in November last year (2022). I was interested in the different Git Branching strategies that people were using and why.

Now I could have called this article "Git Branching Strategies: Which One Should You Use?".

A title like that may have done well on social media as it caters to readers curiosity.

Generally though, people are looking for entertainment on social media and are not looking for git branching tips.

The article was meant for people who were actively searching for the difference between Git Flow and GitHub Flow.

To find keywords I use the tool Keywords Everywhere. It gives you information on how many people search for a particular keyword a month, how difficult it will be to rank for, as well ideas for additional keywords.

Here is an example of the information you get for the term "git flow vs github flow".

I generally try and go for search terms with at least 1,000 searches a month and with difficulties below 70%.

By picking the right keywords we are not trying to scam people into reading our content, it is just giving us a better chance of someone finding it.

Once you have found a keyword phrase you want to rank for, you will need to:

Include it in the first part of your article title
Include it in the URL
Include it in your content.

You will also want to look at the other keywords that Keywords Everywhere mentions and include them where it makes sense.

What difference does it make?

This obviously takes more effort than just publishing an article without doing all this.

So, is it worth it?

If you search Google for the term "git flow vs github flow" you will see that my article is usually around the second result. Not bad considering the first article by Luca Mezzalira was written in 2014 and is the one a lot of websites reference.

The 10th result is also my YouTube video on the same topic, which is awesome.

Possibly more impressive is the fact that for "github flow", I am the first result after github.com and github.io.

It is worth noting that the results won't come quickly. It takes time for Google to index your article and for it to work its way up the rankings.

If we look at my analytics you can see how traffic has increased over the last year for this article:

11.6k of those views have come from Google. Generally this article gets viewed around 2,000 times a month of which 1,200 come from Google. This makes sense considering it is estimated that 1,600 people search for this term a month, and it is at position 2.

The video is a little older as I published it 31st October 2022. If you're wondering about the revenue, it isn't a big earner.

I have only been monetised on YouTube since end of May 2023. I earn roughly £0.0022 per view with this video 🎉.

Again 37.6% of views come from YouTube Search, so the keyword research is working.

This is the approach for most of my content apart from this newsletter which is on a different domain at the moment.

Benefits of Keyword Research

The main benefit of keyword research is that your content will continue to get viewed long after you have written it.

This newsletter is hosted on Substack on a different domain to my main website. This means that my main website currently isn't benefiting from this newsletter, something I plan on changing next year.

I have been working on a course for the past couple of months, so I actually haven't written any blog posts or posted any YouTube videos since 6th October.

For most people this would mean a drastic drop in views on their YouTube channel and website.

Let'shave a look at my stats:

First my website:

Apart from the drop in December as the month is still ongoing, traffic has been fairly stable the past couple of months despite no new content.

Now YouTube:

See the massive drop of views since my last video ..... no me neither. Of course, it isn't growing much either, but I don't have the constant need to produce content.

You can also see here how much my YouTube channel has earned me this year too which is currently averaging around £200 a month. Not bad for a side hustle I haven't touched in 2 months. This is also with me turning off mid-roll ads (ads in the middle of a video) just to make the viewing experience better for everyone.

I have only scratched the surface of how I do keyword research, and it is something I would like to cover in more depth in the future. Hopefully you can see the benefits of doing it before spending hours writing a blog post.

If you want to learn more I highly recommend Erik Dietrich articles on Keyword Research:

❤️ Picks of the Week

📝 Article - Goodbye, Clean Code. I have seen many developers fall into this trap. Why write 10 lines of code when you can condense it down to 1? The reason is usually down to readability and maintainability. If it doesn't affect performance, then minifying source code down to as few lines as possible is only going to cause headaches when you need to change it.

📝 Article - iMessage, explained. This is an interesting article on how iMessage works from someone who has reverse engineered it.

📝 Article - How Google takes the pain out of code reviews, with 97% dev satisfaction. I hadn't heard of Google's Critique tool until this article. I wish there was a public tool available that did this. Anyone fancy building one?

📝 Article - Now I Can Just Print That Video. Another useful use of AI. My wife has a whole load of saved Instagram Reels of tasty recipes. We hardly ever do they them though and when we do, we just end up watching the reel on repeat. This would be perfect for that.

📝 Article - Troy Hunt: A Decade of Have I Been Pwned. I remember when Troy released Have I Been Pwned. He has written some in-depth articles on how he built it and how little it cost to run. It was his articles that got me interested in serverless technologies.

💬 Quote of the Week

Joy is what happens to us when we allow ourselves to recognize how good things really are.
— Marianne Williamson

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →

The Good and Bad of Remote Work

Alex Hyett — Sun, 03 Dec 2023 12:35:03 +0000

This is going to be a slightly different issue of The Curious Engineer, as I am in Berlin this week, so I am writing this issue between work and travelling.

Berlin is really nice but bitterly cold this time of year. I thought it had got cold in the UK this week, but it is nothing compared to here. There is actually snow on the ground!

If you have been an avid reader of my newsletter you will have heard that I have started a new job this week. No I will not be relocating to Berlin, the job is 100% remote, but some things are just easier to do in person.

This is my first time working a fully remote job, before it was always some form of hybrid work (lockdown excluded).

Working remote has some clear benefits which is the reasons why developers love it so much.

🚝 No Commuting - I used to commute 2 hours each way up to London each day. That's 20 hours a week of travel time, no wonder I got burnt out from doing that.
👨‍👩‍👧‍👧 Better Work-Life Balance - Due to the commute I hardly got to see my children during the week when I was working. Not only that but working remotely allows me to have breakfast and dinner with my family every day. I even get to have lunch with my wife if she is home too.
💰 Save Money - I save around £500 a month not having to commute to London and a lot more than that not having to buy lunch each day at London prices.
📦 Extras - There is so many other little benefits as well such as being in to take deliveries, working without all the noise of your co-workers, being in a nice environment and getting use your own equipment.

In our small team and with everyone working in different countries remote work is the obvious choice. It is made a lot easier given that we are all very experienced, and we know each other well.

It does make me wonder how things are going to work should we take on any junior staff. This is the crux of the problem when comes to working remotely.

If you are a junior developer or even an experienced developer working in a new domain, there is going to be a ramp up time before you become a contributing team member.

It is always quicker learning from someone else if that can be done in person. It is the main reason I am in Berlin this week. Even though I know the domain well I don't know the product well, and it is just quicker to go through these things in person.

I think if I was working in a new domain with people I haven't worked with before it would be a lot harder.

Certainly early on in my career I benefitted greatly by being in the office with my coworkers. As it is with product development the key is to keep the feedback loop small. If you are stuck on a problem it is a lot easier just to quickly ask the person sitting net to you vs waiting for someone to respond to you on Slack.

If you are still quite early on in your career I wouldn't stick your nose up at hybrid work setups. Having people around you that you can quickly learn from is really helpful when you are first starting out.

Once you have many years experience under your belt then definitely try and get a remote position if you can as the benefits definitely outweigh the drawbacks.

To finish off this week, here is a nice photo of me freezing my a** off outside the Brandenburger Tor. Typically, it was covered with scaffolding on the day I go to see it.

❤️ Picks of the Week

🛠️ Tool - Animotion — Visual CSS Animation Builder - I am always impressed by what you can do with just CSS in the browser. If you are looking to learn how to do CSS animations this is a great resource to get started.

📝 Article - A good engineer thinks like a product manager - At the end of the day, as engineers we are here to build products. A good engineer has to have a mix of both product and technical skills.

🛠️ Tool - VectorDB - Kagi seem to be doing a lot of great things recently with paid search and their new browser (see below) and they are giving back to open source with VectorDB.

📝 Article - I Fight For The Users - I always like reading Jeff Atwood's articles on Coding Horror, but he doesn't blog as much as he used to. This appears to be this year's blog post, so worth a read.

📝 Article - Writing Documentation for Your House - I have been thinking about setting up my own home wiki. There is so much in this house that only I know how to do. If anything happens to me my wife would be a bit lost. I might use this for inspiration.

📝 Article - PeerTube v6 is out, and powered by your ideas!. I am going to be starting my own PeerTube instance in the new year for all of my YouTube videos. I am glad to see this project is getting better and better.

🛠️ Tool - Orion Browser by Kagi - I am definitely going to have to try this out. A privacy first web browser by the people who have built Kagi search.

📚 Book - Understanding Deep Learning (free) - Everyone loves free stuff, if you are interested in learning about Deep Learning then this free book looks awesome.

📝 Article - Check your AWS Free Tier usage programmatically with a new API. Going with the free theme, I use AWS Free Tier a lot for my blog and other projects. If you need to check your usage you can now do it programmatically.

🛠️ Tool - Darling: Run Mac Apps on Linux. I am only running Linux on my home server at the moment, but I am thinking about putting it on my MacBook as it no longer receives the latest MacOS. This will certainly make things easier.

🛠️ Tool - Whisky: Easy Mac Gaming - Of course if you are sticking with Mac and want to use it as your gaming machine then you can use Whisky to do all the Wine setup for you. Thanks to Valve and the Steam Deck Wine supports pretty much everything these days!

💬 Quote of the Week

Only put off until tomorrow what you are willing to die having left undone.
— Pablo Picasso

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →

Open Source: An Antidote to Closed Source Vulnerability

Alex Hyett — Sun, 19 Nov 2023 12:35:02 +0000

Every job I have had over the last 13 years I have been working on closed source software. I think that is true for most software developers. Companies make their money by producing value but also by creating a mote large enough that other companies can't cross.

If companies were to open source all their code then it would be very easy for a competitor to come along and make something better using their code as a starting point.

That's not the only problem, if something is available for free then there is less chance that people are going to pay for it. This is especially true in the developer space as developers have the technical skills to be able to host software themselves rather than pay someone else for the privilege.

I have a lot of software that I host myself on my home server, partly to save money but also because I want to control my own data. For example, I host VaultWarden which is the open source server for BitWarden. This gives me all the premium features for free with the added bonus of keeping my passwords out of the cloud.

There are a few companies who open source their software but still manage to have a profitable business.

mongoDB
elastic
databricks
HashiCorp
Ghost
Automattic (Wordpress)

These companies tend to do either one or more of the following:

Open Source Core Product - the core product is open source, but a number of features are kept closed source behind a paywall. These are usually enterprise features that individual developers have little use for.
Managed Hosting - companies make their money by charging for managed hosting of the product. Which is what most of the companies above are doing.
Support - big companies using the software tend to need additional timely support which they can get by paying for it.
Mixed licensing - the entire code base is open source but not all of it is licensed under an OSI-approved license. The code under the restrictive licence often includes paid functionality.

Some companies even follow this approach for free software as well. For example the code for Visual Studio Code is licensed under the MIT license. The binary application however is licensed under the Microsoft license.

The binary application includes all sorts of data collection that Microsoft use to improve the product, or their own closed source products. As the saying goes, if a product is free then you are the product.

Of course anyone can compile their own version of VS Code without all the data collection, but Microsoft have already thought of that one! Only the official version of VS Code can download extensions from the marketplace making any derivatives pretty much useless.

The dark side of closed source software

Which brings us nicely on to why open source software is so important.

If Microsoft decided to stop distributing VS Code then you could still build it yourself, but you wouldn't have access to any of the extensions that actually make the editor useful. Without all these extension VS Code is no better than any other text editor.

If this happened you would likely just move on to a different IDE rather than continuing to use VS Code without extensions. Of course, as the majority of the product is open source someone could host their own extension marketplace and keep it alive that way.

Many products however are not open source. What happens to them when the company chooses to stop support or even goes out of business?

For example, I have 5 Amazon Echo's dotted around my house. I mostly just use them for listening to music, controlling lights and getting weather updates. If Amazon decides they no longer want to be part of the smart home market then all of these devices just become expensive paperweights.

When I was a kid I had a Lego Technic set that you could control with a computer. I am pretty sure that software wouldn't run on any modern computer any more, at least you can still use the bricks.

There are so many digital products that either require an app or an online service to work. I have a few TP Smart Plugs for my lights that require an app to work. The same is true for my office lights, my watch and even my bathroom scales.

I know they don't build products to last any more but even if they did what are the chances that the company is still going to be around in 10 years time or that the app will work on the latest phone.

The same is true for our other devices as well. My MacBook Pro from 2016 no longer receives the latest MacOS updates, and I am pretty sure it won't be long before my iPhone XR stops getting the latest version too. At least with my laptop I can switch to Linux if I wanted or use the OpenCore Legacy Patcher to force install the latest version.

The situation is even worse when it comes to medical technology. More than 350 blind people were fitted with implants in their eyes to help restore sight. Unfortunately the company decided to abandon the technology when they nearly went bankrupt, leaving users with no continued improvements or help if there is an issue.

It is a chilling thought when you think about all the devices that would stop working if companies go out of business or choose to stop supporting them. I am certainly going to be a bit more conscious with how I spend money on technology going forward.

If a company goes out of business it would be good if they open sourced their code. This is what Placemark have recently done which I applaud them for. Obviously this isn't always an option, but we can dream.

❤️ Picks of the Week

📝 Article - 67 Weird Debugging Tricks Your Browser Doesn't Want You to Know - I mostly do backend development, but I do dabble with frontend for my website and other projects. There are a lot of tricks on this list that I didn't know about that are worth looking into.

📝 Article - HTML First - This is a great article on using HTML for things rather than relying on external frameworks. It reminds me a lot of an article I read a while back about using JavaScript instead of jQuery. In most cases it took the same amount of code to write in JavaScript as it did in jQuery. It also lasts a lot longer, who is still using jQuery now? At least HTML and JavaScript aren't going anywhere.

🗺️ Map - Paris space invaders map. Not something I would expect to see in Paris. It turns out that it is full of little space invader ships! They should have these in every city!

📝 Article - Archive Your Old Projects - I have a project folder on my computer that has all the code projects that I have started (and never finished). Occasionally I move them into an archive folder, but mostly they get forgotten about. There are some good tips on here about archiving your projects.

📝 Article - A Coder Considers the Waning Days of the Craft. I have been using AI more recently to help with programming. Mostly it has been small scripts that I could have written myself. When it comes to specific Linux commands it has definitely been quicker to ask ChatGPT how do something then to search for it or look in the documentation. Just make sure you test anything that it produces thoroughly. I don't believe that AI will take over a programmers job, but I can see it evolving a lot.

Programming by typing C++ or Python yourself might eventually seem as ridiculous as issuing instructions in binary onto a punch card.

📝 Article - Placemark is going open source and shutting down. As you can guess from this article it is good to see this project going open source, and hopefully it will continue to shine.

📝 Article - GraphCast: AI model for faster and more accurate global weather forecasting. Generative AI is fun, but there are more useful problems to solve than generating a picture of a cat in a tuxedo or yet more anime girls.

📝 Article - Young Animator of the Year UK. This 16-year-old has some skills! Well done Euan, this is some impressive work. I know nothing when comes to 3D rendering, but I can imagine this was a lot of work and looks as good as any Hollywood production.

💬 Quote of the Week

Let all your loves, fears, and interests saturate your work and make it memorable. Who you are is the most important part of your work - never leave it out.

From Feck Perfuction (affiliate link) by James Victore.

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →

Combining Digital and Analog for Better Note-Taking

Alex Hyett — Sun, 12 Nov 2023 12:35:02 +0000

At times, it feels that I have so many ideas rattling around in my head that I struggle to be present.

As a husband and a father it is important that I spend quality time with my family and that can't happen if my thoughts are elsewhere.

Often we find that a large amount of brain power is spent on trying to remember all the things that we need to-do, or the ideas that we have had.

One of the best ways I have found to clear my head, is to just write everything down. It is essentially a brain dump, moving the information from my head onto another medium.

Once I have my thoughts written down, I can free up some much-needed headspace and focus on spending time with my family.

As a self-confessed geek, my natural instinct is to find an app or device to do this, however, I have found a simple notebook is usually the best approach.

Of course the problem with a notebook is that it is not searchable, and you end up filling many notebooks over the years.

To combat this I use both digital and analog tools for taking notes and split what I store in each of them.

Analog Note-Taking

I have had a very nice notebook from Ottergami (affiliate link) sitting on my desk, untouched for the past year. It was present that I got from my Dad for Christmas.

It is such a nice notebook that in my head it was too nice to use without having some purpose. I didn't want to spoil it by adding in my illegible scribbles.

This month however, I have had several nights where sleep has alluded me as too many thoughts were buzzing around my head, so I thought it might be time to put it to use.

Bullet Journal Method

I have started reading The Bullet Journal Method (affiliate link) by Ryder Carroll, I know I am pretty late to the party on this one. Bullet journaling was really popular during lockdown as many people had more time and wanted to live life at a slower pace.

There are some really impressive examples that you can find on Instagram or Pinterest however they have a tendency to put people off. I am not bad at art, but I haven't got the time to make every page look like a masterpiece.

However, the Bullet Journal method isn't really about making things look pretty, it is supposed to be a functional tool.

When I am at work I always have a notebook with me. I mainly use it to keep a running to-do list of what I need to get done that day. I also use it for taking quick notes for ideas.

The Bullet Journal method as the name suggests revolves around using different bullet points as a quick way of adding some context to your to-dos.

These are the main ones, but you are free to create your own as well.

• To Do - item that hasn't been completed yet
x Done - cross through the bullet point when complete
- Note - general notes
> Mitgated - incomplete items that have moved to the next day
< Scheduled - scheduled for another date
○ Event - this can be an appointment or something that happened

Signifiers
* Important - make something as a priority
! Inspiration - great ideas and inspiration

If something becomes irrelevant then you can also cross it out.

There are a few pages the book calls "collections" that you add to the start of the book.

KEY - List out all the types of bullet points you plan to use.
INDEX - As it sounds, you can write down the page numbers for key topics. Leave a couple of pages for this.
FUTURE LOG - This is where you schedule things for the future. Whenever you use < on one of your bullet points it should go in the future log. You need to have a section for each month. I have seen people put 3 months per page, but you may need more space than that.

At the start of each month you then create some more collections for the month ahead:

MONTH LOG - All the days of the month are listed down the left-hand side, and you can add in any appointments, birthdays etc.
MONTH TASKS - A task page of all the things you need to get done in that month. Anything from the future log for this month should move to here.

For the rest of the month you then use a Daily Log to write all of your bullet points down and anything else you want to add.

The key thing with the bullet journal is that it is a blank slate which can be used for anything. Some days I will have 2 pages of notes and tasks while others it may only be half a page. It is handy never running out of space or wasting pages on days I don't use it.

There are plenty of other things that you can add as well such as trackers. Each day I have been writing how many hours I have been sleeping, however, I have realised this would be better off as a dedicated page each month.

I have also seen other people create trackers for:

Habits
Drinking Water
Exercise
Weather
Mood
Spending

The main benefit of a notebook is that it is always available. You don't need to go on a computer or get your phone out to use it. The slower process of writing something out can also help you remember things better.

My plan for my notebook is to use it for the following:

Keeping track of what I need to do on a particular day.
Quick ideas and notes, especially if I am not in front of a computer.
Tracking my sleep, exercise and other habits. This should help me see how one thing affects another easier.
Brain dump, clearing my head at the end of the day.
Gratitude journaling, studies show that those who list what they are grateful for every day, tend to be happier.

There are some downsides of using notebooks though, such as not being able to moving things around and not being able to search through them. Therefore, I still have several digital tools that I use throughout the day.

Digital Tools

I can't use my notebook for everything or at least is not practical to do so.

These are the digital tools that I rely on to keep more organised and help me remember what I am learning.

Calendar

Even though my notebook has a month log, I mostly use it to list things that have happened or events that aren't going to change like birthdays or holidays.

Meetings and appointments regularly get cancelled or rescheduled, and I don't want my notebook looking like a complete mess.

My wife and I share our calendars with each other, and it is really useful to see what the whole family is doing in one place. As I get older I also find I am making more use of the reminders.

Apple Notes

If I am out and don't have my notebook with me, I will often use Apple Notes to write down a quick thought. I also use it for temporary notes such as shopping lists or making sure we have got everyone a Christmas present.

If a note or idea is important then I will often move it to my digital notes (see below).

Obsidian

Obsidian is where I write all of my digital notes as well as the content for my website and this newsletter.

I do occasionally use the daily note plugin and the calendar plugin to write out my daily notes instead of in my notebook. This is mostly when I have something technical rattling around in my brain and there is a chance I will want to include links or screenshots as part of my notes.

My Obsidian notes mostly act as a personal wiki. Any notes related to what I am learning or reading get added to Obsidian. People often call these types of notes a "Second Brain" or a "Digital Garden".

I use Readwise to sync all my highlights and notes from my Kindle books as well as articles I read online into Obsidian. These then later form the basis for my book notes on my website.

At some point I plan on publishing my digital garden on my website too, but I need to split out private and public notes in a better way before I do that.

Currently, I am using the following folder setup:

0. INBOX - This is where all the highlights from Readwise end up. I also use the inbox folder to add new notes that I am working on before moving them to the BRAIN folder.
1. REFERENCES - Once I have added my own notes to the book and article highlights, I move them here. I end up highlighting quite a lot of the book sometimes. Before publishing book notes I tend to copy them and trim them down a bit.
2. BRAIN - This is where I put all my ideas or notes on topics. This is the traditional personal wiki or digital garden.
3. PUBLISHED - Everything I write that is going to be published on my website or newsletter is in here. This includes articles, book notes, courses and newsletters.
ARCHIVE - If a note is no longer relevant it moves to the archive.
EXTRAS - This is where I store all my note templates.
JOURNAL - Daily journal notes get stored here.
LATER - If there is something in my inbox that I don't want to turn into a permanent note I put it in Later, so I can still find it through search.

I haven't had much luck using the Obsidian app on iPhone or iPad. I don't think it works very well if your notes are stored in iCloud. This is why I use my notebook rather than Obsidian if I am not sitting at my computer.

I think it is important to have a system for taking notes. I have lost count of the number of people I have worked with who seem to always forget to do what has been asked or will ask the same question on different days.

Our brains aren't very good at remembering everything and when we do remember something it is usually at the wrong time. We have to make use of the tools we have available to us to make us more efficient.

❤️ Picks of the Week

📝 Article - Cursorless is alien magic from the future. I think it is great that software like this is available. If I ever lose my ability to type it is good to know I will still be able to code using my voice.

⌨️ Font - Monaspace. As developers, we spend a lot of time looking at code, so it makes sense to use a font that is pleasing to the eye. Everyone has their own preference, but this looks like a good option.

👾 Gaming - Steam Deck OLED. I love my Steam Deck and recommend it to anyone who is into PC gaming. The new version has just been released with an OLED screen, a faster processor and better battery life. I won't be upgrading mine just yet but it is great to see things improving in this space.

📝 Article - Omegle. I never actually used Omegle but enjoyed watching people's reactions to good musicians that used to use the platform. I can see why they have had to shut down, but it does beg the question of whose responsibility should it be to police everything. If the telephone is used for evil doings, should we get rid of all telephones or monitor ever call for "policing" purposes, neither is a good option.

📝 Article - 10 hard-to-swallow truths they won't tell you about software engineer job. Software development doesn't look anything like the movies, this is a worthwhile read for anyone getting started in the industry.

📝 Article - What Does and Doesn’t Matter about Apple Shooting their October Event on iPhone 15 Pro Max. I still shoot my YouTube videos on my iPhone XR and before that a Logitech Brio Webcam. The camera only plays a small part in making video look good, which is evident by all the other professional equipment that Apple needed to use.

📝 Article - Building In-Video Search. An interesting read from the Netflix blog on how they are building in video search using LLMs.

👾 Gaming - Why Cities: Skylines 2 performs poorly. I have never played Cities: Skylines but my sister was looking forward to it and bought the game when it first came out. She ended up getting a refund the same day as it was completely unplayable. A developer decompiled it to find out what caused the issues.

📝 Article - Building a high performance JSON parser. I wouldn't recommend building your own JSON parser, but this is an interesting read on how you might do that.

📝 Article - Current thoughts on social media. Martin Fowler has shares his thoughts on the current state of social media. We seem to be on the same page with what direction to take:

The reason I'm posting on Mastodon is because the future I want is one that's based on interoperability. I want to see an open protocol that isn't dependent on a tech-lord's whim. ActivityPub is currently the one that looks most developed ...... By using my domain I can be sure that I can make material available and present it in the way I like.

📝 Article - From IC to manager. A good article in dealing with the transition from IC to manager. Personally I prefer to stay on the technical side rather than go towards management.

📝 Article - Software that supports your body should always respect your freedom. Software is used in everything, but it is a scary thought to think that the medical device that you rely on to stay alive could be abandoned and stop working.

👨‍💻 Latest from me

A quick personal update from me. This week I signed a contract for a new job which I will be starting at the end of the month 🎉.

It's a fascinating project in an area that I know very well, and I get to work with people I like and have worked with before. Those that know me well might be able to guess, but you will have to wait for the official announcement on LinkedIn.

I will be working part-time, which means I still have time to write this newsletter, work on my course and produce the occasional tech video.

As content creation is not going to be my "main gig" any more I have closed down my Patreon page and plan to open up the code to my videos for everyone for free.

Over the coming months I am going to self-host my own Peertube instance for my videos and Forgejo instance for my code. I like the idea of having everything under one domain and in my control.

If you would like to support my work and the hosting costs then I have created a Ko-Fi page for either one time or regular donations. Ko-Fi doesn't take a cut of the donations unlike Patreon which takes 8% and Buy Me a Coffee which takes 5%.

One benefit of optional donations is that they don't attract VAT. As I mentioned in a previous post, anyone selling something online outside the EU has to charge EU VAT to EU customers. This is a huge burden especially as there are no thresholds before this kicks in.

My solution was to move to Patreon who handle the VAT for you, however now everyone gets charged VAT. If you are not actually selling something then VAT doesn't apply.

💬 Quote of the Week

Instead of random and chaotic input, feed your mind focused, high quality input. Read several highly rated books on a subject that interests you. Take notes on what you read, and seek to apply what you learn, so you can test the ideas for yourself.

From the article Caring for Your Mind by Steve Pavlina.

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →

The Art of Abstractions

Alex Hyett — Sun, 05 Nov 2023 12:35:01 +0000

The modern world is built on abstractions.

A good abstraction lets the user focus on doing something rather than concentrating on how they are doing it.

If you take driving for example. A car is a fantastic example of an abstraction that hasn't changed much in over 100 years.

Whether you are driving a 1919 Ford Model T or a Nissan Hyper Force (which looks like it has come straight out of cyberpunk game) they both feature a steering wheel and pedals. You don't need to know how the car works under the hood to be able to drive it.

Having this stable contract means that anyone who has learned how to drive can get in any car and be able to drive it (assuming you didn't learn to drive in an automatic).

I am no mechanic, but I am pretty sure the internals of a petrol car compared to an electric or hydrogen car are very different. For the end user however, it doesn't make much difference until it comes to refuelling.

This is the power of having good abstractions, and it is something we should all strive for when writing software. The problem is, it is incredibly difficult to do.

In order to write a good abstraction you need to predict how your application is going to change in the future.

The reason that cars are such good abstractions is that the requirements are simple.

Must be able to change the direction of the car
Must be able to increase the speed
Must be able to decrease the speed
Must have lights to see in the dark
Must have indicators to show intentions to other drivers

Combine that with some hefty regulation for car safety, and you have a solid contract on what a car must look like that is resistant to change.

One form of abstractions are interfaces, however not all interfaces are abstractions. Mark Seemann has a great article on this, and it is something developers get wrong all the time.

We think we are creating an abstraction but often implementation details leak into the contract spoiling its usefulness.

If you want your interface to stand the test of time, it needs to be resistant to change and describe the functionality not the implementation.

The downside of abstractions

Abstractions are great for software design and ease of use, but they come with a hefty price.

A good abstraction hides from the user the implementation details and design decisions. This is great for end users, but it also makes it too easy to be ignorant of how things actually work.

Abstractions hide a lot of complicated details from users, which is good but as engineers, you aren't going to learn anything by always relying on abstractions.

Many people used to fix issues with their cars themselves, but now they are so complicated that most don't bother. It is the same for a lot of products that we use. We all know how to use a TV, but there aren't many who would be able to fix one or described how it works.

As developers, we use other people abstractions all the time in the form of frameworks and libraries. By using these libraries we can write code faster and save ourselves time on maintaining them but at the same time lose out on all the knowledge we could have gained by implementing it ourselves.

I am not advocating for not using libraries, they are essential in modern development, just take the time to learn how they work.

For those of you that work on the frontend, how many of you can build a UI with just HTML, CSS and raw JavaScript?

I know on the backend I would struggle to build anything useful in C# without using 3rd party packages such as Dapper, FluentValidation or Moq.

Can I really call myself a developer if I can't build an API with just C# and no other packages including the Microsoft ones?

I am sure it would take a very long time, but I am sure I would learn a lot in the process.

❤️ Picks of the Week

📝 Article - Google Brain founder says big tech is lying about AI extinction danger. This is an interesting take on the real threat of AI. AI isn't going to take over the world. The scaremongering is to force legislation to be put in place that will throttle growth in the open source space.

📝 Article - The costs of microservices. This is a good article explaining the costs of switching to microservices. I particularly like the clear diagrams. As I described in my piece on "Monoliths vs Microservices", it is always better to start with a monolith and then switch to microservices when the need arises.

👾 Game - House Flipper. I pretty late to the party on this one. There are very few computer games that my wife and I both enjoy. When we were younger we used to enjoy playing Sims 2 purely for the house design aspects. We have both been enjoying House Flipper this week. It is like Sims 2 without all the annoying people aspect. I have been playing it on my Steam Deck, but they also have versions for XBOX, PS5 and Switch. House Flipper 2 is due out in 6 weeks on Steam which looks even better.

📝 Article - The architecture of today's LLM applications. I find generative AI fascinating, but a lot of the content is difficult to read. This is a great piece on the GitHub blog that goes into how to use AI in your applications.

📝 Article - Hypercritical: Apple’s Blue Ocean. I hadn't heard of the "Blue Ocean Strategy" before, it is all about making the competition irrelevant.

🛠️ Tools - OpenSign: The free & Open Source Alternative to DocuSign. It is always good to see Open Source alternatives to software that would otherwise have a monopoly. OpenSign looks like a great alternative to DocuSign.

📝 Article - Fixing the Volume on my Bluetooth Earbuds. I love the lengths that this person has gone to, to try and fix their headphones. That loud startup sound must have been really annoying to warrant this much effort. Still it has some great tips on how to reverse engineer tech which I found fascinating.

📝 Article - Why you should probably be using SQLite. I must admit I usually jump straight to using MySQL or Postgres for most of my projects. If you just need to store some structured data somewhere you can save yourself a lot of effort by just using SQLite.

📝 Article - How to draw beautiful software architecture diagrams. Half a developers job (at least it should be), is documenting what you have built. They say a picture is worth a thousand words and this couldn't be any more true when it comes to architecture diagrams. If yours end up looking like a spaghetti mess then give this article a read.

💬 Quote of the Week

Efforts to deepen your focus will struggle if you don't simultaneously wean your mind from a dependence on distraction.

Much in the same way that athletes must take care of their bodies outside of their training sessions, you'll struggle to achieve the deepest levels of concentration if you spend the rest of your time fleeing the slightest hint of boredom.

From Deep Work (affiliate link) by Cal Newport.

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →

Protecting Privacy: Can Software Developers Make a Difference?

Alex Hyett — Sun, 29 Oct 2023 12:35:01 +0000

Technology has improved our lives in many ways but the one thing that has got gradually worse over time is our privacy.

Companies like Meta, Amazon and Google know a staggering amount of information about us, from our favourite shows and movies to your sexual preferences and your current location.

The goal of all this data gathering is simple, to make as much money as possible by selling targeted advertising and recommending products based on your preferences.

In 2018, the GDPR regulation came into effect with the goal of protecting peoples privacy. Although this regulation had good intentions, as many of us know, it put a huge technological burden on small companies.

Hopefully these companies were already storing personal data securely, but you now needed to put in place ways for people to request data, delete their data and ensure you aren't keeping it longer than needed.

The one thing that this has ensured is that companies think twice about what data they gather on people. Do you really need to know my date of birth, annual salary and marital status when signing up for a new account?

The regulation puts even stricter protection on sensitive data such as:

race
ethnic background
political opinions
religious beliefs
trade union membership
genetics
biometrics
health
sex life or orientation

This however hasn't stopped large companies from gathering this information anyway, especially those outside of Europe.

It was only last month that X (formerly known as Twitter) updated their privacy policy to include the collection of "biometric data". Why on Earth (or Mars) would Elon need to gather that information?

Technology has come far since GDPR was conceived, and we now have every company rushing to try and use AI, so they don't get left behind.

However, unlike a database, the Large Language Models (LLM) used in artificial intelligence don't have any way to delete information. If your personal details get included as part of the training data, then they are in there for good.

If that wasn't bad enough, Meta are now selling Ray-Ban smart glasses that look like normal glasses but allow the wearer to record everything going on around them.

Don't worry Meta have your privacy covered with a little LED light that turns on, so you know someone is recording.

I know many people who don't like their photo being taken and will actively avoid people who look like they are taking a photo on their phone. However, even with a little light notifying you it's recording, it is still easy to miss.

You can buy them for around £300 which isn't an extortionate amount, so I can see these becoming quite popular, unfortunately.

Online Safety Act

Not to fear, the UK Government is here!

Last Thursday the government passed the Online Safety Act. Which aims to make the internet a safer place for adults and children.

In theory this sounds great.

Unfortunately the bill contains a number of questionable points which completely go against people's privacy.

It seems that the governments take on safety is to keep everyone under surveillance. Forget 2023, it is more like 1984, Orwellian style.

One of the key controversial points was that messaging services would need to use "accredited technology" (meaning government approved technology) to scan all messages for content containing CSAM.

Tech firms have rightly been against this as it poses a fundamental threat to end-to-end encrypted messaging. The government hasn't stepped down despite the controversy but admitted that Ofcom would only be able to intervene once scanning becomes "technically feasible".

Although, scanning peoples files hasn't worked out well in the past as this Dad can attest after being marked as a criminal for sending photos of his son to his doctor.

The other controversial measure which is part of this bill is on age verification. Websites which may contain adult content will need to perform age verification on their users. By the sounds of it this may also include all social media websites as well.

When they say age verification they are not talking about the pop-up that asks you for your date of birth as seen on many alcohol websites. No they are talking about verification using your ID or some other means of personal information.

It seems we have gone from GDPR, where only essential information should be collected, to needing to scan your passport before visiting a website.

The main problem with all this, is that most companies don't have a great track record when comes to protecting our data. You only need to look on Have I Been Pwned to see where your data has been leaked.

Shockingly my data has been leaked by 15 different companies over the years.

So forgive me if I am not overjoyed by the thought of companies needing my passport or driving licence to make sure I am an adult.

What can we do as developers

As software developers many of us will be directly involved with implementing some of the changes for the companies we work for.

Even if you are not based in the UK, there is a lot to be said about the state of data collection in other countries as well.

Luckily as software developers, we are in a good position to make companies a little bit more privacy conscious, especially if we are involved early on in the decision-making process.

It is worth discussing these questions when working on new features:

Do we need to be collecting all this information from our customers?
How are we going to make sure that this data is stored securely?
What security measures have been put in place to ensure no unauthorised access?
How long do we need to keep this data for?
How are we going to automatically delete the data when it is no longer needed?
Is this data being shared with anyone?
How do we ensure the data that we are keeping is kept up to date?

Unfortunately a lot of the data breeches have been due to developers not following best practices when it comes to security. If you are building publicly accessible software then you need to make sure that the proper security is in place and that it is tested for weaknesses.

One such example is from the greetings card company, MoonPig. In 2015, a security flaw was made public which allowed anyone logged into their API to access all of their customers details including name, address, card details and orders. The access even allowed them to make orders on behalf of other customers.

The flaw was found by another developer who told the company, but they failed to fix it for 18 months, so he went public.

Clearly whoever implemented these features didn't understand the difference between authentication and authorisation.

As developers, we need to make sure that we understand the security basics, so we don't accidentally put customers data at risk.

❤️ Picks of the Week

📝 Article - Hopping instead of hustling. The StackOverflow developer survey shows that developers are still job hopping despite the current market with many developers actively looking for a change.

📝 Article - Approaching unconventional problems. Your friend loses their phone in the middle of nowhere with no signal. How do you find it? This is some great out of the box thinking.

📝 Article - Internet Artifacts. Take a trip back in time and revisit how the internet started. Neil has done a great job putting this together and it is fun to look through.

📝 Article - Base64 Encoding, Explained. Akshay has done a great job here explaining how Base64 works. Worth a read.

📝 Article - The Slow Death of Authenticity in an Attention Economy. This is my main issue with social media, everything just seems so fake. If I am going to connect to strangers on the internet, at the very least I want them to be themselves.

💬 Quote of the Week

The real point of doing anything is to be happy, so do only what makes you happy.

From Anything You Want (affiliate link) by Derek Sivers.

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →

Navigating Docker like a Wizard

Alex Hyett — Sun, 22 Oct 2023 11:35:02 +0000

I have been using Docker for many years now. I still remember the times before Docker and having to manually set up servers with all the dependencies needed to run my applications.

Of course, there are ways to provision servers using scripts such as Puppet or Ansible, but these can either go wrong or if you aren't careful you can end up installing the wrong versions for things.

Using Docker images fixes a lot of these issues which is why they have become so popular. I even use Docker images for running software on my home server.

For those lucky enough to not know a time before Docker, it is easy to take it for granted. Not only that but many developers I have met treat Docker as an unknown black box.

When your application is failing to load though you haven't got any choices but to dig into your Docker container and poke around until you can find out what is causing the problem.

This is going to be a mini guide to solving issues with your Docker containers which will hopefully be useful.

I am going to assume for all these examples that your Docker container is running some flavour of Linux.

See what is running

For this step I am going to assume you have your Docker containers running somewhere that you can at least ssh into.

Running docker ps will give you a list of all your running containers and how long they have been running for.

If you have your service set up to automatically restart and your docker container has a problem then you will likely see something along the lines of Restarting (1) 2 seconds ago or Up less than a second.

If you can't see your container at all then try running docker ps -a.

You can see here my Jellyfin container stopped 3 weeks ago, and it wasn't set up to restart.

Check the logs

If your container is running or constantly restarting, as always, the first place to check is the logs.

You can check the logs of any running or stopped container by running docker logs followed by the container ID. You don't need to specify the full ID just the first few characters so that it is unique.

So to look at the logs for my caddy image I can run:

docker logs 502

Gaining access to the container

If the logs aren't overly helpful then it is time to log into the container and see what is causing the issue.

If the container is running then you can use the following command to gain access to a shell that you can use. Again use the Container ID found from docker ps.

docker exec -it 502 /bin/bash

In some cases, bash isn't available on the version of Linux the container is using. If this is the case you will get a message such as OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown

Not to fear we can use sh instead:

docker exec -it 502 /bin/sh

Once you have access to the container you can run any command you would normally on a Linux system to find clues as to what might be causing the problem.

Typical things I usually try:

Find the log files for your application or the system in /var/log. You can use cat or tail to quickly view them.
Try starting your application inside the docker container if it isn't already running.
In the case of connectivity issues, try using ping or curl to see if you can contact other services outside the container. Sometimes I just try ping google.com to make sure it has internet access.

Installing diagnostic tools

For security and to keep Docker images small, the number of packages installed is kept to a minimum.

This is unfortunately not very helpful if you are trying to diagnose why your service can't connect to something, and you don't have access to basic commands such as curl or ping.

Luckily you can still install packages in the same way you would any other Linux system. How you install packages will depend on what Linux version you are using. You can usually find out by running the following command:

cat /etc/os-release

Which should give you an output like this:

NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.18.3
PRETTY_NAME="Alpine Linux v3.18"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

Then it is just a case of looking at how to install packages in that Linux Distro.

In most cases, you need to be running as root in order to install packages. If the commands aren't working try exiting the container (just run exit) and then use the following command to login as root.

docker exec -it -u="root" 502 /bin/sh

There are various ways to install packages depending on your Linux version. If you don't want to look up which version just try one of these and see which works. For example if you are trying to install curl one of the following should work.

apt update
apt install curl

apk update
apk add curl

yum install curl

dnf install curl

Gaining access to a stopped container

If the container you are trying to diagnose doesn't start up then it is a little bit more work to get it started.

Again you need to find the stopped container ID using:

docker ps -a

You then need to commit the container to a new image. It doesn't need to be called test/image:

docker commit 0488e172aa70 test/image

Then you can try running it:

docker run -it --rm test/image /bin/bash

If your container is failing to run because the entrypoint script is broken you can replace the script with just a /bin/bash or /bin/sh with the following command.

docker run -it --rm --entrypoint /bin/bash test/image

Copying files

You have done some digging, and you have found a log file that you want to look at more closely, but you want it on your local machine.

First note the location of the file inside the container e.g. /var/log/dmesg

Then exit the container and run the following replacing the ID and file with your own.

docker cp 0c7b2063b2a2:/var/log/dmesg dmesg

You can even copy whole folders if you need to:

docker cp 0c7b2063b2a2:/var/log/. log

If you need to copy files on to the container for some reason we can do that by switching the parameters around:

docker cp dmesg 0c7b2063b2a2:/var/log/dmesg

Hopefully this mini tutorial has been helpful, if it has then let me know in the comments.

This newsletter is free for everyone, but if you would like to support my work and my YouTube channel you can do so by becoming a patron on Patreon.

❤️ My supporters also get the following benefits:

📝 Exclusive content from me (including my creator income reports)
🔓 Access to the source code from my YouTube videos
💬 A private chat just for supporters
💰 Exclusive generous discounts once my courses are released

❤️ Picks of the Week

🛠️ Tool - localsend/localsend: An open source cross-platform alternative to AirDrop. One of the reasons I use Apple products is the compatibility between all the devices. I use AirDrop a lot for transferring photos from my phone to my computer. There is now an open-source alternative that looks excellent.

📝 Article - Write more "useless" software. It is a common pattern I see amongst developers. We got into programming because we enjoyed it when we were younger. Professional programming however comes with a lot of baggage whether it be unit tests, CI/CD pipelines, issue tracking or vulnerability checks. If you make something for yourself it doesn't need any of that. Just make something for fun for a change.

🛠️ Tool - FediDevs | List of software developers on Mastodon. Twitter/X is turning into a cesspit of toxicity. That is why I am trying to encourage more people to join the Fediverse. When you first join it can be a bit daunting to find like-minded people, but this is a great too if you are looking for fellow developers to follow.

📝 Article - Hacker leaks millions more 23andMe user records on cybercrime forum. The moral of the story, giving your data to anyone is a terrible idea.

🛠️ Tool - Two Stop Bits. If you are interested in retro computing and gaming then this looks cool.

📝 Article - Relative Time Labels. If you are a designer or a frontend developer it is worth reading this. One of my pet peeves with YouTube is the fact the dates say 1 month ago. If the date isn't recent then just put the date like a normal person.

🛠️ Tool - Piped: An alternative privacy-friendly YouTube frontend which is efficient by design. YouTube have started forcing users who are using ad blockers to either disable them or pay for YouTube premium. If you don't want to see the ads then Piped is one way that you can bypass them.

On a side note, I am thinking of creating a PeerTube instance to host my own videos, so people can watch them without the ads.

📝 Article - Organizing multiple Git identities. If you are a freelancer you may have multiple git accounts that you use. If you need to switch between them, it can be a pain, but this post has a great solution for it.

👨‍💻 Latest from me

As I mentioned last week, I am currently working on a course which is taking up most of my time at the moment. I do have a couple of announcements though.

My blog is now on the Fediverse

I am fully embracing the Fediverse everywhere I can. I am not a big fan of feeding algorithms whose sole purpose is to make money from adverts.

Even though I have fewer followers on Mastodon then I did on Twitter, I get much higher engagement on my posts. At the end of the day, that is what really matters, connecting to real people and making a difference.

I have now set up a special @blog account for my website. If you are on Mastodon and want to keep up with my posts on there you can follow @blog@alexhyett.com.

If you have had any issues following me on Mastodon (@alex@alexhyett.com), I did fix an issue with my server that should resolve that now.

Moving to Patreon Chat

One of the benefits of subscribing to my Patreon is getting to chat to me (or at least I think it is a benefit 😉).

I have been using Discord for this but having a Discord community with so few people is just sad. Luckily Patreon have released Patreon Chat this week, so I am switching to that instead.

If you are a patron then you can use the chat feature now from the mobile app.

💬 Quote of the Week

Start placing a higher value on your time: Time is more valuable than money. You can always get more money, but you can never get more time.

From 6 Months to 6 Figures (affiliate link) by Peter Voogd.

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →

Decentralisation, Our Past and Our Future

Alex Hyett — Sun, 15 Oct 2023 11:35:02 +0000

It is easy to forget that the internet is nothing more than an interconnected web of computers.

Many people think of "the cloud" as some sort of space age tech that they could never comprehend. In reality, it is just computers connected together with some massive hard drives.

The Original Decentralised Web

The first website I ever built was done using Microsoft Word.

There was an option to create a web page, which was essentially just a page without borders. Once you were done adding text and images you could publish it to HTML, and it would create your webpage and images in a separate folder.

Then it was just a case of getting an FTP client and uploading your files to the server. Most ISPs at the time gave you a small amount of webspace to host your own website. It was just part of the standard package along with email.

It is a shame that ISPs don't offer webspace any more. How different would the internet be today if everyone had their own space instead of contributing to social media channels that they don't really control?

The internet is built on protocols like HTTP and Email. Everyone can have their own email address hosted with any email provider they want. As with websites they can even host their own.

Protocols made the internet great, and it wasn't just limited to email and websites. We had IRC for chat and Usenet for discussions.

Anyone could host their own IRC server and join other servers anonymously.

You could pick your own client to use and pick which software to run on your server. As long as the software obeyed the rules of the protocol, everyone could interact.

Back then, the internet was truly decentralised, there wasn't a single company or server that everyone had to interact with.

The great thing about protocols and decentralisation is that they tend to stand the test of time.

This is why email is still going strong 40 years on from its inception.

In fact, you can still use IRC and Usenet as well today. Although admittedly, Usenet is mostly used for sharing pirated content and the lack of history in IRC channels make them less useful than alternatives.

Introducing Centralisation

Of course this all changed in the early 2000s, with social networks such as MySpace and later Facebook.

The internet was no longer about communities getting together to talk about like-minded topics.

Instead, it became a popularity contest.

Everyone flocked to which ever platform had the most users, so they could promote themselves in an effort to get the most likes and shares.

The one thing that these platforms had going for them was the ability to share content without needing to be computer-literate. If you wanted to share your own thoughts and photos you didn't need to build a website and work out how to host it. You just needed to type and click send.

These platforms have let millions of people share content but only within the confines of their own platform.

There are no common protocols that let Facebook users interact with Twitter users or Twitter users like Instagram posts. If you want people on other platforms to see your content then you have to post there as well.

These are centralised platforms that not only control how your content is used but also what content you see.

If you have scrolled through your Instagram feed recently you may have noticed that at least half the posts that you see are not from people you follow. Instead, the algorithm decides for you, based on your viewing habits, what content you see.

Decentralisation comeback

Luckily, thanks to Elon Musk making a mess of the Twitter takeover, people are looking for alternatives.

The best alternative to Twitter at the moment is Mastodon.

If you're wondering what sets Mastodon apart from Twitter, Facebook and all the other social media platforms, the answer is protocols.

Mastodon is based on the ActivityPub protocol. This is what Wikipedia says:

ActivityPub is an open, decentralized social networking protocol based on Pump.io's ActivityPump protocol. It provides a client/server API for creating, updating, and deleting content, as well as a federated server-to-server API for delivering notifications and content.

Mastodon is the software that is implementing the ActivityPub protocol.

ActivityPub has been around since 2018, but it is only in the last year that things have really started to take off.

As with email you are completely free to pick your own client and server or even host your own. You don't even have to use Mastodon as it works with all other platforms that use ActivityPub.

Whether you mostly write or share photos and videos there are options available. As they all use the ActivityPub protocol it doesn't matter which one you pick as everyone can see, like and follow people on other servers.

Some options however are more geared to certain types of content, in which case you may want to have separate accounts for each media type. There is nothing stopping you though from sharing your content on the other platforms.

These are the main options at the moment:

Twitter/ Facebook Alternatives

Mastodon - if you are looking for something that is just like Twitter then this is it. Just like Twitter you can also share photos and videos.
Plemora - This is more of a lightweight version of Mastodon with much of the same features.

Also, worth checking out Diaspora, Akkoma and Hometown.

YouTube Alternatives

PeerTube - it is possible to share video on the platforms above but PeerTube is designed to be a decentralised clone of YouTube.

Instagram Alternatives

PixelFed - If you are only interested in sharing photos then you might be interested in PixelFed. As PixelFed also uses ActivityPub you can also follow people on Mastodon and see their photos like you would on Instagram.

Reddit Alternatives

Lemmy - If you are looking for an alternative to Reddit then Lemmy is the main one. A lot of the communities on Reddit moved to Lemmy after Reddit introduced extortionate API pricing.
Kbin - Kbin is another alternative and you will find a few communities using Kbin instead.
Prismo - Not as popular as Lemmy or Kbin but is an option.

WhatsApp / Discord / Messenger Alternatives

Matrix - if you want private encrypted messaging as well as an alternative to Discord or Slack then Matrix is the main one.

Medium Alternative

WriteFreely - if you want a simple blogging platform with a clean design like Medium then check this out.

As all of these platforms are using ActivityPub they all work together. You can sign up for a Mastodon account and then follow someone on PeerTube or PixelFed and even like and comment. You can even follow your favourite Lemmy communities from Mastodon.

It doesn't matter what server they are on or what software they are using, everyone can communicate as one big worldwide community.

If you are building software that is going to be used by the public then it is worth thinking about how you can make it more decentralised or at least accessible using standard protocols.

My main account is on my self-hosted Mastodon server. The easiest way to follow me is to log in on which ever instance you are using and search for "@alex@alexhyett.com".

I also recently joined PixelFed, so I can share my photos. At the moment I am on PixelFed.Social but I will likely move to a self-hosted instance in the near future.

My plan is to have all of my social channels on self-hosted instances. That way everyone can tell it is me as they will all be hosted on alexhyett.com and there is no risk of them disappearing in the future.

This newsletter is free for everyone, but if you would like to support my work and my YouTube channel you can do so by becoming a patron on Patreon.

❤️ My supporters also get the following benefits:

📝 Exclusive content from me (including my creator income reports)
🔓 Access to the source code from my YouTube videos
💬 A private community (currently just me if you want to ask me anything)
💰 Exclusive generous discounts once my courses are released

❤️ Picks of the Week

📝 Article - Spacedrive is an open source cross-platform file explorer, powered by a virtual distributed filesystem written in Rust.. Most of us have our data spread out across devices. Wouldn't it be useful to just have everything available everywhere. This is what Spacedrive tries to solve.

📝 Article - Building a 42-inch E Ink Art Frame. Digital photo frames were all the rage a few years back, but I don't know about you, I don't like the idea of having a photo frame plugged in all the time. This looks really cool even it can't show colour.

📝 Article - Announcing Cloud Spanner price-performance updates. I have not used Google Cloud spanner before, but I know from experience how expensive AWS DynamoDB can get. Apparently Cloud Spanner is now half the cost of DynamoDB!

📝 Article - Roll Your Own All-Sky, Raspberry Pi Camera. I have always loved space, which is one of the reasons I took Physics at University. At the moment I live in an area with too much light pollution. If that changes I can see myself building something like this.... and getting a telescope!

📝 Article - Can't Be F*cked: Underrated Cause of Tech Debt. I think we can all relate to this from time to time. Sometimes the effort of doing it the right way is just too much that day.

📚 Book - Tomorrow and Tomorrow and Tomorrow (affiliate link) by Gabrielle Zevin - I started reading this book this week. In short, it is a love story about two friends who decided to build a game together. I am enjoying it so far.

💬 Quote of the Week

We have an inherent drive towards safety in the sense of not being harmed. But when we default to a comfort zone, we may feel unsafe any time we step outside of it, when really we’re safe, just uncomfortable.

From Why Organizations Fail To Unleash Human Potential by Paul Millerd.

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →

What should a Junior .Net Developer know?

Alex Hyett — Sun, 08 Oct 2023 11:35:04 +0000

Back in March I put together the Backend Developer Roadmap which has been my most popular YouTube video to date and has been watched over 66,000 times.

The point of this roadmap was to cover everything I know as a backend .net developer so that others have a path that they can follow. I have been a software developer for 13 years now, so I have had a while to accumulate all of this knowledge.

Unfortunately, some assumed this was everything you needed to know to even get started as a backend developer.

As I have mentioned in my replies, if you know everything on the roadmap you would likely be considered a senior engineer or higher.

So in this post I thought I would break down the roadmap into what I would expect engineers to know at different levels. This is of course my personal opinion, so you may not agree with my choices!

Junior Developers

The range for what junior developers know is quite broad. Some would have gone to university or college to learn computer science and will therefore have a bit more knowledge than someone who is self-taught.

The majority of computer science courses however do not cover the skills that are actually needed to be a professional software developer.

I would expect most junior .net developers to know the following:

Have a decent understanding of C# and OOO. They don't need to be an expert but should at least know how to do the things that are common across all programming languages.
Can understand and use Git and version control in general.
Understands the main data structures and a few algorithms.
If they are going to be working in web development they should have a basic understanding of how things like domain names, DNS and web hosting work and fit together.
Not required, but I think most developers should know HTML and CSS.
Have a basic understanding of how computers work such as what gets stored to hard drives vs what is stored in memory.
A basic understanding of threads and concurrency and what a race condition is.
Understand REST and be able to code a basic API.
A basic understanding of databases, what they are used for and how to write a SQL query.
Can use Google, StackOverflow and ChatGPT to get answers to their own questions.

Knowing the above should be enough to get you a job as a junior developer and get you through your first year working on tickets as part of a team.

Mid-Level Developers

To progress from junior to mid-level you need to be able to do more than just complete small coding tasks. By mid-level you should be able to code a complete application from scratch to a production standard.

Therefore, adding on from the above I would expect a mid-level developer to be able to know and do the following:

Understand the main coding principles such as SOLID, DRY, KISS and YAGNI.
Have a full understanding of how to build an API including:
- Routing
- Authentication & Authorisation
- Validation frameworks such as FluentValidation
- Dependency Injection
- HATEOAS
- Open API specs
- CORS
Have a decent understanding of RDBMS databases:
- Indexes
- Normalisation
- Stored Procedures and Functions
- Transactions / ACID
- The main differences between Postgres, SQL Server and MySQL
How to use an ORM and the main advantages and disadvantages.
Have a basic understanding of NoSQL databases and how to use them.
Understand how to log properly and some of the frameworks you can use.
An understanding of the Gang of Four design patterns with decent knowledge of a few of them.
Be able to perform code reviews and provide constructive feedback.
Know how to do unit testing and integration testing and understand the other types of testing.
A good understanding of agile meetings sufficient to be able to run them yourself.
Know how to use Docker and how to containerise your application.
Know your way around a Linux terminal (useful for debugging docker containers).

The above should be enough that if you were given a task to produce a whole component as part of a larger system you would be able to do it without too much trouble. I wouldn't expect a mid-level developer to design a whole system themselves or figure out all the infrastructure, but they should be able to take ownership of a single component.

Senior Developers and above

A senior developer should be able to be given a problem and be able to design and develop a scalable solution for it.

We are no longer considering just a single component but a whole system and how all the pieces fit together and the infrastructure supporting them.

Senior developers and above should therefore be able to do the following:

Be able to design a whole system including what components to build, how they interact and how data is stored and flows throughout the application.
Understand the pros and cons of microservices over monoliths and when to use one over the other.
Know how distributed caching works and when to use it.
Know when to use NoSQL vs SQL databases and be able to design the schema for both.
Understand what indexes might be needed on a SQL database and how to write performant queries.
Be able to set up a complete CI/CD pipeline for your applications.
Understand and know how to use key design patterns such as Domain Driven Design, CQRS and Event Driven Architecture.
Have a good understanding of one or more cloud platforms and be able to select and use the right option for the job.
Understand the different types of performance testing and how to do them (see this week's YouTube video)
Understand advanced API options and when to use them such as gRPC, GraphQL and WebSockets.
Know how to monitor your applications and set up dashboards and alerts.
Know how to use infrastructure as code (IAC) to provision all of your infrastructure.

This is what I think developers at various levels should know but of course each company will have their own competency matrices that you need to look at if you are looking to get promoted. There are often a lot of soft skills involved as well that you need to master in order to move up the ranks.

The levels are rarely as clear-cut at this. As a junior you may know things that a senior should know and equally a senior may have missed something from their earlier career paths.

This is essentially the order I have learnt these topics over the last 13 years. It may seem daunting at first, but each bit of knowledge just builds on previous knowledge, and it doesn't seem that daunting once you get there.

I am interested to know what you think of the above levels and whether you agree, let me know in the comments.

This newsletter is free for everyone, but if you would like to support my work and my YouTube channel you can do so by becoming a patron on Patreon.

❤️ My supporters also get the following benefits:

📝 Exclusive content from me (including my creator income reports)
🔓 Access to the source code from my YouTube videos
💬 A private community (currently just me if you want to ask me anything)
💰 Exclusive generous discounts once my courses are released

❤️ Picks of the Week

📝 Article - Where are all the laid-off software developers going? Part 2. With all the lay-offs this year it is interesting see where everyone is going. It seems many developers are getting more luck with companies outside big tech or starting their own companies.

📚 Book - Introducing “Database Performance at Scale”: A Free, Open Source Book. How doesn't like free stuff? This looks like a great book on database performance.

📝 Article - Where does my computer get the time from? An interesting look at NTP and where your computer gets the time from.

📝 Article - No More Vendor Lock-In? The Rise of Sky Computing. I hadn't heard of Sky Computing until this article, although it makes perfect sense. I have done a lot of work with AWS in the past, and it usually involves using their SDK in your code to interact with their services. Sky computing decouples your applications from particular cloud providers making it easier to use multiple clouds or migrating between them.

💬 Discussion - Google Docs infects HTML with tracking links. Google has come a long way from it's "Don't be evil" motto. Its new motto is "Do the right thing", which I can only assume by their actions is "Do the right thing to make the most money and exploit all our users".

📝 Article - BBC Gives Up On Threads (By Instagram), Sticks With Mastodon. I have been really enjoying Mastodon (just search for @alex@alexhyett.com on your instance to find me), however, it still has quite a niche user base, and it will be a while before everyone moves over. I am sure what will help with that is major brands and influencers making the move too. Threads have said they will be federated at some point, but time will tell.

🛠️ Tool - Introducing pgroll: zero-downtime, reversible, schema migrations for Postgres. For those using Postgres I am sure this is going to come in handy.

📝 Article - Hey, Computer, Make Me a Font. Generative AI is here to stay, and it has more uses than just turning Jack Black into all the Disney princesses. I love seeing innovative uses for it and being to make new fonts is quite a cool one. I am sure we will see more things like this especially as the requirements come down, Stable Diffusion XL on a RPI Zero anyone?

🛠️ Tool - Learn and Test DMARC. I have had my email on my own domain for a while now. Some of the things you need to set up are DMARC, SPF and DKIM. This is a fascinating way of testing them.

👨‍💻 Latest from me

I said to myself when I got to 10,000 subscribers on YouTube I would start making some courses. Now that I have reached that milestone I have run out of excuses!

I am going to be spending the next few months focusing on getting that first course out. This will mean fewer YouTube videos however I will still be keeping up with this weekly newsletter.

🎬 YouTube - How to do Performance Testing with k6. I have been wanting to do a video on performance testing for a while but wasn't sure the best way to do it justice. While tidying up my office I found a Raspberry Pi 2, just what I needed to push it to the limits.

💬 Quote of the Week

"To attain knowledge, add things every day. To attain wisdom, remove things every day." - Lao Tzu

From Building A Second Brain (affiliate link) by Tiago Forte.

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →

How to do Performance Testing with k6

Alex Hyett — Fri, 06 Oct 2023 15:02:02 +0000

If you haven't done any performance testing before, then k6 is a really simple free tool that you can use to do it.

Now you might be wondering what is the point of performance testing, and why do I need to bother with it?

You might already be writing unit tests and integration tests to make sure that your application is working as designed. These are both forms of functional tests, as you are testing the functionality of your application.

However, we also need to consider non-functional tests. Non-functional tests are for testing things like security, efficiency, reliability and performance.

We are going to look at performance testing, which will give us an idea of how our application is likely to perform under various real world conditions.

Subscribe for more video content

In particular, we are going to have a look at:

Load Testing
Stress Testing
Spike Testing
Soak Testing

I will cover what each of those are and why we might want to do them as we go through the examples.

Getting started with k6

To get started we need to install k6 on our machines.

There are a number of ways to do this depending on what platform you are using.

If you are on a Mac like myself then you can use Homebrew:

brew install k6

On Windows, if you are using Chocolatey you can do:

choco install k6

There are also packages available for Linux and there is also the option of running it as a docker image.

For the examples shown in this article, I am going to assume that you have k6 installed directly on your machine.

Simple k6 Test

K6 tests are all written in JavaScript, and you don't need many lines of code at all to get started.

So let's start a new test. First we to import the http package from k6.

import http from 'k6/http';

We then add in an export default block and inside we can basically add any JavaScript code we want.

import http from 'k6/http';

export default () => {
  http.get('http://localhost:5157/api');
};

Here I am doing a GET call to an API running locally on my machine.

If I run this then it will call the API once and then stop. Which is obviously not particularly useful by itself, but we can see here the details that we get back in the output:

In particular, we can see how long the request took with http_req_duration and how many requests per second it managed http_reqs.

To make this more useful we are going to add in some options. K6 has the concept of virtual users. This is the number of parallel requests that we are going to send to our API.

If we just add in 1 user with a duration of 10 seconds then this is going to call our API as many times as it can, one call after another, until the time runs out.

import http from 'k6/http';

export const options = {
  vus: 1,
  duration: '10s'
};

export default () => {
  http.get('http://localhost:5157/api');
};

Test API

Now to be able to demonstrate all these different test types I have put together a very simple .Net Core API that if you put in your date of birth, it will calculate how old you are in years, months, days, hours, minutes and seconds.

GET http://localhost:5157/age/1990-03-02

{
    "years": 33,
    "months": 6,
    "days": 25,
    "hours": 10,
    "minutes": 35,
    "seconds": 37
}

This application doesn't read or write from a database, so it should only be bound by CPU and Memory limits.

All the code shown in this article and my video is available to my Patreon subscribers. Subscribers also get access to my private discord community as well discounts to my courses when I release them.

Realistic test set up

The whole point of performance testing is to determine how your application is going to respond under certain conditions.

Your application isn't going to be running on your own machine when its in production, so you really need to do your performance testing against an environment that is as close to production as possible.

Now, I wouldn't recommend running performance tests against production itself, as you will see, some of these tests will really push your application to its limits, and it would likely cause problems for your users.

For this tutorial, I am not going to run my API on a production like system, but I am going to run it on a separate machine, so we can see how it performs under load.

I am using an old Raspberry Pi 2 that I have sitting around with:

900MHz quad-core ARM Cortex-A7 CPU
1 GB RAM

Given this is a low power machine we should be able to really push it to its limits with these tests.

Load Test

The first test we are going to look at is a standard load test. The purpose of a load test is really to determine how your application will respond under an average load.

If on a typical day your application gets 200 request per second then that is what you would set your load test to.

At the moment our script is set up to run as 1 user with one call after another. For this test we need multiple users with 1 call, per user, per second.

To do that we need to add in a sleep which is also included in the k6 package.

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  vus: 1,
  duration: '10s'
};

export default () => {
  http.get('http://localhost:5157/api');
  sleep(1);
};

With load tests we want to slowly ramp up the number of requests to our target of 200 requests per second. Then we keep the test running for a decent amount of time, say 20 minutes, and then ramp it back down again.

We can do this by setting up stages in our options block. So we can get rid of what we have currently and replace it with the stages:

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  stages: [
    { duration: '1m', target: 200 }, // ramp-up
    { duration: '20m', target: 200 }, // stable
    { duration: '1m', target: 0 }, // ramp-down 
  ],
};

export default () => {
  http.get('http://localhost:5157/age/1990-03-02');
  sleep(1);
};

This is good, but it is going to call our API with the same parameters over and over again. I haven't got any caching set up, but generally it is good practice to vary the test data, so it is more realistic to a real life load.

With k6 you can set up a shared array that gives you a place to store data for your tests. As my test needs dates as an input, I can write a quick function that will generate me 100 dates within the last 100 years, and add that to the shared array.

I then just need to select a random date from my array and pass that in as a parameter.

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  stages: [
    { duration: '5m', target: 200 }, // ramp-up
    { duration: '20m', target: 200 }, // stable
    { duration: '5m', target: 0 }, // ramp-down 
  ],
};

const dates = new SharedArray('dates', function () {
    var dates = [];
    var currentDate = new Date();
    var minDate = new Date();
    minDate.setFullYear(currentDate.getFullYear() - 100);

    for (var i = 0; i < 100; i++) {
        var randomTime = Math.random() * (currentDate.getTime() - minDate.getTime());
        var randomDate = new Date(minDate.getTime() + randomTime);
        dates.push(randomDate.toISOString());
    }

    return dates;
});

export default () => {
    const randomDate = dates[Math.floor(Math.random() * dates.length)];
  http.get(`http://localhost:5157/age/${randomDate}`);
  sleep(1);
};

You will also want to make sure that your application is actually responding correctly. There are various ways to do this, but it can be as simple as checking the status code of your responses:

export default () => {
    const randomDate = dates[Math.floor(Math.random() * dates.length)];
  const res = http.get(`http://localhost:5157/age/${randomDate}`);
  check(res, { '200': (r) => r.status === 200 });
  sleep(1);
};

Load tests are usually used to make sure that our application is meeting specific performance requirements. You might have a requirement that says requests need to come back within 100ms.

With k6 you can add in expected performance thresholds as part of the setup, and it will test those for you.

For example, here I am adding in a requirement that 99% of requests should come back within 100ms.

export const options = {
  stages: [
    { duration: '5m', target: 200 }, // ramp-up
    { duration: '20m', target: 200 }, // stable
    { duration: '5m', target: 0 }, // ramp-down 
  ],
  thresholds: {
        http_req_duration: ['p(99)<100'], // 99% of requests must complete within 100ms
   }
};

You can then add this test as part of your CI/CD pipeline to make sure any changes you have made haven't affected the performance of your application.

When running performance tests it is important to make sure you are monitoring your application and server to see how it is responding. You don't want your server running out of memory or staying at 100% CPU for the entire time.

In my case, I am using htop to see how my Raspberry Pi responds at different levels. Ideally you would have some proper monitoring set up using something like either Grafana or Datadog.

Once complete you will get an output like this:

Note: Yes I did only run this for 6 mins just to show the output, but an actual load test would be longer 😉.

You can see the tick next to http_req_duration which is showing that it passed the threshold that we set.

Stress Test

With a stress test we want to see how the application will handle an increased load.

You can pick a realistic number for this which might be a 50% to 100% increase in the number of requests. Alternatively, you could increase the number of requests until your application breaks, so you know what your system is capable of.

In the example below, I am slowly ramping up the number of requests over the space of a minute, keeping it stable for 5 minutes before ramping up the number of requests again.

import http from 'k6/http';
import { sleep } from 'k6';
import { SharedArray } from 'k6/data';

export const options = {
    stages: [
        { duration: '1m', target: 200 }, // ramp up
        { duration: '5m', target: 200 }, // stable
        { duration: '1m', target: 800 }, // ramp up
        { duration: '5m', target: 800 }, // stable
        { duration: '1m', target: 1000 }, // ramp up
        { duration: '5m', target: 1000 }, // stable
        { duration: '5m', target: 0 }, // ramp-down to 0 users
    ],
};

const dates = new SharedArray('dates', function () {
    var dates = [];
    var currentDate = new Date();
    var minDate = new Date();
    minDate.setFullYear(currentDate.getFullYear() - 100);

    for (var i = 0; i < 100; i++) {
    var randomTime = Math.random() * (currentDate.getTime() - minDate.getTime());
    var randomDate = new Date(minDate.getTime() + randomTime);
    dates.push(randomDate.toISOString());
    }

    return dates;
});

export default () => {
    const randomDate = dates[Math.floor(Math.random() * dates.length)];
    http.get(`http://localhost:5157/age/${randomDate}`);
    check(res, { '200': (r) => r.status === 200 });
    sleep(1);
};

This is how my Raspberry Pi responded at each stage.

200 req/s

At 200 req/s things are stable with average CPU under 20%.

800 req/s

At 800 req/s the Raspberry Pi is starting to show a bit of strain with CPU staying around 50%

1000 req/s

Finally at 1000 req/s surprising the Raspberry Pi is still holding with an average load around 55%.

We would have to push this even further if we wanted to push it to breaking point, although a sustained load of that amount might not be realistic.

Here are the results:

We can see that the average and max response time is a lot higher compared to what it was with the load test, but it was still respectable.

Spike Test

There may be occasions where you get a sudden spike of traffic that quickly dies off.

Maybe your application gets on the front page of a HackerNews, and suddenly it has to deal with a very high load in a short space of time.

This is where spike tests come in, where you have a very quick ramp up of traffic that lasts a short amount of time before quickly dying down.

In this example we quickly increase traffic to 2000 req/s, holding for 2 mins and then quickly drop back down again.

export const options = {
    stages: [
        { duration: '30s', target: 2000 }, // ramp up
        { duration: '2m', target: 2000 }, // stable
        { duration: '30s', target: 0 }, // ramp-down to 0 users
    ],
};

You can see at 2,000 req/s my poor old Raspberry Pi is struggling a bit.

Occasionally some of the cores did reach 100%. The memory usage was also a lot higher than it was in the other tests.

I didn't include the 100ms threshold in this test, but you can see from the results that it would have failed as the 95th percentile reached 138ms.

Soak Test

In some cases, running a load test for half an hour isn't going to be enough to gauge the stability of your application.

Things like memory leaks and excessive disk space usage happen gradually, and you aren't likely to notice them unless you run an application for an extended period of time.

The soak test is similar to the load test we did earlier, except you are running it for a very long time, typically around 8 hours, but you can run it for longer if you need to.

Generally, we set the number of virtual users to match the average load on your system, but if you are trying to test for excessive use of resources like disk space, then you may want to increase that amount, so you don't have to run the test for so long.

You can also use soak tests to make sure that you don't lose any requests if your application has to restart, or to check there aren't any rate limits on any third party APIs that you are using.

The stages for a soak test will typically include a short ramp up of traffic followed by the sustained load and then a short ramp down.

export const options = {
  stages: [
    { duration: '5m', target: 200 }, 
    { duration: '8h', target: 200 },
    { duration: '5m', target: 0 }, 
  ],
};

Final Thoughts

I hope this overview of performance testing has been useful. Make sure you check out the k6 documentation as there are a lot of additional options on there that might be useful for you.

📨 Are you looking to level up your skills in the tech industry?

My weekly newsletter is written for engineers like you, providing you with the tools you need to excel in your career. Join here for free →