DEV Community: J

Be friendly and don't ignore Recruiters

Thu, 10 Mar 2022 00:00:00 +0000

Increasingly, I’ve noticed an increased level of resistance to recruiters among engineers. There has always been a love/hate relationship between the two parties. From the beginning of my career, I always heard “recruiters are bad”. And, to be honest, I accepted that as truth for the longest time. But, reflecting on it now, I don’t understand it at all.

I’m writing this article for fellow software engineers to not accept the rhetoric of what others say - see for yourself. In a nutshell, you and I are not “better” because we are engineers. Recruiters are another role that is necessary for the sector to function.

Here’s my advice for software engineers dealing with recruiters.

1. Use canned responses

One of the main aversion to recruiters is sending countless emails to irrelevant jobs. This is understandable annoying. Especially when you are specific about not looking for a job, or the types of job you want. But consider for a second why this happens. Recruiters are paid for successful placements. Because there is a shortage of talented engineers, it figures that most engineers are already employed. If recruiters took the fact that someone was already employed as a reason to not approach them, it would be impossible to hire. For myself, I’ve always been employed when approached about another job.

So, we can’t avoid the inMail and emails. But, what’s the solution? Canned Response

Canned responses save you time by being clear about your expectations for a job and what kind of work you might be open to. You can find many templates for this online, but they are mostly full of snark.

Here is one I use:

Hi X, 

Thanks for your message! This looks like a fantastic opportunity.

Unfortunately, I have recently accepted a position at a new company and am not currently seeking new opportunities.

I’ll be sure to bear you in mind for future roles.

Kind regards,
Josh
https://joshghent.com

Short, friendly, clear. Simple as that.

Or, if you are open to work. Here’s another template I use for that:

Hey X,

Thanks for your message! This looks like a fantastic opportunity.

Although this job doesn't fit me, I am currently looking for a Remote Senior Developer position working with NodeJS, React, and AWS. I have worked with these technologies for over 5 years across an array of projects. Most recently, I have been architecting and building a greenfield project for a large eCommerce company.

I have attached my CV in case you have anything that fits the bill.

Kind regards,
Josh
https://joshghent.com

Again, it’s short and sweet but gets across the message.

2. Recruiters are great at building communities

Due to being involved with a large network of developers, recruiters are incredible at creating communities. Many events that I have run, have been so well supported in large part due to the work of recruiters leveraging their networks. Meeting these people at events can then further your own career.

Additionally, they have more access to commercial ends of businesses. If you have a technical event you’re running, recruitment agencies are usually among the first to sponsor the event. They get the exposure and you get the finances to run a kick-ass event. Win, win. Leverage these resources that they have access to.

3. Don’t let a bad egg put you off the whole batch

Now, I know what you’re thinking. You’ve read this article so far and said to yourself “that’s all well and good. But this recruiter was truly awful”.

I agree.

There are bad recruiters out there. Terrible ones. But there are lots of bad engineers too. There are bad healthcare workers, builders, architects, designers, painters. With anything and everything, there is a “bad” version of. And that’s ok.

At worst, a “bad” recruiter might spam you with some emails or calls. You can easily block these. A bad engineer might give you a haunting nightmare of yarn that you have to untangle over the next year. The effects of these are vastly different in size.

But just as when we recognise a bad engineer, we don’t assume all engineers are bad. We should think the same about recruiters. Don’t let one bad egg spoil the whole batch. There are good eggs out there. This brings me nicely onto my next point…

4. Work with individuals, not companies

A recruitment agency, like any other company, is a faceless emotionless entity. Inside each company, there will be some great people and some not so great people. Find the individuals in those businesses that you get on with and place you into jobs you enjoy. Then work with them throughout your career. Personally, I’ve held 5 jobs given to me by 2 recruiters that I work with and have built up trust over time.

You can build trust with them by running events with them and referring people in your own network to them.

5. Accept them as part of the process

Many believe we should live without recruitment agencies entirely. This is an understandable viewpoint. But this belief underscores a fundamental misunderstanding about business - stuff costs money. And if that “stuff” is hiring people, then it costs a lot. Why? Reviewing resumé’s/CV’s, interviewing and technical skill tests all takes time. Time from someone who is paid by the company. A recruitment agency’s main value offering lies in getting high-quality candidates from a large network and handling all the marketing associated with advertising for a job.

Just as many developers “outsource” their code by using third party libraries to save time, businesses do the same with recruiters. It saves time, and there is no point in reinventing the wheel. It allows them to unlock access to resources that would have taken a considerable amount of time to develop otherwise.

For better or worse, recruiters are part of the process of getting a job and are here to stay.

Hopefully, this post has helped soften your attitude toward recruiters. I’m not trying to win favours with recruiters by writing this. It’s a response to several snarky posts about the recruitment industry and tech. At the end of the day, these are people trying to do their jobs - like you and I. Sure there are some bad apples, but where isn’t there? Default to truth and follow the advice outlined above, it will work out in your best interests to do so.

Mistakes I made as a self-taught developer

Tue, 15 Feb 2022 00:00:00 +0000

Learning to become a software developer is not a trivial task. There is a plethora of guides, tutorials and courses to take. And of course, there is the question of self-teaching or going to university.

But no matter, which path you chose you will always make mistakes.

I made a tonne of mistakes. And I want to share them with people who are learning software development. Here are 5 mistakes I made whilst learning to code.

1. Analysis paralysis of resources

I wasted a lot of time analysing the “best” place to learn to code. Early on, I often found myself reading articles about how “good” Codecademy was vs. a collection of “Head-first” books. This time would have been better used on doing the work. It’s easy to enter this state of panic thinking you’re committed to a certain learning path. But the truth is, you aren’t. Mix and match, try different things and go with what works for you.

I also did this same analysis when choosing a language to learn.

Is Python the best? What about Javascript? My favourite sites use Rails, maybe I should learn that?

That was the list of questions that plagued me. It was exhausting and I shouldn’t have spent so much time overthinking this. My advice for new programmers is to learn Javascript and web technologies. Yes, you might want to learn games, but Javascript is so widely used (now even in space!) that it will allow you to go down any path you want later on.

Lesson : Just learn Javascript. Mix and match resources that work for you to build a consistent practice of learning.

2. Not building early and often

What do I mean by “building”? I mean creating libraries, API’s, demo sites and more. But to begin with, I thought the idea of “building” something was too complex to handle. Translating the code into something practical would have made me familiar solving problems. When I did start building projects, I found it challenging to shift from syntax to finished products. To liken it to real languages, it’s the difference between knowing the word for “Apple” and knowing how to say “Would you like an Apple?“.

This fixation on learning the syntax also had the downside that I didn’t have anything to show for it at the end. To a potential employer, I was just someone who said they had learnt to code and could do some simple problems. By having a portfolio of projects that I could show off, I would have proven to myself and others that I had the skills to work.

Lesson : Apply syntax to real-world style projects after learning it.

3. Being tied to tutorials rather than problems

In line with the above, I spent too much time on specific tutorials. I should have learnt to break a project down into small problems and seek solutions. This practice of breaking larger projects into small problems would have been valuable when I started working. It would have also helped me train my “google-fu” to search out error codes, and problems I needed to solve.

Lesson : Learn to break projects down into small solvable problems.

4. Sweating “interview” preparation

Before I began to interview I spent hours doing code “katas” - small challenges using no libraries in your language of choice. I did this because they are supposedly common interview questions. But, I have never been asked to do specific coding challenges like this on a whiteboard or otherwise (having interviewed for 7 jobs in total). What I have had is take-home projects, and asked general technical questions.

Lesson : If you are interviewing for the FAANG companies you are likely to get these questions. If you are not, then skip this. Focus on projects instead.

5. No SQL Exposure

In self-taught land, starting a new project is quite simple. Install NodeJS, install React, launch Chrome - job done. But, installing and using SQL? That was far too scary for me to tackle. There were ports to configure, connecting it to NodeJS and then the table structure. In part, MongoDB gained popularity because it’s so simple to set up. By not having the exposure to SQL, I struggled when I got a job.

It also meant that my coding style tended to lean on parsing data within the language. Over time, I’ve trained myself (for software performance reasons) to use the database to crunch and mould the data for me.

Lesson : My advice on this point is to set up a free account with Render.com or Heroku and add a MySQL or PostgreSQL instance.

If you’re learning to become a software engineer, don’t give up! It is difficult no matter which path you chose. I hope by listing my failures that you can avoid them. You will make others in your journey and I implore you to write about them and learn from them.

Building Collaboration with Remote Teams

Tue, 08 Feb 2022 00:00:00 +0000

TL;DR : Provide the tools, empower people to use them and embrace remote work for what it is - remote.

Steve Jobs designed Apple Headquarters to maximise the length of time it would take people to get to the bathroom. He did this to increase collaboration stemming from running into others in the corridor.

Since the pandemic, remote working skyrocketed. Owl Labs recorded in 2021 that almost 70% of full-time workers in the US were working from home. And based on the stats from job boards such as RemoteOk.com, you can see a massive uptick in remote job opportunities that are not dying down.

But the question is, how can you “bump into” people in the corridor in the remote-first working world? Many say that weekly meetings to keep people aligned are the answer. But I disagree, meetings are viewed with disdain. The same study by Owl Labs found that 80% of remote workers wanted at least a day per week without any meetings at all.

Instead, I’d suggest a different approach.

Make communication public by default. The main rebuff of remote work is that the communication is too “formalized”, recorded and preserved. People use that excuse to communicate only in direct messages. But, putting messages in public channels, allows everyone to be informed and to take part. Additionally, making individuals comfortable with public communication will open the doorway for asynchronous work. It will discourage people from calling meetings simply to gather a consensus or input.
Don’t recreate the watercooler. Commonly, I’ve seen teams create a #watercooler chat in Slack, an informal space to post memes and have a chit-chat. Although in ultra-large businesses I’ve seen these spaces improve individual relationships, I have not seen them successfully increase collaboration. These spaces are attempting to recreate an in-person space. Whilst well-intentioned, these spaces do not translate to the online realm. Embrace remote work for what it is, remote. It will naturally take time for an in-person organisation to change into the “remote” frame of mind.
Avoid hybrid working. Where possible, avoid having some of the team in an office and the rest remote. I’ve been on both sides of the office divider for this one. And it doesn’t do well on either. On the office side, you can often be blocked by a remote worker not working the same hours and not having the tools to perform a task asynchronously. But on the remote side, you have a constant FOMO for all the undocumented decisions that have been made. Pushing all this communication asynchronous and online alleviates these issues and keeps everyone up to date.
Use the tools. We are all using Zoom and Slack. But that’s not the end. There is a myriad of tools to help you unite a remote team. Use Google Drive for documents and presentations (share them without a meeting!), Tuple for pair programming and GitHub for tickets and code. Directing people toward these channels (asynchronous) and away from meetings (synchronous) will accelerate collaboration for your team by accomplishing meaningful objectives.
Build a documentation culture. In an office, decisions can be made and passed around in small conversations. In remote teams, that vanishes. So, make sure that every decision is documented in a consistent and precise way. Encourage people to write about problems they’ve solved and how they solved them. Help people to write up accurate guides on getting up and running with a system. And document various approaches to a particular ticket. Make sure you lead by example here. Don’t rely on Slack to store a decision. If you ever hear the phrase “I forget exactly what was said”, it is a chance to write it down. Further, address this at its source by making sure to hire individuals with good writing skills.

Not all these techniques will work with your team. I encourage you to experiment and see what works.

Overall, embrace the work situation you are in and capitalize on its advantages. If your business can be in-person, capitalize on the fact that it will likely be more collaborative. If you have a remote business, embrace a diverse global workforce, lower costs and asynchronous work for increased productivity.

Facing the Legacy Code Monster

Tue, 25 Jan 2022 00:00:00 +0000

I start new jobs like a spelunking caver, exploring all the systems, code and pipelines. I time myself to see how long I can ask questions about a system before I get a dreaded response:

“Oh, that’s a critical system that handles our core business. A developer wrote it years ago, who has since left. Now we reboot it when it breaks. You don’t want to mess with that.”

I’m sure we’ve all heard words like that.

It seems almost a law at this point that every mature software company has at least one system that “works”. But, has a few horrendous bugs meaning it needs rebooting and there are workarounds to its inflexibility that have built over time.

It’s software that has no tests. Runs on an extremely specifically configured server (likely in a closet in the office). Where documentation has been passed down in folklore. And the person who originally wrote it now resides on the moon where they are unreachable.

Oftentimes, this software is left to rot because it’s so critical that it’s safer to have the workarounds and reboots than to fix it for good.

But, you shouldn’t be deterred by this rhetoric of what current employees tell you. Why not?

No one deliberately writes broken or bad software. I say no one, there are of course exceptions. But buy-and-large, software is written with good intentions in mind. And as it is running in production, it means it solved the original problem. Adopting this ethos will help dissolve the image of this evil unknown developer spiting you.
Current employees might be misinformed. Depending on how long ago the software was implemented, current employees might have been misinformed. Swaths of developers may have come and gone and simply repeated what they were told on their first day. What started as a healthy respect for a critical system, soon becomes the subject of fear and anxiety.

Fighting these tendencies to buy into the anti-legacy cult will mean you can deal with legacy code. It will be dirty work, but gaining a deep level understanding of these systems will make you a hero among your team and an invaluable employee.

How can you gain insight into these systems?

Understand why it became legacy in the first place. Finding the why on these systems is vital because it can help clarify your own journey’s priorities. In some cases, it might be that the system was impossible to run in a sandbox. If so, that should be your first port of call. Focus on first principles and get to the core of the problem you’re trying to solve.
Get it running in a sandboxed environment. Provide safety for yourself and others, by setting this up to run in a sandbox. This might need the buy-in from someone on the DevOps team. This is a critical dependent step to working with a legacy system as without it you do not have the psychological safety to make changes. This process might take a long time and should be considered just as important as writing tests.
Follow the code, and document it. Target your reading of the codebase by following the various paths it takes. Review the entry points and build a map of the various “journeys”. Then, pretending you are one of those calls, follow the path that the code takes. It’s sometimes helpful to draw these paths on paper for further reference. After reading it through once, read it again, but document the functions as you go. In some places, you might not know “why” something has been written a certain way; mark these areas with a quick TODO comment for later review.
Write integration tests. Not unit, functional or anything else. Focus on integration at the moment. This maximises the test coverage whilst minimizing the level of understanding you need. Early on in this investigation, you likely don’t know all the in’s, out’s and gotchas of the system. So, to avoid getting overwhelmed, it’s best to write a suite of integration tests that mirror the “journeys” you discovered in the previous step.
Share the knowledge. Don’t let yourself become another bus factor. Be quick to share the knowledge, even if there are gaps. This also encourages other developers to get involved, helping write tests and documentation.

This process is by no means perfect. But, be sure not to skip any of the steps, as they are all as critical as each other. It will likely not mean you are a complete expert in this system. Nor will it mean you can rewrite it in a more modern tech stack with the confidence of a solid test suite behind you. But, it does mean you have shed some light on an otherwise dark scary legacy code monster.

How to Ship Software Faster

Tue, 18 Jan 2022 00:00:00 +0000

Remember when software came on a physical medium like discs, USB sticks or punch cards? Me either. Software release lifecycles used to be lengthy - years-long in most cases.

As software flourished on the web, we grew accustomed to “moving fast and breaking things”. This approach has a lot of drawbacks. Not least because some customer bases are more sensitive to problems than others.

They still wanted to “move fast”, but not “break things”. The speed of the web, with the safety of physical releases.

The solution that many teams came up with was to mass up lots of work and then release it every so often. Taking the bad of both release systems. The lack of safety from a “move fast” release and the slow speed of physical releases.

It’s likely that you work in a place like this, or have worked there in the past. Organisations where DevOps are a secondary concern to the application itself. Places where “continuous delivery” is considered voodoo reserved for the FAANG’s.

I have found myself at these organisations all my working life. I quickly noticed the following patterns:

Developers have little to no confidence that a new release will not break something.
That low confidence means there is anxiety when it comes time to release.
The time between releases meant upstream work causes conflicts.
Manual testing cycles had to be done to establish any confidence.
Bugs upon release caused finger pointing; with psychological safety diminishing as a result.

What can you do if you notice these patterns?

The solution is to ship faster. How?

Set expectations of delivery time. Start by opening a discussion, with stakeholders, about what the expected time to ship new versions will be. Establishing these rough boundaries govern the setup of processes used to ship software. Generally speaking, shareholders will want features as soon as possible. But, if you are currently releasing once a month, you should aim to start releasing bi-weekly. Get a bit further along before promising to deploy 30-50 times a day like Instagram.

Make systems observable. Low confidence in releases often originates from systems with little observability. This means that if something does go wrong it’s a nightmare to figure out why. Before starting to increase deployment frequency, you need a system you trust. Focus on the fundamentals - searchable logging, automatic monitoring of key website pages and API endpoints (using UptimeRobot) and automatic tests (integration and unit at least).

Set small concise deliverables. Doing manual releases requires an immense amount of cognitive overhead. Having a small number of tickets and clear deliverables in each release reduces this cognitive load. There is less to remember to test and check. And other areas of the system are less likely to be affected. If releases are simple to do, it’s more likely they’ll get done.

Invest in your DevOps. This is the crucial technical step. There are many other articles about having top quality development tools to aid deployment, so I won’t add to them. But principally, look at the areas that take the most time or have the least confidence, and automate them. For example, a bash script written 2 years ago for bundling the app is unreliable, take time to address this.

Use Feature flags. Often releases get delayed because stakeholders don’t want to reveal new features to customers. Using feature flags allows you to ship unfinished features without breaking things for everyone. A further selling point for stakeholders is that feedback can be gathered from select customers before a full rollout is done.

Make mistakes a non-issue. If the risk of a new release causing a bug is on par with starting a nuclear war, people will shy away from it. By making it easy for developers (or better yet, a blue-green system) to rollback to the last known stable release, it will break down the fear around releasing.

Use checklists. Anything that cannot be automated (or you don’t have time to do so), should be made as programmatic as possible. Using checklists reduces the guesswork out of manual tasks. It means reliable releases are simple to do.

Shipping software faster is a mix of both cultural and technical aspects of an organisation. Both are equally as difficult. Work towards the “release nirvana” that awaits once these systems are set up. Your team will be rewarded with lower blood pressure and your business will be rewarded by getting and retaining more customers.

Save $$$ by Caching Auth0 M2M Tokens

Tue, 11 Jan 2022 00:00:00 +0000

Auth0 is an easy to integrate service that handles all your applications authentication needs. But, if you’ve worked with it before, you’ll know it’s downfalls.

One of them Machine-to-Machine (M2M) tokens; used to authenticate between your services.

But the limits are restrictive for serverless infrastructures. In the free plan you only get 1000 per month. And even on a paid plan, it would be expensive to get the number of tokens you might need in a given month.

The solution is to cache Machine-to-Machine tokens so we don’t need to request new ones until they expire.

In traditional infrastructure, this would be trivial. Save the token globally somewhere and done.

Serverless architectures are a tricky because there is no persistence between instances.

Here’s how to handle caching Auth0 Tokens for AWS Lambda Microservices. But, the same principles apply for other cloud providers.

Create the DynamoDB Table

(or equivalent serverless DB table in other cloud providers)

Set your own name for the table, and set the partition key to token as a String

Add the name of the table as an environment variable CACHE_TOKEN_DB

Retrieve and store tokens

First let’s add a method to store new M2M

// ===
// cacheToken.ts
// ===
import AWS from 'aws-sdk';

const storeNewToken = async (token: string) => {
  const docClient = new AWS.DynamoDB.DocumentClient();
  const response = await docClient.put({ TableName: `${process.env.TOKEN_CACHE_DB}`, Item: { token } }).promise();
  return response;
};

The code is simple enough and fairly self explanatory.

So, let’s move on and add method that we can use in our Lambda Handler to retrieve a new M2M token.

There are two paths for this method

There is a existing unexpired token in DynamoDB, so we use that.
There is no token or only expired ones, so we generate a new one, store it in DynamoDB and use that.

We will design this system to only store one token at a time. This means we do not have to worry about old tokens and filtering them out on each initialization.

So let’s write our method!

// ===
// cacheToken.ts
// ===
import request from 'request-promise';

export const getAuthToken = async (): Promise<string> => {
  const token = await getExistingToken();
  if (token !== '' && hasTokenExpired(token) === false) {
    return token;
  }

  const params = {
    method: 'POST',
    url: `https://${process.env.AUTH0_NAME}.auth0.com/oauth/token`,
    headers: { 'content-type': 'application/json' },
    body: `{"client_id":"${process.env.AUTH0_CLIENT_ID}","client_secret":"${process.env.AUTH0_CLIENT_SECRET}","audience":"${process.env.AUTH0_AUDIENCE}","grant_type":"client_credentials"}`,
  };

  const result = JSON.parse(await request(params));
  if (!result["access_token"]) { throw new Error("No Access Token returned"); }

  await deletePreviousTokens(token);
  await storeNewToken(result['access_token']);

  return result["access_token"];
};

Let’s break this down a little

We first get the existing token in DynamoDB. It returns the token or an empty string.
If it returns a token, we check it’s not expired and then return that token.
If it is expired, or there is no token, we go ahead an generate one from Auth0.
We then delete the old token in DynamoDB, and store the new one.

Potentially, with this flow (and the fact that DynamoDB is non-locking), could mean that multiple instances of your service save a token at the same time. But, this will be minor compared to how much you’re able to save by caching in the first place.

Let’s now create the methods we referenced in the getAuthToken function that help us interact with the tokens storage and validation

// ===
// cacheToken.ts
// ===
import jwt_decode from 'jwt-decode';

const deletePreviousTokens = async (token: string) => {
  const docClient = new AWS.DynamoDB.DocumentClient();
  const tokenRecords = await getAllTokens();

  // Clear down the table
  if (tokenRecords.Items) {
    tokenRecords.Items.forEach(async (row) => {
      const token = row.token;
      await docClient.delete({ TableName: `${process.env.TOKEN_CACHE_DB}`, Key: { "token": token } }).promise();
    });
  }
};

const hasTokenExpired = (token: string) => {
  const decoded = jwt_decode(token) as { exp: number; iat: number; };
  if (decoded) {
    return decoded.exp < (new Date().getTime() / 1000);
  }

  return false;
};

const getAllTokens = async () => {
  const docClient = new AWS.DynamoDB.DocumentClient();
  const response = await docClient.scan({
    TableName: `${process.env.TOKEN_CACHE_DB}`
  }).promise();

  return response;
};

const getExistingToken = async () => {
  const response = await getAllTokens();

  if (response.Items && response.Items.length > 0) {
    return response.Items[0]['token'];
  }

  return '';
};

Again, let’s break this down

In deletePreviousTokens we grab all existing tokens and delete them one by one. This is to avoid concurrency issues where another instance has written a new token which we do not want to delete.
In hasTokenExpired we do a basic JWT validation to make sure that it is not expired. This could be improved by not using the token if it’s only got 1ms left but has worked so far for me.
In getExistingToken we get all rows in the table and return the first token or an empty string if none is found.

Usage in the handler

Now all that’s left to do is to add it to your Lambda functions handler method.

export const handler = async (event: any, context: any) => {
    const token = await getAuthToken();

  // Do something with the token
  await sendResultsToService(token, event.Results);
}

Hopefully you found this interesting and saved some money on your Auth0 Bill!

How You Work

Fri, 07 Jan 2022 00:00:00 +0000

Learning how you work best is a superpower. Imagine, creating and seeking environments where you succeed best. Likely you remember times where you got into a flow state and produced magic, but you can’t pinpoint why.

I was in this position and so took some time to figure out how I work best.

Product user manuals have a section dedicated to “ideal working conditions” - to get the best out of the machine. This includes the maintenance, the temperature and location and how it should be operated. In the same way, you can develop a “personal user manual” documenting how you work best and where.

In my user manual, it’s broken down into five sections.

The kind of work I do best
The environment for doing that work
How I enjoy doing that work
How I receive feedback
How I get motivated

For example, my user manual is included below.

The kind of work I do best

Creating things that I know will benefit people.
Automation and saving time.
Turning rough specs into tangible products.
Thinking of edge cases and being able to dream of scenarios where bugs will occur at scale.
Building clear API’s that are secure and scalable.
Writing technical documentation.
Architecting and building systems that can handle millions of customers.

The environment for doing that work

I’m informed of the bigger picture and the impact of my work.
I like a small to-do list.
Requirements stay reasonably consistent. But I have the autonomy to figure out how to meet those requirements.
The benefit of my work is somewhat measurable.
There is a culture of gathering and reviewing data to make decisions.

How I enjoy doing that work

I prefer to work asynchronously and reserve synchronous work for solving specific problems.
In line with the above, I like to keep meetings to an absolute minimum. Including many of the sprint reviews, standups and retros that are commonplace in most software businesses.
I like to have the flexibility to work inconsistent hours. Often, I find I solve problems better away from the computer rather than bashing my head against the wall.
I believe in transparency and equality, I prefer to work with organisations that foster that same ethos.

How I receive feedback

I love to improve my work and constantly question personally on how to do this. But, I need to understand the reason why something is better or is important.

The Benefit

By preparing a user manual, you can quickly establish a rapport with your co-workers and effectively communicate your “ideal working conditions” to your manager. Having led teams, I ended up putting together versions of a “user manual” for everyone in my team in my head. A written document from that person would have proved vital.

The advantages are that both yourself and the people around you can understand each other and make everyone happy by aiming to keep work within those ideal parameters. Of course, this is not always possible. Tough, stupid things sometimes need doing. But a keen-eyed manager will always be aiming to balance these tasks with work you love.

I’ve found my manual allows me to reaffirm my ideal work. Whenever I am thinking “I hate this work”, I review this manual and figure out why I dislike it.

I’d love to see your manuals! Send them to me on Twitter - @joshghent or via email at me@joshghent.com

Continuous Delivery to ECS with Terraform

Wed, 11 Aug 2021 00:00:00 +0000

Continuous delivery is something that we’re all striving for. I was doing the same, but there was a snag:

My terraform code and API code were in separate projects
I wanted to make updates to the API code and have it build and update the ECS service
I didn’t want to manage the container definition separately as it had too many dependant resources (Datadog sidecar etc.)
You have multiple environments and don’t want to have to use a separate git branch for the API code.
You have a branch for each environment for your infrastructure that is independently deployed.

Maybe you have this problem too. Because Terraform uses a specific ECR image path to build the container definition, how do we ever update it automatically?

There are several ways to solve this problem, many of which are discussed in this thread. But, today, I’m going to show you how I resolved this issue.

Setup your Docker build/deploy pipeline

First things first, we need the API Docker image inside ECR. This will vary according to what CI system you use - in our case we use Azure DevOps.

When the image is being built, we want to tag it uniquely. Ideally, you want something numerable. In our case, we chose to use Azure’s built in “BuildId” parameter to tag the images.

Below you can see the build steps we take in the CI pipeline. After the image is built, it creates a text file with the BuildId in it and ships that as an “Artifact”. This will become important later. But the main thing is you need to trigger a further pipeline for your environments based on that parameter changing.

- task: Docker@2
  inputs:
    command: build
    DockerFile: "$(Build.SourcesDirectory)/Dockerfile"
    repository: ${{parameters.projectName}}
    tags: |
      $(Build.BuildId)

- task: ECRPushImage@1
  inputs:
    imageSource: "imagename"
    sourceImageName: ${{parameters.projectName}}
    sourceImageTag: "$(Build.BuildId)"
    repositoryName: ${{parameters.projectName}}
    pushTag: "$(Build.BuildId)"

- task: Bash@3
  displayName: "Upload Build Artifact of the Docker image Id"
  inputs:
    targetType: "inline"
    script: |
      # Add the build Id to a new file that will then be published as an artifact
      echo $(Build.BuildId) > .buildId
      cat .buildId

- task: CopyFiles@2
  displayName: "Copy BuildId file"
  inputs:
    Contents: ".buildId"
    TargetFolder: "$(Build.ArtifactStagingDirectory)"

- task: PublishBuildArtifacts@1
  displayName: "Publish Artifact"
  inputs:
    pathToPublish: $(Build.ArtifactStagingDirectory)

Make sure to run this pipeline now you’ve created it.

Setup an SSM (Systems Manager) parameter

SSM is an AWS service I had previously never really used. Its parameter store feature will allow us to store a variable that we can update later - in this case, the docker image tag.

Create a new parameter by going to AWS Systems Manager > Application Management > Parameter Store. Name the parameter something like /my-api/${env}/docker-image-tag (where env is the environment, you’ll need to duplicate this parameter for all the environments you have). It should be a “String” variable and be the unique tag that is generated by your CI build pipeline - in my case, the BuildId.

Create your deployment pipeline

Now, we need to define a way to update the image just in a certain environment (e.g., just development). How can we do that? Because of our setup, the duplication effort is fairly minimal. We already have our image build (which is consistent across all environments). We just need to update that SSM parameter to use the unique tag (BuildId) that the build pipeline generated.

In Azure, you can trigger a pipeline based on an artifact as we generated in step 1. I then configured 3 tasks based on this:

Get the BuildId from the file and add it to the runners environment
Update the SSM parameter for that environment to the new BuildID
Trigger the Infrastructure/Terraform pipeline for that environment - this is where the new SSM parameter value will get picked up and used in a container definition.

Update Terraform to use the SSM parameter

Now you’ve got the SSM parameter being updated each time there is a new build, we need to set up Terraform to use the SSM parameter as part of the image name.

// Import the SSM parameter
// This can be done on a module level because it depends on the environment
data "aws_ssm_parameter" "docker_image_id" {
  name = "/my-api/${var.environment}/docker-image-tag"
}

// Use it later on...
container_image = "<AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/my-api:${data.aws_ssm_parameter.docker_image_id.value}"

All done!

Now you’ve set up a complete continuous deployment pipeline with Terraform and ECS. To review, here’s how the system works

Creates and builds a new ECR image tagged in a unique way
The build pipeline notifies the release pipeline of this new image tag in some way (in Azure’s case, a build artifact)
The release pipeline updates the SSM parameter based on the image tag and triggers the Terraform deployment pipeline for that environment.
Terraform picks up the new SSM value and implements it

There are two obvious downsides to this approach:

Multiple updates to multiple API’s could cause locking issues where you have to manually run the Terraform pipeline once.
It’s a bit slower than other approaches, but the best one I could find. Plus if your Terraform repo deploys in less than 2 minutes like ours, then it’s not a big problem.

How to Run Sequelize Migrations in Azure Pipelines

Wed, 07 Apr 2021 00:00:00 +0000

Database migrations are the concept of managing your database schema via reversible, version controlled files. A program is then used to run these “migrations” and keep track of which ones have been run on your database. Migrations are immutable, meaning if you want to change a column name, type or anything else then you have to create a new “migration”. Handling your database programmatically gives you many benefits. Namely, providing a consistent schema across all your environments and portability if something happens to your DB. Further, with these files being committed to source control, migrations can be reviewed by others on your team. If you’re not already using a tool to do this, I’d encourage you to do so.

Throughout this article, I’ll refer to two terms

Migrations - meaning files that change the schema of the database but not the underlying data
Seeders - files that insert anonymous data into our staging environments for testing purposes

Normally, these migration and seed files would have to be run manually from a developers computer against the different databases. But, with my obsession with automation, this wouldn’t fly. I decided to create an Azure Pipeline runner to handle this for us. It will run automatically whenever new commits on our development or master branch are found. It also reduced stress for me as I know that I will make mistakes, whereas a computer, configured correctly, won’t! 😅

Although this article is designed around building azure pipelines for Sequelize migrations. This process can be adapted to other ORM’s such as Knex and TypeORM.

Create Your Artifacts

If you’re not familiar with Azure, it has a concept of “Artifacts”. These are a collection of files that can then be used by other pipelines. We need to create two artifacts, one for our migrations pipeline and the other for our seeding pipeline. In your source control create the following two files - you can copy-paste the code below!

# azure-migrate.yml
pool:
  name: azure-pipeline-runner
pr: none

steps:
  - task: CopyFiles@2
    displayName: "Copy migration scripts"
    inputs:
      contents: "$(Build.SourcesDirectory)/migrations/**"
      targetFolder: $(Build.ArtifactStagingDirectory)

  - task: PublishBuildArtifacts@1
    displayName: "Publish Artifact"
    inputs:
      pathToPublish: $(Build.ArtifactStagingDirectory)
      artifactName: migrate

# azure-seed.yml
pool:
  name: azure-pipeline-runner # the name of your azure pipeline runner
pr: none # Don't run this pipeline for pull requests

steps:
- task: CopyFiles@2
  displayName: 'Publish SequelizeRC'
  inputs:
    Contents: .sequelizerc
    FlattenFolders: true
    TargetFolder: '$(Build.ArtifactStagingDirectory)'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Seed'
  inputs:
    PathtoPublish: seeders
    TargetPath: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: seeders

- task: PublishBuildArtifacts@1
  displayName: 'Publish Sequelize Config Folder'
  inputs:
    PathtoPublish: config
    TargetPath: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: config

In the code above, it first copies the .sequelizerc file that is used to denote various configuration parameters to Sequelize. Next, it creates “build artifacts” for the seeding and migration folders where all the files related to setting up the database are stored. Finally, it publishes them to the azure artifacts library.

Configure Your Runner (Optional)

This step is optional because it depends on your existing setup. All you need to make sure is that your Azure runner (whether self-hosted or not) can access the Database.

We use MySQL Aurora to host our database which sits in a VPC. Our azure-pipeline-runner (defined in the “pool” parameter) is hosted inside the same VPC but a different security group. So, we needed to allow access from the runners’ security group to the RDS’ security group. This is called “ingress” in AWS. The port you need to allow access to may vary - in our case, it’s 3306 which is the default for MySQL.

Getting this setup is a simple process. Check this guide for more info.

Create Your Pipeline

Now we’ve got our runner configured and our build artifacts published, we can move onto creating the actual pipeline. Go to Pipelines and Releases and click ”+ New” and select “Create Release Pipeline” from the dropdown.

You’ll be prompted to select a template but we can click “Empty Job”

Next, click the Artifacts box on the left and then find your artifact by searching for it.

You’ll be able to find the name of the artifact under “Pipelines > Pipelines”. You should see your migration or seeding pipeline that you created earlier. Clicking into one of the runs of this job will reveal the artifact name that the job created.

Now, configure a stage. Click on the “Tasks” tab at the top of the page. This will take you to the list of “tasks” that will run for each stage.

Click to add a new task and search for “npm”. We want to first install Sequelize globally on the command line so that it can be used to run the migrations or seeding process. Because we are using MySQL, we also need to install the mysql2 package.

The job should end up looking something like this:

Now we need to add the stage that runs the migrations or seeding process. Click the plus button again and select the “Command Line” job. This will allow us to run the Sequelize commands.

The command we want to run for migrations is:

sequelize-cli db:migrate --url ${DB_URL}

For the seeding process it is:

sequelize-cli db:seed:all --url ${DB_URL}

The documentation for these commands can be found here.

We aren’t doing anything fancy asides from passing our Database URL. Since this won’t be stored in our Git repo, we need to provide it here as an environment variable.

If something goes wrong. You may need to check that the runner is running files in the correct directory. Ensure that the directory is the root. It should contain a folder called “seeders” or “migrations”. These folders should contain the migration and seed files.

Below is how our job ended up

Now you’ve configured one stage, you can clone it for the others! Go back to the Pipelines view and click the clone button beneath the stage card.

Wrapping Up

I hope this helped you configure your database migrations! Here is a quick summary of what we have learnt.

Database migrations and seeding processes are important to codify for consistency, portability and review purposes.
How to configure Azure runners to allow ingress to RDS
How to create build artifacts in Azure Pipelines
How to write basic azure pipeline jobs
How to migrate and seed your database using Azure pipelines.

I’m glad to have this work sorted as it was a bit of a hassle to configure. But, we got there in the end and this is now a durable process that will scale with the team.

Super Fast React/Node App Testing with GitHub Actions

Thu, 25 Mar 2021 00:00:00 +0000

A seldom thought of component of performance is that of continuous integration performance. Here at York Press, we are big users of both Azure Pipelines and GitHub Actions. Due to us hosting our Azure pipeline runners, “job minute” restrictions were never a concern from a billing perspective. Although, the long running jobs did frustrate the team. Having moved some key processes over to GitHub Actions, I decided it was time that we looked at improving the performance of one of our core repositories. Not only would this mean developers had quicker feedback, but it would also mean we burned through our actions minutes a lot slower. Here’s how I did it.

Initial Investigation

GitHub actions (and I promise this isn’t an ad) has a handy feature whereby you can see how many seconds each stage of the job took.

The first thing I spotted was how long installing npm modules took - nearly 3 minutes! Because of this, I chose to combine the Test and Lint pipeline so that we would not need to duplicate the module installation.

Secondly, I swapped out my normal npm ci command for another action bahmutov/npm-install@v1. This action handles all the cache invalidation and storage of node modules across builds so you can save time with installing them. After those changes, here is what the timings looked like…

Half the time gone! That’s a good start but still not far enough. I found the modules were taking ages to install due to a Webpack plugin responsible for optimizing images, something we didn’t need in the CI process. I moved this out into an optionalDependencies and then set the command to npm ci --no-optional

ESLint

The other big fish to fry was ESLint, it took nearly 2:30 minutes to run. I tried to debug this locally using an environment variable TIMING=1. This gives you a table view of how long each ESLint rule took to check.

Interestingly, it was the import/ rules that were taking the longest. After some google-fu, I discovered that it was due to having to build a dependency graph across the codebase. Our codebase is fairly large so it was understandable why it would take this long. I didn’t want to remove the rule entirely as it was useful, but surely there was a way around it…

Actions to the rescue! Fortunately, a kind internet person has created a github action that will run ESLint only on the files that have changed. I swapped this out like so

- uses: tinovyatkin/action-eslint@v1
  with:
    repo-token: ${{secrets.GH_TOKEN}}
    check-name: eslint

This completely eliminated that time taken if no files had been changed matching the scanning glob.

From there, I spent more time than I care to admit trying to trim the time down. The main blockers were the dependency install (1:20s average) and the Jest test suite (50s average). Although there are ways to run the Jest suite in parallel it sort of seems redundant at this stage. The install is the big job, but the unfortunate battle is that we have Webpack image-loader as a devDependency. This then installs a whole host of binary packages that then get built from source - every. single. time. Anywho, I’m pleased with reducing it by 76%.

Here are my main takeaways:

Spend time speeding up your CI - fast developer feedback is important and saves you money (if you’re restricted on pipeline minutes)
Use the pre-built actions - there is a huge marketplace of actions that solve a bunch of problems and have smart defaults. GitHub Actions is great (and I promise this isn’t an ad), in part, because it’s like code lego.

I hope this helps you with your journey in speeding up your CI.

SpellcheckCI

Thu, 04 Mar 2021 00:00:00 +0000

Making sure you have correct spelling on your blog posts is vital to keep readers attention. Unfortunately, it’s a laborious process and sometimes things fall through the cracks. Being the nerd I am, I decided I needed a shell script to solve this problem.

Thankfully, someone has created an open source markdown based spellcheck module that is Node based - mdspell.

Since I’m using Gatsby, my posts can be found under content/blog/*/index.md - where * is the name of the blog post. The command to run the spell check was then

$ npm i -g node-markdown-spellcheck && mdspell -a -n "content/blog/**/*.md"

This would go through each of my posts and then validate the spelling is correct. When it comes across an incorrect spelling, it notifies me and asks me if I want to correct it, or add it to a local dictionary.

But, because I often blog from my iPad, where I don’t have a terminal, I wanted this feedback to be visible on the CI for the new blog posts. My workflow for creating new posts is create a new git branch, create the file and write the post, push to github and create a new pull request. You can find this exact blog post’s pull request here.

Time to Automate

I’m a big user of GitHub Actions so I went with that to setup this process.

Initially, I went down the road of installing all the node dependencies, then installing mdspell and then running the spellcheck. However, I found that it took over a minute to download all the node modules! It turns out, I could have used npx to use mdspell without having to install the project.

Here is the complete GitHubCI - which, across over 50 blog posts, takes around 10 seconds to run!

# ./.github/workflows/spellcheck.yml
name: Spellcheck

on: [pull_request]

jobs:
  spellcheck:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [14.x]

    steps:
      - uses: actions/checkout@v2
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm i markdown-spellcheck -g
      - run: mdspell -a -n -r "content/blog/**/*.md"
        name: Spellcheck

I hope this proves useful to you for your own blog. If you don’t have one already, I’d highly recommend creating one!

Setting up LightHouse CI for React in GitHub Actions

Tue, 16 Feb 2021 00:00:00 +0000

At York Press, we noticed that our pages were gaining weight. In some cases, pages were loading over 1MB of resources before showing for the customer. This was unacceptable considering the modal broadband speed is around 1MB/s. So, we decided we needed stricter checks. This would ensure that pages are lighter than an ants leg made of clouds. And, faster load times would mean customers could get to studying faster - which I trust they yearn for.

Lighthouse to the Rescue!

Lighthouse is a tool developed by Google. It analyses a page and gives it a score, out of 100, on SEO, Performance, Accessibility, PWA and Best Practises. Although these are arbitrary numbers, they give a rough guide to how your website is doing. These scores are also used to rank your page in Google search rankings. So they are vital to maintain for business reasons, not technical prowess.

The challenge is how to get this tool setup as there are lots of outdated articles and guides. Furthermore, none of these seem to cover a regular use case - setting up Lighthouse for your React app.

Here’s a definitive guide on how to setup LighthouseCI for your React app - and have it tracked in Github Actions.

Setup Lighthouse CI

First, you will want to install LighthouseCI and http-server locally for testing purposes.

$ npm i -g @lhci/cli http-server

The former is the LighthouseCI tool. The latter is a small module to run the React app after it has been built.

Next you can create a file called lighthouserc.json. This should have the following contents

{
  "ci": {
    "collect": {
      "url": [
        "http://127.0.0.1:4000"
      ],
      "startServerCommand": "http-server ./build/client -p 4000 -g",
      "startServerReadyPattern": "Available on",
      "numberOfRuns": 1
    },
    "upload": {
      "target": "temporary-public-storage"
    },
    "assert": {
      "preset": "lighthouse:recommended",
    }
  }
}

The section under “collect” is where the server that runs the React app is defined. The interesting properties are the startServerCommand and startServerReadyPattern. The first tells Lighthouse how to start your application. And the second, tells Lighthouse what text to look for to see that the server is running and the test can begin. In this case, it starts the server via http-server and then it listens for the text Available On. Run the command shown above for yourself and see what text it displays in your terminal. You may need to change /build/client to the directory where your application gets built

Now you can give your LighthouseCI a whirl! Build your application (if you used create-react-app then run npm run build), then run:

$ npm run build
$ lhci autorun

You should then see an output like this:

✅ .lighthouseci/ directory writable
✅ Configuration file found
✅ Chrome installation found
Healthcheck passed!

Started a web server with "http-server ./build/client -p 4000 -g"...
Running Lighthouse 1 time(s) on http://127.0.0.1:4000
Run #1...done.
Done running Lighthouse!

Checking assertions against 1 URL(s), 1 total run(s)

33 result(s) for http://127.0.0.1:4000/ :

Setting up GitHub Actions CI

Now, let’s automate that. The best way to enforce these sorts of checks is to make them part of your pull request workflow. This means preventing merge on requests that fail to meet these standards.

All we need to do with GitHub Actions is imitate the commands we did in the setup process. Paste the following into a new file called /.github/workflows/lighthouse.yml

# ./.github/workflows/lighthouse.yml
name: LighthouseCI

 on:
   pull_request:

 jobs:
   lighthouse:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2

       - name: Setup node
         uses: actions/setup-node@v1
         with:
           node-version: "14.x"

       - name: Install
         run: npm ci && npm i -g http-server @lhci/cli

       - name: Build
         run: npm run build

       - name: LighthouseCI
         run: lhci autorun

Next, push up your changes and create a new pull request. You should see your Action running at the bottom of the pull request.

And that’s that! I hope that has saved you a lot of time if you were struggling to get your React app to play nice with GitHub Actions.