DEV Community: Tom Wright

Interviews are not a zero-sum game (FizzBuzz postscript)

Tom Wright — Fri, 15 Jul 2022 15:13:19 +0000

My last post sparked some controversy in the comments on Reddit. Whilst the majority seemed on board, there was a sizeable minority who took umbrage. Two objections, in particular, caught my eye, because I think they both relate to the same underlying issue – adversarial interviews. In this (short) post, I’d like to explore this and explain my approach. Finally – unrelated to the bulk of this post, but too fun to ignore – let’s look at some of the wacky FizzBuzz implementations shared in the Reddit comments.

Not how an interview should go. [Image courtesy of David Trawin on Flickr]

So, I published an article recently about how useful FizzBuzz remains as an interviewing tool. I submitted it as a link to Reddit, as I often do when I write a new blog post. Overall I was very gratified with the response – at the time of writing it had attracted over 800 upvotes and a large number of interesting comments. Inevitably, however, there were some less positive responses too.

I won’t dwell on the responses from people who had clearly not read (or, more charitably, had misunderstood) the actual post. For instance, I don’t feel the need to respond to suggestions that TTD and interfaces are overkill for FizzBuzz…

That said, there were quite a few comments that fall into a couple of related themes: the modulus operator and live coding in interviews. I thought it would be good to rebut these first, before going into what I consider to be the underlying theme.

The modulo “gotcha”

There was a recurring theme of comments that objected to the use of FizzBuzz as a simple “do they know modulo” test. It’s true that not everyone will be familiar with modulo. One commenter (/u/mindbleach) put it nicely: “[modulo] is an accidental shibboleth” – i.e. an idiom whose use belies membership to a particular group. And whilst it is possible to complete FizzBuzz without using modulo, it’s probably the most natural approach

So, given these factors, how come I didn’t mention the modulo operator in my original post? Because, In short, on the rare occasions that a candidate doesn’t know about it, we’ll introduce them to it. We’ll have a chat with them about what they are trying to do, and then straight up tell them what the % operator does. Problem solved! And, as a bonus, the candidate has learnt something.

Live coding in interviews

The second theme in the comments that caught my eye was concerned with the inherent fairness of asking a candidate to do live coding in an interview. I completely sympathise with people who get nervous when having to program in front of a stranger, but unfortunately, there really isn’t a good alternative:

Take-home tasks disadvantage people with busy lives. Plus it can make a recruitment process less attractive to well-qualified candidates, who will likely be interviewing with several companies at once.
Presenting a portfolio isn’t an option for many people who don’t code outside of work. I really love to see a GitHub link on a CV and will always discuss it at interview, but it can’t be relied upon and I don’t like to penalise candidates on this basis.
“Whiteboard interviews” suffer the same problem of a coding task, whilst telling me less about a candidate’s abilities.

Additionally, if we’re doing the evaluation non-live/asynchronously, there’s no (time efficient) way to explore new ideas together or to adjust the complexity on the fly. We’re stuck with whatever the candidate prepared beforehand. Yes, we can discuss it, but that’s also true of a live coding task.

Finally, I reject the notion that you never have to code in front of other people as part of a software development job. In my teams, I encourage pair programming as a normal part of writing code. It’s not something I’d suggest people do for 40 hours a week, but it can be really helpful when working on something more complex, or when close collaboration is going to be required.

So if we are sticking with a live coding exercise, I suppose we better make it as painless as possible. To minimise stress, here are the steps I take:

Be as open as possible about the format of the interview, right from the start of any conversation.
Send the skeleton solution over in advance and encourage the candidate to try running the app and the tests ahead of the interview.
Make it clear that they can use whatever IDE/tools they feel comfortable with.
Set the whole thing up as a pair programming exercise – it’s not a quiz and you really can ask the interviewer for support.
Explain that it’s not a test of what syntax/design patterns/keywords they remember, but rather a chance to see how they think about problems, communicate with colleagues, and, yes, how they write code.

There will still be candidates who will be nervous. Heck, candidates will be nervous about interviewing, even if there’s no live coding element. It’s a stressful experience. With a bit of empathy and compassion, however, I maintain it’s entirely possible to coax a great performance from a nervous-but-talented developer.

Non-adversarial interviewing

The thing that struck me about both the emergent themes was that they seemed to stem from really negative experiences. Like, “who hurt you, dude?” It made me really sad to think of all the awful interviewing experiences the commenters must have had to make the assumptions they did. It seems so obvious to me that you need to set the candidate up for success, that it didn’t occur to me that I’d need to explain some of this stuff.

Interviewing is not a zero-sum game. You don’t “win” it by making the other person feel stupid. The interviewer should not be looking for reasons to disqualify a candidate. There’s no good reason for the interview to have “gotchas”.

In fact, it’s the opposite: the incentives for the candidate and interviewer are aligned! The interviewer has a vacancy to fill. The candidate wants a job. Both parties have gambled some of their precious time on the hope that it will be a mutual fit.

Maybe this is a step too far? [Image from PxHere]

In many ways, an interview is like a date. Unless you’re an “incel”, a date is usually a chance for both parties to put their best self forward, have a nice time, and hopefully create a positive lasting impression on the other. And it really does go both ways – after making time to interview someone, the worst outcome for an interviewer is for a good candidate to withdraw. Using the modulo operator as an opportunity to make someone feel small is clearly not compatible with this, nor is it to anyone’s advantage.

NB: In case it needs saying, please don’t confuse an interview with a date. Don’t be creepy or make things weird!

I’m reasonably confident that my interviews are pretty good. I attribute this to the age of maxim of “not being a dick”. Why do I feel that people like my interviews? Because I seek feedback and it’s usually pretty good. Crucially, it’s not just the successful candidates that seem to enjoy my interviews – I also get positive feedback from unsuccessful candidates and recruiters. Blowing my own trumpet like this has a purpose beyond self-gratification, by the way: I sincerely hope that someone reading this (hiring manager, or prospective candidate) will change the way they approach interviews in the future.

And now for something lighter…

OK, with that out of the way (and off my chest) I thought we could wrap up with a bit of fun. One of the cool things that happened in the Reddit comments was that all the crazy versions of FizzBuzz came out of the woodwork. Some are parodies, others are just insane demonstrations, but they’re all worth a look.

Extremely high throughput FizzBuzz
FizzBuzz in TensorFlow
EnterpriseFizzBuzz
FizzBuzz from scratch (i.e. the base of Yggdrasil itself)

So there we have it. By the way, if you enjoyed these, keep an eye on the blog for an upcoming FizzBuzz competition. For now, since I didn’t intend to blog twice in a week and I promised to keep this post short, let’s leave it there. Ciao!

FizzBuzz is FizzBuzz years old (and still a powerful tool)

Tom Wright — Thu, 14 Jul 2022 08:02:49 +0000

This year marks 15 years since FizzBuzz was popularised as an interview tool for developers. I’m a big fan and have watched over 100 candidates try their hand at my version of the task. In today’s blog post I’d like to take a moment to celebrate what makes FizzBuzz so helpful, discuss some common patterns I’ve observed in the many attempts I’ve witnessed, and finally explore some tweaks that can be deployed to keep the challenge fresh.

The big one-five! Sweet fifteen! [Image courtesy of gratuit on freeimages.co.uk]

Happy Birthday, FizzBuzz! Well, happy belated birthday, I suppose, as it was really January 2022 that marked 15 years of FizzBuzz being used as a programming challenge.

We can trace the “birth” to a 2007 blog post by Imran Ghory. This was the same year as the iPhone was announced, when Nicolas Sarkozy was elected as president of France, and when sub-prime mortgages triggered the “Great Recession”. 2007 was the year that the last of the main Harry Potter books was published when LCD Soundsystem released Sound of Silver and the year that brought us The Simpsons Movie. In other words, 15 years is a long time!

The motivation for Imran’s post touched a nerve in the programming community at the time. The post went as close to “viral” as existed in 2007. Jeff Atwood (who went on to co-found Stack Overflow) wrote about Imran’s article later in the same year, pulling together lots of commentary in agreement with the central thesis: too many candidates for development jobs struggle to write even simple code. Being able to implement FizzBuzz would at least demonstrate that a candidate could do the basics.

As originally stated, a FizzBuzz program should output a sequence of integers from 1 to 100, except that multiples of 3 should be replaced with “Fizz”, multiples of 5 with “Buzz”, and multiples of both 3 and 5 with “FizzBuzz”. This would result in something like 1, 2, Fizz, 4, Buzz, Fizz, 7, 8, Fizz, Buzz, 11, Fizz, 13, 14, FizzBuzz… The interesting thing about FizzBuzz is that it is easy to express but requires some nuance to implement correctly.

Over the course of the last 15 years, FizzBuzz has become quite widely known. For instance, many of the people I’ve interviewed have come across it before (not that this matters – see below). In fact, it has got to the point where it’s almost a cliché. Tom Scott did a great video on it back in 2017 (i.e., when FizzBuzz was just Buzz years old), during which he goes through a solution in Javascript and touches on some of the pitfalls.

One area where Mr Scott and I may disagree, however, is whether FizzBuzz can still be useful and interesting. He attributes the continued use of FizzBuzz to the fact that “some interviewers can’t think of anything better”. My view, on the other hand, is that FizzBuzz is still well suited to evaluating core competencies in developers and offers an adaptable, domain-neutral foundation for exploring more complex ideas.

My experience of setting FizzBuzz in interviews

FizzBuzz has been a key tool in my interviewing arsenal since 2015. Or, put another way, I’ve been using FizzBuzz for the most recent half of its life. I’ve used it at every company where I’ve recruited developers. Over 100 candidates from the UK, Australia, and India have tried their hand at my special brand of FizzBuzz using C# and .NET. Whilst there were obviously patterns and trends, the thing that fascinates me is that every attempt was unique, with its own quirks.

My version of the FizzBuzz task is set up as a pair programming (“tour guide” flavour) exercise following strict (perhaps artificially strict) TDD. The interviewer delivers small, atomic requirements and the candidate is expected to start addressing each requirement with a unit test, before moving on to the implementation. In proper pair programming style, the candidate is encouraged to discuss their approach with the interviewer. Likewise, they are free to use any online reference materials if they forget a method name or some syntax.

Setup and general observations

The candidate is provided with a .NET Core solution containing three projects: a class library, an XUnit project, and a console application. There are stub implementations so to act as a guiding template for the candidate and demonstrate how the three projects interact. There is an example unit test that calls an example method, which is also called by the console application. I even provide interfaces that contain the signatures of the methods I’m expecting the candidate to implement.

It really is a “batteries included” experience.

The aim of having all these guide rails is equally to…

Minimize the chance of “analysis paralysis”
Avoid the candidate having to waste a load of time writing boilerplate
Prevent this task becoming a syntax spelling bee

Because ultimately, I’m interested in seeing them write some code, rather than watching them flounder and panic.

Nevertheless, a candidate’s reaction to some of this stuff can tell you a lot about their professional experiences. For instance, it will quickly become obvious whether they’ve ever written a unit test before. Crucially, however, there’s enough of a template in there for even a unit test virgin to have a go at it, which means the rest of the task can still be useful in probing other areas of competency.

Other general-purpose indicators of competence I look for include:

Do they immediately head to the console app to start making changes? This is a behaviour that I’ve come to associate with more junior developers and particularly those whose past experience tends to be more in the realm of “scripting” than application development.
Are they just (for want of a better word) messy? Do they have any regard for things like indentation or capitalisation? Do they try to use meaningful names for methods and variables? Since code is usually read more than it’s written, these questions aren’t just about me being fussy. Experienced developers understand the importance of producing code that is easy to grok quickly.
Do they leave loads of vestigial code hanging around? This ties into the point above, but is worth calling out as a special case. I often see developers leaving in chunks of commented-out code. I find this can indicate a lack of confidence and experience, or sometimes just a wariness regarding source control.

To be clear, I’d never rule a candidate out for displaying any of these behaviours. After all, I’ve seen many talented developers who don’t conform to all my preferences. That said, these sorts of observations can be helpful in contributing to the overall impressions of a candidate.

Delivering the requirements

The functional requirements provided to the candidate start off very minimal but get larger as the exercise progresses. This provides a bit of a warm-up, which can be helpful for less experienced candidates. To cater for the other end of the spectrum, there are two stretch goals, with the interviewer being able to switch to one or the other depending on how the candidate is getting on. By these means, I find I can use one task for all candidates, dynamically adapting it during the interview as the candidate’s capabilities are made apparent.

_ NB: If you’d like to see the task I give to candidates, I’ve uploaded it to Github: github.com/tdwright/FizzBuzzInterviewTask In fact, if you’d like to have a go at completing the task, stay tuned for details of an upcoming competition!_

In the last two companies I’ve worked at, I’ve ended each FizzBuzz challenge by asking the developer to commit their changes to a new branch in git. Not only does this allow me to assess their git comfort levels, but it also means I have developed a reasonably large corpus of candidate attempts. This has allowed me to compare them, and find patterns and themes. In the next section of this post then, let’s take a look at the components of the task and what each part can tell the interviewer.

Core task

In my version of FizzBuzz, I always start with the string replacement aspect of the problem. The first four requirements handle the four possible outcomes for a single number in the classic FizzBuzz problem. We start by implementing the IFizzBuzz interface, by adding a simple method to the provided class. For now, the method just needs to return a string containing the digits of the integer it was passed as an argument. For example, when passed 1, return “1”. For most developers, this is a gentle introduction to the task which allows them to get acquainted with the solution structure, XUnit, and the TDD approach. For some developers, however, we find that we have to explain how to use Int32.ToString(), which is in itself highly instructive. For some other developers, this step reveals their inexperience in working with interfaces.

The next two requirements are about extending the same method to handle the cases of Fizz for 3 and Buzz for 5. This is usually pretty quick and is when confidence builds. Depending on whether a candidate has prior knowledge of FizzBuzz (and depending on how strictly they are following TDD), we tend to see one of two main approaches here – those approaching it naively will chuck in some return statements inside simple if statements, whereas those who know (or can guess) what’s coming will typically start appending things to an initially empty string.

The fourth requirement is where candidates typically come unstuck. This is not a feature unique to my implementation of the challenge either – Tom Scott mentions this in his video and it is mentioned on the FizzBuzz page in the venerable C2 Wiki. The fourth requirement is all about handling the case for multiples of both 3 and 5 (e.g., 15) where the output should be FizzBuzz. The simplest implementation will involve an extra if statement with an extra return, but may still fail if the order of the various if statements means that the condition is caught by one of the individual checks – for instance returning Fizz due to being a multiple of 3 before evaluating for the FizzBuzz condition. Even if a candidate implements this correctly, it’s a good place to have a discussion about refactoring – Can we reduce the nesting? Can we reduce the repetition of the logical checks? If a candidate had gone straight to a string appending implementation, we can discuss string immutability, the difference between reference and value types, StringBuilder, etc.

As an aside, it’s this fourth requirement where the penny often drops about the usefulness of unit tests. In trying to solve the case of 15, it is common for other things to get broken. Being able to re-run a suite of unit tests can highlight this for the candidate and provide valuable feedback.

Variant 0 – Loop to 100

I’ve called this the zeroth variant since most FizzBuzz challenges include this as a core requirement. Although I specify this behaviour in requirements 5 & 6 of the main task, I will often skip them for very junior and very senior developers. My motivation for this is that I will have been able to gauge a developer’s level by the time we have gone through steps 1-4. Time pressures mean that I’d often like to switch to something more challenging for the more competent candidates. Conversely, for candidates who struggle with the main task, we may already have run out of time by this point.

When I do run this section of the task, there are a few things I’ll keep an eye on. Firstly, I’m obviously watching carefully for “off-by-one” errors. These are usually caused by incorrectly bounded loops and aren’t a big deal, but it’s always interesting to see if the developer can spot the cause. The other thing I look out for (and will ask about) is whether they’ll use something more exotic than a simple loop – perhaps Enumerable.Range feeding a LINQ statement, or maybe using yield return. A loop will clearly do the job, but it can be a nice opportunity for the candidate to show off.

For reference, these are the numbers between 1 and 100. [Image courtesy of Leo Reynolds on Flickr]

Variant 1 – FlexiFizzBuzz

For candidates who breeze through the first set of requirements, I love to challenge them with this variant. The requirements here aren’t enumerated in the same granular detail, but instead, we point to a new interface to be implemented, describe the desired behaviour in general terms, and provide some examples. In short, the change we want the candidate to make is to generalise FizzBuzz, so that any arbitrary collection of int/string combinations can be supplied as replacement pairs. Perhaps instead of replacing multiples of 3 with Fizz and 5 with Buzz, we want to replace multiples of 2 with the word “Even”, for example.

The provided interface specifies the data structure – a Dictionary of ints and strings as a property. What isn’t specified is whether this should be populated via the constructor, by setting the property, or by calling some method(s) to manipulate the dictionary. This is great for exploring how the candidate thinks about API design and the way in which a caller might expect the lifecycle of the FizzBuzz class to behave.

The really interesting part of this variant, however, is the way candidates adapt their previous code to handle this more general case. For a start, it can be instructive to observe whether they make any attempt to maintain backwards compatibility with the unit tests (and possibly console app) that they’ve already written. To be specific, about half of candidates don’t initially set out to retain the 3: Fizz; 5: Buzz replacement pairs as a default, which obviously leads to all their prior unit tests failing.

Typically, developers will land on some solution involving iteration over the dictionary to check whether the key is a factor of the input number and append the value to some aggregator string.

One interesting exception to this that I’ve seen recently is to ignore the dictionary altogether and instead allow for Funcs to be added to the class, with each Func fully encapsulating the logic about what string (if any) to return. This means that the classic FizzBuzz can either be implemented as one or two Funcs, depending on the calling developer’s preferences. It also means that the replacement logic can deviate from simply looking at multiples. You could, for instance, supply a Func that returned a certain word based on whether the number is a prime, or the number of digits, or whether the digits formed a palindrome – the sky is the limit. This type of approach could be somewhat controversial (is it even still FizzBuzz?), but that just provides an intriguing line of discussion with the candidate.

Variant 2 – CloudFizzBuzz

This one I can’t claim credit for, as I experienced it on the receiving end… Back in 2018, as I was moving to Australia and applying for jobs, I was delighted when HeadUp Labs tasked me with a variant of FizzBuzz as a technical challenge. Kudos to the excellent James Devlin for this!

Their twist was to implement FizzBuzz as a pair of Azure Functions (one for generating the sequence, the other for handling the replacements) linked with a queue. I remember having a lot of fun with the fusion of FizzBuzz (which I was already using in my own interviews) and cloud technologies.

Let’s see you implement FizzBuzz on these! [Image courtesy of Larisa Koshkina on PublicDomainPictures.net]

Fun things to explore with this challenge could include:

What cloud services to use?
How to connect the various services?
How could we make it scalable?
Since we’re hosting it, should we cache or generate sequences ahead of time?
How could we secure the service and/or add billing? (Are they aware of things like Azure API Management?)

In other words, the actual implementation (provided it works) will likely be of less interest than the sorts of discussions it can trigger. What I’d be looking for here is not just the extent of the candidate’s knowledge, but also how collaborative and open they are as we bounce ideas around.

This variant is a lot more involved than the one mentioned above, to the point where I wouldn’t ask a candidate to do this as part of a live coding exercise. Instead, this variant could work as a take-home follow-on task, particularly for more senior or cloud-oriented roles. I personally try not to issue homework as part of my interview process, but I know a lot of places do.

The beauty of FizzBuzz in this context is again related to the fact it’s a task that’s easy to understand without being tied to any domain or depending on any specific cultural knowledge.

If any candidates see this…

If you ever have an interview scheduled with me and have read this post, please don’t be shy about letting me know. The fact that you’ve read this demonstrates that you’re plugged into the developer community and is a strong positive signal even before we start coding.

One of the great things about delivering this task as part of a live programming exercise is that it isn’t spoiled by prior knowledge. It doesn’t rely on the candidate being ignorant of the potential pitfalls – the point isn’t to shout “gotcha!” Even if someone aced the programming because they’d memorised this post, I could still evaluate their competencies with follow-on questions. For example, asking why something was done a certain way, or asking if the candidate has ever used a technique in another context.

Here’s to another FizzBuzz years of FizzBuzz!

As you may have guessed by now, I’m a big fan of FizzBuzz. I think it has the perfect level of complexity for an interview task – rich enough to allow insights into the candidate’s skills, whilst simple enough to get from zero to something meaningful in a reasonable time. Like all good interview tasks, FizzBuzz can be fully explained very easily, even to someone who has never heard of it before. It does not depend on any cultural references or domain knowledge.

I hope I’ve shown that FizzBuzz can easily be extended to cater for a wide range of abilities. In fact, the sort of dynamic adaptation I’ve described above means that you don’t need to select a task based on preconceptions about a candidate’s abilities but can instead tailor the task on the fly depending on how they perform.

I find this sets FizzBuzz apart from other common tasks set in technical interviews. Reversing a string or generating a sequence of Fibonacci numbers are both very straightforward to explain, for instance, but have limited scope for extension. Where, after all, does one go on from a successfully reversed string? At the other end of the spectrum, some companies ask candidates to do some work on a real codebase. This is not something I’m a fan of (although I will concede that it could be informative with regards to code comprehension). Aside from tricky issues about unpaid labour, this sort of task has a high barrier to entry, often without any guarantee that it will reveal anything interesting. The flexibility of FizzBuzz (as described) allows it to cover all bases, which is super valuable.

There’s also something to be said for familiarity. I’ve seen so many attempts now that I know what to expect and what to look out for. This can make the interview process a lot more productive than it otherwise might be. On the other hand, we have to acknowledge that a candidate’s familiarity may contribute to their perceived membership of the “in-group”. To improve diversity in the profession, we must reduce gatekeeping, which in this context means focusing on each candidate’s abilities, regardless of whether they’ve encountered the task before or not.

In summary, I can absolutely see myself continuing to use FizzBuzz in interviews for the foreseeable future. Not, as might be suggested, because I’m too lazy to come up with anything better, but rather because after considering the alternatives, I’ve concluded that FizzBuzz is still at the front of the pack. It’s not the only task I’ll set, but it’s a useful default that works in most cases.

So then, here’s to another FizzBuzz years of FizzBuzz!

[Image courtesy of Warner Bros.]

If you’ve made it this far, you may be interested to know I’m planning a FizzBuzz competition. Subscribe to my blog to be sure not to miss it…

Only the great die at 11 (RIP Stack Overflow Jobs)

Tom Wright — Sat, 29 Jan 2022 23:19:19 +0000

Stack Overflow have announced they are going to ~~kill off~~ sunset their popular Jobs product. In this post, I talk about what Jobs was, why it was great, and why so many (myself included) will be sad to see it go.

Stack Overflow Jobs – an obituary

On the 13^th of January 2022, a post on Meta announced thatStack Overflow Jobs will soon be “sunsetted”. This is a euphemism I’d normally let slide. With Stack Overflow jobs, however, its use is a huge disservice to all the many users who have benefited from this service over the years. I mean “users” in this context to refer to those from both sides of the aisle – candidates and employers. Let me go back and recap what Stack Overflow Jobs was and it’ll be clear why…

(NB: I’ll be using the name “Jobs” throughout, but please treat this as synonymous with “Careers” and aspects of “Talent”.)

What was Stack Overflow Jobs?

The Jobs section of Stack Overflow launched in 2011 and was, at its heart, a jobs board – albeit one that was barely recognisable compared to its peers. Employers could post high-quality job ads linked to helpful company profiles. Candidates could maintain a “developer story” linked to their Stack Overflow profile, indicate their status (active or passive), and could of course browse through the job ads. As well as candidates responding to ads, hiring managers could use a powerful search to identify and message candidates that would likely be a good fit.

The genius of Stack Overflow Jobs was that it was a pure value add, which did not detract at all from the core user experience of the main Stack Overflow Q&A site. It was never forced on anyone and the service on the main site was not degraded for those who opted not to engage with it. On the other hand, Jobs benefited massively from the close integration with the main site. Linked profiles, for example, added a degree of transparency for both parties – candidates could easily demonstrate their communication and technical skills via their interactions on the main site, whilst employers could showcase their team by linking their company profiles to those of their current employees.

I used Stack Overflow Jobs as both types of user. In late 2016 I was searching for my next job and Stack Overflow was the clear place to look. I applied for a job I found on the site and by December I had started a new role. Whilst at that firm I had cause to hire more developers and got my first taste of using the site as a hiring manager. I used the platform as a hiring manager again in my next role in Australia. Each time, my experience was overwhelmingly positive. The only complaint was the costs, which were justified by the results.

What was it like to use?

As a candidate, the first thing one noticed was the refreshing lack of clutter. Anyone who has ever searched for a job will be familiar with the normal cesspool of unstructured and barely relevant content on sites like Indeed or TotalJobs. On Stack Overflow Jobs, we benefited from two big wins:

Any employer posting a job ad on the site was at least aware of Stack Overflow and therefore probably not total garbage.
It’s easier to add some structure to a fairly focused section of the employment market. General sites have to accommodate beauticians alongside engineers. On Stack Overflow, it is assumed you will have some programming languages you prefer, which really helps refine a search.

These factors were also very helpful to the hiring manager, who could easily find relevant candidates and screen applicants based on their skill set. The added bonus for a hiring manager was the ability to click through and see some of the written output of a candidate. If they’d ever asked or answered a question on Stack Overflow, one could review this material and gain insight into the author. Were they polite? Did they communicate well? Do they appear to be knowledgeable?

A great example of how Stack Overflow Jobs approached the market differently was the “Developer Story”. This was like a CV, but more free-flowing and with the candidate more able to highlight interesting content of varying types. At its core was a timeline of items, which could be employment history, open-source contributions, courses completed, etc. Here’s a screenshot of mine:

And a link, whilst it still works. (Side-note: my main domain tdwright.co.uk currently acts as a redirect to my developer story – what am I going to do with that?)

Another interesting aspect of Stack Overflow Jobs was the close link with the developer survey conducted by Stack Overflow. This primarily manifested itself in the salary calculator:

The salary calculator was my go-to tool for benchmarking salaries – my own and those of my team. It’s a shame the data hasn’t been updated since 2019, but the power of objective salary estimates is invaluable. I’ve often linked to results from the calculator when discussing budgets with non-technical colleagues.

All in all, I am an unashamed fan of Stack Overflow Jobs. It added a lot of value to the developer ecosystem and I personally benefited from it a great deal.

Praise for Stack Overflow Jobs

So it was a great product? Well, don’t just take my word for it. In response to the announcement of its imminent demise, there has been an outpouring of regret and disappointment, alongside praise and fond memories. (The below are all taken from the replies to the announcement.)

“I don’t use a resume/CV anymore, I send people my SO Dev story!” – Chris Schaller
“Terrible news. SO Jobs was at the very top of my list of reputable job lists. And Dev Story landed me some of the best interviews Very sad news!” – Luis Alvarez
“I’m really disappointed in this. I’ve always found SO much better than other job sites for finding quality positions.” – TrueWill
“As someone who recently found a job through SO Jobs, it was light years ahead of many other places, although I believe it could certainly have used a few improvements it’s really sad to see it go. I loved that I could filter and find jobs that are actually relevant to me without having to sift through 35345 other job postings that just happened to contain a keyword or something like many other jobs websites do, I’m really disappointed in this.” – maxshuty

In fact, I’d encourage anyone interested to scroll through the comments. The affection for Stack Overflow Jobs is very real and plainly evident in these messages.

Commiserations to the team

Being a community for developers, it’s poignant to read some of the replies left by those who have worked on the product over the years:

“Bye bye /jobs, we had some fun times building ya <3” – Dean Ward
“Sad day. I spent four years of my life building and maintaining those things.” – rossipedia
“Well, Dev Story, you had a good run. I once had high hopes in you, but it looks like you weren’t meant to be forever.” – balpha

I also know one of the early employees in a personal capacity, so I understand how committed this team was. (NB: They haven’t worked there in a while and we did not discuss any details whilst I was writing this post.)

It’s always disappointing when something you’ve worked on is shut down, but it can only be worse when the product is so well-liked.

The decline

So why is such a popular product being closed down? According to the announcement, it boils down to strategic direction. As former Jobs developer Jon Erikkson points out “it’s easier to see how the new products fit into the strengths of Stack Overflow”. The official line is that this closure will allow the company “to refocus on products that build on our core strengths: knowledge reuse and building communities at scale.”

In truth, the writing appears to have been on the wall for a while now. Back in April ’21, CEO Prashanth Chandrasekar revealed in a blog post that something was going to happen with Jobs. The acquisition by Prosus in June can surely only have sped up this process.

Anecdotally, at around the same time as the acquisition, I tried unsuccessfully to once again become a customer. I was peeved to realise that my business was no longer worth getting out of bed for:

At the time I wondered if this was a bad sign for Jobs – would good candidates continue to use it if the smaller startups weren’t there? With hindsight, however, this email feels symptomatic of a gradual retreat from the job ad market.

Better to die the hero

Long time readers will know I’ve got a lot of affection for Stack Overflow. I previously described the Q&A site as my alter alma mater. Jobs has played an equally significant role in the later stages of my career in software development.

Having had some time to digest this news, I’ve been able to think about this from multiple angles. Despite the significant loss to the sector and the disappointment of the users and SO team, it might be better that the move was at least made decisively. Once the business had decided that this ugly duckling didn’t belong, it really stood no chance. And isn’t it better to know when to hang up the gloves? The alternative would be a neglected product limping down a sad path of decreasing relevance. So, whilst I will sorely miss Stack Overflow Jobs, and whilst I am still very disappointed in this decision, I suppose it’s better this way.

I can’t help but return to the euphemism central to the announcement. Rather than being like a sunset, this is more akin to the calculated euthanasia of a racehorse.

Goodbye Stack Overflow Jobs – we had a great run.

Three ways the lockdown improved our agile culture

Tom Wright — Tue, 07 Jul 2020 08:22:30 +0000

This article was written by Paul Boylan. Paul is a Scrum Master at GPI, establishing a supportive environment where the development team can work at its best.

At the end of our Coleridge sprint on the Horizon project, we had our first major demo for the leadership team. Hardly settled into our new offices, we were a little in denial we would soon be working out of our homes. Before the end of our Duxford sprint we were all in lockdown, and we faced the challenge of sprints Ely through Hauxton as individuals.

1. Self-organisation

This was a new team, who hadn’t been working together for long. Normally a challenging time. Still in the polite stage, differences in working styles were showing as a theme in retrospectives. Sending everybody home would likely exacerbate this. In fact, with change now inevitable for everyone, I now saw the team finding ways to work. Practice, the ‘how’, emerged from the team.

2. Stand-up engagement

I have three stand up meetings each morning. I am not in the same room as any of the teams. They might be split across a couple of locations, but they are in a group. This changed with lockdown. The purpose of the stand-up meeting is for the developer to tell their own story to themselves, but also to the team. It is easy to slip into the habit of making each update sound like it is to the Scrum Master, especially when the Scrum Master is the most remote. It is an update to the person who wasn’t here yesterday and won’t be here today. Unexpectedly, with everyone working from home, a more complete circle was formed, increasing participation and collaboration.

3. Story selection

Another area refreshed by the lockdown has been story selection. Over time, the cadence of sprints begins to work against value. We want to deliver the highest value possible each sprint, but we know there is a next sprint coming. In lockdown, I have noticed a new importance in having something great to show. We began to visualize the product, one sprint on, either with or without a feature. It is easy to imagine a craving for interaction and a need for purpose as driving this, but another idea is that we are living with more ‘cannot do’ now and this has sharpened our focus on the ‘can do’: what if the user can do this… or this?

If agile thrives in environments of adversity, unpredictability, and change, then it follows that there is a risk that it loses ground where these are absent. The challenge, I think, is to keep with us the spirit of these times.

How has lockdown affected your agile practice? Let me know in the comments below.

Running a static site for free with Azure Blob Storage and Cloudflare

Tom Wright — Wed, 24 Jun 2020 21:23:58 +0000

Lately, it feels like there’s an arms race in the “cheap static hosting” space. In today’s blog, I share my current favourite approach, using Azure Blob Storage and Cloudflare. I explain the problems I’ve encountered with other Blob Storage approaches, as well as how they can be overcome using Cloudflare. Oh, and it costs less and £0.01 per month. Intrigued? Read on…

With Gen 2 of Azure Blob Storage, we gained the ability to serve static websites directly from a blob container. All we need to do is flick a switch and put our files in the $web container. I won’t go into the details of this here, as my colleague Sneha recently published a great write-up.

Blob storage is so cheap!

The advantage of this approach is that it is cheap, cheap, cheap. Like, really inexpensive. So cheap, it’s virtually free. Go take a look at the pricing for blob storage, then consider that the static website option is free, and remember that your first 5GB of bandwidth (outbound, as inbound is always free) is free too. I challenge you to spend more than $1 / month.

The downside of just using blob storage is the almost total absence of any “mod cons”. You won’t find much configuration or monitoring beyond what a vanilla storage account offers. You can specify what your index and 404 pages are and, well, that’s about it.

You can configure a custom domain, but this comes with some limitations:

It’s not possible to use the “apex” (root) domain, so must use a subdomain. Using “www” is the most common workaround, but the experience for users who fail to include this won’t be graceful.
Similarly, HTTP is not supported. All requests have to be over HTTPS. This isn’t a problem in itself, but (again) the fallback is a less than ideal error message.

So how can we have our cheap cake and eat it too? How can we enjoy the effectively free storage on offer, whilst retaining some of the bells and whistles that the discerning developer expects?

Azure CDN?

One option is to use Azure CDN. You can read how to do that over in this post by Chad Shulz. This approach does give you some more fine-grained controls. Moreover, it does allow one to solve one of the two problems mentioned above – seamlessly redirecting non-HTTPS traffic to HTTPS.

Significantly, however, it seems as though Azure CDN no longer supports apex domains. There is a feature request for this, which has been open for two years now. Some people (including Arlan Nugara) have reported success using workarounds, but none of them worked for me in June 2020.

On top of this, there’s the price point. Azure CDN is by no means expensive, but it isn’t free. If we’re honest, it is basically free, with monthly costs probably coming in at the 10s of cents mark. Still, why would I pay anything when there’s a 100% free service I can use?

Cloudflare

I have been aware of Cloudflare for a while but first came to use their services when I moved this blog to SiteGround. I wrote about my experiences a few weeks ago and would continue to sing the praises of both SiteGround and Cloudflare.

When I was struggling to configure Azure CDN to use my apex domain, I started to wonder if I could do the same thing with Cloudflare. Let me tell you, I was pleasantly surprised by how simple it was. At first, I was actually trying to configure Cloudflare in front of Azure CDN, but quickly realised I could simplify things by pointing Cloudflare at blob storage directly.

Here’s what I did:

Add a new website to my Cloudflare account using their free tier
Configure the custom domain in blob storage
Create CNAME records to point my www subdomain at my blob storage hostname
Added the verification records to my DNS records
Force all traffic to be HTTPS by toggling the “Always Use HTTPS” switch
Create a “Page Rule” to perform a 301 redirect from my apex domain to the www subdomain.

And that’s basically it. The two main problems with naked blob storage are fixed in steps 5 and 6. Since these are pretty significant, I should probably go into a little more detail.

Well, perhaps there isn’t a lot of detail to go into for the first of these… As I said, setting up an HTTP-to-HTTPS redirect really is as simple as toggling an option from “off” to “on”. Whilst we’re here though, I could point out a wealth of other SSL-related options. You can see here that I’ve set the minimum TLS version to 1.2, for instance.

With the apex-to-www redirect rule, however, there is something more interesting to look at. Here’s how we configure a “Page Rule”:

The interface is intuitive and uses natural language to simplify the process. It’s very straightforward to intercept URLs matching a pattern (in my case lacking the www subdomain) and send them somewhere else. This is probably the least exciting kind of Page Rule. Other uses might include preventing sensitive details from being cached, for instance. (There’s even one intriguingly called “Rocket Loader”.) You can read more about Page Rules in the Cloudflare docs.

Overall, Cloudflare is a very feature-rich offering. The dashboard is packed with statistics and configuration options to explore. The free tier is obviously a loss-leader for them – you’ll be teased by many of the “pro” features as you look around – but it’s impressive what you get for free.

Final thoughts

I’ve been running my little static site for almost a month now, and my Azure bill is on track to be less than a penny. I’m quite interested to know how Microsoft will bill me for this! Even if they round it up to £0.01, I think I’ll still consider it to be effectively free.

Truth be told, I would have gladly paid more for the simplicity of having everything in one place. If the Azure CDN had done what I needed it to do, I might have been willing to overlook the £0.50/month (or whatever) as merely the price of convenience.

Now though, if Microsoft did improve their CDN offering, I think I’d have to give it a lot of thought. I really like the Cloudflare dashboard; the insights it and flexibility it offers. The Azure CDN team have a lot of catching up to do before I’d be tempted back based on features.

It’s a problem that will remain academic for the time being. I’m delighted with my setup as it is. It’s fully functional, ferociously fast, and best of all… free! Azure Blob Storage and Cloudflare make a great team.

What’s your preferred method of hosting static sites? Let me know in the comments.

Accelerating data ingestion with Bartscript

Tom Wright — Tue, 23 Jun 2020 08:28:46 +0000

Pricing and reimbursement data are the lifeblood of the pharmaceutical industry. The maintenance of GPI Pulse involves retrieving, standardising, and interpreting data from over 100 different sources. Automating this data collection and processing is vital to our commercial success.

Historically, our data processing pipelines have been built by developers. These work well, until the data source changes. Unfortunately, when the source drifts, it takes a developer to make the corresponding adjustment to the pipeline. One of our significant challenges in the engineering team has been balancing this sort of maintenance with feature development. At the same time, the analysts were facing delays.

Our response has been to invest in bespoke tooling for our data analysts. Taking the lessons learnt from the development of our traditional pipelines, we have been able to produce a generic, customisable pipeline. This approach has put our analysts back in control of data processing and increased the agility with which we can respond to upstream changes in data formats.

Central to this strategy has been the development of an internal “Domain Specific Language” (DSL). For the sake of familiarity, we modelled this new language on Excel formulas. This language allows for the declarative articulation of a group of data transformation actions, arranged in a sequence of phases. We call this new language “Bartscript” in honour of the contractor who lead the development efforts.

Even though it is so niche, Bartscript has many of the features you’d expect from a fully-fledged programming language:

Branching logic
Loosely typed and liberal when it comes to coercion.
Full sandboxing
Comprehensive error handling

Despite this sophistication, Bartscript remains deceptively simple to use. This simplicity has facilitated a team of two analysts and one developer to automate sources at a rate of 5 per sprint. For comparison, under the old approach, we would have expected one developer and one analyst to work on one country full time for one sprint (give or take). In other words, we have reduced development time by two thirds.

The real test will come when one of the pipelines requires some modification due to a change in its source. We hope that, in many cases, the analysts can make the necessary changes with minimal support from the developers. We are still in the early days, but the initial signs are good.

The Joy of Serverless

Tom Wright — Sun, 24 May 2020 20:11:04 +0000

Today’s post is a little bit different to usual. I want to address a discrepancy between how aficionados talk about serverless and how it’s covered in written material. In this post, I shall unashamedly extoll the subjective (perhaps even emotional) virtues of serverless software development. I shall draw on some quotes from members of the serverless community and discuss what it is about serverless that makes it so joyful.

In recent years the technical virtues of serverless architectures have been extolled ad infinitum. Holy wars have reigned over precisely what “serverless” means, whether fears of “vendor lock-in” dictate that we should stay away, or whether it’s all a fad. I’m not planning on weighing-in on these discussions. Nor am I going to be drawn into the “there are still servers in serverless” squabbles. No, what I’d like to do is offer another perspective.

In this post, I’ll be talking about the joy of serverless development. Hopefully, I can tempt somebody who hasn’t yet explored the world of serverless architectures to do so.

Developing serverless applications is fun.

That’s a big claim, but I’m going to back it up. Let’s start with personal experience.

Over the past four years, I’ve built and maintained a wide range of applications. Some were internal back-office systems. Others powered popular mobile apps. What unites them all is that they all feature serverless elements, or in some case entirely serverless. I’ve worked mostly with Azure, but have some experience with AWS and GCP.

Regardless of the technology, the developers I’ve worked with on these projects have enjoyed working with serverless technologies. And I don’t just mean that they’ve enjoyed not having to deal with servers, but something more fundamental than that.

Outside of my day-to-day jobs, I’ve also had some exposure to the broader serverless community. I was one of the organisers of the inaugural ServerlessDays Melbourne last year and also had the opportunity to attend our sister event ServerlessDays Sydney.

Both events were jubilant celebrations of creativity. Everyone there seemed to be passionate about the technologies we were discussing. Now, I’ll accept that this may partly be due to a degree of selection bias. Still, I maintain that the vibe of these events was more energised than any other software-related conference I’ve attended.

What is it that serverless developers enjoy?

To explore why serverless developers enjoy the paradigm, I decided to try and draw on experienced beyond my own. I asked my network on LinkedIn for their thoughts and also started a discussion on Reddit. To my surprise, 11 people took the time to share their thoughts. So, before diving into my findings, I’d like to thank everyone who responded!

As you might have guessed, each of the respondents was broadly very positive about serverless. (There were, however, some caveats applied, which reassures me that I wasn’t just talking to fanatics.) Many of the answers included themes that align with the frequently cited technical and organisation benefits, but there were some surprises too.

I’ve been through the answers and tried to code them to extract themes.

Scaling, costs, and speed

Given that they are the most commonly cited benefits of a serverless approach, it’s no surprise that ease of scaling, low costs, and speed of development were mentioned many times in the responses.

I’ve considered cost and scaling as one theme since they are often mentioned in tandem. It seems not only to be the overall cost per se that is attractive but the predictable way in which cost scales with usage.

Mishal Sanghvi mentioned that developers can “start with an idea and a small budget“. Likewise, @xavi_lefevre appreciates “knowing (at least believing) that it’s only consuming resources when needed“. Sneha Biswas explains that one of the reasons she loves serverless is “scaling up and down and being so cost-effective“.

Overall, four of the eleven respondents cited costs and scalability as one of the reasons they enjoy working with serverless.

Speed of development is another significant factor. Three respondents cited this directly. Rodgerjm put it succinctly: “I’ve observed and built serverless/microservices projects in days instead of weeks or months.” Szotrj echoes this sentiment: “I can […] iterate extremely quickly, and deploy a new capability in a matter of hours.”

Fewer distractions, less worrying, more focus

Another group of themes from the responses all relate to manifestations of simplicity. Whether it’s the reduction in cognitive overhead, the ability to focus on solving business problems, or respite from the fear of something going wrong, the underlying cause is a lower level of embodied complexity for the developer to handle.

I was surprised by the number of responses that used the word “worry”. It was the most prevalent of all the themes I extracted from the answers. With hindsight, it makes sense: worry is unpleasant, can be chronic, and often lacks a clear route to resolution.

My favourite comment about worry was from mightydjinn: “[Serverless] increases safety with staged rapid iteration, and reduced blast radius of bad events when they do occur.”

VAST_BLINKER_SHRINK is all about the simplicity: “Love never having to SSH into another server ever! Love never worrying about server crashing ever! Love not having to worry about application state! Love the simplicity the mental model!“

Matt Gillard neatly explains how simplicity gives us the freedom to focus on what matters: “I have spent too many years building infrastructure from the ground up, just to host business logic, and having to deal with procurement, building snowflake OS’s, patching, identity management. Serverless (or should I say thinking with a serverless frame of mind; i.e. thinking about business events and event-driven architectures!) removes the need to worry about these things and allows the investment to be made on stuff that is directly beneficial to business outcomes.“

The fascinating thing about this theme is that it is the flip-side of one of the prevalent arguments against serverless. A recurring criticism is that simplicity comes at the expense of control (see this article). To a large extent, this is true – the complexity is still there, but we’ve offloaded it to a specialist provider, and we no longer need to worry about it. This concern may be legitimate for some projects. In my experience, however, this critique is very often just a modern version of Not Invented Here syndrome.

The fact that so many of the respondents mentioned these themes (less to do, less to worry about, increased focus on business problems) signals to me that developers truly value the simplicity once they’ve embraced serverless. The big question is whether serverless aficionados gravitate towards the paradigm because they are predisposed to prefer simplicity, or whether they have had to overcome negative feelings associated with loss of control?

Other notable mentions

The themes mentioned above all have more than one response supporting them. Perhaps the most interesting comments, however, were mentioned only once. Although we can’ really generalise from these, I think they each say something profound about the experience of working with serverless.

First off, we have Sneha, who ends her response with “It is just absolute fun!” And I fully understand what she means. Creating, inventing, building – there’s a real thrill to it. Without the other complexities and overheads, we are free to just, well, enjoy ourselves.

The next one (from spopgg3) is fascinating: “I feel guilty sometimes, because someone else is managing the hardware and software of the server. It’s like magic.” This reaction was not one I had anticipated, but it makes complete sense. It’s natural to feel spoiled when we’re used to having to worry about all the little details and suddenly have these taken care of for us. It almost feels a little too good to be true!

Finally, @xavi_lefevre said something that resonated deeply with me: “Talking about pure selfish enjoyment: feeling like a pioneer discovering a new territory full of promise.” For a lot of us, the transition to serverless has been a voyage into the unknown – one that has certainly paid off. Like a weary traveller who has hiked for days, we crest a hill, and spy verdant pastures of untold opportunity unfurl ahead of us.

What does it all mean?

For this post, I wanted to go beyond the dry technical discussions of merit. While I firmly believe that serverless approaches do tend to have concrete benefits over their more server_ful_ counterparts, it felt like the draw for me was as much emotional as rational. I wanted to probe into this aspect – why do serverless developers seem to enjoy serverless so much?

I’m so grateful to those who responded to my question. What I think I’ve found is a range of different qualities of serverless development that bring joy to developers. While some of these are derived from (and closely related to) the technical benefits of serverless, there is another significant grouping of qualities that are more connected to the mental agility and wellbeing that serverless affords. Further, there seem to be a third, more idiosyncratic, group of responses that seem more personal.

It should not be surprising that there is considerable overlap between the sterile, advertised advantages (scaling, cost, speed of development) and the more emotional responses. After all, as professional developers, we take pride in our work, and thus we gain satisfaction from tools that enable us to do our job more effectively.

Similarly, it makes sense that lowering the complexity and eliminating causes of stress would be desirable. Nevertheless, I was somewhat surprised by the prevalence of these themes. The word “worry” cropped up repeatedly. It strikes me that serverless developers may feel liberated from a burden of which they may previously have been unaware. The developers who responded seemed to be glad of the relief from having to carry around arcane knowledge, from being stressed by the possibility of significant downtime, or from simply being distracted from core goals. In short, these developers enjoy being able to focus on what matters.

Finally, we have responses that are pure emotion: The feelings of being spoiled by the luxury of managed services, the wonder of exploring new technologies, and the sheer joy of developing serverless applications. These feelings come from the heart and, in my opinion, are probably under-represented in the responses I got. As engineers, we naturally tend to be somewhat stoic sorts. Insights like this give us a peek behind that curtain.

The joy of serverless

People will identify many reasons for their fondness for serverless, and just as many ways of expressing those sentiments. What is clear is that serverless developers are often passionate about the approach. This passion, I would wager, is a significant factor in the commercial success (and continued growth) of serverless offerings. No matter how stoic some engineers appear, we are ultimately all emotional animals.

It is essential to recognise that not all engineers are alike. As Ant Stanley wrote: “Serverless is a buzzword that has marmite appeal.” Although I strongly suspect that the ranks of serverless proponents will continue to swell in the coming years, there will be those who genuinely prefer a more server_ful_ approach.

If there are any reading this far down who are not already converts to serverless, I strongly encourage you to give it a go. It may seem alien at first, but there’s a good chance you’ll end up loving it. Or perhaps you tried it before and didn’t enjoy it? As serverless continues to mature, the tooling, documentation and community all become richer, making for a more accessible and rewarding experience. In other words, you should take another look!

For those of you that happen to like the taste of ~~marmite~~ serverless, I would very much like to continue to hear your reasons. What is your favourite aspect of serverless? Let me know in the comments.

Naturally, having had our passion for serverless suitably validated, the instinct would be to go and share it with the uninitiated/hesitant/resistant. When evangelising, however, we must be sensitive to the preferences of others. No matter your sincerity, pushing too hard will switch people off. We must also remember that their choices do not invalidate your own – you are not less of a developer for preferring less server.

Personally, I was thrilled to see the topic of serverless evoking so many expressions of joy. And so varied too! It bodes well for the continued health of the community and for the longevity of a development style that I find so enjoyable.

It feels as though the most fitting ending to a post like this would be some sort of rallying call, so I will do my best… Rejoice in the fact that serverless is fun! Feel no guilt for enjoying your craft. Dear reader, I implore you: Go forth and (gently) spread the joy of serverless!

Goodbye GoDaddy! Hello SiteGround!

Tom Wright — Tue, 12 May 2020 10:00:00 +0000

I’ve recently ditched GoDaddy for my personal hosting needs, moving my stuff (including this blog) to SiteGround. In this post, I talk about why I moved and the rewards for doing so. Also, I’ll share my initial experiences of using Let’s Encrypt and Cloudflare, which SiteGround has enabled.

When I first bought my domain back in 2008, the internet was a different place and I was pretty clueless when it came to the process of registering a domain and connecting it to hosting. I did what a lot of people do – I bought the whole lot from GoDaddy. I’ve been a customer ever since, even if this is more a reflection of inertia than loyalty.

Out with the old…

Having recently decided to start blogging again, I’ve been giving my site a bit of a “spring clean”. In between updating plugins and clearing out drafts, I noticed that WordPress was warning me about an insecure version of PHP.

It’s not exactly subtle! Hard to ignore.

Suitably chastised, I immediately logged into my GoDaddy account and set to work. Unfortunately, I soon found that my options were very limited. For my hosting, I was able to choose between PHP version 5.4 or 5.6. The good news is that I was already on version 5.6. The bad news is that this active support for 5.6 ended in early 2017 and in January of this year it had reached “end of life” – no more fixes, even for security.

As per the official documentation, even versions 7.0 and 7.1 are now officially at end of life. The version to use now is 7.4.

Now, I’m not too proud to admit that my site was running such an outdated version of PHP. Heck, I’m not over the moon to be confessing my use of PHP at all. (I jest! I jest!) To be honest, though, I had assumed that GoDaddy would update PHP in the same way they update the OS. It didn’t cross my mind that this would be a manual task. I suppose I spend so much time on managed cloud platforms these days that I’d developed a bit of a blind spot. This site is, after all, the only one I run that uses shared hosting.

I spoke to one of GoDaddy’s support agents, who told me that the issue was the age of the hosting package I was using. In other words, I was being punished for being the fact I’d been a customer so long! I was offered a migration to a more modern package – for a fee of $99.

The end result is that my dissatisfaction with GoDaddy rapidly grew to the point where it could overcome 12 years of inertia. I decided to take my business elsewhere.

… and in with the new

After shopping around a bit, reading reviews, and speaking to some friends, I eventually settled on SiteGround for my hosting needs. At the moment they’re doing a COVID-19 promotion: £0.99 for the first three months. Bit of a no-brainer really.

Jumping ahead a bit, but… I’m so pleased with SiteGround that I’ve signed up to be an affiliate. If you are considering signing up, please consider using my link: https://siteground.co.uk/go/tt38p9uy8z (it doesn’t cost you anything)

The things that drew me to SiteGround included:

Great reputation and reviews
UK-based hosting
Mature and professional
Transparent about their tech
Lot’s of nice perks (more on these below)

Registration was super easy and they’ve clearly put a lot of thought into the user experience of managing a hosting account. It’s a long way from CPanel! I very quickly managed to configure DNS, a MySQL database, and an FTP user. In other words, getting my basic WordPress site set up took no time at all.

Getting the site set up so quickly meant I could start to explore a couple of the perks: Let’s Encrypt and Cloudflare integrations.

SSL Certificates from Let’s Encrypt

Let’s Encrypt is a provider of 100% free SSL certificates that launched in 2015. This was hugely disruptive, as until then SSL certificates were quite expensive; not super expensive, but certainly expensive enough to dissuade most amateurs and hobbyists.

Not that the impacts of Let’s Encrypt are limited to personal projects. My team at work are now making extensive use of the free certificates. HTTPS is now standard across all environments.

Unfortunately, despite my being very familiar with Let’s Encrypt, the GoDaddy hosting I was using had, for a long time, made it impractical for me to use it for this blog. Beyond simply not providing a one-click integration, my GoDaddy Linux environment was incapable of running the Let’s Encrypt tool, despite my repeated attempts.

Why wouldn’t GoDaddy make this easier? They’re a big company and I’m sure it has plenty of smart employees – surely they could figure it out? Well, I don’t mean to be cynical, but perhaps it has something to do with the amount of money GoDaddy makes from SSL certificates. Right now they want £44.99 for a single website. Their costs on this are basically zero, meaning they are basically printing money.

Anyway, SiteGround isn’t in the gouging business. They have an integration with Let’s Encrypt baked into the account dashboard. It took me seconds to get a completely free certificate for this site. Oh, and it will automatically renew too!

I count maybe 3 clicks? Super simple.

They also have an option to “force HTTPS”, which I think is the user-friendly term for HSTS. Honestly, the hardest part of moving this blog over to HTTPS was finding out which PHP file contained the hardcoded HTTP reference to Google Fonts!

Cloudflare

To be entirely frank, I’m including Cloudflare here because it’s so interesting, rather than because I’ve decided it’s essential for my blog. Obviously, since I’ve not been living under a rock, I have a basic understanding of what Cloudflare does. My understanding of their product, however, has been based on an assumption that only massive sites need it.

Seeing Cloudflare as a push-button option in the account dashboard of a hosting account for which I’ve paid £0.99 just seemed, well, a little incongruous. I mean I obviously pushed the button. (Who amongst us wouldn’t?) But I’m yet to be convinced that I need it.

Part of the reason I find this so interesting is that Cloudflare has some pretty advanced features. I know Troy Hunt, for instance, really likes their edge serverless compute offering. Not quite sure how I’d put that to use on my blog, but I’m chuffed it’s an option!

Another intriguing feature is the built-in firewall. At work, we have a site protected by a GoDaddy managed firewall. We paid (not under my watch) north of £150. The free firewall offered by Cloudflare has all the features we care about.

I’m in the early stages of evaluating Cloudflare but am impressed so far. Anecdotally, the vast amount of spam my contact form used to attract has slowed to a trickle. This corresponds with several blocked requests from Tor users trying to access that form. Hardly a scientific test, but encouraging nonetheless.

One thing to note is that Cloudflare supports TLS 1.0 and 1.1 by default. These are now widely considered to be inadequate and insecure. Luckily, I was alerted to the issue by the Qualsys SSL Server Test tool. (This, by the way, is an excellent tool. I’d recommend it to anyone who works with websites.) This blog initially got a B – this is the max score you can get whilst still offering TLS 1.0 and 1.1. To their credit, Cloudflare does provide an option to set the minimum supported version of TLS for your site. Changing it to TLS 1.2 was all it took to boost this site to a grade of A+ – much better!

Wrapping up

It’s been a really eye-opening few days for me. Having been sufficiently annoyed to take action, I now can’t believe I held out for so long. GoDaddy and SiteGround are worlds apart. Where the former seems to hold their customers in contempt, the latter are going out of their way to add value.

I’m really pleased to have finally got my blog running on HTTPS and am really excited to give Cloudflare a proper evaluation. Neither would have been impossible with GoDaddy, but the barriers were just high enough to put me off. With SiteGround, I was encouraged and supported to give both a go.

Interestingly, as I was writing this, a story broke about GoDaddy being hacked. Apparently, hackers gained access to a file controlling SSH access and made off with as many as 28,000 sets of credentials. As an SSH user, I presume I’m included in this figure, although haven’t had any notification from GoDaddy.

If you’re looking for a new hosting provider, I can wholeheartedly recommend SiteGround. I feel very confident that readers of this blog will love SiteGround.

If you want to support this blog (and help me cover the whopping £0.99 hosting costs!) you can use my affiliate link. It won’t cost you a penny, but it’ll help me out.

And to those of you still putting up with GoDaddy, I implore you to consider moving elsewhere. It’s really not that much hassle and the rewards are enormous. You can do it!

9 things Westworld can teach us about software engineering

Tom Wright — Thu, 07 May 2020 13:19:10 +0000

Watching Westworld recently, I couldn’t stop seeing parallels with software engineering. In particular, the series is rife with cautionary tales about how not to do software development. In this article, I’ll share 9 lessons we can take from the show to improve the effectiveness of software engineering teams – and hopefully prevent them from going “full Westworld”…

Like a lot of people, I’m watching a lot more TV at the moment because of the Covid-19 lockdown. Despite being recommended to me ages ago (thanks Alex!), I’ve only just got around to watching Westworld. When we finally started watching, it immediately displaced all other shows and we crashed through season 1 in no time at all. (We haven’t started season 2 yet, so this article is just about season 1.) I’ll try to avoid spoilers where possible, but, well… the episodes I’m talking about aired back in 2016.

For those that haven’t see the show, Westworld is basically a massive open-world RPG made reality; a western-themed park inhabited by robot NPCs. Guests pay large sums of money to come and play out their wildest fantasies, including the killing and raping the robotic “Hosts”. A lot of the story in the show takes place behind the scenes, giving us a window into how the “Body Shop”, “Behaviour”, “Narrative”, and “QA” teams interact, as well as the effect this has on the smooth running of the park.

As someone interested in processes that can enable development teams to be effective and the cultural trait effective teams exhibit, I regularly found myself spotting areas for improvement whilst watching Westworld.

NB: Without giving too much away, there are some really interesting ethical questions raised by the show. For the purposes of this article, I’m going to ignore these and just assume we all want the park to be run effectively. I’m also going to gloss over some of the twists and simply take the park at face value.

So, without any further ado, 9 things that Westworld (season 1) teaches us about software engineering:

1. Dev, Ops, QA should all work together

At several points in the story, we see problems arising due to poor communication – or even outright hostility – between various teams of Westworld staff. If the Body Shop had been more able to communicate freely with the other teams, perhaps they would have raised the alarm when a Host starts displaying behaviours that should be impossible. Or, if there hadn’t been such a power struggle between QA and Behaviour, perhaps they could have collaborated to more quickly resolve a botched update.

The parallels are clear: The Body Shop look after hardware, Behaviour is the coders, and QA is, well, QA. Time and again the show reveals the consequences of the adversarial relationships between these fictional teams. I bet most engineers have experienced some degree of this in the workplace, as well as some of the consequences (though hopefully not so catastrophic). The lesson here is to remember we’re really all on the same team, with the same overall goal. If QA, DevOps, Development (and all the other associated teams) can all pull in the same direction we’re all going to be able to do our jobs better.

Members of three teams arguing over a malfunctioning Host.
Westworld images courtesy of Anastasia over at thetvshows.us

2. Complexity is the enemy

Westworld is effectively a massive interconnected collection of computers – each Host has many layers of its own set of motivations stacked on top of a common set of “Prime Directives”. This complexity causes tension even when the park is running without bugs, bad updates, or glitches. A team of analysts nervously keeps an eye on activity, whilst others act as enforcers with itchy trigger fingers.

The problem with complexity is that it decreases the chance that anyone will be able to be rational when debugging. We can only hold so many causal links in our heads, only so many triggers and reactions. When a system grows to the sort of complexity that exceeds this limit, we begin to rely on heuristics rather than reason. Even if we think we understand the system, we can encounter “emergent behaviours” when features interact in unanticipated ways.

How do we keep systems simple? Some might say architecture, but you can have a complex set of microservices just as you can have complex monoliths. In part, we can reduce complexity by carefully designing components so that they interact in simple ways. And we can try to manage any residual complexity with tests, monitoring, and documentation.

3. Make sure you understand legacy code

One of the big themes of season one is the possibility that a former contributor to the team has left in some code. This is a possible explanation when the Hosts appear to gain abilities they shouldn’t have. Malicious intent is suspected, but this is hardly the point. The fact that it’s a mystery is the real problem here.

When we are new to a legacy codebase, we really need make the effort to try and understand it. Too often we find ourselves producing layers of new code before we get a chance to really become familiar with the old code. This can be especially true when we’re working on a very complex, unreadable, impenetrable codebase.

Certain circumstances make it especially important to really comb through the code of an inherited legacy system. First, if we suspect that our predecessors were either malicious or incompetent (or both!) then we really must do our due diligence. Secondly, if the application is particularly high-risk, the consequences of not understanding the code can make it foolish not to do so. What makes the Westworld case so egregious is that both these factors applied!

Anthony Hopkins is incredible as Dr Ford.
Westworld images courtesy of Anastasia over at thetvshows.us

4. Don’t skip code review

Early in the season, we hear that one of the original developers, now very senior, has bypassed the code review process to add in a feature he’s developed. Out of deference to the developer’s seniority, the head of the Behaviour team permits the feature to be deployed to production. This backfires when the code is later linked to a serious defect. The story shows how important code review can be – and how it should even apply to very experienced developers.

Code review would also have helped the developers to better understand the legacy codebase. Further, code review can be an important tool in managing a complex system, as the knowledge of how each part operates becomes more distributed throughout the team.

5. Remember why we test

Despite having a well-staffed and powerful QA team, Westworld’s testing processes seem to leave a lot to be desired. As mentioned above, senior employees are apparently free to deploy features into production without any oversight, let alone testing. Where were the acceptance criteria? Why was there no regression testing?

When developing software, quality is everyone’s responsibility. Before writing any code, we should ensure that all parties share a common understanding of a feature entails. Developers should then test their own code as much as possible (e.g. manual testing, unit tests) before handing it over to QA.

In Westworld, many of the park staff seem to view the Hosts with a degree of superstition. Likewise, the QA team’s approach to testing seems largely ritualistic. This leads to a very superficial form of testing. Following processes is an important part of QA, but it isn’t why we test. Your list of checkboxes is there as a foundation, but effective QA can only be achieved if all contributors approach the system with an inquisitive mind.

Ultimately, we all want to ship the best product possible. That’s why we test.

The Body Shop can be a messy place.
Westworld images courtesy of Anastasia over at thetvshows.us

6. Tidy up after yourself (and maybe beforehand too)

The Hosts are supposed to be completely reset between deployments, but we find that in some cases their memories aren’t fully wiped. In some cases, even when a Host is radically reconfigured to play a different character, elements of the old character may linger. This laziness isn’t of much consequence until some of these memories start to interact with new behaviours in unexpected ways.

The equivalent in day-to-day software engineering is to have a nice, empty context to work in whenever our program runs. If we cause some files to be written to the filesystem, for instance, we may have to tidy these up to prevent them from affecting a future task. Or if we need to create a temporary table in a database, this should be dropped when it is no longer needed. Where our code has side effects, we should manage these carefully to prevent pollution of future scopes.

7. Assess the logs with a healthy dose of scepticism

Without giving too much away, it becomes clear that the action we’re watching isn’t all playing out at the same time. Events that appear contemporaneous are actually, in some cases, separated by decades.

This reminded me of trying to solve a particular bug. The logs were apparently telling me that events had taken place in an impossible sequence. How could B have happened before A? It didn’t make sense! Well, like the rest of the application, the logging had been written by developers and turned out to be less reliable than expected.

Separately, we find that (at least some) members of the Behaviour team can edit the interaction histories of the Hosts to cover their tracks. Depending on how your logging system is implemented, consider whether it would be possible for a colleague to alter your logs, either accidentally or intentionally.

Either way, remember that your logs (like everything else) have the potential to be imperfect. If your gut tells you something doesn’t add up, try to bring in evidence from other sources and see if it tallies.

Although impressive, this massive map of the park never seems to be useful for very much…
Westworld images courtesy of Anastasia over at thetvshows.us

8. Make your application (easily) observable

Aside from the logging issues noted above, we also find out that aspects of the legacy system are rather hard to inspect. Hosts are, it seems, debugged primarily via voice commands. Westworld staff frequently have to enter the park to identify issues/malfunctions with the hosts. One member of the Behaviour team even has to travel to a location within the park to access logs from a particular part of the infrastructure.

It should go without saying that having to be physically close to a system to extract diagnostic information is hardly ideal. It might be tolerable in some consumer electronics, but Westworld was built by one corporation on a single (admittedly large) site. I could perhaps forgive them the lack of remote diagnostics if it weren’t for the fact they seem to have advanced telecommunications throughout the park.

Where possible, having all logs, monitoring and diagnostic data sent to a central, accessible location (e.g. the cloud) is going to make it loads easier to manage your application. Simple questions about the general health of your application are much easier to answer when you’ve got a steady stream of data coming in.

No-one should ever have to travel to a creepy old theatre in order to access system logs…
Westworld images courtesy of Anastasia over at thetvshows.us

9. Beware the 10x developer

Remember the very senior dev we mentioned earlier on? To my mind, he’s the embodiment of a “10x developer” – and I don’t mean that as a compliment. Although undeniably highly intelligent, he sees himself as above the rest of the team, hoards knowledge, and acts unilaterally. He’s a maverick in all the wrong ways.

Many of the points raised in this article can stem from (are at least be exacerbated by) this sort of behaviour. Having a single senior developer with this much knowledge and influence can quickly lead to an infantilization of their colleagues if they have the wrong mindset. This can lead to the decline of critical thinking and the decay of healthy processes. Code review ceases to apply to the senior dev. No-one else understands the system well enough to test it effectively. Corners get cut and teams become siloed. Intelligent, collaborative software development gets reduced to the performative following of process.

Worst of all, 10x developers and unmanageably complex systems seem to be symbiotic. 10x developers often favour their own baroque style of coding, which can be impenetrable to others. This can lead to a vicious circle of an increasingly complex system, understood by an ever-shrinking ring of developers.

So, the obvious thing to do is to avoid hiring 10x developers. The problem with that is that 10x developers are rarely hired – many 10x developers are born in the crucible of bad culture. This means that the advice here is more nuanced than simply “don’t hire 10 developers”. Instead, the advice is to be vigilant for the red flags that might signal that your team’s culture is souring, that your application is becoming unmaintainable, or that one of your developers is on the road to 10x.

Finally

So there we have it, 9 cautionary lessons I think we can all learn from the ill-fated Westworld team. I really hope you found them useful and maybe gave you something to think about in the context of your own team.

Bizarrely, Westworld was one of the most relatable representations of professional software development I’ve seen on screen in a long time. Even though the premise is so fantastical, the cultural and organisational problems exhibited by the team are very realistic.

Hopefully, your team isn’t in as bad a place as the Westworld team, but it pays to be vigilant for the sorts of behaviour that can lead to problems. From time to time, it helps to step back and reflect on what makes a good team, and whether your team exemplifies this.

After starting to write this article, I found that there are already versions for DevOps and QA. There is some overlap with this post, but I hope I’ve given a useful developer perspective. Since one of the main points is to collaborate with allied teams, understanding what lessons our DevOps and QA colleagues took from the show can be really informative – go check them out!

Think I’ve missed something? Leave me a comment below.

Tools of the trade – COVID-19 edition

Tom Wright — Mon, 04 May 2020 21:51:45 +0000

Back in 2017, I wrote about the tools I use at work and why good tools are important. I wrote about three bits of kit that back then and I still use two of them. In this post, I’ll introduce three new tools: noise-cancelling headphones, a USB-C monitor, and a standing desk.

In these turbulent COVID-19 times, it’s reassuring to find some stability. Three years is a long time in technology, but the post I wrote back in 2017 is still (IMHO) pretty darn relevant. I still believe in the importance of investing in the tools we use for 40+ hours a week. A carpenter would not put up with blunt chisels – nor should we put up with bad keyboards.

NB: Just like the original post, I’ll be using Amazon affiliate links in this article. They won’t mean you pay more, but I will get a bit of a kickback if you do make a purchase. If this offends you, the products are also very Google-able…

Like the article in which they featured, two of my favoured peripherals are still in active service. I’m still very much in love with both my Filco Majestouch-2 keyboard and my Logitech MX Master mouse. There may be newer versions available, but I have no reason to go looking – both these items are still fantastic to use.

Of course, Coronavirus has turned all our lives upside-down. Like a lot of people, I’m now working from home full-time. Since the birth of my second child, I no longer have a spare room. Instead, the baby and I are using her room in shifts – she sleeps there in the night and I work there in the day. Why am I mentioning this? Because my recommendations in this post are strongly influenced by my current situation!

My headphones: Sony MDR-ZX770BN

Yes, OK, I admit it – I’m a big fan of Sony’s headphones. I only got rid of my last pair (Sony MDRRF865RK) when they died. For me, Sony is bang on the mark when it comes to the trade-offs between comfort, audio quality, and price.

I bought the MDR-ZX770BNs (side note: why do Sony insist on such awful names?) back in 2018 and they have travelled with my to Malaysia, Australia and back to the UK. The noise-cancelling does a great job of eliminating engine noise – a godsend on long flights.

Of course, no-one is flying right now. Perhaps a more topical example would be how competent they are at cancelling out the wails of my infant and toddler office-mates.

Insert head here.

Besides the audio quality and noise-cancelling abilities, there are a few more things I really like about these headphones. For starters, they have excellent battery life. Secondly, the controls on the ear-pieces are very intuitive. Finally, they are really, really comfortable.

If you’re in the market for some new cans, you can get Sony MDR-ZX770BN on Amazon.

My monitor: Dell Ultrasharp U2419H

For this one, I have to thank my colleague Sneha. I’ve never been fussy about monitors previously, but this one has changed my outlook.

Despite being firmly mid-range in terms of price and size, I think this monitor packs a lot of punch. On top of the excellent image quality (apparently the product of great colour reproduction and pretty good brightness), there are a couple of little features that I like more than I’d have anticipated.

Mountain scene not included.

For starters, the stand is adjustable in all the right ways. There’s a decent amount of range and it’s easy to use. No more hunching over a badly positioned screen. No more stacking books on the desk to gain height. Good work Dell!

Secondly, I love the fact I can use my monitor as a docking station. By supporting USB-C, it becomes more than just a display; it becomes my charger and my USB hub as well. This means the cable from the monitor is the only one I ever plug into my laptop. My mouse, keyboard, etc. all plug into the monitor instead. When I want to move away from my desk, it’s just a case of unplugging one cable. Sounds like a small thing, but I love it.

If you’re in the market for a new monitor, the Dell Ultrasharp U2419H is available on Amazon.

My desk: Jarvis Adjustable

One of my favourite things I miss most about my old office in Melbourne (aside from the people obviously!) is the ubiquitous standing desks. Given the choice, I will preferentially spend 50%+ of my time stood up at my desk. When I have the option to stand, I find I get less restless and am generally more comfortable.

Before COVID, our office space had plenty of opportunities to stand. These were all in communal areas, so couldn’t be my main desk, but their existence made me reluctant to invest in an adjustable desk for myself. Since working from home, the algebra has altered. I finally caved and bought a Jarvis adjustable desk for my birthday.

The Jarvis desk is a motorized adjustable desk. With the push of a button, it will rise up from a sitting position to a standing position. With another push of a button, it’ll go back down again. I opted for the controller with memory, so I get four customisable heights to choose from, or I can use the up/down controls as needed.

Yes, of course my home office is that tidy! What are you implying?

I’ll admit that it was somewhat expensive (at ca. £650 for mine), but I have zero regrets. As you’d expect at this price, I was able to choose all the details and I am very pleased with the result. It looks great, there’s plenty of room, and I’m 100% more comfortable than when I was sat at the dining table.

The team at Fully should be commended on their excellent customer service too. I initially ordered the bamboo top but was informed that COVID-related delays would mean I’d be waiting a while. Over a chat, I was guided towards a combination that was in stock, and would, therefore, be quicker to ship. (Important for someone as impatient as me!)

Fully don’t sell on Amazon, presumably because they wouldn’t be able to support the same degree of customisation as they do on their website. That said, if you’re thinking about a standing desk, do go and check them out.

That’s all folks

There you have it, three new tools of the trade on top of the two that have survived from last time.

As professional developers, we already spent a lot of our time sat at desks, hammering at keys and pushing mice around. Now that I’m working from home, I have new challenges to contend with and new opportunities as well.

In the age of COVID-19, many people are struggling to make ends meet – I’m very lucky to be in a position where my biggest adjustment has been to work from home. Part of me feels a bit self-conscious to be showing off my purchases at this time, but I do strongly believe that these tools are important.

If you’re lucky enough to be working at the moment, give serious consideration to the tools you’re using. Whether you realise it or not, your choice of tools will determine your productivity and comfort.

Prefer some other kit? Let me know in the comments!

Aggressively tuning Cosmos DB (the long way round)

Tom Wright — Sat, 29 Jun 2019 12:11:53 +0000

How many ways are there to tune Cosmos DB? In our (eventually pretty desperate) attempts to coax it into scaling nicely, we tried 9 approaches. As they became more arcane, we saw diminishing returns. Some were flat-out counter-productive! The eventual solution in our case was a single-line fix (how embarrassing!), but the journey was really interesting. It’s this journey that I want to share today.

As regular readers will know, I work for HeadUp Labs. We make a health app that applies data science to our user’s data in order to help them understand their bodies. We use a range of Azure technologies to ingest data from wearable devices and apply data science at scale.

Since launching the app back in October, we’ve rapidly scaled to reach many tens of thousands of users. On one hand, scaling at this rate has validated a lot of the strategic decisions we took early on about the technologies to use: on the whole we’re very pleased with Azure Functions and Cosmos DB. On the other hand, our rapid intake of users has been unforgiving when it comes to the details of our implementation.

One of the main battles we’ve had is with overloading Cosmos DB. Since the early days, we’ve found that provisioning enough “Request Units” (“RUs”) for our peak times has been very expensive. On the flip side, failing to provision enough RUs means that requests get throttled and we start to see the dreaded 429 status code in our logs.

As we scaled, we found that there were periods when we were managing to completely swamp the database. During these periods, the number of requests eliciting a 429 status (RequestRateTooLarge) could dwarf the number of successful (200, 201) requests.

We eventually realised that our problems were primarily caused by a really simple misconfiguration. (Skip to the end if you want the skinny on this issue and the fix.) The solution isn’t really the interesting bit though. The reason I wanted to write this blog post was to talk about the things we did before we worked out what the big issue was. The journey we took and the techniques we tried are the interesting topics here. I hope you’re sitting comfortably…

Spreading the load

The first way we tried to tame Cosmos was a simple, but highly effective architectural approach. Many of our jobs are driven by timer-triggered Azure functions. For instance, we might run a certain data model for each of our users once a day. By staggering the execution of each task, we are able to perform the same amount of work with a much lower RU ceiling.

Instead of triggering 10,000 jobs at midnight UTC, we would instead place that many items on an Azure Storage Queue, each with a different VisibilityTimeout. This property prevents the item from being dequeued until the specified amount of time has elapsed.

Depending on the type of task, we use a variety of strategies to determine the amount of delay for each item. A common pattern is:

Use the user’s timezone to offset the task by a whole number of hours, such that the task will always happen at roughly the same time of day, local to the user.
Apply an offset within the hour based on the user’s cohort (an arbitrary designation), which might result in groups being spaced 10 minutes apart.
Within each cohort band, apply a random number of seconds, so that the users are distributed approximately evenly throughout the 10 minute period.

The overall effect is to smooth a massive daily spike into a smaller, more constant stream of work. Since we need to provision Cosmos to be able to handle the biggest spike, this strategy results in a big cost saving.

Implementing index documents

Another problem of our initial naive implementation was around complicated query logic. For each user, we may have a large collection of “ratings”, relating to different aspects of their health. We maintain several categories and each rating may have a different status. A common query we might make would be to find the most recent Rating document for each combination of category and status. This sort of query quickly became very expensive as both the number of users and the number of ratings per user grew.

We were able to buy some time by switching our query from using a text-based representation of a DateTime to the inbuilt _ts property, which is a numeric representation of the time the document was last changed.

Eventually, though, we needed to rethink our approach altogether. The route we ended up taking was to introduce a new type of document: a RatingsIndex. This document is basically a grouped collection of IDs of Rating documents elsewhere in Cosmos. We update this document whenever we change a rating, and query it to work out which rating to use.

This might seem counter-intuitive. After all, we’ve replaced one query with two. The trick lies in the fact that we were actually replacing one expensive query with a combination of a very cheap query and read. Read operations are more like a call to the filesystem than a database query. They are insanely efficient.

We’ve effectively shifted the logic of selecting ratings from read-time to write-time. Since we read a lot more than we write, we’ve found this to be highly beneficial. The cost of the combination of an extra write (when we update the index) and extra read (after we’ve queried the index) is more than offset by the gains we made by removing an expensive and frequently-called query.

Shielding Cosmos with Redis

Even after spreading the load, our growth meant that costs kept rising. As with most applications, we read a lot more than we write, so we decided to expand our Cosmos implementation.

I say “expand” because we were already using Redis to cache database calls made from our API project. As is commonplace, we were using a layered design for our ASP.NET-based API. Having all our database logic centralised in our repository layer made it very straightforward to weave in caching as a defensive layer.

The problem was that our API is actually pretty thin – most of the heavy lifting is performed by our (numerous) Azure Functions. The idiomatic way to access a database in a Function is by accepting a client via a “binding” parameter. The example below is lifted straight from the documentation:

[FunctionName("DocsByUsingDocumentClient")]
public static async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)]HttpRequest req,
    [CosmosDB(databaseName: "ToDoItems", collectionName: "Items", ConnectionStringSetting = "CosmosDBConnection")] DocumentClient client,
    ILogger log)
{
    // do stuff with client
}

We had dozens of functions written in this form. Despite having caching logic implemented in repositories sitting in our API project, we had no real way of using them here. Re-implementing the caching in the Functions would have meant a massive amount of duplicate code, as well as being error-prone and hard to debug.

Our luck changed when we discovered an unofficial implementation of AutoFac for Azure Functions: AzureFunctions.Autofac. Suddenly we could inject dependencies into our Functions. Indeed, our API project was already using AutoFac for DI, so it was extremely simple to start sharing the services and repositories and remove all the duplicated functionality. Much code was deleted that day.

By removing all direct bindings to Cosmos from our Azure Functions and replacing them with cache-aware repository calls, we were able to slash the number of requests making their way to the database.

But still, they came…

For a while, things were looking a lot better. We were able to grow our user-base without unduly cranking up the provision of RUs or succumbing to an unacceptable number of throttled requests (429s).

Then we started to see some unpleasant symptoms during peak hours. The number of requests would rise beyond capacity and stay there for an hour or two. During this time, we would see higher volumes of RequestRateTooLarge exceptions and 429 status codes.

Those purple hills are definitely downers.

These events became more frequent and more significant over time. Worse, we were beginning to see the application misbehave as a result. Jobs wouldn’t finish. Data would end up in strange states. You get the idea… Well, as you might imagine, the drive to find a solution became more and more urgent.

Tweaking retries

One idea we latched onto early on was that somehow it was the combination of using Storage Queues, Azure Functions and the .NET Cosmos DB SDK. Both queues and the database will attempt to retry a failed operation, you see.

If the database is unable to handle a query, for instance, the SDK will back off and try again. It will do this a set number of times before throwing an exception. This exception will cause the Function to fail, at which point its claim on the queue item will lapse, meaning that another instance of the Function is free to swoop in and try again.

The implication is that a single queue item may result in many times more requests to the database than might be expected. According to this helpful doc, the default value is 9 retries for the Cosmos SDK and 3 for Storage Queues. In other words, a single item on our queue could cause 27 database hits. Could this be why we were seeing such severe failure rates?

If true, this implied two things. Firstly, it suggests that raising the provisioned throughput by a little bit might have a big effect on the failure rate, as each additional success might prevent 26 further requests. Secondly, it prompted us to think about how we might want retries to behave – do all operations really need 27 chances to succeed?

Encouraged to bump up throughput a little bit, we also reduced the number of retries for our backend application. (We left the API project alone, figuring that the impact of failure there was likely to be greater.) The reasoning was something along the lines of “eventual consistency”.

As you might have guessed, this was not a great strategy. Not only were the rates of 429s barely affected, but we found that the system managed to get itself into some pretty strange states, as an increasing number of operations failed hard. We reverted this change.

Logging Cosmos DB transactions

One of the key lessons from our adventures with our failed experimentation with retries was that we had exhausted all the normal ideas and were now flying blind. Accordingly, we started to look at the available data with a newfound vigour.

The metrics built into Azure, as well as Application Insights, were able to tell us in broad strokes where the issues lay. We could tell, for instance, that we were spending a higher proportion of our RUs on upserts that we had been previously (the effect of caching, I suppose). Digging deeper isn’t really possible in the Azure Portal, however, and we found ourselves unable to derive anything actionable. We couldn’t confidently point at any particular “hotspot” in our code.

Frustrated, we borrowed a trick from our pasts, from the world of SQL Server… We decided to try profiling our Cosmos DB usage.

Thankfully, our earlier move to centralise database logic into a set of shared repositories paved the way. These repositories all used the same underlying Cosmos DB client class, so it was relatively painless to add some code to log each request. We decided to write the request type, the query, the calling method, and the cost in RUs to a new Azure Storage Table.

After leaving the profiling code enabled for a few days, we were able to hook up Power BI to the table of transactions. The data didn’t quite match the overall patterns we were seeing (the final section makes the reason for this painfully obvious), but we had some strong leads to pursue.

Can’t outsmart the SDK

A few of the things we tried after looking at the profile data yielded no fruit. I’ll put these all together in a class of bittersweet realisations. Bitter, because we didn’t see the improvements we’d been hoping for. Sweet because we realised that the .NET SDK for Cosmos was already smarter than we gave it credit for. I won’t dwell on these, but I’ll give an example.

We had left an inappropriate setting in place. Deep in the bowels of our implementation, we had the following:

EnableCrossPartitionQuery = true

So we had the thought that, perhaps, maybe each query was incurring RU costs once for each partition. Given that our database now spanned many partitions, it seemed as though this might explain the explosion is RU consumption.

It was a simple thing to try. For all the queries where we had access to the partition key (which, thankfully, is most of them), we disabled EnableCrossPartitionQuery and explicitly set the PartitionKey within the FeedOptions object.

The result? No change whatsoever. The SDK had clearly been working it’s magic away from prying eyes; optimising our queries without us even knowing. Whilst, on the one hand, it’s great to know that the SDK is so smart, on the other hand, our search for a solution would have to continue.

Beware the update!

One thing that we learnt from our profiling efforts was the seemingly disproportionate cost of an update operation. As Maxime Rouiller points out in a blog post on the subject of calculating RU costs, an update is effectively a delete followed by a create. This results in an RU cost 10 times greater than would be incurred for reading the same document.

Looking at the output of our profiling clearly showed some “hot spots” caused by updates. In the most significant case, we were frequently updating a single property in a relatively large document. Since the cost in RUs is proportional to the size of the document, we decided to split this document into two: one small document containing just the properties we often need to update and one large document with the rest.

Having to read two documents does come at a cost. Each of the two documents will both have at least a standard set of core properties (e.g. _ts, etag), which means you’ll be necessarily be reading more data after splitting your documents. But because of the 10x cost difference between reading and writing, writing to a smaller document results in a big enough saving to overcome the penalty incurred by reading two documents.

We can use the Azure Cosmos Capacity Calculator to estimate the economics of the split. I’ll leave all the general settings at their default values and just change the section relating to our documents. Imagine we have a single big document (like this one; 1.99kb) and we need to read and update 100 documents like this per second. This would require us to provision 734 RU/s – 105 for reads and 629 for writes. Now imagine we took a couple of properties from the big doc (to leave it slightly smaller; 1.94kb) and moved them to a small one (0.47kb). If we read both 100 times per second, but only wrote the smaller one at this frequency, the total throughput drops to 700 RU/s (205 for reads + 495 for writes). This is a toy example, but we can already see a 5% improvement.

	Naive	Split	%
Reads	105	205	+95%
Writes	629	495	-79%
Total	734	700	-5%

In our real-life example, by reducing the size of the document we update most frequently, we were able to shave off close to 15% of the RU cost of one of our most frequent operations.

Pre-emptive scaling

Whilst our efforts to this point had resulted in noticeable gains, we were still facing big peaks and troughs in consumption. The peaks were invariably seeing huge numbers of throttled requests as demand exceeded the throughput we’d provisioned. Keeping the database at a scale that would satisfy this demand looked to be prohibitively expensive.

In order to handle the peaks without paying over the odds during quieter periods, we wrote a Function to automatically scale Cosmos to anticipate demand. We were fortunate that our periods of high demand were fairly predictable, so we could scale up and down on a schedule.

It turns out that altering the number of provisioned RUs per second is possible through the DocumentClient itself, so the code to scale the database up and down is very straightforward:

var coll = cosmosDBClient
    .CreateDocumentCollectionQuery(UriFactory.CreateDatabaseUri(databaseName))
    .Where(c => c.Id == collectionName)
    .AsEnumerable()
    .Single();

var offer = cosmosDBClient
    .CreateOfferQuery()
    .Where(o => o.ResourceLink == coll.SelfLink)
    .AsEnumerable()
    .Single();

var newOffer = new OfferV2(offer, newRUs);
await cosmosDBClient.ReplaceOfferAsync(newOffer);

As you can see, the hard part is creating a new Offer object with the new provision.

We run this from a timer-triggered function which looks at a Storage Table for its schedule. It checks the schedule at quarter-to and quarter-past the hour and scales the database up or down depending on the known patterns of demand.

Granted, this doesn’t solve the problem of higher-than-anticipated RU demand. What it does do is make it significantly cheaper whilst we find a proper solution.

Tuning the index policies

In our increasing desperation to reduce our RU spend, we started to look at Cosmos’ indexing policies. The default and recommended approach is to allow Cosmos to index every property on every document. We’d initially accepted this but were now sceptical of everything. Could it be that our index metadata was now so vast that it was harming performance?

We tried switching away from the default policy of having every property indexed. Instead, we used the data from our profiling exercise to identify the properties we were actually using in our queries and instructed Cosmos to only index these. The motivation behind this was to reduce the cost of write operations.

The documentation seems to support this as a strategy, but the results were not as impressive as we had hoped.

A one-line fix

As we started to run out of ideas – and still no closer to a performant system – we began to despair. In our despair, we started aimlessly combing over our code.

We even started looking at code that we knew worked. We knew it worked because we’d written it months ago and had used it ever since. It hadn’t been modified since long before the start of our current woes. It couldn’t possibly be at fault, but still, we leafed through the files, scrolling hopelessly through ancient methods.

One of the classes we looked at was our Cosmos client singleton. When we’d first started writing code to access Cosmos, we’d followed the performance tips and had implemented our client as a shared singleton. It all looked fine, but we wanted to double check that Autofac was properly resolving it as a singleton, so we decided to step through it.

F11, F11, F11… “Yep, it’s definitely sharing a single instance.”

“But… what is that method doing? And why did it take 750ms to run?” So we ran it again. Same behaviour. Very strange.

Ladies and gentlemen, it turns out that we had left a particular method call in the query method instead of the constructor. Not just any method call either, but a call to the method that proactively fetches all the metadata for the collection in order to speed up each query. Given the size of our database, these metadata requests were now taking hundreds of milliseconds and costing us a large number of our provisioned RU/s.

The fix was comically simple in the end. We moved OpenAsync(); out from the method performing the actions/queries and into the method that instantiated the singleton.

As soon as we deployed this change, the plateaus of huge numbers of 429 responses went away. Throughput dropped so much that we were able to reduce the number of RU/s we were provisioning to 20% of what it had been. (In other words, this fix instantly slashed our database bill by 80%.)

Everyone was delighted. Costs were way down, exceptions were non-existent, and overnight the app started behaving as it was designed to.

Reflections

It’s been a few weeks since we wrapped this up, but I’ve been thinking about writing this blog post and thinking about a whole load of related questions.

Chief among these has to be “why did it take us so long to get right?” And I think the answer to that is nuanced and interesting. Fundamentally, it breaks down into the following factors:

Code consolidation. The problematic class was only used by our API at first. We started using it in our Functions project as we started injecting dependencies to our repositories. In other words, we spread the problem as part of a fix.
A growing problem. OpenAsync is a bit of a black box (the docs say very little about what it’s doing under the hood), but we think it pulls a dictionary of partition keys and partitions. This would make sense, as it would allow the SDK to make most queries lightning fast. This would also explain why we didn’t notice this issue when the database was so much smaller (or on our non-prod DBs). The upshot is that the problem was literally growing in parallel with the database.
Lack of documentation. In a similar vein, what is a “ReadFeed” operation? We could see these in our Azure metrics. They were clearly something to do with our problems, but our Googling didn’t answer our questions We now know that OpenAsync performs a ReadFeed operation, but we didn’t manage to make this connection until afterwards.
Partial logging. Remember when I said that the data from our profiling work didn’t quite match what we were seeing in Azure? Well, we hadn’t considered that OpenAsync might be costing RUs. We were only tracking the costs of our queries, upserts etc. With our imperfect data, we were able to target some sensible incremental optimisations, but missed the fundamental reason that the system was misbehaving.

Underpinning all this is, I suppose, a good dollop of “new tech” an “inexperience”. With hindsight, we had already provisioned more throughput than should have needed – we just weren’t familiar enough with the technology for those alarm bells to ring.

On the other hand, I can’t us ever learning some of the things we’ve learnt about Cosmos DB by any other route. Sometimes necessity is the best instructor.

As well as the knowledge we’ve gained, we’ve kept a lot of the changes we made:

We still use a variety of techniques to spread load and smooth out spikes. In fact, this is a central part of our architectural thinking now. Regardless of any issues in our code, the availability model of Cosmos is suited more to a baseload than spikes.
Index documents are something we’ve not just kept, but extended. For documents that are frequently read, but infrequently written, it’s just a lot more efficient. We’ve rolled this technique out to a few other document types.
Injecting repositories with caching logic was still a smart move, even if it initially spread a defect in our DB client. We still use repositories throughout our application to encapsulate all the logic around caching and database access.
We’ve left in the code that logs calls to Cosmos, so we can do profiling of new subsystems in the future. All we need to do it flip a flag in our config and all database calls get logged, along with details of the responsible repository method.
We’ve retained the ability to scale the database on a schedule. Even with everything else we’ve fixed/implemented, we still have peak and off-peak times. This is mainly due to us having biases in the geographic spread of our users and the desire to have certain actions run during certain times of the day in users local timezones. So we’ve kept the code that scales our provisioned throughput up and down throughout the day.

So, in a lot of ways, the problems we’ve encountered have made us wiser and driven us to create tools with ongoing utility.

Conclusion

The last couple of weeks prior to us finding the fix were pretty unpleasant. The team were feeling the pressure and doubt was creeping in around whether Cosmos was even the right tool for the job.

Now that we’ve emerged through to the other side, the change is dramatic. The whole system now behaves as it was designed to and we’re paying much less than we were before. It’s safe to say that any doubt we had about Cosmos has now evaporated.

We tried a lot of tactical solutions for a problem that was, in the end, just far more fundamental. Along the way we learnt loads and built some cool tools, so ’twas not all for nought.

I decided to publish a list of the things we’d done when we were still deep in the woods. I didn’t guess then that the answer would be as simple as moving one method call. I imagined that I’d be writing a definitive playbook for tuning Cosmos. When we stumbled on our solution, my first thought was to dismiss all the other things we’d tried.

Ultimately though, it was the journey that was interesting to me. I hope, having read the full post, that you’ll agree.

Reticulating splines…

Tom Wright — Sun, 28 Apr 2019 10:56:25 +0000

It’s been a little while since my last post, so I thought I’d better do a quick status check. What have I been up to recently? What on earth could possibly keep me so busy that I’ve been unable to blog? It’s mainly a mix of work-stuff and a big project. Read on to find out more about both, as well as a few other bits and pieces, and also maybe some teasers about future posts.

Changes at work

In March I was asked to step up and start managing the development team at the startup where I work.

One of the big things that’s kept me from writing more blog posts is that I’ve been spending a lot of time doing managery things: hiring, onboarding, planning, forming processes… You get the idea. It’s also meant that I’ve backed off from the interesting tech stuff a little bit. So I’ve had less to write about and less time to do my writing in.

“But Tom,” I hear you say, “when you left your last job (and the one before that) you said you were sick of being a manager and that you just wanted to write code.” Yep. That was me. And my own words rang in my ears when I was talking to my boss about the changes. So why am I doing it all again?

Well, for a start, I’m finally at an organisation where I feel I can develop my own management style. I can take what I’ve learnt from being a manager in two companies previously and apply it to the fertile soil of a startup packed with talented people.

The other factor is more inward-looking. I’m at peace now with the “manager” part of my identity. I guess I’ve mature enough to feel comfortable acknowledging that I’m pretty good with people. Plus, I’m OK with the process side of things.

Anyway, I’m still more software engineer than manager, as I hope the rest of this post will prove.

ServerlessDays Melbourne

This is a big bit of news. I’m organising a “ServerlessDays” conference here in Melbourne in August.

ServerlessDays Melbourne 2019 is taking place at the Melbourne Museum on the 29th of August. The CFP is open and we’ve had some cracking submissions already. We have secured some great sponsors, but still have some slots left (download our prospectus). Tickets will go on sale any day now, so keep an eye on our Twitter feed.

ConTabs and dotnet whois

ConTabs has been a victim of my busy schedule recently. We’re pretty close to being ready for the v2 release, but just have one big feature left to tackle.

As well as v2, I have a post brewing about moving to .Net Standard 2.0. The short version is that this makes the dependency stuff (that caused me so much pain before) loads easier.

I’ve also started a project I’m calling dotnet whois. It’s sorta like business cards for the dotnet CLI. It’s pretty early days, but take a look at the Github repo and dive in if you’re interested.

Future posts

I’m going to make a concerted effort to blog more. In the not-too-distant future, I hope to publish posts about…

My interview process
How I can’t get any women to apply for my jobs
All the stuff involved in organising a conference
ConTabs v2 (when I finally finish it)
Moving ConTabs to .net standard 2.0
Dotnet whois and why y’all should fork it

Anyway, that’s the plan. Stay tuned kiddos.