DEV Community: Mario Fernández

Book Review: Software Engineering at Google

Mario Fernández — Sun, 10 Dec 2023 00:00:00 +0000

Software Engineering at Google is, unsurprisingly, a book about how Google engineers develop software.

For years, Google has captivated the curiosity of software engineers. After all, they built things like Bigtable, Spanner, and many others.

However, this book delves behind the scenes, emphasizing three key areas:

Processes
Tools
Culture

Let’s cover each, as I think some parts are more insightful than others.

Remember that You Aren’t Google

Keep in mind one thing while reading this book: Most companies don’t have the problems, the scale, or the resources that Google has. For most people, many of the solutions presented in this book won’t apply.

I’ve only realized this after I joined a BigTech company. It’s hard to picture the difference in scale until you see it firsthand. For instance, I find stacked diffs quite useful. However, I think they work in a very specific environment, and I’ve got ex-colleagues who just couldn’t imagine that being useful. And that’s fine.

Anyways, back to the book.

Processes: Making it Work At Scale

Google is renowned for its massive mono repo. What you do can affect a lot of other engineers, thus they developed a strong culture around reviews. They have systems around who can review which technology or aspect that’s quite interesting. In a past gig, they copied the system with terrible results. Beware of copying anything if you don’t have the necessary resources.

Another unique area is testing. At this size, running tests becomes kind of a probabilistic exercise. Any test that can be flaky will become so if it runs millions of times. Google is also famous for having popularized the SRE role. However, operations aren’t covered in this book that much.

Tools: The Secret Weapon

Tooling is my favorite part of this book. Developer tools that seamlessly work are extremely satisfying. Also, this is the last thing a small company should try to copy. Internal tools are challenging to nail, and the incentives skew against them.

Moreover, it’s an existential thing at a certain scale. A mono repo as gigantic as Google’s needs top-notch tooling. Otherwise, you can’t even browse files properly. The same goes for navigating code. You need a supercharged grep that works quickly and reliably across the whole repo.

I really like the chapter that touches on large-scale changes. A change across hundreds or thousands of files feels magical when it actually works.

Interestingly, CI/CD is pretty vanilla at Google. Granted, it seamlessly works with much bigger volumes of changes. Other than that, it doesn’t do anything out of the ordinary. Check this old article I wrote for comparison.

Culture: Have the Right People

Culture is the weakest part of the whole book. It left me pretty cold. My theory is that BigTech has such a huge pool of candidates willing to join that it can rely on a pretty decent average for engineers. Some of the agile methodologies seem to make less sense in this context.

Reading through these chapters didn’t feel too insightful. I’d rather check Team Topologies, Accelerate, or plenty of other books.

Verdict

A few years ago, reading about Google practices could feel like science fiction. Nowadays, cloud providers and a host of SaaS tools for observability, feature toggles, and whatnot have made this tooling accessible. It doesn’t feel as foreign as it once did, although we’re still talking about a completely different game in terms of scale.

Nevertheless, it’s an enjoyable read. You get a reasonable idea of how BigTech works internally, in case you want to get in. The book gets ⭐⭐⭐⭐ stars from me.

My Approach to Taking Notes

Mario Fernández — Mon, 11 Sep 2023 00:00:00 +0000

Taking notes is kind of like flossing but for the corporate world. It’s essential yet frequently overlooked. We all know we should do it more, but we let other things get in the way.

Good notes are helpful in pretty much every possible situation. Let’s say you want to give feedback.

You can wing it and hope you somehow remember what happened months ago. Or you can rely on your carefully collected notes and make your life way easier in the process. I know I prefer the second approach. Let me share how I go about it.

When To Take Notes

I’m going to start with when to take notes. I think the cadence is the most important aspect of this whole endeavor. Yes, more important than what you write or where you maintain your notes. Consistency beats any fancy system you can think of.

If it were so easy, though, everybody would be doing it already. The challenge lies in finding moments to take notes amidst a busy day. Over the years, I’ve figured out the specific times that work best for me. By using these opportunities to collect as many notes as possible, I’ll have an easier time once I sit down to think about the feedback I want to give.

During Meetings

It never looks so clean in reality

Meetings offer an excellent opportunity to observe how others present their ideas. It’s not just about the content. You can focus on what they ask, how they contribute, or how they explain their arguments. Small improvements in your communication increase your effectiveness in spreading your ideas. Thus, you can find plenty of suggestions to share with them.

However, this isn’t easy to do! I struggle to take notes during meetings, more so if I’m actively participating. So, I limit myself to very short notes. Otherwise, I tend to get lost and miss the actual content of the meeting. That’s not what you want.

Shortly After the Meeting

Taking notes straight after the meeting works better for me, while the information is fresh in my mind. It’s the perfect time to write it down and keep it for later.

What if you have overlapping meetings without a break? I try to squeeze some quick note-taking during the short breaks in between them. It can make your day stressful as you sacrifice your breathing time. It’s not something to abuse. Even then, it’s so effective that it’s worth it for me.

At the End of the Week

Before signing off for the weekend, I like to reflect on the week’s events. I check places like my calendar, open documents, or anywhere else where I can find the stuff I was involved with for the past few days. I discover a lot of things to share that way, including some that seemed unimportant at the moment.

It’s also a convenient time to balance between positive and critical points. I sometimes drift too much towards negativity, but I’ve found that having the weekend in front of me puts me in the mood to counteract it 😎.

At the End of the Month/Quarter

This moment is the scaled-up version of the previous point. It’s also the one that many people are probably doing already, as it sort of aligns with retrospectives, quarter summaries, or similar. Still, it’s another good moment to look back and check if anything’s missing.

How I Take Notes

Efficiency is crucial when taking notes between other tasks. I optimize for speed and volume.

I create separate notes for individual topics or people. I aim to write the shortest thing I can get away with, in chronological order and with timestamps. For future feedback, I keep context minimal. Instead, I limit myself to observed actions or behaviors. I may leave out the impact, as that’s something I can add later based on the facts.

This note is quite raw and not meant for sharing. I leave enough of a trail to trace it back to the source. This is a lightly edited example from a previous gig:

06.10

- update in the standup: what do you want to convey? what’s the level of details?
-- think of a structure: what happened, what’s coming up, what blocks

13.01

- lack of sync before the workshop, we had two conflicting messages

27.01

- kudos for fixing the CG dependency

23.02

- didn't leave an update before being off, was a bit confusing

05.04

- seemed like you were jumping straight into fixing the bug, without quantifying it first
-- could be taken as lack of prioritization
- what's next

Sometimes, I outsmart myself with some garbled notes so cryptic that I can’t decipher them afterward. That’s far from ideal. However, I can’t commit to writing tidy notes all the time. The occasional failure is an acceptable trade-off for me.

The Logistics of Taking Notes

I wouldn’t be too concerned about optimizing logistics as long as your system fits your workflow.

Choosing between digital and handwritten notes is a personal preference. I prefer digital notes for easy access across devices, but sometimes, I resort to pen and paper for quick notes. Regardless of the format, ensure you can find your notes later. Otherwise, all the effort gets wasted.

There are other points of view. For instance, this recent article from Werner Vogels is full of praise for handwritten notes. If you want to be a CTO someday, maybe it’s worth copying that approach instead.

Be Mindful of Where You Put Things

Trying every software ever made

It goes without saying, but you should only store notes where it’s allowed. I try very hard to avoid proprietary details in my notes. That stuff is better suited for official documents, wikis, or similar. Your notes can contain a reference that links to it.

Feedback notes focus on behaviors. It’s relatively easy to write them in a way that you only talk about an individual’s actions and leave any unwanted internal detail. It’s worth getting used to writing things that are safe to share, in case you do it inadvertently.

As for concrete software, there are a ton of solutions to choose from. I use Apple Notes for my personal stuff. I got notes about feedback I received or gave that go five or six years back there.

When to Make Notes Public

As a general principle, you should consider making public anything that could help somebody else. This post, for instance, started as a direct response, but I thought it might benefit others 😊.

Feedback isn’t the best example, as it’s something to share privately. For less personal topics, it’s good to ask yourself if somebody else would read it. Defaulting to sharing leverages what you already did. You can be more impactful with low extra effort. For instance, while migrating one service to the cloud, I had a lot of notes to understand AWS permissions in more detail. At some point, I realized I could show it to others, so I converted it into a post about IAM. I shared it broadly and even reused it on different projects.

You can see that it’s not completely true that making it public is for free. To publish a private note you need to edit it more heavily than when you’re the only consumer.

Notes for Everything

While I’ve focused on feedback throughout this article, it’s not the only thing that benefits from note-taking. The possibilities are endless, even something like learning in public.

A long time ago, I read that John Carmack tracked his work in .plan files. I liked the idea so much that I adopted it, although I keep mine private.

Every now and then, I write down what I’ve been doing lately. I put a timestamp and document what happened. What I’m focusing on, what went well, anything that comes to mind. It’s insightful to go back and see my progress. It gives me perspective on how things are going without too much recency bias. That’s helped me a lot in tracking my growth.

Start Taking Notes

The TLDR is that taking notes is inexpensive if you organize yourself. Moreover, it’s an extremely beneficial investment of your time. If you aren’t doing it now, you have to start. If you do, do it more often.

Thanks to Anna for the feedback.

A Method to Give Feedback

Mario Fernández — Mon, 07 Aug 2023 00:00:00 +0000

Feedback, when used thoughtfully, is a superpower. It accelerates personal growth faster than anything you can achieve through introspection alone.

Over time, I’ve witnessed developers hinder their careers by repeatedly making the same fundamental mistake. Imagine influencing somebody’s trajectory by simply giving good feedback. That screams Senior Engineer to me.

Why is it, then, that so many people wait until the last day before the performance review submission deadline to finally write down something that’s way too superficial to be useful? I don’t know, man. Maybe it’s for lack of a method? In this article, I’m talking about my approach to delivering feedback.

What Constitutes Good Feedback?

There are two aspects of giving feedback:

The content: The observation you’re attempting to share
The delivery: How you share it with the intended recipient

Let’s begin with the content. You don’t need deep, career-changing insights to be helpful. More mundane observations are valuable and easier to come by. For instance, I tend to speak over others in meetings. While it might sound like a small thing to point out, receiving that feedback has helped me become more aware of it.

Be sure to refer to concrete events. Be as precise as possible to help the other person understand what you’re talking about. For the same reason, talk about recent events. Memory gets hazy over time. It’s significantly easier to discuss something when it’s fresh on your mind. At the same time, it’s probably good to avoid giving feedback immediately after the fact. Waiting a bit helps cool down.

actionable insights

Next, your input should be actionable. What’s there to do differently next time? A situation with no realistic alternative is a dead end. Feedback is about finding ways to improve yourself, not grading your performance.

Lastly, it should be constructive. You suck isn’t feedback, even if you phrase it nicely. If you can’t find a way to frame it so that it helps the person grow, it’s probably better to let it go.

As an example of what not to do, I remember a manager in a previous company telling me something akin to Why do you repeat yourself so much?, in a way that sounded more like an accusation than a question. I actually went and asked a few colleagues if they had observed this, as I was taken aback. That feedback didn’t land well at all.

In summary, keep the following in mind when preparing the content:

It should be recent
It should be concrete
It should be actionable
It should be constructive

Situation-Behavior-Impact

Let’s say you have the content ready. The next step is figuring out the delivery. Giving feedback can be difficult, especially if it’s not positive. That’s why I like to stick to a structured pattern commonly referred to as Situation-Behavior-Impact. You have three distinct phases:

Situation describes the context, so that the receiver knows what you’re talking about. This ties back to using a concrete example. Moreover, it being a recent situation means both of you are more likely to remember the details, avoiding misunderstandings. Your goal should be to set up the stage, not to bore the other person. After all, they were there as well. Make it as short as you can.

Behavior is about describing the actions you observed. Key word being observed. Acting as if you know exactly what somebody else was thinking is condescending and counterproductive. You’ll probably be wrong, and it won’t land very well.

Instead of reading minds, focus on your observations. Make it clear that this is your perspective, which might be incomplete or even incorrect. Use a neutral tone and avoid making judgments. Your goal is to talk about actions.

Impact is the last part of the puzzle. What did the previous behavior cause, on you or somebody else? Talking about impact can be uncomfortable, but it’s a necessary part of a complete piece of feedback. Based on this impact, the receiver can reflect and decide to do things differently the next time.

Similarly to the behavior part, focus on what you saw. Using a fixed structure can feel forced, but it ensures consistency. When you’re raising a thorny issue, this formula helps you concentrate on being as objective as possible.

Landing some impact

Another example: I used to work in consulting. In my first project, we developed some CI/CD infrastructure for a traditional German company. I was a bit overwhelmed as I wasn’t familiar with the tooling. A senior colleague gave me some harsh feedback about me being hesitant to make decisions, which was diminishing my credibility with the client. It wasn’t easy to hear, I can tell you that much.

However, my colleague was structured in how he delivered the feedback. He was concrete about the meeting he was referring to. He described the actions he was seeing from me. Lastly, while he was clear about the impact it was having, he wasn’t judgemental. On the contrary, he was keen to support me. The structure made this tough message more digestible. Building a plan to address the issues felt a lot easier.

Practice!

Sharing imperfect feedback is preferable to sharing nothing, as long as you remain respectful and constructive. Remember that any formula isn’t a substitute for practice. The only way to get better is to do it regularly. Don’t leave it for the last possible moment. Take notes and share them regularly, and it’ll progressively get easier over time.

I prefer giving feedback in person. I always thought that what you write for an official review shouldn’t surprise the receiver. However, if you feel more comfortable over chat or email, use that medium. Remove the barriers that are preventing you from doing it more often.

Another option is to go to the person’s manager directly and share the feedback with them. Honestly, that sounds very foreign to me, so I have a hard time picturing myself doing it. I believe that building trust with others to a point where you can freely share and receive feedback is a better long-term investment for yourself. But, hey, I guess that some feedback is better than just eschewing the practice altogether.

Beware of Rigid Formulas

Even though a structured method like SBI can help, you should be ready to abandon any method and be adaptable instead. Feedback is a very personal thing. People have different preferences regarding format, medium, or anything else. Adapting your delivery to accommodate these preferences will improve effectiveness. Especially so for people with whom you collaborate more closely and have more to share.

For instance, some resources recommend using the feedback sandwich, where you wrap a negative piece of feedback in between two good ones. I despise this method and find it fake and artificial. It just doesn’t work for me. Everybody has their anti-patterns, so it’s a good idea to figure out those over time.

Using a formula allows you to be consistent. However, it’s more important to be genuine.

It’s Not Up to You Anymore

Alright, so your carefully crafted feedback is ready, and you go out and share it with the intended recipient. What’s next?

Not much, really. In my opinion, you did your job already. Now, it’s up to that person to digest it, reflect upon it, and decide how to act on it. It might well be that they choose to discard it. That’s not necessarily bad! Remember that you only have a limited view of how somebody else acts and behaves.

Book Review: Unit Testing

Mario Fernández — Tue, 18 Apr 2023 00:00:00 +0000

Unit Testing is a book that, despite its title, isn’t just about unit testing. While that’s a significant portion of the book, there’s more.

You get a lot of advice on how to write tests. How to make them effective and leverage those tests to improve code through refactorings. It touches on other topics, such as mocks or even integration tests. There’s a lot of good stuff to mention in just a few lines.

Anyhow, two parts stood out the most to me.

Pillars of a Good Test

How do you recognize a good test? As a big proponent of testing, I’ve thought a lot about that through the years. It should be worth the effort has been my intuition for a long time.

The author has more concrete ideas. He provides four pillars for a good unit test:

Protection against regressions
Resistance to refactoring
Fast feedback
Maintainability

They all make sense to me and don’t merit an extended explanation. More interesting is the relationship between them. The first three are related to each other. Resistance to refactoring is rather binary. A test is either resistant to refactoring or it isn’t. Thus, it comes down to balancing between protection against regressions and fast feedback. Coincidentally, that’s the continuum between end-to-end tests and unit tests.

I recommend reading this chapter carefully, as it’s a novel idea that I hadn’t seen before.

Where to Deploy Tests Effectively

What types of code are there? The book lists four types: Domain model/Algorithms, trivial code, overcomplicated code, and controllers. They’re all pretty self-descriptive. And they all have well-defined actions:

Trivial code isn’t worth testing
The domain model is covered with unit tests
Controllers are covered with integration tests
Overcomplicated code shall be refactored by splitting it into domain model and controllers

It makes a lot of sense! Testing is all about return on investment. You have limited resources and have to deploy them wisely. And it turns out, there’s code that’s too easy to test, and there’s code that’s too hard to test. I like the idea of accepting that neither is the target.

A common complaint against testing stems, I believe, from treating all code equally instead of following a guideline like this to maximize impact.

Verdict

Testing is a fundamental part of software engineering. Thankfully, that’s fairly accepted throughout the industry today. However, there’s a lot of variance in the effectiveness of testing efforts.

If you want to increase your testing skills, which you should, this book is one of the best resources I know. It packs a remarkable amount of highly actionable information in fewer than 300 pages. It’s concrete enough to be broadly applicable in many situations. I found it one of the most useful books I’ve read in a while, and I plan to re-read it more than once. It gets ⭐⭐⭐⭐⭐ stars from me.

The Humble Extract Method

Mario Fernández — Mon, 27 Feb 2023 00:00:00 +0000

After reviewing a lot of code in the past couple of months, I’ve realized that I regularly leave comments like this one:

Can we break this down into smaller pieces?

Turns out, I like to advocate for shorter methods. Is this the right thing to pay attention to? I think so. I strongly believe that breaking down code is often the most cost-effective way to increase maintainability.

For that, the best alternative is extracting smaller methods. It’s easy and works consistently. It should be part of every developer’s toolkit.

In this article, I’m writing about the simple yet powerful extract method refactoring.

What Is Extract Method, Exactly?

Extract method is a refactoring where you move a code fragment to a separate function with a name that describes its purpose. Consider this snippet:

fun splitNumbers(source: String): List<Int> {
    val split = source.split(",")

    // convert each string to a number
    val numbers = split.map { str ->
        str.toInt()
    }

    return numbers
}

This example is too small to truly benefit from breaking it down, so bear with me for the sake of the example. Anyway, there’s a section that converts a string to a number. It’s a prime candidate for extraction:

fun splitNumbers(source: String): List<Int> {
    val split = source.split(",")

    val numbers = convertStringToNumber(split)
    return numbers
}

fun convertStringToNumber(list: List<String>): List<Int> {
    return list.map { str ->
        str.toInt()
    }
}

There it is! We now have a proper submethod instead of an inline comment. For a more realistic scenario, imagine this was a hundred lines long instead of five. The good thing is, the approach would work the exact same way. Moreover, you can repeat the procedure and break down the new methods.

All in all, this refactoring is pretty straightforward. In fact, it’s been supported by IDEs like IntelliJ for basically forever. And yet! Despite its simplicity, this humble change brings quite a lot of value. Let’s talk about its benefits.

Shorter Methods Improve Readability

The most obvious outcome of using Extract Method is that it makes methods short. Is that a good thing, though?

Yes! Let me argue why. Shortening methods is necessary to get them to do one thing only (also known as the Single Responsibility Principle). Methods that do one thing are generally less complex. They require less knowledge about the surrounding context.

When you navigate a big codebase, you can’t keep every detail in your head. If you have short, single-purpose methods you can tune out some complexity and focus on one part at a time. That, in my mind, leads to a massive readability boost.

One argument I’ve heard against extracting methods is that it’s easier to understand code if it’s all in the same place. I find the reasoning unsatisfying. In my opinion, it assumes you’ll be able to zoom in and focus on the relevant parts as you read through a possibly massive block. To do that, you need to be pretty familiar with the codebase already. Otherwise, you’re going to struggle.

In my view, a more sustainable alternative is to rely on abstract methods that hide implementation details. That way, even if your understanding is incomplete you’ll still be effective. Note that this only holds true if the abstractions are good. If the division doesn’t make any sense, you’ll be without a doubt worse off.

How short is short enough? That’s a matter of debate and preference. Hear Uncle Bob for some guidance:

My rule is “extract till you drop”. A function is too big if I can meaningfully extract another function from it.

Extracting Methods is an Opportunity for Good Naming

Naming is hard. And yet, without good names extracting methods isn’t worth it. Even if the abstraction is good, a bad name makes it inaccessible. Thus, good naming is crucial for readability.

A name that describes the method’s purpose serves as lightweight documentation. Documentation that’s supported by the language, unlike comments. Automated tooling can find and change methods in a way that’s usually impossible in ad-hoc comments.

If you’re struggling to find a name for an extracted method, it can be a sign of two things:

An unclear boundary, and thus a fuzzy responsibility. That makes it hard to find a proper name
The method is still too big and needs to be split further

Not Bad, and There Is More

The combination of smaller methods and better names reduces cognitive load. Each method serves as a small reusable abstraction. For instance, something as typical as iterating over a list to apply a function to every element:

val input = listOf(1..10)

val result = mutableListOf<String>()
for(item in input) {
    result.add(item.toString())
}

return result

A better alternative is to use a higher-level abstraction, in this case, map:

val input = listOf(1..10)

val result = input.map { it.toString() }
return result

Using map encapsulates the implementation and lets you focus on what you want to express.

Anyhow. While shorter methods and better names are the big two aspects, other considerations that speak for using this refactoring.

Nesting Is Another form of Complexity

Excessive length isn’t the only way complexity creeps in. Code that’s nested multiple layers deep is as hard to read as an overly long piece of code. It can hide bugs as you inadvertently put things in the wrong scope.

Luckily, extracting methods reduces nesting the same way that it reduces length.

Small Methods Reduce Coupling

Dependencies make it harder to reason about code. They make it harder to change as well.

Suppose you have a massive method. There are plenty of local variables capturing intermediate results. While they provide an opportunity for meaningful names, they lack the encapsulation that methods provide. You have access to everything that happened before within the same scope, which can entangle the logic beyond salvation.

Not only that, you start getting worried about unexpected changes. You can’t be sure that the variable wasn’t modified unless you painstakingly go through the whole logic. Doubly so in languages that don’t allow read-only variables.

What About Testability?

And lastly, we come to testability. Does extracting methods improve testability? You could argue it doesn’t, as you should have your tests ready before you perform the refactoring.

However! Reality isn’t always so kind. You might face a huge, gnarly class that lacks tests. Writing tests might prove impossible unless you make the problem more manageable.

As described in the book Working Effectively with Legacy Code, extracting methods allows you to break down smaller pieces. At some point, they become small enough that you can write some tests for them.

It’s not pretty, but it gets the job done. Much better than the alternative of not really knowing if your changes will screw something.

Notice I Didn’t Talk about Reuse

I haven’t mentioned reusability throughout the article. It is, in fact, not the primary goal of Extract Method.

A smaller auxiliary method is valuable even if we only use it once. Making code more readable is already a significant benefit, worth doing on its own. If that method encapsulates a reusable abstraction, that’s a welcome benefit. But it’s not something to force or expect.

Is Extracting Methods Worth The Effort?

In any big codebase, the code developers write lives for a long time. Often, it’s still there while the original developers moved somewhere else. Meanwhile, new developers will read it many times and try to figure out what it does. Capturing the context of why a decision happened is hard, so these poor developers are left without context.

Thus, it will be read many times by multiple developers. Investing some effort in making it more readable pays off more and more over time.

Not that you should overdo it, though. An endless collection of one-liners isn’t an improvement, either. Aim for methods that stand on their own and have one purpose. That’s a reasonable guideline to follow.

Book Review: Effective Remote Work

Mario Fernández — Sun, 27 Nov 2022 00:00:00 +0000

Effective Remote Work is a book about, well, remote work. The author, James Stanier, works for Shopify, a company with a long tradition of working remotely.

These past few years have been a large-scale experiment in working remotely for most tech companies. However, the way it happened was so abrupt and unprepared that many people had a miserable experience.

It’s time to correct that! Despite the bumpy start, many companies still offer at least hybrid work. It seems this practice is here to stay. This book is a great resource about making remote work effective, productive, and sustainable. Let’s go over some of the things it mentions.

No Second-class Citizens

The book advocates treating everybody as a remote worker. I like that goal. You start with basic stuff like ensuring you don’t forget to include remote people in meetings. But it goes beyond that. Are they included in decision-making? Do they have the same career opportunities as people going to the office daily?

Proper Onboarding

A good onboarding plan is a must whether you work remotely or not. Working remotely magnifies a bad experience. Sitting at home feeling helpless because you don’t know something is frustrating. Remote workers need to be able to unblock themselves, and you need to help them with that.

Durable Artifacts

This point completely matches my experience in the past three years of working remotely almost full-time. Collaboration around durable artifacts should be at the center of everything you do. If you don’t regularly create easily accessible documents, you’ll repeat the same conversations over and over. Or, even worse, decisions will happen ad-hoc in secret meetings.

Good technical writing makes written communication better. Go check Google’s courses for a great resource.

The Spectrum of Synchronousness

People might not answer immediately when you have a question. You need to balance between things that require synchronous communication and the ones that can wait. If you have worked with people in different time zones you’ll know what I mean.

The Remote Test

Similar to the Joel Test, the book ends with a list of criteria for remote work. The questions that help determine if you’re ready are:

Do you treat everyone as remote?
Do you provide a remote workspace setup?
Do you spend money equally on in-office and remote staff?
Do you optimize for asynchronous communication?
Do you create artifacts from synchronous interactions?
Do you measure staff by their impact?
Do you allow staff to choose flexible hours?
Are the members of the executive team remote workers?
Do you use the best collaborative tools that money can buy?
Do you hire staff anywhere in the world?
Do you support families as well as employees?
Do you give back to an employee’s local community?

Verdict

This book is very poignant now that remote work has normalized to a never-seen-before degree. Many companies have embraced the practice. Thus, it’s worth investing in understanding how to be effective in this environment. Compared to another book like Remote: Office Not Required, there’s a lot of actionable advice that you can apply starting today. It gets ⭐⭐⭐⭐ stars from me.

Anatomy of a Successful Retro

Mario Fernández — Sun, 16 Oct 2022 00:00:00 +0000

Retrospectives are an integral part of Agile. When they are effective, they provide a powerful tool for continuous improvement. However, when they go wrong they become a catalyst for frustration and pointless venting. You don’t want to end your week on such a negative note, do you?

While I won’t claim to be an expert facilitator, I’ve experienced my fair share of crappy retros to know what I like and don’t like. Let’s talk about how to run retros productively. You don’t need a dedicated agile coach to get meaningful outcomes.

What Is a Retro?

Short and sweet: It’s a regular meeting where the team comes together and reflects on all that has happened lately. You discuss the most promising topics and seek actions to improve.

How do you recognize a good retro? Well, what about the opposite question? What are some retro smells that can doom the whole endeavor?

Retro Smells

Healthy debate

The one thing that grinds my gears is poor time management. For example, collecting stickies for 55 minutes and then realizing that you have only five minutes to half-ass some action items.

Another questionable habit is talking about things that you can’t influence. Don’t get me wrong: sometimes it’s good to vent and blow off steam. But it can’t become the norm. Otherwise, you’ll create a fountain of negativity.

My last personal favorite is the topic that comes up repeatedly, without ever getting to any resolution. Extra points if it’s an overly generic thing like not enough communication.

Everybody has their pet peeves. As a facilitator, it’s your job to sidestep them and foster a productive ritual.

Following A Structure

I tend to follow a familiar structure. I want repeatability and predictability. That’s easier to do with a general framework. At the same time, you want to adapt to what people want to talk about. This is my rough script for a 60 minutes slot.

Set the Tone
Icebreaker
Previous Actions
Reflection
Actions
Wrap-Up

Let’s go a bit more into detail.

Set the Tone

Start with a quick (I repeat, quick) reminder that a retrospective is a safe space. Have you read the Prime directive? It does sound a bit tacky, to be fair. Behind the “Star Trek”-like message hides an important question: Are people comfortable talking about potentially uncomfortable topics? If not, you have serious problems that can’t be fixed by gathering people in a room. Sorry for that.

Icebreaker

Icebreakers are small activities to lighten the mood. It seems silly, but it doesn’t hurt, provided you don’t invest too much time. Focus on the positive.

The icebreaker that consistently works best for me is collecting kudos. Spend a few minutes letting people thank others for their work and contributions. Share it broadly.

Previous Actions

It’s crucial to keep track of the actions decided in previous retrospectives. Are you accumulating a long list of unclaimed actions that won’t ever happen? That’s a huge red flag.

I like to go over previous actions to check progress. For the incomplete ones, it’s worth ensuring they’re still relevant and necessary. Prune the obsolete ones, and carry the rest as part of this iteration. It’s fine, but keep an eye on actions that never go anywhere.

Reflection

Now comes the meaty part. You’re going to do an exercise where you leave time for the participants to collect their thoughts and write them down in stickies. There are many formats for this. A typical one is to use three buckets like this:

Start Doing
Stop Doing
Continue Doing

Areas that aren’t going well hold the most potential for improvement. Nevertheless, don’t tunnel too much on those. A reminder of activities worth keeping alive can be very useful, even if it doesn’t merit further debate.

Be very, very careful of sticky overdose. Collecting 50 stickies is great until you realize you won’t get to discuss 95% of them. Here is the spot where many retros go awry. Limit the number of stickies to keep the logistics manageable.

Once you collect all these wonderful ideas it’s time to read them to ensure they’re clear to everybody. Then, group them and let people vote for the ones they want to discuss. This is a clarification process, don’t allow discussions to start just yet. Some people love to talk about their pet issues regardless of priority. They might monopolize the limited time if you aren’t careful.

The most important tool in any retro

No matter how efficient you are, you’ll have consumed about half the time by now. You’ll have time for about three topics. I like to use a very rigid structure for this:

Pick the most-voted topic
Start a timer for 7 minutes. Use the slot to explore the problem and think of actions
Once the timer is over, I move on to the next topic. If I feel that people still want to talk a bit more, I do a quick thumbs up / thumbs down vote to confirm the suspicion. You might discuss fewer topics, but you don’t leave a conversation unfinished

You might read this and recoil in horror. It’s so formulaic! However, there’s a reason behind it. If you want to finish on time, you’ll have to be ruthless with time management. If you explain the rules in advance, it feels less rude and awkward. Remember, we’re committed to finishing on time.

Actions

Realistic actions

A healthy discussion has value on its own, but we want to foster improvement. That requires some actions. As the facilitator, you have to be on the lookout for them. Pounce when given the opportunity.

There are plenty of catchy acronyms to define good action items, such as SMART. Follow the one you like. Remember that any action needs to be doable in a reasonable time frame. Ideally before the next retro. I refuse to write down commitments unless I can assign a person to them. I think that actions without an owner are as good as useless.

What if you don’t find any actions to take? It’s not ideal, but it happens. I’ve seen facilitators push to get some action, any action will do. I disagree with this approach. If you couldn’t find a meaningful thing to do, so be it.

Wrap-Up

If you did everything well until now, you’ll have about one minute left to wrap up, summarize the actions, and thank everybody for their time. Well done, recruit!

Write this summary down somewhere, especially the actions. Make sure that you aren’t sharing things you shouldn’t.

Scaling it Up

So we got one decent retro. Good! Now let’s make it a regular occurrence. There’s no reason that you can’t get a reproducible result.

I believe that team rituals shouldn’t be owned by an individual. Spreading them across the team increases ownership. Thus, I think moderation should rotate among the people on the team. It’s good practice for them and an antidote against having the same people talking all the time.

As a parting piece of advice, shake things up! Repeating the same format all the time puts people on auto-pilot. Switch formats occasionally to keep it fresh. Every now and then, do a completely different thing. A futurospective about where the team wants to be next year. Focus on only one topic that’s important for the team.

What Do We Actually Get Out of Retrospectives?

Enlightment state

Getting quality action items is good. Implementing them within a reasonable time frame is even better. Let’s start with that.

That said, I think retrospectives are also about creating a culture of reflection. It’s easy to let routine take over and do things the same way as before. Reflective teams are proactive in finding ways of improving on their own. A good retrospective is one small step towards that lofty goal.

The Behavioral Interview

Mario Fernández — Wed, 07 Sep 2022 00:00:00 +0000

The behavioral interview is the last component of the interview loop at BigTech. It’s not, as its name implies, a technical interview. Instead, it focuses on your previous experience. You’re there to show that you understand how to act and communicate. It’s similar to the System Design round in that it weighs more heavily for senior candidates.

Well, I don’t need preparation to talk about my experience, you might think. And you’d be wrong. Don’t underestimate this interview or you will fail it miserably. A behavioral interview isn’t an informal chat where you reminisce about your previous experience and have a good laugh in the end. There are clear expectations.

I’m ending my series about interviewing talking about this interview format.

Structure of a Behavioral Interview

A behavioral interview centers around open-ended questions. There are many, many sample questions on the internet. They tend to sound similar, though:

Tell me about a project where you had a big impact
Tell me how you solved a conflict between multiple teams
Tell me about a time when you had to make a quick decision

Note that these questions focus on your previous experience, not hypothetical scenarios. The reasoning is that it’s easier to understand how you think by hearing about what you did already.

These questions are vague and undefined. However, that doesn’t mean you should answer with the first thing that comes to mind. Doing so is a recipe for rejection. Instead, the canonical advice is to use the STAR method. While that’s the right starting point, we can build on top and do better. Allow me to explain.

STAR is Flawed

This response could be shorter

STAR (Situation, Task, Action, Result) is a way of structuring your answers to any behavioral question. You start explaining the context. Then, you talk about your task. You follow with your actions, and in the end, you round it up with the results of those actions. Using this structure ensures that you tell a coherent story.

There’s a problem, and it has to do with repetition. In practice, Situation and Task often overlap significantly. If you follow the method rigidly you’ll end up mostly saying the same thing twice. It might sound innocuous, but you can’t afford to use your limited time inefficiently.

In these interviews, time is your enemy. At Meta, for instance, they last 45 minutes. That includes the introduction and your questions, mind you. You won’t have enough time to talk about all the wonderful things you have done. Repeating yourself leaves you less time to provide useful signal. It makes the interviewer’s evaluation harder, which reduces your chances.

I prefer using a streamlined version of STAR. Let’s call it PAR: Problem, Action, Result.

Using PAR

Let’s say you get one of the aforementioned questions. You come prepared and want to answer it following this method. What are the relevant bits to include for each section?

Problem

This part is about giving context. Where were you, when was that, and what was your role. Make it short and crisp. Skip any detail about your position that’s not fundamental for understanding the situation.

After that, it’s time to describe the problem. Remember that you have to make it understandable for people who weren’t there. Talk about the timelines. If you were measuring the problem with some KPI, that’s worth mentioning.

Be succinct! An interviewer will get frustrated if you bore them with irrelevant details. Practice saying the least possible that is still enough to understand what’s going on.

Action

You set up the stages, now it’s time for the important bit. What did you do? Explain that without domain-specific context or internal terminology. Focus on what you did. A behavioral question isn’t the time to talk about the team. They are looking to hire you , not your team. Ensure that there’s no ambiguity about that.

Don’t go too broad with your answer. For a complex project, you did a lot of things. You don’t have time for all of them. Pick the top three that are the most relevant for the question and had the most impact. Again, you’re looking for the right amount of detail.

Result

The result is where you round up your answer and come to a satisfying conclusion. You did a whole bunch of things. Talk about the outcome. There has to be some outcome. If all your actions changed nothing, find another example.

Ideally, you’ll have numbers that you can mention. You don’t always have solid, well-measured numbers, but you can make reasoned estimations. Don’t make outlandish claims unless you’re ready to justify them.

Depending on time, consider including:

Next Steps: What would come next?
Scaling: How can you use what you did to scale it to a bigger audience?
Learnings: What would you do differently next time?

You put all three together, and that’s your answer. See this post from Will Larson to see an example.

Building a Portfolio of Experiences

In my experience, the best way to prepare for the behavioral interview is to build and curate a portfolio of examples based on your experience. There are way too many questions out there. You can’t prepare specific answers for all of them. Instead, I focused on having strong examples that cover multiple behaviors. That way, you can adjust them to the situation instead of scrambling to think of something unique on the spot.

A future portfolio of success

The Amazon Leadership Principles are a good base. They cover plenty of ground, so you get to think about many different aspects of your previous jobs. I have a table with examples spanning my career that I’ve refined carefully. I usually follow the same approach to extend this table.

First, I write down a sketch of my answer. Just enough that I know all the points I want to cover. It’s fine to bring your notes to an interview, but don’t read a whole answer straight out of them. Once the case is ready it’s time to polish it! Your first version will inevitably suck. You probably made it too long. Or didn’t make the results clear enough. In any case, you need to iterate relentlessly. Get feedback from somebody you trust for extra perspective.

Lastly, test it out. Read the scenario aloud to yourself or somebody else. Do a mock interview, or use it in a real interview. While your experience remains the same, your ability to present it will improve massively.

Some Extra Thoughts

Don’t make stuff up. Honestly, you’ll probably get caught and fail the interview. Even if you somehow put it past your interviewer, you’re setting yourself up for failure.

Try to cover a decent span of your career. People want to see that you have acquired a broad array of experiences. At the same time, your most recent work is the most valuable. It’s easier to remember, and it’s a closer representation of who you are today. Your strongest examples should come from the last few years.

You should prepare enough examples. You might get behavioral questions in every interview you do. Expect a couple of them. Talking about the same scenario over and over won’t be beneficial.

Lastly, don’t limit yourself to positive experiences exclusively. We all have failed at some point. Talking about your failures openly is a great way to show self-reflection and growth. What did you learn, and what would you do differently? Needless to say, don’t confront failure by blaming others. Even if it happens to be true you’re showing a very concerning side to somebody who’s trying to gauge your contributions.

Measure your Outcomes

Let’s be clear: You can’t change your experience. You can make present it better. You can show your ability to reflect and learn. Don’t get discouraged if you think that your examples are too modest.

If you don’t have impactful examples, think about how to get there over time. Use this to guide your work. If you’re working on a problem at work, go deep and understand its true impact. Seek metrics that will reflect changes. If you’re doing good work, you’ll be able to show it sooner or later. If you somehow didn’t burn all your motivation yet, this video from Jackson Gabbard.

The System Design Interview

Mario Fernández — Mon, 08 Aug 2022 00:00:00 +0000

System design interviews are a staple of interviewing loops at BigTech. In a system design interview, your job is to design a complex system, such as a video platform like YouTube. You aren’t required to code anything. Instead, you focus on the high-level architecture and discuss aspects like scalability for your design.

Usually, only somewhat senior candidates get this interview. The assumption is that you need to gather a certain amount of experience before you can talk about designing systems in depth. However, it wouldn’t be out of the ordinary if you haven’t done one of these before. It all depends on your career path. In this article, I want to share what I’ve learned while preparing for them.

What Am I Supposed to Show in a System Design Interview?

In this interview, you show your technological breadth and depth. You consider the different trade-offs and work with the interviewer to get the best solution you can in the allotted time.

It’s not just about technology, though. The problem is almost always unclear and loosely defined. It’s up to you to bring clarity, understand what to build in more detail, and discuss the alternatives.

Keep in mind that system design can be decisive in your leveling, even more so than the coding part. Check this video from Jackson Gabbard is an excellent introduction to the topic.

A Solid Foundation

The first layer of preparation is your day-to-day job. If you’re designing systems, you have the opportunity to hone your skills and acquire experience.

So, practice as much as you can! Write your proposals as RFCs and get feedback from senior engineers in your organization. Review other people’s designs. If this isn’t a well-established process, maybe there’s an opportunity to drive that initiative. It’ll do wonders to your engineering palate. Even if you don’t operate at BigTech scale, it’s a valuable experience.

If your company doesn’t provide any opportunity to engage in these activities, you should consider moving to a different place. Technical decisions made without any debate? That sounds suspect to me.

Preparation Materials

The bible of System Design

Designing data-intensive applications (DDIA) is the canonical book for system design. It’s almost a mandatory read as it covers such a vast amount of topics that are relevant to system design. Additionally, it helps polish your vocabulary to communicate more efficiently with the interviewer.

Specialized courses are popping up that cover the topic. Alex Wu’s course is probably the best and completely worth your money.

YouTube has a ton of content. For instance, this channel is a good source of inspiration. QCon’s talks are good as well.

Lastly, BigTech companies almost always maintain an engineering blog. That’s an excellent resource to understand how they fix their problems.

Practicing Actual Interviews

Once you have a solid base, it’s time to practice. Unlike coding interviews, system design is hard to practice alone. System design interviews are very reliant on timing. You’re looking for an intuitive feeling of how long to spend on each part of the interview. I don’t think you can get the timings right without testing them under realistic conditions.

Mock interviews are by far the most effective way to practice system design. You can use interviewing.io if you’re in a rush and don’t mind paying. Or check one of the many interviewing-oriented Discord channels and find a buddy to run some mocks.

It took me a lot of mental effort to start doing mock interviews. I just found it awkward. I tried to get by doing it by myself. I can tell you it’s not the same. Get over it as fast as possible. It’s worth it.

Don’t go into a system design interview without practice. You’re almost guaranteed to fail.

Tips And Tricks

There’s a vast amount of materials out there. I won’t cover the basics, or tell you to do capacity planning or anything like that. Instead, I’m focusing on some realizations that helped me polish my interviews.

Manage Time. Interviews are shorter than you think. To pass a senior loop, you have to make sure that you handle time effectively
Reduce Scope. This is related to time management. System design problems are almost always bigger than what you can handle in an hour. You have to find ways to reduce the scope. Otherwise, you might leave the problem unfinished, or fail to dive into a concrete area.
- Your main scoping tool is the functional requirements. Don’t define too many of them
- Be prepared to descope during the high-level design. If you realize that you’re trying to do too much, offer to skip some parts or skip details. Whatever you do, don’t risk a half-finished solution
Don’t repeat yourself. Train to not repeat the same thing multiple times. You need all the extra time you can buy
Have an agenda. I’ll get to this in a second
It’s not a quiz. Some people memorize extremely specific details. That misses the point. I would say that presenting a coherent and reasoned solution is better, even if you skip some details
Talk about trade-offs. Every design has pros and cons. You should ensure that you explain the consequences of your decisions and how they inform your design
Who uses the System? It’s tempting to dive into replication strategies, partitioning, and a multitude of other technical challenges. But you should always keep the user in your mind. Who is using the system? What’s their success? If I introduce a change in my design, will it result in a better experience for them?

A lot of these suggestions are very fuzzy. How much scope is too much? How do you know how much time you can spend in an area? That’s why you practice. You need to get a feeling of that, and that’s only happening if you practice in advance.

Present a Clear Structure

We're getting somewhere here

There’s so much you can talk about in a system design interview. It’s easy to get lost or start aimlessly rambling. That’s why it’s useful to create a rough outline of the topics you’re going to cover. It has two crucial benefits:

You have a structure to follow to keep you on track
You align with the interviewer. If they want to dive into a topic, make sure to integrate that into your plan and cover it

Note that you should never repeat the same structure for each interview. You’ve got to figure out the relevant parts for the problem at hand and consider your interviewer’s expectations. They surely want to touch on some topics. Don’t ignore those.

With that being said, this is a sample agenda I’ve used before:

Problem Exploration
1. Functional Requirements
2. Cross-Functional Requirements
3. What is the success case for the user, and how to measure it
Plan
1. Propose a list of topics
2. Align on items to cover
Quantitative Analysis
1. QPS
2. Data amounts
API
High-Level Design
1. Main services
2. Interactions between them
3. What kind of data do you produce and store
Deep Dive
1. Find the most promising topic to dive in
2. Go as deep as you can
3. Trade-offs!
Operational concerns
1. Points of failure (error handling)
2. Monitoring
3. Analytics -> Business metrics
Scaling Up
1. What are the bottlenecks
2. Elasticity

Remember that you won’t have time to talk about everything.

Another way to use an agenda to your benefit is to come back to it as you address different topics. If your design is close to complete, go back to the requirements and check that you covered them.

Huge Disclaimer : If you sketch a plan, it’s fundamental to make sure you hold yourself to it. If you only manage to run through half of it, you’re pretty much priming your interviewer to fail you. Practice until you can manage the timeslot. Be prepared to descope things as you move along, but don’t ever get to the point where the interviewer cuts in the middle of a sentence with something like “We don’t have any more time, let’s stop here”.

Logistics

Use a diagramming tool. Get familiar with one so that you don’t waste time trying to draw circles. I like Whimsical or Excalidraw.

Write down things! Don’t leave empty boxes. If you’re talking about a topic, write it down, even if only on a high level. The interviewer might go back to the board after the interview. It’s always a challenging balance to strike, as writing absolutely everything you say will slow you down significantly.

Putting it All Together

As you can see, the system design interview focuses on you showing how to apply your experience in technology to fix a nebulous problem in the best way possible. It’s fuzzy and open-ended, and it requires you to think clearly and communicate efficiently. Improve your chances of success by working on all these pieces in advance!

Tips for Coding Interviews at FAANG

Mario Fernández — Mon, 11 Jul 2022 00:00:00 +0000

Have you ever considered a position in BigTech as an engineer? Lots of people do. There’s no shortage of articles where people talk about their experiences.

If you’ve looked into it, you know that you’ll have to pass a coding interview as part of the process. Possibly multiple rounds, depending on the company. You’re unlikely to get an offer without a decent performance. Better be prepared for them.

Let’s talk about coding interviews. There are way too many articles about this topic already. I’m assuming you’re familiar with the format already. I’m going to focus on a bunch of things that helped me during the interview process at FAANG.

If you’re not up to speed, check out this article from the pragmatic engineer. He’s an invaluable source on all things related to tech.

Coding Interviews Have Moved Online

Coding interviews are about writing a small program. Typically it’s an algorithmic challenge. Traditionally, you’d do that in person, writing your code on a whiteboard.

Nowadays, most interviews happen remotely using some online editor. It’s a pretty significant improvement, as coding on a whiteboard is quite uncomfortable. Still, don’t expect much more than simple code highlighting. You don’t get to compile or run your code.

Thus, the very first thing to do is to get familiar with the environment you’ll be using in the real interview.

Preparing for a Coding Interview

It goes without saying, but if you’re going through the process, you’ll have to spend time preparing. If you can’t commit to that, or you fundamentally oppose the idea, then you should skip the whole thing. I don’t think you can realistically pass a FAANG interview loop without any preparation.

With that in mind, here are six tips for a more successful coding interview:

The Right Amount of Grinding
Speed is Key
Share your Thoughts
Get Used to Write Readable Code
A Clear Structure
Don’t Give Up

The Right Amount of Grinding

Putting that subscription to use

There’s a whole industry built around preparing for technical interviews. Books, videos, courses, websites, you name it. I would stick to LeetCode. LeetCode is a highly convenient place to practice exercises similar to the ones you’ll encounter.

LeetCode has a ton of exercises, though. It’s easy to get lost. You read horror stories of people that spent months doing it full time. I don’t think you need to get to those extremes to have success.

Aim for breadth. I used LeetCode to develop a broad portfolio of techniques you can apply to any problem. Some concrete recommendations:

Cover a variety of structures, like Arrays, Hashes, Linked Lists, Heaps, Trees, or Graphs
Cover a variety of techniques, like Binary Search, BFS/DFS, Sort, Backtracking, or Sliding Windows

Other structures like Tries or Disjoint Sets make some difficult problems super easy. It’s never a bad idea to expand your toolkit.

Certain algorithms are just too hard to implement in the context of an interview. Nobody other than a sadist will ask you to write a Red-Black Tree in an interview. Then you have some edge cases, like Dynamic Programming. Google is supposed to like them a lot. Get familiar with some of them, but remember that plenty of people struggle mightily solving them.

LeetCode categorizes problems as easy/medium/hard. I doubt you will pass an interview for BigTech if you can only handle easy problems. If you are good at finishing medium problems, you’re probably good to go. If you can reliably solve hard problems under the constraints of an interview you probably don’t need my advice.

Speed is Key

The king of terse code

Interviews are short. You have about 15-20 minutes per problem to solve two problems, at best. Minimizing the amount of typing gives you back some valuable time.

For that reason, I recommend using the highest-level language you’re comfortable with. I don’t know if esoteric languages are a good bet, though.

Python is a common choice because it’s both popular and high-level. I’m not a C++ person, but it feels a bit too low level. I find Java too verbose (consider using a newer version if you pick it). I don’t like JavaScript as it doesn’t have a very comprehensive standard library.

I ended up using Kotlin. Similar to Java, but much more succinct. You can use all the standard libraries. I like the language a lot, so it made my preparation easier. I think you should stick to one language, though. Most companies are fairly accommodating in that respect.

Share your Thoughts

You can’t solve problems silently. Even if you’re correct, you might lose your interviewer. However, don’t mistake this ubiquitous advice for constant talking. Just avoiding silence isn’t good enough.

You should talk about the options that you’re considering. What are the tradeoffs? Maybe you have a great idea that won’t fit within the interview. Share your thoughts and let the interviewer participate. Many are happy to support you and offer valuable tips and input.

Once you start coding, shift to discuss why you are doing things. Explain the choices you make in data structures. Walk the interviewer through the structure of the code. Show her a coherent path.

Talking while coding isn’t easy to practice just by doing LeetCode. Consider walking a friend through a problem, or doing a mock interview.

Get Used to Writing Readable Code

Good variable naming

The pressure of getting something done quickly will tempt you to take shortcuts. Don’t do that.

Use good variable names and split chunks into functions. Write idiomatic code. If needed, add a comment or two. It will help you, and it will help the interviewer evaluate you.

Splitting the code has another benefit; you can tackle things in order of importance. Most interviewers won’t care if you leave boilerplate functions unimplemented. I’ve had interviewers that completed some of my methods while I was writing the core part.

You won’t have linters or formatters to help produce readable output. It’s on you to get this ingrained as a habit. Doing it naturally means you won’t have to focus on it, which leaves space to focus on other meatier parts of the interview.

A Clear Structure

Following a clear structure helped me a lot. If you get nervous you might forget important steps. Take this as an example:

Understand the problem. State it in your own words.
Input/Output. Write down some inputs and the outputs. Think of edge cases.
Approach. Think of an approach, and explain the tradeoffs.
Code the solution
Read through your solution carefully. Spot errors
Test the code. Run a sample input manually
How does the solution perform? Use the O-notation, for time and space
Can you improve it?

Notice how coding is just one among many steps. That’s comparable to how you’d approach a problem in real life.

Programmers estimating the time complexity of a double loop

Being disciplined is key. It gives you time to explore the boundaries of the problem. Maybe you misunderstood the problem. Establishing a conversation helps you work on what the interviewer wants, not what you have in mind. Depending on seniority, you won’t pass the interview if you only code a solution without thinking about edge cases and testing the solution by yourself.

If you don’t find the optimal approach, take any solution that you can think of and iterate. Iterate continuously until you get to a good place. Clear code and discipline help tremendously here.

Given that you’re diligently preparing, there’s a chance that you get a problem that you’ve seen before. Don’t pretend to struggle and “magically” find the optimal solution. You’ll make an ass of yourself and fail in the process. Be honest in what you know. Let the interviewer change the problem if they need to, but don’t force the issue yourself.

Learn to do reasonable O-notation estimations on the fly. Nobody is going to expect hard proof in the middle of the interview. A few heuristics go a long way.

Don’t Give Up!

If you get stuck, don’t panic. Keep trying things, and keep voicing your thoughts. A naive solution is better than no solution. You might think that you’re bombing the interview, but some problems are too big by design.

If you’re running out of time, explain how you would continue with the parts that are missing. Whatever you do, don’t give up. Even if your performance isn’t great, show as much as possible.

What if I Always Get Nervous During Coding Interviews?

Welcome to the club! Very few people are naturally good at this. An interview is a stressful situation, no matter what. Preparing well gives you the backbone to rely on when things get rough. Interviewers want you to succeed, for the most part.

Even if you are well prepared, you’ll fail some interviews miserably. Shit happens. Don’t take it as a personal failure. Everybody has gone through that.

Book Review: Site Reliability Workbook

Mario Fernández — Mon, 23 May 2022 00:00:00 +0000

Site Reliability Workbook is the successor of Site Reliability Engineering. In the first book, Google introduced its vision of site reliability to the world, launching countless discussions about the distinction between SRE and DevOps.

One common criticism of that treaty was that it was hard to apply for organizations with fewer resources than Google. The second book aims to fix that. This time around, they’re focusing on discussing practices that anybody can implement to become more effective at operating live systems.

So Much Content

This book is a gigantic HOWTO on how to implement SRE. There’s a good amount of theory, there’s is source code, and there are case studies. It covers three areas throughout its almost 500 pages:

Foundations

Foundations are the base building blocks you’ll use daily if you adopt an SRE approach. SLOs, monitoring, toil, and a call for simplicity in software architecture.

Practices

What kind of practices help to continuously improve and iterate? Having a healthy On-Call rotation for one. Handling incidents in general features prominently, including reflecting on causes and next steps in postmortems.

Interestingly, how to handle configuration is another big topic here. I was pleasantly surprised to learn that Google emphasizes manageable configuration definitions. They mention jsonnet as a tool to handle that complexity.

Processes

The last part of the book is about processes. How should an SRE team engage with other teams? How do they recognize if the time for collaboration is over? In a way, this section reminds me of another book, team topologies. I’ve lost count of how many times I’ve said that a platform team and an enablement team have completely different mandates. This part really resonated with me.

Let’s Talk About SLOs

Chapter 5 focuses on service level objectives (SLOs). It’s by far my favorite chapter. I’ve re-read it over and over. I’ve spent a lot of time thinking about alerts over the past year.

Understanding how to implement multiwindow, multi-burn-rate alerts improved my quality of life massively when I was on a project that suffered from poor alerts. And let me tell you, it’s not an easy thing to understand. Don’t underestimate the complexity behind implementing high-quality alerts.

This book is the best source to understand SLOs. It’s worth getting just for this chapter alone.

Verdict

This book covers a lot of ground. There’s a lot to unpack, much of it extremely actionable. I’ve found myself coming back to it a lot lately. It seems to have more sticking power than the first book. I think it’s a very recommendable read. It gets ⭐⭐⭐⭐ stars from me.

Making a Microservice More Resilient Against Downstream Failure

Mario Fernández — Sun, 08 May 2022 00:00:00 +0000

I like thinking about monitoring and alerting a lot. SLOs have been one of my latest obsessions over the past few months.

However, all these metrics are worthless if left unattended! Worse than worthless. If you have comprehensive alerts that don’t get fixed, you’ll get swamped by a barrage of alerts.

That’s precisely what I want to talk about. This post covers making services more tolerant against downstream failures, using the wonderful resilience4j.

Instability Can Ruin Your Day

In my last project, I maintained a service with high visibility. The service didn’t own any data, relying instead on several downstream dependencies. Sadly, these dependencies were very unstable and experienced frequent downtime. The issues propagated to our service, causing errors. It made us look bad. Check these error spikes:

Shit is hitting the fan

Not only were our users getting a mediocre experience, but our On-Call rotations were becoming a shitshow. We had to do something about it. It was leading to a dangerous spiral that could only end in burn-out.

Enter Resilience4j

Our service is written in Java. That makes resilience4j a natural fit. In case you don’t know, it’s a library written by Netflix focused on fault tolerance. As a bonus, it’s based on Vavr. Thus, it benefits from things like the Either type and all that good stuff. It’s right up my alley.

To prevent the nasty error spikes I showed above, we’re focusing on two things:

Using resilience operators to make our microservice a better-behaved citizen
Building a fallback cache to prevent downstream errors from affecting our users

Using Resilience Operators

resilience4j provides a bunch of operators out of the box that increases the fault tolerance of your remote calls. One big plus of this library is its composability. There’s little difference between using a Retry alone or piping other operators like a CircuitBreaker. I’m not getting into the details of how they work, head over the documentation to get more details.

If you use Spring Boot, there’s support for using annotations. There’s always support for annotations if SpringBoot is involved. Interestingly enough, I feel the annotations create a bit too much duplication. See, I wanted to apply a common set of operators to six or seven remote calls. Annotating the methods and adding the fallbacks led to a messy solution.

Instead, I built a small abstraction, a FallbackCache. Before we consider the caching part, let’s focus on instantiating these operators. The code is quite declarative:

Retry getRetry(String name, MeterRegistry meterRegistry) {
  RetryRegistry retryRegistry = RetryRegistry.ofDefaults();
  TaggedRetryMetrics.ofRetryRegistry(retryRegistry).bindTo(meterRegistry);
  return retryRegistry.retry(name);
}

CircuitBreaker getCircuitBreaker(String name, MeterRegistry meterRegistry) {
  CircuitBreakerRegistry circuitBreakerRegistry = CircuitBreakerRegistry
      .ofDefaults();
  TaggedCircuitBreakerMetrics
      .ofCircuitBreakerRegistry(circuitBreakerRegistry).bindTo(meterRegistry);
  return circuitBreakerRegistry.circuitBreaker(name);
}

There are a vast amount of settings that you can configure. The default settings work well to get started. On top of that, I’m adding metrics for micrometer.

Building a Fallback Cache

Building resilience operators is only one part of the equation. Yes, we’re retrying failed requests automatically. Thanks to the circuit breaker we won’t DDOS downstream services. But the user won’t care. As it is, the errors propagate back to them.

My answer is a fallback cache. The flow is as follows:

That's a happy user

This boils down to three steps:

Make a call to the downstream service with retries and a circuit breaker
If the call works, store the result in a cache and return it
If the call fails, use the cache to serve a fallback so that the user experience isn’t compromised

Before you ask, caching proactively wasn’t an option as we didn’t have a way to invalidate the cache when the underlying data changed. Anyhow, these three steps map nicely to a class with the following methods:

public class SyncFallbackCache<T, R extends Cacheable<T>> extends AbstractFallbackCache<T, R> {
  public T get(Supplier<String> keySupplier, Supplier<R> valueSupplier);
  private T getAndCache(String key, Supplier<R> valueSupplier);
  private T getFromCacheOrThrow(String key, RuntimeException exception);
}

Let’s go one by one.

Making the Call

The fallback cache doesn’t do anything on its own, as it’s rather a wrapper. We use a Supplier to pass the operation to perform. We need a key to identify the element in the cache, which uses another supplier for extra flexibility.

public T get(Supplier<String> keySupplier, Supplier<R> valueSupplier) {
  Supplier<T> decoratedSupplier = Decorators.ofSupplier(() -> getAndCache(key, valueSupplier))
      .withRetry(retry)
      .withCircuitBreaker(cb).decorate();

  var key = keySupplier.get();

  return Try.ofSupplier(decoratedSupplier)
      .recover(RuntimeException.class, (exception) -> getFromCacheOrThrow(key, exception))
      .get();
}

We decorate the supplier with our operators. If the call works, we return the result. If not, there’s a Try wrapping it that looks for something to serve from the cache. An empty cache means we’re out of options, and we can only propagate the error upstream.

Store the Result

Whenever we get a valid result, we store it in the cache. Let’s assume that valueSupplier.get() returns without any errors:

private T getAndCache(String key, Supplier<R> valueSupplier) {
  /*
    This has the disadvantage that the [get result, cache it] is not atomic. So if two requests try to update the same key at the same time, it might create an undetermined result.

    However, two calls to the same service should return the same value, so we should be able to live with that.

    This is not happening inside the compute method anymore due to problems acquiring the lock that led to significant timeouts.
   */
  var result = valueSupplier.get();
  if (result == null) {
    return null;
  }

  return Try.of(() -> {
    cache.save(result);
    return result;
  })
      .recover(Exception.class, e -> {
        log.error("Could not save result to cache", e);
        return result;
      })
      .map(Cacheable::data)
      .get();
}

This action should be as seamless as possible. We want to avoid blocking anything while we save the data. Moreover, we swallow any error that happens while saving. Failing to save to the cache shouldn’t result in a failed request.

Use the Fallback

If the supplier didn’t work, our last hope is to find some content in the cache for the associated key. We can show valid, albeit possibly outdated, information to the user. A cache miss ends up with the original exception.

private T getFromCacheOrThrow(String key, RuntimeException exception) {
  var value = cache.findById(key);

  value.ifPresent((v) -> {
    log.info(String.format("FallbackCache[%s] got cached value", cb.getName()), v);
  });

  return value
      .map(Cacheable::data)
      .orElseThrow(() -> {
            log.error(
                String.format("FallbackCache[%s] got an error without a cached value", cb.getName()),
                exception
            );
            throw exception;
          }
      );
}

The Caller’s Perspective

As a user, I need to instantiate a SyncFallbackCache and call the public method with the operation I want to safeguard.

private SyncFallbackCache<Collection<SupplierUserPermission>> permissions;

private Collection<SupplierUserPermission> getPermissions(
    int userId, 
    int supplierId) {
  return permissions.get(
      () -> String.format("id:%s;suId:%s", userId, supplierId),
      () -> client.getUserPermissions(userId, supplierId)
          .getData()
          .getSupplierUserMyInformation()
          .getPermissions()
  );
}

As you can see, we don’t need to know about what’s happening underneath. It’s pretty nifty. You could argue that the return type isn’t explicit enough, as the method may throw an exception that’s not part of the signature. An alternative is to model it with an Either.

Where’s the Cache?

I haven’t spoken about the cache. The easiest alternative is using an in-memory cache like Caffeine. Something like this:

static <R> Cache<String, R> cache(String name, MeterRegistry meterRegistry) {
  Cache<String, R> cache = Caffeine.newBuilder()
      .recordStats()
      .expireAfterWrite(Duration.ofHours(24))
      .maximumSize(6000)
      .build();

  CaffeineCacheMetrics.monitor(meterRegistry, cache, name);
  return cache;
}

This approach is convenient yet flawed. Our application runs as a container. Every time it’s restarted, the cache is wiped out. It leads to a bad hit rate percentage, which hovered around 65-70%. That’s not great for a cache and still propagates way too many failures to the end-user.

We switched to a persistent cache based on Aerospike to mitigate the problem. SpringBoot has some integrations to make this easier:

public interface NavigationRepository extends
    AerospikeRepository<CachedNavigation, String> {
}

@Value
@Document(collection = "navigation_CachedNavigationMenu")
public class CachedNavigation implements Cacheable<List<LegacyNavMenuItem>> {
  @Id
  String id;
  List<LegacyNavMenuItem> navigation;

  @Override
  public List<LegacyNavMenuItem> data() {
    return navigation;
  }
}

Using the cache for failure scenarios means invalidation isn’t that relevant. The persistent cache made a tangible difference that brought our hit rate percentage to the high 90s. That’s a lot better, isn’t it?

A Journey With a Happy Ending

Was all this effort worth it? Hell, yes. Our error rate went from 2% to 0.03%. We’ve set an SLO for the service’s availability of 99,95%. Our rotations became significantly smoother. I had so much time available that I could write blog posts about it!