DEV Community: Alberto Guerra González

Can't cache this 🎶

Alberto Guerra González — Mon, 06 Dec 2021 16:15:45 +0000

Introduction

We often see companies publishing a short incident recap after they had a major outage. A couple of recent ones were Facebook and Cloudflare.

Although knowing the actual root cause is interesting, I think there is a lot to learn from how the engineers managed to troubleshoot the issue.

So I decided to take a stab at it and walk you through the troubleshooting process that took place in one of our recent incidents.

The beginning of it all

A few weeks ago, my team was preparing for the launch of an alpha version of a new product. Engineers were heads down wrapping up the last features and smashing bugs when our QA raised a bug in production.

The users should have been redirected to the payment page. However, quite often, the underlying request would timeout when anyone was attempting to click on the Buy button.

This was a regression and a major one, hence we decided to jump on it.

Investigation

The first step was to dive into the backend app logs. Here is what we saw:

Judging by the logs, it looked like the request froze 🧊 at some point during the execution path. This was hard to believe since the app is simple enough that shouldn't lead to any deadlocks. However, we should have seen an Exception somewhere and that wasn't the case here.

Also, what made it even more interesting was the fact that we also saw some requests that executed normally.

Nevertheless, since the bug was fairly easy to reproduce, we needed to keep digging.

Diving into the code

The next step in the investigation was to analyze the log lines in the code and pin point 🎯 the section that was causing the request thread to freeze. This can simply be done by looking at the last log line reported and the next log line that should have been reported.

This exercise brought to light the first suspect. As part of the request, we fetched a large JSON file from an external service. Since this file didn't change very often, we added a caching layer to save some network requests and reduce the latency of the request.

To implement this caching layer, we decided to use a Guava cache. The cache was configured so that it would return old values while the new values were being fetched.

Here is a fragment of code showing how the cache was configured:

var cache = CacheBuilder.newBuilder()
        .refreshAfterWrite(Constants.CACHE_REFRESH_TIME_MS, TimeUnit.MILLISECONDS)
        .build(buildCacheLoader(baseUrl));

private static CacheLoader<String, String> buildCacheLoader(
      String baseUrl) {
    return new CacheLoader<>() {
      @Override
      public String load(final String environment) throws Exception {
        log.info("Fetching data");
        var builder =
        HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(Constants.TIMEOUT_IN_SECONDS));

        var client = builder.build();
        final HttpRequest request =
            HttpRequest.newBuilder()
            .uri(URI.create(baseUrl))
            .timeout(Duration.ofSeconds(Constants.TIMEOUT_IN_SECONDS))
            .build();

        final HttpResponse<String> response =
            client.send(request, HttpResponse.BodyHandlers.ofString());

        return mapper.readValue(response.body(), Data.class);
      }
  };

At this point, it was clear that the cache wasn't behaving as we expected since:

It wasn't logging the potential Exception that would explain what was causing the request thread to hang.
It was causing the request thread to hang.

Thus, I decided to spend some time trying to understand in depth how Guava caches work.

Setting up the playground

In situations like this one, I usually create a playground project where I can remove all the noise (business logic) and focus on what matters 🧐.

I had to come up with a setup as close to production to ensure I was able to reproduce the exact same issue.

To achieve that, I decided to:

Interact with the cache in separate threads. Since the web server framework we rely on executes each request on a different thread (within a thread pool).
Use WireMock to simulate the calls to the external service. That way we can make sure the network stack is exercised and that it is easy to set up edge case conditions (delays, error responses, etc).
Use JUnit to setup the different scenarios and run them.

To ensure we have visibility in everything that happens during the different test cases, the threads will run the following Runnable which includes several log lines:

@Slf4j
public class CacheReader implements Runnable {
  private LoadingCache<String, Data> cache;

  public CacheReader(LoadingCache<String, Data> cache) {
    this.cache = cache;
  }

  @Override
  public void run() {
    try {
      log.info("Reading value at {}", System.currentTimeMillis());
      log.info("Value {} at {}", this.cache.get("something"), System.currentTimeMillis());
    } catch (ExecutionException e) {
      log.error("Exception while loading value at {}", System.currentTimeMillis(), e);
    }
  }
}

❓ Mystery 1 - Exception MIA

The first mystery to solve was the missing Exception. For that, there are two potential failures we could have experienced in production that should have reported an Exception:

The cache failed to be populated on the first attempt.
The cache was populated but an attempt to refresh the cache later on failed (either due to an error on the external service or a network timeout).

🧪 Test 1 - Given an empty cache and a slow network connection, when fetching the cache, it should throw an Exception

@Test
@SneakyThrows
void givenEmptyCache_whenRemoteServiceTimeout_shouldThrowException() {
    var cache = CacheFactory.buildCacheWithoutExceptionHandling(wireMockServer.baseUrl());

    setupRemoteResponse("value", 2 * Constants.TIMEOUT_IN_SECONDS);

    var thread = new Thread(new CacheReader(cache));
    thread.start();
    thread.join();
}

Which reports the following logs:

1861 [Thread-8] INFO  org.example.CacheReader [] - Reading value at 1637941200937
1872 [Thread-8] INFO  org.example.CacheFactory [] - Fetching data
2925 [Thread-8] ERROR org.example.CacheReader [] - Exception while loading value at 1637941202006
java.util.concurrent.ExecutionException: java.net.http.HttpTimeoutException: request timed out 
... <omitted stacktrace> ...

When the cache is empty, the thread loading the cache seems to be the one that receives the Exception.

Since we didn't see any Exception in our logs, this means this wasn't the failure we observed in production. Let's keep going.

🧪 Test 2 - Given an empty cache and a external service outage, when fetching the cache, it should throw an `Exception`

@Test
void givenEmptyCache_whenRemoteServiceReturns500_shouldThrowException() throws InterruptedException {
    LoadingCache<String, String> cache =
            CacheFactory.buildCacheWithoutExceptionHandling(wireMockServer.baseUrl());

    setupFailedResponse();

    var thread = new Thread(new CacheReader(cache));
    thread.start();
    thread.join();
}

Which reports the following logs:

1793 [Thread-8] INFO  org.example.CacheReader [] - Reading value at 1637944243058
1807 [Thread-8] INFO  org.example.CacheFactory [] - Fetching data
2070 [Thread-8] ERROR org.example.CacheReader [] - Exception while loading value at 1637944243341
java.util.concurrent.ExecutionException: com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
 at [Source: (String)""; line: 1, column: 0]
... <omitted stacktrace> ...

In this case, the calling thread is getting a serialization Exception since the response body was not the expected one.

Again, since we didn't see any Exception in our logs, we need to continue the investigation.

🧪 Test 3 - Given a populated cache due for refresh and a slow network connection, when fetching the cache, it should get the old value

@Test
@SneakyThrows
  void givenCacheDueForRefresh_whenRemoteServiceTimeout_shouldGetOldValue() {
    var cache = CacheFactory.buildCacheWithoutExceptionHandling(wireMockServer.baseUrl());

    setupRemoteResponse(new Data("old_value"));

    var thread = new Thread(new CacheReader(cache));
    thread.start();
    thread.join();

    waitForCacheToExpire();

    setupRemoteResponse(new Data("new_value"), 2 * Constants.TIMEOUT_IN_SECONDS);

    thread = new Thread(new CacheReader(cache));
    thread.start();
    thread.join();
}

Which reports the following logs:

1834 [Thread-8] INFO  org.example.CacheReader [] - Reading value at 1637945326283
1846 [Thread-8] INFO  org.example.CacheFactory [] - Fetching data
2099 [Thread-8] INFO  org.example.CacheReader [] - Value Data(value=old_value) at 1637945326554
3114 [Thread-9] INFO  org.example.CacheReader [] - Reading value at 1637945327569
3115 [Thread-9] INFO  org.example.CacheFactory [] - Reloading
3115 [Thread-9] INFO  org.example.CacheFactory [] - Fetching datacom.google.common.cache.LocalCache$Segment$1 run
WARNING: Exception thrown during refresh
java.util.concurrent.ExecutionException: java.net.http.HttpTimeoutException: request timed out
    at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566)
4296 [Thread-9] INFO  org.example.CacheReader [] - Value old_value at 1637941240139
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:527)
... <omitted stacktrace> ...

In these logs, we see that the thread that attempts to load the value after the cache is due for refresh will block (we will follow up on this later) in attempt to reload the new value. However, if that fails, it will get the old value and an Exception will be logged in the background.

When we look into the Guava documentation for the type of cache we are using (refreshAfterWrite), we find the following:

If an exception is thrown while refreshing, the old value is kept, and the exception is logged and swallowed.

This explains why we never saw the Exception in the logs. It was logged by Guava and shallowed which was being filtered out when our app sends those logs to NewRelic. This was a decision we made to avoid polluting NewRelic with logs that we don't care about but it ended up biting us in the arse 🤦

Mystery 1 solved.

❓ Mystery 2 - Request thread hanging

While troubleshooting the previous issue, we found that, when the cache is due for refresh, the thread that attempts to fetch the new value will block and, if it fails, it will eventually get the old value.

The next thing to figure out is what happens if many theads attempt to get the value from the cache at the same time when it is due for refresh

🧪 Test 4 - Given a populated cache due for refresh and a slow network connection, when many threads attempt to fetch the cache, they should all get the old value.

@Test
@SneakyThrows
  void givenCacheDueForRefresh_whenRemoteServiceTimeout_allThreadsShouldGetOldValue() {
    var cache = CacheFactory.buildCacheWithoutExceptionHandling(wireMockServer.baseUrl());

    setupRemoteResponse(new Data("old_value"));

    var thread = new Thread(new CacheReader(cache));
    thread.start();
    thread.join();

    waitForCacheToExpire();

    setupRemoteResponse(new Data("new_value"), 2 * Constants.TIMEOUT_IN_SECONDS);

    runManyAndWait(cache, 3);
  }

Which reports the following logs:

1861 [Thread-8] INFO  org.example.CacheReader [] - Reading value at 1637945351638
1871 [Thread-8] INFO  org.example.CacheFactory [] - Fetching data
2118 [Thread-8] INFO  org.example.CacheReader [] - Value Data(value=old_value) at 1637945351899
3132 [Thread-11] INFO  org.example.CacheReader [] - Reading value at 1637945352913
3132 [Thread-10] INFO  org.example.CacheReader [] - Reading value at 1637945352913
3132 [Thread-9] INFO  org.example.CacheReader [] - Reading value at 1637945352913
3132 [Thread-9] INFO  org.example.CacheFactory [] - Reloading
3132 [Thread-10] INFO  org.example.CacheReader [] - Value Data(value=old_value) at 1637945352913
3132 [Thread-9] INFO  org.example.CacheFactory [] - Fetching data
3132 [Thread-11] INFO  org.example.CacheReader [] - Value Data(value=old_value) at 1637945352913com.google.common.cache.LocalCache$Segment$1 run
4185 [Thread-9] INFO  org.example.CacheReader [] - Value old_value at 1637942442820
WARNING: Exception thrown during refresh
java.util.concurrent.ExecutionException: java.net.http.HttpTimeoutException: request timed out
... <omitted stacktrace> ...

Here, we observe that the unlucky Thread-9 blocks to fetch the new value while all the other threads get the old value and move on with their lives.

This gives us enough information to conclude what happened in production!

When our QA was running their regression test, their requests would often be the ran by an unlucky thread that gets blocked while fetching the new value. Since we have multiple instances of this application, this could be reproduced easily.

Funny enough, when multiple people were trying at the same time, if they were lucky enough to hit the same instance, everything would work just fine, which is what we saw in the logs.

Mystery 2 solved!

Are we done?

Not quite!

We have learnt that, when the cache is due for refresh, there is always an unlucky thread that has to pay the latency cost of refreshing the value.

To provide the best experience for our users, we should make sure the cache refresh takes place asynchronously and that all requests get the old value while the new one is being fetched.

So ... how do we do that? 🤔

Actually fixing the problem

If we look again into Guava's documentation we can see a couple of comments about asynchronous reload:

[...] As specified in LoadingCache.refresh(K), refreshing a key loads a new value for the key, possibly asynchronously [...]

[...] If CacheLoader.reload is implemented to be asynchronous, then the query will not be slowed down by the refresh [...]

Unfortunately, they forgot to add an example on how to do it! 😅

It's alright, 💩 happens. A couple of DuckDuckGo searches later, we found that we could achieve this by configuring the cache to use asyncReloading:

public static LoadingCache<String, Data> buildCacheWithAsyncLoading(
      String baseUrl, ExecutorService pool) {
    return CacheBuilder.newBuilder()
        .refreshAfterWrite(Constants.CACHE_REFRESH_TIME_MS, TimeUnit.MILLISECONDS)
        .build(CacheLoader.asyncReloading(CacheFactory.buildCacheLoader(baseUrl), pool));
}

In this case, the refresh will take place on a thread from a separate thread pool we have to pass on during instantiation.

To validate this works as expected, we write another test 🧪:

@Test
@SneakyThrows
void solution_whenFutureResponsesTimeoutAndAsyncLoading_returnOldValue() {
    ExecutorService pool = Executors.newFixedThreadPool(10);
    try {
      var cache = CacheFactory.buildCacheWithAsyncLoading(wireMockServer.baseUrl(), pool);

      setupRemoteResponse(new Data("old_value"));

      runAndWait(cache);

      waitForCacheToExpire();

      setupRemoteResponse(new Data("new_value"), 2 * Constants.TIMEOUT_IN_SECONDS);

      runManyAndWait(cache, THREAD_COUNT);
    } finally {
      Thread.sleep(10000);
      pool.shutdown();
    }
}

Which reports the following logs:

1772 [Thread-8] INFO  org.example.CacheReader [] - Reading value at 1637945095606
1783 [Thread-8] INFO  org.example.CacheFactory [] - Fetching data
2032 [Thread-8] INFO  org.example.CacheReader [] - Value Data(value=old_value) at 1637945095871
3049 [Thread-9] INFO  org.example.CacheReader [] - Reading value at 1637945096888
3049 [Thread-10] INFO  org.example.CacheReader [] - Reading value at 1637945096888
3050 [Thread-11] INFO  org.example.CacheReader [] - Reading value at 1637945096889
3050 [Thread-11] INFO  org.example.CacheReader [] - Value Data(value=old_value) at 1637945096889
3050 [Thread-10] INFO  org.example.CacheReader [] - Value Data(value=old_value) at 1637945096889
3053 [pool-2-thread-1] INFO  org.example.CacheFactory [] - Reloading
3053 [pool-2-thread-1] INFO  org.example.CacheFactory [] - Fetching data
3065 [Thread-9] INFO  org.example.CacheReader [] - Value Data(value=old_value) at 1637945096904
4063 [pool-2-thread-1] ERROR org.example.CacheFactory [] - Loading Exception at 1637945097902
... <omitted stacktrace> ...

We can see that there isn't an unlucky thread anymore and that the Exception is reported by the thread in the pool.

Conclusion

When we encounter an issue in production, depending on its severity and impact, we often perform an RCA (root cause analysis) investigation.

The goal of this process is to find the root cause of the problem, fix it and also to come up with ways we can prevent issues like this happening in the future or, at least, improve our automated monitoring to ensure we catch these issues as soon as they happen.

For this particular problem, apart from fixing the bug, we decided to:

Use NewRelic scripted browser synthetics to periodically run a smoke test similar to what QA was doing as part of the regression test. This will allow us to catch user-impacting issues early.
Tweak our request latency alerts. With a similar goal as the one above.
Come up with a plan to implement browser-based E2E automation. This will allow us to catch regressions in internal environments before they reach production.

And that's the end of it all!

If you are curious to see the playground code, you can find it in

AnotherDevBoy / guava-playground

This repository contains some tests that I used to investigate a tricky caching bug.

You can find more information about it in my blog albertowar.me.

View on GitHub

Name your threads

Alberto Guerra González — Sat, 27 Nov 2021 21:08:27 +0000

A few weeks ago, my team decided to start leveraging NewRelic JFR which is a relatively new feature from NewRelic to provide visibility on JFR events, and I noticed something that got me thinking 🤔

Who the hell is creating all these threads?

By the way, you don't need to use NewRelic or JFR to have a look at your threads. Since JVM threads are actually OS threads, you can also view them with ps.

Simply, type ps -ef to find the ID of your java process and then ps -T -p <pid>.

In my case, it gave me the following output (redacted):

  PID  SPID TTY          TIME CMD
    2     2 ?        00:00:00 java
    2     3 ?        00:00:08 java
    2     4 ?        00:10:17 GC Thread#0
                     ...
    2    17 ?        00:10:17 GC Thread#1
    2    34 ?        00:00:01 New Relic Jar A
                     ...
    2    50 ?        00:00:02 pool-3-thread-1
    2    51 ?        00:00:02 redisson-netty-
                     ...
    2    89 ?        00:00:39 pool-5-thread-1
                     ...
    2   109 ?        01:16:09 pool-6-thread-1

Threads everywhere

As our applications grow larger, we start making use of libraries to make external web requests, database queries, etc. Often times, these libraries make use of threads (or thread pools) behind the scenes to improve their throughput.

The JVM assigns threads some default names like the ones we observed above (Thread-%, pool-%-thread-%).

Although these names make sense as default values, they are not as helpful when we try to understand if a particular set of threads is misbehaving.

This could be:

Allocating too much memory.
Taking too much CPU time.
Spinning up too many threads (which also has a memory usage impact).
Deadlocks.

Name your pets

Mike Wazowski once said:

In this case, I would recommend not to follow his advice and name your 🧵

This will save you time in the future since you will be able to use effectively many tools that will help you troubleshooting any performance issues.

How do I do that?

Threads

If you are the one creating the thread, you can simply do:

thread.setName("boo");

Pools

If you are creating the thread pool, the Executors class allows you to pass a ThreadFactory to its different factory methods.

If you combine that with Guava's ThreadFactoryBuilder, you can easily make sure your threads are named with the following:

var factory = 
  new ThreadFactoryBuilder().setNameFormat("boo-%d").build();
var pool = Executors.newFixedThreadPool(1, factory);

Closing words

Once Project Loom lands on the SDK, things may change and this advice may not be as useful, since there will no longer be a 1-1 mapping between OS threads and JVM virtual threads.

However, that will take some time to be widely available and even more so to an LTS version (next LTS is planned for September 2023).

To end, I will go against the pet vs cattle DevOps's mindset and say:

Treat your threads as pets, not cattle ... for now

Mob programming revisited

Alberto Guerra González — Sun, 02 May 2021 19:58:44 +0000

At the beginning of the year, I wrote a post about the first mob programming session that I organized with my team.

The results were promising, yet we never had a chance to use it in real world examples.

That changed last week.

Background story

For the last 6 months, my team has worked on a project that had a very optimistic deadline from day one.

We had to take many shortcuts in order to make it and one of these shortcuts was siloing.

In order to ship the required features on time, we distributed the ownership of the different parts of the application across the team and worked on them separately.

💩 Shit storm

Fast forward to a week ago, we were in a tricky situation. We had:

One component in our application that only 1 developer knew how it worked.
A bunch of new features lined up in our roadmap that we committed to deliver by the end of May.
The need to perform a significant refactor on that component to prepare for those features.
The main SME was off sick.

Mob programming to the rescue!

The situation was less of ideal. This was the result of team debt accumulated over time, and it was time to pay it off.

I brought up the idea of using mob programming for the refactoring work. In my mind, this would improve the bus factor since:

All engineers would work together on the same task at the same time.
Refactoring forces you to understand the code before changing it to ensure it still behaves in the same way after the changes.

Gladly management bought onto the idea and we spent a full week working on it.

Retrospective

💻 Preparation

In order to save valuable time, make sure you share instructions on how to set up the development environment prior to the session.

I forgot to do it, therefore we had to repeat the same steps every time we rotated to a new driver 🤦‍♂️.

➡️ One direction

It is important that all navigators follow the same train of thought.

It's hard follow the conversation when different people are trying to tackle different parts of the problem, specially if you are the driver.

⚖️ Facilitation is key

I found it really hard to contribute to the session and facilitate it at the same time. However, having a facilitator is critical to ensure you make the most of your time.

The facilitator will be in charge of the following.

🎯 1. Defining a goal for the session

At the beginning of the session, the facilitator should state what you want to achieve by the end of the day.

This will help to prioritize the areas of focus and when it is OK to move on.

☮️ 2. Creating a safe environment

As a facilitator, you need to ensure everyone in the room has an opportunity to share their ideas.

In every team, there will be people with stronger voices in the room. As a facilitator, you can intervene with prompts like:

Hey X (person who has been silent), what are your thoughts on Y (the problem at hand)?

Excuse me X (person who's speaking over somebody else), I would like to hear what Y has to say, I think they have a valid point.

Occasionally, people may get hung up on some details, react sarcastically, make negative remarks about the code that's been written or the driver.

This can be quite distracting and some people might not feel safe sharing their ideas in such an environment.

If you, as the facilitator, observe this is happening, you should also intercede in a respectful way:

Apologies for interrupting, I would like to kindly ask that we leave sarcasm or non-constructive feedback outside of this session. It is distracting some members of the mob and we all want to get the most out of this session.

Then, to avoid an awkward silence, you can quickly redirect the attention back to code:

So, Y (navigator), how do you think we should solve X (problem at hand)?

🛣️ 3. Staying on the right track

Although one of the main values of mob programming is the discussion that takes place while multiple people are working on the same piece of code, it is also important to ensure the mob doesn't get stuck with things that are not relevant.

As developers, we sometimes get hung up with details such as variable names, libraries to use, etc.

If this happens, the facilitator should intervene and bring the session back on track. Here are some ways you could do it:

Hey folks! I think you all have valid points. Also, we have been discussing this point for a while now, can we compromise on one solution and move on to the next problem?

It seems this topic needs more discussion. Should we keep track of this item and come back to it at a separate meeting?

📝 4. Track decisions

During the session, the mob will make decision that impacts the feature or product that's been developed.

This could be anything from the programming language to use, structure of the code, picking a library to solve a problem, the architecture of the system ...

When that happens, the facilitator should make sure that such a decision and the reason behind it are tracked.

For this, you could use an architectural decision record.

ℹ️ 5. Progress updates

Mob programming is still a fairly new way to solve problems and many people are still unfamiliar with its value.

An easy way to keep everyone informed is to send a recap at the end of the mail.

Here, you can share:

What the mob accomplished during the session.
What is still left to do.
Any questions or potential blockers that might have popped up during the session and need to be addressed separately.

Conclusion

Having a practical example where we could apply this way of solving problems was quite illustrative.

Not only we achieved our goal of sharing knowledge across other engineers, but we also learnt more about mob programming and what type of problems are best suited for it.

With that said, there are still some details that aren't quite clear to us yet.

Should the driver also be part of the discussion?
How often should we rotate?
Should we involve people from other disciplines?
What's the optimal number of engineers?

Next steps

To continue discovering more about mob programming, I plan to:

👀 Continue looking for more opportunities to put it into practice at work.
📖 Read The Mob Programming Guidebook and Remote Mob Programming to see if I can find answers for the questions above.

Setting goals in the middle of a pandemic

Alberto Guerra González — Mon, 11 Jan 2021 20:52:34 +0000

In a previous post, I looked back at my 2020 goals and what I managed to achieve by the end of the year.

I have been going through that exercise for a few years now. Nonetheless, 2020 taught me a lesson when it comes to goal setting in a volatile context and I wanted to share that with you.

Setting goals

Before I get to my recommendations for goal setting in the middle of a pandemic, first I will describe my approach to setting goals in general.

1 - Areas of improvement

I always start the process defining areas that I want to improve on during the year. These could be your finances, health, relationships, work, etc.

You can choose as many as you want with the caveat that, the more areas you choose the less focused you will be. This means you will make some progress in every area but you won't be able to make big changes.

If this is the first time you are setting goals, I would suggest you start small. Pick one or two areas where you can make a big impact.

2 - GROW

Once you have picked your areas of focus, the next step would be to define your goals and how you will achieve them.

For this, I usually follow the GROW model. This framework has been used in coaching for many years and has proven to be fairly successful.

Goal 🏁

Start by defining what it is that you want to achieve.

Coming up with "good" goals can be challenging. In the next section, I will share more details on how to do so.

Reality 🌍

Before you can elaborate a path to achieve your goal, you need to understand where you are now towards that goal. This will help you assess how much time you will actually need to meet the goal.

It might be useful to find a buddy that can give you their objective point of view in the matter. For career development, your manager or someone more senior would be an ideal candidate for this. For personal growth, a friend or a partner can also help.

Options ⚖️

Now that you know where you want to get and your starting point, it is the time to brainstorm ideas on how you can get there.

Most of the time, there are multiple paths you can take. This is your opportunity to evaluate their pros/cons so that you can make an informed decision.

Way forward 🛣️

Finally, it is the time for you to narrow down the options and choose the best one.

Depending on the scope of your goal or the amount of time you have to complete it, you might need to more than one action item from the list you came up with in the previous step.

As an example, imagine your goal (G) is to lose weight by the end of the year. In particular, 10 kg. You current weight is 100 kg (R).

There a few ways you can tackle this goal (O): exercise, changing your diet, going on caloric deficit, etc.

Since you enjoy playing tennis, you decide to join a club and play 3 times a week (W).

If you want to learn more about coaching or the GROW model, I recommend you read Coaching for Performance.

SMART goals

A nuance I introduce as part of the GROW model is the use of the SMART criteria when it comes to defining goals.

Specific 🎯

Your goal should be specific. Otherwise, it will be hard to know whether you achieved it or not. In the previous example, we set 10 kg as a weight loss goal which is easy to track.

Measurable 📏

With a measurable goal you can keep track of your progress which will help you stay focused and motivated.

Achievable 👌

The goal should be realistic and attainable.

Les Brown once said:

"Shoot for the moon. Even if you miss, you'll land among the stars".

This is a great quote and I personally follow it when it comes to goals for things that I am passionate about. If you put yourself outside of your comfort zone, you will be able to achieve greater things.

With that said, some of your goals may be about things that you are not so passionate about. In that case, I recommend you play it safe and aim for something that is relatively easy to achieve. Then, if you make it, you can raise the bar and continue on your growth journey.

Relevant 🧐

The goal should be important to you and there should be a reason behind the goal. This will also help you stay motivated and will increase your chances of success.

Following the previous example, you may want to lose weight to be healthier and live longer.

If you don't feel connected to your goal, you will rely only on will power to achieve it. That could be ok for short term goals or if you have a strong will power.

For long term goals, chances are that things will get in the way: a bad night of sleep, stress at work, etc. When that happens, being connected to your goal will help you persist on the efforts and eventually achieve it.

Time bound ⏰

It is important that you set a deadline for your goal. This will not only help you stay motivated but ensure you take the time to evaluate whether you have succeeded or not.

You should choose a deadline wisely.

Give yourself enough time to achieve it.
Feel free to stretch yourself if you are willing to take on the challenge.
The goal must still be relevant once you achieve it.

In the previous example, we picked one year for the weight loss goal. We could have picked 20 years for that goal if we were not sure we could make it.

Yet, if you are already experiencing health issues or if your doctor has given you a recommendation, reaching the goal in 20 years time might be too late.

3 - Goal setting in the middle of a pandemic

You might be thinking ...

How can I come up with goals when everything around me is a mess?

2020 was a crazy year and 2021 won't be any different for the time being. I admit that this volatile environment we live in is not ideal for self development.

With that said, if you still have enough mental energy to do so, here are a couple of recommendations that will help you succeed.

Circle of influence

The Circle of Influence contains the things that concern you that you can do something about.

In the context of goal setting, the idea is to choose goals that you are certain you can have an impact on in this current climate.

As an example, I had a few goals in 2020 that involved traveling as a way to invest in my relationships with friends and family. Unfortunately, because of the travel restrictions, I failed in all of them.

For 2021, knowing that my circle of control has reduced because of COVID-related restrictions, I will look for alternative ways to achieve the same goal: video calls, texting, etc.

Checkpoints

This is an idea borrowed from the concept of sprint retrospectives in agile development.

By having regular checkpoints on your way to the goal, you will have the opportunity to reassess whether the actions you chose during the GROW exercise are still applicable in the new reality.

If you find that you no longer can continue with those actions or they are not as effective as you initially thought, you will still have time to pivot and choose a new way forward. With that said, changing too often might be counterproductive and delay you over the long run.

It is important that you evaluate the tradeoffs before you pivot on a new direction to reach the goal. You might be making progress even though you can't observe it just yet.

Going back to the example, many people who start exercising for weight loss purposes take longer to see results because their bodies go through a rebalancing process. They lose fat at the same time that they gain muscle and their net mass remains the same.

At this stage, changing the strategy completely may turn in a complete waste of effort. It may be more productive to pivot the way to measure progress to a metric that discerns between the two, such as body fat percentage.

2020 Goals in review

Alberto Guerra González — Sat, 02 Jan 2021 15:54:57 +0000

For many of us, the end of the year brings an ideal atmosphere for introspection. I don’t know if it is superstition, the need to escape from unbearably long family dinners and dodge poking questions about our relationship status; or simply the wish to stop repeating the same mistakes over and over again. At the end of 2019, I also couldn’t stop the urge to reflect and plan for the year to come.

2020 was supposed to be not only the beginning of a new year but also the beginning of a new decade and the year I will turn 30. With the extra motivation, I rolled myself in a blanket, found a comfortable spot on the sofa and spent a good couple of hours crafting the perfect list of goals for 2020.

Little had I known what was about to happen.

Dealing with the unexpected

2020 has been a hard year for many of us and a terrible year for many others. I believe it is important we acknowledge this and take the time to heal if we need to.

Personally, I try to focus on the learnings and the positives from this complicated year as a way to cope with it.

The outcome

My list of 2020 goals could be divided in 4 areas of focus.

Personal growth (3 proposed, 1.2 met)

The goals were:

Blog writing. I always wanted to share knowledge through blogs but never dared to do so. I wanted 2020 to be the year where I would overcome my impostor syndrome, step up and just do it. I set a goal of at least 6 blog posts by the end of the year ❔
Learn Go. At the beginning of 2020, my team moved to a project where I needed to use Go for backend development. In order to prepare for that, I aimed to complete Advent of Code in Go by the end of Q1 2020 ✅
Learn Game Development. I have been working for a video game developer company for a few years now, but I have learnt little about what game development is about. I believe it is important I understand the differences between what I do regularly and what game development is so I can collaborate with game teams more effectively. To do so, I aimed to complete the Multiplayer Game Development Mini-Degree in Zenva by the end of the year ❌

Although I didn’t meet the goal of 6 posts, since the goal was to start blog writing, I am happy I was able to take time to write publish my first blog post on mob programming and the post that you are reading now.

When it comes to Go, I managed to learn enough to do what I needed to do at work, hence why I consider this goal met. However, I didn’t get enough time to finish Advent of Code in Go. I uploaded my progress to GitHub in case you are curious to see how far I got.

Unfortunately, this year was quite busy at work with many tight deadlines. So I couldn’t find the time and energy to commit to explore game development.

Health and wellness (5 proposed, 2.5 met)

The previous year, I focused too much on doing great at work and I struggled to disconnect from it. Over time, this affected my sleep, which initiated a butterfly effect. With bad sleep, my energy levels plummeted, which influenced my mood and impacted the people around me. Something had to change, otherwise I was going to end up burning out.

The goals were:

Sleep more. At a minimum, go to bed before 11 PM at least 3 times per week ❔
Exercise more. Go to the gym at least 3 times per week ❌
Maintain reading habit. Reading is one activity that helps me disconnect and relax. In 2019, I aimed to read one book per month and ended up reading about 30. This was something I enjoyed a lot and wanted to continue doing in 2020. For 2020, my goal was also 1 book per month ❌
Watch series & movies. Similarly, this is another activity that helps me disconnect and I wanted to make time for it. At a minimum, I aimed for one movie night per week ✅
Play more games. Towards the second half of 2019, I stopped playing the games we develop in house and explored titles developed by other companies. Playing games is another activity that helps me disconnect, and I had an entire console generation to catch up with. For 2020, my goal was to play at least 3 games developed by other studios ✅

Although the quality of the sleep improved, the number of hours was sometimes not enough. This year, it had more to do with the fact that we have been working with teams across multiple time zones (China and US West coast) which meant we had to start work early and finish late.

With the series of lockdowns, tight deadlines we had to face at work and everything that was going on around the world, I didn’t have enough mental energy to exercise or read.

To keep myself sane, I over-indexed in the other two activities which kept me going throughout the year.

If you are curious, I ended up spending a lot of hours in: The Witcher 3, Dragon Ball Kakarot, Final Fantasy 7 Remake, AC Origins, AC Odyssey and AC Valhalla.

Relationships (6 proposed, 1 met)

If 2020 has taught us anything it was the importance to spend time with our loved ones. Fortunately, this was something that I already wanted to change before the year started.

The goals were:

Travel to India in October. To meet my partner’s family and spend at least one week there ❌
Travel to my hometown in August and spend at least two weeks there ❌
Move in with my partner by July ✅
Spend the end of 2020 also in my hometown. To celebrate my 30th birthday with my friends there ❌
Travel to Lyon to visit friends ❌
Travel to Munich to visit another friend ❌

As you can imagine, I couldn’t meet any of the goals that required traveling. Although flights were still operating, I cancelled all my travel plans as they were non essential.

With that said, I managed to move in with my partner before the lockdowns started. My previous apartment was close to a busy road, which didn’t help with my sleeping problems. So I pushed that goal a bit forward and we ended up moving together in by the end of March.

Finances and spending (3 proposed, 3 met)

My master plan was to move in with my partner in 2020 and, if that went well, start searching for a house to buy together. In order to save for that deposit, there were certain things of my careless spending habits that I had to bring back in line.

The goals were:

Reduce book purchases. In 2019, I bought 22 books for a total of €257. Unfortunately, I only managed to read 13 of those books since I had plenty more in my library that were waiting for me to pick up. For 2020, I wanted to save some money by cutting book purchases down to less than 10. ✅
Reduce takeaway orders. After living alone for many years, I had become a bit lazy when it comes to cooking and relied on Deliveroo more than I should have. Over the year, I made 14 orders and spent about €400 on it. This was another opportunity for me to save some money in 2020 and reduce the number of orders to one per month at most. ✅
Maximize company allowance. My company has many perks I wasn’t fully leveraging in previous years (travel, game, sports, etc). This year I wanted to make sure I maximized them as much as I could. ✅

When it comes to books, I only bought 7 books in 2020 but they were also a bit more expensive, so ended up saving only about €100 there.

Deliveroo was a bit of a different story. The number of orders was also 14 this year, but some of them came from company allowance. The total was €100 over the previous year, but it was shared between two people, since I am now living with my partner. Overall, I think we did better than the previous year considering that we had to cook a lot more in 2020 because of the move to WFH.

Overall, I saved for the new house’s deposit and also realized that the impact of these spending habits to my overall saving power is not that great as I initially thought.

Other achievements

Apart from the goals I laid out at the beginning of the year, I have accomplished other things that I wasn’t initially planning for:

I got promoted to Senior Software Engineer 🥳
We made an offer to buy a house that was accepted and we are in the process of purchasing it 🥳
Contributed to the launch of 3 games 3️⃣✖️🎮
Led an initiative to upgrade my team’s on-call process, including product triage, reduce mis-escalations and onboarding process 🚨

Conclusion

2020 has been a hectic year and not much is going to change until the vaccine has been rolled out across the globe.

With only 7.7 out of 17 (or 12 if we exclude the travel-related goals) met, you might think that I have missed a lot of the goals I signed up for, and you are right.

However, I strongly believe is important to celebrate our small victories, especially when they take place under less than ideal circumstances.

As someone once said:

Life is a marathon, not a sprint …

Every improvement we can make, regardless of how small, it is still a step forward in the right direction.

I hope you enjoyed my 2020 goals story. I’m also planning to share what my 2021 goals will be in a follow up post where I will explain in detail the process I follow to come up with them.

Keep tuned and stay safe!

Lessons learnt from my first mob programming session

Alberto Guerra González — Tue, 29 Sep 2020 09:28:52 +0000

A few months ago, the YouTube recommender algorithm baited me with this presentation about Mob programming from Woody Zuill.

It was the first time I heard about the term, hence I followed my curiosity and gave it a go.

On this post, I am going to walk you through the steps I took to introduce mob programming to my team and what I learnt from it.

First impressions

Wikipedia says that Mob programming is:

Mob programming (informally mobbing) is a software development approach where the whole team works on the same thing, at the same time, in the same space, and at the same computer. [...]

I’m sure you are thinking ...

How the **** can this be productive?
If we don't even do pair programming at work, how can I convince my peers to try this?

That’s at least what I thought after watching the presentation 😅, so I moved on and a few months passed.

Opportunity awaits

Fast forward to a week ago, my team was feeling pretty drained. We had been battling deadline after deadline for the past year, and it has been long since the last time we did something fun together.

This was the perfect moment to bring it up!

I knew everyone would jump on anything that would pull them away from work, but it couldn't be so long that it could raise some manager’s eyebrows.

With that in mind, I booked a one hour meeting in everyone's calendar on a Friday and hoped for the best.

To my surprise, everyone loved the idea (even the managers 😮🎉). It was time to get to work.

Preparation

Mob programming agenda

The key to a successful learning environment is structure - Cara Carroll

The first thing to figure out was the structure of the session. This would be a one hour trial, so I had to make sure I made the most out of the limited time.

I broke the hour down as follows:

10 minutes for introduction. Where I would explain the dynamics of pair programming and the problem we were about to solve.
4x10 minutes blocks for problem solving. This would ensure that a few of us would get a sense of what being a driver is.
10 minutes for discussion. This would allow me to get a sense of what my peers thought of the experiment and understand whether this is something we want to try for real.

Collaboration

Together we are stronger - Lots of people

The next thing to define was how we were going to work together.

Again, because of the time constraint, I decided to play it safe on this one so that the tools wouldn't slow us down: Git, Java, Maven.

During the session, the designated driver would periodically commit to the repository (always making sure that the build was passing before that) and, once they were ready to pass the baton, they would push to the repository.

With that in mind, I created a repository with:

A Maven project with 1 class and 1 test class.
A README with a brief explanation on mob programming, a description of the structure of the session and a list of tools needed to take part.

After that was ready, I added a link to the repository to the meeting invite and shared in Slack, so that everyone could setup before the session.

In case you want to reuse something like this, I have pushed an example repository here.

The problem

Wax on, wax off - Mr. Miyagi

After a bit of browsing, I ran onto a site called Katalyst from Codurance that has a ton of code katas to practice the craft.

I picked a problem called String Calculator. My intention was to keep the focus of the session around getting familiar with mob programming; therefore, I picked a simple problem that would not require a lot of brain cycles for experienced programmers.

We don't want people to start solving the problem in their heads and come with an idea on how to tackle it. We want to solve the problem together. Do not share the problem prior to the session.

3-2-1, action!

To ensure the session would go as expected, I played the role of a facilitator in disguise.

I was the first driver, so that everyone could see what that was about, and continuously probed individuals for ideas on how to continue. This helped us to get the ball rolling and encouraged everyone to be engaged.

Once my driver shift was over, I took a passive approach with small suggestions here and there but making sure that everyone would get their voices heard.

Key takeaways

Onboarding. I wanted to call this one out first because it caught me by surprise. We had a couple of engineers joining the team that week; and I noticed that it was the perfect way to show them how we code and what aspects we care about when improving code.
Bus factor → ∞. With all the engineers working on the same code, the team is more resilient to attrition or people getting sick.
It was productive. I wasn't expecting this one but, with a lot more 👀 on the code, we were able to fly through the kata and almost finish it by the end of the session.
Knowledge sharing. One engineer was not familiar with the Java Stream syntax, and someone else explained how it works. Another engineer suggested using Regex to solve the problem, and everyone pitched in to discuss performance implications. This kept going spontaneously for the entire session.
Planted the seed. Finally, the best takeaway was the fact that everyone started thinking about it. The next day, people already had ideas on how we could incorporate mob programming to specific areas of the next project to tackle 🥳.

DEV Community: Alberto Guerra González

Can't cache this 🎶

Introduction

The beginning of it all

Investigation

Diving into the code

Setting up the playground

❓ Mystery 1 - Exception MIA

🧪 Test 1 - Given an empty cache and a slow network connection, when fetching the cache, it should throw an Exception

🧪 Test 2 - Given an empty cache and a external service outage, when fetching the cache, it should throw an Exception

🧪 Test 3 - Given a populated cache due for refresh and a slow network connection, when fetching the cache, it should get the old value

❓ Mystery 2 - Request thread hanging

🧪 Test 4 - Given a populated cache due for refresh and a slow network connection, when many threads attempt to fetch the cache, they should all get the old value.

Are we done?

Actually fixing the problem

Conclusion

AnotherDevBoy / guava-playground

Name your threads

Threads everywhere

Name your pets

How do I do that?

Threads

Pools

Closing words

Mob programming revisited

Background story

💩 Shit storm

Mob programming to the rescue!

Retrospective

💻 Preparation

➡️ One direction

⚖️ Facilitation is key

🎯 1. Defining a goal for the session

☮️ 2. Creating a safe environment

🛣️ 3. Staying on the right track

📝 4. Track decisions

ℹ️ 5. Progress updates

Conclusion

Next steps

Setting goals in the middle of a pandemic

Setting goals

1 - Areas of improvement

2 - GROW

Goal 🏁

Reality 🌍

Options ⚖️

Way forward 🛣️

SMART goals

Specific 🎯

Measurable 📏

Achievable 👌

Relevant 🧐

Time bound ⏰

3 - Goal setting in the middle of a pandemic

Circle of influence

Checkpoints

2020 Goals in review

Dealing with the unexpected

The outcome

Personal growth (3 proposed, 1.2 met)

Health and wellness (5 proposed, 2.5 met)

Relationships (6 proposed, 1 met)

Finances and spending (3 proposed, 3 met)

Other achievements

Conclusion

Lessons learnt from my first mob programming session

First impressions

Opportunity awaits

Preparation

Mob programming agenda

Collaboration

The problem

3-2-1, action!

Key takeaways

🧪 Test 2 - Given an empty cache and a external service outage, when fetching the cache, it should throw an `Exception`