DEV Community: Willian Antunes

Query compressed logs that are stored in S3 using AWS Athena

Willian Antunes — Mon, 23 May 2022 11:54:17 +0000

Recently someone asked me to create an easy way to consult all the logs stored in S3. Unfortunately, the person who was trying to check all the log files couldn't consult them suitably because of the following:

20.3 GB of data compressed with GZIP.
Each file has more than 40 thousand lines.
Many folders, with each containing various compressed files.

Moreover, those logs have been exported from a Log Group that is not available on CloudWatch anymore; otherwise, we could use Log Insights right away 😅.

Ideas that I had at first glance

Some options that I thought of when I received the request:

Restore from S3 to a Log Group: I'd have to create a serverless function that would read all the objects in S3, check if one is a GZIP, if true, then uncompress it, read the log file and send each line using The PutLogEvents API to the Log Group.
Download files from S3 and fill them into Elasticsearch: Simply put, I'd expend some time programming something to understand the underlying data from the logs to populate an index in Elasticsearch. A simple LOREM would be enough to do the ETL followed by the Kibana startup afterward.
Use some ETL service from AWS and push what has been processed to a Log Group: AWS Glue is the service that can be used for this purpose. So it's another option to make everything inside the cloud itself.

Before executing one of these options, I sought cookbooks, lessons learned, and tutorials to help me with something that I couldn't grasp at first. That's when I met AWS Athena 😍! If you look at its overview, you can get a quick idea of what it may deliver to you:

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.

But how would I query the bucket with the compressed log files 🤔?

Understanding AWS Athena and applying a solution

When I found an AWS blog post describing an example of how to consult access logs, I quickly get the point! For my case, basically, the plan was to arrange the following topics:

Identify the folder in your bucket where you'd like AWS Athena to analyze to create the table. It's important to mention that it will look over all objects recursively.
Understand a REGEX pattern that can capture each line of your log file in a meaningful way (below, I show you my occasion).
Define the table schema to match the capturing group you configured and some extra metadata to inform AWS Athena to create everything properly.

The first was relatively easy to handle. My folder structure is something like the following:

bucket-name/
│
└── 0smpprubzz/  <--- API Gateway ID
    └── 2574f75d-7a93-4387-bed3-2ea5f4e2be59/  <--- folder with many other folders
        ├── 00107dc08cd17d3ea18805491763048c/  <--- sort of aggregation of log files
        |   ├── 000000.gz  <--- Compressed log file 0
        |   ├── 000001.gz  <--- Compressed log file 1
        |   └── 00000X.gz  <--- Compressed log file X
        |       └── 00000X <--- If you open the compressed log file, you get this file. It has no extension!
        ├── 00249903990fb8d8cc29b88e4b0d3a1a/  <--- sort of aggregation of log files
        |   ├── 000000.gz  <--- Compressed log file 0
        |   ├── 000001.gz  <--- Compressed log file 1
        |   └── 00000X.gz  <--- Compressed log file X
        |       └── 00000X <--- If you open the compressed log file, you get this file. It has no extension!
        └── X/  <--- there are more than 1000 folders like this

Thus I had to use the following:

s3://bucket-name/0smpprubzz/2574f75d-7a93-4387-bed3-2ea5f4e2be59/

For the second topic, I just had to look at some entries from a random log file to recognize a pattern. For instance:

2021-06-18T16:41:50.920Z (3e08ac47-b24b-417f-9d8c-13a4bb335ac3) Starting execution for request: Ze08ac47-b24b-417f-9d8c-13a4bb335ac3
2021-06-18T16:41:50.920Z (3e08ac47-b24b-417f-9d8c-13a4bb335ac3) HTTP Method: OPTIONS, Resource Path: /jafar-application/v1/lamps
2021-06-18T16:41:50.920Z (3e08ac47-b24b-417f-9d8c-13a4bb335ac3) Method request path: {}
2021-06-18T16:41:50.920Z (3e08ac47-b24b-417f-9d8c-13a4bb335ac3) Method request query string: {}
2021-06-18T16:41:50.920Z (3e08ac47-b24b-417f-9d8c-13a4bb335ac3) Method request headers: {Origin=https://agrabah.gov.eg, sec-fetch-mode=cors, X-Akamai-SR-Hop=1, Akamai-Origin-Hop=2, sec-fetch-site=same-site, Accept=*/*, Referer=https://agrabah.gov.eg/, User-Agent=Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G570M) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/14.0 Chrome/87.0.4280.141 Mobile Safari/537.36, X-Forwarded-Proto=https, Host=api.agrabah.gov.eg, Accept-Encoding=gzip, Pragma=no-cache, True-Client-IP=X.P.219.112, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=X-60ccccce-P84a5379108e3a2773817ca1, Via=1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost), X-Akamai-CONFIG-LOG-DETAIL=true, Access-Control-Request-Method=GET, Cache-Control=no-cache, max-age=0, Access-Control-Request-Headers=authorization,x_etag, X-Forwarded-For=X.P.219.112, X.P.63.4, X.P.247.39, Accept-Language=pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7, sec-fetch-dest=empty}

Then I came up with the following regex (see it on RegEx website):

^(\d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}[+-\.]\d{1,3}Z?) (\(.+?\)) (.+)$

Check out the value of each group from a sample line:

It extracts three groups; hence we can use them to finally define the table schema. Through the User Guide, which describes how "create table" works, I wrote the following DDL statement:

CREATE EXTERNAL TABLE `jafar_database.agrabah`(
  `requestdatetime` STRING,
  `identifier` STRING,
  `message` STRING)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
  'input.regex'='^(\\d{4}-\\d{2}-\\d{2}T\\d{2}\\:\\d{2}\\:\\d{2}[+-\\.]\\d{1,3}Z?) (\\(.+?\\)) (.+)$')
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  's3://bucket-name/0smpprubzz/2574f75d-7a93-4387-bed3-2ea5f4e2be59/'

In my case, STRING for all columns wasn't a problem (if it will ever be, I update this article, but you should know that you can use many other types, like TIMESTAMP). I didn't specify the compression type because it will get it by the file extension by default. By the way, if you look at the regex again, I had to use double backslash because RegexSerDe follows the Java Standard: the backslash is an escape character in the Java String class.

Given you created everything accordingly, you can use some DQL and explore your data!

Conclusion

This approach to consulting data is compelling, but it's worth mentioning that it can become expensive pretty quickly though. Also, it depends on the requirements and how a person will use it. Therefore some care with performance concerns is fundamental.

Posted listening to Shout, Disturbed 🎶.

Discover issues with performance testing

Willian Antunes — Mon, 16 May 2022 12:22:06 +0000

It's incredible when your back-end project is well implemented in terms of unit and integration tests, covering most of the business rules that your application should respect to deliver what is expected for those who will use it. Now the moment has come! You think it's prepared to be deployed in production, and when you do, you start seeing some problems 😵‍💫 such as:

The cache is not working correctly.
The REST API is too slow for most of the requests.
The database uses 100% of the CPU during peak hours, but it seems dead even in some basic circumstances.
Some users receive data from other users, which could denote race conditions (or perhaps even worse things).

One way to mitigate these issues (and many others possibly) is doing performance testing. I'm not saying it will save you. Instead, it's another layer of protection that you can have beyond a code review process, let's say. But, of course, it may have some problem if it's not correctly implemented, so it will all depend. Here's an image that illustrates some gains that you have depending on which approach you're using (there are many, but I'm showing just some):

This is known as the test pyramid. As you can see above, you can identify bottlenecks with performance testing and some other situations. Before doing anything to create a test plan, it's a good idea to start with the functional requirements of your project, if it has one.

Gathering functional requirements

Before creating your performance testing script, it's a good idea to start answering how many requests your application should handle (in the case of a REST API), given its business requirements. To illustrate, imagine that your application is an ad-hoc one specially built to deal with incoming requests from your partners, and it should be capable of working with 3,500 requests per minute. In addition to that, your app should answer each request until the limit of 500 ms. There we go! We have something to start our script 😄. By the way, I'm roughly speaking here. I'm just giving a fictional example. Sometimes you might not even have this information 😬, which is why some measurements are required to retrieve that first. You can check out many cases from the setting performance goal on the Wikipedia article about the related topic.

Now, if you are in a condition where you have nothing, literally, then it's a good idea to start with the APDEX (Application Performance Index); besides, even if you have many cases, it's always fundamental to use APDEX.

Apdex: Measure user satisfaction

According to the APDEX official website:

The Apdex methodology was first described by Peter Sevcik in 2004. The objective was to have a simple and uniform open standard to report on application performance regardless of how it was measured.

The following is from Newrelic (which I think it's easier to understand):

It's a simplified Service Level Agreement (SLA) solution that helps you see how satisfied users are with your app through metrics such as Apdex score and dissatisfaction percentage instead of easily skewed traditional metrics such as average response time.

Now that we understand a few of what it is, which tool can we use? Furthermore, how can we apply it in our working flow 🤔?

JMeter

There are many tools out there like Gatling, Locust, and K6, but here I'll explain just a bit of JMeter. It is totally visual and pretty easy to follow and understand 🤗; it also includes a module that generates dashboards that gives us an APDEX result, which is what we want. By default, I have it on my system; you can use the same idea or simply download it from the official website and run it afterward.

Sample project and creating a script

We'll use the REST API Version of the Tic Tac Toe game that I have written using C#! Now let's see the performance test script. Ah! It's worth mention that I won't explain in detail each configuration step. JMeter has an excellent assistant feature. You just add a resource and click on the help icon. It will redirect you to the manual related to the same resource you added. It's awesome 😛. Thus, if you don't understand a given component, either go to the help page or comment below that I will happily help you 🤟.

OK! Download this file and open it on your JMeter. You should see something like this:

The thread group is configured to meet our fictional functional requirement, including a safe margin. Thus we have the following:

775 users (threads). Each user will execute 10 requests (see the green dropper icon symbolizing an HTTP request), which means we will have 7750 samples.
All the users will make their first request in 60 seconds. After this ramp-up, the test will be fully up to speed.
Each user can live up to 120 seconds.

By default, JMeter has the following parameters configured, which are used for the APDEX formula:

apdex_satisfied_threshold: Sets the satisfaction threshold for the APDEX calculation in ms. By default is 500.
apdex_tolerated_threshold: Sets the tolerance threshold for the APDEX calculation in ms. By default is 1500.

If we run it (see this script), we may receive the following result:

This means that our test was asserted as expected 🚀! So now we get back to the question I wrote here previously: how do we implement it in our workflow? Our next topic 👀!

Simple workflow to start with

I believe half a loaf is better than none. So starting from this idea, with the help of containers, given our performance testing script is done and well tested, we can create a compose service to execute the test plan so that the result is available in some format. This output can be used to assert that the result matches our expectations. Look at this simple C# code that we can use to verify the result:

public class PerformanceTestingResultTest
{
    private JObject _generalStatistics;
    private JObject _applicationPerformanceIndex;

    [SkippableFact(DisplayName = "Should have no error in all requests")]
    public void TestScenario1()
    {
        // Arrange
        PreparePropertiesAndSkipTestIfNeeded();
        // Assert
        _generalStatistics.Count.Should().Be(11);

        foreach (var token in _generalStatistics)
        {
            var errorCount = token.Value["errorCount"].As<int>();
            errorCount.Should().Be(0);
        }
    }

    [SkippableFact(DisplayName = "Should APDEX result be greater than 0.9 to all evaluated requests")]
    public void TestScenario2()
    {
        // Arrange
        PreparePropertiesAndSkipTestIfNeeded();
        var rawOverallResult = _applicationPerformanceIndex["overall"]["data"];
        var rawRequestsResults = _applicationPerformanceIndex["items"];
        // Assert
        rawOverallResult.Should().NotBeNullOrEmpty();
        rawRequestsResults.Count().Should().Be(10);

        foreach (var token in rawRequestsResults)
        {
            var apdexResultForGivenRequest = token["data"][0].Value<double>();
            apdexResultForGivenRequest.Should().BeGreaterThan(0.9);
        }
    }

    private void PreparePropertiesAndSkipTestIfNeeded()
    {
        // Skip test if required
        var value = Environment.GetEnvironmentVariable("EVALUATE_PERFORMANCE_TESTING");
        bool.TryParse(value, out var shouldNotSkipTest);
        Skip.If(shouldNotSkipTest is false, "It was informed to be skipped");
        // General statistics
        _generalStatistics = FileHandler.ReadFileAsDictionary("tests-jmeter/statistics.json");
        // APDEX sadly is not in only one file, we should extract it
        var lineWhereApdexResultIs = @"createTable($(""#apdexTable"")";
        var regexPatternToCaptureResult = @", ({"".+]}), ";
        var dashBoardFile = FileHandler.EnumerableFromFile("tests-jmeter/content/js/dashboard.js");
        foreach (var line in dashBoardFile)
        {
            if (line.Contains(lineWhereApdexResultIs))
            {
                var match = Regex.Match(line, regexPatternToCaptureResult);
                var rawApdexResult = match.Groups[1].Value;
                _applicationPerformanceIndex = JObject.Parse(rawApdexResult);
            }
        }
    }
}

If something is wrong, the test fails. Thus we can use it to break on purpose our pipeline, avoiding bugs in production 🤩. Looking at the code above, you may have noticed an environment variable on it. It can trigger the test execution or not because you normally wouldn't have the outputs from JMeter. That is especially important when you want to control whether this test should be executed or not.

Conclusion

When I started my career as a developer, I wish I'd had this advice right from the start, so that's why I created this post. JMeter can help you learn even more programming topics such as concurrency, parallelism, threading, coroutines, and many others. In addition, it opens a new broad set of tools that are at our disposal. I hope it can help you in any way and, as always, feel welcome to post any question you have.

Posted listening to Time (Clock Of The Heart), Culture Club.

RegEx, Data Classes and Type Hints with Python: Learning from tweet text

Willian Antunes — Mon, 02 May 2022 12:28:14 +0000

Before we jump into the code, let's first define our fictional business rule ✍:

After the mention, the word "concedes" must be present. It is identified as the trigger keyword.
After the keyword, at least one hashtag must be present. The tweet may contain more than one if the user wishes to.
The hashtag must be written using Upper Camel Case (also known as Pascal Case) to identify as separate words to create slugs. For instance, #VillainJafar turns into villain-jafar.

Thus if we receive something like @GenieOfTheLamp concedes #FirstWish #SecondWish we should get two hashtags and their slugs: first-wish and second-wish, as well as if the tweet is valid or not.

Starting from the test

It's pretty typical to start from the tests when you have a well-defined business rule. This approach of development is known as TDD. If we consider the sample given with #VillainJafar:

def test_should_evaluate_text_as_valid_given_keyword_and_one_hashtag_presence():
    # Arrange
    sample_tweet = "@GenieOfTheLamp concedes #VillainJafar"
    # Act
    result = None
    # Assert
    assert result.is_valid
    assert result.hashtags == ["VillainJafar"]
    assert result.slugs == ["villain-jafar"]

We can also use the second one:

def test_should_evaluate_text_as_valid_given_keyword_and_two_hashtags_presence():
    # Arrange
    sample_tweet = "@GenieOfTheLamp concedes #FirstWish #SecondWish"
    # Act
    result = None
    # Assert
    assert result.is_valid
    assert result.hashtags == ["FirstWish", "SecondWish"]
    assert result.slugs == ["first-wish", "second-wish"]

Now let's think in terms of flows that don't match what is expected. Some test cases that we can assert:

The text is missing the keyword.
The text has the wrong keyword.
The text is none or empty.

We can write tests like the following:

def test_should_evaluate_text_as_invalid_given_missing_keyword():
    # Arrange
    sample_tweet = "@GenieOfTheLamp #FirstWish #SecondWish"
    # Act
    result = None
    # Assert
    assert not result.is_valid


def test_should_evaluate_text_as_invalid_given_wrong_keyword():
    # Arrange
    sample_tweet = "@GenieOfTheLamp creates #FirstWish #SecondWish"
    # Act
    result = None
    # Assert
    assert not result.is_valid


def test_should_throw_exception_when_text_is_none_or_empty():
    # Arrange
    sample_tweet = "@GenieOfTheLamp creates #FirstWish #SecondWish"
    # Act and assert
    with pytest.raises(Exception):
        # Empty String case
        pass
    with pytest.raises(Exception):
        # None as an argument
        pass

Let's jump into the concrete implementation! By the way, as you may have noticed, I'm using pytest, but you can see how it would be with unittest if you check the repository.

Defining method contract using type hints and data classes

As you've seen in our tests, we didn't define our method contract. Let's start with the following:

def check_text_and_grab_its_details(text: str):
    pass

We defined the argument's type, but what about the returned type? That's a good situation to use Data Classes! From our tests, we know what the expected behavior is. Let's define our data class and use it as the returning type of our method:

@dataclass(frozen=True)
class TextDetails:
    is_valid: bool
    hashtags: Optional[List[str]] = None
    slugs: Optional[List[str]] = None


def check_text_and_grab_its_details(text: str) -> TextDetails:
    pass

The defined data class has the parameter Frozen as true, which means that its instance will be read-only, and any assignment attempt to fields will generate an exception. Learn more about Frozen Instances in Python's documentation. It has two fields as optional and with value none. It allows us to instantiate the data class like the following:

TextDetails(True)
TextDetails(True, ["FirstWish"], ["first-wish"])

The method contract is adequate, thanks to type hints. We can take advantage of it and use mypy to guarantee all the methods contracts, but let's configure it in the end. Let's dive into our method's implementation first.

Method implementation and the RE module

To have a valid regular expression, I used the site RegExr to create it. If you don't know how it works, I recommend the book Piazinho. The regex pattern can be defined in our Python code like the following:

regex_valid_text = re.compile(r".* ?@GenieOfTheLamp concedes ( ?#([a-zA-Z]{1,}))+$")

The method compile does the following according to Python documentation:

Compile a regular expression pattern into a regular expression object, which can be used for matching using its match(), search() and other methods.

We'll use another regex pattern to transform our hashtag into a slug. We just need to identify the upper case letters, ignoring the first matched group to avoid inserting a dash at the beginning. Now that we have our regex expressions determined, it's essential to clean our text before running them. Wrapping everything up, here's our code so far:

@dataclass(frozen=True)
class TextDetails:
    is_valid: bool
    hashtags: Optional[List[str]] = None
    slugs: Optional[List[str]] = None


regex_valid_text = re.compile(r".* ?@GenieOfTheLamp concedes ( ?#([a-zA-Z]{1,}))+$")
regex_camel_case_conversion = re.compile(r"(?!^)([A-Z]+)")


def check_text_and_grab_its_details(text: str) -> TextDetails:
    cleaned_text = strip_left_and_right_sides(text)

    if not cleaned_text:
        raise TextIsFalsyException

We'll use the methods match, findall, and sub; they are all related to regex patterns. The first will check if the tweet is valid, the second will retrieve all matched hashtags with the help of another regex pattern, and the last one will replace a given match to a particular character; in our case, to transform the hashtag to slug. This is the final code:

@dataclass(frozen=True)
class TextDetails:
    is_valid: bool
    hashtags: Optional[List[str]] = None
    slugs: Optional[List[str]] = None


pattern_valid_text = re.compile(r".* ?@GenieOfTheLamp concedes (#[a-zA-Z]{1,} ?)+$")
pattern_hashtags = re.compile(r"(#[a-zA-Z]{1,})")
pattern_camel_case_conversion = re.compile(r"(?!^)([A-Z]+)")


def check_text_and_grab_its_details(text: str) -> TextDetails:
    cleaned_text = strip_left_and_right_sides(text)

    if not cleaned_text:
        raise TextIsFalsyException

    match = pattern_valid_text.match(text)

    if not match:
        return TextDetails(False)

    all_hashtags = pattern_hashtags.findall(text)

    hashtags = []
    slugs = []

    for hashtag in all_hashtags:
        tag = hashtag.replace("#", "")
        almost_slug = pattern_camel_case_conversion.sub(r"-\1", tag)
        slug = almost_slug.lower()
        hashtags.append(tag)
        slugs.append(slug)

    return TextDetails(True, hashtags, slugs)

Running the tests

We just need to replace the following snippets:

# Act
result = None

# Act and assert
with pytest.raises(Exception):
    # Empty String case
    pass
with pytest.raises(Exception):
    # None as an argument
    pass

With these:

# Act
result = check_text_and_grab_its_details(sample_tweet)

# Act and assert
with pytest.raises(TextIsFalsyException):
    check_text_and_grab_its_details("")
with pytest.raises(AttributeError):
    check_text_and_grab_its_details(None)

Now, if we run the tests 🚀:

Bonus: static analysis with mypy

Nowadays, a production-ready Python project must have a static type checker. You can gradually type your project, then make it safer to work with and ship to your environment. One that you can use is mypy; what is it according to the documentation:

Mypy is an optional static type checker for Python. You can add type hints (PEP 484) to your Python programs, and use mypy to type check them statically. Find bugs in your programs without even running them! You can mix dynamic and static typing in your programs. You can always fall back to dynamic typing when static typing is not convenient, such as for legacy code.

I've found many bugs with the help of mypy since I started using it two years ago, so let's make our project a bit better. Let's apply black and isort also. Here's the script to evaluate our project:

#!/usr/bin/env bash

TARGET_PROJECT=regex_dataclasses
TARGET_TEST_PROJECT=tests
TARGET_FOLDERS="$TARGET_PROJECT $TARGET_TEST_PROJECT"

echo "######## ISORT..."
isort $TARGET_FOLDERS --check-only --diff
echo "######## BLACK..."
black --check --diff $TARGET_FOLDERS
echo "######## MYPY..."
# mypy will only target the project folder
mypy $TARGET_PROJECT

If we run it, here's the output:

▶ ./scripts/start-lint.sh
######## ISORT...
######## BLACK...
All done! ✨ 🍰 ✨
7 files would be left unchanged.
######## MYPY...
Success: no issues found in 4 source files
(regex-dataclasses-with-python-learning-from-tweet-text)

Conclusion

If you've been working with strong typing languages and dynamic ones for a time, once you start gradually typing projects, you'll notice how fast you can produce and deliver good quality code. Instead of typing everything, you can create types and apply them to important places on your code. It's been some years that I understood that 100% of pure dynamic code or typed one is a bad thing, depending, of course, in which context you are. Static type checkers enable Python projects to support gradual typing for our luck, and data classes are a fantastic way to help us with it. This technique must be used wisely, or more problems are brought up actually, though.

You can take a look at the whole project on GitHub 🤟. By the way, how about if you modify TextDetails class? Perhaps you can create a dedicated data class to contain the hashtag and its slug and then produce a list of it. There are many ways! Try it out 😜!

Posted listening to It's My Life, Talk Talk.

C# Web API: How to call your endpoint through integration tests

Willian Antunes — Tue, 13 Jul 2021 13:13:08 +0000

It's fantastic when you have all of your unit tests returning green signs everywhere. Still, when you execute your project, it raises an error because an infrastructure setup is wrong, making it impossible to run your Web API properly 😑. It can be how you set up your logs, database, providers, well, many other things. One approach to fix it or minimize its impact is through integration tests. So how can you do a quick setup for that 🤔? By the way, in this blog post, we're going to consider the following technologies:

Describing the sample project

Here's the image that shows a sample flow:

It is composed of three steps:

The user calls the endpoint /api/v1/movies.
The application will do fake processing.
A random movie is returned to the user.

To take care of this business rule, here's our controller:

[ApiController]
[Route("api/v1/[controller]")]
public class MoviesController : ControllerBase
{
    private readonly IFilmSpecialist _filmSpecialist;

    public MoviesController(IFilmSpecialist filmSpecialist)
    {
        _filmSpecialist = filmSpecialist;
    }

    [HttpGet]
    public Movie Get()
    {
        Log.Information("Let me ask the film specialist...");
        var movie = _filmSpecialist.SuggestSomeMovie();
        Log.Information("Suggested movie: {Movie}", movie);
        return movie;
    }
}

Who will be responsible for doing fake processing:

public class FilmSpecialist : IFilmSpecialist
{
    private static readonly Movie[] Films =
    {
        new("RoboCop", "10/08/1987", new[] {"Action", "Thriller", "Science Fiction"}, "1h 42m"),
        new("The Matrix", "05/21/1999", new[] {"Action", "Science Fiction"}, "2h 16m"),
        new("Soul", "12/25/2020", new[] {"Family", "Animation", "Comedy", "Drama", "Music", "Fantasy"}, "1h 41m"),
        new("Space Jam", "12/25/1996", new[] {"Adventure", "Animation", "Comedy", "Family"}, "1h 28m"),
        new("Aladdin", "07/03/1993", new[] {"Animation", "Family", "Adventure", "Fantasy", "Romance"}, "1h 28m"),
        new("The World of Dragon Ball Z", "01/21/2000", new[] {"Action"}, "20m"),
    };

    public Movie SuggestSomeMovie()
    {
        Log.Debug("OKAY! Which film will I suggest 🤔");
        Random random = new();
        var filmIndexThatIWillSuggest = random.Next(0, Films.Length);
        Log.Information("Will suggest the film with index {FilmIndex}!", filmIndexThatIWillSuggest);

        return Films[filmIndexThatIWillSuggest];
    }
}

Let's first do the unit testing to ensure that our methods contracts are being respected.

Starting from unit testing

Here we'll do a simple unit test on the service responsible for returning a random movie. We can write something like the following, as it's not our focus:

public class FilmSpecialistTests
{
    private readonly IFilmSpecialist _filmSpecialist = new FilmSpecialist();

    [Fact]
    public void ShouldReturnRandomMovieWhenAsked()
    {
        // Act
        var suggestedMovie = _filmSpecialist.SuggestSomeMovie();
        // Assert
        var expectedTiles = new[]
        {
            "RoboCop", "The Matrix", "Soul", "Space Jam", "Aladdin", "The World of Dragon Ball Z"
        };
        suggestedMovie.Title.Should().BeOneOf(expectedTiles);
    }
}

Making an actual HTTP request to our API

To call our endpoint, we can use a class fixture with the help of WebApplicationFactory (know more about it at the section Basic tests with the default WebApplicationFactory in Integration tests in ASP.NET Core guide). In our class test constructor, we can use the factory to create a HttpClient, hence allowing us to do HTTP calls to our endpoint. Moreover, let's say you'd like to replace an injected service with a mock: you can do that through ConfigureTestServices. To illustrate a complete example:

public class MoviesControllerITests : IClassFixture<WebApplicationFactory<Startup>>
{
    private readonly IFilmSpecialist _filmSpecialist;
    private HttpClient _httpClient;

    public MoviesControllerITests(WebApplicationFactory<Startup> factory)
    {
        _filmSpecialist = Mock.Of<IFilmSpecialist>();
        _httpClient = factory.WithWebHostBuilder(builder =>
        {
            // https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-5.0#inject-mock-services
            builder.ConfigureTestServices(services =>
            {
                services.RemoveAll<IFilmSpecialist>();
                services.TryAddTransient(_ => _filmSpecialist);
            });
        }).CreateClient();
    }

    [Fact]
    public async Task ShouldCreateGameGivenFirstMovementIsBeingExecuted()
    {
        // Arrange
        var requestPath = "/api/v1/movies";
        var movieToBeSuggested = new Movie("Schindler's List", "12/31/1993", new[] {"Drama", "History", "War"}, "3h 15m");
        Mock.Get(_filmSpecialist)
            .Setup(f => f.SuggestSomeMovie())
            .Returns(movieToBeSuggested)
            .Verifiable();
        // Act
        var response = await _httpClient.GetAsync(requestPath);
        response.StatusCode.Should().Be(HttpStatusCode.OK);
        var movie = await response.Content.ReadFromJsonAsync<Movie>();
        // Assert
        movie.Should().BeEquivalentTo(movieToBeSuggested);
        Mock.Get(_filmSpecialist).Verify();
    }
}

By the way, to use WebApplicationFactory, you must install the package:

Microsoft.AspNetCore.Mvc.Testing

It's pretty simple 🤗. I'll leave it as it is, but we could abstract our integration test to avoid creating a HttpClient every time for each of our class tests 😏.

To end things off

I think you can get enormous benefits from doing a cheap integration test sometimes because, as I told you at the beginning, it can almost guarantee that your code will be shipped as expected at the infrastructure layer. In this article, I gave a somewhat simple example, but things can be more challenging, let's say when it comes to broker connection — the subject of another blog entry 😜.

You can consult the code I showed here on this GitHub repository. You can use Docker Compose to run the project as well as execute its tests. Check the README for more details.

Posted listening to Toy Soldiers, Martika.

GKE Ingress: How to configure IPv4 and IPv6 addresses

Willian Antunes — Tue, 01 Jun 2021 18:34:41 +0000

In the past, when you were about to release a website to be accessed worldwide, you usually would have the following:

A domain where your users would use to access your website.
Reserved IPv4 public address.
A configuration that consists of a DNS entry of type A mapping your IPv4 address to your domain.
Some infrastructure stuff that would deliver your website.

Nowadays, we must have an IPv6 address as well! There are certain areas in the world where IPv4 connections only are not supported anymore due to addresses exhaustion. Let's see how we can do it using Terraform to apply it on GKE Ingress!

Reserving IPv4 and IPv6 addresses

We can use the resource google_compute_global_address:

resource "google_compute_global_address" "gke_ingress_ipv6" {
  name = "external-address-gke-ingress-ipv6"
  ip_version = "IPV6"
  address_type = "EXTERNAL"
}

resource "google_compute_global_address" "gke_ingress_ipv4" {
  name = "external-address-gke-ingress-ipv4"
  ip_version = "IPV4"
  address_type = "EXTERNAL"
}

After their creation, you can check them out by accessing VPC Network and then External IP addresses through the Web Console.

Creating certificate managers

The following GCP resource:

google_compute_managed_ssl_certificate

It allows us to create certificate managers. To illustrate our fictional sample:

resource "google_compute_managed_ssl_certificate" "jasmine_certs" {
  provider = google-beta

  name = "jasmine-certs"

  managed {
    domains = [
      "agrabah.com",
    ]
  }
}

Now we have everything to create our Ingress 🚀.

Creating the GKE Ingress

Here we'll follow the guide Configuring Ingress for external load balancing. First, let's start understanding which annotations we must use.

Static IP address

To use only one of the reserved addresses, we should use the annotation kubernetes.io/ingress.global-static-ip-name. Its description:

Use this annotation to specify that the load balancer should use a static external IP address that you previously created.

By the way, I wrote only one address because, sadly, this annotation supports only one at the current time. We will circumvent that later 😉.

Certificate managers

There are two ways to bind a certificate manager with the Ingress. If you see the guide Using Google-managed SSL certificates, you will see a ManagedCertificate resource; that's not our case. As we created our certificate managers on GCP, we must use the ingress.gcp.kubernetes.io/pre-shared-cert annotation. Its specification:

You can upload certificates and keys to your Google Cloud project. Use this annotation to reference the certificates and keys.

We created only one certificate manager, but let's suppose we had two certificate managers, this annotation would have the following value:

"ingress.gcp.kubernetes.io/pre-shared-cert" = "cert_manager_1,cert_manager_2"

Terraform manifest

Wrapping everything up, this is our resource kubernetes_ingress:

resource "kubernetes_ingress" "sample_ingress" {
  metadata {
    name = "sample-ingress"
    namespace = "production"

    annotations = {
      "kubernetes.io/ingress.global-static-ip-name" = "external-address-gke-ingress-ipv4"
      "ingress.gcp.kubernetes.io/pre-shared-cert" = "jasmine-certs"
    }
  }

  spec {
    rule {
      host = "agrabah.com"
      http {
        path {
          backend {
            service_name = "agrabah-np-service"
            service_port = 8000
          }
        }
      }
    }
  }
}

Now you can execute terraform apply followed by your confirmation. After its creation, if you run the command kubectl -n production get ingress, you'll see something like the following:

▶ kubectl -n production get ingress
NAME               CLASS    HOSTS         ADDRESS             PORTS   AGE
sample-ingress     <none>   agrabah.com   34.X.X.X            80      42d

We're almost there. Now we're at the part where we have to make our hands dirty 😬.

Manual configuration

As this configuration hasn't been supported yet, we have two approaches: either create two Ingresses or configure only one manually. Let's do the latter. When you create an Ingress, a native load balancer is automatically made for you. You can get its name through the command:

kubectl get ingress sample-ingress -o jsonpath='{.metadata.annotations.ingress\.kubernetes\.io/url-map}'

Let's open it on the page Load balancing in Network Services. You'll see the frontend table more or less like the following:

You can click on edit and then click on frontend configuration.

On the panel Frontend configuration, we can click on Add Frontend IP and port and then configure two new entries for ports 80 and 443 for the IPv6 address that is missing. You can base your configuration following what has been set for you automatically. Sample:

After saving the new setup, it should be working accordingly if you access your website either through IPv4 or IPv6.

Possible caveats 🤏

A friend of mine said that this setup wasn't working as expected two years ago because GKE would override what you had done manually. It's been more than one month that I released a project with this approach, and so far, so good. I create some new hosts and certificate managers on the Ingress, and GKE only applied the new configuration and left what had been set intact. Be careful and do your tests as well 👍.

Conclusion

At the end of the blog entry where I posted about how to fix a 502 error returned by GKE Ingress, I described an issue regarding health check configuration that has been opened for over three years. The one I mentioned here to support multiple addresses it's been opened for over four years. I think GKE Ingress is a remarkable resource. It can help you quickly release an application using K8S and cloud-native features wrapped in abstracted manifests, but it seems a bit left aside in some aspects.

You can check the entire code out on GitHub. As always, don't forget to execute terraform destroy after your test! See you next time ✌.

Posted listening to Toy Soldiers, Martika.

GKE Ingress: How to fix a 502 bad gateway error

Willian Antunes — Fri, 28 May 2021 12:56:58 +0000

Past few days, I was configuring a GKE cluster for my personal projects. Some of them had to be accessed through the internet; hence I created an ingress to do this job. Here's an example of ingress using Terraform:

resource "kubernetes_ingress" "default" {
  metadata {
    name = "sample-ingress"
    namespace = "production"

    annotations = {
      "ingress.gcp.kubernetes.io/pre-shared-cert" = "spacejam-cert-manager"
      "kubernetes.io/ingress.global-static-ip-name" = "my-honest-public-ip"
    }
  }

  spec {
    rule {
      host = "spacejam.com"
      http {
        path {
          backend {
            service_name = "spacejam-np-service"
            service_port = 8000
          }
        }
      }
    }
  }
}

The service named spacejam-np-service was bound to a deployment like the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spacejam-web-deployment
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spacejam-web-deployment
  template:
    metadata:
      labels:
        app: spacejam-web-deployment
    spec:
      containers:
        - name: spacejam-web-container
          image: gcr.io/agrabah-project/spacejam:latest-prd
          envFrom:
            - configMapRef:
                name: spacejam-configmap
          ports:
            - name: http
              containerPort: 8000
          livenessProbe:
            httpGet:
              path: "/health-check"
              port: 8000
            initialDelaySeconds: 10
            timeoutSeconds: 5

Notice that it does not have a readiness probe entry 🤔. After creating all of the manifests, I tried to access the host spacejam.com, but I received a 502 error 😯.

Is the service running correctly?

My first idea was to check the service. Is it healthy or not? The command kubectl describe deployments spacejam-web-deployment returned that it was fine, then I ran another command kubectl port-forward deployment/spacejam-web-deployment 8000:8000 so I could access the service directly through http://localhost:8000/health-check. Everything was working as expected. My second idea was to understand how the ingress checks if a service is healthy or not.

Checking backend health status and finding the root cause

When you create an ingress, GKE will create a native load balancer afterward. You can know the LB's name through the command:

kubectl get ingress sample-ingress -o jsonpath='{.metadata.annotations.ingress\.kubernetes\.io/url-map}'

Then I opened its configuration, and I saw that one of my backends was unhealthy 👀:

I could understand the root cause of the error when I noticed which path the backend service was using:

The targeted service returns 404 instead of 200 for the path /.

Understanding GKE behavior

Looking over GKE Ingress for HTTP(S) Load Balancing guide, the section Default and inferred parameters shows us that the ingress will check the POD's spec:

 containers[].readinessProbe.httpGet.path

Then using it value to get the path to verify the service health, but if it's empty, it will use the default value /. Moreover, this statement is important:

When you expose one or more Services through an Ingress using the default Ingress controller, GKE creates a Google Cloud external HTTP(S) load balancer or a Google Cloud internal HTTP(S) load balancer. Both of these load balancers support multiple backend services on a single URL map. Each of the backend services corresponds to a Kubernetes Service, and each backend service must reference a Google Cloud health check. This health check is different from a Kubernetes liveness or readiness probe because the health check is implemented outside of the cluster.

Fixing the problem

To quickly solve it, I added the following to my deployment manifest:

readinessProbe:
  httpGet:
    path: "/health-check"
    port: 8000
  initialDelaySeconds: 10
  timeoutSeconds: 5

Afterward, the backend service was updated to the correct path:

Summing-up

There is a considerable debate on this very topic in the issue Ingress Healthcheck Configuration on kubernetes/ingress-gce repository. Looking through issues on GitHub is an excellent way to understand how an application or service is evolving. It's been more than three years, and it's not resolved yet.

In my next blog entry ✍, I'll explain how I configured IPv4 and IPv6 public addresses using only one ingress on GKE. Till next time!

Production-ready shell startup scripts: The Set Builtin

Willian Antunes — Thu, 27 May 2021 12:09:21 +0000

When you build an application and make it available through container technology, either you have an ENTRYPOINT or CMD instructions at the end of its Dockerfile. Depending on which framework you're using and some requirements you have, sometimes it's better to have a bash script responsible for running your project. When it's available, generally, you'll see a bunch of commands that are executed, like the following script I created for the project Django Multiple Schemas:

#!/usr/bin/env bash

python manage.py makemigrations
python manage.py migrate
python manage.py seed --create-super-user

python manage.py runserver 0.0.0.0:8000

Let's suppose the command python manage.py migrate failed its execution. What would happen 🤔? The answer is counter-intuitive, but the script would run fine, even with the error 🤯. It would execute the command python manage.py seed --create-super-user followed by python manage.py runserver 0.0.0.0:8000. How to fix that? Let's know a bit about some arguments of The Set Builtin.

Exit immediately if any command returns a non-zero status

Let's suppose the following script:

#!/usr/bin/env bash

python the_set_builtin/sample_1.py
python the_set_builtin/sample_2_force_error.py
python the_set_builtin/sample_3.py

If we execute it, we'll have the following output:

▶ docker-compose up why-use-argument-e-with-bug
Creating the-set-builtin_why-use-argument-e-with-bug_1 ... done
Attaching to the-set-builtin_why-use-argument-e-with-bug_1
why-use-argument-e-with-bug_1  | I am the /app/the_set_builtin/sample_1.py!
why-use-argument-e-with-bug_1  | I am the /app/the_set_builtin/sample_2_force_error.py!
why-use-argument-e-with-bug_1  | Let me force an error 👀
why-use-argument-e-with-bug_1  | I am the /app/the_set_builtin/sample_3.py!
the-set-builtin_why-use-argument-e-with-bug_1 exited with code 0

The exit code is 0 😠. Now, if you include the option set -e and execute it again, the output changes, fixing the unexpected behavior:

▶ docker-compose up why-use-argument-e-with-fix 
Creating the-set-builtin_why-use-argument-e-with-fix_1 ... done
Attaching to the-set-builtin_why-use-argument-e-with-fix_1
why-use-argument-e-with-fix_1  | I am the /app/the_set_builtin/sample_1.py!
why-use-argument-e-with-fix_1  | I am the /app/the_set_builtin/sample_2_force_error.py!
why-use-argument-e-with-fix_1  | Let me force an error 👀
the-set-builtin_why-use-argument-e-with-fix_1 exited with code 1

Now the exit code is 1 🥳. According to the documentation about the argument -e:

Exit immediately if a pipeline, which may consist of a single simple command, a list, or a compound command returns a non-zero status.

This argument is quite good! It can protect us from bugs in production. Though it can solve many potential problems, we may need an environment variable. To illustrate, see the PORT:

#!/usr/bin/env bash

python manage.py migrate
python manage.py collectstatic --no-input

gunicorn -cpython:gunicorn_config -b 0.0.0.0:${PORT} aladdin.wsgi

If the variable is not provided, our application might be running without a critical parameter. Is there another argument that can stop the script if this variable is missing?

Treat unset variables as an error when substituting

Now we have the following script:

#!/usr/bin/env bash

python the_set_builtin/sample_1.py
python the_set_builtin/sample_2_force_error.py
python the_set_builtin/sample_3.py
python the_set_builtin/sample_4_env_variable.py "$ALADDIN_WISH"
python the_set_builtin/sample_5.py

If you run, you'll get the following output:

▶ docker-compose up why-use-argument-u-with-bug
Starting the-set-builtin_why-use-argument-u-with-bug_1 ... done
Attaching to the-set-builtin_why-use-argument-u-with-bug_1
why-use-argument-u-with-bug_1  | I am the /app/the_set_builtin/sample_1.py!
why-use-argument-u-with-bug_1  | I am the /app/the_set_builtin/sample_2_force_error.py!
why-use-argument-u-with-bug_1  | Let me force an error 👀
why-use-argument-u-with-bug_1  | I am the /app/the_set_builtin/sample_3.py!
why-use-argument-u-with-bug_1  | I am the /app/the_set_builtin/sample_4_env_variable.py!
why-use-argument-u-with-bug_1  | Look! I received the following arguments: ['the_set_builtin/sample_4_env_variable.py', '']
why-use-argument-u-with-bug_1  | Length: 2
why-use-argument-u-with-bug_1  | I am the /app/the_set_builtin/sample_5.py!
the-set-builtin_why-use-argument-u-with-bug_1 exited with code 0

With the option set -u, here is what you get:

▶ docker-compose up why-use-argument-u-with-fix
Starting the-set-builtin_why-use-argument-u-with-fix_1 ... done
Attaching to the-set-builtin_why-use-argument-u-with-fix_1
why-use-argument-u-with-fix_1  | I am the /app/the_set_builtin/sample_1.py!
why-use-argument-u-with-fix_1  | I am the /app/the_set_builtin/sample_2_force_error.py!
why-use-argument-u-with-fix_1  | Let me force an error 👀
why-use-argument-u-with-fix_1  | I am the /app/the_set_builtin/sample_3.py!
why-use-argument-u-with-fix_1  | ./scripts/with-argument-u.sh: line 10: ALADDIN_WISH: unbound variable
the-set-builtin_why-use-argument-u-with-fix_1 exited with code 1

It's rocking 🤘! Let's see the explanation of -u:

Treat unset variables and parameters other than the special parameters ‘@’ or ‘*’ as an error when performing parameter expansion. An error message will be written to the standard error, and a non-interactive shell will exit.

Including both options in our scripts (set -eu) and we are almost good to go! I wrote "almost" because we can add another layer of protection. Let's see the last one.

A pipeline should produce a failure return code if any command fails

To illustrate, imagine you have the script below. Notice that we're using set -e argument to make it safer:

#!/usr/bin/env bash

# https://www.gnu.org/software/bash/manual/bash.html#The-Set-Builtin
# • -e:  Exit immediately if a command exits with a non-zero status.
set -e

this-is-a-fake-command-my-friend | echo "You...are late."
echo "A thousand apologies, O patient one."

In case the second command fails in the pipeline, here's the output:

▶ docker-compose up why-use-option-pipefail-with-bug
Starting the-set-builtin_why-use-option-pipefail-with-bug_1 ... done
Attaching to the-set-builtin_why-use-option-pipefail-with-bug_1
why-use-option-pipefail-with-bug_1  | ./scripts/without-option-pipefail.sh: line 7: this-is-a-fake-command-my-friend: command not found
why-use-option-pipefail-with-bug_1  | You...are late.
why-use-option-pipefail-with-bug_1  | A thousand apologies, O patient one.
the-set-builtin_why-use-option-pipefail-with-bug_1 exited with code 0

Including the option set -e -o pipefail, the output changes to the following:

▶ docker-compose up why-use-option-pipefail-with-fix
Starting the-set-builtin_why-use-option-pipefail-with-fix_1 ... done
Attaching to the-set-builtin_why-use-option-pipefail-with-fix_1
why-use-option-pipefail-with-fix_1  | ./scripts/with-option-pipefail.sh: line 9: this-is-a-fake-command-my-friend: command not found
why-use-option-pipefail-with-fix_1  | You...are late.
the-set-builtin_why-use-option-pipefail-with-fix_1 exited with code 127

Its explanation:

If set, the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully. This option is disabled by default.

Shield you from working outside working hours

Wrapping everything up, this is the entry your bash script should have at the top: set -eu -o pipefail. Always insert it in all of your scripts, always. This simple protection will help you a lot during TSHOOT 🙏. One example is when you're running your application without a migration that should have been applied.

You can check out all the code I put in this article on GitHub. There you'll find a docker-compose file with all the services testing each of the arguments I showed here 🤙.

The easiest way to run a container on GCE with Terraform

Willian Antunes — Tue, 25 May 2021 20:29:58 +0000

Recently I developed a worker that would listen to published tasks through a database. For this sole purpose, I used Django Q. When I finished its development, I started wondering how I would enable it in production. As my whole stack is on GCP for personal projects, I found an exciting way to deploy it with Google Compute Engine using a container image. Let's see how we can do it pretty quickly with the help of Terraform.

Some limits you must be aware of before we start

There are some limitations that you should pay attention to before spending any time on this solution. These two, in particular, can bring trouble to you:

You can only deploy one container for each VM instance. Consider Google Kubernetes Engine if you need to deploy multiple containers per VM instance.
You can only deploy containers from a public repository or from a private Container Registry repository that you have access to. Other private repositories are not supported.

In our example, I'll use a private image from Container Registry, and for that, we'll need a user that can read and download it.

How to create a service account that can download images on Container Registry

You can create a service account with Terraform too, but let's stick with the command line. To make the account, you can issue the following:

gcloud iam service-accounts create custom-gce-dealer \
--display-name "Custom GCE Dealer"

You must check which storage is used for your container registry. Let's suppose it's artifacts.agrabah-project.appspot.com. Then you can execute the following command, using the generated ID for your service account:

gsutil iam ch \
serviceAccount:custom-gce-dealer@agrabah-project.iam.gserviceaccount.com:roles/storage.objectViewer \
gs://artifacts.agrabah-project.appspot.com

That's all we need to check out images from the container registry.

Terraform manifests

I've found a module that handles the metadata needed to set up the container configuration for the resource google_compute_instance. Let's use it to create our own module; thus, our main.tf file will not be immense and difficult to understand. First, let's start a project with the following structure:

root-folder-of-your-project/ <--- Main project
│
├── gce-with-container/  <--- Our custom module
|   |
│   ├── main.tf
│   └── variables.tf
│
├── main.tf <--- We'll use gce-with-container here
├── terraform.tfvars <--- Values for what we defined in variables.tf
├── variables.tf <--- terraform.tfvars has the values for each defined variables
└── versions.tf  <-- Here you will find the terraform block which specifies the required provider version and required Terraform version for this configuration

Custom module

The folder gce-with-container contains our custom module. I've checked all the examples available on terraform-google-container-vm to create my own. Let's see how it's defined the main.tf:

locals {
  # https://www.terraform.io/docs/language/values/locals.html
  instance_name = format("%s-%s", var.instance_name, substr(md5(module.gce-container.container.image), 0, 8))

  env_variables = [for var_name, var_value in var.env_variables : {
    name = var_name
    value = var_value
  }]
}

####################
##### CONTAINER SETUP

module "gce-container" {
  # https://github.com/terraform-google-modules/terraform-google-container-vm
  source = "terraform-google-modules/container-vm/google"
  version = "~> 2.0"

  container = {
    image = var.image
    command = var.custom_command
    env = local.env_variables
    securityContext = {
      privileged : var.privileged_mode
    }
    tty : var.activate_tty
  }

  restart_policy = "Always"
}

####################
##### COMPUTE ENGINE

resource "google_compute_instance" "vm" {
  name = local.instance_name
  # gcloud compute machine-types list | grep micro | grep us-central1-a
  # e2-micro / 2 / 1.00
  # f1-micro / 1 / 0.60
  # gcloud compute machine-types list | grep small | grep us-central1-a
  # e2-small / 2 / 2.00
  # g1-small / 1 / 1.70
  machine_type = "f1-micro"
  # If true, allows Terraform to stop the instance to update its properties.
  allow_stopping_for_update = true

  boot_disk {
    initialize_params {
      image = module.gce-container.source_image
    }
  }

  network_interface {
    network = var.network_name

    access_config {}
  }

  metadata = {
    gce-container-declaration = module.gce-container.metadata_value
  }

  labels = {
    container-vm = module.gce-container.vm_container_label
  }

  service_account {
    email = var.client_email
    scopes = [
      "https://www.googleapis.com/auth/cloud-platform",
    ]
  }
}

I created some custom variables (see the file variables.tf) and a helper to make the configuration of environment variables easier (see locals.env_variables).

Main project

Our main.tf file in our root folder has the boilerplate part that configures the provider (see variables.tf and terraform.tfvars) and the module we created previously. I'm only using the variable custom_command because the container image I provided already has a CMD entry on its Dockerfile; thus, it must be overridden.

provider "google" {
  project = var.project
  credentials = file(var.credentials_file)
  region = var.region
  zone = var.zone
}

module "gce-worker-container" {
  source = "./gce-with-container"

  image = "gcr.io/${var.project}/jafar@sha256:6b71cebad455dae81af9fcb87a4c8b5bca2c2b6b2c09cec21756acd0f1ae7cec"
  privileged_mode = true
  activate_tty = true
  custom_command = [
    "./scripts/start-worker.sh"
  ]
  env_variables = {
    Q_CLUSTER_WORKERS = "2"
    DB_HOST = "your-database-host"
    DB_PORT = "5432"
    DB_ENGINE = "django.db.backends.postgresql"
    DB_NAME = "db_production"
    DB_SCHEMA = "jafar_prd"
    DB_USER = "role_jafar_prd"
    DB_PASS = "this-is-my-honest-password"
    DB_USE_SSL = "True"
  }
  instance_name = "jafar-worker"
  network_name = "default"
  # This has the permission to download images from Container Registry
  client_email = "custom-gce-dealer@${var.project}.iam.gserviceaccount.com"
}

That's it! You can type terraform init at the root folder and then terraform apply followed by your confirmation. It's a good idea to access the created VM and check if everything is okay through the command docker logs.

If you see the container with the image gcr.io/gce-containers/konlet:v.0.11-latest, you don't have to worry 👀. It's the job responsible for downloading the one you configured 😅.

In this article I used Terraform v0.14.9. You can check the entire code out on GitHub. Don't forget to execute terraform destroy after your test! See you next time ✌.

You should configure env variables in one place

Willian Antunes — Sun, 23 May 2021 14:57:52 +0000

Recently I had to create a new JavaScript project. I chose a framework and some libraries that I haven't used before. First, I tried to find lessons learned and projects on GitHub targeting the tools I decided on and then use them as a way to create my own project. Suddenly, I noticed a problem in all examples and even in big projects: they had process.env been invoked in many different files.

This approach may be seen as harmless because it's relatively trivial and works; nevertheless, isn't it better to help people who get in touch if your project regarding its settings right away?

Explicit is better than implicit

When you are learning Python, sooner or later, you will face The Zen of Python. To give you an idea, if you've ever had to deal with projects where many developers were involved, you will probably sympathize with the statements contained in this easter egg. I use them as a means to endorse some practices in this article.

One of its statements is exactly explicit is better than implicit. Never, ever expect that somebody will understand your intentions, try to always leave this in doubt. To illustrate, if your team is growing, including people who have never programmed with the targeted language or framework, one of your concerns should be reducing cognitive load across projects. There are many ways to minimize this impact. One is creating a place where all your environment variables are defined. Let's see an example in the following image.

Instead of searching which files have process.env, we can simply look at the settings.js file and then understand what the application needs to run correctly.

Readability counts, and a lot

This is another statement of the zen. Having all your environment variables in one place counts a lot. That is easy for someone with minimal "domain context" to read and quickly understand what your application does. As a result, it's pretty simple to statically find where the variables are being used. You can do that with process.env as well. Still, you would have trouble adding some spices to protect you from problems across many files. Let's see one ahead.

Errors should never pass silently

Let's say the variables PARTNER_API_ENDPOINT and ACTIVE_MQ_HOST must be present in production, and then you forgot to add them, but your application can still be built. Can you imagine the problem that you would have when it's identified because of customer complaints? Let' solve that:

function getEnvOrRaiseException(envName, defaultValue = null) {
  const value = process.env[envName]

  if (!value && defaultValue === null) throw new EnvironmentError(`Environment variable ${envName} is not set!`)
  if (!value && defaultValue !== null) return defaultValue

  return value
}

function evalEnvAsBoolean(envName, defaultValue = false) {
  const value = process.env[envName]

  if (!value && defaultValue) return defaultValue
  if (!value) return defaultValue

  const valueAsLowerCase = value.toLowerCase()
  const trueValues = ["true", "t", "y", "yes", "1"]
  return trueValues.includes(valueAsLowerCase)
}

const PARTNER_API_ENDPOINT = getEnvOrRaiseException("PARTNER_API_ENDPOINT")
const ACTIVE_MQ_HOST = getEnvOrRaiseException("ACTIVE_MQ_HOST")
const DATABASE_USE_SSL = evalEnvAsBoolean("DATABASE_USE_SSL", true)

export { PARTNER_API_ENDPOINT, ACTIVE_MQ_HOST, DATABASE_USE_SSL }

What about now? It can simply break your pipeline during build time. Your project will never advance to production without the required environment variables again. Is something unexpected? Throw an exception 😉

Now is better than never

How about trying to use some of the zen statements in your work or even in your life? By the way, now is better than never is another declaration that I took from it. See you next time!

Why did I create a blog from scratch?

Willian Antunes — Sat, 22 May 2021 16:00:12 +0000

If I had to recommend someone to start a blog, taking into consideration things like time to market, huge ecosystems, tutorials, and no expected advanced knowledge in programming at all, I would tell them to start right away with Blogger, or if more options are needed, then Wordpress would be the best choice probably. Now, if the idea is to know how stuff works, and utilize a technology that handles not only on the front-end but on the back-end also, thinking carefully in terms of its usage on the job market, I'd advise using a technology built on top of JavaScript.

Why JavaScript and not PHP or another language?

Let's dive a bit into some more concerns. What I usually do is look over data. To illustrate, what I've been doing to answer such questions so far is the following:

Search for projects through hashtags on GitHub.
Analyze the latest survey called The State of Developer Ecosystem made by JetBrains.
Study carefully the latest developer survey made by StackOverflow.
Which tags are most answered on StackOverflow.
Scan which jobs are offered on the job market. LinkedIn is an excellent place to look over.
Determine how expensive the technology is to deploy and serve its purpose.
Does the technology have documentation?
Consulting with skilled contacts 😅.

There is no silver bullet to choose a technology to create a blog, and some people might prefer a language that they already know, so this can be quite complex. Some may prefer a technology built from Java, Ruby, and so forth. But the thing is, you have to look carefully what technologies have companies been using so far, and start with those.

My research pointed out that would be the best idea to use JavaScript either using Gatsby or Next.JS as the framework for the blog. Nowadays I'm more used to Python, though, so what should I do? I mean, getting out of our comfort zone is quite challenging, but it's necessary.

Knowing new technologies opens your mind

When people discuss this topic, I like to compare it to learning a new language. In the beginning, it can be dull, monotonous, boring, but the game starts to change when you see your evolution, and people start to tell you about it. It's satisfying! When you notice, you will have a new tool to use to communicate with people and even apply it in your job. The same goes for technology. We must strive to learn new technologies, and understand other possibilities.

When you study technologies through playgrounds, leave them open-source

Isn't it amazing when you need to create/build something and then find a playground project to base your work on? Apart from that, when you create an open-source project, it can be used as your portfolio too. So use it wisely! Sell yourself! You will reap the fruits of your hard labor.

Final words and honest advice

One thing is discovering new tools and using them on your personal projects, and another entirely different thing is applying them to your job. Know the consequences because there are many variables to consider. If you want to switch services to use Hanami instead of Rails, you are supposed to answer at least the following questions:

Will it bring business value?
Will the customers be positively impacted by it?
Does it have support from the community?
Does it have support from companies out there?
Is it hard to find people that can work with it?
Is it arduous to teach people that have never been in contact with the language or the framework?

There are others of course, but you get the point 😁.

This is the first blog entry and I expect to post once per month at least. I hope I can achieve this goal! 😄