DEV Community: Jake Witcher

Which "this" is This in a Node Module?

Jake Witcher — Tue, 05 Oct 2021 01:23:25 +0000

Understanding which object context the keyword this is referencing is one of the more confusing language features in JavaScript. Even if you have a firm understanding of the rules for this binding there is still opportunity for it to surprise you, like when using this in the outermost scope of a Node module.

For comparison purposes, let's see what happens when we use this in the outermost scope of a script run in the browser before we try using it in a Node module.

<html>
<head>
    <title>Which This Is This</title>
</head>
    <body>
    <script>
        this.context = "is this the global context?"

        function whichThisIsThis() {
            alert(this.context)
        }

        whichThisIsThis()
    </script>
    </body>
</html>

Opening the above HTML in a browser creates an alert modal with the text "is this the global context?" displayed, confirming that both references to this are for the same context object. If we replace this.context = ... with window.context = ... in the first line of the script we get the same text in the alert modal as before, confirming that this is the window object when whichThisIsThis is executed.

Based on the above experiment in the browser, what do you think the following code will print to the console if executed by the Node runtime?

this.context = "is this the global context?"

function whichThisIsThis() {
    console.log(this.context)
}

whichThisIsThis()

Surprisingly, it will print undefined. Unlike in the browser script, this in the outermost scope of a Node module is not a reference to the global object, rather it is a reference to the module.exports object.

However when whichThisIsThis is executed in the example above, this inside the function's scope is the global object. Because the outermost scope is setting a context property on the this that's referencing module.exports, we get an undefined response when trying to access a context property on the this that references the global object, the this inside the whichThisIsThis function scope.

You can test this out by changing this.context in the call to console.log to module.exports.context. Run the code again and you should see the string value assigned to this.context in the outermost scope print to the console.

this.context = "this is the module.exports context, not the global context!"

function whichThisIsThis() {
    console.log(module.exports.context)
}

whichThisIsThis()

However if your intention is to use the global object you can keep this.context as the argument for console.log but you will need to explicitly create and set a context property on the global object.

global.context = "this is the global context!"

function whichThisIsThis() {
    console.log(this.context)
}

whichThisIsThis()

Keep in mind that the this in the whichThisIsThis function is only referencing the global object in the above example because no other context has been provided. If this function were to be called in a way that implicitly or explicitly provided a different object to assume the role of this, the outcome would be different.

global.context = "default global context"

function whichThisIsThis() {
    console.log(this.context)
}


let implicitContext = {
    context: "implicitly bound context",
    whichThisIsThis
}
implicitContext.whichThisIsThis()


let explicitContext = whichThisIsThis.bind({context: "explicitly bound context"})
explicitContext()

Running the above code will print "implicitly bound context" and "explicitly bound context" to the console but not "default global context".

Functional Interfaces in Java: What are Functional Interfaces?

Jake Witcher — Sat, 21 Aug 2021 20:31:29 +0000

A key concept in functional programming is the notion of higher order functions. Higher order functions are functions that can be used as input parameters and output values for other functions or methods. However in a typed language like Java we need to be able to define our parameters and return values as types.

When defining an integer parameter, we use the type int. For a collection of string values we can use Collection<String> or we can be more specific with types like List<String> or Set<String>. To benefit from the use of higher order functions, we need to be able to describe a lambda expression or method reference in a similar way.

Functional interfaces are a language feature added in Java 8 along with lambda expressions that provide a means for expressing a lambda or method as a specific type by using an interface. Any interface with a single abstract method can be considered a functional interface and can be used in the type signature of a method that expects a lambda expression as a parameter or output value.

Video Rental Inventory Management

To illustrate how functional interfaces work and the usefulness of higher order functions, let's create a simple inventory management system. Video rental stores have predominately become a thing of the past, but for those that still exist, keeping track of available movie inventory is a must. The requirements for this application are as follows:

It should be able to count the available inventory of all movies.
It should be able to count movies by media type — either VHS or DVD.
It should be able to count movies by title.
It should be extensible for including additional inventory counting methods in the future.

First we define a couple of domain models for Movie and Inventory using Lombok to reduce the amount of boilerplate required.

public enum MediaType {
    VHS, DVD
}

@Data
public class Movie {
    private final String title;
    private final MediaType mediaType;
    private final int quantity;
}

@RequiredArgsConstructor
public class Inventory {
    private final List<Movie> movies;
}

We could add a method to the Inventory class with the logic for counting available movies however with each inventory counting requirement we would either need to write separate methods or utilize object composition.

Writing separate methods would mean our class is not closed for modification, violating one of the SOLID principles. Object composition would be preferable, but it would require a new interface to be made and added as a field on the Inventory class as well as the creation of new classes that implement the interface for every known and future requirement.

// adding new methods for each requirement modifies the class's public contract with each new use case
@RequiredArgsConstructor
public class Inventory {
    private final List<Movie> movies;

    public int countAll() {
        return movies.stream().mapToInt(Movie::getQuantity).sum();
    }

    public int countAllVhs() {
        return movies.stream()
                .filter(movie -> Objects.equals(movie.getMediaType(), MediaType.VHS))
                .mapToInt(Movie::getQuantity).sum();
    }

    public int countAllByTitle(String title) {
        return movies.stream()
                .filter(movie -> Objects.equals(movie.getTitle(), title))
                .mapToInt(Movie::getQuantity).sum();
    }
}

// object composition is preferable but requires new classes to be made for each use case
public interface InventoryCounter {
    int count(List<Movie> inventory);
}

@RequiredArgsConstructor
public class Inventory {
    private final List<Movie> movies;
    private final InventoryCounter counter;

    public int count() {
        return counter.count(movies);
    }
}

Leveraging Higher Order Functions

Higher order functions provide us with a third option. We can define the interface of a function that takes a list of movies and returns an integer value, which at a high level accounts for the functionality of all our movie counting use cases. We use the @FunctionalInterface annotation to document our intended purpose for the InventoryCounter interface.

@FunctionalInterface
public interface InventoryCounter {
    int apply(List<Movie> inventory);
}

Other than changing the name of the single function on this interface, it is identical to the interface in the object composition example above. However while the interface itself has not changed, the way we use it has.

Next we add a count method to the Inventory class. As its only parameter, this method accepts a lambda function or method reference with the same type signature as the apply method on the interface; any method or lambda that has List<Movie> as an input parameter and int as an output value.

@RequiredArgsConstructor
public class Inventory {
    private final List<Movie> movies;

    public int count(InventoryCounter counter) {
        return counter.apply(movies);
    }
}

Testing the Implementation

Now we can test our solution. As if renting physical copies of movies wasn't niche enough, our particular client deals only in movies from the DC cinematic universe.

public class InventoryCounterTest {
    private static Inventory inventory;

    @BeforeAll
    public static void setup() {
        inventory = new Inventory(getMovies());
    }

    private static List<Movie> getMovies() {
        return asList(
                new Movie("Aquaman", MediaType.DVD, 15),
                new Movie("Aquaman", MediaType.VHS, 5),
                new Movie("Batman v. Superman", MediaType.DVD,  25),
                new Movie("Batman v. Superman", MediaType.VHS,  10),
                new Movie("Justice League", MediaType.DVD, 30),
                new Movie("Justice League", MediaType.VHS, 12),
                new Movie("Man of Steel", MediaType.DVD, 12),
                new Movie("Man of Steel", MediaType.VHS, 3),
                new Movie("Wonder Woman", MediaType.DVD, 35),
                new Movie("Wonder Woman", MediaType.VHS, 10)
        );
    }
}

How they managed to acquire VHS copies of those films is a mystery.

Our first use case is to count the total number of movies available. To accomplish this we can use an anonymous lambda expression, a lambda expression assigned to a variable, or a method reference.

// anonymous lambda expression
@Test
public void itShouldCountAllMoviesInInventory() {
    var inventoryCount = inventory.count(movies -> movies.stream()
            .mapToInt(Movie::getQuantity)
            .sum());

    assertEquals(157, inventoryCount);
}


// lambda expression assigned to a variable
@Test
public void itShouldCountAllMoviesInInventory() {
    InventoryCounter counter = movies -> movies.stream()
            .mapToInt(Movie::getQuantity)
            .sum();

    var inventoryCount = inventory.count(counter);

    assertEquals(157, inventoryCount);
}


// method reference
@Test
public void itShouldCountAllMoviesInInventory() {
    var inventoryCount = inventory.count(this::countMovieInventory);

    assertEquals(157, inventoryCount);
}

private int countMovieInventory(List<Movie> movies) {
    return movies.stream().mapToInt(Movie::getQuantity).sum();
}

We can easily test the second requirement — counting the available VHS movies — this time using only the method reference approach. This is the preferred way of satisfying a functional interface for anything other than short lambda expressions with intent easily derived from the expression itself.

When in doubt, favor a well-named method reference.

@Test
public void itShouldCountAllVhsMoviesInInventory() {
    var vhsInventoryCount = inventory.count(this::countVhsMovieInventory);
    assertEquals(40, vhsInventoryCount);
}

private int countVhsMovieInventory(List<Movie> movies) {
    return movies.stream()
            .filter(movie -> Objects.equals(movie.getMediaType(), MediaType.VHS))
            .mapToInt(Movie::getQuantity).sum();
}

We have one last requirement to fulfill — counting the available movies by title. To meet this requirement we need to use the InventoryCounter functional interfaces in a new way, as the return type of a method.

@Test
public void itShouldCountAllMoviesInInventoryWithSpecifiedTitle() {
    var aquamanInventoryCount = inventory.count(countMovieInventoryWithTitle("Aquaman"));
    var wonderWomanInventoryCount = inventory.count(countMovieInventoryWithTitle("Wonder Woman"));

    assertEquals(20, aquamanInventoryCount);
    assertEquals(45, wonderWomanInventoryCount);
}

private InventoryCounter countMovieInventoryWithTitle(String title) {
    return movies -> movies.stream()
            .filter(movie -> Objects.equals(movie.getTitle(), title))
            .mapToInt(Movie::getQuantity).sum();
}

The countMovieInventoryWithTitle method cannot be used as a method reference like in the previous two examples because it does not satisfy the InventoryCounter functional interface. What it does do is return a lambda expression that satisfies the functional interface and allows us to include some dynamic information in the lambda, the title of the movie.

With this last test we have met all of our requirements. The three known use cases are covered and by using functional interfaces we have ensured our code is extensible for future use cases.

Standard Functional Interfaces

We have eliminated a lot of unnecessary code by using functional interfaces over object composition, however there is one opportunity to refactor that allows us to remove the InventoryCounter interface. Most function type signatures can be expressed by a small number of interfaces with one or more generic type parameter. The function package leverages this fact to provide several standard functional interfaces so that you do not have to define your own.

import java.util.function.Function;
import java.util.function.ToIntFunction;

public class Inventory {
    private final List<Movie> movies;

    // the Function interface accepts two type parameters, one for the input parameter and one for the output
    public int count(Function<List<Movie>, Integer> counter) {
        return counter.apply(movies);
    }

    // to avoid unnecessary boxing, each functional interface has several alternatives for working with primitives
    public int count(ToIntFunction<List<Movie>> counter) {
        return counter.applyAsInt(movies);
    }
}

There are times it may be necessary or more expressive to define your own functional interfaces, but for all other use cases, you should use the standard functional interfaces.

More to Explore

Although this introduction to and overview of functional interfaces has come to an end, there is a whole lot more to explore. In future blog posts I will review each of the primary functional interface types, describing their use cases, providing code examples of how they work, and exploring additional functionality exposed by their individual APIs.

Keyboard Shortcuts for Moving the Cursor in a Text Editor

Jake Witcher — Sat, 27 Mar 2021 13:08:25 +0000

One of the skills you can develop to increase productivity is your ability to navigate around a text editor using keyboard shortcuts instead of a mouse. However not all shortcuts are equally useful nor are they all universal across text editors.

If you are new to using keyboard shortcuts, one of the best places to start is with the shortcuts that move the cursor around a page. These shortcuts are universal across most text editors as well as many other desktop applications like Slack, Microsoft Teams, Goodle Docs, and Microsoft Word.

It can take some time to develop the muscle memory but a few days or weeks of awkward and slow key presses will pay off in dividends once the shortcuts become second nature and you can quickly move the cursor as you code, write, or send your coworkers a message.

Below are the keyboard shortcuts for both Windows and Mac that I use the most frequently for moving the cursor around.

Moving the Cursor on a Windows PC

Move Cursor To	Keyboard Shortcut
the start of the line	HOME
the end of the line	END
the top of the page	CTRL + HOME
the bottom of the page	CTRL + END
one word to the right	CTRL + RIGHT ARROW
one word to the left	CTRL + LEFT ARROW

Moving the Cursor to Select Text on a Windows PC

Select Text From The Cursor To	Keyboard Shortcut
the start of the line	SHIFT + HOME
the end of the line	SHIFT + END
the top of the page	SHIFT + CTRL + HOME
the end of the page	SHIFT + CTRL + END
one word to the right	SHIFT + CTRL + RIGHT ARROW**
one word to the left	SHIFT + CTRL + LEFT ARROW**

**the right or left arrow can be repeatedly pressed to select multiple words

Moving the Cursor on a Mac

Move Cursor To	Keyboard Shortcut
the start of the line	CMD + LEFT ARROW
the end of the line	CMD + RIGHT ARROW
the top of the page	CMD + UP ARROW
the bottom of the page	CMD + DOWN ARROW
one word to the right	OPTION + LEFT ARROW
one word to the left	OPTION + RIGHT ARROW

Moving the Cursor to Select Text on a Mac

Select Text From The Cursor To	Keyboard Shortcut
the start of the line	SHIFT + CMD + LEFT ARROW
the end of the line	SHIFT + CMD + RIGHT ARROW
the top of the page	SHIFT + CMD + UP ARROW
the end of the page	SHIFT + CMD + DOWN ARROW
one word to the right	SHIFT + OPTION + RIGHT ARROW**
one word to the left	SHIFT + OPTION + LEFT ARROW**

**the right or left arrow can be repeatedly pressed to select multiple words

Once you are able to move the cursor around the page with ease, you can combine the above keyboard shortcuts with several other universal and editor-specific shortcuts in your workflow like copying text, renaming methods or variables, moving blocks of code up or down the page, and extracting methods.

Deploying Twemproxy to GCP

Jake Witcher — Sat, 20 Mar 2021 13:13:01 +0000

Now that the cloud resources have been provisioned and the docker image for the Twemproxy server has been created, it's time to deploy the proxy to GCP and make a few changes to the client services that will be communicating with the proxy.

Pushing the Twemproxy Docker Image to Google Container Registry

First we need to push the docker image to Google Container Registry (GCR). You may have to enable the GCR API for the project containing the Twemproxy server if you have not used the container registry service for any other applications in the project. You can enable the API for GCR using the Cloud SDK.

gcloud services enable containerregistry.googleapis.com

To push an image to a container registry in GCP, Docker must be authenticated on your machine. Detailed information about configuring authentication can be found here with the recommended method being the gcloud credential helper.

You will need to have a service account created with the "Storage Admin" role as well as a key file with the service account's credentials in JSON format in order to use the credential helper. With those two things Docker can be authenticated with the following commands.

# make sure you are logged in before running the Docker authentication commands
gcloud auth login 

gcloud auth activate-service-account your-service-account-name@your-project-name.iam.gserviceaccount.com --key-file=path/to/key-file.json

gcloud auth configure-docker

With Docker authenticated you can push the image to GCR and a new container registry will be created for you.

For the image to be pushed successfully it must be tagged either when it is created or before you push the image to GCR with the registry name gcr.io/your-project-name/ followed by the name of the image and the version. If a version is not specified the latest version will be used.

# if you are creating the image for the first time, you can tag it with the flag '-t'
docker build . -t gcr.io/your-project-name/twemproxy:v1

# or if the image already exists, use the 'docker tag' command
docker tag twemproxy gcr.io/your-project-name/twemproxy:v1

docker push gcr.io/your-project-name/twemproxy:v1

Creating an Image Template from the Docker Image

Next an image template must be created. This is the virtual machine image that will be used by the managed instance group (MIG) when creating new instances of the Twemproxy server.

To create the image template you will need to have the digest of the Docker image you just pushed to GCR. The digest is a unique id generated by GCP to distinguish different versions of the same Docker image. You can find the digest manually using the browser-based console or you can script out the process of retrieving the digest which is useful for automating the deployment process.

(All script examples will be written in Powershell but can be adapted as Linux shell scripts)

$Digest = (gcloud container images list-tags "gcr.io/your-project-name/twemproxy" `
    --limit=1 `
    --format="get(digest)")[0]

$Image = "gcr.io/your-project-name/twemproxy@{0}" -f $Digest

This script retrieves a list of the twemproxy Docker image tags with a limit of "1" so that only the first one is retrieved. This should be the most recent image pushed to the registry. The --format flag reduces the output to only the digest property. You can run this command with the format changed to --format=json if you would like to see all properties available. Once the digest is retrieved it is appended to the fully qualified name of the image in your registry.

You can now use the Docker image name to create an image template with the Docker container.

gcloud compute instance-templates create-with-container twemproxy-v1 `
    --container-image=$Image `
    --machine-type=f1-micro `
    --tags=fw-allow-lb-and-hc,fw-allow-twemproxy-6380 `
    --no-address `
    --scopes=cloud-platform `
    --network=your-network-name `
    --subnet=your-subnet-name `
    --region=us-east1

Note that the tag names following the --tags flag must match the target tags defined by the firewall rules created in the previous blog post.

Performing a Rolling Update on the Managed Instances Group

Lastly, we will need to perform a rolling restart of the MIG so that the newly created template with the Twemproxy server is used instead of the placeholder template used in the Terraform code.

gcloud compute instance-groups managed rolling-action start-update twemproxy-mig `
    --version="template=twemproxy-v1" `
    --zone=us-east1-b

Once the rolling update completes your Twemproxy server will be live and ready to test. For your microservices to communicate with the Twemproxy server instead of directly with a Memorystore instance you may need to make some changes to the configuration of any Redis client libraries used by your code.

For example in C# the Nuget package StackExchange.Redis has a configuration object that must register the Twemproxy server to work correctly. The only difference between using this package directly against Memorystore and indirectly through Twemproxy is the Proxy property of the ConfigurationOptions object. The API for setting and retrieving data from the cache remains the same.

private IConnectionMultiplexer CreateConnectionMultiplexer(string ip, int port)
{
    var config = new ConfigurationOptions
    {
        AsyncTimeout = 1000,
        EndPoints = {{ip, port}},
        Proxy = Proxy.Twemproxy
    };

    return ConnectionMultiplexer.Connect(config);
}

You will also need to set the IP and port number to match that of the load balancer for the Twemproxy MIG.

Test and Configure to Meet Your Requirements

Now you can redeploy your client services with any changes to the code and start testing your newly created Memorystore cluster! Every use case is different so you may run into a few issues that I did not encounter or you may need to go back and further configure the Twemproxy server to meet your requirements.

Before concluding this series on using Twemproxy with Memorystore in GCP there is one last script you may find useful when automating your deployment process. Whenever you add, remove, or recreate an instance of Memorystore the list of IP addresses in the YAML file used to configure the Twemproxy server will need to be updated. Rather than doing this manually, you can use a script.

The first step is to remove the existing list of IP addresses from the YAML file.

Get-Content -Path .\nutcracker.yml | Where-Object {$_ -notmatch ":6379:1"} | Set-Content .\tmp.yml
Get-Content -Path .\tmp.yml | Set-Content .\nutcracker.yml
Remove-Item .\tmp.yml

Then use the gcloud SDK to retrieve a list of the IP addresses of your deployed Memorystore instances and append them to the end of the YAML file. For this to work it's important that the servers property of the configuration YAML be the last one listed.

gcloud redis instances list --region us-east1 --format "value(host)" `
| ForEach-Object {"        - {0}:6379:1" -f $_} `
| Out-File -FilePath .\nutcracker.yml -Append -Encoding utf8

With the scripts in this blog post you should have most if not all the tools you need to completely automate the process of redeploying your Twemproxy server whenever you need to change the configuration or scale up the number of Memorystore instances.

Magical Mystery XOR

Jake Witcher — Sat, 13 Mar 2021 19:43:22 +0000

Binary logical operators are fundamental to software development and chances are you use them almost every time you write code. The operators && (and) and || (or) fall into this category and are regularly found in predicates for control flow statements in application code.

if (a && b) {
    // do something...
} else if (a || b) {
    // do something else...
}

&& and || are referred to as binary logical operators because they perform an operation on two values (binary) and the two values (or operands) must be propositional expressions that evaluate to either true or false (logical).

Combining two propositional expressions with a binary logical operator produces a new propositional expression which is itself a valid operand for a binary logical operator. This means that we can use an expression with a logical operator as one of the operands of another expression, for example (a && b) || c where the expression (a && b) serves as the left operand of the || operator.

While less common in most application code, there is a third binary logical operator that can be used in control flow statements. It is known as the xor operator or the exclusive or operator. Not all programming languages have a built in xor operator but of those that do, the symbol ^ is often used to represent xor in code, allowing for expressions like a ^ b and (a ^ b && c).

The && and || Operators

Before getting into what exactly ^ does and when to use it, let's first take a brief look at && and || so that we can consider the xor operator as it relates to and and or.

The operator && evaluates to true if both operands in the expression evaluate to true, otherwise it evaluates to false. The relationship between && and all possible combinations of left and right operands are expressed in the following truth table where column a represents the left operand of an expression, column b represents the right operand of an expression, and column && represents the result of applying the && operator to both a and b.

-----------------------
 a     | b     | && 
=======================
 True  | True  | True
-----------------------
 True  | False | False
------------------
 False | True  | False
------------------
 False | False | False
------------------

In contrast to &&, the operator || evaluates to true if either one or the other operand in the expression evaluates to true or if both operands evaluate to true. If both operands are false, then the || expression will be false.

------------------
 a     | b     | || 
=======================
 True  | True  | True
------------------
 True  | False | True
------------------
 False | True  | True
------------------
 False | False | False
------------------

The ^ Operator

Then there is the ^ operator. It evaluates to true if either operand in the expression evaluates to true but not if both operands evaluate to true. If both operands are false or if both operands are true, the ^ expression will evaluate to false. This is what the exclusive in exclusive or is communicating. For the expression to evaluate as true, one or the other operand must be true, not both. One of them must be exclusively true.

------------------
 a     | b     | ^ 
=======================
 True  | True  | False
------------------
 True  | False | True
------------------
 False | True  | True
------------------
 False | False | False
------------------

You may come across languages that have a bitwise xor operator however this is not the same as the logical xor operator. Conceptually they are similar but in practice they are used for different purposes. It is outside the scope of this blog post to discuss bitwise operators and their relationship to logical operators however it is a topic worth exploring separately. For now it is important to know that they are not the same and that the focus of this blog post is on logical operators, not bitwise.

Using Exclusive Or in Code

The xor operator should be used when you require one and only one of two expressions to be true. For languages that do not have an xor operator, it is possible to check if only one expression is true however it is more verbose and does not communicate intent as clearly.

If the language you are working with has an xor operator, using it will make your code easier to read at a glance. It will clearly communicate the purpose of your code — to find an exclusively true expression.

// these expressions takes much longer to read and understand
boolean c = (a || b) && !(a && b)
boolean d = (a && !b) || (!a && b)

// using xor clearly communicates the purpose of the expression
boolean e = a ^ b

Using Terraform to Provision Cloud Resources for a Twemproxy Server

Jake Witcher — Sun, 07 Mar 2021 13:02:02 +0000

With the Twemproxy docker image created, we can now turn our attention to provisioning the resources required to deploy and run a Memorystore Redis cluster in GCP with a proxy. Using IAC (infrastructure as code) to manage your cloud resources is a great way to enforce consistency across environments and to ensure visibility on all infrastructure.

Terraform is one of the more popular IAC solutions and is what we'll be using to plan out the infrastructure required for a Redis cluster in GCP.

It would take several blog posts to go through every detail of each resource, so we will focus on a few key details for each one. Some dependencies, like the network and subnetwork required for the load balancer or the details regarding the client services that will be making requests to the proxy server, will not be explicitly included in the example code. The assumption will be that these resources are already deployed. This blog post will focus on the additional resources required to add a caching layer.

Memorystore

The first resources we will define are the Memorystore instances.

locals {
    reserved_ips = ["10.0.0.24/29", "10.0.0.32/29", "10.0.0.40/29"]
}

resource "google_redis_instance" "redis_memorystore" {
    count = length(local.reserved_ips)

    provider           = google-beta
    region             = "us-east1"

    name               = "memorystore${count.index + 1}"
    tier               = "BASIC"
    memory_size_gb     = 1
    redis_version      = "REDIS_5_0"
    authorized_network = var.network.id
    reserved_ip_range  = local.reserved_ips[count.index]
    display_name       = "Memorystore Redis Cache ${count.index + 1}"
}

For this example we have a list of CIDR ranges, one range for each Memorystore instance. The IP address can be whatever you would like it to be however the IP network prefix must be 29. You can optionally remove the reservered_ip_range argument and let GCP assign the CIDR ranges at which point you can set count to an integer value for the number of instances desired.

The Memorystore instances created by this resource block will work well for testing but will likely require a few changes for production environments. For production you should consider changing the tier to STANDARD_HA and the memory_size_gb to a size that satisfies your application's requirements.

Managed Instance Group

Next we will define the resources required to deploy the Twemproxy server as a managed instance group of VMs (virtual machines) running our containerized server. There are quite a few resources needed to deploy a complete managed instance group solution. We'll start with the instance template and managed instance group resources.

resource "google_compute_instance_template" "twemproxy_template" {
    region       = "us-east1"

    name_prefix  = "twemproxy-"
    machine_type = "f1-micro"

    tags = ["fw-allow-lb-and-hc", "fw-allow-twemproxy-6380"]

    disk {
        source_image = "cos-cloud/cos-stable"
        boot         = true
    }

    network_interface {
        network    = var.network.name
        subnetwork = var.subnetwork.name
    }

    lifecycle {
        create_before_destroy = true
    }
}

resource "google_compute_instance_group_manager" "twemproxy_instance_group" {
    zone               = "us-east1-b"

    name               = "twemproxy-mig"
    base_instance_name = "twemproxy"
    target_size        = 0

    named_port {
        name = "proxy"
        port = 6380
    }

    version {
        name              = "twemproxy-template"
        instance_template = google_compute_instance_template.twemproxy_template.self_link
    }

    lifecycle {
        ignore_changes = [
        version,
        target_size
        ]
    }
}

The image template resource is primarily a placeholder allowing us to provision the infrastructure for the managed instance group independent from the deployment of our code. The template is required for an initial deployment of a managed instance group but will be replaced every time our application code changes and our proxy service is redeployed.

The creation and deployment of the real instance template — one that includes our docker image — will be discussed in the next blog post. When creating a new template as a part of the deployment process, we will need to include the same tags listed on the placeholder. These tags are used to attach the firewall rules defined below to the VM instances created from our instance template.

The managed instance group starts with a target size of 0 instances so that we do not incur unnecessary cost before deploying the proxy server. The instance group also includes lifecycle information that ensures Terraform will ignore changes to the instance template version and number of instances running.

If we do not tell Terraform to ignore these changes, on subsequent runs of terraform apply it would reduce the number of instances in our managed instance group back to 0 and would use the placeholder template which would override the current template in use.

Load Balancer

Now we need to create an internal load balancer for the managed instance group. Internal load balancers are made up of three resources, a forwarding rule, a backend service, and a health check. The forwarding rule defines the client facing properties of the load balancer including the IP address and the available ports. The backend service provides a health check for the instances, defines the way traffic is distributed to the instances by the load balancer, and also contains a reference to the managed instance group resource.

resource "google_compute_forwarding_rule" "twemproxy_internal_lb" {
    provider = google-beta
    region   = "us-east1"

    name                  = "twemproxy-internal-lb"
    ip_protocol           = "TCP"
    load_balancing_scheme = "INTERNAL"
    backend_service       = google_compute_region_backend_service.twemproxy_backend_service.self_link
    ip_address            = "10.0.0.50"
    ports                 = ["6380"]
    network               = var.network.name
    subnetwork            = var.subnetwork.name
}

resource "google_compute_region_backend_service" "twemproxy_backend_service" {
    name          = "twemproxy-backend-service"
    region        = "us-east1"
    health_checks = [google_compute_health_check.twemproxy_health_check.self_link]

    backend {
        group          = google_compute_instance_group_manager.twemproxy_instance_group.instance_group
        balancing_mode = "CONNECTION"
    }
}

resource "google_compute_health_check" "twemproxy_health_check" {
    name               = "twemproxy-health-check"
    timeout_sec        = 1
    check_interval_sec = 1
    tcp_health_check {
        port = 6380
    }
}

Firewall Rules

The last resources we'll need to define are a set of firewall rules.

resource "google_compute_firewall" "allow_lb_and_hc" {
  name    = "fw-allow-lb-and-hc"
  network = var.network.name

  source_ranges = ["130.211.0.0/22", "35.191.0.0/16"]
  target_tags   = ["fw-allow-lb-and-hc"]

  allow {
    protocol = "tcp"
  }
}

resource "google_compute_firewall" "allow_twemproxy" {
    name          = "fw-allow-twemproxy-6380"
    network       = var.network.name
    source_ranges = ["0.0.0.0/0"]
    target_tags   = ["fw-allow-twemproxy-6380"]

    allow {
        protocol = "tcp"
        ports    = ["6380"]
    }
}

The first firewall rule defined above allows both the load balancer and the health check to communicate with the VMs in your managed instance group. The IP ranges assigned to the source_ranges argument are provided by Google in their documentation for setting up a load balancer and need to be exactly as defined in the example code.

The second firewall rule will allow other resources in your network to communicate with the Twemproxy server. You will need to change the source_ranges attribute to match the IP addresses of the applications that use the proxy. In addition to IP ranges and tags, there are other options for specifying the source and target resources of this firewall rule. You can find information on these options and their restrictions here.

With all of the required resources defined, we can now run terraform validate and terraform plan to make sure everything was done correctly and to see the final list of resources Terraform will provision for our given plan.

Once everything is correct we can run terraform apply -auto-approve. Terraform will begin to provision all of our resources and let us know if the deployment was a success or if there are any issues to resolve.

In the next blog post we will deploy the Twemproxy server to our newly provisioned cloud resources as well as make a few changes to the code of our client services so that they are able to use the proxy server instead of directly communicating with Memorystore.

Creating a Docker Image for a Twemproxy Server

Jake Witcher — Sat, 27 Feb 2021 14:01:45 +0000

Twemproxy is an open source proxy server created by Twitter and is one of the methods recommended by Google for managing a cluster of Memorystore instances in GCP (Google Cloud Platform). Memorystore is the managed service provided by GCP for application caching with either Redis or Memcached. Twemproxy provides a way to evenly distribute cached data between multiple Redis instances to improve the performance, reliability, and resilience of a distributed system.

This blog post is the start of a series that will provide you with the foundational information you need to go from creating a Docker image of a Twemproxy server to deploying a managed instance group of containers created from that image to serve as a proxy for several instances of Memorystore in GCP.

The first step is to download a copy of the distribution tarball using the link provided by Twitter in the Twemproxy Github repo. The link can be found under the "Build" section. This example uses version 0.4.1 of Twemproxy, (also referred to as nutcracker).

Create a directory for your Twemproxy server project and place the nutcracker-0.4.1.tar.gz file in it along with an empty Dockerfile and an empty nutcracker.yml file. The YAML file is where you will write your configuration for the server.

Next, copy and paste the following code in your Dockerfile.



FROM debian:buster as builder

COPY nutcracker-0.4.1.tar.gz /

RUN apt-get update && apt-get -qy install \
    automake \
    libtool \
    make \
    && rm -rf /var/lib/apt/lists/*

RUN tar xzf /nutcracker-0.4.1.tar.gz
RUN cd nutcracker-0.4.1 && CFLAGS="-ggdb3 -O0" ./configure --enable-debug=full && make && make install


# Main

FROM debian:buster

COPY --from=builder /usr/local/sbin/nutcracker /usr/local/sbin/nutcracker
COPY nutcracker.yml /etc/nutcracker.yml

EXPOSE 6380
CMD ["nutcracker", "-c", "/etc/nutcracker.yml"]

There are two stages to this Dockerfile. The first stage installs the tools required to build Twemproxy, unzips the nutcracker-0.4.1.tar.gz file, and then uses the make tool to build it.

The second stage copies over the nutcracker directory that was created by the build process as well as the configuration file, nutcracker.yml. Lastly, this stage defines the command that will start the server once the container is running. nutcracker is the name of the executable and the -c flag followed by /etc/nutcracker.yml directs the application to your customized configuration file.

There are a lot of options for customizing the Twemproxy server based on your requirements, most of which can be defined in the configuration file. Other customization options, like changing the size of the mbufs used to ferry requests and responses, can be changed through optional parameters passed to the nutcracker command. Changes to logging must be done at the compilation step as illustrated in the Dockerfile with the --enable-debug=full parameter passed to the ./configure command.

You can find a complete list of configuration options in the Twemproxy README file.

As a starting point, you can use the configuration below in your nutcracker.yml file and adjust as needed. You will need to make sure that the port in the listen field matches the exposed port in your Dockerfile and you will need to change the IP addresses and port numbers listed under the servers property to match your Memorystore instances once they have been created.



pool:
    listen: 0.0.0.0:6380
    hash: fnv1a_64
    distribution: ketama
    redis: true
    auto_eject_hosts: true
    timeout: 2000
    server_retry_timeout: 10000
    server_failure_limit: 3
    server_connections: 40
    preconnect: true
    servers:
        - 127.0.0.1:6379:1
        - 127.0.0.2:6379:1
        - 127.0.0.3:6379:1

To make sure the Dockerfile builds and that the image runs locally in a container, run the following commands from within the same directory as your Dockerfile.



docker build . -t twemproxy-test
docker run --name twemproxy -it --rm twemproxy-test

You will see the following message once the server has successfully started running along with a couple of error messages due to the fact that the proxy is unable to connect with Redis instances that do not exist yet.



nutcracker-0.4.1 build for Linux 4.19.121-linuxkit x86_64 started on pid 1
run, rabbit run / dig that hole, forget the sun / and when at last the work is done / don't sit down / it's time to dig another one

In the next blog post we will use Terraform to define all of the resources required to deploy a managed instance group of Twemproxy servers and a cluster of three Memorystore instances. Once the cloud resources have been provisioned you will be able to replace the arbitrary server IP addresses and ports in the example configuration file above with actual address information for your deployed Redis instances.

Abstracting Application IO Using Higher Order Functions

Jake Witcher — Sat, 20 Feb 2021 19:49:58 +0000

In a previous blog post I examined how interfaces can be used to abstract away the details of an application's IO creating code that is easier to test and easier to extend. Although interfaces are a very effective tool for abstraction, they are not the only tool available for separating business logic and IO. Higher order functions are another method we can use to separate the details about file systems or databases from our core business logic.

Higher order functions are functions that rely on other functions as either a parameter type or a return type. Functions like map and filter in Java, Python and many other languages fit the definition of a higher order function. Both of these functions accept another function as an argument; one that either describes how the elements of a collection should be filtered or how the elements should be transformed.

nums = [1, 2, 3, 4, 5]

# using higher order functions in Python
squared_nums = map(lambda n: n * n, nums)
even_nums = filter(lambda n: n % 2 == 0, nums)

Or if you have used the http package in Go, you may be familiar with another higher order function, the http.HandleFunc method. This method takes a function that defines how an http request should be handled for a given route as its second argument.

func handleHomePage(writer http.ResponseWriter, _ *http.Request) {
    _, err := fmt.Fprintf(writer, "This is the Home Page")
    if err != nil {
        log.Fatalln(err)
    }
}

func main() {
    http.HandleFunc("/", handleHomePage)
    log.Fatalln(http.ListenAndServe(":8080", nil))
}

We can illustrate how higher order functions are able to hide IO details using the same example from the previous blog post; an application that calculates an order total from a list of line items. We'll start by writing a version of the CalculateOrderTotal function that takes a function as an argument rather than a struct that satisfies the OrderProvider interface.

func CalculateOrderTotal(providerFunc func() []LineItem) float64 {
    lineItems := providerFunc()

    var orderTotal float64
    for _, lineItem := range lineItems {
        orderTotal += lineItem.Price
    }

    return orderTotal
}

When using higher order functions, the type signature of the function is key. The new parameter providerFunc is a function that takes no arguments and returns an array slice of line items. Any function that matches the type signature func() []LineItem can be used as an argument for this version of CalculateOrderTotal.

The implementation details regarding how this function will retrieve and then return a list of line items is kept hidden from CalculateOrderTotal just as it was before when using the OrderProvider interface.

To test out this new version of CalculateOrderTotal, we can define a function that returns a list of line items stored in memory. However instead of just writing a function that returns a predetermined list of line items, we'll create a provider function factory that creates a function with the desired signature func() []LineItem using a list of line items passed as an argument to the factory function.

func InMemoryLineItemsProviderFuncFactory(lineItems []LineItem) func() []LineItem {
    return func() []LineItem {
        return lineItems
    }
}

This implementation may seem unnecessarily complex, and if all we needed was a static list of line items it certainly would be. But the goal is to keep IO details away from CalculateOrderTotal and to have the freedom to swap out those details on the fly. This factory approach will allow us to test the new version of CalculateOrderTotal with different sets of values stored in memory to make sure it still works the same as before.

var testCases = []struct {
    lineItems []LineItem
    expected  float64
}{
    {
        lineItems: []LineItem{
            {Description: "A", Price: 85},
            {Description: "B", Price: 15},
        },
        expected: 100,
    },
    {
        lineItems: []LineItem{
            {Description: "A", Price: 35.25},
            {Description: "B", Price: 95.5},
        },
        expected: 130.75,
    },
}

func TestHigherOrderFunctionCalculateOrderTotal(t *testing.T) {
    for _, test := range testCases {
        providerFunc := InMemoryLineItemsProviderFuncFactory(test.lineItems)

        if actual := CalculateOrderTotal(providerFunc); actual != test.expected {
            t.Fatalf("expected %.2f, actual %.2f", test.expected, actual)
        }
    }
}

This test suite is nearly identical to the one we used to test the final version of CalculateOrderTotal in the previous blog post. The line items and expected output are the same however instead of encapsulating the line items with an InMemoryOrderProvider struct that implements the OrderProvider interface we are using InMemoryLineItemsProviderFuncFactory to encapsulate the details in a function.

When running the test suite using the new CalculateOrderTotal it produces the same results as the previous version. The new implementation required very few changes to our code and still prevented the details of our IO from leaking into CalculateOrderTotal.

Now lets take a look at how the same function signature can be used to retrieve the line item data from a JSON file.

func FileBasedLineItemsProviderFuncFactory(filepath string) func() []LineItem {
    return func() []LineItem {
        file, err := ioutil.ReadFile(filepath)
        if err != nil {
            log.Fatalf("unable to read file %s", filepath)
        }

        var lineItems []LineItem
        err = json.Unmarshal(file, &lineItems)
        if err != nil {
            log.Fatalf("unable to parse json for file %s", filepath)
        }

        return lineItems
    }
}

The function FileBasedLineItemsProviderFuncFactory is another provider function factory that returns a function with the signature func() []LineItem based on the filepath parameter of the factory function.

func main() {
    providerFunc := FileBasedLineItemsProviderFuncFactory("data/order_items.json")
    orderTotal := CalculateOrderTotal(providerFunc)

    fmt.Printf("Your total comes to %.2f", orderTotal)
}

At this point our higher order function solution is providing more or less the same functionality as the interface solution from the previous blog post. For most programming languages this would be a good place to stop, however Go has a unique language feature that allows us to use function types as receiver arguments for methods opening up an additional means of abstraction using both higher order functions and interfaces together.

A receiver argument determines the type that can invoke a method in Go in the same way that defining a public method in a C# or Java class allows you to invoke the method from an instance of that class type using the receiver.methodName() syntax.

In Go a receiver can be any type, not just a struct. This allows us to attach a method to a function type and then use that method to satisfy the requirements of an interface.

To start off, we'll need to take our function signature func() []LineItem and assign it to a named type.

type ProviderFunc func() []LineItem

With the function type defined, we can now create a GetLineItems method that uses the ProviderFunc type as a receiver.

// the OrderProvider type from the previous blog post for reference
type OrderProvider interface {
    GetLineItems() []LineItem
}

func (f ProviderFunc) GetLineItems() []LineItem {
    return f()
}

The GetLineItems method of the ProviderFunc type returns the results of calling the function f. In other words we are able to satisfy the GetLineItems method of the OrderProvider interface by having the ProviderFunc function type invoke itself.

Before we can use our two factory functions to create ProviderFunc functions that satisfy the OrderProvider interface, we'll need to change their return type from func() []LineItem to ProviderFunc. Nothing needs to be changed in the body of the factories as the functions they return already satisfy the ProviderFunc type. They just have to be explicitly returned as such.

func InMemoryLineItemsProviderFuncFactory(lineItems []LineItem) ProviderFunc {
    // ...function body remains the same as before
}

func FileBasedLineItemsProviderFuncFactory(filepath string) ProviderFunc {
    // ...function body remains the same as before
}

Now the functions returned by both factories satisfy the parameter type of the new CalculateOrderTotal function (a ProviderFunc can still be used as a func() []LinetItem type) as well as the previous version of CalculateOrderTotal with its OrderProvider parameter type. Whether we use the old or new version of CalculateOrderTotal we are still able to switch data sources with relative ease, going from in memory to file based data retrieval without having to rewrite either version of CalculateOrderTotal.

func NewCalculateOrderTotal(providerFunc func() []LineItem) float64 {
    lineItems := providerFunc()

    var orderTotal float64
    for _, lineItem := range lineItems {
        orderTotal += lineItem.Price
    }

    return orderTotal
}

func PrevCalculateOrderTotal(provider OrderProvider) float64 {
    lineItems := provider.GetLineItems()

    var orderTotal float64
    for _, lineItem := range lineItems {
        orderTotal += lineItem.Price
    }

    return orderTotal
}

func main() {
    fileBasedProviderFunc := FileBasedLineItemsProviderFuncFactory("data/order_items.json")

    // using the file based provider function as a 'func() []LineItem' function type
    orderTotal := NewCalculateOrderTotal(fileBasedProviderFunc)
    fmt.Printf("Your total comes to %.2f", orderTotal)

    // using the file based provider function as an 'OrderProvider' interface type
    orderTotal = PrevCalculateOrderTotal(fileBasedProviderFunc)
    fmt.Printf("Your total comes to %.2f", orderTotal)

    lineItems := []LineItem{
        {Description: "Leather Recliner", Price: 2499},
        {Description: "End Table", Price: 249},
    }

    inMemoryProviderFunc := InMemoryLineItemsProviderFuncFactory(lineItems)

    // using the in memory provider function as a 'func() []LineItem' function type
    orderTotal = NewCalculateOrderTotal(inMemoryProviderFunc)
    fmt.Printf("Your total comes to %.2f", orderTotal)

    // using the in memory provider function as an 'OrderProvider' interface type
    orderTotal = PrevCalculateOrderTotal(inMemoryProviderFunc)
    fmt.Printf("Your total comes to %.2f", orderTotal)
}

Determining which method you use — a struct satisfying an interface type, a higher order function, or a function satisfying an interface type — will depend on details about the problems you're trying to solve, the application you're building, and personal preference. It is outside the scope of this blog post to discuss how those factors might impact your design decisions.

Regardless of which method best suites the requirements of your application, creating separation between business logic and IO through these abstractions will make your code resilient to change and easier to test.

All example code from both blog posts on abstracting application IO can be found here.

Abstracting Application IO Using Interfaces

Jake Witcher — Sun, 31 Jan 2021 01:23:15 +0000

Dependency on input and output (IO) resources like databases, APIs, and user input is inevitable in software development and application design. Where you choose to include these touchpoints with IO in your code will have a substantial impact on the long term resuse and extensibility of your code as well as your ability to test it. Interfaces provide a means for hiding the implementation details required by IO interactions so that you can write the majority of your code independent of concerns about where data is coming from or where it is going.

To illustrate how functions and methods can be refactored from knowing everything about your application's IO to knowing nothing about it, we will create a small application that calculates the total price of a sales order from a list of line items. The order information is retrieved by the application from a locally stored JSON file. Each line item is represented by a struct with a description and a price.

type LineItem struct {
    Description string  `json:"description"`
    Price       float64 `json:"price"`
}

As a first attempt we can write a function that knows everything about where the order details are stored and how to retrieve them. This function reads in the file, deserializes the contents, and then iterates over each line item to produce the total cost.

func CalculateOrderTotal() float64 {
    filePath := "data/order_items.json"
    file, err := ioutil.ReadFile(filePath)
    if err != nil {
        log.Fatalf("unable to read file %s", filePath)
    }

    var lineItems []LineItem
    err = json.Unmarshal(file, &lineItems)
    if err != nil {
        log.Fatalf("unable to parse json for file %s", filePath)
    }

    var orderTotal float64
    for _, lineItem := range lineItems {
        orderTotal += lineItem.Price
    }

    return orderTotal
}

This approach is problematic for a number of reasons. The first red flag is that this function is not focused on doing just one thing. It does more than calculate order totals as the name implies; it is reading in files and deserializing JSON as well.

The function is also restricted to retrieving the order information from one specific source. This could be addressed by passing in the file path as an argument but what happens if requirements change and order information must be retrieved from additional forms of persistent storage like a relational database or blob storage in the cloud? This first attempt at implementing CalculateOrderTotal does not allow us to reuse the order total calculation logic for anything other than JSON files.

And there's one more problem with this function. It's difficult to test.

func TestCalculateOrderTotal(t *testing.T) {
    var expected float64 = 3476
    actual := CalculateOrderTotal()

    if actual != expected {
        t.Fatalf("expected %.2f, actual %.2f", expected, actual)
    }
}

Running this code actually causes ioutil.ReadFile to return a file not found error because the hardcoded file path is relative and the test directory uses a different root path when resolving relative file paths than the one used by the application's main function. Again, this could be resolved by passing in the file path as an argument but tests will be dependent on the file system and CalculateOrderTotal will still be responsible for too many things and dependent on a very specific data retrieval implementation.

We can move a step in the right direction by extracting the file reading and JSON deserialzing code into a method of a struct designed specifically to deserialize JSON from a file.

type JsonOrderProvider struct {
    FilePath string
}

func (r JsonOrderProvider) GetLineItems() []LineItem {
    file, err := ioutil.ReadFile(r.FilePath)
    if err != nil {
        log.Fatalf("unable to read file %s", r.FilePath)
    }

    var lineItems []LineItem
    err = json.Unmarshal(file, &lineItems)
    if err != nil {
        log.Fatalf("unable to parse json for file %s", r.FilePath)
    }

    return lineItems
}

Now the file path is no longer hard coded making it easier to change which files are read. CalculateOrderTotal can be rewritten to accept this new struct as an argument and is now focused on calculating order totals rather than dealing with local file systems.

func CalculateOrderTotal(provider JsonOrderProvider) float64 {
    lineItems := provider.GetLineItems()

    var orderTotal float64
    for _, lineItem := range lineItems {
        orderTotal += lineItem.Price
    }

    return orderTotal
}

Testing is much easier as well, we can now adjust the relative file path so that the test project is able to locate the JSON file and we can even use several JSON files to test our function with a variety of inputs.

func TestCalculateOrderTotal(t *testing.T) {
    var expected float64 = 3476
    provider := JsonOrderProvider{FilePath: "../data/order_items.json"}
    actual := CalculateOrderTotal(provider)

    if actual != expected {
        t.Fatalf("expected %.2f, actual %.2f", expected, actual)
    }
}

However this solution still isn't ideal because both our tests and the CalculateOrderTotal function are dependent on order information being persisted through the local file system. If our requirements for persistence changed and we needed to calculate order totals from information stored in a SQL database or a cloud datastore we would not only need to write a new provider struct, but also a second CalculateOrderTotal function with a different parameter type.

Both problems can be solved by introducing an interface.

type OrderProvider interface {
    GetLineItems() []LineItem
}

By creating an OrderProvider interface we are able to describe a contract about the type of behavior we expect without knowing anything about the implementation details. The JsonOrderProvider already satisfies the requirements for this interface (in Go, interface implementation is implicit) so let's create a new provider, one that serves order information from an in-memory list of line items so that we can improve our testing capabilities.

type InMemoryOrderProvider struct {
    LineItems []LineItem
}

func (r InMemoryOrderProvider) GetLineItems() []LineItem {
    return r.LineItems
}

We also need to change the parameter of CalculateOrderTotal to be an OrderProvider interface type instead of the concrete type, JsonOrderProvider.

func CalculateOrderTotal(provider OrderProvider) float64 {
    lineItems := provider.GetLineItems()

    var orderTotal float64
    for _, lineItem := range lineItems {
        orderTotal += lineItem.Price
    }

    return orderTotal
}

Because our previous parameter type, JsonOrderProvider, already satisfied the GetLineItems method of the OrderProvider interface, the body of CalculateOrderTotal does not need to change, only the function's parameter type. Everything else works the same.

Now let's see if this shift from concrete type to interface type improves our testing capabilities.

var testCases = []struct {
    provider InMemoryOrderProvider
    expected float64
}{
    {
        provider: InMemoryOrderProvider{
            LineItems: []LineItem{
                {Description: "A", Price: 85},
                {Description: "B", Price: 15},
            },
        },
        expected: 100,
    },
    {
        provider: InMemoryOrderProvider{
            LineItems: []LineItem{
                {Description: "A", Price: 35.25},
                {Description: "B", Price: 95.5},
            },
        },
        expected: 130.75,
    },
}

func TestCalculateOrderTotal(t *testing.T) {
    for _, test := range testCases {
        if actual := CalculateOrderTotal(test.provider); actual != test.expected {
            t.Fatalf("expected %.2f, actual %.2f", test.expected, actual)
        }
    }
}

There is some additional code in this test, a list of anonymous struct types with provider and expected output fields. This list serves as a suite of test cases to be used by the test method TestCalculatedOrderTotal. This is a testing convention common to Go unit tests and is an alternative to the more common convention of having separate methods for each test case.

Testing conventions aside, we can now define several test cases in the same file where our test method is defined without any dependency on the local file system. The function CalculateOrderTotal no longer contains implementation details for retrieving the data nor any hint as to where the data is coming from. The OrderProvider interface ensures that if the method GetLineItems is invoked, the function CalculateOrderTotal will receive a list of LineItem structs and that's the only guarantee this function needs to perform its intended work.

In our main function we can prove that CalculateOrderTotal works the same whether we use a JsonOrderProvider or an InMemoryOrderProvider.

func main() {
    jsonProvider := JsonOrderProvider{FilePath: "data/order_items.json"}

    orderTotal := CalculateOrderTotal(jsonProvider)
    fmt.Printf("Your total comes to %.2f", orderTotal)

    inMemoryProvider := InMemoryOrderProvider{
        LineItems: []LineItem{
            {Description: "Leather Recliner", Price: 2499},
            {Description: "End Table", Price: 249},
        },
    }

    orderTotal = CalculateOrderTotal(inMemoryProvider)
    fmt.Printf("Your total comes to %.2f", orderTotal)
}

At some point you may have been wondering, "why not pass in the list of line items as an argument rather than going through all this trouble to create an interface?" and for a function like CalculateOrderTotal that is likely how you would write it in a real application. However this does not remove the need for the data to be retrieved. By refactoring CalculateOrderTotal to take a list of line items instead of an OrderProvider interface you are deferring the responsibility of data retrieval to its caller.

It might make sense for the calling function to know about file systems and to use functions like ioutil.ReadFile but in most applications you will have this middle layer of code that needs to retrieve data that ultimately comes from an IO resource while still deferring the specifics as to which resource it is coming from to the edges of the application, keeping the core business logic free of such details.

Using interfaces to abstract away the specifics of your IO operations provides a way to push those IO implementation details further out, keeping them close to the edges while still granting functions closer to the middle the ability to influence what data is retrieved and when. This flexibility allows you to design solutions that are adaptable to change and testable in isolation from IO dependencies.

Interested in learning more? I've written a second blog post on abstracting away the implementation details of application IO using higher order functions.

All example code from both blog posts can be found here.

Interfaces and the Principle of Least Privilege

Jake Witcher — Tue, 12 Jan 2021 13:46:19 +0000

Interfaces are an incredibly powerful abstraction for writing extensible and readable code yet developing an intuition for when and how to use them does not always come easily. While there's no substitute for time and practice, the principle of least privilege — a concept most commonly applied to infosec concerns like IAM (Identity and Access Management) — can provide a helpful framework for thinking about the role interfaces play in your code.

The Principle of Least Privilege

When writing a function with an interface rather than a concrete type as a parameter, you are allowing the function to access only the behavior exposed by the interface. The actual object or struct used as an argument may have additional data or functionality but these properties are hidden from the function.

Limiting the visible behavior of a function's parameter is similar to restricting the services and resources the members of a cloud based application can access. For your product you may want an end user to be able to view data but not update it or add additional entries. For that same application you may have a serverless function that needs not only to view the data but to make changes to it as well. And what about deleting content? That may be an action reserved for developers with administrative privileges.

When managing IAM, you could provide all three of these members with administrative level access thus meeting their combined access needs. While this solution enables the developers to conduct their work appropriately it places your data at risk of being illegitimately updated or deleted by end users and the serverless function. This "least common denominator" approach to IAM violates the principle of least privilege.

The principle of least privilege states that a user of a given resource or service should only have the bare minimum permissions required to perform their expected, legitimate tasks.

In the case of a function or method, we might rewrite this principle to state that a function's parameter type should only reveal enough about the type's behavior for the function to perform its expected, legitimate tasks.

Illustrating the Principle of Least Privilege in Code

To illustrate this idea we're going to create interfaces and structs in Go based on resources and roles in GCP (Google Cloud Platform).

We will be building an application that simulates Google Cloud Storage — a service for blob storage that can be divided into logical groupings called buckets and is useful for storing many types of data like images or zip files.

Playing the role of the cloud storage bucket is a Go struct (similar to a class in object-oriented languages). This struct has one property — Blobs — which is a map of strings as keys and any data type as the values.

// interface{} in Go is an interface that defines no behavior
// and is similar to the `any` type in other languages
type CloudStorageBucket struct {
    Blobs map[string]interface{}
}

Next we'll define a function for creating a new instance of CloudStorageBucket as well as some helper methods for viewing, creating, and deleting blobs.

func NewCloudStorageBucket() *CloudStorageBucket {
    return &CloudStorageBucket{
        Blobs: make(map[string]interface{}),
    }
}

// the variable 'ok' is a boolean value that will return true if 
// the key you're trying to access exists for the given map
func (cs CloudStorageBucket) ViewBlob(key string) (interface{}, error) {
    if blob, ok := cs.Blobs[key]; ok {
        return blob, nil
    }

    return nil, fmt.Errorf("no blob found with key %s", key)
}

func (cs CloudStorageBucket) CreateBlob(key string, contents interface{}) error {
    if _, ok := cs.Blobs[key]; !ok {
        cs.Blobs[key] = contents
        return nil
    }

    return fmt.Errorf("blob with key %s already exists", key)
}

func (cs CloudStorageBucket) DeleteBlob(key string) (interface{}, error) {
    if blob, ok := cs.Blobs[key]; ok {
        delete(cs.Blobs, key)
        return blob, nil
    }

    return nil, fmt.Errorf("no blob found with key %s", key)
}

With that, the basic implementation of a cloud storage bucket is complete.

Defining the Storage Bucket Permissions as Interfaces

Turning our attention to the access permissions on the cloud storage bucket, we'll create three interfaces that expose the behaviors available to the users of this service — viewing, creating, and deleting.

type CloudStorageViewer interface {
    ViewBlob(cs *CloudStorageBucket, key string) (interface{}, error)
}

type CloudStorageCreator interface {
    CreateBlob(cs *CloudStorageBucket, key string, contents interface{}) error
}

type CloudStorageDeleter interface {
    DeleteBlob(cs *CloudStorageBucket, key string) (interface{}, error)
}

Each interface defines only one method because it is focused on exposing only one behavior. They can also be composed together when more complex use cases emerge.

// creating an admin interface by composing together three 
// single method interfaces
type CloudStorageAdmin interface {
    CloudStorageViewer
    CloudStorageCreator
    CloudStorageDeleter
}

Creating the Storage Bucket Users

To see this in action, we will implement these interfaces on three different types of users: an end user who should only be able to view the data in cloud storage, a serverless cloud function that should be able to view existing blobs and create new ones, and a developer that has admin access and can perform all three actions.

// Go does not have explicit interface implementation. Any struct 
// with a method signature identical to the one defined on an 
// interface is treated as an implicit implementation. In that 
// respect it is a bit like duck typing in languages like Ruby.

type EndUser struct {}

func (EndUser) ViewBlob(cs *CloudStorageBucket, key string) (interface{}, error) {
    return cs.ViewBlob(key)
}


type CloudFunction struct {}

func (CloudFunction) ViewBlob(cs *CloudStorageBucket, key string) (interface{}, error) {
    return cs.ViewBlob(key)
}

func (CloudFunction) CreateBlob(cs *CloudStorageBucket, key string, contents interface{}) error {
    return cs.CreateBlob(key, contents)
}


type Developer struct {}

func (Developer) ViewBlob(cs *CloudStorageBucket, key string) (interface{}, error) {
    return cs.ViewBlob(key)
}

func (Developer) CreateBlob(cs *CloudStorageBucket, key string, contents interface{}) error {
    return cs.CreateBlob(key, contents)
}

func (Developer) DeleteBlob(cs *CloudStorageBucket, key string) (interface{}, error) {
    return cs.DeleteBlob(key)
}

Applying the Principle of Least Privilege to Function Design

With the interfaces defined and implemented for the three types of users, we can now create functions that use these interfaces as parameter types, beginning with the function ViewCloudStorageBlob which will accept any type that implements the ViewBlob method of the CloudStorageViewer interface, restricting its ability to do more than view blobs in our cloud storage struct.

func ViewCloudStorageBlob(cs *CloudStorageBucket, viewer CloudStorageViewer) {
    contents, err := viewer.ViewBlob(cs, "foo")
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Printf("key foo contains %v", contents)
}

func main() {
    storage := NewCloudStorageBucket()

    endUser := EndUser{}
    cloudFunction := CloudFunction{}
    developer := Developer{}

    ViewCloudStorageBlob(storage, endUser)
    ViewCloudStorageBlob(storage, cloudFunction)
    ViewCloudStorageBlob(storage, developer)
}

Because all three of our user structs implement the ViewBlob method, they are all acceptable candidates for the second argument of ViewCloudStorageBlob. This function does not need to know about creating or deleting blobs, nor does it need to know additional details about the users that implement CloudStorageViewer. The only thing ViewCloudStorageBlob needs to know is that ViewBlob can be invoked. That's it. By creating small modular interfaces we provide single-purpose functions with only the information required for them to satisfy that purpose.

The same steps can be applied to a function that accepts a CloudStorageCreator type.

func CreateCloudStorageBlob(cs *CloudStorageBucket, creator CloudStorageCreator) {
    err := creator.CreateBlob(cs, "foo", []int{1,2,3})
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println("new blob with key 'foo' successfully created")
}

func main() {
    storage := NewCloudStorageBucket()

    endUser := EndUser{}
    cloudFunction := CloudFunction{}
    developer := Developer{}

    CreateCloudStorageBlob(storage, cloudFunction)
    CreateCloudStorageBlob(storage, developer)
}

This time our function can only be called with two of the three user types as arguments. And even though both CloudFunction and Developer still have the ability to view a storage blob, CreateCloudStorageBlob has no knowledge of this behavior because it does not need to know about it.

If we try to use the EndUser struct as a CloudStorageCreator type, we get this message when building the project:

cannot use endUser (type EndUser) as type CloudStorageCreator in argument to CreateCloudStorageBlob:
EndUser does not implement CloudStorageCreator (missing CreateBlob method)

EndUser does not satisfy the requirements of a CloudStorageCreator type and therefore we get a compile time error message letting us know that the function is trying to invoke behavior on a type that does not have access to the CreateBlob action.

Lastly, let's write a function that requires a type with full access, one that uses the CloudStorageAdmin interface as a parameter type. First we'll create a function for deleting cloud storage blobs, similar to the view and create functions we've already made.

func DeleteCloudStorageBlob(cs *CloudStorageBucket, deleter CloudStorageDeleter) {
    contents, err := deleter.DeleteBlob(cs, "foo")
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Printf("successfully deleted %v", contents)
}

Before we write a function that requires the CloudStorageAdmin type, let's confirm something about the developer struct, that it works as an argument for all three of the other functions that were made for viewing, creating, and deleting.

func main() {
    storage := NewCloudStorageBucket()

    developer := Developer{}

    ViewCloudStorageBlob(storage, developer)
    CreateCloudStorageBlob(storage, developer)
    DeleteCloudStorageBlob(storage, developer)
}

Because the Developer struct satisfies the CloudStorageViewer, CloudStorageCreator and CloudStorageDeleter interfaces, it works as an argument for the three functions that accept those types.

If it works as all three types individually, it should also work as an argument for a function that explicitly takes a CloudStorageAdmin type.

func PerformAdminActions(cs *CloudStorageBucket, admin CloudStorageAdmin) {
    ViewCloudStorageBlob(cs, admin)
    CreateCloudStorageBlob(cs, admin)
    DeleteCloudStorageBlob(cs, admin)
}

func main() {
    storage := NewCloudStorageBucket()

    developer := Developer{}

    PerformAdminActions(storage, developer)
}

The previous functions required very little information about the concrete types they were accepting as arguments and the small, one method interfaces enabled us to hide any additional behavior. Now that we have a function that requires the ability to perform multiple actions against the cloud storage struct, we can leverage those small, one method interfaces in a different way, by composing them together into a new interface.

The end result is that our Developer struct is useable by functions that require very little information, as with the ViewCloudStorageBlob function, as well as by functions that require a lot of information like PerformAdminActions. In either case only the required behaviors are exposed to the function by its parameter type even though the underlying struct is the same.

The Benefits of Understanding and Applying the Principle of Least Privilege

The benefits of applying the principle of least privilege to interface design extend to more than just code written to mimic a cloud service. The same thought process can be applied to the design of any application, making your code more extensible and your intent clearer.

By considering what a function should be permitted to know about its parameters — what those parameters can do rather than what they are — your functions will benefit from greater reuse. Just like predefined roles in cloud IAM make it easier to grant new users a set of related permissions, when an interface is used as a function's parameter type, new concrete types that implement the interface can also be used as an argument without having to change the function itself.

Interfaces designed and named with exposed behavior in mind also communicate intent. In the same way that the GCP IAM role cloudfunctions.invoker communicates what actions a user assigned that role can perform, so too do interfaces like io.Reader or io.Writer communicate the intent of the functions that use them as parameter types. By communicating intent through interfaces, you are letting other developers and your future self know what behaviors a function is and is not expected to invoke on its parameters.

In addition to the benefits to your code, understanding and applying the principle of least privilege will also benefit you as a developer by providing a tangible way of thinking about the role of interfaces, helping you spot opportunities to refactor your code to make use of them. When your ability to spot legitimate use cases for interfaces improves, so too will the readability and extensibility of your code.

Setting Up Authorization for HTTP Cloud Functions in GCP

Jake Witcher — Sun, 27 Dec 2020 15:20:45 +0000

Cloud functions in GCP (Google Cloud Platform) are a lightweight, stateless, and serverless option for executing code that is triggered by an event. They are equivalent to lambdas in AWS (Amazon Web Services) or Azure Functions in Microsoft Azure. Events that trigger a cloud function include Pub/Sub messages, Cloud Storage events like creating or deleting a storage object, and HTTP requests.

As with any cloud service, you want to make sure that the people and resources interacting with your cloud function are authorized to do so. With Pub/Sub and Cloud Storage triggers the responsibility for restricting who has the ability to invoke your cloud function is deferred to the IAM (Identity and Access Management) associated with those resources, however with HTTP triggers you will have to manage permissions by setting up and assigning one or more Google account and/or service account with the authorization to invoke your cloud function.

By default, a new cloud function created in the GCP Console will require authentication. When deploying a new cloud function from the gcloud SDK, you will be asked whether or not you would like to allow unauthenticated invocations.

gcloud functions deploy my-new-function `
--entry-point handle_request`
--runtime python38 `
--trigger-http

Allow unauthenticated invocations of new function
[my-new-function]? (y/N)?

By allowing unauthenticated invocations, you are making your cloud function's endpoint available to the open internet. If allowing unauthenticated invocations is the desired behavior, you can bypass this question when deploying via the SDK by using the optional flag --allow-unauthenticated with the gcloud functions deploy command. Unauthenticated cloud function invocations should be an exception; for most use cases you will want some form of authentication and authorization.

Depending on who or what is invoking your cloud function the process for setting up authentication will vary, however there are two requirements common to all types of authentication:

The person or service authorized to invoke the cloud function must be assigned the cloudfunctions.invoker role or some other role with the cloudfunctions.invoke permission.
The person or service authorized to invoke the cloud function must send a token along with the HTTP request to prove that they are authorized to invoke the cloud function.

Describing the process for all use cases is beyond the scope of this article. Instead the focus will be on setting up authentication for one cloud function to invoke another cloud function and on setting up authentication locally so that you can test your secure cloud functions.

Function-to-Function Invocation

Before setting up authentication you will need to have a function written in one of the supported languages with a few lines of code covering the basics of a response to an HTTP request. All of the example code in this post has been written in Python however the principles will be the same for all supported languages.

This first cloud function is the caller function and it is responsible for validating the initial HTTP request, applying some business logic to the contents of that request, and finally invoking the second cloud function with an HTTP request that is authenticated by a token.

import os
import requests

from dotenv import load_dotenv


load_dotenv()
CALLED_CLOUD_FUNCTION_URL = os.getenv('CALLED_CLOUD_FUNCTION_URL')


def send_request_to_called_cloud_function(request):
    request_body = request.get_json(silent=True)

    if not is_valid_request(request_body):
        return 'bad request, missing required field "foo"', 400

    called_func_request = prepare_called_func_request(request_body)

    token = request_token()
    headers = create_request_headers(token)

    called_func_response = requests.post(CALLED_CLOUD_FUNCTION_URL, json=called_func_request, headers=headers)

    return called_func_response.text, called_func_response.status_code

To create the token, the caller cloud function makes an HTTP request to its metadata server.

def request_token():
    metadata_server_url = 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience='
    token_request_url = metadata_server_url + CALLED_CLOUD_FUNCTION_URL
    headers = {'Metadata-Flavor': 'Google'}

    token_response = requests.get(token_request_url, headers=headers)
    token = token_response.text

    return token


def create_request_headers(token):
    headers = {
        'content-type': 'application/json',
        'authorization': f'bearer {token}'
    }

    return headers

Whether it's a virtual machine, a cloud function, or an app engine app, all of the compute resources in GCP have a server where relevant metadata about the resource is stored. The request to the metadata server url in the request_token function is checking the caller cloud function's metadata and verifying that the service account attached to it is authorized to invoke the intended audience of the request, the called cloud function. If the service account for the caller was setup correctly, your function will receive a valid token to send to the second cloud function.

How do you create a service account that is authorized to invoke a cloud function? You can either create a new service account using the Console or gcloud SDK and give it the cloudfunctions.invoker role, or you can manage your IAM resources using an IAC (Infrastructure as Code) solution like Terraform.

resource "google_service_account" "caller_cloud_function_sa" {
    project      = "my-cloud-function-project"
    account_id   = "caller-cloud-function-sa"
    display_name = "Caller Cloud Function Service Account"
}

resource "google_cloud_functions_function" "caller_cloud_function" {
    project               = "my-cloud-function-project"
    name                  = "caller-cloud-function"
    entry_point           = "send_request_to_called_cloud_function"
    runtime               = "python38"
    service_account_email = google_service_account.caller_cloud_function_sa.email
    trigger_http          = true
}

resource "google_cloudfunctions_function_iam_member" "cloud_function_invoker" {
    project        = google_cloud_functions_function.caller_cloud_function.project
    region         = google_cloud_functions_function.caller_cloud_function.region
    cloud_function = google_cloud_functions_function.caller_cloud_function.name

    role   = "roles/cloudfunctions.invoker"
    member = "serviceAccount:${google_service_account.caller_cloud_function_sa.email}"
}

With the service account created and the caller cloud function setup to request a token from its metadata server, the only thing left to do is write the code for the called cloud function

from flask import jsonify


def respond_to_caller_cloud_function_request(request):
    request_body = request.get_json(silent=True)

    if not is_valid_request(request_body):
        return 'bad request, missing required field "bar"', 400

    new_entity = save_request_body(request_body)

    return jsonify(new_entity), 201

and deploy both cloud functions from the Console, the gcloud SDK, or through some other means like Terraform.

# using powershell and the gcloud sdk to deploy both cloud functions
gcloud functions deploy caller-cloud-function `
--entry-point send_request_to_called_cloud_function `
--runtime python38 `
--service-account caller-cloud-function-sa@my-cloud-function-project.iam.gserviceaccount.com `
--trigger-http `
--allow-unauthenticated

gcloud functions deploy called-cloud-function `
--entry-point respond_to_caller_cloud_function_request `
--runtime python38 `
--trigger-http

Note that the caller function is currently setup to allow unauthenticated access. To make this example more secure, you will either want to change this function to use one of the other triggers or set up HTTP authentication for it as well.

Testing Your Secured Cloud Function

Now that the called cloud function is secure, testing it from your local machine will require you to send a token along with the request just like the caller cloud function does.

To get a token, first make sure you are logged into your Google account using the gcloud SDK, and that you have the authorization to invoke a cloud function. This requires the cloudfunctions.invoker role or any other role that includes the cloudfunctions.invoke permission.

If you have been assigned one of the basic roles of editor or owner you will have the cloudfunctions.invoke permission already, otherwise you will need to check the list of roles/permissions assigned to your account and possibly request to have a role with the cloudfunctions.invoke permission added by someone on the project with the ability to grant IAM roles.

To log into a Google account using the gcloud SDK, use the command

gcloud auth login

This will open up a tab in your browser where you will be prompted to log in using your account's email address, password, and any multi-factor authentication required by your account's security settings.

Once you are logged in, you can request a token with the command

gcloud auth print-identity-token

You can also do this programmatically using Python

def request_identity_token():
    stream = os.popen('gcloud auth print-identity-token')
    token = stream.read()

    return token.strip()

and then send your HTTP request in the same way the caller function does.

import requests
from dotenv import load_dotenv


load_dotenv()
CALLED_CLOUD_FUNCTION_URL = os.getenv('CLOUD_FUNCTION_URL')


def send_test_request():
    content = {'foo': 'bar'}
    token = request_identity_token()
    headers = { 'content-type': 'application/json', 'authorization': f'bearer {token}'

    response = requests.post(CALLED_CLOUD_FUNCTION_URL, json=content, headers = headers)

With that you are ready to start deploying HTTP cloud functions that require authentication and to test those functions from your local machine. If your use case requires authorization for end users or some other service or resource the particulars may vary but the general process of attaching the cloudfunctions.invoker role to a Google account or service account and having the user or resource create a token before calling the function will be the same.

Using .env Files for Environment Variables in Python Applications

Jake Witcher — Thu, 24 Dec 2020 15:25:58 +0000

Applications are made to be deployed. At some point during development you will need to think about the environment in which your application will run and the potentially sensitive or environment specific information your application will need to perform its tasks.

Environment variables are one of the key ways software developers provide an application with this kind of information, however running and testing an application locally that is dependent on environment variables can be a hassle if you are setting those variables on your local machine's environment.

The process of setting or changing an environment variable is time consuming and over time the number of environment variables you have to manage grows out of control. Eventually naming conflicts becomes an issue and every new variable requires a lengthy prefix to distinguish itself from similar variables.

Using a .env file will enable you to use environment variables for local development without polluting the global environment namespace. It will also keep your environment variable names and values isolated to the same project that utilizes them.

A .env file is a text file containing key value pairs of all the environment variables required by your application. This file is included with your project locally but not saved to source control so that you aren't putting potentially sensitive information at risk.

# environment variables defined inside a .env file
GCP_PROJECT_ID=my-project-id
SERVICE_ACCOUNT_FILE=path/to/serviceAccountCredentials
STORAGE_BUCKET_NAME=my-super-important-data

Nearly every programming language has a package or library that can be used to read environment variables from the .env file instead of from your local environment. For Python, that library is python-dotenv. Once the library is installed, an average use case for python-dotenv only requires adding two lines of code to your project.

from dotenv import load_dotenv

load_dotenv()

load_dotenv() will first look for a .env file and if it finds one, it will load the environment variables from the file and make them accessible to your project like any other environment variable would be.

import os
from dotenv import load_dotenv

load_dotenv()

GCP_PROJECT_ID = os.getenv('GCP_PROJECT_ID')
SERVICE_ACCOUNT_FILE = os.getenv('SERVICE_ACCOUNT_FILE')
STORAGE_BUCKET_NAME = os.getenv('STORAGE_BUCKET_NAME')

If an environment variable is not found in the .env file, load_dotenv will then search for a variable by the given name in the host environment. This means that when your project is running locally and the .env file is present, the variables defined in the file will be used. When your project is deployed to a host environment like a virtual machine or Docker container where the .env file is not present, the environment variables defined in the host environment will be used instead.

By default load_dotenv will look for the .env file in the current working directory or any parent directories however you can also specify the path if your particular use case requires it be stored elsewhere.

from dotenv import load_dotenv
from pathlib import Path

dotenv_path = Path('path/to/.env')
load_dotenv(dotenv_path=dotenv_path)

For most applications, that's all the information you should need to be productive with python-dotenv however there are a few additional features which you can read about in the python-dotenv documentation.

One of the benefits of using a .env file is that it becomes much easier to develop with environment variables in mind early on. Thinking about these things at an earlier stage of the development process will make it easier for you to get your application ready for deployment.

If you are deploying your application in a Docker container, you can seamlessly transition to running and testing your application in a locally run container by using the flag --env-file .env with your docker run command. Docker will then set the same environment variables you've been using through python-dotenv in the container's environment.

Without a .env file, environment variables are a cumbersome albeit necessary part of testing your application locally. By using a .env file and the python-dotenv library, working with environment variables becomes much more manageable and will get you developing with deployment in mind right from the start.