DEV Community: Hector Castro

Parsing Data in Rust with Nom

Hector Castro — Fri, 23 Dec 2022 05:00:00 +0000

This is my third year participating in Advent of Code, but the first using Rust! Since I’m new to the Rust ecosystem, I’ve been dependent on others to steer my third-party library selections. As an example, Day 15 (like most days) presented some interesting string parsing requirements. Luckily, I was guided toward an excellent parser combinator library, affectionately named nom, via Chris Biscardi¹.

Beacon exclusion zone

The Day 15 challenge requires you to track sensors, beacons, and their coordinates. The raw input for this looks like:

Sensor at x=2, y=18: closest beacon is at x=-2, y=15
Sensor at x=9, y=16: closest beacon is at x=10, y=16
Sensor at x=13, y=2: closest beacon is at x=15, y=3
Sensor at x=12, y=14: closest beacon is at x=10, y=16
Sensor at x=10, y=20: closest beacon is at x=10, y=16
Sensor at x=14, y=17: closest beacon is at x=10, y=16
Sensor at x=8, y=7: closest beacon is at x=2, y=10
Sensor at x=2, y=0: closest beacon is at x=2, y=10
Sensor at x=0, y=11: closest beacon is at x=2, y=10
Sensor at x=20, y=14: closest beacon is at x=25, y=17
Sensor at x=17, y=20: closest beacon is at x=21, y=22
Sensor at x=16, y=7: closest beacon is at x=15, y=3
Sensor at x=14, y=3: closest beacon is at x=15, y=3
Sensor at x=20, y=1: closest beacon is at x=15, y=3

While this text is parsable with regular expressions, or a combination of well-placed string splits, using a parsing library helps break things down in a structured way (which can sometimes be beneficial for part 2 challenges).

Presuming we have structs for Sensor and Beacon that look like the ones below, we can start building out the parsing logic.

struct Sensor {
    x: i64,
    y: i64,
}

struct Beacon {
    x: i64,
    y: i64,
}

Parsing with Nom

First, we’ll parse out each line of input, along with the part of the line relevant to either a Sensor or a Beason. Second, we’ll parse out the coordinates and populate them into instances of Sensor and Beacon.

For the first part, everything is contained in a function that takes the raw input as a string slice (&str) and returns an IResult. An IResult is a container for the result of a nom parsing function. The string slice component of an IResult is the remaining unparsed input, and the Vec(Sensor, Beacon) is our expected parsing result.

fn map(input: &str) -> IResult<&str, Vec<(Sensor, Beacon)>> {
    let (input, reports) = separated_list1(
        line_ending,
        preceded(
            tag("Sensor at "),
            separated_pair(
                position.map(|(x, y)| Sensor { x, y }),
                tag(": closest beacon is at "),
                position.map(|(x, y)| Beacon { x, y }),
            ),
        ),
    )(input)?;

    Ok((input, reports))
}

Inside the map function, we start off with separated_list1, which helps us break up the input into lines. The first argument is line_ending, which matches line endings of both the \n and \r\n variety. The second argument starts with preceded, which isolates everything after the Sensor at tag in the line and supplies it to separated_pair. separated_pair in turn helps parse out what is on either side of the : closest beacon is at tag. In this case, those are the coordinate pairs for Sensor and Beacon, respectively. To parse them, we’ll define another function called position.

The position function helps extract the values of coordinate pairs. As you can see, it has similar arguments to map, and an IResult return value. However, the types in the IResult are a bit different here. The second argument is a tuple, for the x and y coordinates, both i64.

fn position(input: &str) -> IResult<&str, (i64, i64)> {
    separated_pair(
        preceded(tag("x="), complete::i64),
        tag(", "),
        preceded(tag("y="), complete::i64),
    )(input)
}

Right away, we jump into separated_pair again. This parses out both sides of the ,, while preceded isolates the value after either x= or y=. The second argument of preceded is another parsing function—a character::complete::i64, which matches the coordinate integer value.

Coming back to the map function, we (somewhat confusingly) call map on the position parsing result to get the parsed values. That allows us to destructure the tuple and use the values to construct the Sensor and Beacon struct literals.

Now, if we use the dbg! macro on the result of a call to map with test input, we should see something like:

map = [
    (
        Sensor {
            x: 2,
            y: 18,
        },
        Beacon {
            x: -2,
            y: 15,
        },
    ),
    (
        Sensor {
            x: 9,
            y: 16,
        },
        Beacon {
            x: 10,
            y: 16,
        },
    ),

// . . .

]

Look at that beautifully structured data!

Conclusion

Reasonably painless, and well-structured—that’s parsing data with Rust and Nom! If you’re interested in taking a closer look at Nom, I encourage you to review this handy list of its available parsers and combinators.

I highly recommend checking out Chris’ phenomenal Advent of Code solution videos. I could not have dreamt of a better resource to get up-to-speed quickly, with Rust. ↩

Every Other Friday Off Work Schedule

Hector Castro — Mon, 27 Jun 2022 04:00:00 +0000

For the last six months, I’ve adopted a work schedule where you tally up extra hours in the first nine days of a two-week range, and take the second Friday off (also known as a 9/80 work schedule¹). Below is a diagram that illustrates one way to do it.

When I first adopted this work schedule, I didn’t think I’d value it as much as my peers. For better or worse (increasingly, worse), I’ve come from a long line of work environments where long hours were rewarded. Every other Friday off? Sure, that’s nice—but that’s not for leaders.

However, after a month of practice, it easily became one of the best work schedule arrangements I’ve ever participated in. Below are some supporting reasons why (from the perspective of an employee—me). If you consider yourself a forward-thinking leader, and have the authority to implement an alternative work schedule like this for your teams, please give it some serious consideration.

Relief in knowing others are off too

This may be a side effect of the previous work cultures I referenced above, but for me, I enjoy days off much more when I know other members of the team are off too. The probability of having your day interrupted by a chat DM goes down significantly. There is also a reduced concern of missing previously scheduled meetings or timely emails.

More opportunity for decompression

The older I get, the more responsibilities I assume outside of work. Increasingly, the weekend is less a block of time to unwind before the next work week, and more the only opportunity to meet personal obligations that require more time than weekday evenings afford.

Maybe it’s cleaning out the garage, or rearranging the home office, or picking out a nice gift for a loved one. Whatever it is, now there is a whole additional day every two weeks to get it done. That leaves the traditional weekend days with more of an opportunity for much-needed decompression.

More quality time with my kid

This is a bit biased toward folks with younger kids (i.e., not yet in school, or not yet in a serious grade), but the time I spend with my kid on these Fridays off is much higher quality than that of a usual weekend.

When we plan to go somewhere, like the aquarium or a museum, it is much easier to get tickets. There are also generally a lot less people around (❤️ introverts). It’s enabled us to comfortably enjoy experiences like bowling, mini golfing, and wizarding² during a pandemic.

Conclusion

A 9/80-style work schedule is personally, very compelling. It unlocks a level of work/life balance I haven’t experienced in a work setting since I started working from home.

It may not be for everyone, though. Work schedule changes require a base level of individual and team-level maturity. But, if that’s present, most software development teams should be able to adopt a work schedule change like this and not miss a beat.

In 2022, hiring for software-focused roles is tough and differentiators are hard to come by. An alternative work schedule that improves work/life balance and doesn’t compromise the business can go a long way on the recruiting front.

Turning Lemons into Topologically Sorted Lemonade

Hector Castro — Fri, 09 Apr 2021 00:00:00 +0000

In a recent interview, I was asked to pair on a coding problem. Like most live coding exercises, I didn't do very well. So, in an effort to redeem myself (in the eyes of myself), I studied up on the problem and worked through several solutions.

Hopefully, you don't find yourself in a similar situation. But, if you do, I hope reading through these solutions helps you fair better than I did!

Courses and prerequisites

Without further ado, the pairing exercise problem statement:

Given a set of courses and a corresponding set prerequisites, produce a valid ordering of courses such that the courses can be taken in that order without bypassing any of the prerequisites (there are multiple correct solutions).

And, using a Python dictionary, the input data:

COURSES = {
    "Algebra 1": [],
    "Algebra 2": ["Algebra 1"],
    "English 1": [],
    "English 2": ["English 1", "History 1"],
    "English 3": ["English 2"],
    "English 4": ["English 3"],
    "History 1": [],
    "History 2": ["History 1"],
    "Pre-Calculus": ["Algebra 2"],
    "Statistics 1": ["Algebra 1"],
    "Statistics 2": ["Statistics 1"],
}

Given the input above, a valid ordering of courses is:

['History 1',
 'History 2',
 'English 1',
 'English 2',
 'English 3',
 'English 4',
 'Algebra 1',
 'Statistics 1',
 'Statistics 2',
 'Algebra 2',
 'Pre-Calculus']

Now that we have the problem defined, let’s look at some solutions!

Sorting out the correct terminology

I had a sense that the solution to this problem involved modeling the data with a graph data structure, but I wasn’t sure what to do with it after that. So, I started looking for graph related libraries in Python, which led me to the most excellent NetworkX.

After navigating the NetworkX API documentation a bit, I noticed an entire section dedicated to algorithms. Under the algorithms section was a subsection specific to Directed Acyclic Graphs (DAGs). The reference to DAGs caught my eye because DAGs are often used to model data processing workflows with complex dependencies. In the problem statement above, the course prerequisites are a lot like data processing workflow dependencies.

Continuing through the DAG related algorithms, the description for topological_sort(G) stood out:

A topological sort is a nonunique permutation of the nodes such that an edge from u to v implies that u appears before v in the topological sort order.

That sounds promising! An edge can be produced by connecting a course v to a prerequisite u. If a topological sort can help ensure u appears before v in an ordering, then it aligns with our goal. Let’s give it a spin!

NetworkX to the rescue

Following the guidance in the topological_sort(G) description, I iterated over each combination of course and prerequisite and created an edge between them with the add_edge method:

>>> import networkx as nx
>>> graph = nx.DiGraph()
>>> for course, prerequisites in COURSES.items():
... for prerequisite in prerequisites:
... graph.add_edge(prerequisite, course)
... 
>>>

From there, the only thing left to do was to call topological_sort(G) with the DiGraph as an argument:

>>> pprint.pprint(list(nx.topological_sort(graph)))
['History 1',
 'History 2',
 'English 1',
 'English 2',
 'English 3',
 'English 4',
 'Algebra 1',
 'Statistics 1',
 'Statistics 2',
 'Algebra 2',
 'Pre-Calculus']
>>>

That ordering looks valid to me!

The standard library can do it too

After identifying the class of algorithm necessary to solve the problem (i.e., topological sort), I began to use the term in searches for other types of solutions. Eventually, that led me to a module in the Python standard library called graphlib.

According to the Python commit history, graphlib is pretty new (added in Python 3.9). It promises to provide a set of functionality for operating on graph-like structures. But, right now it only appears to have one class worth of functionality. Luckily for us, that one class is called TopologicalSorter!

Instantiating the class takes an argument, graph, which:

…must be a dictionary representing a directed acyclic graph where the keys are nodes and the values are iterables of all predecessors of that node in the graph (the nodes that have edges that point to the value in the key).

Hm. That sounds very similar to the COURSES data structure defined above. Let’s pass it through in the Python interpreter, and then call the static_order method:

>>> from graphlib import TopologicalSorter
>>> pprint.pprint(list(TopologicalSorter(COURSES).static_order()))
['Algebra 1',
 'English 1',
 'History 1',
 'Algebra 2',
 'Statistics 1',
 'English 2',
 'History 2',
 'Pre-Calculus',
 'Statistics 2',
 'English 3',
 'English 4']
>>>

Whoa! A different ordering from the one above, but still valid. We can even use another NetworkX function to confirm the TopologicalSorter solution is valid by checking it against all possible topological sorts, as reported by all_topological_sorts(G):

>>> solution = list(TopologicalSorter(COURSES).static_order())
>>> solution in nx.all_topological_sorts(graph)
True
>>>

Excellent—looks like we are two-for-two so far!

Show your work

Using algorithms built-in to libraries like NetworkX and graphlib is fun and all, but how would we solve this problem algorithmically? Well, according to Wikipedia, there are two existing algorithms to draw from:

I’m going to focus on Kahn’s algorithm—primarily because it doesn’t involve recursion. Although recursion is a fascinating technique, I don’t see it too often in day-to-day code, and I find that it creates confusion in most engineers (myself included).

Note: If the variable names below become confusing, please refer to the pseudocode in the Wikipedia link for Kahn's algorithm. I'm trying to match the variable names to that, so it is easier to follow along.

To start things off, let’s use the Python interpreter to define L and S. Here, L is being set up to contain the final course ordering and S made up of all the nodes in the graph with zero edges pointing to them (e.g., in_degree == 0):

>>> L = []
>>> S = [node for node in graph if not graph.in_degree(node)]
>>>

Next, we need to create a while loop set to run until S is empty. Inside, we pop n from S and immediately append it to L:

>>> while S:
... n = S.pop()
... L.append(n)

After that, we need to identify all nodes connected to n so that we can remove each edge from the graph, one-by-one. As they’re removed, we check to see if there are any remaining edges pointing to the node m. If not, append m to S.

... edges = list(graph.neighbors(n))
... for m in edges:
... graph.remove_edge(n, m)
... if not graph.in_degree(m):
... S.append(m)
>>>

After these steps are complete, L should contain a valid course ordering. Again, we can confirm with all_topological_sorts(G):

>>> L in nx.all_topological_sorts(graph)
True
>>>

That’s it! Now we have three different solutions to determine a valid ordering for a set of courses and prerequisites. Enjoy all the flavors of topologically sorted lemonade! 🍋

Twelve-Factor Methodology Applied to a Django App

Hector Castro — Tue, 16 Mar 2021 00:00:00 +0000

In the past few weeks, I’ve participated in a handful of DevOps/Site Reliability Engineer (SRE) interviews. Several interviewers have asked for guidelines configuring and operating cloud-native applications. My mind immediately goes to the Twelve-Factor App methodology, originally created by the folks who built Heroku—one of the first publicly accessible platforms as a service (PaaS).

Combined, the points serve to abstract applications from the infrastructure they run on, paving the way for configurability, scalability, and reliability. To illustrate how this works in practice, I set up a Django application and use it to explain how each 12 Factor point applies. I hope you find it useful!

Codebase
Dependencies
Config
Backing services
Build, release, run
Processes
Port binding
Concurrency
Disposability
Dev/prod parity
Logs
Admin processes

Note : The code snippets in the following sections do not chain together perfectly. The snippets are there primarily to help communicate what’s going on in ways that only code can.

Codebase

A codebase is the complete source material of a given software program or application. Its structure will vary based on technology, but for a Django application called mysite created with django-admin startproject, it looks like this:

$ git init
Initialized empty Git repository in /home/hector/Projects/django-blog/.git/
$ git add .
$ git status
On branch master

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file: .gitignore
        new file: Pipfile
        new file: Pipfile.lock
        new file: mysite/manage.py
        new file: mysite/mysite/ __init__.py
        new file: mysite/mysite/asgi.py
        new file: mysite/mysite/settings.py
        new file: mysite/mysite/urls.py
        new file: mysite/mysite/wsgi.py
        new file: setup.cfg

Excellent—we have ourselves a codebase! We’ll gradually cover converting codebases into deploys in the following sections.

Dependencies

Applications have dependencies. 12 Factor wants us to explicitly declare these dependencies so they can be managed in a repeatable way. The first step toward achieving this happens with a Pipfile. It was created by a Python dependency management tool called pipenv after the following commands were run:

pipenv install django~=3.1
pipenv install black --dev --pre # --pre is needed because of black's versioning scheme
pipenv install flake8~=3.8 --dev
pipenv install isort~=5.7 --dev

The inside of a Pipfile is written in Tom’s Obvious Minimal Language (TOML) and contains a manifest of the Python dependencies needed for a project:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
django = "~=3.1"

[dev-packages]
black = "*"
flake8 = "~=3.8"
isort = "~=5.7"

[requires]
python_version = "3.8"

[pipenv]
allow_prereleases = true

Nowadays, we try to take this a step further by capturing all the necessary application dependencies in a container image. In most cases, the pursuit of creating a container image leads to using Docker, which implies the addition of a Dockerfile:

FROM python:3.8

ENV PYTHONUNBUFFERED=1

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

COPY ./Pipfile* .
RUN pip install pipenv
RUN pipenv install --system --deploy --ignore-pipfile
COPY ./mysite .

ENTRYPOINT ["python", "manage.py"]

To make sure things are in working order, we can build and test the container image using the following commands. Here, the runserver argument launches the Django development server:

$ docker build -t mysite .
$ docker run --rm mysite runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
March 01, 2021 - 20:45:33
Django version 3.1.7, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Looks good! We now have everything needed to spin up the application captured in a container image. In addition, we have all the associated instructions to build the image defined in a declarative way (e.g., Pipfile, Dockerfile).

Config

In the Twelve-Factor world, configuration is defined as anything that can vary between deploys of a codebase. This allows a single codebase to be deployed into different environments without customization. Some examples of configuration include:

Connection strings to the database, Memcached, and other backing services.
Credentials to external services (e.g., Amazon S3, Google Maps, etc.).
Information about the target environment (e.g., Staging vs. Production).

Once we’ve identified the configuration for our application, we need to work toward making it consumable via environment variables. In the example below, we focus on changing the way Django’s SECRET_KEY and DEBUG settings are set in settings.py (the home for all Django configuration settings).

diff --git a/mysite/mysite/settings.py b/mysite/mysite/settings.py
index d541c62..3a99d45 100644
--- a/mysite/mysite/settings.py
+++ b/mysite/mysite/settings.py
@@ -9,7 +9,7 @@ https://docs.djangoproject.com/en/3.1/topics/settings/
 For the full list of settings and their values, see
 https://docs.djangoproject.com/en/3.1/ref/settings/
 """
-
+import os
 from pathlib import Path

 # Build paths inside the project like this: BASE_DIR / 'subdir'.
@@ -20,10 +20,10 @@ BASE_DIR = Path( __file__ ).resolve().parent.parent
 # See https://docs.djangoproject.com/en/3.1/howto/deployment/checklist/

 # SECURITY WARNING: keep the secret key used in production secret!
-SECRET_KEY = "#v5hnkypk39qex@9zb2j2as3n9f7)jgvz05*9t&0@2y$kx$7lw"
+SECRET_KEY = os.getenv("DJANGO_SECRET_KEY", "secret")

 # SECURITY WARNING: don't run with debug turned on in production!
-DEBUG = True
+DEBUG = os.getenv("DJANGO_ENV") == "Development"

 ALLOWED_HOSTS = []

Here, we made use of the Python standard library os module to help us read configuration from the environment. Now, the two settings can be more easily reconfigured across deploys.

To prove it works, we can change the environment with the -e flag of docker run:

$ docker build -t mysite .
$ docker run --rm \
    -e DJANGO_SECRET_KEY="dev-secret" \
    -e DJANGO_ENV="Development" \
    mysite runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
March 01, 2021 - 21:25:57
Django version 3.1.7, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
^C%

OK. Everything continued to work the way it was working before. Now, let’s see what happens if we try to make DJANGO_ENV=Production, which will cause the DEBUG setting to evaluate to False:

$ docker run --rm \
    -e DJANGO_SECRET_KEY="prod-secret" \
    -e DJANGO_ENV="Production" \
    mysite runserver
CommandError: You must set settings.ALLOWED_HOSTS if DEBUG is False.

Aha! This CommandError looks ominous, but it is an indicator that our change of DJANGO_ENV made its way into the application’s execution environment successfully!

Backing services

A backing service is any service the application consumes over the network as part of its normal operation. Emphasis is placed on minimizing the distinction between local and third-party backing services such that the application can’t tell the difference between them.

As an example, say you have a PostgreSQL database instance running on your workstation that’s connected to your application to persist data. Later, when it comes time to deploy to production, the same approach to configuring the local PostgreSQL instance should work when it gets swapped out for an Amazon Relational Database Service (RDS) instance.

To achieve this with Django, we need to change the way connectivity to the database is configured. That happens via the DATABASES dictionary in settings.py:

diff --git a/mysite/mysite/settings.py b/mysite/mysite/settings.py
index 3a99d45..fcff52a 100644
--- a/mysite/mysite/settings.py
+++ b/mysite/mysite/settings.py
@@ -75,8 +75,12 @@ WSGI_APPLICATION = "mysite.wsgi.application"

 DATABASES = {
     "default": {
- "ENGINE": "django.db.backends.sqlite3",
- "NAME": BASE_DIR / "db.sqlite3",
+ "ENGINE": "django.db.backends.postgresql",
+ "NAME": os.getenv("POSTGRES_DB"),
+ "USER": os.getenv("POSTGRES_USER"),
+ "PASSWORD": os.getenv("POSTGRES_PASSWORD"),
+ "HOST": os.getenv("POSTGRES_HOST"),
+ "PORT": os.getenv("POSTGRES_PORT"),
     }
 }

Here, we modified DATABASES so that all the necessary settings for the default database are pulled from the environment. Now, it doesn’t matter if the application is launched with HOST equal to localhost or mysite.123456789012.us-east-1.rds.amazonaws.com. In either case, the application should be able to connect to the database successfully using the settings found in the environment.

Build, release, run

In the Dependencies section we produced a build in the form of a container image. But, we also need a unique label to identify and differentiate between versions of the container image. Uniqueness can come in the form of a timestamp, or an incrementing number, but I personally like to use Git revisions. Below is an example that uses the current Git revision to tag a container image:

$ # Get a reference to the latest commit of the current
$ # branch and make it short (only 7 characters long).
$ export GIT_COMMIT="$(git rev-parse --short HEAD)"
$ docker build -t "mysite:$GIT_COMMIT" .
$ docker images | grep mysite
mysite e87b8c4 4f3dc2772c57 2 minutes ago 978MB

As you can see from the output, the reference mysite:e87b8c4 is unique to the container image we built. If we make additional changes to the codebase and commit them to the underlying Git repository, following these same steps will result in a new container image with a new unique reference.

Next, we need to combine the container image build above with a relevant set of configuration to produce a release. Here, we’ll use a lightweight Docker Compose configuration file to describe the connection between the two (builds and releases) in a declarative way. In a production system, you’d likely do something similar using a Kubernetes deployment or an Amazon ECS task definition:

version: "3"
services:
  web:
    image: mysite:e87b8c4
    environment:
      - POSTGRES_HOST=mysite.123456789012.us-east-1.rds.amazonaws.com
      - POSTGRES_PORT=5432
      - POSTGRES_USER=mysite
      - POSTGRES_PASSWORD=mysite
      - POSTGRES_DB=mysite
      - DJANGO_ENV=Staging
      - DJANGO_SECRET_KEY=staging-secret
    command:
      - runserver
      - "0.0.0.0:8000"
    ports:
      - "8000:8000"

This bit of Docker Compose configuration ties together the mysite:e87b8c4 build with a set of environment specific configuration to produce a release. If the container image and Docker Compose configuration snippet are available on the same host, then the application is ready for immediate execution on that host.

Lastly, we have the run stage. For Docker Compose, that’s as simple as using docker-compose up to launch the web service. For a more sophisticated container orchestration system, several more steps would likely be involved:

The container image is published to a centrally accessible container registry.
The deployment manifest is submitted for evaluation to a container scheduler.
Compute is connected to the container scheduler with adequate resources to place instances of the application.

Processes

The Twelve-Factor methodology emphasizes applications as stand-alone processes because when they share nothing, they can be made to more easily scale horizontally. Therefore, striving to store all dynamic state in a backing service (e.g., a database) to make a process stateless is important.

However, sometimes whole components of an application need to be dynamically built, like its associated CSS and JavaScript. To be truly stateless, we want to generate those components during the build phase and capture them in the container image.

Django has several built-in mechanisms to handle static assets, but I prefer to use a third-party library named WhiteNoise. Primarily, because it helps package both the application and its supporting static assets together in a way that enables thinking about a deploy as an atomic operation.

After installing WhiteNoise using pipenv with a command similar to the one we used in Dependencies to install Django, we need to configure the Django application to use WhiteNoise for static asset management. Here, we inject WhiteNoise into the Django INSTALLED_APPS and MIDDLEWARE hierarchy to take over static asset management in development and non-development environments:

diff --git a/mysite/mysite/settings.py b/mysite/mysite/settings.py
index 216452b..f4e32c6 100644
--- a/mysite/mysite/settings.py
+++ b/mysite/mysite/settings.py
@@ -31,6 +31,7 @@ ALLOWED_HOSTS = []
 # Application definition

 INSTALLED_APPS = [
+ "whitenoise.runserver_nostatic",
     "django.contrib.admin",
     "django.contrib.auth",
     "django.contrib.contenttypes",
@@ -41,6 +42,7 @@ INSTALLED_APPS = [

 MIDDLEWARE = [
     "django.middleware.security.SecurityMiddleware",
+ "whitenoise.middleware.WhiteNoiseMiddleware",
     "django.contrib.sessions.middleware.SessionMiddleware",
     "django.middleware.common.CommonMiddleware",
     "django.middleware.csrf.CsrfViewMiddleware",
@@ -122,3 +124,7 @@ USE_TZ = True
 # https://docs.djangoproject.com/en/3.1/howto/static-files/

 STATIC_URL = "/static/"
+
+STATIC_ROOT = "/static"
+
+STATICFILES_STORAGE = "whitenoise.storage.CompressedManifestStaticFilesStorage"

The two settings at the bottom (STATIC_ROOT and STATICFILES_STORAGE) tell Django where to store the collected files on the container image file system and what preprocessing operations to apply.

Next, we need to ensure that Django preprocesses all static assets as part of the container image build process. For Django, that means adding an invocation of the collectstatic command to the container image build instructions:

diff --git a/Dockerfile b/Dockerfile
index 4653278..6420680 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -10,4 +10,6 @@ RUN pip install pipenv
 RUN pipenv install --system --deploy --ignore-pipfile
 COPY ./mysite .

+RUN python manage.py collectstatic --no-input
+
 ENTRYPOINT ["python", "manage.py"]

Statelessness achieved!

Port binding

Now that we have the application source code, dependencies, and supporting static assets inside a container image, we need a way to expose the entirety of it in a self-contained way. Since this is a web application, our goal is to use the HTTP protocol instead of lower level APIs like CGI, FastCGI, Servlets, etc.

We’ve seen our application bound to a port over HTTP several times already via the docker run invocations above, but they were all using a development-grade HTTP application server (e.g., runserver). How do we achieve something similar in a production-grade way?

Enter Gunicorn and Uvicorn. Gunicorn is a production-grade Python application server for UNIX based systems, and Uvicorn provides a Gunicorn worker implementation with Asynchronous Server Gateway Interface (ASGI) compatibility.

After installing Gunicorn and Uvicorn using pipenv install, we need to tweak the Docker Compose configuration from Build, release, run to use Gunicorn as the entrypoint. We also add a few command-line options to ensure that the ASGI API is used (between Gunicorn and Django) along with the Uvicorn worker implementation:

diff --git a/docker-compose.yml b/docker-compose.yml
index f5f693d..bac885d 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -20,8 +20,12 @@ services:
     build:
       context: .
       dockerfile: Dockerfile
+ entrypoint: gunicorn
     command:
- - runserver
- - "0.0.0.0:8000"
+ - "mysite.asgi:application"
+ - "-b 0.0.0.0:8000"
+ - "-k uvicorn.workers.UvicornWorker"

After all of these changes, Docker Compose should be able to bring the service up bound to port 8000 using Gunicorn:

$ docker-compose up web
Starting django-blog_web_1 ... done
Attaching to django-blog_web_1
web_1 | [2021-03-06 19:57:43 +0000] [1] [INFO] Starting gunicorn 20.0.4
web_1 | [2021-03-06 19:57:43 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
web_1 | [2021-03-06 19:57:43 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
web_1 | [2021-03-06 19:57:43 +0000] [8] [INFO] Booting worker with pid: 8
web_1 | [2021-03-06 19:57:43 +0000] [8] [INFO] Started server process [8]
web_1 | [2021-03-06 19:57:43 +0000] [8] [INFO] Waiting for application startup.
web_1 | [2021-03-06 19:57:43 +0000] [8] [INFO] ASGI 'lifespan' protocol appears unsupported.
web_1 | [2021-03-06 19:57:43 +0000] [8] [INFO] Application startup complete.

We can confirm by creating a second terminal session, hitting the /admin/ endpoint, and inspecting the response:

$ http localhost:8000/admin/
HTTP/1.1 302 Found
cache-control: max-age=0, no-cache, no-store, must-revalidate, private
content-length: 0
content-type: text/html charset=utf-8
date: Sat, 06 Mar 2021 19:59:36 GMT
expires: Sat, 06 Mar 2021 19:59:36 GMT
location: /admin/login/?next=/admin/
referrer-policy: same-origin
server: uvicorn
vary: Cookie
x-content-type-options: nosniff
x-frame-options: DENY

It’s alive!

Concurrency

As load against an application increases, the ability to address it by quickly and reliably adding more stateless processes is desirable. Gunicorn has built-in support for a process level worker model, but using it to scale an application in cloud based environments can cause contention with higher level distributed process managers. This is because both want to manage the processes, but only the distributed process manager has a wholistic view of resources across machines. Instead, we can set the number of Gunicorn worker processes low and defer process management to a higher level supervisor.

Specifying different process types can’t really be done with Gunicorn either. Usually, that’s more tightly coupled with the container orchestration engine you use. Later on in Dev/prod parity we’ll see a Docker Compose configuration with both a database and web process type. Within a more production-oriented container orchestration system like Kubernetes, you’d achieve something similar by creating separate sets of pods—one for each process type to enable independent scaling.

Disposability

In cloud environments, application disposability is important because it increases agility during releases, scaling events, and failures. An application exhibits disposability when it properly handles certain types of asynchronous notifications called signals. Signals help local supervisory services (e.g., systemd and Kubelet) manage an application’s lifecycle externally.

Gunicorn has built-in support for signal handling. If you use it as your application server, it will automatically handle signals like SIGTERM to facilitate a graceful shutdown of the application.

Dev/prod parity

Configuration allows a single build of a codebase to run locally, in staging, and in production. Leveraging that to maintain parity across environments keeps incompatibilities from cropping up as software is being developed. This results in a higher degree confidence that the application will function the same way in production, as it did locally.

Still, maintaining development and production parity is an ongoing challenge. Much like speed and security, you have to be constantly thinking about it, or else you lose it.

Nowadays, operating system support for namespacing resources through containerization, along with higher level tooling like Docker and Docker Compose, go a long way toward making this pursuit easier to achieve. As an example, see the following Docker Compose configuration file:

version: "3"
services:
  database:
    image: postgres:12.6
    environment:
      - POSTGRES_USER=mysite
      - POSTGRES_PASSWORD=mysite
      - POSTGRES_DB=mysite

  web:
    image: mysite
    environment:
      - POSTGRES_HOST=database
      - POSTGRES_PORT=5432
      - POSTGRES_USER=mysite
      - POSTGRES_PASSWORD=mysite
      - POSTGRES_DB=mysite
      - DJANGO_ENV=Development
      - DJANGO_SECRET_KEY=secret
      - DJANGO_LOG_LEVEL=DEBUG
    build:
      context: .
      dockerfile: Dockerfile
    entrypoint: gunicorn
    command:
      - "mysite.asgi:application"
      - "-b 0.0.0.0:8000"
      - "-k uvicorn.workers.UvicornWorker"
    ports:
      - "8000:8000"

Within this relatively small file, we have defined all services needed to run our application locally. Each service (database and web) run as separate processes within their own containers, but are networked together. From the perspective of our Django application, this setup differs minimally from a true production container orchestration setup.

Logs

Logs emitted by an application provide visibility into its behavior. However, in cloud environments you cannot reliably predict where your application is going to run. This makes it difficult to get visibility into the application’s behavior—unless, you treat application logging as a stream. Treating application logs as a stream makes it easier for other services to aggregate and archive log output for centralized viewing.

Django uses Python’s built-in logging module to perform system logging, which allows it to be set up in some pretty sophisticated ways. However, all we want is for Django to log everything as a stream to standard out. We can make that happen by specifying a custom logging configuration dictionary in settings.py that looks like:

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
        },
    },
    "root": {
        "handlers": ["console"],
        "level": "WARNING",
    },
    "loggers": {
        "django": {
            "handlers": ["console"],
            "level": "WARNING",
            "propagate": False,
        },
    },
}

This configures the parent root logger to send messages with the WARNING level and higher to the console handler (e.g., standard out). It also has support to tune the default Django log levels via the DJANGO_LOG_LEVEL environment variable. A dynamic override like this can be extremely helpful when troubleshooting because it allows logging settings to be modified without requiring a new release.

Admin processes

Administrative tasks are essential to every application. It is important for the code associated them to ship with the application to avoid synchronization issues as they are invoked in the same execution environment as the application.

Most of Django’s supporting administrative tasks, like applying database migrations, sending test emails, and adding users, can already be executed as one-off processes. In addition, Django provides a robust framework for adding more that are specific to your application (e.g., toggling feature flags, orchestrating data imports, etc.).

As an example, we can apply outstanding database migrations (there should be some for a newly initialized Django project) with the built-in migrate command:

$ docker-compose run --rm --entrypoint "python manage.py" web migrate
Creating django-blog_web_run ... done
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying sessions.0001_initial... OK

Here, we dynamically override the previously referenced Docker Compose configuration with --entrypoint set to python manage.py instead of gunicorn. We also specify that we want the migrate subcommand to be run. This execution leads to a series of cross-container communications that ensure our database schema aligns with the current state of Django’s data model.

That’s it! Whether you were aware of the 12 Factor methodology before or not, I hope that seeing it applied to a Django application enables you to more easily integrate it with whatever web framework you use. May it lead to more configurable, scalable, and reliable applications. Amen.

Thanks to Dave Konopka for providing thoughtful feedback on my drafts of this post.

How I Make Slack Work for Me

Hector Castro — Sat, 13 Feb 2021 00:00:00 +0000

As a daily Slack user for the last seven years, I’ve spent a lot of time exploring ways to get the most out of it as a collaboration tool. While I have mixed feelings about its impact on productivity, I figure Slack isn’t going away any time soon, so I may as well learn how to make it work for me.

The sections below capture some Slack features and general tactics I’ve employed to make the most out of Slack as a tech lead and engineering leader in a software development focused organization. I hope you find some of them useful.

All unread entrypoint
Follow thread
Reminders on messages to track commitments
Strategic keyword notifications
Saved items as source for feedback
Quiet hours
Team CHANGELOG channel

1. All unread entrypoint

By default, the all unread feature of Slack is disabled. When enabled, it adds a new top level entry to the left-hand Slack navigation that allows you to browse all of your unread messages, grouped by channel, in a single view. While in this mode, you can scan messages, but also access all of the individual message shortcuts.

I use this feature as the entrypoint for digesting all Slack messages because it enables the move Denise Yu so eloquently summarized as, The Art of the Rollup.

enter what i've been mentally noting as "the art of the rollup". read the backscroll, take a deep breath, wait 5 mins, and write to entire channel:

"To summarize: the problem is X. Possible paths forward are A, B, C. Sounds like we're leaning towards A. have I missed anything?"

— Denise Yu (@deniseyu21) February 5, 2021

The ability to digest top level channel discussion as it develops, while still leaving it all marked an unread, allows me to bookend the context necessary to assemble an effective rollup.

2. Follow thread

The follow thread feature is easily the Slack feature I use most on this list. It allows you to subscribe to messages published in a thread without contributing any messages to the thread. I use it pretty liberally on any interesting message that pops up in a channel, then mark the channel as read via the All unread view referenced above.

It is worth cautioning that heavy use of this feature can quickly escalate into behavior that becomes indistinguishable from micromanagement—especially, if you use it to inject yourself into lots of conversations where people are trying to build their own problem solving skills.

3. Reminders on messages to track commitments

The Kahn Academy career development guide emphasizes a top level attribute called Maturity. An important indicator of maturity they cite is following through on your commitments (i.e., doing what you say you are going to do).

As a typical work day progresses, tons of micro commitments come up, and many of them occur in chat. Setting a reminder on a message provides an effective in-context way to track, snooze, and reschedule commitments so that they don’t get lost in the shuffle.

4. Strategic keyword notifications

Most folks are familiar with (and possibly loathe) Slack notifications. Most of the time, notifications happen when you get a direct message, when someone mentions you, or when someone mentions a group alias you’re a member of. But, Slack also provides a way to set up an open-ended list of keywords that trigger notifications.

In the past I’ve taken advantage of this feature to target certain keywords that have a tendency to lead to architecturally significant events:

bad idea
cache
lock
redis
should work
trivial

5. Saved items as source for feedback

It has been said that feedback is a gift. But, as with any great gift, feedback can be difficult to identify and deliver.

One way to make the feedback delivery process more effective is to connect it to specific events. For example, telling someone that they did a really good job at disambiguating a complex topic in a meeting last week. Or, that their testing instructions on a pull request from yesterday were detailed and easy to follow.

Neither of these examples are tied to chat, but many others are. To persist these events across different Slack channels, I repurpose the Slack save messages and files feature solely to track examples of both exemplary and poor communication. Any time I see a good candidate, I don’t have to think—I just click on the bookmark (used to be ⭐) icon. Later, I draw upon that list to support feedback in venues like one-on-ones, performance reviews, calls for kudos, etc.

6. Quiet hours

A couple of years back, I was exposed to the concept of Slack quiet hours by Nassim Kammah in a talk about remote-first team practices. Quiet hours are periods of blocked-off time, mutually agreed upon in advance by the team, when the team does not actively engage in Slack conversations. Colleagues are encouraged to save questions, requests, and conversations for outside of these periods.

Reserved blocks of time off of Slack aim to help enable deep work and mitigate the amount of context switching and FOMO that can occur as we bounce between completing tasks and keeping up with the never-ending Slack firehose.

7. Team `CHANGELOG` channel

Also sourced from Nassim’s talk above is the use of Reacji Channeler. Reacji Channeler is a Slack application that routes messages annotated with specific reactions to a designated channel. It can be configured in many ways to target a wide variety of use cases, but the use case described in the talk is particularly interesting: using it to produce a team CHANGELOG.

As significant events occur throughout a team’s day-to-day, someone summarizes (or rolls up) the event into one message that includes the surrounding context. When the appropriate reaction is applied to the message, it gets routed to a team CHANGELOG channel (e.g., #sre-team-changelog).

The goal is to produce a log such that if someone goes on vacation for a week, they can come back, read just that channel’s backscroll (not all of the actual team channel backscroll), and be caught up.

Special thanks to Terence Tuhinanshu for encouraging me to write this.

Creating Go Application Releases with GoReleaser

Hector Castro — Mon, 18 Jan 2021 00:00:00 +0000

A few weeks ago, I set out to upgrade the version of Go (1.6 to 1.15) used to build an old command-line utility I developed, named Heimdall. Heimdall provides a way to wrap an executable program inside of an exclusive lock provided by a central PostgreSQL instance via pg_try_advisory_lock.

Now, Heimdall is nice little utility and all (if you’re intrigued, check out the README), but the most interesting part of the upgrade process came after I got everything working and started to think about how to create a new release. That’s when I came across GoReleaser.

GoReleaser

GoReleaser is a release automation tool specifically for Go projects. With a few bits of YAML configuration, GoReleaser provided me with:

Hooks into the Go module system for managing library dependencies
The ability to easily produce a set of build artifacts for multiple operating systems and computer architectures
Checksums for each of the build artifacts
Easy integration with GitHub Actions to automate publishing releases on tagged commits

If you are responsible for Go applications that are in need of a uniform release process, I find it really hard to beat GoReleaser.

Validating Data in Python with Cerberus

Hector Castro — Tue, 29 Dec 2020 00:00:00 +0000

This year was my first participating in Advent of Code—and I’m glad I did, because solving one of the challenges exposed me to an excellent data validation library for Python named Cerberus.

What’s in a valid passport

Below are some excerpts from the challenge, along with specific field level validation rules:

You arrive at the airport only to realize that you grabbed your North Pole Credentials instead of your passport. While these documents are extremely similar, North Pole Credentials aren’t issued by a country and therefore aren’t actually valid documentation for travel in most of the world.

It seems like you’re not the only one having problems, though; a very long line has formed for the automatic passport scanners, and the delay could upset your travel itinerary.

…

The line is moving more quickly now, but you overhear airport security talking about how passports with invalid data are getting through. Better add some data validation, quick!

You can continue to ignore the cid field, but each other field has strict rules about what values are valid for automatic validation:

byr (Birth Year) - four digits; at least 1920 and at most 2002.

iyr (Issue Year) - four digits; at least 2010 and at most 2020.

eyr (Expiration Year) - four digits; at least 2020 and at most 2030.

hgt (Height) - a number followed by either cm or in:

If cm, the number must be at least 150 and at most 193.

If in, the number must be at least 59 and at most 76.

hcl (Hair Color) - a # followed by exactly six characters 0-9 or a-f.

ecl (Eye Color) - exactly one of: amb blu brn gry grn hzl oth.

pid (Passport ID) - a nine-digit number, including leading zeroes.

cid (Country ID) - ignored, missing or not.

Your job is to count the passports where all required fields are both present and valid according to the above rules.

For completeness, here are some invalid passports (delimited by \n\n):

eyr:1972 cid:100
hcl:#18171d ecl:amb hgt:170 pid:186cm iyr:2018 byr:1926

iyr:2019
hcl:#602927 eyr:1967 hgt:170cm
ecl:grn pid:012533040 byr:1946

hcl:dab227 iyr:2012
ecl:brn hgt:182cm pid:021572410 eyr:2020 byr:1992 cid:277

And, some valid passports:

pid:087499704 hgt:74in ecl:grn iyr:2012 eyr:2030 byr:1980
hcl:#623a2f

eyr:2029 ecl:blu cid:129 byr:1989
iyr:2014 pid:896056539 hcl:#a97842 hgt:165cm

hcl:#888785
hgt:164cm byr:2001 iyr:2015 cid:88
pid:545766238 ecl:hzl
eyr:2022

Most of the validation rules look straightforward in isolation, but less so when you think about composing them all together.

Validating passports with Cerberus

Step one involved getting familiar with Cerberus validation rules. The library supports rules like the following:

contains - This rule validates that the a container object contains all of the defined items.

>>> document = {"states": ["peace", "love", "inity"]}

>>> schema = {"states": {"contains": "peace"}}
>>> v.validate(document, schema)
True

regex - The validation will fail if the field’s value does not match the provided regular expression.

>>> schema = {
... "email": {
... "type": "string",
... "regex": "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$"
... }
... }
>>> document = {"email": "john@example.com"}
>>> v.validate(document, schema)
True

required - If True the field is mandatory. Validation will fail when it is missing.

>>> v.schema = {"name": {"required": True, "type": "string"}, "age": {"type": "integer"}}
>>> document = {"age": 10}
>>> v.validate(document)
False

Step two involved converting the passports into Cerberus documents. This was mostly an exercise in parsing uniquely assembled text into Python dictionaries.

# Split the batch file records by double newline.
for record in batch_file.read().split("\n\n"):
    # Split the fields within a record by a space or newline.
    record_field_list = [
        tuple(field.split(":")) for field in re.compile(r"\s").split(record.strip())
    ]

That leaves record_field_list looking like:

>>> record_field_list
[('ecl', 'gry'),
 ('pid', '860033327'),
 ('eyr', '2020'),
 ('hcl', '#fffffd'),
 ('byr', '1937'),
 ('iyr', '2017'),
 ('cid', '147'),
 ('hgt', '183cm')]

From there, dict converts the list of tuples into a proper Cerberus document:

>>> document = dict(record_field_list)
>>> document
{'byr': '1937',
 'cid': '147',
 'ecl': 'gry',
 'eyr': '2020',
 'hcl': '#fffffd',
 'hgt': '183cm',
 'iyr': '2017',
 'pid': '860033327'}

Putting it all together

Equipped with a better understanding of what’s possible with Cerberus, and a list of Python dictionaries representing passports, below is the schema I put together to enforce the passport validation rules of the challenge. Only one of the rules (hgt) required a custom function (compare_hgt_with_units).

SCHEMA = {
    "byr": {"min": "1920", "max": "2002"},
    "iyr": {"min": "2010", "max": "2020"},
    "eyr": {"min": "2020", "max": "2030"},
    "hgt": {
        "anyof": [
            {"allof": [{"regex": "[0-9]+cm"}, {"check_with": compare_hgt_with_units}]},
            {"allof": [{"regex": "[0-9]+in"}, {"check_with": compare_hgt_with_units}]},
        ]
    },
    "hcl": {"regex": "#[0-9a-f]{6}"},
    "ecl": {"allowed": ["amb", "blu", "brn", "gry", "grn", "hzl", "oth"]},
    "pid": {"regex": "[0-9]{9}"},
    "cid": {"required": False},
}

# Provide a custom field validation function for a height with units.
def compare_hgt_with_units(field: str, value: str, error: Callable[..., str]) -> None:
    if value.endswith("cm"):
        if not (150 <= int(value.rstrip("cm")) <= 193):
            error(field, "out of range")
    elif value.endswith("in"):
        if not (59 <= int(value.rstrip("in")) <= 76):
            error(field, "out of range")
    else:
        error(field, "missing units")

With a schema in place, all that’s left to do is instantiate a Validator and validate each document:

>>> v = Validator(SCHEMA, require_all=True)
>>> v.validate(document)
True

Thanks, Cerberus!

Centralized Scala Steward with GitHub Actions

Hector Castro — Wed, 18 Nov 2020 00:00:00 +0000

Keeping project dependencies up-to-date is a challenging problem. Services like GitHub’s automated dependency updating system, Dependabot, go a long way to help make things easier, but that is only helpful if your package manager’s ecosystem is supported. In the case of Scala based projects, it is not.

Enter Scala Steward.

Scala Steward provides a similar, low-effort way to keep project dependencies up-to-date. You simply open a pull request against the Scala Steward repository and add a reference to your project’s GitHub repository inside of a specially designated Markdown file. After that, Scala Steward (which manifests itself as a robot user on GitHub) keeps your project dependencies up-to-date via pull requests.

Unfortunately, this easy-mode option requires that your repository be publicly accessible. There are options for running Scala Steward as a service for yourself, but that path is less trodden and requires a bit more effort.

Scala Steward and GitHub Actions

So what other options do you have if your Scala project is inside a private repository? Well, if your project is on GitHub, then you likely have access to their workflow automation service, GitHub Actions. Scala Steward’s maintainers created a GitHub Action that lowers the bar to adding Scala Steward support to projects via the GitHub Actions execution model.

By default, the Action supports dependency detection through a workflow defined inside of your project’s repository. This approach makes it easy to simulate the public instance of Scala Steward on a per repository basis. But, there is also a centralized mode that allows you to mimic the way the centrally managed instance of Scala Steward works.

This centralized mode gives us an opportunity to have the best of both worlds: a low-effort way to keep multiple project dependencies up-to-date (similar to the public instance of Scala Steward), and the ability to do so across both public and private repositories!

Putting things together

First, create a GitHub repository for your instance of Scala Steward and put a file in it at .github/workflows/scala-steward.yml with the following contents:

name: Scala Steward

on:
  schedule:
    # Schedule to run every Sunday @ 12PM UTC. Replace this with
    # whatever seems appropriate to you.
    - cron: "0 0 * * 0"
  # Provide support for manually triggering the workflow via GitHub.
  workflow_dispatch:

jobs:
  scala-steward:
    name: scala-steward
    runs-on: ubuntu-latest
    steps:
      # This is necessary to ensure that the most up-to-date version of
      # REPOSITORIES.md is used.
      - uses: actions/checkout@v2

      - name: Execute Scala Steward
        uses: scala-steward-org/scala-steward-action@vX.Y.Z
        with:
          # A GitHub personal access token tied to a user that will create
          # pull requests against your projects to update dependencies. More
          # on this under the YAML snippet.
          github-token: ${{ secrets.SCALA_STEWARD_GITHUB_TOKEN }}
          # A Markdown file with a literal Markdown list of repositories
          # Scala Steward should monitor.
          repos-file: REPOSITORIES.md
          author-email: scala-steward@users.noreply.github.com
          author-name: Scala Steward

Hopefully, the inline comments help minimize any ambiguity in the GitHub Actions workflow configuration file. For completeness, below is an example of the Markdown file as well:

- organization/repository1
- organization/repository2
- organization/repository3

The last step is to ensure that any private repositories add the user associated with the GitHub personal access token as a collaborator with the Write role permissions. Also, to slightly improve usability and maintainability, consider the following suggestions:

Add Dependabot support to your Scala Steward repository to keep the Scala Steward GitHub Action up-to-date.
Avoid tying Scala Steward to an individual user GitHub account. Consider creating a bot account first, then create a personal access token with it to use with Scala Steward.
Create a custom Scala Steward team (e.g., @organization/scala-steward ) and add the bot account above to it. Now, instead of remembering to add the bot account to your Scala project repository as a collaborator, you can add the more intuitive Scala Steward team.

Haskell Code Katas: Counting Duplicates

Hector Castro — Sun, 17 Dec 2017 00:00:00 +0000

For the past few weeks, I’ve been starting off my days with Haskell flavored code katas from Codewars. Today I started with the kata below and figured it would be a good exercise to walk through my solution.

Write a function that will return the count of distinct case-insensitive alphabetic characters and numeric digits that occur more than once in the input string. The input string can be assumed to contain only alphabets (both uppercase and lowercase) and numeric digits.

To help clarify the specifications for this kata, the Hspec test suite is below:

module Codwars.Kata.Duplicates.Test where

import Codwars.Kata.Duplicates (duplicateCount)
import Data.List (nub)
import Test.Hspec
import Test.QuickCheck

main = hspec $ do
  describe "duplicateCount" $ do
    it "should work for some small tests" $ do
      duplicateCount "" =?= 0
      duplicateCount "abcde" =?= 0
      duplicateCount "aabbcde" =?= 2
      duplicateCount "aaBbcde" =?= 2
      duplicateCount "Indivisibility" =?= 1
      duplicateCount "Indivisibilities" =?= 2
      duplicateCount ['a'..'z'] =?= 0
      duplicateCount (['a'..'z'] ++ ['A'..'Z']) =?= 26
    it "should work for some random lists" $ do
      property $ forAll (listOf $ elements ['a'..'z']) $ \x ->
        let xs = nub x
        in duplicateCount (concatMap (replicate 2) xs) =?= length xs
  where (=?=) = shouldBe

Sorting & Grouping

To start things off, we are given the following snippet:

module Codwars.Kata.Duplicates where

duplicateCount :: String -> Int
duplicateCount = undefined

My first step is to figure out how to deal with case-insensitivity. Within Data.Char is toLower, which can be used to map over each character in the input String.

Prelude> x = "aaBbcde"
Prelude> x
"aaBbcde"
Prelude> import Data.Char
Prelude Data.Char> :t toLower
toLower :: Char -> Char
Prelude Data.Char> map toLower x
"aabbcde"

Next, I want to group like characters together. To do that, I need to sort and then group the characters together.

Prelude Data.Char> import Data.List
Prelude Data.Char Data.List> :t sort
sort :: Ord a => [a] -> [a]
Prelude Data.Char Data.List> sort . map toLower $ x
"aabbcde"

The sort doesn’t do very much in this case because the input string was already sorted. Either way, now we can work on grouping like characters with group:

Prelude Data.Char Data.List> :t group
group :: Eq a => [a] -> [[a]]
Prelude Data.Char Data.List> group . sort . map toLower $ x
["aa","bb","c","d","e"]

Home Stretch

Now, how do we go from a list of [Char] to an Int length that can be used for filtering characters that only occur once? filter, with a >1 condition applied to the length, should get us there.

Prelude Data.Char Data.List> z = group . sort . map toLower $ x
Prelude Data.Char Data.List> filter ((>1) . length) z
["aa","bb"]

Here, the . allows us to compose length and >1 together so that both can be applied to the [Char] provided to filter. The result rids the list of any characters that only occur once in the original input.

Lastly, we need the count of distinct characters from the input String that occur more than one, which is as simple as getting the length of the filtered list.

Prelude Data.Char Data.List> f = filter ((>1) . length) z
Prelude Data.Char Data.List> length f
2

Putting it all together, and breaking out some of the pipelined functions into a variable in the where clause, we get the duplicateCount function below.

module Codwars.Kata.Duplicates where

import Data.List (group, sort)
import Data.Char (toLower)

duplicateCount :: String -> Int
duplicateCount = length . filter ((>1) . length) . grouped
  where
    grouped = group . sort . map toLower

DEV Community: Hector Castro

Parsing Data in Rust with Nom

Beacon exclusion zone

Parsing with Nom

Conclusion

Every Other Friday Off Work Schedule

Relief in knowing others are off too

More opportunity for decompression

More quality time with my kid

Conclusion

Turning Lemons into Topologically Sorted Lemonade

Courses and prerequisites

Sorting out the correct terminology

NetworkX to the rescue

The standard library can do it too

Show your work

Twelve-Factor Methodology Applied to a Django App

Codebase

Dependencies

Config

Backing services

Build, release, run

Processes

Port binding

Concurrency

Disposability

Dev/prod parity

Logs

Admin processes

How I Make Slack Work for Me

1. All unread entrypoint

2. Follow thread

3. Reminders on messages to track commitments

4. Strategic keyword notifications

5. Saved items as source for feedback

6. Quiet hours

7. Team CHANGELOG channel

Creating Go Application Releases with GoReleaser

GoReleaser

Validating Data in Python with Cerberus

What’s in a valid passport

Validating passports with Cerberus

Putting it all together

Centralized Scala Steward with GitHub Actions

Scala Steward and GitHub Actions

Putting things together

Haskell Code Katas: Counting Duplicates

Sorting & Grouping

Home Stretch

7. Team `CHANGELOG` channel