DEV Community: Craig Franklin

A Review of "Professor Fisby's Mostly Adequate Guide to Functional Programming"

Craig Franklin — Tue, 05 Oct 2021 03:47:49 +0000

After years of mostly working with object-oriented languages such as Ruby and Python and, as a result, focusing on learning the best practices of object-oriented code design, I've recently changed jobs to a company whose applications are mostly written in TypeScript. What's more, their in-house style avoids classes entirely, preferring a more functional approach to organising the codebase. Although, the principals of good code design are applicable across languages, I felt a little unsure of myself trying to write code in this unfamiliar environment. Therefore, I decided to read up a bit on Functional Programming to learn the specific techniques and patterns of FP that I could use to achieve the nebulous goal of "clean code". Unfortunately, just as many of the popular OOP books use languages that I can't be bothered to learn, like Java and C++, many of the top FP books use functional languages, like Haskell and Scala, that I don't anticipate working with anytime soon. In both of these cases, I have nothing against these languages; it's just that I'm a practical guy, and if I'm going to put the time and effort into learning programming concepts or techniques, I want to be able to use them. Otherwise, I'll just forget them, and if I'm going to read something for personal enrichment, I'd rather read a good novel than pour over pages upon pages of code in a language I can only half-understand. Thankfully, there are FP books whose authors have chosen to meet the majority of programmers where they're at and use JavaScript for their code examples. Professor Fisby's Mostly Adequate Guide to Functional Programming by Brian Lonsdorf is one such book. Given that it was one of the top results in my searches, and the comments and reviews that I found were generally positive, I decided to read it in the hopes of getting a better handle on how to write good functional code, so that I might contribute to my new job's functional TypeScript codebase with more confidence.

At 146 pages (according to GoodReads), Professor Fisby's Mostly Adequate Guide to Functional Programming (MAG from now one for brevity's sake) is a fair bit shorter than a lot of programming books that I have read. I see this as a strength, because I often find such books to be a bit bloated with extended code examples and in-depth explanations of said code. Sometimes it's necessary, but often it drags on way too long and probably could have used a hard-headed editor forcing the author(s) to get to the point already. For people looking for a deeper exploration of FP, with more examples to really clarify some of the more-complex mathematical concepts, I can see how this might be viewed as a weakness. I, however, was looking for a quick introduction that would get me writing better functional TS code in short order, so erring on the side of brevity, both in the examples and the explanations of underlying theory, worked well for me. Another overall strength of the book is Lonsdorf's jokey writing style. Admittedly, the jokes are as likely to elicit a roll of the eyes as a chuckle, but I respect him for trying to keep what can be a very dry topic light and amusing. Yet another reason that programming books often drag at some point (at least for me) is the authors are so concerned with conveying information that they neglect to make their writing engaging, perhaps believing that the content is engaging enough on its own. Now, I'm not expecting Lord of the Rings when learning about how to refactor for-loops, but having a writer with a sense of their own voice, as opposed to an aggressively-neutral presentation of information, makes a big difference in how likely I am to stick with a technical book till the end. One last thing to have in mind about MAG is that, per its own "plans for the future", it is unfinished. The book is broken into three sections, with the first being a practical introduction to FP syntax and basic concepts, the second going deeper into the theory and using more-abstract structures in the code, and a planned third section that will "dance the fine line between practical programming and academic absurdity", but which was never added. Given my practical goals for learning from this book, and my reaction to the moderately theoretical second section (more on that below), I don't view this as a serious omission. MAG does a good job of introducing the techniques and theory of FP, and I imagine if someone wants to really get into the weeds, they'd probably be better off picking up a book that uses one of the pure FP languages anyway.

The first section of MAG, covering seven chapters, serves as an introduction to why FP is useful in codebases and the sort of low-level syntax and structures required to make it possible. Though I was familiar with the concept of pure functions, Lonsdorf's statement that "The philosophy of functional programming postulates that side effects are a primary cause of incorrect behaviour" struck me as an excellent distillation of the benefits of pursuing FP as the organising paradigm of a codebase. Flaky tests, conflicting component states in React, old invalid records just sitting in the database, all of these are common examples of problems caused by statefulness in software, which we manage via side effects. As I'm sure many of you know, a bug that you can't reproduce consistently is one of the most difficult to fix, and it's usually a specific and highly unlikely combination of states that make it so difficult to reproduce. For example, I remember trying to figure out a bug while working at an ecommerce company, where all the products in a user's cart were available and ready to be purchased when they began the checkout process, but when they tried to pay, the products were no longer available, and we raised an error. After days of pouring over logs looking for clues and trying to recreate the bug anyway I could think of, I finally figured out that the user had opened a second browser tab during checkout and made some changes to their cart before proceeding with payment in the original tab. The cart's state had changed in one part of our system, but that change hadn't been propogated to all parts of the system. Now, some statefulness in an application is probably unavoidable, or at least avoiding it would be horribly impractical, but minimisation of dependence on that statefulness greatly simplifies code, because it reduces how much you have to keep track of when writing it. This limits your attention to two things: input and output. Side effects, on the other hand, are theoretically infinite, there's no limit to the number of database, API, or logging calls you can make in a given function. Therefore, regardless of what language I'm working in, something that I like to keep in mind is that you can use pure functions or methods anywhere, even in largely-OOP codebases. Python and Ruby (and JavaScript for that matter) often offer two variations of a function or method: one that takes an object and changes it, and another that returns a new object (list.sort() vs sorted(list) in Python for example). I think that this is one of the most useful lessons from learning about different programming languages or paradigms: you can take the useful pieces from each, mixing and matching them in the code that you write in order to derive some of the benefits of each while mitigating some of the costs.

Now, if one of the great costs of OOP is the pervasiveness of state, what then is the cost of applying FP, which largely avoids state? In my opinion, that would be how abstract and mathematical FP gets once you start wandering down the rabbit hole. I found Lonsdorf's introductions to currying, function composition, and pointfree style to be useful. These are techniques and syntatical styles that I can use in my own code, I thought. Starting around chapter 7, however, Lonsdorf begins to focus a bit more on some of the theoretical underpinnings of FP in order to introduce higher-level structures that enable the kind of mathematical correctness that adherents of FP promise. At this point, I found myself doing a lot more skimming than I had previously, nodding at the explanations for how functors work and why they are useful, content in getting the gist of it, and not even bothering with the exercises at the ends of the later chapters. The reason for my disengagement is that I didn't really see myself ever applying these more-advanced techniques or using these more-complex structures in my code. Writing pure functions and composing them with maps, filters, or pipe operators is something you can do in almost any codebase, and the code will likely be easier to read, understand, and change because of it. But functors, applicative or otherwise, well, that's pretty much an all-or-nothing proposition. Maybe I suffer from a limited imagination, but I don't see how one could write code in that style in a piecemeal fashion. So, for me, the second half of MAG was pure theory, even when it was explaining the practical application of concepts from set theory. When it comes to code, I'm not particularly interested in theory. I can understand, however, why some coders get inspired by FP and can get so adamant about its benefits. Codebases are messy affairs, containing a few languages, each written in at least a half dozen styles, all based on the momentary preferences of dozens (hundreds? thousands?), of coders that have contributed over the years, and around every corner is a bug, just lying in wait for the right combination of parameters to raise that unanticipated error. So, the idea that your code could have the elegance and provability of a mathematical theorem is a powerful one. It just seems too impractical to me, as we can't very well expect each new developer who joins our team to spend their first few months reading textbooks on set theory and Functional Programming before they can make their first commit.

One thing that sometimes bothers me about proponents of Agile, OOP, TDD, etc. is that their rhetoric can wander into No True Scotsman territory: it's not that these techniques or processes or principles are flawed or fail to deliver their promised benefits; people are just doing them wrong. I believe that OOP, done exceptionally well, can provide the kind of readability and maintainability promised by its experts, but it's really hard to write OOP code at that level. How many coders can claim to be masters with a straight face (or with those around them maintaining a similarly-straight face)? On the other hand, even poorly-written OOP code has some basic organising principles that aids future devs in trying to understand and modify it. You have classes that represent business concepts (sometimes more abstract or technical concepts), and those objects have behaviours represented by their methods. This makes the learning curve much more manageable, because early practitioners have some basic, concrete ideas and techniques that they can apply while they're still learning the methods for writing truly clean code. My impression of FP is that it's like the classic bit about learning to draw an owl: make your functions pure, compose them, and now here's a bunch of set theory to explain why implementing an entire system of functor containers for any data that might pass through your system is totally worth it. The leap from a few basic design principles to abstract structures, without any real-world analog, is large, and I imagine that I'm not the only one who finds that juice to not be worth the squeeze. It feels like it would be easier to write bug-free (or at least lightly-bugged) code if one followed FP to the letter, but sometimes mediocre code is enough to get the job done and move on with your life, and it seems quite difficult to just write mediocre FP code.

I've already begun using pointfree style and some light function composition in my code, and being introduced to the JS package ramda really went a long way toward easing me into a more functional style of coding. I also found that the explanations of functors gave me a better appreciation for what languages like Rust do to avoid unhandled errors and nulls. However, at least for now, I think that's the extent of the impact of Professor Fisby's Mostly Adequate Guide to Functional Programming on how I read and write code. Even though I remain unconverted to the full FP path, I feel like I learned some useful concepts and techniques and would definitely recommend this book to anyone who is FP-curious but unwilling to commit to a 400-page tome with code examples written in Haskell.

How to Set up FaunaDB for local development

Craig Franklin — Sun, 06 Dec 2020 05:01:27 +0000

If you're feeling impatient and want to skip to the end, all the code is available at this repo.

What is FaunaDB, and why should I try it?

FaunaDB is a serverless database that is an ideal choice for serverless applications, because it has the same benefits as the latter: auto-scaling, pay-for-what-you-use billing, not requiring server configuration or maintenance. FaunaDB accomplishes this by executing database operations via calls to its HTTP API rather than maintaining a connection to a database server. There are other options available for serverless database services. DynamoDB from AWS is one of the most commonly referenced when looking up information about serverless databases, but I find a basic key-value data store to be too limiting when trying to model an application's business domain, as it makes modelling relationships among entities very difficult. Relational databases are ideal for this, and AWS has another serverless option, Aurora Serverless, which combines the advantages of serverless architecture with the expressiveness of relational databases and SQL queries. Although I haven't used it myself, Aurora Serverless seems to be a good option. It comes with a large caveat, however, in that Aurora Serverless requires all applications and services that access its databases to be in the same Amazon Virtual Private Cloud. This means committing yourself even further to vendor lock-in, as any move away from using AWS, even partially, would mean having to completely change your database infrastructure. Also, my knowledge of ops stuff is pretty basic, and setting up an Amazon VPC is just the sort of extra complication that I wanted to avoid by going serverless in the first place. FaunaDB isn't strictly a relational database, but it still offers some of the same advantages by allowing for the inclusion of all the usual entity relationships (e.g. one-to-one, one-to-many). Also, FaunaDB databases can be called from any application hosted on any cloud service, giving you more flexibility in how and where you deploy applications, thus reducing vendor lock-in.

FaunaDB supports two different ways of querying its databases: FaunaDB Query Language (FQL) and GraphQL (GQL). For now, the GQL interface is somewhat limited, making FQL a better option if you need the sort of functionality that you can get from SQL. If you're not sure, or if you anticipate only needing basic queries, I recommend starting with the GQL interface, because it's much simpler to work with (if you already know GQL, then there's not much more to learn), and it offers the benefit of having a schema file as the source of truth about the structure of your database. If you start with GQL and find out later that you need more-complex queries, you can mix in FQL functions by defining custom resolvers, so you shouldn't ever need to completely migrate to FQL. Since I prefer the GQL interface (and it's all I've worked with so far), that's what I'll use for this tutorial.

If you want more information on how FaunaDB works in general, feel free to check out their Getting Started guide, but it's not necessary for following the rest of this tutorial.

Requirements

To start, make sure you have the following installed on your machine:

Docker, so we can run an instance of FaunaDB in a container.
fauna-shell to be able to interact with local FaunaDB databases.
An API testing tool (e.g. Postman, Insomnia) to easily send GraphQL queries (or you can use curl if you really want to). For the examples below, I will use Insomnia, because its GraphQL support is a bit better than Postman's (for example, it can fetch a GraphQL schema from an API endpoint).
Python if you want to set up integration testing.

Setting up local FaunaDB

Running FaunaDB in a Docker container

FaunaDB is kind enough to provide us with an official Docker image, which greatly simplifies running it on our local environment. Run the following commands in your terminal to get an instance of FaunaDB running:

docker pull fauna/faunadb
docker run --name faunadb -p 8443:8443 -p 8084:8084 fauna/faunadb

We expose port 8443 for standard database interactions and 8084 for calls to the GraphQL API. As with other database images, you can use other Docker container options for data persistence and such. See the FaunaDB documentation for alternatives.

Create a database in the FaunaDB instance

Now that we have FaunaDB up and running, the easiest way to interact with it is to use fauna-shell. In a new tab or window, run the following to create a database and an API key for it.

fauna add-endpoint http://localhost:8443/ --alias localhost --key secret
fauna create-database development_db --endpoint=localhost
fauna create-key development_db --endpoint=localhost

You can change the alias for the endpoint (currently localhost) and the name of the database (currently development_db) to anything you want. If one doesn't already exist, create a .env file in the root of the project, copy the API key printed by fauna create-key, and assign its value to FAUNADB_KEY in .env like below:

FAUNADB_KEY=<copied API key>

Using GraphQL with FaunaDB

Create a GraphQL schema

A full introduction to GraphQL is outside the scope of this tutorial, so if you want more information, you can check out the official introduction. Also, if you want to get a better view of the GraphQL queries that FaunaDB creates by default for entities, check out their getting started page.

To keep things simple, we'll create a basic schema file for a blogging platform, where we have users who write posts. Such a schema for FaunaDB looks like this:

type User {
  username: String! @unique
  password: String!
  posts: [Post!] @relation
}

type Post {
  title: String!
  text: String!
  author: User!
}

type Query {
  allUsers: [User!]
  allPosts: [Post!]
}

This schema creates collections (FaunaDB's version of data tables) of users and posts, with a couple general-purpose query fields. It doesn't matter where you save this file, but it's best to name it following GraphQL conventions to take advantage of tools such as linters (I use schema.gql, but there are a few other acceptable alternatives). Most of this is standard GraphQL schema syntax, but notice the FaunaDB directives that start with @. These give extra information to FaunaDB about the structure of collections and documents.

@unique indicates that the value for this attribute must be unique within the collection. FaunaDB will return an error response if we try to create a user with a duplicate username.
@relation indicates a bi-directional relationship between two collections. In this case, a user has many posts, and a post has one user, which we call its author. Notice that we only need to use the relation directive on the "many" side of the relationship. FaunaDB has more information on how to model different relationships among collections.

These are the directives that I use most, but there are others for more-advanced use cases. You can see all the available directives here.

Also, by default, FaunaDB creates a few basic queries and mutations for each collection that you create (e.g. createUser, updateUser, findUserById), but doesn't add any queries for multiple records, so we add allUsers and allPosts queries for convenience.

Import the GraphQL schema to the FaunaDB database

From now on, we're going to interact with the FaunaDB database via HTTP calls to its GraphQL API, which should be running on http://localhost:8084. The simplest way to do this is with an API testing tool like Insomnia, but for an application, you'll obviously want to automate these calls with an HTTP library like requests for Python or axios for Node.

Our first call will be to import our GraphQL schema. Open Insomnia, and create a request that will POST to http://localhost:8084/import. Remember that secret key that we saved? Use it in the Authorization header with the format Bearer <FaunaDB secret key>. Finally, let's add our schema file to the request (in the case of Insomnia, select "Binary File" from the body options, then "Choose File" to upload the schema).

Create and query records in the database

Since the FaunaDB instance has a standard GraphQL API layer accessed at http://localhost:8084/graphql, you can use GraphQL tooling to export the schema and documentation, and load it in an interactive tool like GraphiQL or GraphQL Playground. The easiest solution, however, is to use Insomnia's GraphQL documentation feature by selecting "GraphQL Query" for the request body, then clicking "schema" and selecting "Refresh Schema". This will give you access to auto-generated documentation for your GraphQL API without needing any extra setup.

To start, we will want to create some data that we will then be able to query. The following mutations will create two users with two posts each. We return the IDs in the response for future reference, or you can use the allUsers query to get them again later.

mutation createBob {
  createUser(data: {
    username: "burgerbob",
    password: "password1234",
    posts: {
      create: [
        {
          title: "Burgers Are Great!",
          text: "Burgers have meat, and bread, and..."
        },
        {
          title: "Top 10 Burgers",
          text: "1. Cheeseburger, 2. Hamburger..."
        },
      ]
    }
  }) {
    _id
    posts {
      data { _id }
    }
  }
}

mutation createLinda {
  createUser(data: {
    username: "momsense",
    password: "password1234",
    posts: {
      create: [
        {
          title: "Why Baked Ziti Is Overrated",
          text: "Baked ziti is really not that great..."
        },
        {
          title: "Top 10 Wines to Pair with Burgers",
          text: "1. Pinot Noir, 2. Merlot..."
        },
      ]
    }
  }) {
    _id
    posts {
      data { _id }
    }
  }
}

Now that we have some records in the database, we can query them with the built-in find<Collection>ByID query or fetch all of a collection's records with one of the all<Collection> queries that we included in the schema. If you're using Insomnia, explore the schema documentation and try out different queries.

Bonus: Using FaunaDB while testing a Python application (with Pytest)

A common challenge when testing application code is handling database transactions in integration tests. Thankfully, we have frameworks like Rails and Django that, with a little configuration, can handle database setup and teardown for us. Unfortunately, such frameworks' database connectors generally have limited support for NoSQL databases like FaunaDB, and even then, it's only for the most-popular options like MongoDB. So, how can we use our local FaunaDB instance for integration tests without corrupting our development database?

When setting up integration tests, I wanted to start by making sure that I didn't accidentally change data in my development database. Since FaunaDB identifies the specific database that you're calling with the API token that you use, the easiest way to avoid accidental calls is to make sure that the token for your development database isn't available in the code. Assuming that you're using environment variables for your tokens and secrets, the easiest way to hide the dev API token is to make it blank in os.environ before every test. Pytest allows us to automatically run code before tests with conftest.py files. What's more, these files can be nested to run different pre-test code in different test modules. So, we can define a conftest.py file at the root of our tests directory with the following code to prevent unwanted FaunaDB calls:

# src/tests/conftest.py
import os

os.environ["FAUNADB_KEY"] = ""

I tried doing the same thing using a Pytest fixture or unittest's patch, which can work as well, but require you to use them before every test or at least before every test suite, and such details are easy to forget when writing tests for the next feature. The code above, however, is guaranteed to run before any tests, making it a foolproof way of keeping our development data safe.

Now that we've prevented unwanted calls to our dev database, how do we set up and call our test database for integration tests? Why, with another conftest.py of course! I have tests separated into unit and integration submodules. Inside integration we can include the following conftest.py:

# src/tests/integration/conftest.py

import os
from unittest.mock import patch

import pytest

# Names of module & FaunaDB client class depend
# on your application code
from src.app.faunadb import FaunadbClient


# Use "session" scope and autouse to run once before all tests.
# This is to make sure that the "localhost" endpoint exists.
@pytest.fixture(scope="session", autouse=True)
def _setup_faunadb():
    os.system(
        "npx fauna add-endpoint http://faunadb:8443/ --alias localhost --key secret"
    )


# Scope "function" means that this only applies to the test function
# that uses it
@pytest.fixture(scope="function")
def faunadb_client():
    # We create and delete the database for each test,
    # because it's reasonably quick, and simpler than manually deleting
    # all data.
    os.system("npx fauna create-database test --endpoint localhost")

    # Creating an API key produces output in the terminal
    # that includes the following line: secret: <API token>
    create_key_output = (
      os.popen("npx fauna create-key test --endpoint=localhost").read()
    )
    faunadb_key = (
        re.search("secret: (.+)", create_key_output).group(1).strip()
    )

    client = FaunadbClient(faunadb_key=faunadb_key)
    client.import_schema()

    # For any test that uses this fixture, we patch the environment
    # variable for the FaunaDB API key and return the client. This way,
    # all FaunaDB calls will use the test DB, and the test function
    # will have a valid client to make DB calls for test setup
    # or assertions.
    with patch.dict(
      "os.environ", {**os.environ, 'FAUNADB_KEY': faunadb_key}
    ):
        yield client

    os.system("npx fauna delete-database test --endpoint localhost")

You might need to modify the code above depending on the specifics of your app and tests (for example, I patch my app's settings module rather than os.environ directly), but I've been using a similar file for a few weeks, and it's been working well. Now, if you wanted to test user creation with actual database transactions, you could run something like the following:

# src/tests/integration/test_user.py

def test_user_creation(faunadb_client):
  username = "burgerbob"
  created_user = faunadb_client.create_user(
    username=username,
    password="password1234"
  )
  assert created_user['username'] == username

  all_users = faunadb_client.all_users()
  assert len(all_users) == 1

If you want to see the full test and FaunaDB client code, I have an example in the repo for this tutorial.

Conclusion

There you have it: a full FaunaDB setup for local development and testing (as long as you're using Python). Thanks to its GraphQL interface, service-agnostic architecture, and its ability to model complex data relationships, I think FaunaDB is a solid choice for data persistence for any serverless application. The documentation for setting up a local instance of FaunaDB, however, is a bit sparse. So, I had to figure it out by piecing together disparate posts and code samples, but, as you can see, it's not too complicated once you know the necessary commands and configuration. So, give FaunaDB a try for your next serverless project.

Footy Tipping with Machine Learning: 2019 Season Review

Craig Franklin — Wed, 06 Nov 2019 09:37:20 +0000

I've recently finished up my second season of ML footy tipping (if you're interested, you can check out an earlier post for more context on what the hell this means), and, though I did not repeat my path to office-tipping-comp-glory (no prize money, no trash talking, at least not for me), the season was not without personal highlights. Unfortunately, they were more of the lessons-learned, character-building variety rather than the crushing-of-foes, hearing-the-lamentations-of-their-women variety. So, what have I learned over the past year to have made all the heartache of defeat worthwhile? Well, I definitely improved my machine-learning workflow and coding practices, both of which were major pain points the year before, but I also made plenty of new mistakes that will inform my goals for next year.

Got to admit: it's getting better

Last year, I wrote a post (same one linked to above) laying out the mistakes I had made when creating my first footy-tipping model and how I hoped to improve on my processes and implementations while building a new model from scratch. I ended up with the following goals:

Do more exploratory data analysis
Don't jump straight to fancy, complicated models
Put some effort into code design to make extending it easier
Come up with a better name for the model

1. Do some EDA

I had skimped on exploratory data analysis (i.e. I hadn't done any) while developing my model for 2018, relying on a brute-force method of shoveling a bunch of data on top of an algorithm till it looked right. I avoided this in the run-up to the 2019 season, looking into basic heuristics for predicting match results (e.g. always pick the home team, always pick the favourite), as well as analysing trends and the data (or lack thereof) behind momentum. One can always do more and better analysis, but I gained additional insight into Aussie Rules Football, how it's played, and what stats matter, which informed my feature building and model tuning later in the process.

2. Compare different models

Being a machine-learning novice, I had bought into the hype and started with deep learning and ensemble models without even looking at simpler linear models. I did not repeat this mistake and, though I ended up with a bagging ensemble of XGBoost models, I learned an important lesson by comparing multi-layer neural nets to linear models and basic ensembles (e.g. random forests, gradient boosters) and finding that, at least without more data, their performance was underwhelming, especially given how much longer they take to train.

3. Write better code

As the 2018 AFL season approached, my first tipping model had become a tangled mess of scripts, half-baked classes, and hard-coded values, making any extension of functionality, even something as simple as adding new columns to the data set, a herculean task on the order of cleaning out the Augean stables. I had learned much about coding best practices since then and vowed to write more-flexible code for the next model and its surrounding application. Much like EDA, one can always do better, but I'm going to count this goal as achieved as well. I did some mixing-and-matching of functional and object-oriented approaches, building pipelines with a reduced list of transformation functions, but organising the wider application with various classes. I occasionally rewrote large sections of the application to adapt to new requirements or my own changing opinions on the best way to organise the code, but the foundation was strong enough to permit this with less pain and fewer bugs than before. In particular, composing pipelines of functions that each perform one transformation was a major improvement over having a few classes whose transform methods amounted to do_all_the_feature_building_at_once.

4. Come up with a better name

Okay, at least this one is unequivocal: I nailed it. If you can't appreciate the many layers on which 'Tipresias' works, I don't know what to tell you.

Tipresias 1.0: 2019 performance

Despite the various improvements in my processes and code, the results were not exactly worthy of deification. In my office competition, which is what really matters, I finished 4th with 132 tips (i.e. correct predictions) at the end of the regular season, well behind the winner, who got 137. By the end of the season, after the confetti had been swept from Punt Road, I had 137 tips (66.18% accuracy), well behind the betting odds, which had favoured the eventual winner 140 times (67.63% accuracy). My mean absolute error (MAE) (i.e. how far off I was in predicting the winning margin) was also a bit higher at 26.77 vs 26.25 for the betting odds. This was a harder-than-average season for predicting winners, as odds-on favourites tend to win roughly 72% of the time (over the last 10 years), and the top model on Squiggle AFL got 139, whereas last year's winner got 147. It's possible that Tipresias 1.0 is particularly weak in an upset-heavy season, but it's more likely that I would have gotten subpar performance regardless of the contours of the schedule and teams' consistency. In the analysis below I'll use betting odds as a benchmark, because it's a simple, publicly-available heuristic for predicting winners that performs as well as the top, publicly-available statistical models. If I can't beat the betting odds, then I'm at a disadvantage to every Joe and Flo Schmo who has the humility to just pick the favourite from beginning to end.

This comparison of the rolling accuracy of the betting odds and Tipresias shows that both performed poorly in the early seasons, which is typical, as teams change rosters and sometimes coaches in the offseason, and it can be difficult to predict which changed for the better and which for the worse. It is also clear that Tipresias was consistently below the betting odds throughout the season, save for a single round near mid-season, and a few late in the year, which is entirely due to a particularly atrocious round 22, which had a lot of coin-flip matches of which the oddsmakers only picked three of nine and Tipresias picked seven.

Even with the anomaly of round 22, and the sudden dip in the late-teen rounds, the general trend is for accuracy to increase through the early rounds, as oddsmakers and statistical models adjust to those off-season changes, flatten around the middle of the season, then fall a little for the final third or so. Finals are a bit more difficult to predict than mid-season matches, because there is greater parity between teams (even the worst finals team is in the top half of the competition after all), but that doesn't explain the poorer accuracy in the late rounds of the regular season. A piece of conventional footy wisdom is that, as the remaining rounds fall away like shards of marble before the sculptors chisel, the shape of the season is revealed: teams get a sense of which position they'll finish at, and ones safely at the top might rest some players, increasing their chances of losing a throw-away match, and teams just out of finals contention might not do everything in their power to beat lower-ranked teams, because a few losses might result in better draft picks for next year. One of my goals for the offseason is to take a closer look at these changing dynamics across rounds and how they might affect model predictions.

Tipresias's overall performance relative to the betting odds is demonstrated even more clearly in their cumulative accuracies shown below.

Although the rolling accuracy, with a short window, shows Tipresias having a good few rounds and passing the betting odds as a result, its overall accuracy was not higher at any point during the season. Although the model's performance needs to improve in general, one area that I will be focusing on this offseason will be trying to increase accuracy in the early rounds, as there is much more room for improvement during that part of the season than later on when most heuristics and models are consistently tipping 70% - 80%. Squeezing an extra 5% - 10% accuracy out of a model that's already getting most matches right becomes increasingly difficult, and I think there are still some easier gains to be made elsewhere.

Features, features everywhere

So, which features steered Tipresias off course into the empty waste of the wine-dark sea? Since the underlying estimator is XGBoost, we can get the weights of all the features to see which had the largest impact on the model's decisions. It's possible to get these values from the model itself, but I used the model-explanation package eli5, because it offers a few functions to make extracting and visualising the underlying attributes a little easier. Even with the help of this package, I found working with an ensemble of XGBoost estimators nested in a bagging estimator nested in my own custom wrapper class a bit of a pain, which I hope to simplify for the next model to aid in analysing performance. I managed to loop over the the instances of XGBoost and average their feature weights to get a sense of which features contributed most to the final model's predictions. I used the default importance metric 'gain', in part because it's the default. Looking into it a bit more, however, I found that the other popular metric, 'weight', tends to diminish the importance of categorical variables, and I wanted to avoid that, as I believed some categories would prove influential in the model's predictions. So, below are the 20 most important features for Tipresias by gain.

What sticks out is that elo_pred_win is by far the most important feature in the model, with a gain of 0.39, compared to at_home, which is the second most important with just 0.049. Even beyond elo_pred_win, you'll notice many other features that start with elo. This is because I incorporated an Elo-based model as a group of features that I fed into the ensemble model. The problem is that since a decent statistical model is going to be better at predicting match results than almost any other single data point, it is bound to have an outsized influence on the model that it's a part of. Unfortunately, Elo underperformed even more than the betting odds or Tipresias, only achieving 60.87% accuracy (it was closer to 70% during training). So, being overly influenced by a possibly-overfitted Elo model (or one that was overly susceptible to failing in an upset-heavy season if we're being charitable) probably has something to do with Tipresias performing below expectations in 2019.

Since creating Tipresias, I've read Google's Rules of ML, which is a good, practical guide to the machine-learning development process. One rule that stuck in my mind was #40: "Keep ensembles simple". I definitely violated this rule by incorporating a statistical model into the base data set, treating it as though model outputs were of the same type as raw data about matches and players. Therefore, one of my first big changes to Tipresias will be restructuring the model into a voting ensemble rather than a bagging ensemble, so I can separate my raw data from the base models and have a clear hierarchy of inputs and outputs, finishing with the meta-estimator that will make predictions based on predictions of the sub-models. I'll have to see what the performance is like for this change, but I'm willing to accept a short-term hit in the interest of making the model easier to interpret. I'm hoping that this will offer insight into new possibilities for improving long-term performance.

I was surprised by the presence of 'Regular' and 'Finals' (i.e. whether a match was in the regular season or finals) toward the top of the feature-importance list. As I mentioned above, I want to look more closely at how predictions change during different phases of the season and if there are any tendencies that hold from year to year. The importance of these round-type features further suggests that this could be a fruitful area of investigation.

Aggregating prediction explanations by feature and round didn't prove to be particularly illuminating. We see that there isn't much movement for most of the top features save for a slight downward trend for some of the Elo-based features and a sizeable jump in the importance of round-type features with the start of finals. Again, this requires further digging, but I suspect that the model may be under-fitting the changes in context during different phases of the AFL season, optimising for the high-accuracy middle rounds and suffering poor performance early and late in the season as a result.

Tipresias 2020

Analysing the performance and pitfalls of the current version of the model offers some guidance on how I can improve performance for next season.

Make the model easier to interpret
This includes separating model-output features from raw data, and simplifying the class structure (i.e. no so many layers of wrapper classes) so that model-interpretability tools are a little easier to use. There are more packages and forms of analysis than I covered here, but I was unable to use them, because they're picky about which models they accept (BaggingEstimator certainly didn't play nice with most of them) and which classes are exposed to the relevant analysis functions.
Investigate ways to optimise the model for different parts of the season
I will pay particular attention to the early rounds, because that's where there's the most room for a model to gain an advantage over competitors.
Make better use of player data
I incorporated player stats into my model for the first time this season, but had a difficult time figuring out how best to use these features, so I ended up just aggregating rolling averages into per-team stats and calling it a day. As a result, roster changes didn't move Tipresias's predictions much (rarely more than a few points). Since a big part of teams' changes in the off-season are due to gaining or losing players, increasing the importance of these features could be part of improving early-season accuracy.
Add confidence percentage as part of the predictions
This isn't related to model performance, but statistically-oriented tipping competitions include a metric called 'bits' that measures a model's error according to its confidence in a given prediction. A confidence figure comes naturally with ML classifiers, but not regressors. However, I require the predicted margin that comes with a regressor, because that is a part of all tipping competitions. One option is to add a classifier as part of the model to get an extra percentage output. A possible alternative is to learn more about conformal prediction, which is a way of adding confidence intervals to regressors. I'm still unsure of the feasibility of this approach, but it might allow me to engineer some sort of prediction confidence percentage from a regressor's output.

Highlights from PyCon AU 2019

Craig Franklin — Sun, 11 Aug 2019 06:21:07 +0000

I recently attended PyCon Australia, which was my first Python conference anywhere and only my second proper tech conference, but I can state, despite the small sample size, with confidence that PyCon AU 2019 was a great conference. Below are some personal highlights from last weekend.

Day 1: Specialist Track Day

My coworkers and I bought our tickets after the T-shirt order deadline, so we didn't get shirts, and, on the walk to the convention centre alongside the lovely Darling Harbour, a seagull dive-bombed me and got a big ole chunk of my cinnamon role for its trouble (I quickly stuffed the rest in my mouth, and have sworn biblical levels of vengeance on that seagull if ever again we should meet: I'm basically like John Wick for seagulls now). My PyCon weekend had not gotten off to an auspicious start.

Camelot and Excalibur for PDF data

Although I don't currently have any use cases for them, I was really impressed with Vinayak Mehta's talk on the Camelot and Excalibur packages for extracting data tables from PDF files. The way it can use spacing for tables that aren't visually separated into cells and even has a GUI for manually separating rows and columns for tricky documents is really cool. I've never worked with government reports in PDF form, but I have written scripts for converting horribly-formatted government-issued spreadsheets into something resembling a pivotable table, and I have nothing but respect for the tooling (there's a GUI and a CLI!) and flexibility (you can define character filters, column counts, and more!) that he and the other contributors have built into these packages. It almost makes me want to get data from PDFs. Almost.

Unity 3D environments for reinforcement learning

In another case of information-I-won't-use-in-the-foreseeable-future-but-was-very-cool-nonetheless, Paris Buttfield-Addison talked about building virtual environments in Unity 3D for training reinforcement learning models that can then be used to direct robots or self-driving cars. I had heard of Unity 3D, knew it powered some of the games I play, knew that my wife worked with it once, but that was about it. So, it was interesting to see Paris quickly simulate objects and environments with realistic physical interactions by dragging and dropping a few elements in the Unity GUI tool. He then demonstrated how to combine Unity with Unity Machine Learning Agents Toolkit (for plugging TensorFlow models into Unity) by walking through the creation of a virtual racetrack, the architecture for applying reinforcement learning, the code for a reinforcement learning model, then playing the video of his bulldozer careening through his virtual, python-themed track without so much as rubbing the barrier, despite the well-known fact that such activity is racing.

I had seen the use of virtual environments to train AI for robots before at the local ML/AI meetup, but it seemed like some sort of mathy black magic at the time; the combination of Unity's user-friendly UI and Paris's humble speaking style made it all seem so accessible, seem that I too could one day create a self-driving car.

Threat modelling the Death Star

I know nothing about security, and I don't find the subject particularly interesting, but my Star-Wars-fanatic wife insisted that I go to this talk. So, partly for the sake of my marriage, partly intrigued by the talk's conceit, I went to listen to Mario Areias talk about threat modelling. Due to my current role as a basic backend dev, I'm unlikely to put into practice the techniques that I learned about, but for sheer entertainment value, Mario's talk is worth watching. Also, if you're interested in speaking at a conference, note how the description of the talk is written to make one want to attend even if threat modelling might not interest them personally, and how the talk itself, from slides to speech, is both informative and fun.

Day 2: Main Conference Day 1 (don't think too hard about it)

Lessons learned building Python microservices

I've really been getting into DevOps and microservices over the past year, and Richard Jones's talk was one of the two at PyCon from which I got really great information about concepts, techniques, and tools that I can implement in projects in the near future. Hearing about the challenges that he and his team have faced in stitching together microservices that use different frameworks and languages, and the design decisions that they've made reinforced some things that I've heard elsewhere and made me aware of some new possibilities. In particular, their separation of the backend into multiple layers of services (e.g. a layer that connects to the frontend and another that connects with the database) was interesting. Also, I'm really eager to try out some of the tooling that Richard mentioned. I've used cookiecutter a little before, but never created my own, and the idea of using it to give a consistent, framework-like directory structure to all microservices sounds like a good idea. pact.io sounds like an absolute godsend for some of the problems that I've been having lately with keeping track of valid request and response bodies. One solution would be to become less of a careless flake, but having a tool that enforces contracts between services sounds more reasonable than completely changing my personality.

Sufficiently advanced testing (deep-dive talk)

And this is the other talk that left me thinking, wow, I've got to try this stuff out, like yesterday. Zac Hatfield-Dodds's talk, being a deep-dive, gave him time to cover some advanced testing concepts, then get into the details of the package hypothesis, which implements many of them. I was already using faker to generate random values for most of my test fixtures, but in this talk, Zac took it further by introducing me to the concept of fuzzy testing, specifically, using algorithmically-generated randomness to probe the limits of what your code can handle without breaking. In theory, this will create combinations of inputs and situations that you never thought of, hopefully uncovering edge-case bugs before they pop up in production. In my day-to-day, I work on a giant Rails monolith with many, many interconnected parts, and the number of times our imaginative users create an absolutely inconceivable menagerie of conditions that break some far-flung corner of the codebase that hasn't been changed since that whitespace git commit from four years prior makes me really appreciate the potential value of such a technique.

Zac then presented hypothesis, of which he is a maintainer, and the ways in which we mere mortals can implement fuzzy testing in our own projects without having to understand all those fancy algorithms. I have yet to really dig into the documentation, but the package seems really intuitive, with options to limit the randomly-generated inputs within given constraints, which serves to get your tests passing, but also makes explicit your assumptions about what can and cannot happen in your app. And, as we all know, explicit is better than implicit. There also seems to be a mechanism for saving input values from failed test runs, which goes a long way toward reducing the pain of flaky tests.

When software and law are the same thing

Another example of a talk and speaker that are the whole package, Brenda Wallace's talk was interesting, entertaining, and Brenda is a compelling speaker, combining humour and serious issues with ease. If you thought converting business requirements into code was difficult, have you ever tried to encode the logic of decades-old laws that have been changed piecemeal by this party, and that minister, over the years? Turns out different people have different ideas about what defines a 'child' and exactly how old four-and-a-half years is. The idea behind this project is something that had never occurred to me, and its goals (democratising understanding of laws and simplifying people's interactions with government bureaucracy) are laudable. Having recently gained permanent residency in Australia myself (not to mention having interacted with immigration departments in four different countries), I can guarantee that even when you try to fill out all the forms correctly and go to all the right desks at all the right offices, you're just guessing half the time, because government instructions are never so clear nor so explicit as a good old if/else.

Day 3: Main Conference Day 2

By Sunday, I was pretty exhausted. Whether it was the two full days of talks and learning and socialising, or the beers I had drunk the night before, or the whiskey I had drunk the night before, who can say? Regardless, I was pounding caffeine throughout the day, and I missed a few more talks than on previous days just to give myself a break from paying attention to stuff.

The new COBOL

For me, the principle charm of Benno Rice's talk on COBOL and how we talk about different technologies was his survey of the history of programming languages, their roots, principle uses, and coders' (often biased) perceptions of them. I have far more formal training in iambic pentameter than the Law of Demeter, so much of this was new information to me. Like any good talk, this one certainly inspired me to want to read more on the subject when I have some time. A simple history of the likes of COBOL, Lisp, FP, and OOP, however, would have been a little dry, so the fact that Benno was able to tie his history lesson into a broader point about how we talk about various languages and, more importantly, the people who work with them added immediate relevance to make it really engaging.

Bonus: Lightning Talks

I won't go into specific lightning talks (though there were many good ones, some informative, some funny, some just plain weird, some all of the above), but I just wanted to say that the whole exercise was definitely a highlight of the conference for me. They were perfectly timed toward the end of Saturday and Sunday, right when the caffeine from afternoon tea was fading and the hunger for dinner was rising, to energise everyone in attendance, so that we could finish the day full of vim and vigour. When there's only five minutes to cover one's material, not all of the talks will go according to plan, and some speakers had to be cut off, but the consistent, quick tempo and the intellectual whiplash of hearing about so many different topics, techniques, and technologies in such a short period of time meant that even the chaos, the unexpected, contributed to my enjoyment. If you have the chance to attend a future PyCon AU, you should definitely attend the lightning talks sessions, and I highly recommend watching the videos from this year, because there is some really good stuff in there.

Docker, Django, React: Building Assets and Deploying to Heroku

Craig Franklin — Sun, 02 Jun 2019 02:10:59 +0000

Part 2 in a series on combining Docker, Django, and React. This builds on the development setup from Part 1, so you might want to take a look at that first. If you want to skip to the end or need a reference, you can see the final version of the code on the production-heroku branch of the repo.

Update: ohduran has created a cookiecutter template based on this tutorial if you want a quick-and-easy way to get the code.

Now that we have our app humming like a '69 Mustang Shelby GT500 in our local environment, doing hot-reloading donuts all over the parking lot, it's time to deploy that bad boy, so the whole world can find out how many characters there are in all their favourite phrases. In order to deploy this app to production, we'll need to do the following:

Set up Django to use WhiteNoise to serve static assets in production.
Create a production Dockerfile that combines our frontend and backend into a single app.
Create a new Heroku app to deploy to.
Configure our app to deploy a Docker image to Heroku.

Use WhiteNoise to serve our frontend assets

Update settings for different environments

Since we only want to use WhiteNoise in production, we'll have to change how our Django app's settings work to differentiate between the dev and prod environments. There are different ways to do this, but the one that seems to offer the most flexibility, and has worked well enough for me, is to create a settings file for each environment, all of which inherit from some base settings, then determine which settings file to use with an environment variable. In the backend/hello_world, which is our project directory, create a settings folder (as usual, with a __init__.py inside to make it a module), move the existing settings.py into it, and rename it base.py. This will be the collection of base app settings that all environments will inherit. To make sure we don't accidentally deploy with unsafe settings, cut the following code from base.py, and paste it into a newly-created development.py:

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.2/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = "<some long series of letters, numbers, and symbols that Django generates>"

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = ["backend"]

Double check now: have those lines of code disappeared from base.py? Good. We are slightly less hackable. At the top of the file, add the line from hello_world.settings.base import *. What the * import from base does is make all of those settings that are already defined in our base available in development as well, where we're free to overwrite or extend them as necessary.

Since we're embedding our settings files a little deeper in the project by moving them into a settings subdirectory, we'll also need to update BASE_DIR in base.py to point to the correct directory, which is now one level higher (relatively speaking). You can wrap the value in one more os.path.dirname call, but I find the following a little easier to read:

BASE_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../"))

Django determines which module to use when running the app with the environment variable DJANGO_SETTINGS_MODULE, which should be the module path to the settings that we want to use. To avoid errors, we update the default in backend/hello_world/wsgi.py to 'hello_world.settings.base', and add the following to our backend service in docker-compose.yml:

environment:
  - DJANGO_SETTINGS_MODULE=hello_world.settings.development

Add production settings with WhiteNoise

The reason we want to use WhiteNoise in production instead of whatever Django does out-of-the-box is because, by default, Django is very slow to serve frontend assets, whereas WhiteNoise is reasonably fast. Not as fast as professional-grade CDN-AWS-S3-bucket-thingy fast, but fast enough for our purposes.

To start, we need to install WhiteNoise by adding whitenoise to requirements.txt.

Next, since we have dev-specific settings, let's create production.py with settings of its very own. To start, we'll just add production variations of the development settings that we have, which should look something like this:

import os
from hello_world.settings.base import *

SECRET_KEY = os.environ.get("SECRET_KEY")
DEBUG = False
ALLOWED_HOSTS = [os.environ.get("PRODUCTION_HOST")]

We'll add the allowed host once we set up an app on Heroku. Note that you can hard-code the allowed host in the settings file, but using an environment variable is a little easier to change if you deploy to a different environment. The SECRET_KEY can be any string you want, but for security reasons it should be some long string of random characters (I just use a password generator for mine), and you should save it as an environment/config variable hidden away from the cruel, thieving world. Do not check it into source control!.

To enable WhiteNoise to serve our frontend assets, we add the following to production.py:

INSTALLED_APPS.extend(["whitenoise.runserver_nostatic"])

# Must insert after SecurityMiddleware, which is first in settings/common.py
MIDDLEWARE.insert(1, "whitenoise.middleware.WhiteNoiseMiddleware")

TEMPLATES[0]["DIRS"] = [os.path.join(BASE_DIR, "../", "frontend", "build")]

STATICFILES_DIRS = [os.path.join(BASE_DIR, "../", "frontend", "build", "static")]
STATICFILES_STORAGE = "whitenoise.storage.CompressedManifestStaticFilesStorage"
STATIC_ROOT = os.path.join(BASE_DIR, "staticfiles")

STATIC_URL = "/static/"
WHITENOISE_ROOT = os.path.join(BASE_DIR, "../", "frontend", "build", "root")

Most of the above comes from the WhiteNoise documentation for implementation in Django, along with a little trial and error to figure out which file paths to use for finding the assets built by React (more on that below). The confusing bit is all the variables that refer to slightly different frontend-asset-related directories.

TEMPLATES: directories with templates (e.g. Jinja) or html files
STATICFILES_DIRS: directory where Django can find html, js, css, and other static assets
STATIC_ROOT: directory to which Django will move those static assets and from which it will serve them when the app is running
WHITENOISE_ROOT: directory where WhiteNoise can find all non-html static assets

Add home URL for production

In addition to changing the settings, we have to make Django aware of the path /, because right now it only knows about /admin and /char_count. So, we'll have to update /backend/hello_world/urls.py to look like the following:

from django.contrib import admin
from django.urls import path, re_path
from django.views.generic import TemplateView
from char_count.views import char_count

urlpatterns = [
    path("admin/", admin.site.urls),
    path("char_count", char_count, name="char_count"),
    re_path(".*", TemplateView.as_view(template_name="index.html")),
]

Note that we've added a regex path (.*) that says to Django, "Any request that you don't have explicit instructions for, just respond by sending them index.html". How this works in practice is that in a dev environment, React's Webpack server will still handle calls to / (and any path other than the two defined above), but in production, when there's no Webpack server, Django will just shrug its shoulders and serve index.html from the static files directory (as defined in the settings above), which is exactly what we want. The reason we use .* instead of a specific path is it allows us the freedom to define as many paths as we want for the frontend to handle (with React Router for example) without having to update Django's URLs list.

None of these changes should change our app's functionality on local, so try running docker-compose up to make sure nothing breaks.

Create a production Dockerfile

In order for WhiteNoise to be able to serve our frontend assets, we'll need to include them in the same image as our Django app. There are a few ways we could accomplish this, but I think the simplest is to copy the Dockerfile that builds our backend image and add the installation of our frontend dependencies, along with the building of our assets, to it. Since this image will contain a single app that encompasses both frontend and backend, put it in the project root.

FROM python:3.6

# Install curl, node, & yarn
RUN apt-get -y install curl \
  && curl -sL https://deb.nodesource.com/setup_8.x | bash \
  && apt-get install nodejs \
  && curl -o- -L https://yarnpkg.com/install.sh | bash

WORKDIR /app/backend

# Install Python dependencies
COPY ./backend/requirements.txt /app/backend/
RUN pip3 install --upgrade pip -r requirements.txt

# Install JS dependencies
WORKDIR /app/frontend

COPY ./frontend/package.json ./frontend/yarn.lock /app/frontend/
RUN $HOME/.yarn/bin/yarn install

# Add the rest of the code
COPY . /app/

# Build static files
RUN $HOME/.yarn/bin/yarn build

# Have to move all static files other than index.html to root/
# for whitenoise middleware
WORKDIR /app/frontend/build

RUN mkdir root && mv *.ico *.js *.json root

# Collect static files
RUN mkdir /app/backend/staticfiles

WORKDIR /app

# SECRET_KEY is only included here to avoid raising an error when generating static files.
# Be sure to add a real SECRET_KEY config variable in Heroku.
RUN DJANGO_SETTINGS_MODULE=hello_world.settings.production \
  SECRET_KEY=somethingsupersecret \
  python3 backend/manage.py collectstatic --noinput

EXPOSE $PORT

CMD python3 backend/manage.py runserver 0.0.0.0:$PORT

The Dockerfile above installs everything we need to run both Django and React apps, then builds the frontend assets, then collects those assets for WhiteNoise to serve them. Since the collectstatic command makes changes to the files, we want to run it during our build step rather than as a separate command that we run during deployment. You could probably do the latter under some circumstances, but I ran into problems when deploying to Heroku, because they discard post-deployment file changes on free-tier dynos.

Also, note the command that moves static files from /app/frontend/build to /app/frontend/build/root, leaving index.html in place. WhiteNoise needs everything that isn't an HTML file in a separate subdirectory. Otherwise, it gets confused about which files are HTML and which aren't, and nothing ends up getting loaded. Many Bothans died to bring us this information.

Create an app on Heroku

If you're new to Heroku, their getting started guide will walk you through the basics of creating a generic, non-dockerized Python app. If you don’t have it yet, install the Heroku CLI. We can create a Heroku app by running heroku create within our project. Once you've created your new Heroku app, copy the URL displayed by the command, and add it to ALLOWED_HOSTS in settings.production. Just like adding backend to our allowed hosts on dev, we need this to make sure Django's willing to respond to our HTTP requests. (I can't even begin to count the number of blank screens I've repeatedly refreshed with a mix of confusion and despair due to forgetting to add the hostname to ALLOWED_HOSTS when deploying to a new environment). If you want to keep it secret, or want greater flexibility, you can add os.environ.get("PRODUCTION_HOST") to the allowed hosts instead, then add your Heroku app's URL to its config variables. I'm not sure how strict it is for which URL elements to include or omit, but <your app name>.herokuapp.com definitely works.

For environment variables in production, we can use the Heroku CLI to set secure config variables that will be hidden from the public. Heroku has a way of adding these variables with heroku.yml, but I always have trouble getting it to work, so I opt for the manual way in this case. This has the added benefit of not having to worry about which variables are okay to commit to source control and which we need to keep secret. To set the config variables, run the following in the terminal:

heroku config:set PRODUCTION_HOST=<your app name>.herokuapp.com SECRET_KEY=<your secret key> DJANGO_SETTINGS_MODULE=hello_world.settings.production

As stated earlier, PRODUCTION_HOST is optional (depending on whether you added the app URL to ALLOWED_HOSTS directly). DJANGO_SETTINGS_MODULE will make sure that the app uses our production settings when running on Heroku.

Deploy to Heroku

There are a couple of different ways we can deploy Dockerized apps to Heroku, but I like heroku.yml, because, like docker-compose.yml, it has all the app configs and commands in one place. Heroku has a good introduction to how it all works, but for our purposes, we only need the following:

build:
  docker:
    web: Dockerfile
run:
  web: python3 backend/manage.py runserver 0.0.0.0:$PORT

We also need to run heroku stack:set container in the terminal to tell our Heroku app to use Docker rather than one of Heroku's language-specific build packs. Now, deploying is as easy as running git push heroku master (if you're on the master branch; otherwise, run git push heroku <your branch>:master).

Once Heroku is finished building our image and deploying, we can open a browser to <your app name>.herokuapp.com and count characters on the CLOOOOOUUUUUD!!!

Summary

Conceptually, putting the frontend and backend together into a single app that we can deploy to Heroku is very simple, but there are so many little gotchas in the configurations and file structure (not to mention the lack of meaningful error messages whenever one makes a mistake) that I found it devilishly difficult to get it all working. Even going through this process a second time while writing this tutorial, I forgot something here, added the wrong thing there, and spent hours trying to remember how I got it working the first time around, and what terrible sin I might have committed to cause the coding gods to punish me now.

But here we are, having just accomplished the following:

Configure environment-specific settings for Django.
Set up WhiteNoise to serve static assets in production.
Create a production Dockerfile that includes frontend and backend code and dependencies.
Create a Heroku app and deploy our code to it using heroku.yml and the container stack.

Creating an app with Docker Compose, Django, and Create React App

Craig Franklin — Thu, 16 May 2019 23:03:10 +0000

Final code for this tutorial if you want to skip the text, or get lost with some of the references, can be found on GitHub.

Update: ohduran has created a cookiecutter template based on this tutorial if you want a quick-and-easy way to get the code.

Inspired by sports data sites like Squiggle and Matter of Stats, in building the app that houses Tipresias (my footy-tipping machine-learning model), I wanted to include a proper front-end with metrics, charts, and round-by-round tips. I already knew that I would have to dockerize the thing, because I was working with multiple packages across Python and R, and such complex dependencies are incredibly difficult to manage in a remote-server context (and impossible to run on an out-of-the-box service like Heroku) without using Docker. I could have avoided exacerbating my complexity issue by using basic Django views (i.e. static HTML templates) to build my pages, but having worked with a mishmash of ancient Rails views that had React components grafted on to add a little interactivity (then a lot of interactivity), I preferred starting out with clear separation between my frontend and backend. What's more, I wanted to focus on the machine learning, data engineering, and server-side logic (not to mention the fact that I couldn't design my way out of a wet paper bag), so my intelligent, lovely wife agreed to help me out with the frontend, and there was no way she was going to settle for coding in the context of a 10-year-old paradigm. It was going to be a modern web-app architecture, or I was going to have to pad my own divs.

The problem with combining Docker, Django, and React was that I had never set up anything like this before, and, though I ultimately figured it out, I had to piece together my solution from multiple guides/tutorials that did some aspect of what I wanted without covering the whole. In particular, the tutorials that I found tended to build static Javascript assets that Django could use in its views. This is fine for production, but working without hot-reloading (i.e. having file changes automatically restart the server, so that they are reflected in the relevant pages that are loaded in the browser) is the hair shirt of development: at first you think you can endure the mild discomfort, but the constant itching wears you down, becoming the all-consuming focus of your every waking thought, driving you to distraction, to questioning all of your choices in life. Imagine having to run a build command that takes maybe a minute every time you change so much as a single line of code. Side projects don't exactly require optimal productivity, but, unlike jobs, if they become a pain to work on, it's pretty easy to just quit.

What we're gonna do

Create a Django app that runs inside a Docker container.
Create a React app with the all-too-literally-named Create React App that runs inside a Docker container.
Implement these dockerized apps as services in Docker Compose.
Connect the frontend service to a basic backend API from which it can fetch data.

Note: This tutorial assumes working knowledge of Docker, Django, and React in order to focus on the specifics of getting these three things working together in a dev environment.

1. Create a dockerized Django app

Let's start by creating a project directory named whatever you want, then a backend subdirectory with a requirements.txt that just adds the django package for now. This will allow us to install and run Django in a Docker image built with the following Dockerfile:

# Use an official Python runtime as a parent image
FROM python:3.6

# Adding backend directory to make absolute filepaths consistent across services
WORKDIR /app/backend

# Install Python dependencies
COPY requirements.txt /app/backend
RUN pip3 install --upgrade pip -r requirements.txt

# Add the rest of the code
COPY . /app/backend

# Make port 8000 available for the app
EXPOSE 8000

# Be sure to use 0.0.0.0 for the host within the Docker container,
# otherwise the browser won't be able to find it
CMD python3 manage.py runserver 0.0.0.0:8000

In the terminal, run the following commands to build the image, create a Django project named hello_world, and run the app:

docker build -t backend:latest backend
docker run -v $PWD/backend:/app/backend backend:latest django-admin startproject hello_world .
docker run -v $PWD/backend:/app/backend -p 8000:8000 backend:latest

Note that we create a volume for the backend directory, so the code created by startproject will appear on our machine. The . at the end of the create command will place all of the Django folders and files inside our backend directories instead of creating a new project directory, which can complicate managing the working directory within the Docker container.

Open your browser to localhost:8000 to verify that the app is up and running.

2. Create a dockerized Create React App (CRA) app

Although I got my start coding frontend Javascript, I found my calling working on back-end systems. So, through a combination of my own dereliction and the rapid pace of change of frontend tools and technologies, I am ill-equipped to set up a modern frontend application from scratch. I am, however, fully capable of installing a package and running a command.

Unlike with the Django app, we can't create a Docker image with a CRA app all at once, because we'll first need a Dockerfile with node, so we can initialise the CRA app, then we'll be able to add the usual Dockerfile commands to install dependencies. So, create a frontend directory with a Dockerfile that looks like the following:

# Use an official node runtime as a parent image
FROM node:8

WORKDIR /app/

# Install dependencies
# COPY package.json yarn.lock /app/

# RUN npm install

# Add rest of the client code
COPY . /app/

EXPOSE 3000

# CMD npm start

Some of the commands are currently commented out, because we don't have a few of the files referenced, but we will need these commands later. Run the following commands in the terminal to build the image, create the app, and run it:

docker build -t frontend:latest frontend
docker run -v $PWD/frontend:/app frontend:latest npx create-react-app hello-world
mv frontend/hello-world/* frontend/hello-world/.gitignore frontend/ && rmdir frontend/hello-world
docker run -v $PWD/frontend:/app -p 3000:3000 frontend:latest npm start

Note that we move the newly-created app directory's contents up to the frontend directory and remove it. Django gives us the option to do this by default, but I couldn't find anything to suggest that CRA will do anything other than create its own directory. Working around this nested structure is kind of a pain, so I find it easier to just move everything up the docker-service level and work from there. Navigate your browser to localhost:3000 to make sure the app is running. Also, you can uncomment the rest of the commands in the Dockerfile, so that any new dependencies will be installed the next time you rebuild the image.

3. Docker-composify into services

Now that we have our two Docker images and are able to run the apps in their respective Docker containers, let's simplify the process of running them with Docker Compose. In docker-compose.yml, we can define our two services, frontend and backend, and how to run them, which will allow us to consolidate the multiple docker commands, and their multiple arguments, into much fewer docker-compose commands. The config file looks like this:

version: "3.2"
services:
  backend:
    build: ./backend
    volumes:
      - ./backend:/app/backend
    ports:
      - "8000:8000"
    stdin_open: true
    tty: true
    command: python3 manage.py runserver 0.0.0.0:8000
  frontend:
    build: ./frontend
    volumes:
      - ./frontend:/app
      # One-way volume to use node_modules from inside image
      - /app/node_modules
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=development
    depends_on:
      - backend
    command: npm start

We've converted the various arguments for the docker commands into key-value pairs in the config file, and now we can run both our frontend and backend apps by just executing docker-compose up. With that, you should be able to see them both running in parallel at localhost:8000 and localhost:3000.

4. Connecting both ends into a single app

Of course, the purpose of this post is not to learn how to overcomplicate running independent React and Django apps just for the fun of it. We are here to build a single, integrated app with a dynamic, modern frontend that's fed with data from a robust backend API. Toward that goal, while still keeping the app as simple as possible, let's have the frontend send text to the backend, which will return a count of the number of characters in the text, which the frontend will then display.

Setting up the Django API

Let's start by creating an API route for the frontend to call. You can create a new Django app (which is kind of a sub-app/module within the Django project architecture) by running the following in the terminal:

docker-compose run --rm backend python3 manage.py startapp char_count

This gives you a new directory inside backend called char_count, where we can define routes and their associated logic.

We can create the API response in backend/char_count/views.py with the following, which, as promised, will return the character count of the submitted text:

from django.http import JsonResponse


def char_count(request):
    text = request.GET.get("text", "")

    return JsonResponse({"count": len(text)})

Now, to make the Django project aware of our new app, we need to update INSTALLED_APPS in backend/hello_world/settings.py by adding "char_count.apps.CharCountConfig" to the list. To add our count response to the available URLs, we update backend/hello_world/urls.py with our char_count view as follows:

from django.contrib import admin
from django.urls import path
from char_count.views import char_count

urlpatterns = [
    path('admin/', admin.site.urls),
    path('char_count', char_count, name='char_count'),
]

Since we're changing project settings, we'll need to stop our Docker Compose processes (either ctl+c or docker-compose stop in a separate tab) and start it again with docker-compose up. We can now go to localhost:8000/char_count?text=hello world and see that it has 11 characters.

Connecting React to the API

First, let's add a little more of that sweet config to make sure we don't get silent errors related to networking stuff that we'd really rather not deal with. Our Django app currently won't run on any host other than localhost, but our React app can only access it via the Docker service name backend (which does some magic host mapping stuff). So, we need to add "backend" to ALLOWED_HOSTS in backend/hello_world/settings.py, and we add "proxy": "http://backend:8000" to package.json. This will allow both services to talk to each other. Also, we'll need to use the npm package axios to make the API call, so add it to package.json and rebuild the images with the following:

docker-compose run --rm frontend npm add axios
docker-compose down
docker-compose up --build

My frontend dev skills are, admittedly, subpar, but please keep in mind that the little component below is not a reflection of my knowledge of React (or even HTML for that matter). In the interest of simplicity, I just removed the CRA boilerplate and replaced it with an input, a button, a click handler, and a headline.

import React from 'react';
import axios from 'axios';
import './App.css';

function handleSubmit(event) {
  const text = document.querySelector('#char-input').value

  axios
    .get(`/char_count?text=${text}`).then(({data}) => {
      document.querySelector('#char-count').textContent = `${data.count} characters!`
    })
    .catch(err => console.log(err))
}

function App() {
  return (
    <div className="App">
      <div>
        <label htmlFor='char-input'>How many characters does</label>
        <input id='char-input' type='text' />
        <button onClick={handleSubmit}>have?</button>
      </div>

      <div>
        <h3 id='char-count'></h3>
      </div>
    </div>
  );
}

export default App;

Now, when we enter text into the input and click the button, the character count of the text is displayed below. And best of all: we got hot reloading all up and down the field! You can add new components to the frontend, new classes to the backend, and all your changes (short of config or dependencies) will be reflected in the functioning of the app as you work, without having to manually restart the servers.

Summary

In the end, setting all this up isn't too complicated, but there are lots of little gotchas, many of which don't give you a nice error message to look up on Stack Overflow. Also, at least in my case, I really struggled at first to conceptualise how the pieces were going to work together. Would the React app go inside the Django app, like it does with webpacker in Rails? If the two apps are separate Docker Compose services, how do you connect them? In the end we learned how to:

Set up Django in a Docker container.
Set up Create React App in a Docker container
Configure those containers with Docker Compose
Use Docker Compose's service names (e.g. backend) and package.json's "proxy" attribute to direct React's HTTP call to Django's API and display the response.