DEV Community: Michael Morehouse

The 35 Words You Need to Python

Michael Morehouse — Sun, 28 Jul 2019 00:00:00 +0000

In the Beginning were Python's Keywords

Let’s face it, learning to write computer programs is hard. Doing so involves picking up a lot of specialized skills and jargon, most of which is unfamiliar, and it also means cultivating a mental approach to the world that, at first, might seem downright alien. It’s a long journey, but fluency starts with learning how to read computer programs, and that really shouldn’t be so hard.

Programs must be written for people to read, and only incidentally for machines to execute.

Harold Abelson

One of the defining traits of the Python programming language is that it’s designed to be readable by humans. For the most part it fulfills that promise, however, after several years of helping others understand Python, I’ve come to realize that there’s an important caveat to that statement; it’s really only true for someone with a significant and specialized English-language vocabulary.

The truth is you need enough mastery of the “rules” of English grammar and wordplay to recognize and understand, at a glance, words written in English that are not actually words you’ll find in an English dictionary. Nonstandard terms like def (a contraction), elif (a portmanteau) and nonlocal (a neologism) all abound, so even for those with quite advanced native English fluency the task of learning Python is very much like trying to learn a foreign language that’s slightly related to their native tongue. For those without that fluency learning Python is like learning a foreign language inside a foreign language, and is therefore far more difficult.

To make that task a little easier I’m going to try, in this post and the ones that follow, to shed some light on the meaning of – and a little of the etymological history behind – the fundamental units of Python fluency. In this first part we will start with the most basic of those units, Python’s 35 keywords¹.

That’s right; the entire vocabulary of Python you actually need to know to start to do meaningful work is just 35 keywords. It’s not the smallest language, but it’s far from the largest, and just compare it to the roughly 10,000 words required to achieve basic native fluency in a non-programming language.

First, Some Conventions

Python is what is known as a statement-oriented language; but what is a statement? Well, for the purposes of this article we’re just going to say that in Python a statement is a single line of code that does something. What it does, specifically, depends on the building blocks of that statement.

But what are those building blocks? Well, let’s define them quickly and very roughly, since we’ll go into more detail about them in later posts. I’ll use UPPERCASE letters to make it easier to visually distinguish these abstract forms from the specific instances we’ll talk about later.

KEYWORD: A reserved word the meaning of which cannot be changed by the user. We will visit all 35 of these in the next section of this article.

OPERATOR

A reserved symbol that indicates an action to be performed. For example, = is the assignment OPERATOR and + is the addition OPERATOR. There are quite a few others, but we’ll save them for the next post. A small number of KEYWORDs behave like OPERATORs, and I’ll point those out below.

These are both provided by Python and you can’t directly change their meaning, which means that they’re somewhat inflexible. To do most work you’ll need something more flexible, which is why Python gives you the ability to represent anything.

OBJECT: An individual thing you can interact with. Unlike KEYWORDs and OPERATORs, you can directly manipulate these, though the degree to which you can manipulate them depends on what type of OBJECT they are. You can also use KEYWORDs to define entirely new types, which makes them a very expressive way of building new things of your own. So expressive, in fact, that practically speaking everything you interact with in Python will be an OBJECT.

That can be a bit abstract and hard to wrap your head around at this point, though. For now just know that OBJECTs tend to fall into three main categories.

VALUE: An OBJECT that represents a single, concrete thing; for the purposes of this discussion what that thing actually is is irrelevant, but as an example, 4 is a VALUE of the int (short for integer) type and hello is a VALUE of the str (short for string) type. These are both examples of primitive types, which have a single meaningful value, but there are also composite types for describing things the meaning of which is defined by more than one attribute. A real-world example would be a rectangle, which cannot be defined without both height and width. As you’ll see below three special KEYWORDs all behave like VALUEs, though as before you cannot change their meaning.

COLLECTION

An OBJECT that groups together or contains other OBJECTs; there are many different types of COLLECTIONs in Python, but for the moment all we care about is that a COLLECTION contains zero or more OBJECTs. For example the statement [2, 3, 4] creates a COLLECTION of the type list that holds three VALUEs inside of it. A COLLECTION can contain any OBJECT, so you can nest a COLLECTION inside another COLLECTION.

CALLABLE

An OBJECT that represents some action to perform: it performs that action when you call it with some number of arguments then it returns (or gives back) an OBJECT. For instance sum is a CALLABLE and when we call it using sum([2, 3, 4]) it gives us back the VALUE 9. There are several different kinds of CALLABLE, and we’ll touch on them in more detail below.

It wouldn’t be very efficient to type out the same OBJECT every time you needed to refer to it, though. It’s often very helpful to be able to refer to things indirectly.

NAME: Any word that is not a KEYWORD, and that is used as an alias to refer to some specific OBJECT. Unlike a KEYWORD the meaning of a NAME may change over the course of a program, which is why these are often – if a little incorrectly – thought of as variables. There are several ways to create new NAMEs (and one to destroy them), as we’ll see below, but as a simple example in number = 2 the assignment OPERATOR = creates the NAME number and assigns it to refer to the VALUE 2. When later that is followed by number += 2, however, the augmented assignment OPERATOR += will re-assign number to refer to 4.

Now we’ve got all the simple building blocks defined and we can start organizing them into composite structures.

EXPRESSION: Any composite form of one or more of the above that can be evaluated to an OBJECT. For example, 4, 2 + 2, 1 * 3 + 1 and sum([1, 1, 1, 1]) are all EXPRESSIONs that evaluate to 4. The EXPRESSION represents the smallest discrete unit of work in Python.

STATEMENT

Any single line of code that is composed of at least one of the above. These can get quite complex, but to do anything they’ll usually need to include KEYWORDs and/or OPERATORs plus EXPRESSIONs. You’ve already met a useful STATEMENT in number = 2. If you read each STATEMENT in a program out in turn you can track the program as it does its work.

That covers any given line of code, but there are also a couple of higher level structures we need to define for the moment:

BLOCK: At least two STATEMENTs that are bound together; the first STATEMENT will end in a : character and indicates the start of the BLOCK. The second and all further STATEMENTs inside that BLOCK will be indented further right than the initial STATEMENT, to indicate that they belong to the same BLOCK. The last such indented STATEMENT represents the end of the BLOCK.

MODULE

A single Python .py file; it’s composed of some number of STATEMENTS. All Python programs are comprised of at least one MODULE. As you’ll see below we write all of our functionality inside MODULEs, and we use KEYWORDs and NAMEs to import functionality from other MODULEs.

There are many other concepts you’ll need to become familiar with, but with these building blocks we can investigate all 35 words in Python’s relatively small vocabulary, and thus understand the skeleton of any Python program.

On to the Keywords

One That Does Nothing

pass 🗎: A placeholder; technically known as Python’s null operation, pass does nothing whatsoever; it exists solely to allow you to write a syntactically valid BLOCK.

The general meaning here comes from a Middle English verb borrowed via Old French from the Latin passus that implies “to move by [a place, without stopping]”. More specifically the meaning in Python is borrowed from its use in sequential-play card games such as Bridge, where if you do not wish to do anything on your turn you pass control of the game to the next player, doing nothing.

The need for this is simple; once you begin a BLOCK in Python it must contain at least one indented STATEMENT to be considered valid syntax. The pass statement exists to allow you to write a valid BLOCK structure before you’re ready to start writing meaningful statements.

STATEMENT:
    pass

Because it’s mainly used early on when building the rough structure of a program you’ll rarely, if ever, see pass in working code, but it’s good to know it exists.

Three That Are Objects

The next three keywords are specialized because they each behave like a primitive VALUE. This means they can be assigned to a NAME, kept within a COLLECTION, and can be the result of evaluating an EXPRESSION. They’re also the only keywords that start with a capital letter, which makes them easy to distinguish.

Boolean values

These two are used in most, if not all, programs, and are essential whenever performing Boolean logic.

True 🗎: Indicator of logical truth and the opposite of False ; behaves like the integer 1, except that it will always be displayed as True .

From the Old English adjective triewe , which has German roots; the general meaning is of being “worthy of trust” and “consistent with fact”. In logic, though, the specific meaning is really just “that which is not false”, and in computer programming it’s usually a proxy for the binary digit 1.

False 🗎: Indicator of logical untruth and the opposite of True ; behaves like the integer 0, except that it will always be displayed as False .

From late Old English, borrowed via the Old French faus from the Latin falsus ; the general meaning is of “fake, incorrect, mistaken, or deceitful”. In logic the meaning is not so sinister, it just means “that which is not true”, and in computer programming it’s usually a proxy for the binary digit 0.

In some lower-level languages you would probably just use 1 and 0 for these, but in Python they’ve been given special keywords to make it obvious that you’re working with the logical meaning instead of the numerical one.

It’s important to understand that in Python every OBJECT (hence ever VALUE and COLLECTION, and therefore every EXPRESSION), has a logical value, in that it’s considered to be logically equivalent to either True or False. Testing the state of that logical value is known as truth-value testing, and as you’ll see below keywords like and, or, and if all rely on truth-value testing for their operation.

I will go into deeper detail about the specifics of truth-value testing in later articles, but for now you just need to know that most things are considered True by default, except for “no value” VALUEs like 0, None, and False itself, as well as “no content” COLLECTIONs like [].

The null value

This is most commonly used to represent the absence of any other VALUE.

None 🗎: A special name for nothing whatsoever; technically Python’s null object; it’s considered equivalent to False for truth-value testing, but essentially represents no value at all. Is very commonly used in Python, and will always appear as None .

A Middle English pronoun from the Old English nan , meaning “not one” or “not any”. The meaning in programming, however, relates more to null , which means “not [any] thing” or just “nothing”. Python chose to use none because it’s a more commonly familiar word. It also helps to distinguish it from the use of the special value NULL in the C programming language, which has a similar meaning but behaves in a very different way.

The notion of making something that explicitly represents nothing might seem a little odd, at first, but the need for None becomes obvious when you start building useful code.

Three for Making Decisions

Being able to tell if something is considered True or False isn’t very useful unless you have the means to take different actions based on that knowledge. For that most programming languages have some notion of conditional operations. In Python there are three keywords dedicated to conditional tasks.

if 🗎: Starts a conditional BLOCK by checking the truth-value of the EXPRESSION that follows it; the STATEMENT(s) indented underneath the if will be executed only if the EXPRESSION is considered True .

A Middle English conjunction from the Old English gif which means, “in the event” or “whether” or, oddly, just “ if ”. Has many Scandinavian/Germanic relatives, and possibly arrives via an Old Norse term for “doubt, hesitation”. The general use is to make one word or phrase conditional on another being true, as in “if it is raining, open your umbrella”. The sense in computing is more formal but essentially the same; “if [this condition is true], then [do some action].

elif: Optionally continues a conditional by adding another BLOCK; if present it must follow either the initial if or another elif . Behaves exactly like an if , except that its conditional EXPRESSION will only be evaluated when no previous if /elif STATEMENT has evaluated as True .

Not a proper English word, but instead a portmanteau that contracts else and if into a single artificial word, elif. Together it means “otherwise, if” or “as an alternative, if”, both of which imply that the action controlled by the elif is contingent on the outcome of some previous test or tests. So, in computing, “ else [after checking some prior condition] if [this other condition is True], then [do some other action]“.

else: Optionally terminates a conditional by adding a final BLOCK; if present it must follow the last if /elif in the BLOCK. If no previous if /elif STATEMENT evaluated to True then the indented STATEMENT(s) below else will be run.; Can also be used to terminate blocks started with other KEYWORDs; see for , while , and try below.

An adverb from the Old English elles , meaning “[to do] instead of [some other action]” or “as an alternative”, or just “otherwise”. In computing it means “[check if any prior condition is true] else [perform some final action]“. It is used to take some default or fallback action when no better, more specific action should be taken.

Conditionals are key to a lot of Python programming, and are needed to better explain some of the keywords that follow, so I’ll provide a few examples of how they work.

Sometimes you only want to take any action if some condition is met; this is the simplest form:

if EXPRESSION:
    STATEMENT

Many situations are binary, though, and so you’ll always want to take some fallback action if the condition is not met:

if EXPRESSION:
    STATEMENT_A
else:
    STATEMENT_B

In complex cases you may need to have any number of alternative actions based on mutually exclusive conditions, as well as a fallback:

if EXPRESSION_A:
    STATEMENT_A
elif EXPRESSION_B:
    STATEMENT_B
else:
    STATEMENT_C

For the middle, “either/or” case there’s another form you will sometimes see, known as the ternary operator form. This is useful mainly because, unlike a standard if conditional, it behaves like an EXPRESSION, and the value it evaluates to can be directly assigned to a NAME:

NAME = STATEMENT_A if EXPRESSION else STATEMENT_B

Which is a much shorter way of writing:

if EXPRESSION:
    NAME = STATEMENT_A
else:
    NAME = STATEMENT_B

We’ll find this useful when we look at the OPERATOR-like keywords below.

Five That Are Operators

The next five keywords all behave like an OPERATOR to denote actions performed on OBJECTs and/or EXPRESSIONS.

Boolean logic operators

These are used for making meaningful comparisons between things based on their truth-value.

not 🗎: A unary OPERATOR that inverts the truth-value of whatever follows it, as in not EXPRESSION.; Can be used to invert the meaning of another KEYWORD; see is not and not in below.

A Middle English adverb from the Old English nawiht, implying “nothing” or “zero”. The general meaning today negates (or flips) the meaning of the word or phrase that follows it. Compare “I do have apples” to “I do not have apples”. In programming, however, the specific meaning comes from logical negation, and thus not negates true to false, and vice versa.

This is Python’s Boolean negation OPERATOR, used whenever you need the opposite of the truth-value of a thing. It is unary, which means that it acts on whatever is to its immediate right.

The usage of not is straightforward:

not EXPRESSION

If EXPRESSION is considered True then not EXPRESSION evaluates to False and otherwise it evaluates to True.

That can be easier to understand if you think of not as working like the following ternary if:

False if EXPRESSION else True

and 🗎: A binary OPERATOR that checks the truth-value of two things, evaluating to the thing on the left if it tested False , otherwise to the thing on the right.

An ancient Old English conjunction with Germanic roots vaguely meaning “thereupon” or “next [to]” and used to combine two words or phrases, as in “coffee and tea”. The meaning in Python, however, comes entirely from logical conjunction, and implies that either both things it combines are true or the whole combination is false.

This is Python’s Boolean conjunction OPERATOR; used whenever you need to test if both sides of the and are considered True. It is a short-circuiting operation; if the left-hand EXPRESSION is considered False the entire operation is considered False and the right-hand side will never be evaluated at all. And unlike not it does not necessarily evaluate to either True or False, evaluating instead to either the left side (if considered False) or the right side.

Thus the usage:

EXPRESSION_A and EXPRESSION_B

Can be thought of as working like the following ternary if:

EXPRESSION_B if EXPRESSION_A else EXPRESSION_A

Which is why you may one day find yourself surprised to find that True and 1 evaluates to 1 while 1 and True evaluates to True.

or 🗎: A binary OPERATOR that checks the truth-value of two things, evaluating to the thing on the left if it tested True , otherwise to the thing on the right.

Derived from the Old English conjunction oþþe , meaning “either”, and implying that either of the ideas conjoined are acceptable, as in “coffee or tea”. In Python the meaning comes from logical disjunction, and implies that either one of the things it combines is true or the whole combination is false.

This is Python’s Boolean disjunction OPERATOR; used whenever you need to test if either side of the or is considered True. It is a short-circuiting operation; if the left-hand EXPRESSION is considered True the entire operation is considered True and the right-hand side will never be evaluated at all. Also, unlike not it does not necessarily evaluate to either True or False, evaluating instead to either the left side (if considered True) or the right side.

Thus the usage:

EXPRESSION_A or EXPRESSION_B

Can be thought of as working like the following ternary if:

EXPRESSION_A if EXPRESSION_A else EXPRESSION_B

This subtlety can catch you out when you find that True or 1 evaluates to True but 1 or True evaluates to 1.

Identity checking operator(s)

is 🗎: A binary OPERATOR that tests if the OBJECT on the left has the same identity as the OBJECT on the right and then evaluates to either True or False .; Can be inverted by not to become the is not operator.

An Old English verb from the Germanic stem *es- ; it’s the third person singular present indicative form of the word be , so it generally means “to be [a thing]“. In Python its meaning is specific to identity, and implies something more like “to be [some unique thing]“.

The usage is straightforward:

EXPRESSION_A is EXPRESSION_B

And the usage with not is:

EXPRESSION_A is not EXPRESSION_B

The notion of identity is a little abstract, but think of it like this: Tom and Bob are twins, they’re the same height, age, and weight, and they share the same birthday, but they do not have the same identity. Tom is Tom, and Bob is Bob, but Tom is not Bob.

In most implementations of Python the identity of an OBJECT can be thought of as its unique address in memory, and since every OBJECT you work with will have its own unique address, is is usually only useful for telling if a NAME refers to a specific OBJECT:

NAME is OBJECT
NAME is not OBJECT

Or for testing if two different NAMEs refer to the same OBJECT in memory.

NAME is OTHER_NAME
NAME is not OTHER_NAME

There are however some special cases: for instance, True, False, and None are all singletons in memory, meaning there is only ever one copy of True in any Python program. For the most part this is just a space-saving detail you don’t need to worry about, but it explains why in Python we use VALUE == True and not VALUE is True when checking if something is considered equivalent to True. Testing identity is not the same as testing value.

For this reason the uses of is are limited and specific, which is why you’ll only rarely see is and is not used in practice.

Membership testing operator(s)

in 🗎: A binary OPERATOR that tests if the OBJECT on the left is a member of the COLLECTION on the right and then evaluates to either True or False . Also known as Python’s inclusion operator.; Can be inverted by not to become the not in exclusion operator.; Also used with for , see below.

A Middle English merger of the Old English words in , meaning “among”, and inne , meaning “within” or “inside”. The merged word has many usages and meanings, but the general sense here is from the prepositional form, which implies that some thing is contained within or inside some larger thing, as in “a page in a book” or “a book in a library”. In Python it is specifically used for membership testing, when checking if an item is contained within a group of items.

The usual usage of in is to test if an OBJECT is a member of a specific CONTAINER:

OBJECT in CONTAINER

Or to test if OBJECT is not a member of CONTAINER:

OBJECT not in CONTAINER

It can be tempting to think you can use is with in, but that’s invalid syntax. It helps to remember that since is, is not, in, and not in are all binary OPERATORs they must have either an OBJECT or EXPRESSION on either side, not another KEYWORD.

Four Used to Loop

The above keywords give you everything you need to perform simple decision making and to take basic actions, but they’re useless whenever you need to do something repeatedly; that’s where looping comes in. In Python the following four keywords give you everything you need to do that.

Starting a loop

There are two ways to start a loop in Python, and they’re conceptually pretty similar, but your choice of which depends a great deal on exactly what you need to do with that loop.

Looping until some condition is reached

while 🗎: Starts a loop BLOCK by testing a the truth-value of an EXPRESSION; will iterate continuously until EXPRESSION evaluates to False.

From the Old English word hwile for “a duration of time”, but here we’re using the conjunctive form which implies “[during the] time [that something is true]” or “for as long as [something is true]“. In programming languages the keyword is always associated with something to test and something to do, so the meaning becomes “ while [the test is true], [do something]“.

The form of a while loop BLOCK is always the same:

while EXPRESSION:
    STATEMENT

If the EXPRESSION evaluates as True then STATEMENT will be reached and executed. When the end of the indented BLOCK is reached, control returns immediately to the top, then EXPRESSION is tested again, and so on. So long as EXPRESSION evaluates as True the entire indented BLOCK will be run again and again.

The while loop can also optionally be terminated by an else BLOCK:

while EXPRESSION:
    [...]
else:
    STATEMENT

In this case the STATEMENT inside the else block will executed if the while loop runs all the way to completion (its test evaluates to False) without encountering break. This can be useful when there is some cleanup action that needs to occur when a while loop has exited naturally.

The while is the most basic form of loop, but it’s a little dangerous if not used with care. This is because any EXPRESSION that always evaluates as True will run forever. For this reason it’s generally important to design the EXPRESSION so that it will eventually evaluate to False.

Looping through the members of a COLLECTION

for 🗎: Starts a loop BLOCK that will iterate once over a COLLECTION, visiting every item in it.; Can also be marked with async , to start an async for loop, see below.

An Old English word via the German für with a great many meanings; the general meaning is taken from a prepositional sense of “[performing an action] on behalf of [some thing]“. In computing, though, the meaning is actually taken from the contraction of the word for with either every (meaning “each [item] in a group”) or each (meaning “all [of a group]“) to form for every or for each , both of which mean “[to perform an action] on behalf of each item [in a group]“. In programming languages that descend from ALGOL for has traditionally been the most common name for such a loop, with do used in a smaller number of languages. Python takes the name from the traditional usage in ALGOL via C, however it is more accurate to describe Python’s version as a foreach loop, because there is no explicit counter and the thing being looped over must be iterable.

The usage of for consistently involves assigning a user-assigned NAME to every OBJECT in a COLLECTION.

for NAME in COLLECTION:
    STATEMENT

Thus for every item in the COLLECTION the NAME will refer to that item inside the scope of the BLOCK; this allows the STATEMENT to use NAME to act on that thing.

This can be a little easier to grasp if you think of for as a specialized form of while loop:

while [items remain in COLLECTION to visit]:
    NAME = [increment to the next item]
    STATEMENT

But obviously the for loop is much simpler to work with, as you don’t need to worry about implementing the machinery necessary to track the start and stop conditions of the loop. You’ll find that you can, and usually should, use a for loop whenever you’re visiting the individual contents of a COLLECTION, rather than risk a runaway while loop.

The for loop can also optionally be terminated by an else BLOCK:

for NAME in COLLECTION:
    [...]
else:
    STATEMENT

In this case the STATEMENT inside the else block will executed if the for loop has run to completion without encountering break. This can be useful when there is some cleanup action that needs to occur when a for loop has exited naturally.

Controlling a loop

You don’t always want to simply run the loop to completion; there may be good reason to exit early, or to skip a round of the loop.

break 🗎: Used to immediately interrupt the current loop iteration, ending the BLOCK it is found within. For this reason must only be used within a loop BLOCK.

From the Old English word brecan , which has several forms and meanings. The noun form generally means “to damage, destroy, or render unusable”, as in “to break a leg”. Here, however, we use the alternative meaning “to interrupt [a continuous sequence]“, as in “to break an electrical circuit”. In programming it specifically means to interrupt a loop from inside that loop.

The break statement always forms a line of its own, and it must be used in either a for or while loop. The most common use is to stop a loop immediately if some particular condition is reached:

while True:
    if EXPRESSION:
        break
    STATEMENT

for NAME in COLLECTION:
    if EXPRESSION:
        break
    STATEMENT

This is particularly useful if there’s some situation that warrants stopping the loop before it would normally be completed.

continue 🗎: Immediately skips the rest of the current loop BLOCK, allowing the loop to continue on to the next iteration. For this reason must only be used within a loop BLOCK.

A Middle English verb borrowed via the Old French continuer from the Latin continuare. The general meaning used here is to “go forward or onward”, “carry on”, or “proceed”. In programming it means to cause a loop to start executing the next iteration, skipping any instructions that follow it.

The continue statement always forms a line of its own, and it must be used in either a for or while loop. The most common use is to skip to the next iteration of a loop immediately when some particular condition is reached:

while True:
    if EXPRESSION:
        continue
    STATEMENT

for NAME in COLLECTION:
    if EXPRESSION:
        continue
    STATEMENT

This is particularly useful if there’s some situation that warrants skipping the current loop; for example if you only wanted to act on every second iteration.

Three for Importing Other Things

All of the above, plus the builtin functions we’ll talk about in a later article, are sufficient to let you start using Python as a scripting language, where you glue together things others have written with your own code to do some task you want to accomplish. But you need to be able to access those “things others have written” to do so. That’s what Python’s import mechanism is for.

import 🗎: Used to bring the functionality of an external MODULE into your own code.

A verb from the Middle English importen , via the Old French from the Latin importare. The general meaning is to “bring/carry [goods into this country] from abroad”. In computing it means to bring or import some functionality exported by another program written in the same language into the current program.

The most common usage is very simple, the keyword is followed by the MODULE you want to access:

import MODULE

You can also import multiple MODULEs separated by commas:

import MODULE_A, MODULE_B

And for organizational purposes you can put parentheses around them as well:

import (MODULE_A, MODULE_B)

Any NAME inside the imported MODULE(s) can then be accessed within your own program using the dot access pattern in the form MODULE.NAME. So, for instance, if you wanted to get the circumference of a circle:

import math

radius = 3
circumference = math.tau * radius

Usually you’ll find all the import usage at the top of a MODULE, which makes it pretty easy to determine where such functionality comes from.

from: Modifies import to allow you to import specific NAMEs from within an external MODULE.; Can also be used with raise and yield , see below.

A Middle English word from the Old English fram , here we use the preposition form, with a general sense of “departure or movement away [from something]“; in computing we use a more specific sense of “taken from a source”.

It’s used to modify import to import a specific NAME from a MODULE, rather than the entire MODULE:

from MODULE import NAME

If you wish to import more than one NAME they can be separated by commas:

from MODULE import NAME_A, NAME_B

And for organizational purposes you can put parentheses around them as well:

from MODULE import (NAME_A, NAME_B)

In all cases you can then use the imported NAME directly, so for instance if you want the area of a circle:

from math import tau

radius = 3
area = tau * radius ** 2 / 2

The main reason for from is to remove the need to have many references to a MODULE peppered throughout your code, but it’s best reserved for when the MODULE has many NAMEs that it exports and you want to just use one or two.

as: Modifies import to create an alternative NAME (or alias) for an imported NAME.; Can also be used with except and with , see below.

From the Old English eallswā meaning “just so” or simply “all so”, which makes it a reduced form of also. The usage here comes from the adverb form meaning “[to act] in the manner or role [of some other thing]“. In Python it very specifically means “[from here on refer to this thing] as [this instead]“.

Because from exists there are two forms of usage:

import MODULE as MODULE_ALIAS
from MODULE import NAME as NAME_ALIAS

The point of as is to allow you to import some MODULE or NAME but refer to it by some other name. This is useful if the original name is particularly long, would conflict with one already in use in your code, or could simply use some extra information in context.

from math import pi as half_as_good_as_tau

Five for Exceptional Situations

Now that you’ve got the basics down, you’re getting into more complicated territory. What happens if you find yourself reaching a point in the code where you’re in an obvious error state and you don’t want to continue? This is where the notion of exception handling come into play. We’ll go into more detail on the standard Exceptions in later articles, but for now let’s just say that an EXCEPTION is a special type of VALUE that signifies a specific issue that has come up in your program. That issue might be an error, in which case you might want to crash out of the program, or it might be a signal that something unusual but expected has occurred. Either way you need to be able to emit those signals from your own code, as well as catch and react to such signals emitted by other code.

To signal that there’s a problem

raise 🗎: Used to raise a specified EXCEPTION, which will cause the program to stop immediately and exit if not handled by an except BLOCK.; If used without an argument inside an except or finally BLOCK, re-raises the EXCEPTION being handled by the BLOCK.

A Middle English word with many meanings, but in this case it comes from the verb form meaning “to lift upright, build, or construct” or “to make higher”. The meaning in computing is more specifically from a newer sense of “to mention [a question, issue, or argument] for discussion”, as in “to raise attention [to an issue]“. In several other programming languages throw is used with similar meaning.

The usage is usually going to be:

raise EXCEPTION

Quite rarely you might see the chained form:

raise EXCEPTION from OTHER_EXCEPTION

Which is used to indicate that the EXCEPTION being raised was caused by (or came from) some other EXCEPTION. This isn’t used often, but sometimes is useful when you attempt to handle an exception raised by other code but somehow cannot do so.

Lastly inside an except BLOCK you can simply:

raise

Which will re-raise whatever EXCEPTION is currently being handled inside that BLOCK.

To signal that a particular condition is not met

assert 🗎: Used to test if some EXPRESSION is considered True, and if not raise an AssertionError.

From the Latin assertus , with the general sense of “declared, protected, or claimed”. In programming it specifically means to specify that a condition must be met at a particular point in the code, and to error if it is not.

The usage of assert is usually going to be in the form:

assert EXPRESSION

Which will simply raise an AssertionError if EXPRESSION does not evaluate as True.

An alternative form allows you to specify a message:

assert EXPRESSION, "something is wrong!"

And this message will be incorporated into the AssertionError.

The assert statement exists for you to test the things that must be True for your program to continue to work (we call these your invariants). This can be very helpful when developing and debugging, but should not be relied on, at all, in production code, as the person running your program can elect to disable all assert statements by passing the -O command line option to Python. Basically you can imagine that assert works like:

if [the -O command line flag was not passed]:
    if not EXPRESSION:
        raise AssertionError

… except that if the -O flag was passed the assert statement will simply be replaced by pass and nothing whatsoever will happen.

To catch a signal and react to it

With both raise and assert in your toolkit you know how to signal an EXCEPTION, but how do you catch and react to (or ignore) one? This is where Python’s exception handling mechanism comes into play.

try 🗎: Starts an exception handler BLOCK; must be followed either an except BLOCK, an else block or a finally BLOCK in order to be valid syntax.

A verb borrowed from the Old French word trier , meaning to “test”, “experiment”, or “attempt to do”. The meaning in Python is essentially the same, “[start a] test [of something that may error]“.

except: Optionally continues an exception handler BLOCK by catching EXCEPTIONs; can (and should) be limited to specific types of EXCEPTION. More than one of these can follow a try and each will be checked in turn until either the EXCEPTION is handled or no more except statements remain.

A verb borrowed from the Middle French excepter and Latin exceptus ; the original meaning is “to receive”, but the more general uses it to “exclude [something]” or “object to [something]“, as in “every fruit except apples”. The meaning in Python is a little vague, but can be thought of as “catch, capture, or trap [an exception]“. In fact in many other languages catch is used for the same purpose; Python’s except is the exception to that rule.

finally: Optionally cleans up an exception handler BLOCK to provide a means of always performing some action whether or not the EXCEPTION was handled. Must appear after any except BLOCKS that are present, as well as the optional else BLOCK if that is also present. If no except BLOCK is present then finally must terminate the exception handler.

From the Middle English fynaly meaning “at the end or conclusion” or just “lastly”. It implies the very last thing to do in a sequence, which is the meaning here as well.

Exception handling is the first circumstance you’ve encountered in which one BLOCK must be followed by another, and the rules are somewhat more complicated than anything you’ve yet seen. For instance there are two different minimal syntactically valid forms of exception handler:

try:
    BLOCK
finally:
    STATEMENT

This form does not actually handle any EXCEPTION raised within BLOCK, it merely ensures that STATEMENT is run in all circumstances. Any EXCEPTION not handled will continue to “raise up” until it is either handled or crashes your program. This is useful for situations in which you want to do the same action (such as closing a file or database connection) whether or not there has been an error.

The second form does handle an EXCEPTION:

try:
    BLOCK
except:
    STATEMENT

Note : because this form will catch any EXCEPTION, including several that are used by Python to perform critical operations, you should, practically speaking, never use the above form. Instead use the minimum safe form:

try:
    BLOCK
except EXCEPTION:
    STATEMENT

This ensures that the exception handler only handles the EXCEPTION type specified. The hierarchy of built-in exceptions in Python is complex and touches on concepts we can’t cover in this article, but for now just know that except Exception is the least specific except statement you should ever use in production code.

That minimum safe form, however, isn’t particularly useful, because the STATEMENT can’t actually know what EXCEPTION it’s handling. Most of the time you’ll see something like this:

except EXCEPTION as NAME:
    STATEMENT

Which uses as to provide an alias NAME that can be used to inspect the actual EXCEPTION that was raised. This can be very helpful for seeing precisely what went wrong, which will help you decide if you can ignore the problem or need to respond in some way.

You can also specify multiple different types of EXCEPTION to catch, which can be helpful if you want to respond to any of those forms in the same way:

except (EXCEPTION_A, EXCEPTION_B, EXCEPTION_C) as NAME:
    STATEMENT

And if you want to respond to them each in a different way you can just stack except BLOCKS:

except EXCEPTION_A as NAME:
    STATEMENT_A
except EXCEPTION_B as NAME:
    STATEMENT_B

You can also optionally use else with an exception handler, so long as at least one except statement is used, and as long as the else is positioned before any finally:

try:
    BLOCK_A
except EXCEPTION as NAME:
    BLOCK_B
else:
    STATEMENT

However this is relatively rare, because the STATEMENT indented under else will only be executed when no EXCEPTION was raised inside the try BLOCK and no break, continue, or return statements were encountered within it.

As you’ve seen exception handling has some fairly dense syntax compared to the rest of Python. A fully fleshed out exception handler in a program that does something fairly complex, like interacting with a database, might involve all these parts.

try:
    [connect to database]
    [query the database]
except ConnectionError as error:
    [log the error]
else:
    [log success]
finally:
    [close the connection]

But, thankfully, it’s often not that complex, and you usually only have to deal with exception handling when something truly exceptional has happened.

Four for Writing Functions

Now that you’ve got all the structures you need to write an arbitrarily complex program, with the ability to make decisions, loop, and handle errors, your biggest problem is going to be organizing those structures into re-usable units. For instance you don’t want to type out a complex exception handler for every single time you connect to and query a database, as that would quickly lead to an unmanageable amount of repetition.

One of the key mantras of programming is Don’t Repeat Yourself (aka DRY ); the less boilerplate, the better, and so you want to be able to create subroutines, which are the most basic form of code re-use. In Python the most common form of subroutine is the function, which comes in two primary forms.

Anonymous (unnamed) functions

A lot of people teaching (and learning) Python tend to skip over anonymous functions, or treat them as an advanced feature, but that’s really just because they’re poorly named. The “anonymous” part just means the function isn’t given a specific name at creation time. That might not sound like the easiest thing to re-use – and you’re right – but it’s useful to understand them before we head on to named functions because they’re fundamentally simpler and more constrained.

Unfortunately the powers that be decided that anonymous functions, already saddled with a bad and confusing name, should definitely get a worse name in Python.

lambda 🗎: Used to define a CALLABLE anonymous function and its signature.

The 11th letter of the Classical Greek alphabet, λ ; it has no general meaning in English. In Python it is used because anonymous functions are a fundamental unit of Alonzo Church’s Lambda Calculus, which provides much of the mathematical underpinnings of modern computation. As an honor, that’s nice; in reality lambda would be used more often if it had a more fun name.

Despite the name a lambda is actually the conceptually simplest form of function in Python, because you can think of it as a just a way of creating a delayed-evaluation EXPRESSION that’s stated with a specific form:

lambda : EXPRESSION

Which evaluates to a CALLABLE that will evaluate the EXPRESSION that comes after the colon only when the CALLABLE is itself called. To call it you use the call syntax that is common to all Python CALLABLEs, however since the lambda is itself an EXPRESSION you need to surround it with parentheses to do that.

(lambda : EXPRESSION)()

And voila, everything you’ve just written will instantly be replaced by whatever the inner EXPRESSION evaluates to.

Thus all three of these are entirely identical:

x = (lambda : 2 + 2)()
x = 2 + 2
x = 4

But just delaying an EXPRESSION isn’t really the most useful tool. So the lambda introduces the idea of a function signature: you can add a NAME to the left side of the colon and the thing that NAME refers to will able to be used inside the inner EXPRESSION when it evaluates. This name is known as a parameter of the function.

Here’s a lambda with a single parameter:

lambda NAME: EXPRESSION

And here’s one with two parameters:

lambda NAME_A, NAME_B: EXPRESSION

And when we want to use the lambda we just call it, passing in a concrete VALUE for every parameter in the signature:

(lambda NAME_A, NAME_B: EXPRESSION)(VALUE_A, VALUE_B)

Which you can imagine can be useful if we want to calculate the area of a rectangle:

lambda width, height: width * height

But, again, we’re going to end up writing that a lot if we don’t assign it to a NAME:

area = lambda width, height: width * height
square = area(2, 2)
rectangle = area(3, 5)

Great! The square is now 4 and rectangle is 15; we’ve got the basis of code re-use!

But lambda is going to get pretty nasty when the EXPRESSION starts to get long. And beyond that there are some pretty significant limitations, since we actually cannot execute many kinds of STATEMENT within a lambda, much less an arbitrary BLOCK. They have their place, but maybe there’s a better and more flexible way?

Since we’ve just gone and assigned what is supposed to be an anonymous (which means “has no name”) function to a NAME, let’s look at how we should usually write functions in Python.

Named functions

The named function builds on the ideas of the lambda but take them to a much more flexible place, allowing you to re-use large chunks of code quite easily. They’re a little more subtle to master too, because they don’t evaluate exactly like an EXPRESSION. In fact by default they’ll always evaluate to None unless you explicitly tell them to do otherwise.

def 🗎: Used to define a named function and its signature, the indented BLOCK that follows can then be re-used by calling that NAME using the function() syntax.; If used inside a class defines a named method instead, which is called using the class.method() syntax.; Can also be marked with async , to start an async def , see below.

A contraction of the word define , which comes via the Middle English deffinen from Old French and Latin roots. It’s a verb that means “to specify or fix [the meaning of a word or phrase]“. In Python it is used specifically to create a named subroutine. In other languages define , fn , fun , func , function , and let are often used instead.

Because def can be used to create arbitrarily complex CALLABLEs, it’s going to require some explanation. Essentially it’s used to create a new NAME that refers to a CALLABLE. In fact, from now on let’s just use FUNCTION to mean exactly that:

def FUNCTION():
    STATEMENT

This defines a FUNCTION that can be called using FUNCTION(), which will then execute every STATEMENT inside the indented BLOCK. As you can see it’s conceptually equivalent to the “named” lambda form you saw earlier. Compare these two forms:

def FUNCTION(NAME_A, NAME_B): STATEMENT
FUNCTION = lambda NAME_A, NAME_B : STATEMENT

And you’ll see they’re very similar, except that the def version can run an arbitrary number of STATEMENTs².

These STATEMENTs will be executed within the local scope of the function, meaning that any NAME assigned inside the function cannot be directly seen or accessed by any code outside the function. In this example there are no such names, but there will be as soon as we change the signature of the function:

def FUNCTION(NAME):
    STATEMENT

And just as with lambda we can have multiple parameters:

def FUNCTION(NAME_A, NAME_B):
    STATEMENT

Which let’s us build the area function we made before with a lambda:

def area(width, height):
    width * height

square = area(2, 2)
rectangle = area(3, 5)

Except, wait a minute … didn’t I say just above that def functions evaluate to None when called? Yep, right now both square and rectangle are assigned to None. We’ve calculated two areas, but then discard them again as soon as the local scope of area is closed. Now, how the heck do we get the area out of area, anyway?

Stop the function and give back a value

return 🗎: Used to immediately give up control and end execution of the function at the point at which it is encountered. If followed by an EXPRESSION, that is evaluated first and the resulting OBJECT is given back to the caller of the function. If no EXPRESSION is present None is returned instead. Has no meaning outside a function, thus if present at all it must be inside a BLOCK that follows a def .

A Middle English verb from the Old French retourner , meaning “to turn back” or “to go back [to a former position]“. In computing it has two meanings:

The intransitive meaning is “to give back (or relinquish) control [to the calling procedure]“, as in “when the function exits it will return control”.

The transitive meaning is “to pass [some data] back to the calling procedure”, as in “this function will return the current time”.

In Python both meanings are combined, since a function will always return both control and data to the caller.

The most basic usage of return immediately ends the function and returns None to the caller:

return

This form is relatively rarely used; as mentioned before a function always evaluates to None by default; thus you can imagine that return is the implicit last line of every function.

Because of that it is much more common to see:

return EXPRESSION

Which immediately evaluates the EXPRESSION and returns the resulting OBJECT back to the caller.

You can also use return to pass back more than one OBJECT:

return EXPRESSION_A, EXPRESSION_B

This evaluates each EXPRESSION in turn, ends the function, and returns a tuple (a type of fixed-length COLLECTION) containing one OBJECT for each EXPRESSION.

This multiple form is also relatively rare, as it can be a bit of a surprise to the user of the code, and so requires a bit more effort in documentation, but it can be convenient for internal functions that you don’t intend to be used by others.

What you’ll notice is that returnterminates a function at the moment it’s evaluated. Anything below that point in the function essentially doesn’t exist (with one exception, the finally block inside an exception handler). So how would you handle situations in which you needed to give back data, but continue to work afterwards?

Pause the function and give back a value

yield 🗎: Used to immediately pause execution and temporarily give up control at the point at which it is encountered. If followed by an EXPRESSION, that is evaluated first and the resulting OBJECT is yielded back to the caller of the function; if no EXPRESSION is present None is yielded instead. Has no meaning outside a function, thus if present at all it must be used inside a BLOCK that follows a def .; Can be modified by from to form yield from , see below.

Etymologically the oddest word in this list; derives from the Middle English yielden and the Old English gieldan , both of which mean “to pay”, and share their root with the Old Norse gjald and the German geld , both of which mean “money”. Today geld means an ancient form of compelled tax or ransom, but it also means “to castrate”. Historically the Dangeld was a tax raised on the English by their king. This tax was raised to pay waves of Danish Vikings to, presumably, not castrate the English (or, at the very least, their king). None of this is directly important here, but might explain a little of why the meaning in computing derives from both “to give way and relinquish control”, as in “ yield to oncoming traffic”, and “to give back [a result or return on investment]” as in “the fund has a yield of 5% per year”. In both cases the implication is that the situation is not yet final, and is likely recurring: you yield , then you wait, then you yield again.

In Python it helps to remember an allegory: return is Death, and comes exactly once, while yield is Taxes, and may only end when Death arrives.

The usage of yield is similar to return:

yield

yield EXPRESSION

yield EXPRESSION_A, EXPRESSION_B

However any function that uses yield actually returns a special kind of COLLECTION known as a GENERATOR when called. Unlike a normal COLLECTION a GENERATOR does not hold all of its items in memory simultaneously, but instead “generates” each item as required by running through the function until yield is encountered and it gives forth an item. The GENERATOR pauses execution at that point, allowing the code using the GENERATOR to work with that item. When desired it can request the next item, at which point the GENERATOR continues running to the next yield, and so on until there are no more yield statements left to run or a return is encountered.

This is what allows the next form:

yield from GENERATOR

Which is a specialized use case that allows you to write a GENERATOR that will yield every item, in turn, from another GENERATOR.

A thorough explanation of GENERATORs is outside the scope of this series of articles, as they’re a fairly advanced topic. For now it’s sufficient to know that if you encounter yield you’re looking at a GENERATOR.

Three for Manipulating Namespaces

Now that you understand the basics of function, you’ll need to know about namespaces, as they become very important when you start building more complex programs.

At its simplest a namespace is essentially just the place where Python stores any NAME you create, so they’re used whenever Python needs to look up the OBJECT referred to by that NAME. Each namespace has a scope, which limits how long the namespace “lives” and whether or not any NAME within it is readable and/or writable by a given STATEMENT. Namespaces form a hierarchy, however, so the rules can be a bit tricky to remember.

In any given namespace a STATEMENT can:

both read and write any NAME defined in its own namespace
read, but not write, any NAME defined in an enclosing parent namespace
neither read nor write any NAME defined in a sibling or child namespace

At the top level is Python’s own namespace, which contains all the builtin NAMEs that Python provides for you. This namespace cannot be written to, and any NAME in it essentially lives forever.

Next comes the global namespace, which is the namespace of the MODULE you’re working in; it becomes the parent of all other namespaces created within that MODULE. Any NAME defined here usually lives for the duration your program is running.

Next, each CALLABLE you create gets its own unique local namespace when it is called; any NAME created here lives only as long as the CALLABLE is running, and when it finishes both the NAME and the OBJECT it refers to will be destroyed, unless either return or yield is used to pass it back to the calling scope.

Since you can define a CALLABLE that is nested inside another CALLABLE you can build a namespace hierarchy of arbitrary depth, so any given NAME might exist in any number of namespaces further and further removed from where you’re actually trying to use that NAME. This can get unwieldy to think about pretty quick, though, which is why we try to limit the amount of nesting we do in practical code.

The assignment of a new NAME is pretty straightforward: unless modified by one of the keywords below a NAME is always assigned in the scope in which the assignment STATEMENT occurs.

But the use of a NAME is a little less obvious: when used in a STATEMENT Python first searches for the NAME in the immediate local scope. If it cannot find the NAME then it searches the immediate parent (known as the nonlocal) scope, and then it keeps searching each successive parent scope until it either finds the NAME or cannot resolve it and errors.

All of this usually somewhat invisible machinery exists to allow you to write code using NAMEs that are appropriate and clear to the scope in which you’re working. So long as it isn’t a keyword and you don’t need to use the NAME from an ancestor’s scope you can use whatever NAME you want locally and not worry about it overwriting anything outside its own scope.

Most of the time you don’t want to fool around with that machinery, but every so often there’s a good reason to. Or a bad one that’s gotten you confused.

Write to the top from anywhere

In extremely rare circumstances you want to directly manipulate the global namespace in a way you wouldn’t normally be allowed.

global 🗎: Used to declare a NAME as part of the global namespace; the NAME cannot have been used previously in the same namespace. In effect this allows a local STATEMENT to both create and assign to a global NAME it otherwise could only read. Can be used within the global namespace, but has no effect.

An adjective borrowed from the French, with the general meaning of “worldwide” and “universal”. In computing the meaning is specifically “[a NAME that is] accessible by all parts of a program”. In Python it is both the top-most namespace, which is readable from within all functions and methods, but is also a keyword that binds a local NAME into the global namespace, allowing it to be writable from within that local namespace.

The usage is quite simple; the main restriction is that it cannot appear after the NAME has been used:

global NAME
NAME = EXPRESSION

You should be aware that the use of global is an immediate code smell, as its use is almost always an indication of bad coding habits, because it usually can – and almost certainly should – be replaced with argument passing and return values. There are data structures and dynamic code generation tasks that would be impossible to build without it, but the odds are if you’re thinking about using it, don’t.

Write to the parent from the child

Slightly more often there’s a need for the child to tell the parent what to do.

nonlocal 🗎: Used exclusively inside nested functions / methods (also known as closures) to bind a pre-existing NAME in the parent’s namespace. This allows a STATEMENT in the child to have write access to a NAME defined in its parent. Must appear before any reference to that NAME in the local scope.

An adjective formed by combining the prefix non- (“not”) and local (“pertaining to a particular place”). This is a primarily scientific and technical neologism (literally “new word”) that has no true general meaning, only specific meanings within specific fields. In programming it signifies a non-local variable, meaning “[a NAME that is] accessible in neither the local nor the global namespace”. In Python it very specifically applies to the namespace of the enclosing function from the point of view of a nested function, and is also a keyword that binds a local NAME within that nested function into the nonlocal namespace, allowing it to be writable from within the nested function.

Because it is only of use inside a nested function, usage will always involve some enclosing structure:

def OUTER_FUNCTION():
    NAME = VALUE
    def INNER_FUNCTION():
        nonlocal NAME
        NAME = EXPRESSION
    INNER_FUNCTION()

The key thing to understand is that nonlocal is used in a similar fashion to global, but with the more limited goal of making a variable in the parent of a nested function writable. It has similar caveats as well, though the fact that its impact isn’t felt by the entire program makes it inherently less dangerous. Still, its uses are very narrow and relatively few, so try not to abuse it when argument passing will do.

Kill it with fire

So far you’ve seen how to add a NAME to a namespace, but how do you remove a NAME from a namespace?

del 🗎: Used to delete a NAME (or NAMEs) from a namespace; if no other references exist to the OBJECT that NAME referred to, the underlying OBJECT is deleted as well.; Can also be used to delete an attribute of an OBJECT or a member (or members) of a COLLECTION if the specific type has allowed this operation.

A contraction of delete , which is in turn a verb derived from the Latin deletus. Its general meaning is to “destroy or eradicate”, “erase or smudge out”, and “utterly remove”, and while the specific meaning in computing is a little less violent it has similar effect. In Python the meaning is a little indirect: del is used to delete a reference that the user directly controls, which may indirectly trigger deletion of the thing referred to from process memory, which the interpreter controls.

The most common usage is to provide a single NAME:

del NAME

However it is also valid to provide multiple NAMEs, comma separated:

del NAME_A, NAME_B, NAME_C

In either case Python will proceed left to right, deleting each NAME from the namespace in which the del invocation has occurred.

Because the effect is limited to the closest namespace, you have to use global or nonlocal to delete a NAME outside the local namespace:

global NAME
del NAME

You can also use del to delete attributes of OBJECTs and member(s) of COLLECTIONs, however what actually happens depends on the type of the OBJECT / COLLECTION involved. The list COLLECTION, for instance, allows you to use del to delete both individual indices and slices of its contents:

test = [1, 2, 3]
del test[0] # deletes the first item
del test[:] # deletes all items

We’ll leave a further exploration of this for when we discuss the builtin types in later articles. In the meantime, let’s finally meet the keyword that will let you start defining such type-specific behaviors for yourself.

One for Defining New Types of Object

Up until this point I’ve been pretty vague about what an OBJECT actually is. In the Object-Oriented Programming paradigm, also known as OOP, an object is, basically, a thing that has both state (in the form of named attributes) and behavior (in the form of callable methods). Two individual things of the same type may have different specific values for those attributes – we call them different instances of the same type – but they share the same overall interface. However there’s a bit more to it than that; the programmer should be able to define partial interfaces, which we’ll call traits, that types with similar needs can implement in different ways, such that they all share some common attributes and behaviors (a property known as polymorphism). Additionally these traits should be able to be passed from more generic types to more specific types via an inheritance mechanism, much as a parent passes on traits to their children.

You can take an example from nature; a duck is a type of bird, and it inherits certain traits it shares will all birds. All birds are a type of animal, and thus inherit certain traits shared will all animals. An animal is a type of life, and so on. Thus Howard, a specific duck, has all the attributes and behaviors of life, animals, birds, and ducks, all rolled up nicely in one instance of the duck type.

In fact it is from just such an example of biological classification that Python takes the keyword it uses for defining new types of OBJECT.

class 🗎: Used to define a template for a new type of OBJECT.

A noun borrowed via the French classe from the Latin classis , which meant “a division, army, fleet”, “the people of Rome under arms”, and, oddly specifically, “any one of the six orders into which Servius Tullius divided the Roman people for the purpose of taxation”, which goes a long way towards explaining why English is such a difficult language to master. The general meaning we use here, though, is loosely borrowed from evolutionary taxonomy, implying “a group that shares certain inheritable traits”. The specific meaning in Python comes from Object-Oriented Programming and implies “a template for creating objects that share common inheritable state (attributes) and behavior (methods)”.

In Python you define a new type by creating a new NAME that is also a CALLABLE; from now on we’ll call that a CLASS. The class keyword is used to start a new BLOCK that defines the implementation of that CLASS:

class CLASS:
    pass

The CLASS can now be called to create a new instance of its type:

NAME = CLASS()

Now from the above you might expect that instance to have no attributes or behavior, but in fact it does have the minimal interface that all OBJECTs share (it can be printed and used in comparisons, for instance, though not very usefully). It receives this because all new class definitions implicitly inherit from object, which is Python’s base type.

You can also specify a more generic CLASS (also known as a superclass) to inherit from:

class CLASS(SUPERCLASS):
    pass

In fact you’ll often want to inherit from more than one superclass in order to mix together their various traits. This is done by separating the superclasses with commas:

class CLASS(SUPERCLASS_A, SUPERCLASS_B):
    pass

The notion of multiple inheritance can get quite complicated when more than one superclass defines the same attribute or method NAMEs, so for now let’s just keep our classes simple and assume no such overlap.

Just defining the CLASS and its superclasses without giving it some attributes and methods of its own is unusual, so you’ll normally see at minimum a custom initializer method, which by convention is done by defining the special double underscore (or “dunder”) instance method __init__ :

class CLASS:
    def __init__ (self, NAME):
        self.NAME = NAME

As you can see an instance method is just another kind of function definition made using def. In fact a METHOD is just a FUNCTION that’s been added to the CLASS’s namespace. That’s right, the class keyword also creates a new kind of local namespace, this one remains attached to the CLASS and allows names to be looked up on either the instance or the class itself using the dot access patterns INSTANCE.NAME and CLASS.NAME. Notice the self parameter name in the method definition above? That’s the name given by convention to the first parameter of any instance method, and it’s how you access attributes set on the instance itself.

This all sounds a little abstract, so let’s demonstrate it by building a CLASS for thinking about circles.

from math import tau

class Circle:
    def __init__ (self, radius):
        self.radius = radius

    def diameter(self):
        return self.radius * 2

    def circumference(self):
        return self.radius * tau

    def area(self):
        return tau * self.radius ** 2 / 2

Now if we create an instance representing the unit circle:

unit = Circle(1)

We can access and print the self.radius attribute:

print(unit.radius)

And we can also call any of its methods, but notice how we never have to pass the radius of the circle as an argument, since the methods can all access that via self :

print(unit.diameter())  
print(unit.circumference())
print(unit.area())

Since they’re the fundamental building block of everything in Python’s data model, there’s a lot than can be said about classes in Python, and there’s a lot of subtleties that can go into their proper use (and improper abuse), so we’ll come back to them often in later articles in this series. For now though the important thing is to start to recognize how class is used to create them, and how instances are used when accessing their attributes and methods. It’s also helpful to know that there are quite a few “dunder” methods, and that these are used to implement a lot of the common “under-the-hood” functionality that supports all the builtin functions we’ll start to meet in later articles.

One for Working Within a Context

Sometimes there are actions you always want to perform within a specific context, such as committing a database transaction or closing a network connection. These sort of things usually involve some form of exception handling or other boilerplate specific to the task, and that can lead to a lot of boilerplate that needs to be repeated. An example is something as simple as reading a file: opening the file must necessarily mean getting some resources from the underlying operating system, and you always want to free up those resources, even if you accidentally tried to open a file that didn’t exist or which you did not have permissions to read. The file is a type of OBJECT that will always carry that contextual need to manage a resource with it, and so you’ll need to write the same try and finally boilerplate every time you open a file.

Unless the language you’re working in provides a convenient means of ensuring that it happens for you.

with 🗎: Starts a context manager BLOCK, which ensures that the indented STATEMENT(s) below it are performed within the context of the OBJECT being managed.; Can also be marked with async , to start an async with , see below.

A Middle English preposition that takes its pronunciation from one Old English term, wið (“against”), but takes its modern meaning from another, mid (“in association with”) which in turn comes from the German mit. The meaning in Python derives from within , meaning “inside the scope or context of [some thing or event]“.

Any use of with requires an EXPRESSION that evaluates to an OBJECT that satisfies the context manager type’s interface, and thus has implemented both the __enter__ and __exit__ “dunder” methods. From now on we’ll call that EXPRESSION a CONTEXT_MANAGER, to make the examples clearer.

In the most basic usage you simply start a new BLOCK:

with CONTEXT_MANAGER:
    STATEMENT

Which will enter the context by calling CONTEXT_MANAGER. __enter__ (), execute the STATEMENT within the context, and then exit the context by calling CONTEXT_MANAGER. __exit__ ().

But most often you’ll want to use as to assign the OBJECT returned by CONTEXT_MANAGER. __enter__ () to an alias NAME so the BLOCK can work with it:

with CONTEXT_MANAGER as NAME:
    STATEMENT

This can save you a lot of boilerplate, especially with the many standard OBJECTs that already implement the context manager interface. For instance the recommended way to read a file’s contents:

with open(path) as src:
    contents = src.read()

Is much simpler than what you’d otherwise have to write:

src = open(path)
try:
    contents = src.read()
finally:
    src.close()

Occasionally you’ll want to work within more than one CONTEXT_MANAGER:

with CONTEXT_MANAGER_A as NAME_A, CONTEXT_MANAGER_B as NAME_B:
    STATEMENT

Which works exactly the same as nesting one with inside another:

with CONTEXT_MANAGER_A as NAME_A:
    with CONTEXT_MANAGER_B as NAME_B:
        STATEMENT

You’ll get to meet quite a few context managers as you work your way through the Python standard library. And now you know that building your own is just a matter of creating a new class and implementing a couple of “dunder” methods.

And, Finally… Two for Working Asynchronously

Any function or method you write with def alone is, by definition, synchronous, meaning that the moment you call it your running code has to stop everything else and wait, however long it takes, for your function to either return or yield control back to it before the rest of your code can be executed. For most tasks that happen on a local machine this is sensible and perfectly fine, especially when you need the answer before you can proceed any further.

In much modern programming though it’s often necessary to interact with things that don’t respond quickly, like network interactions with servers that could be a world away, and sometimes there’s plenty of work you could be doing while you wait for them to respond. You still need the answer, but you need it eventually. One way this can be addressed is via asynchronous programming, which as of Python 3.5 has become a first-class part of the Python language.

Asynchronous programming is an advanced topic that is fairly specialized, so using it is well outside the scope of this series. It’s also quite a new addition to the language, and its usage has fluctuated from version to version. Rather than try to summarize those changed I’ll just describe the very basics of using these keywords in Python 3.7 below. If, however, you’re curious there’s a quite good async primer available here that seems to be kept up to date.

Because asynchronous code does the same work as synchronous code, but schedules the execution of that work differently, only two new keywords needed to be added to the language.

async 🗎: Used to mark another KEYWORD as one that works asynchronously. As such, async cannot appear on its own.; With def as async def to define an asynchronous function, also known as a COROUTINE.; With for as async for to loop over an asynchronous iterator inside an async def .; With with as async with to use an asynchronous context manager inside an async def , see below.

As you may have guessed, this is a contraction of the Modern English word asynchronous , which is formed by combining the Latin roots a- (“not”) and syn- (“together”) with Khronus , the Ancient Greek personification of time. It unambiguously means “not occurring at the same time”. In Python it more specifically marks an operation as “not occurring in the same time as the caller”, which allows the caller to wait for the result of that operation, which will occur at some point in the future.

Because the meaning of async is tied to the KEYWORD it marks, its usage is always the same:

async KEYWORD

However, unlike other any other keyword we’ve seen, the usage of async will always begin with the definition of a new COROUTINE:

async def COROUTINE():
    STATEMENT

Both of the other forms of async exist to allow you to work within a COROUTINE with other COROUTINEs, and thus they can only exist inside an async def.

For instance you can use async for to loop over an asynchronous iterator such as a GENERATOR COROUTINE (which is simply a COROUTINE that uses yield instead of return).

async def COROUTINE():
    async for item in GENERATOR_COROUTINE:
        STATEMENT

You can also use async with to perform work within the context of an asynchronous context manager:

async def COROUTINE():
    async with CONTEXT_MANAGER_COROUTINE as NAME:
        STATEMENT

Virtually every COROUTINE will need to wait on other COROUTINE(s), which is why there’s another keyword that can only be used within an async def.

await 🗎: Used to suspend the execution of the COROUTINE it is found within and waits for the COROUTINE to its right to complete; can only be used inside an async def .

From the Middle English verb awaiten (“to wait for”) from the Old French awaitier / agaitier (“to lie in wait for, watch, or observe”). The general sense is more active and hostile than wait , which it’s a modification of. In asynchronous programming it means to “[suspend execution] and wait [for something to finish]“.

The await keyword is always going to be used to call a COROUTINE inside an async def:

async def OUTER_COROUTINE():
    await COROUTINE()

Which will pause the execution of the outer COROUTINE and wait until the called COROUTINE, eventually, returns control. Any VALUE returned by the called COROUTINE is passed through await, so you can think of await COROUTINE() as an asynchronous EXPRESSION that will, eventually evaluate to whatever the COROUTINE returns when called.

Of course now that you’ve got the basics of doing asynchronous work down, how do you actually perform such work? Well, as of Python 3.7 that still requires using functionality provided by the asyncio module, the details of which are well outside the scope of these articles. See the async primer I mentioned earlier for details.

Whew…

There you have them, Python’s 35 keywords: in and of themselves not enough to make you fluent in Python, but if you’ve read this far and digested it you’re well on your way to truly understanding (and not merely using) the skeleton of what is going on in one of the fastest growing general programming languages around. In the next post in this series we’ll take a step away from words and look instead at symbols, with a dive into Python’s slightly smaller, but thankfully much simpler, list of operators.

There are 35 keywords as of Python 3.7, the current major version of the language as of the time of writing. New keywords are added quite rarely, and it’s even more rare for keywords to be removed, but in whatever version you’re on you can use from keyword import kwlist; print(kwlist) to view the current list. ↩
Putting def on a single line makes the equivalence with lambda more obvious, but for the sake of readability don’t do this very often. ↩

Look Before You Leap Year

Michael Morehouse — Wed, 12 Jun 2019 00:00:00 +0000

Leap Years are Stupid

Actually, leap years are absurd, and consequently extremely stupid. From the day some long-dead forbear of astronomers¹ and astrologers² peered up at both the Sun and Moon in the same sky, put two and two together, and realized that they were in the same place as the last time it was this bloody cold³, we’ve been stuck with them, whether we realized it or not. In a sad quest to pretend that an ellipse⁴ is, nevertheless, a circle, we doomed ourselves to forever having to maybe offset our mental clocks when looking sufficiently far forward or back in time, lest history and reality drift dangerously out of sync.

In the modern West the leap year is usually attributed to Julius Caesar, who so succesfully promoted the concept that it only took the Romans fifty years after his death⁵ to reliably incorporate it into what history now tries to forget as the Julian calendar. Unfortunately Caesar and his scholars – being pretty poor programmers – had made a wee bit of a rounding error, and so in 1582 Pope Gregory XIII was forced to “correct” the situation⁶ by shifting the average year from 365.25 to 365.2425 days. And so was born the Gregorian calendar we all know and use today⁷. This approximation was good enough for the Church, and so history has been positively littered with February 29th every so often ever since.

I feel contempt for the leap year. So why am I writing about it?

It’s an Ideal Programming Exercise

It turns out that determining if a particular Gregorian calendar year⁸ is, or is not, a leap year is a problem with some unusual properties that make it an excellent test of a programmer’s mastery of some of the most fundamental tools they have available to them. That, in turn, means that it’s not only a very good challenge for the budding programmer, but it could – and I’d argue should – also be used as an impressively diagnostic question to ask a candidate in a technical interview.

In every programming language that I’ve yet encountered there exists a short, simple, and performance-optimal solution for this problem. Indeed in most cases the optimum implementation is a single line long⁹, but most who attempt it will end up with something quite far from elegant on the first try. And, unlike most problems with such an ideal answer, if solved inelegantly it’s also almost certainly solved inefficiently.

The beauty of the exercise is that the pathway to the ideal solution can easily be discovered just by employing the sort of straightforward reasoning that is the bread and butter of professional programming. It encourages you to think through your solution by first understanding what you’re trying to solve.

Why does all this matter?

From the masochist to the sadist: "Hurt me."

From the sadist to the masochist: "No."

Clive Barker, for one

Well maybe you’ve just joined Exercism. Or perhaps you’re just looking for a good code kata. Or you’ve been assigned to a team working with calendars. Maybe you’re simply a masochist. Or worse, perhaps your interviewer is a sadist.

One way or another, let’s assume you’ve just been assigned the task of implementing a function is_leap that takes an integer argument year and returns a Boolean result.

Luckily There’s an Algorithm for That

Every year that is exactly divisible by 4 is a leap year, except for years that are exactly divisible by 100, but these centurial years are leap years if they are exactly divisible by 400.

United States Naval Observatory

Unusually for a programming problem the basic algorithm will most likely be presented to you up front, and you need merely implement it. The algorithm is very old, and has been stated in more or less precise ways over the centuries, so it helps to transcribe it into pseudocode:

IF the year is a multiple of 4 THEN
    IF the year is a multiple of 100 THEN
        IF the year is a multiple of 400 THEN
            RETURN True
        RETURN False
    RETURN True
RETURN False

Notice the chevron-like > shape that results from the multiple layers of nesting? It turns out that’s fairly typical of a first-pass mental model of the algorithm, at least in a language where indentation is meaningful. It’s not the most elegant form – we’ll get to that – but it conveys some important properties about our algorithm:

It’s simple : You have one argument, you perform (up to) three tests on it using a basic math operation, and it’s the same operation for all three tests. The most complicated thing you need to know to solve it is how your language performs the modulo operation. Once you’ve got that, comparison with zero, simple conditions, and how to return a Boolean-equivalent value in your language of choice you’re golden. After five minutes of well-focussed instruction a novice to any high-level language should be able to bash out a general solution that mimics the above pseudocode.
It’s progressive : After each successive test you either know the answer or you know what to ask next. Most algorithms – indeed most non-trivial problems – aren’t nearly so cut and dried. There is no need for complicated branching, or your language’s version of else_if, you need merely proceed forward until the moment you are certain you know the answer.
It’s robust : There are no bugs, no weird or exceptional edge cases; every possible (integer) input maps directly to a single, unwavering output. Additionally there are no side effects, no state to mutate… you don’t even need to carry forward the results of each modulo test. It’s as pure as a function can get, and so constrained that you can know from glancing at it if it will always return the correct answer.

Know Thine Enemy and Thine Self

Understanding the algorithm is great, but understanding the problem it’s addressing is better, and the insights gained above tell us nothing about the domain we’re working with. We’ve treated the various modulo tests as black boxes, but really they’re specialized filters that apply to specific, progressively smaller, subsets of the overall domain of Gregorian years.

So what can we glean from looking a little more closely at those tests?

The answer is usually no : From the very first test we know that 3 out of every 4 years won’t be a leap year. By looking at the rest we can see that an additional 3 out of every 4 centuries won’t be either… returning False by default will correctly answer >75% of all possible questions without any additional effort.
They rapidly cull the herd : Just 1 in every 4 years is even a potential leap year, just 1 year in every 100 is a century, and just 1 year in 400 is a century that’s also a leap year… we learn a lot fast about our year.
They have a natural order : Each test is a special case of the previous one, and from the above we know that each applies to a progressively smaller number of cases. Sure, if we test division by 400 first we solve the problem very quickly in an extremely small number of cases, but if it doesn’t give us the answer it also doesn’t give us any useful knowledge about the question being asked. Proceed in the natural order, however, and each step refines the last.

Taking the first of these to heart we can rewrite our pseudocode so that we need only ever consider the paths to a True answer:

IF the year is a multiple of 4 THEN
    IF the year is NOT a multiple of 100 THEN
        RETURN True
    IF the year is a multiple of 400 THEN
        RETURN True
RETURN False

But this is interesting. Glancing at the above we can now see that there are exactly two criteria that result in a year being a leap year, and that those criteria are exclusive. So for any given year the answer is True if either of the following is True:

the year is a multiple of 4, but not of 100
the year is a multiple of 4, 100, and 400

The Path of the Righteous

From either of the above the path to the ideal form can pretty neatly unroll in front of you. Especially if we introduce some abstraction and call our three tests a , b , and c.

For instance, if we progressively introduce Boolean logic operations to our pseudocode:

IF a THEN
    IF NOT b OR c THEN
        RETURN True
RETURN False

… becomes…

IF a AND (IF NOT b OR c) THEN
    RETURN True
RETURN False

… which, in a language that allows you to return an expressions becomes:

RETURN a AND (NOT b OR c)

Oooh, that’s pretty concise.

Or We Could Go The Other Way

And if we’d started from the criteria we derived alone?

IF a AND NOT b THEN
    RETURN True
IF a AND b AND c THEN
    RETURN True
RETURN False

… can be combined with another Boolean logic operator as:

IF (a AND NOT b) OR (a AND b AND c) THEN
    RETURN True
RETURN False

… which allows us to extract the common operation:

IF a AND ((NOT b) OR (b AND c)) THEN
    RETURN True
RETURN False

… but there’s no reason to perform the same Boolean test twice, so:

IF a AND (NOT b OR c) THEN
    RETURN True
RETURN False

… and since we’re returning expressions brings us right back to:

RETURN a AND (NOT b OR c)

All Roads Lead Here

And here lies the ultimate, optimal form. In a language that has short-circuiting Boolean operations this is also the most performant form, because neither side of the or will be evaluated for the vast majority of cases. Thankfully most languages either do short-circuit Boolean operations, or they tend to provide an alternate short-circuiting operator like and_also and or_else, so if you’re not sure yours does you’ll need to do a bit of research.

Speaking of research, you might be tempted to remove those parentheses, which certainly does look nicer. But therein lies the final lesson is_leap can teach you; your ability to elide those parentheses depends entirely on the evaluation strategy and operator precedence rules for your language. If your language evaluates left to right and binds or at the same or lower precedence as and, then no, you cannot remove those significant parentheses. At least not without paying the price of checking division by 400 for a year that doesn’t even divide by 4.

Which, like leap years, would be stupid.

Those who look up at the night sky and find it full of stuff. ↩
Those who look up at the night sky and find it full of nonsense. ↩
They’ve never actually occupied the same absolute positions twice, of course, and never will, but neither angular momentum or orbital mechanics were yet available to our prototypical yutz. ↩
Actually something closer to an ellipse being frantically sketched over and over on an imaginary two dimensional plane being pulled away from the artist in four dimensional spacetime. ↩
Caesar’s assassination, which famously happened on March 15th, 44 BCE, rather threw off the adoption of leap years. Things got a bit wonky until 8 CE and, since no one in this period could be relied on to write anything down consistently, we know exactly when he died, just not when in our calendar. ↩
Gregory XIII may have been liturgically infallible, but the fact that we’ve since had to introduce leap seconds tends to imply he wasn’t mathematically infallible as well. ↩
Unless you use the Hindu, Islamic, or Buddhist calendars, are Thai or Chinese, subscribe to the Juche ideology of North Korea, or measure everything in Unix time. ↩
Specifically, for the truly pedantic, the proleptic Gregorian calendar with astronomical year numbering and a year zero. Which is fine for the purposes of an exercise. To accurately reflect leap years actually observed by humans you’d need at minimum a quite convoluted lookup table before the 9th century CE, and probably a time machine. ↩
Your mileage may vary; in Perl it may even be less, but no one will know because of the braces. If you’re writing in 8086 Assembly Language then, honestly, why are you even reading this? ↩

Hugo Probably Isn't For You

Michael Morehouse — Wed, 05 Jun 2019 00:00:00 +0000

Yes, It’s Quick Enough to Break Your Neck

As it says on the proverbial box, the Hugo static site generator is fast. Written in Go, Hugo promises blistering speed that it certainly delivers, and I’ve got all the redness, pain, and burning discomfort needed to prove it.

As I’m writing this – in Markdown, using Vim – all I have to do is save the file and the hugo --server process I’ve left running in the background will render to HTML and update my browser window by the time I can shift my eyes over to it. Near instant feedback, even if I’ve made significant changes to multiple posts. Hugo is quick, no question, and once you’re up and running your site will do nothing but benefit from that speed, especially when you’ve built up a significant catalogue of content.

But…

Prepare ship for Ludicrous Speed!

Colonel Sanders

There’s something to be said for the idea that fast isn’t necessarily efficient, and in the hands of the ill-prepared it’s more often than not downright lethal. To get to the point where you could be reading this post I needed to do vastly more than is implied by the quickstart tutorial, and to even do that I had to learn vastly more than I’d originally been prepared for. For months the only thing Hugo actually delivered to me at blistering speed was a blistering headache. To be fair that wasn’t entirely Hugo’s fault – I made several false starts based on my aspirations more than my established needs – but I let my desire for speed and some exciting marketing copy trip me up right out of the gate. Which means I spent most of those months on my face, or often worse. All Hugo really did to help me during that time was fail blisteringly fast; it often did things that hindered me.

Frankly it’s been an agonizing journey. Pray you do not repeat it. Read on, and perhaps you won’t.

A Bit of Background

I am an experienced programmer, but I am emphatically not a web programmer. Before finally deciding to build out my personal site as a home for my technical musings I’d only ever blogged with WYSIWYG tools like Tumblr, and then only sporadically. Professionally I mainly write backend and process automation code for the Visual Effects industry, so it’s charitable to say that I’ve developed the UX/UI design chops of your average wild-born gibbon. The last serious experience I had with HTML/CSS predated the Marvel Cinematic Universe, and let’s just say there have been some plot developments since then.

So I don’t have a lot of experience in web development, but I’m also not seeking to gain that experience. I’ve also got no large back catalogue of existing content that I need to port over, so I’m basically a greenfield blogger who wants to get up and running relatively quickly. I just need to make a choice about how I’m going to do that.

I wasn’t naive about that choice; I knew that any choice I made would involve a learning curve, I just wanted it to be slightly under the slope of, say, picking up Dwarf Fortress, so I tried to approach the choice of tooling critically.

Step 1: Establish Key Criteria

After a fair amount of offline research I’d eventually narrowed down exactly what I thought I wanted out of my Minimum Viable Website:

A static site I can host on the free tier of a CDN-backed service like Github Pages or Netlify
A responsive, mobile-first theme with syntax highlighting and readable typography
Modern markup practices; JSON-LD Structured Data, OpenGraph, and semantic HTML5
Reasonable privacy practices; I live in the EU, so I must make a good-faith effort at GPDR compliance
Easy social sharing to Facebook, LinkedIn, and Twitter, but without unnecessary tracking
Some form of commenting on posts, so I can hear from the readers I sincerely hope will arrive
Minimal ability to sign up for a mailing list; a lot of people left RSS after Google Reader died
Fast load time; I’ve been badly bitten by Tumblr; Google’s AMP pages seemed ideal
A decent nod to accessibility; I’m getting older and I’ve got friends that are visually impaired

Now, those nine criteria might seem like a lot – and, in hindsight, maybe they were – but really they can be boiled down to deliverable , discoverable , and readable , and since my visual needs are pretty bare-bones they didn’t seem like too much of an ask.

Of course I’d quicky learn that some of these had natural, even insurmountable conflicts – AMP would eventually be abandoned because it renders commenting and mailing list signup unfeasible, for instance, and social sharing and privacy don’t exactly go hand in hand – but I had hope.

Step 2: Assess the Field

Dynamic tools like Wordpress were obviously eliminated by my first criterion, so I set out to understand the options for static site generators, of which there are many. Though I came across Hugo early in my research I really didn’t want to go in full bore on anything without reading up on the competition, so I set it aside for the moment.

I tried to narrow the field by eliminating generators written in what were, to me, exotic languages. As I’m extremely comfortable with Python – far more so than I am, or ever intend to be, with Go – I started by investigating Pelican, Nikola, and Hyde¹. In each case it quickly became apparent that these weren’t the right choice for me, as none seemed either very deep or in very wide use, and I wanted something with relatively broad adoption and a community to help me through the learning curve. No options in my number one language, then, but really how often was I going to need – or want – to monkey around with the generator’s internals, anyway? If the tool worked I shouldn’t need to, I reasoned, though the ability to comprehend how it worked would be nice. Plus, at the end of the day, my itch for speed stems in part from many years waiting on Python, so the siren song of fast lured me a little further out of my comfort zone.

Of course I also checked out Jekyll, the Ruby-written, Github Pages first class member; an extremely solid alternative that’s polished enough to be on its third major release, with a community large enough to demand some level of stability, and which certainly seemed friendly enough. But I’m not at all interested in Ruby, so lingering fear of having to get under the hood at least once, coupled with the – by all accounts quite obvious – speed differential, tipped the balance towards Hugo. Besides, Jeykll felt long in the tooth, with some long unaddressed issues and theme showcases bloated by what certainly looked to me like a large amount of swine and only a few pearls. Even its own website seemed dated and rather dull, which added to the perception that it wasn’t keeping up with the times.

Step 3: Embrace Confirmation Bias

By contrast there was an obvious, churning buzz around Hugo among tech bloggers I admire, a solid amount of recent commit history, and some definitely cool sounding features like Hugo Pipes. As a neophyte to static, self-managed blogging I really hoped for something that might save me from the confusing tide of tooling that seems to inundate the world of web development², so Pipes looked like a big shiny win.

And, glancing through the theme browser, there were a high proportion of options that looked good – if not ideal – for the minimalist tech blog I was obviously going to finally have up and running any day now. I quickly found themes that met most of my functionality criteria, and others that seemed to implement some of what was missing. So I might need to figure out how to transplant some elements, Frankenstein-like, from one theme into another, but the templating syntax looked simple enough, there seemed to be a lot of documentation, and there were obviously active forums with lots of meat to chew through.

So, after some more comparison on drearier topics like CI deployment and hosting options, the decision was made. Hugo had been an early contender, and now it was Hugo for the win! I was confident; I’m a greenfield blogger, no big – or for that matter any – back catalogue to worry about importing… all I’ve got to do is pick a theme, start writing my first post, and deploy! I’ll be up in days! Maybe even hours!

What could go wrong?

It Doesn’t Exactly Work Out of the Box

You never get a second chance to make a first impression.

Head & Shoulders

Maybe it was bad luck, maybe it was the winds of change, most likely it was an errant commit on the Ananke theme – who knows? – but on the fateful day I first did the quickstart I rapidly and tenaciously made it all the way to the moment of truth, breathlessly ran hugo server -D and then, aching with anticipation, opened a browser window and pointed it at the localhost address conventiently displayed in the terminal. Where I found absolutely bupkiss being served. Nada. A blank screen with an empty source file. Back in the terminal where I’d launched the command there were several warnings, but nothing that could obviously explain the error³.

This was disappointing. But at the end of the day I had zero interest in using Ananke, so I decided to try the rest of the quickstart with one of the more me-appropriate themes I’d already bookmarked.

… it didn’t work. Neither did the next.

Neither did the following three.

The Agony of Teething

Hugo is an open source project; as of right now its most recent release is 0.55. Unlike Jekyll it hasn’t yet reached a 1.0 release, and thus has made no promises about backwards compatability. And that’s fine. Each of the themes is similarly an open source effort, typically – though not always – with a single developer who is very possibly the sole invested consumer as well. And so things have broken, badly. Which maybe is to be expected, in a project this immature.

But for each and every one of these themes the “official” hosted demo of the theme appears to work fine, which gives a Hugo-newbie like me the impression that there are many more out-of-the-box functional themes than there actually are.

So what gives? Well it turns out it’s because the hosted showcase uses the exampleSite directory embedded in each submitted theme, something not even mentioned in the quickstart, and very often the templates in the theme depend on configuration that simply won’t be present in your quickstart site.

Eventually, after much reading of old forum posts and tripping through a lot of documentation well past its best before date, I discovered the following usually gives you a local theme demonstration that behaves roughly as the author intended:

mv config.toml config.toml.bak
cp -r themes/YOUR_THEME/exampleSite/* .
find . -maxdepth 1 -type f \( -name "config.toml" -or -name "config.yaml" -or -name "config.json" \) -print -quit | grep -q "." || cp config.toml.bak config.toml
cp themes/YOUR_THEME/archetypes/*.* ./archetypes/

The first line backs up the default config.toml file before the contents of the theme’s exampleSite directory are copied into the root. We need to move config.toml because that file, if present, will take precedence over config.yaml, which will in turn take precedence over config.json. That’s right, Hugo has had three configuration file formats over its short life, and all three remain viable options, though for some reason TOML has the edge. Since any of them may have been present in the theme, we’ll only move the backup back into place if none of them were.

Similarly the final line overwrites the archetypes/default.md file with any and all files in the theme’s archetypes directory; this is because currently Hugo will always assume that a default.md in the root archetypes directory should override any archetype from the theme, no matter how specialized it may be. The end result is that the hugo new posts/my-first-post.md command mentioned in the quickstart won’t include theme-critical elements in the post’s front matter, quite often leading to errors or missing content.

And as I said the above fixes will only usually give you a working example of the theme you’ve chosen. The Ananke theme itself, for instance, will need further steps:

sed -i 's/gohugo-theme-ananke/ananke/' config.toml
sed -i 's/^themesDir/#themesDir/' config.toml

These are required because the quickstart instructions had you rename the theme directory as you made the submodule, and the themesDir configuration key was set apparently as a hack to make the theme easier to test.

These sorts of issues mostly derive from the various themes having been written at various stages in Hugo’s rapid evolution, and many of them have committed to old choices that no longer remain the best practice, or are now pending deprecation⁴. One phylogentic tree of themes requires content to be in content/post instead of content/posts, another will fall down if it’s not the other way around.

These are, though, the relatively minor issues; take a dew days, pore over the forums and the documentation, make a few key configuration changes, possibly clear out a few of the cobwebs in a partial or two, and you’re good, right?

Failure is a Recursing Theme

This isn't life in the fast lane, it's life in the oncoming traffic.

Terry Pratchett

If I’ve learned anything on this saga it’s that themes for static blogs tend to be fairly and surprisingly complex beasts, and that human beings can make them far more complex than that. As a rule you should assume, going in, that whoever wrote the theme you’ve chosen only understood – or at least only encountered – a fraction of the potential issues. As just one example of an issue repeatedly encountered, a small and seemingly innocuous change to a configuration value can suddenly turn Hugo’s vaunted speed into an equally impressive amount of thumb twiddling, as you wait for the server to time out and spit out a complaint about excess recursion. Unfortunately this is one of a family of errors that leaves you with no clear idea where the problem child might be, or even what file was being processed.

My encounters with these recursion issues weren’t extremely common, but frustrating enough that I tried to figure out what was going on; they seem to be related to the usage of the Scratch area, a sort of variously site / page / template localized “scratchpad” that the Hugo team bolted on to overcome an apparent deficiency in the Go template language⁵. The Scratch facility seems to have been added to attend to a narrow problem, but it was apparently adopted enthusiastically – and perhaps too widely and, at times, inappropriately – by theme authors. Unfortunately they’ve done that off of sparse documentation on what is an obviously powerful, but seemingly dangerous, feature, and more importantly a feature you, the theme consumer, can’t know is in use without knowing your theme’s various templates both deeply and intimately.

Debugging these sort of issue is hard, and if you’re not a reasonably seasoned developer you won’t have a hope of doing so. And, beyond Scratch, the Go templating language is impressively flexible and powerful, allowing you to embed significantly more complex logic into templates than I’ve seen in other such template DSLs. But this brings with it a consequent and potentially explosive increase in the complexity and number of possible errors. Since there’s – as of yet – no meaningful or obvious way to test a template, you can imagine that the reliability of the various themes in the showcase is extremely variable.

Lessons Hard Won

None of the above is to say that there aren’t themes out there that work, and work well, out of the box, just that I never found them. However I also got nowhere near testing them all, and it’s entirely possible that I just chose un-wisely, so your mileage may vary. And of course your needs may simply not overlap with mine; if you need a single page landing site or a product launch page there are slick looking, seemingly professional offerings that may work perfectly with only the most minimal configuration.

But if you are like me, with relatively low experience in web programming and a strong desire to start producing content quickly, and with as much out-of-the-box support as conceivably possible, then be forewarned that the implicit promise of getting up and running fast with Hugo is quite a hollow one.

If you’re not interested in – or capable of – monkeying around with theme internals, then Hugo is almost certainly not the right choice for you, at least not yet. If you are interested and capable, then realize that even if you don’t end up rewriting every line of your chosen theme – and there’s a very good chance you will want to – then you’re definitely going to invest a significant amount of time browsing, installing, shake-down testing, and rejecting themes in the showcase. And ultimately you’re going to find that you will be required to understand what is happening internally to a degree that simply isn’t immediately true with other more mature, stable, and well documented projects in this space.

And if you don’t intend to be especially prolific – or if you haven’t already got a substantial back catalogue of posts – then there’s a very substantial chance that the dividends paid by Hugo’s extremely impressive rendering speed simply won’t ever repay that investment, much less bring you real returns on it.

That said if you do have the temerity to start from scratch – as I eventually ended up doing with my own theme – then pat yourself on the back when those frustrating months of effort are done, because that first post – this one – feels good, even if it took a very long time to see it finally rendered.

Hyde’s own site looking different when served over http vs https didn’t inspire confidence. ↩
I’m staring balefully at you npm, grunt, yarn, webpack, yack and pork, the last two of which I’m not even sure I just made up. ↩
As of the day I’m writing this running through the quickstart instructions does in fact work, though with a great many deprecation warnings. You’ll end up with a site that looks nothing like the theme demo page, though; even the gargoylish top splash image doesn’t load. ↩
Hence the warnings you’ll see in the quickstart with Ananke, even with the fixes describe above. ↩
Variables could be assigned in templates with the := assignment operator, but not updated. This appears to have been quietly fixed at some point (there is now a = update operator), reducing or possibly eliminating the need for the Scratch space, at least within a single template’s scope. ↩