DEV Community: Derek D.

Nobody Likes a DRY PASTRY

Derek D. — Mon, 29 Mar 2021 15:36:45 +0000

Welcome to another one of my vendetta articles. Last time it was A Pythonic Guide to SOLID Design Principles proving the SOLID Design Principles are indeed Pythonic. This time I'm taking aim at the most abused clean code principle Don't Repeat Yourself, or DRY, as it is commonly abbreviated. It's a pretty simple principle to follow. All you have to do to is not repeat yourself, meaning don't write the same code twice. It's even got catchy phrases like, "Can you DRY this code up", or "This code could be DRYer" which I often see in code review. I don't disagree with the DRY principle. I do disagree with how it is used as an excuse for clean code as if just because there is no duplicate code the code is also clean. That's why I like to say nobody likes a DRY PASTRY. Let me say that again nobody likes a DRY PASTRY. I had to repeat myself because PASTRY stands for Please Always Stop To Repeat Yourself.

History

Let's start with the origin of the DRY Principle. It comes from the Pragmatic Programmer written by Andrew Hunt, and David Thomas published in 1999. In the article the DRY Principle is defined as

"Every piece of knowledge must have a single, unambiguous, authoritative representation within a system."

Ironically over the years the DRY Principle has been the subject of many technical articles. Each echoing the definition found in The Pragmatic Programmer, and then proceeding to interpret what that definition means through code snippets. So what has happened is we've repeated the very principle that say's Don't Repeat Yourself! Since most of dev's now are being introduced to DRY by reading someone's interpretation of it instead of reading the authoritative definition from the Pragmatic Programmer the understanding of DRY has been corrupted. The DRY principle has been reduced to mean duplicate code is bad, don't write duplicate code or you are bad and as this popular image for the DRY Principle shows... "Repetition is the root of all software evil".

In lies my disdain for how DRY is being used. Repeating code is not bad and I can prove it.

Why Repeated Code is Good

I like the analogy of baking to help explain why repeated code is a good thing. Take for instance chocolate chip cookies. You need wet ingredients like eggs, butter, and oil. They are necessary for the chemical reactions to take place that make cookies, soft, chewy, and delicious. Have you ever tried making cookies without the eggs, butter or oil? They are not very good. The moisture is not just a good thing it's vital for a quality cookie. The same is true for code, and just as the cookies dry out as they are developed our code should dry out as it is developed. When you write code multiple times and repeat yourself you are introducing your wet ingredients, when the code is functional you go back to refactor. While refactoring you'll see the actual use cases of your code allowing you to create DRY abstractions that avoid some of the gotcha's you couldn't have anticipated if you started with an abstraction. That being said here are some practical tips to help you to avoid ending up with a DRY PASTRY.

Code for your use case not your reuse case
Premature abstraction is the root of all fragile software
Duplication is cheaper than the wrong abstraction
Code is not reusable. Abstractions are.

Code for your Use Case not your Reuse Case

This is a quote from a senior developer I worked with at my first job out of college. He would always tell me, "code for your use case and not your reuse case". I didn't understand what it meant until I experienced DRY being done wrong. In essence what he was trying to say here was don't try and anticipate use cases that are not a current requirement. Anticipating use cases is often what happens when developers are trying to DRY up their code. They see similar code being written and try to "abstract" it by putting it all in a class or a function. I put "abstract" in quotes there because these function and classes often are more of a distraction than they are an abstraction. Consider this example where a developer has implemented two functions, post and put, that use the same code to get query string parameters, and body from an HTTP request.

import json


def post(request):
    params = {}
    for param in request['query_string'].split('&'):
        field, value = param.split('=')
        params[field] = value

    body = json.loads(request['body'])
    ...


def put(request):
    params = {}
    for param in request['query_string'].split('&'):
        field, value = param.split('=')
        params[field] = value

    body = json.loads(request['body'])
    ...

to DRY up this code they might create a function to abstract the logic of parsing HTTP requests.

import json

def parse_request(request):
    params = {}
    for param in request.query.split('&'):
        field, value = param.split('=')
        params[field] = value

    body = json.loads(request.body)
    return params, body

def post(request):
    params, body = parse_request(request)
    ...


def put(request):
    params, body = parse_request(request)
    ...

That looks pretty good, but there is a problem. What happens when you want to use parse_request() for an HTTP Get Request? You can't because Get requests don't use the body which means a None value would be passed to json.loads() causing it to blow up. Just because code is DRY doesn't mean it's reusable. What would have been more reusable is a get_query_params function and a get_body function. The only way the developer would have seen this is by duplicating the code for getting query string parameters and request bodies in the post, put, and get functions, then going back to refactor and creating abstractions for the actual patterns in the code and not what they anticipated the pattern to be. It's worth noting most IDE's have refactoring features that will find common patterns and abstract them for you. Very few IDE's have features that try to predict how code will be used though. An IDE's refactoring features would likely have resulted in two functions, one for getting the query string parameters and another for getting the request body because those were repeated patterns in the code. The get_query_params and get_body functions are good abstractions because they are decoupled from one another, and have a single responsibility making them SOLID as well. They also happen to be DRY code. The point is when you try to predict use cases you create premature abstractions which lead to bloated classes, functions and abstractions.

Premature Abstraction is the Root of all Fragile Software

This brings me to my second point. Premature abstractions are the root of all fragile software. I say this because premature abstractions put you at risk of coding wrong assumptions. Wrong assumptions lead to bad abstractions, bad abstractions lead to hacks, and too many hacks lead to fragile not easily maintainable code. Trying to always parse the request body was a wrong assumption, because not every HTTP request has a body. In this case there's a simple solution, but that's not always the case, and the more features stuffed into an abstraction the more hacks will be needed to support all of its use cases. This creates a knot in your code, especially when the bad abstraction is used a lot. When fixing a highly used bad abstraction you have to change a lot of calling code. Not only is that tedious it violates the open/closed principle from SOLID, and increases the risk of introducing a bug. This is why I say duplicate code is far cheaper than the wrong abstraction.

Duplication is cheaper than the wrong abstraction

This could also be phrased as duplication is easier to correct than the wrong abstractions. Removing duplication is like picking up toys and putting them in a toy box, while fixing the wrong abstraction as I hinted at earlier is like untying a knot. That's because duplicate code follows a pattern and the abstraction to fix it will also follow a pattern so the fix is calling the function or class everywhere the duplicate code was and passing in the appropriate variables when needed. You saw this previously when the parse_request() function was created. Untying the knot is usually more difficult though because you need to identify all the use cases and where they need to be handled. You then need to find and remove any hacks that were put in place to work with the bad abstraction. Here is an example of untying the knot that would have been created if the developer forced the abstraction to handle the special case of get requests.

Here is the code with a try/except block around the json.loads() call which is the hack.

import json


def parse_request(request):
    params = {}
    for param in request.query.split('&'):
        field, value = param.split('=')
        params[field] = value

    try:
        body = json.loads(request.body)
    except:
        body = {}

    return params, body


def post(request):
    parse_request(request)
    ...


def put(request):
    parse_request(request)
    ...


def get(request):
    parse_request(request)
    ...

Here is the code after untying the knot.

import json


def get_query_params(request):
    params = {}
    for param in request.query.split('&'):
        field, value = param.split('=')
        params[field] = value

    return params


def get_body(request):
    return json.loads(request.body)


def post(request):
    params = get_query_params(request)
    body = get_body(request)
    ...


def put(request):
    params = get_query_params(request)
    body = get_body(request)
    ...

This example wasn't too bad, but it still required

creating a new function
renaming and old one
removing the try/except block

After all that you still had to make sure the functions are called in the correct places which was more work than DRYing up the duplicate code which was basically copy/paste. The longer the bad abstraction lives on the more use cases it accumulates and the more places it gets called from. Every time the bad abstraction is touched or used the knot tighter and harder to fix.

Code is not Reusable, Abstractions are

It all comes down to this, code is NOT reusable, abstractions are. The definition of DRY even say's as much. "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system". Meaning there should be one abstraction for each piece of knowledge. So one abstraction that has the knowledge on how to parse query string parameters, one abstraction with the knowledge to parse a request body, one abstract representation of a user and so on and so forth. Maintaining one abstraction per piece of knowledge keeps code SOLID, DRY and it all starts when you repeat yourself, so Please Always Stop To Repeat Yourself.

Finding the Skeleton Buried In The Code

Derek D. — Wed, 10 Mar 2021 03:33:48 +0000

So, that title is total clickbait and I apologize for that. Reading it you think this is going to be a story of how I uncovered some nasty hacks buried deep within a legacy code base that changed the very fabric of reality itself. That's not what this article is about though. This article is about seeing through language specific syntax and discovering the fundamental skeletal structure of programming constructs like conditionals, loops, and classes.

My initial thoughts for this article came to me early in 2020 while I was writing Go vs Python a Technical Deep Dive. I figured if I wanted any authority on the topic I should be able to write some Go. So I set aside a weekend to go (pun intended) through the official Tour of Go. At the end of the weekend I didn't just know the features, and data types of Go I could write syntactically correct Go from memory. This made me wonder why did my first programming language, Java, take 6 months to learn but Go only took a weekend? My theory is there is a fundamental skeletal structure in the core building blocks of every programming language, and if you can understand those you can teach yourself and programming language in literally a weekend. It's kind of like knowing Latin. When you know what every other language was derived from those languages become easier to learn.

Without further gilding, the lily here is an in-depth examination of programming's core constructs that cuts away language specific syntax and shows the skeletal structure of programming. These are the constructs that will be examined in this article.

Code Comments
Output
Variables
Conditionals (if, else if, else)
Loops (for, while)
Functions
Classes/Objects
imports

Code Comments

I always start with code comments so I can use them to explain code in later sections.

Code comments are important because it's how you leave notes, explanations, and documentation in source code for yourself and other developers. A code comment always starts with some special character(s). In most languages, a comment starts with //, and everything on that line following // is ignored by the compiler/interpreter. Some languages, like Python use # to start a code comment. Regardless of the character(s) used to start a comment every language has them and they are incredibly useful. Here's an example in Python, Javascript and Go.

Python

# This is a comment.

Javascript & Go

// This is a comment in both javascript and go

There are also multi-line comments. Most languages start a multi-line comment with /** and end them with **/ very similar to opening and closing tags in HTML. Here are examples of multi-line comments in Python, Javascript, and Go.

Python

"""
Python is weird and uses the triple quote for mutli-line comments.
Note there is a triple quote used to close the comment.
"""

Javascript & Go

/**
 * This is a multi-line comment in Javascript and Go
 * Note each line starts with a star but it is not needed as long as the
 * comment exists between the opening and closing tags
 **/

Outputs

Outputs are ultimately what benefit the user. Without output, a program could solve the most difficult problems in the world but the operator would have no idea of what the solution was. Output can be something printed to console, a file written to disk, or technically event something printed on paper. There is always one built into the language though and that's printing to console.

The first program many developers write is Hello Word. Hello World is a simple program that prints the string of characters "Hello World!" to console. Every programming language has some built-in function that handles this for you, all you have to do is pass the string of characters you want to be printed. In Python that function is quite literally print(), Javascript has console.log() while Go blends the 2 with fmt.Println(). Here is Hello World in each of these languages.

Python

print('Hello World!')

Javascript

console.log('Hello World!')

fmt.Println("Hello World!")

In these examples Python's print() is a function while Javascript's log and Go's Println are methods on an object. Objects and methods will be discussed in more depth later.

Variables

Variables are the first building block that can also be a stumbling block. Comments and Outputs are very static. It's easy to understand what they do. Variables change though, it's why they are called variables. There are only 2 things you can do with a variable; assign it a value and reference that value.

Here's an example of assigning a value to a variable.

Python

name = 'Derek'

Javascript

var name = 'Derek'   // A variable accessible throughout the entire program
let name = 'Derek'   // A variable accessible only within the current scope
const name = 'Derek' // A variable whose values can't change known as a constant

var name str = "Derek" // Creates a variable of type str with the value Derek
var name = "Derek"     // Creates a variable that derives the type str from its value Derek
name := "Derek"        // Shorthand for creating a variable that derives its type from its value

Here you see why Python is so popular, there is only one way to assign variables. You don't have to worry about data types or if the variable is accessible globally or not. At this point, it's sufficient to understand an assignment statement has exactly 3 parts. The variable, the operator, and the value. This is still true for statements with expressions in them like a = 2 + 2. For these kinds of statements, the expression is evaluated and the result gets assigned to the variable. The expressions that appear on the right-hand side of the operator can get incredibly complex but at the end of the day, the entire expression is evaluated into a single value before the assignment occurs.

Conditionals (if, else if, else)

Conditionals are how your program makes decisions. Conditionals have 2 parts; the condition statement that evaluates to true or false, and the condition body which only executes when the condition statement evaluates to true. Conditionals can be chained using if/else if/else condition statements.

If conditionals are the most common conditional and start every chain of conditionals.

Here's an example of an if statement.

Python

count = 1
if count > 0:
    print('count is positive')

Javascript

const count = 1
if (count > 0) {
    console.log('count is positive')
}

count := 1
if (count > 0) {
    fmt.Println("count is positive")
}

In these examples if count > 0: and if (count > 0) are the condition statement. and the print statements, print('count is positive') etc. are the condition body.

The key syntax differences are Python doesn't surround the expression in () while Javascript and Go do, and Python uses a : to indicate the end of the condition statement and the start of the condition body, while Javascript and Go just wrap the condition body in {}. What's known as the conditional expression count > 0 is the same in all 3 languages though.

To continue a conditional chain an else if conditional can be used after the body of a preceding conditional. The else if conditional will only be evaluated if the condition statement that comes before it evaluated to false.

Here are the previous examples now with else if statements.

Python

count = 10
if count > 5:
    print('count is positive')
elif count < 0:
    print('count is negative')

Javascript

const count = 1
if (count > 0) {
    console.log('count is positive')
} else if (count < 0 ) {
    console.log('count is negative')
}

count := 1
if (count > 0) {
    fmt.Println("count is positive")
} else if (count < 0) {
    fmt.Println("count is negative")
}

The key difference here is Python uses the elif keyword instead of else if. elif doesn't do anything special it's just shorthand for else if.

Else conditionals are used to end conditional chains. An else conditional does not have a conditional statement since it only executes when every preceding if and else if condition statement in the chain evaluates to false. This is why else statements are commonly used to define default behaviors in a program.

Here are the examples with else statements.

Python

count = 10
if count > 5:
    print('count is positive')
elif count < 0:
    print('count is negative')
else:
    print('count is zero')

Javascript

const count = 1
if (count > 0) {
    console.log('count is positive')
} else if (count < 0 ) {
    console.log('count is negative')
} else {
    console.log('count is zero')
}

count := 1
if (count > 0) {
    fmt.Println("count is positive")
} else if (count < 0) {
    fmt.Println("count is negative")
} else {
    fmt.Println("count is zero")
}

Loops

Loops are where the languages start to diverge a little bit, mostly because there are types of loops available in some languages and not others, however, those loops that are available in multiple languages generally have similar syntax.

Loops are used to loop over and execute the same code until a condition is true. The main loops are for and while loops which we will start with.

The anatomy of while loops are similar to conditionals. There is a condition statement that evaluates to true or false and immediately following that is the loop body which is executed every time the loops condition statement evaluates to true.

Here is an example of a while loop that will search for the letter a in a jumbled up alphabet. Note Go only has for loops so there is no code snippet for while loops in Go.

python

needle = 'a'
haystack = 'rsthidlmjpqknefgabczowxyuv'
curr = ''
index = 0
while curr != needle:
    curr = haystack[index]
    index += 1

print('found needle at index', index)

Javascript

needle = 'a'
haystack = 'rsthidlmjpqknefgabczowxyuv'
var curr = ''
index = 0
while (curr != needle) {
    curr = haystack[index]
    index += 1
}

console.log('found needle at index', index)

The same differences from conditionals apply here as well. Python separates the loop condition statement with : while Javascript wraps its condition statement in () and the body in {}.

For loops are tend to be more verbose than while loops and can come in 2 forms. The classic for loop and a for each/for in loop.

A classic for loop has 2 parts to it the for statement and the loop body. Subsequently, the for statement is made up of 3 parts, which I call the initializer, the condition, and the step. The initializer sets up a variable used to keep track of the number of
iterations, the condition determines if the loop body should be executed again, and the step which moves the variable created in the initializer to its next step.

Here is an example of a classic for loop in Javascript and Go since Python only has for in loops.

Javascript

count = 0
for (var i = 0; i < 10; i += 1) {
    count += 1
}

console.log('count is', count)

count := 0
for i := 0; i < 10; i += 1 {
    count += 1
}

fmt.Println("count is", count)

In these examples var i = 0 and i := 0 are the initializer, i < 10 is the condition, and i += 1 is the step. Everything between the {} is the loop body. Both of these programs will print out the value 10.

For each and for in loops are used to iterate over every element in a collection. They have 2 parts as well the for in/for each statement and the loop body. Just like a classic for loop for each/for in statements can be broken up into parts as well. These parts are the iteration variable and the collection. The collection is just as it sounds. It's a collection of values that will be iterated over. The iteration variable is a variable that holds the current value being looked at in the collection.

Python

numbers = [1, -1, 10, -5, 8, 9, -4]
positives = 0
negatives = 0
for n in numbers:
    if n >= 0:
        positives += 1
    else:
        negatives += 1

print('Negatives', negatives, 'positives', positives)

Javascript

const numbers = [1, -1, 10, -5, 8, 9, -4]
let positives = 0
let negatives = 0
for (const n in numbers) {
    if (n >= 0) {
        positives += 1
    } else {
        negatives += 1
    }
}

numbers := []int{1, -1, 10, -5, 8, 9, -4}
positives := 0
negatives := 0
for _, n := range numbers {
    if n >= 0 {
        positives += 1
    } else {
        negatives += 1
    }
}

fmt.Println("Negatives", negatives, "Positives", positives)

In these examples, there is some magic performed by the language in that each iteration of the loop the next value in the collection is stored in the iteration variable. In this case n is the iteration variable and [1, -1, 10, -5, 8, 9, -4] is the collection. This is usually a mind-bending concept the first time you see it so for those of you just getting started. on the first iteration of the loop n is equal to 1 and the second iteration n is equal to -1.

Functions

Functions are very similar to algebraic functions. They receive input(s) and produce an output. Consider the following algebraic function that doubles the value of x.

f(x) = x * 2
f(2) = 2 * 2
f(10) = 10 * 2

f(x) could be called the function declaration (i.e. what the function does) f(2) calls the function with the input 2, which results in the output 4 f(10) calls the function with the input 10, which results in the output 20

This mathematical concept of doubling an input can be expressed in Python like so.

def double(x):
    return x * 2

A function has 2 parts, its signature, and its body. The signature is made up of the function name and the parameters that the function accepts. In some languages, the data types of the parameters and the data type returned by the function are also part of the signature. Here are the declarations for the double function in Javascript and Go since you've already seen it in Python.

Javascript

function double(x) {
    return x * 2
}

// Note this is the same thing just using ES6 arrow functions
double = x => x * 2

func double(x int) int {
    return x * 2
}

def double(x), function double(x) and func double(x int) int are the signature, while return x * 2 is the body. Go is one such language where the data type of the parameters and return value are used in the signature. That's why you see x int for the parameter and int following the closing parens ) for the return type.

The functions are called the same way in all the languages.

Python

double(2) # returns 4
double(10) # returns 20

Javascript

double(2) // returns 4
double(10) // returns 20

double(2) // returns 4
double(10) // returns 20

Classes and Objects

Classes are used to model real-world objects/concepts. Some languages, like Go, have things that are class like while some others, mostly functional programming languages, don't have classes and objects at all.

My favorite example to explain classes is a table. That's right like your run of the mill coffee or dining room table. Classes are complex so I can't say, "here is the format all classes use" as I could for conditionals, loops, and functions. Using tables as an example, however, makes it simple to understand what components make up a class. There are 3 basic components to every class. The constructor, properties, and methods.

The properties are fields, properties, or attributes of the thing you are modeling. In our case, this is a table and those attributes might be height, length, width, height, and legs representing the number of legs a table has.

The constructor is a special method on the class that builds an instance of the class which is an object. Here is an example of a Table class that defines the height, width, length, and legs properties and a constructor for constructing instances of the class. Remember instances of the class are objects.

Python

class Table:
    def __init__(self, height, width, length, legs):
        self.height = height
        self.width = width
        self.length = length
        self.legs = legs

Javascript

class Table {
    constructor(height, width, length, legs) {
        this.height = height
        this.width = width
        this.length = length
        this.legs = legs
    }
}

type Table struct {
    Height int
    Width int
    Length int
    Legs int
}

In these examples, Python and Javascript have explicit constructor methods while Go has an implicit constructor. In Python, the constructor is __init__ and in Javascript constructor. These methods take in parameters that are used to set the properties for the instance of the class being constructed. This is seen with self.height = height in Python and this.height = height in Javascript. Go does the same thing implicitly which is why no constructor needs to be defined in Go. Here is an example of creating 2 tables, myTable and yourTable.

Python

myTable = Table(3, 4, 8, 4)
yourTable = Table(3, 4, 8, 4)

Javascript

myTable = new Table(3, 4, 8, 4)
yourTable = new Table(3, 4, 8, 4)

myTable := Table(3, 4, 8, 4)
yourTable := Table(3, 4, 8, 4)

I like this example because it's easy to see that mytable and yourTable are not the same table even though they have the same height, width, length, and the number of legs. This is an important concept for Classes and objects since they help to model the real world. After all, when I bought my dining room table I didn't buy every instance of that table just because they had the same properties as mine. More importantly, though tables with different properties can be represented using the same Table class. Here's an example of creating a coffee table and dining room table.

Python

coffeeTable = Table(2, 2, 4, 4)
diningRoomTable = Table(3, 4, 8, 4)

Javascript

coffeeTable = new Table(2, 2, 4, 4)
diningRoomTable = new Table(3, 4, 8, 4)

coffeeTable := Table(2, 2, 4, 4)
diningRoomTable := Table(3, 4, 8, 4)

In this case, it's obvious that coffeeTable isn't diningRoomTable because they don't have the same values. An important take away though is regardless of the table represented a table in its general abstract sense can be described as having a height, width, length, and number of legs no matter what those values are. The object's properties can be accessed like so.

Python

print(coffeeTable.height)

Javascript

console.log(coffeeTable.height)

fmt.Println(coffeeTable.Height)

The final components to a class are methods which we briefly talked about with constructors. A method is just a function bound to a class meaning the function cannot be called without referring to the class. Calling a method syntactically is a hybrid between calling a function (i.e. foo()) and accessing an object property (i.e. myTable.height). Here's the code for the rather contrived example of folding tables.

Python

class Table:
    def __init__(self, height, width, length legs):
        self.height = height
        self.width = width
        self.length = length
        self.legs = legs

    def fold(self):
        self.length = self.length / 2

    def unfold(self):
        self.length = self.length * 2

Javascript

class Table {
    constructor(height, width, length legs) {
        this.height = height
        this.width = width
        this.length = length
        this.legs = legs
    }

    function fold() {
        this.length = this.length / 2
    }

    function unfold() {
        this.length = this.length * 2
    }
}

type Table struct {
    Height int
    Width int
    Length int
    Legs int
}

func (t Table) fold() {
    t.length = t.length / 2
}

func (t Table) unfold() {
    t.length = t.length * 2
}

There are 2 main takeaways from these examples. The first is that methods are defined the same way as functions. Functions and methods both use the def keyword in Python, function in Javascript, and func in Go. The difference is they are bound to the class. This is shown very explicitly in Go with the (t Table) in the signature of fold and unfold functions. It quite literally binds those functions to that struct. Python's use of self as the first parameter is a little less explicit but does the same thing. In Javascript, it's completely implicit but it's still happening and you can see that by the use of this. The second takeaway is that self in Python, this in Javascript, and t in Go are references to the object that the method was called from. So for example, if I had 2 variables myTable and yourTable representing 2 coffee tables and fold is called on myTable only the length for myTable will change. Here's the code demonstrating this.

Python

myTable = Table(4, 4, 2, "Wood", 4)
yourTable = Table(4, 4, 2, "Wood", 4)
myTable.fold()
print(myTable.length) # prints 1
print(yourTable.length) # prints 2

Javascript

myTable = Table(4, 4, 2, "Wood", 4)
yourTable = Table(4, 4, 2, "Wood", 4)
myTable.fold()
print(myTable.length) // prints 1
print(yourTable.length) // prints 2

myTable := Table{4, 4, 2, "Wood", 4}
yourTable := Table{4, 4, 2, "Wood", 4}
myTable.fold()
print(myTable.Length) // prints 1
print(yourTable.Length) // prints 2

Imports

Imports come in 2 varieties, full and partial. A full import is the simplest and most common import bringing in an entire module of code. They look like this.

Python

import datetime

Javascript

import moment

import "time"

Sometimes you only want a specific variable, function or class from a module and that's when you use a partial import. Go doesn't support partial imports but they look like this in Javascript and Python.

Python

from datetime import timedelta

Javascript

import { timedelta } from moment

Conclusion

This is by no means a comprehensive guide to learning programming. What is hopefully has done though is provided a mental model of programming languages themselves that you can use to teach yourself a new language and/or decipher code written in any language. As always I want to hear from you and if you enjoyed the article I encourage you to follow me on Twitter @d3r3kdrumm0nd and subscribe to the Namespace Podcast that I co-host.

A Test Driven Approach to Python Packaging

Derek D. — Mon, 15 Jun 2020 04:02:55 +0000

Recently I reached a breaking point with importing local packages in python. I consistently found myself dealing with ImportError or ModuleNotFoundError's when starting on a new project. A lot of it is my fault for never sitting down and properly understanding Pythons importing mechanism until now, but a little of it has to do with a plethora of answers, and advice online that tell you how to fix a problem, but not why the fix works. I hope to bridge that gap today.

There are a few key questions I want to answer about this topic that I think will clear up most of the confusion.

Is an __init__.py file needed in every folder?
Is a package you import the same thing as the package you install from PyPI?
Does Pytest and Python's builtin unittest module import packages differently?

The simple answers to these questions are

Yes if using Python 3.2 or earlier, and No if using Python 3.3 or newer.
No, they are different even though they represent the same code.
Yes

Historical Perspective

Although Python 2 reached end of life this year, it's influence on Python devs has been profound. Even when I started writing Python several years ago every project I worked with only supported Python 2, and in Python 2 an __init__.py file was required in every directory, otherwise it wasn't importable. You can test this for yourself. Create a directory strucutre as shown below, open up a Python REPL and try to import greeter.lang. It doesn't work until you add an __init__.py to the lang folder.

experiment/
|_greeter
|  |_ __init__.py
|  |_lang
|  |  |_en.py
|  |  |_en.py
|  |
|  |_greeting.py
|
|_tests
|  |_test_greeting.py

This behaviour was present even in the early days of Python 3. PEP-420 finally removed that requirement in Python 3.3, but the trend of adding __init__.py files has been continued by long time Python developers because they never knew to change their habits and the habit has been passed down to newer devs either from these older devs or from historical answers on Q&A sites like Stack Overflow.

If you don't believe me that __init__.py files are no longer needed in every directory you can read it for yourself in the Python 3 docs here or in this screenshot of the paragraph that confirms it.

I have some examples of common scenarios you'll find yourself in that I want to share with you, but before that, I figure we should define what a package actually is.

Paraphrasing from the setuptools documentation (setuptools is the Python lib for building and distributing PyPI packages),

"A package in the context of PyPI is a distribution of bundled > software. A package in the context of Python is a container of > modules".

Simply put a module is just a Python source code file.

Additionally there are 2 types of python packages now, "Regular Packages" which adhere to the old Python 3.2 and earlier requirements of a package (i.e. __init__.py in every directory) and "Namespace Packages" that follow the requirements from PEP-420, which bring a whole new meaning to being 420 friendly.

With that in mind let's get down to the examples. Since most of my confusion came while I was following Test Driven Development all of these examples use pytest and Python's builtin unittest module to run a test module that imports some other python module/package of varying complexity.

Example 1

The simplest case is where the test is in the same module as the code it is testing. The directory structure would look like this.

example-1/
 |_ greeter.py

Both pytest greeter.py and python -m unittest greeter.py pass. Since there is nothing to import this should be expected. If you want to see the source code for this example it's available on GitHub.

Example 2

A slightly more complex, but more common case is when there is a test module separate from the module being testing. The directory structure would look like this.

example-2/
 |_greeter.py
 |_test_greeter.py

In this case pytest test_greeter.py and python -m unittest test_greeter.py once again pass even without an __init__.py file. The source code for this example is also available on GitHub.

More complex projects need better organization than a single source code module and a single test module. This is where python packages come into play. Examples 3, 4 and 5 all deal with the various ways a package can be structured (i.e. nesting the test module with the source code package or keeping it separate from the source package).

Example 3

This example covers the case where the test module is nested within the source code package. The directory structure looks like this.

example-3/
 |_translator/
 |  |_greeter.py
 |  |_tests/
 |  |  |_test_greeter.py

This time only python -m unittest translator/tests/test_greeter.py passes the test. pytest translator/tests/ fails with the error ModuleNotFoundError: No module named 'translator'. That's because pytest only looks for PyPI packages to import. You can get pytest to pass by adding an __init__.py file under the translator directory, adding a setup.py file under the example-3 directory and running pip install .. You can see the source code here and the PR that fixes it here.

Example 4

This is the same as example 3 except the tests package is outside the source code package.

example-4/
 |_ translator/
 |  |_ greeter.py
 |
 |_ tests
 |  |_ test_greeter.py

Once again python -m unittest tests/test_greeter.py passes but pytest tests/ fails with a ModuleNotFoundError. The fix is exactly the same as example 3. Add an __init__.py file under the translator directory and a setup.py file under the example-4 directory then run pip install .. The source code can be seen here and the PR to fix it can be seen here.

Example 5

Example 5 is a little more complex because there is a subpackage under the translator package. The directory stucture will look like that.

example-5/
 |_translator/
 |  |_ greeter.py
 |  |_ lang/
 |  |  |_ en.py
 |  |  |_ es.py
 |
 |_tests
 |  |_test_greeter.py

As you've probably already guessed python -m unittest tests/test_greeter.py passes and pytest tests/ failes with ModuleNotFoundError. If you apply the same fix from examples 3 and 4, pytest still fails, but this time with the error ModuleNotFoundError: No module named 'lang'. This error is caused by the setup.py file i've been using. In setup.py I use the function find_packages() from setuptools which traverses the directory structure and tries to find packages, but it's only looking for "Regular Packages". That's right the ones that require an __init__.py file to be present. So to fix this example an __init__.py needs to be added under the lang directory as well. Unless you used the -e flag in yout pip install command you'll need to re-run pip install . again since the __init__.py has to exist before the PyPI package was installed.

Conclusion

I hope that can save you hours of debugging import issues so you can focus on the more enjoyable parts of coding. Here are some best practices I've taken away from this journey.

__init__.py files should only be used when needed by setuptools or when a package needs some intial setup on import.
Tests should remain outside the source code package (Can help with things like Docker which only wants Prod code)
pip install -e . will pick up new subpackages that have __init__.py w/o needing to rerun pip install .

A good next step would be to look into importlib which exposes
the implementation of the import statement.

Thanks for reading. If you enjoy this content check out my other articles on DEV.to, or tune into the Namespace Podcast which I co-host with another TDD, Python junkie like myself.

Go v Python A Technical Deep Dive

Derek D. — Mon, 18 May 2020 06:55:34 +0000

Go v Python a Technical Deep Dive

Python is well known for its simple, natural language like syntax, and ease of use. Since the official Python language is written in C/C++ you might think Go, a language inspired by C with an emphasis on greater simplicity would have similar syntax to Python. I’ve found, Go is more similar to C++ than it is Python. That's not to say there are no
similarities, but they are fairly high level.

If you're a Python developer wanting to learn Go this is the article for you. My goal is for anyone comfortable with Python, can, by the end of the article, read, write, and understand Go well enough to write simple programs and understand the code
snippets they'll find on sites like Stack Overflow.

Before the technical deep dive, I’m going to briefly talk about the common use cases of each language, fundamental differences and the Go syntax rules that are likely to trip you up as you learn Go.

Use Cases

Python has found its place in scientific computing, data science, web development, artificial intelligence, and machine learning. Go was designed with a narrower set of use cases in mind, primarily distributed systems and applications that rely on running lots of tasks concurrently.

In general if your programming task deals with distributed systems, or heavily relies on concurrency Go is an excellent choice. For more general programming tasks or for analyzing data there is no better choice than python.

Fundamental Differences

The two big fundamental differences are:

Go is a compiled language while Python is an interpreted language.
Go is a statically typed language while Python is a dynamically typed language.

In case you are not familiar with the differences between statically, and dynamically typed languages here is what I mean when using these terms.

Statically typed languages require a variable’s data type to be declared explicitly in source code. Statically typed languages are often strongly typed, meaning once a variable has been assigned a data type that variable cannot switch data types. Dynamically typed languages on the other hand use type inferencing to determine a variable’s data type at runtime and are often weakly typed meaning the variables data type can change throughout the program.

Not all statically typed languages are strongly typed and not all dynamically typed languages are weakly typed, however in Go’s case it is statically and strongly typed and in Python’s case it is dynamically and weakly typed.

Syntax Differences

Go is statically and strongly typed while Python is dynamically and weakly typed.
Python allows strings to be surrounded by single quotes, or double quotes while Go requires strings to be surrounded by double-quotes. (i.e. "This is a string in Go" while 'this is a compiler error in Go')
Go uses true/false while Python uses True/False.
Comments in Go begin with // while in Python they begin with #.
Go enforces privacy of functions and properties while Python only supports privacy by convention using underscore for private and double underscore for really private.
Go doesn't care about indentation while Python uses indentation to determine scope.

Technical Deep Dive

Your journey into Go starts with the basic data types available in each language. You’ll work your way through variables, lists/arrays, dicts/maps, functions and classes/structs ending with concurrency and packaging.

Data Types

Go has a lot more data types than Python and Go's data types are a lot more specific. The table below shows the equivalent Python to Go data types.

Python	Go
bool	bool
str	string
int	int, uint, int8, uint8, int16, uint16, int32, uint32, int63, int64
float	float32, float64
list	array
dict	map

NOTE: When you see a single Python type that corresponds to multiple Go types it means that one Python type can be used to represent any one of the Go types listed.

The takeaway is data types in Go (int8, float32, etc.) hint at the number of bits used to store values. Binary representation of numbers is beyond the scope of this article but there is an excellent write-up here if you are curious. For now, you can use the int, uint, and float64 data types in Go without experiencing many problems.

Variables

Declaring variables in Python is simple, and consistent. The variable name goes on the left of the assignment operator, =, and the value goes on the right. Go is a little more verbose because it requires a data type on the left hand sign of the = but all in all the syntax is similar.
this.

a = 12

var a int = 12 // Variable `a` is of type int and has value 12

Go also supports type inferencing which is the mechanic Python uses to determine data types. In Go type inferencing is used when the keyword var replaces an actual data type, or the := operator is used in place of =.

var a = 12
a := 12

The := operator feels the most pythonic to me so I’ll use that in the rest of the examples. Occasionally you'll see the long-hand syntax which is either required or to exaggerate a point.

Assignment is the same between Python and Go.

a = 1337

a = 1337

That is until you want to change the type of the value stored. Python will let you overwrite the original variable with a new data type, but Go requires an explicit type conversion. Type conversions look a lot alike in Python and Go though.

a = 1337
a = 1337.0
# This is an explicit type conversion in Python
a = float(a)

a := 1337
c := float32(a)

It’s Go’s strong typing system that forces the converted value to be stored in a variable of the new type.

Just like Python, there are some conversions, such as converting an int to a list, that cannot be done automatically. The proper way to handle such a conversion is to construct a new instance of the object like type (i.e. list/array or dict/map) and use the converted value as an initial value which is a great lead into arrays v lists.

Arrays v Lists

The closest thing Go has to Python's lists are arrays. Arrays and lists both serve the same purpose, to hold a collection of items but they function very differently. The big differences are.

Arrays can only hold items of the same type while lists can hold items of any type.
Arrays have a fixed length while lists will grow as more items are added.

The first differences you’ll find between Arrays and Lists is that they are instantiated differently. Python is still straight forward having only one way to instantiate a list. Go gets a little bit complicated because arrays have capacity, and size.

shopping = ['coffee', 'milk', 'sugar']

var shopping [3]string
var shopping_v1 = []string {"coffee", "milk", "sugar"}
shopping_v2 := []string {"coffee", "milk", "sugar"}
shopping_v3 := [3]string {"coffee", "milk"}

All of the arrays in the Go example have a capacity of 3, but different sizes. The shopping array is an empty array, shopping_v1 and shopping_v2 have a size of 3, and shopping_v3 has a size of 2.

In Python, you won’t have to think about the difference between capacity and size. A list is always as big as it needs to be and the number of items in the list is the same as the length of the list (i.e. len(shopping)). With Go arrays, you will need to know and understand capacity and size. Capacity is the number of items the array CAN hold, while size is the number of items the array IS CURRENTLY holding. This table breaks down the capacity and size of each array in the previous example.

expressions	capacity	size
`var shopping [3]string`	3	0
`var shopping_v1 = []string {"coffee", "milk", "sugar"}`	3	3
`shopping_v2 := []string {"coffee", "milk", "sugar"}`	3	3
`shopping_v3 := [3]string {"coffee", "milk"}`	3	2

The items in a list/array can be retrieved or saved to a list/array by index as shown in the following code snippets. Note arrays and lists both start and index 0.

shopping = ['coffee', 'milk', 'sugar']
print(shopping[0]) # prints coffee
shopping[0] = 'decaf coffee' # Forgive me :-(
print(shopping) # prints ['decaf coffee', 'milk', 'sugar']

shopping := []string {"coffee", "milk", "sugar"}
fmt.Println(shopping[0]); # prints coffee
shopping[0] = "decaf coffee"
fmt.Println(shopping) // prints ["decaf coffee", "milk", "sugar"]

The most common way to add an item to a list in Python is calling the append() method. Go doesn’t allow items to be appended to an array because it would increase the capacity of the array which is static once the array has been declared. That being said if the size is less than capacity any unused space is filled by zero values and those zero values can be overwritten the same way another other value can be overwritten by using an index.

shopping = ["coffee", "milk", "sugar"]
shopping.append("filters")

shopping := string[4] {"coffee", "milk", "sugar"}
shopping[3] = "filters" // Arrays start at index 0

Items can be removed from a list in Python using the del keyword but in Go an item is removed by creating two slices of the array that exclude the item to be removed and saving those slices back to the original array.

shopping = ['coffee', 'milk', 'sugar']
del shopping[0] # Again forgive me I love coffee too
print(shopping) # prints ['milk', 'sugar']

shopping := [3]string {"coffee", "milk", "sugar"}
shopping = append(shopping[2], shopping[3:])
fmt.Println(shopping) // prints ["milk", "sugar"]

Slices are a common concept between Python and Go. The syntax is very similar but they behave completely different. In Python, a slice is a list made up of items from another list. It's a complete copy and what happens to the slice does not affect the original list.

In Go, a slice is a window into an array so anything done to the slice affects the underlying array, and changing the indexes of the slice only changes which items are being seen through the window.

Other than that slices have the same syntax.

[n] gets the item at the nth index
[n:] gets the items from the nth index through the last item and including the last item.
[n:m] gets the items from the nth index through the mth index excluding the item at the mth index.
[:m] gets the items from the 0th index through the nth index excluding the item at the nth index.

NOTE: Go does not support a third parameter in slices to specify the size of the step between items.

Map v Dict

When you need more than a numerical index into a collection of data the next step up is key, value pair storage. In Python you have dictionaries more commonly referred to as dicts, and in Go you have maps. Maps have the same limitation arrays had in that they are limited to storing items of the declared data type. Maps also have to be constructed using the make() function. make() handles allocating the appropriate amount of memory which is once again a detail you won't have to consider in Python.

ace_of_spades = { "suit": "spades", "value": "A"}

ace_of_spades = make(map[string]string)
ace_of_spades["suit"] = "spades"
ace_of_spades["value"] = "A"

The syntax for creating a map probably looks strange for Python developers. What you’ve seen from arrays can help simplify it though. [3]string, created an array capable of holding 3 strings, each of which could be accessed by an integer index. make(map[string]string) creates a map where each item can be accessed by a string index. The added bonus of maps is that they are not limited to a specific size the way arrays were and will continue to grow as more items are added.

Maps also can be constructed using the := operator when seeding the map with initial values.

ace_of_spades := map[string]string{"suit": "spades", "value": "A"}

You already got a sneak peek of adding a value to a map. In Go it is the same as adding a value to a dict in Python.

ace_of_spades = {"suit": "spades", "value": "A"}
ace_of_spades['numeric_value'] = 1

ace_of_spades = map[string]string{"suit": "spades", "value": "A"}
ace_of_spades["numeric_value"] = "14

Removing an item for a map is also quite similar to removing an item from a Python dict. In Python, you would use del while in Go you use the builtin delete function.

ace_of_spades = {'suit': 'spades', 'value': 'A'}
del ace_of_spades['value']

ace_of_spades := map[string]string{"suit": "spades", "value": "A"}
delete(ace_of_spades, "value")

Pythons dictionaries have the .get() function providing a safe way to check if a key exists. Maps also have a mechanism for checking if a key exists. It looks like this.

item, exists = ace_of_spades["does not exist"]

In this example exists would be false, and item would be an empty string. This same pattern works for all maps. The only thing that changes is what value is assigned to item. When the key exists in the map item is the value associated with that key, and exists is true. When the key does not exist exists is false, and item is the zero value for the data type being stored in the map.

Functions

Functions in Python and Go both allow multiple return values, support higher-order functions and use a specific keyword to declare a function. That keyword for Python is def and func in Go. The primary difference between the languages is that Go requires a data type declaration for function parameters and return values, while Python does not. Python’s type hinting is very similar to Go’s data type declarations so if you’ve been type hinting you’ll be one step ahead.

Here is an example showing the required data type declarations for a greeting function in Go and the same function in Python using type hints.

# Hints the function takes a string as a parameter and returns a string
def greeting(name: str) -> str:
    return f'Hello {name}!'

print(greeting('Derek')) # Prints Hello Derek!
print(greeting(12)) # Prints Hello 12!

func greeting(name string) string {
  return fmt.Printf("Hello %s!", name)
}

func main() {
  fmt.Println(greeting("Derek")) // Prints Hello Derek!
  fmt.Println(greeting(12)) // Cause a runtime error.
}

This example demonstrates Go’s enforced data typing compared to Python’s type hinting. Python doesn’t mind accepting an integer when the type hint is for a string, where Go encounters a runtime error when calling greeting(12).

In Go if a function will return multiple values both need a data type declaration and both return values need to be saved to their own variable. Python also allows multiple return values, however it will group them into a tuple unless each value is explicitly stored in it’s own variable. Handling multiple return values looks like this.

def greeting(name: str)-> Tuple(str,str):
    return 'Hello', name

result = greeting('World')
print(type(result)) # prints <type 'tuple'>

func greeting(name string) (string, string) {
   return "Hello", name
}

result_1, result_2 := greeting("World")
fmt.Printf("%T, %T", result_1, result_2) // Prints string, string

Both the Go and Python communities use the convention of storing a return value in a variable named _ can be ignored. It looks like this.

def greeting(name):
  return 'Hello', name

salutation, _ greeting('Derek')

func greeting(name string) (string, string) {
  return "Hello", name
}

salutation, _ = greeting("Derek")

Time to cover the advanced topics of higher-order functions, callbacks, and closures.

Although Python and Go both support high-order functions Go once again has more explicit and verbose syntax. In Python assigning a function to a variable (closure), passing a function as an argument (callback) and expecting a function as an argument
(higher-order function) is basically the same syntax as any other variable.

def english(name: str) -> str:
  return f'Hello {name}!'


def spanish(name: str) -> str:
  return f'Hola {name}!'


def greeting(lang, name: str) -> str:
    return lang(name)


en = english
es = spanish

print(greeting(en, 'Derek')) # Prints Hello Derek!
print(greeting(es, 'Derek')) # Prints Hola Derek!

In this example the variables en and es are closures, the lang parameter in the greeting function is the callback, and the greeting function itself is the higher-order function. Go has the same capabilities but is very explicit about the data types for
Parameters and return values not only for the function being passed but also the function it is being passed to. It’s easier to understand the data type declarations required by looking at code.

func english(name string) string {
  return fmt.Sprintf("Hello %s!", name)
}

func spanish(name string) string {
  return fmt.Sprintf("Hola %s!", name)
}

func greeting(lang func(string)string, name string) string {
  return lang(name)
}

func main() {
  en := english
  es := spanish

  fmt.Println(greeting(en, "Derek")) // Prints Hello Derek!
  fmt.Println(greeting(es, "Derek")) // Prints Hola Derek!
}

The same statements about the Python code are true for this code as well. en and es are closures, the lang parameter in the greeting function is a callback, and the greeting function itself is still the higher-order function. The only thing that has changed is the function declaration for greeting has become a monster. It’s stating the greeting function is expecting two parameters. The first is a function that takes in a string and returns a string and the second parameter is a simple string. The function declaration for greeting can be cleaned up using a function pointer.

Function pointers are exactly as they sound. They are pointers to functions. Another way to look at it is function pointers are variables that store references to a specific function, which is exactly what a closure is. The syntax for declaring a new function pointer looks like this.

type PointerNoParamNoReturn func()
type PointerIntParamNoReturn func(int)
type PointerIntAndStrParamsStrReturn func(int, string)string

All of these examples create function pointers, but each creates a new type that can only point to functions that match a specific signature. The last example which is the most complex creates the new type PointerIntAndStrParamsStrReturn which
is a function pointer that can point to any function whose first parameter is an int and whose second parameter is a string and returns a string. Basically PointerIntAndStrParamsStrReturn becomes an alias for func(int, string)string. Here is an example using a function pointer to clean up the monstrous function declaration from the previous greeting example.

type LangPtr func(string)string

def greeting(lang LangPtr, name string) string {
  return lang(name)
}

func main() {
  // Declares es and en variables of type lang_ptr (i.e. are function pointers)
  var en, es langPtr
  en = english
  es = spanish

  fmt.Println(greeting(en, "Derek")) // Prints Hello Derek!
  fmt.Println(greeting(es, "Derek")) // Prints Hola Derek!
}

Structs v Classes

Structs are a concept that comes from C, which was not designed to be an Object-Oriented language, so typical OO constructs like classes were omitted from its grammar. There was still a need to group variables together in a single block of memory which C accomplished with structs. C++ came after C adding in classes while still supporting structs. Then Python came after C++ doing away with structs altogether. When using a CPython implementation of Python structs are being used without you knowing it, but that's enough history.

A class in Python can define properties, and methods. Structs can only define properties, however, methods are still supported in the way of bolting a function on to a struct giving that function access to the struct’s properties and that’s basically a method. Here is an example of a simple Class and Struct definition for a Playing Card showing the syntactical differences in Go v Python.

class Card:
  def __init__(self, suit, value):
    self.suit = suit
    self.value = value

ace_of_spades = Card(1, 'S')

type Card struct {
  suit string
  value int
}

func main() {
  ace_of_spades := Card{1, "S"}
}

Go allows structs to be instantiated using positional arguments as shown in the above example, or with keyword arguments. When instantiating a struct with positional arguments or object literals as they are called in Go the first value goes into the first property defined, the second value into the second property, and so on. Object literals can take keywords as well which makes them act more like Python's **kwargs. It looks like this.

type Card struct {
  suit string
  value int
}

ace_of_spades := Card{value: 1, suit: "S"}

In this example the order of the values doesn't matter. Whatever value is provided for the suit keyword argument gets stored in the suit struct property and whatever value is provided for the value keyword argument gets stored in the value struct property.

Regardless if you use positional or keyword arguments to instantiate a struct, if you do not provide a value for one of the struct properties that property will implicitly get set to the zero value for that properties data type. This differs from Python which allows the developer to specify the default value for an unspecified argument.

class Card:
  def __init__(self, suit="", value=0):
      self.suit = suit
      self.value = value

suitless_king = Card(value=13)
valueless_heart = Card(suit="H")

type Card struct {
  suit string
  value int
}

suitless_king = Card{value: 13}
valueless_heart = Card{suit: "H"}

Once a class/struct has been instantiated properties are accessed the same way in Python and Go.

print(ace_of_spades.suit) # Prints S

fmt.Println(ace_of_spades.suit) // Prints S

In Python a function becomes a method when it is nested under a class declaration as shown below. Go bolts functions to a struct giving that function access to the struct's properties. The code below shows how a fold method can be added to a
Hand class/struct.

class Card:
  def __init__(self, suit, value):
    self.suit = suit
    self.value = value

class Hand:
  def __init__(self, cards: list):
    self.cards = cards

  def fold(self):
      card_1 = f"{self.cards[0].value}:{self.cards[0].suit}"
      card_2 = f"{self.cards[1].value}:{self.cards[1].suit}"
      print(f'Folded with hand: {card_1}, {card_2}')

my_hand = Hand([Card("S", 1), Card("H", 2)])
my_hand.fold() # Prints Folded with hand: 1:S, 2:H

type Card struct {
  suit string
  value int
}

type Hand struct {
  cards [2]Card
}

func (h Hand) fold() {
  card_1 := fmt.Sprintf("%d:%s", h.cards[0].value, h.cards[0].suit)
  card_2 := fmt.Sprintf("%d:%s", h.cards[1].value, h.cards[1].suit)
  fmt.Printf("Folded with hand: %s, %s", card_1, card_2)
}

func main() {
  ace_of_spades := Card{"S", 1}
  two_of_hearts := Card{"H", 2}
  my_hand := Hand{[2]Card{ace_of_spades, two_of_hearts}}
  my_hand.fold() // Prints Folded with hand 1:S, 2:H
}

The declaration of the fold function should feel odd to Python developers, however, it becomes a lot more familiar when h is changed to self.

func (self Hand) Fold() {
  card_1 := fmt.Sprintf("%d:%s", self.cards[0].value, self.cards[0].suit)
  card_2 := fmt.Sprintf("%d:%s", self.cards[1].value, self.cards[1].suit)
  fmt.Printf("Folded with hand: %s, %s", card_1, card_2)
}

That looks a lot more like Python. On that note did you know it's only convention to name the first parameter to a method self? Truthfully that parameter can be named anything, this, me, or something totally arbitrary like h. Here is the Python code using h instead of self.

class Hand:
  def __init__(h, cards: list):
    h.cards = cards

  def fold(h):
      card_1 = f"{h.cards[0].value}:{h.cards[0].suit}"
      card_2 = f"{h.cards[1].value}:{h.cards[1].suit}"
      print(f'Folded with hand: {card_1}, {card_2}')

Knowing that structs were taken from C, that Python is written partially in C, and how functions are bolted on to structs you should have a decent idea of what the underlying C/C++ code looks like when dealing with self.

Concurrency

Threads, multiprocessing and asynchronous I/O have been supported in Python since 3.4. Each has its own module that can be imported and used to run code asynchronously or in its own thread or process. There are no such modules in Go. It's actually quite the opposite. There is a sync package that helps make code synchronous because Go is inherently multithreaded and asynchronous. Every Go program is executed from a goroutine which is a light-weight thread managed by the runtime.

This is where Go is finally less verbose than Python. To start a new thread all you need to do is call go before a function call.

import threading
def sum(a, b):
    print(a + b)

t = threading.Thread(target=sum, args=(11, 1,))
t.start() # Kicks off a thread that prints 12

func sum(a int, b int) {
  fmt.Println(a + b)
}

go sum(11, 1) // Prints 12

Getting results from a thread, or passing values between threads has always been difficult. Go solves the problem with channels, abbreviated to chan. Using channels it is simple to pass values between goroutines. In Python, you have to use a ThreadPoolExecutor which isn't bad but is still more verbose.

import concurrent.futures

def sum(a, b):
  return a + b

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
  future = executor.submit(sum, 11, 1)
  print(future.result())

func sum(a int, b int, c (chan int)) {
  c <- a + b // Sends the results of a+b to the channel c
}

func main() {
  int_channel := make(chan int) // Creates a channel for passing integers
  go sum(11, 1, int_channel)
  res := <-int_channel // Gets the value from int_channel and stores it in res
  fmt.Println(res) // Prints 12
}

When you pass a channel to another goroutine you are literally providing a communication channel between those 2 goroutines. Channels only have the one operator, <-, which states where the data is flowing from and to. Hint data flows in the
direction of the arrow. So in the sum function data is flowing into the channel, and in the main function data is flowing out of the channel into the res variable. At the risk of editorializing, I think this is easier to think about conceptually and looks cleaner too.

Concurrency is handled so differently between the 2 languages I'm going to stop comparing side by side code samples at this point. You should have a good enough grasp of goroutines and channels to prototype an implementation of the publisher/consumer pattern though.

Packaging

Python and Go both use the project's directory structure for import paths. The concepts will feel familiar. A module is a file ending in .go or .py. A package is a directory on your file system and any module under a package is considered a module of that package. The big differences are

Python allows granular imports of packages, modules, classes, functions, or variables while Go only supports importing a complete package including the modules, classes, functions, and variables exported by that package.
Private is a real concept in Go, unlike Python which uses the _ and __ convention to indicate something is private but still giving access to that thing.
Go requires the package name to be stated at the top of every module.
The package name in the package declaration doesn't have to match the directory name.

Let's look at an example.

sampleapp/
  |_ main.py
  |_ main.go
  |_ messages /
    |_ important.py
    |_ important.go

Given this directory structure, both Python and Go will use sampleapp as the main package name. The imports of important.py and important.go look very different though. Here is what main.py and main.go might look like.

from sampleapp.messages import important

important.important()

import "sampleapp/messages"

func main() {
  messages.Important()
}

This tells you there is a function named important in important.py and a function named Important somewhere in the sampleapp/messages Go package. Since important.go is the only module under sampleapp/messages ou know the Important function is declared in that module. You can also derive that the package declaration in important.go is package messages because the function is called with messages.Important. Remember the declared package name does not have to match the import path's package name.

There is a small subtlety hidden in this example. The Python function is important and the Go function is Important with a capital I. I've been very intentional to keep casing consistent until now so this example would stand out. I did this because Go by convention only exports functions classes and variables whose names start with a capital letter. Anything starting with a lower case letter is private. It's similar to Pythons convention that _ is private and __ is really private, but with Go, you cannot access private properties or methods.

Distributing Your Package

Real Python already has an excellent tutorial on
Publishing a Package to PyPi. If you want to know more about packaging check it out.

The essentials are, the source code should be stored in a code repository of some kind, and a metadata file describing name, version, dependencies, etc needs to be included. The metadata file in Python is setup.py and go.mod in Go. Once the source code is available through a public repository you have published your Go package and there is nothing left to do. With Python, you'll need to build the wheel file from source code, create a PyPi account, and use a tool called Twine to publish the wheel to PyPi.

Downloading & Using Packages

Published packages are easy to get in either language. With Python, it's pip install <package-name> while in Go its go get <package-url>.

Once a package has been downloaded it can be imported similarly to local packages and modules. For example, if you downloaded a package named haiku that generates random haikus. To import the package in Python it would be import haiku. In Go, it
depends where you got the package from. If this package was downloaded from my Github the go get command would be go get github.com/ezzy1337/haiku and the import would be import "github.com/ezzy1337/haiku". However, if it was stored on
Test Testoffersons Github the go get command would have been something like go get github.com/ttestofferson/haiku and the import would be import "github.com/ttestofferson/haiku". Regardless of where the package came from after being imported, it will be used like any other imported package.

Conclusion

Wow, thanks for sticking with me through all of that. You should be able to go play around with Go and be confident in what you are doing. Google has a self-guided Tour of Go which I highly recommend especially if you just want to play around with what you've learned in this article. Keep in mind the syntax differences from the beginning of this article I said that are likely to trip you up.

I'd love to hear from you! Hit me up @d3r3kdrumm0nd on Twitter with comments, questions, or if you just want to stay up to date with my next project.

Get Going With Go

Derek D. — Wed, 15 Apr 2020 04:25:45 +0000

This article is meant to be a quick introduction to installing the Go programming language, running your first script, and building a binary that can be executed on Windows, Mac OS X, and Linux. I've included links to resources from Google and Digital Ocean that go into much greater detail on these topics but if you are just looking for a quick 5-minute introduction this is the article for you.

Downloading Go

Google provides an installer for the major operating systems (Windows, macOS X, and Linux). You can also download a tarball of binaries and install directly from source code, but that's out of scope for this article. For this step download and run the installer for your OS, following the prompts.

You've probably heard or seen articles about the GOPATH before. This was a top-level directory where all of your go projects had to be located. Fortunately, it's no longer needed. At least not on Mac OS X. At this point, a little code is needed to test the installation.

Go Say Hello

Unfortunately, Go doesn't have a REPL (Read Evaluate Print Loop) so you'll be saving and running code from a file.

Copy the code below and save it as greeting.go. Golang requires the .go extension but the rest of the filename could be anything you want. Without covering Go syntax to much this simple script when executed, prompts for a name and says hello to whatever name was entered.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    reader := bufio.NewReader(os.Stdin)
    fmt.Print("What is your name? ")
    text, _ := reader.ReadString('\n')
    fmt.Println("Hello ", text)
}

There are 2 important structural parts to the sample code. The first is package main which identifies this file contains the entrypoint to your code. The second is func main() which is the actual entrypoint. If either of these is missing or is not named main the code will not execute.

Go Run V Go Build

There are 2 ways to run the sample script. During development, you'll mostly use go run which compiles the code and executes it in one step. When it comes time to deploy the code you should use go build which creates a binary that can be executed by calling directly from the terminal.

Go Run

Go Run is pretty simple and has one required argument, a package name. In this case, you will use the filename. Here is an example of the greeting.go package. go run greeting.go.

Go Build

Go build is a little more complex. The simplest form is go build <package-name>. This creates a binary named the same as the package name, minus the .go extension. So go build greeting.go would create a binary named greeting. The new binary can be executed with ./greeting.

The more advanced form is using the -o flag which allows you to name the binary produced by go build. Rebuilding the greeting.go package using the -o flag would look like this go build -o hello.exe greeting.go which would produce a binary file named hello.exe.

Conclusion

Use go run <package-name> when testing code in development.
Use go build <package-name> when preparing to ship code.
The file with package main at the top of the file, must contain the func main() function which is the entrypoint to the app.
Your Go code no longer has to go under the GOPATH.

Additional Resources

Official Getting Started

How To Build And Install Go Programs by Digital Ocean

A Pythonic Guide to SOLID Design Principles

Derek D. — Fri, 20 Mar 2020 22:39:36 +0000

People that know me will tell you I am a big fan of the SOLID Design Principles championed by Robert C. Martin (Uncle Bob). Over the years I've used these principles in C#, PHP, Node.js, and Python. Everywhere I took them they were generally well-received...except when I started working in Python. I kept getting comments like "It's not a very Pythonic way to do things" during code review. I was new to Python at the time so I didn't really know how to respond. I didn't know what Pythonic code meant or looked like and none of the explanations offered were very satisfying. It honestly pissed me off. I felt like people were using the Pythonic way to cop-out of writing more disciplined code. Since then I've been on a mission to prove SOLID code is Pythonic. That's the wind up now here's the pitch.

What is SOLID Design

Michael Feathers can be credited for creating the mnemonic SOLID which is based on principles from Robert C. Martin’s paper, “Design Principles and Design Patterns”.
The principles are

Single Responsibility Principle
Open Closed Princple
Liskov's Substitutablilty Principle
Interface Segregation Principle
Dependency Inversion Principle

We will cover these in more detail shortly. The most important thing to note about the SOLID design principles is they are meant to be used holistically. Choosing one and just one is not going to do much for you. It's when used together you start to see the real value in these principles.

What is the Pythonic Way

Although there is no official definition for the Pythonic way a little Googling gives you several answers along this general vain.

"Above correct syntax Pythonic code follows the normally accepted conventions of the Python community, and uses the language in a way that follows the founding philosophy." - Derek D.

I think the most accurate description comes from the Zen of Python.

The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than right now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

One Last Thing

Before I jump right into the principles and how they relate to the Zen of Python, there's one thing I want to do that no other SOLID tutorial does. Instead of using a different code snippet for each principle, We are going to work with a single code base and make it more SOLID as we cover each principle. Here is the code we are going to start with.

class FTPClient:
  def __init__(self, **kwargs):
    self._ftp_client = FTPDriver(kwargs['host'], kwargs['port'])
    self._sftp_client = SFTPDriver(kwargs['sftp_host'], kwargs['user'], kwargs['pw'])

  def upload(self, file:bytes, **kwargs):
    is_sftp = kwargs['sftp']
    if is_sftp:
      with self._sftp_client.Connection() as sftp:
        sftp.put(file)
    else:
      self._ftp_client.upload(file)

  def download(self, target:str, **kwargs) -> bytes:
    is_sftp = kwargs['sftp']
    if is_sftp:
      with self._sftp_client.Connection() as sftp:
        return sftp.get(target)
    else:
      return self._ftp_client.download(target)

Single Responsibility Principle (SRP)

Definition: Every module/class should only have one responsibility and therefore only one reason to change.

Relevant Zen: There should be one-- and preferably only one --obvious way to do things

The Single Responsibility Principle (SRP) is all about increasing cohesion and decreasing coupling by organizing code around responsibilities. It's not a big leap to see why that happens. If all the code for any given responsibility is in a single place that's cohesive and while responsibilities may be similar they don't often overlap. Consider this non-code example. If it is your responsibility to sweep and my responsibility to mop there's no reason for me to keep track of whether or not the floor has been swept. I can just ask you, "has the floor been swept"? and base my action according to your response.

I find it useful to think of responsibilities as use cases, which is how our Zen comes into play. Each use case should only be handled in one place, in turn, creating one obvious way to do things. This also satisfies the, "one reason to change" portion of the SRP's definition. The only reason this class should change is if the use case has changed.

Examining our original code we can see the class does not have a single responsibility because it has to manage connection details for an FTP, and SFTP server. Furthermore, the methods don't even have a single responsibility because both have to choose which protocol they will be using. This can be fixed by splitting the FTPClient class into 2 classes each with one of the responsibilities.

class FTPClient:
  def __init__(self, host, port):
    self._client = FTPDriver(host, port)

  def upload(self, file:bytes):
    self._client.upload(file)

  def download(self, target:str) -> bytes:
    return self._client.download(target)


class SFTPClient(FTPClient):
  def __init__(self, host, user, password):
    self._client = SFTPDriver(host, username=user, password=password)

  def upload(self, file:bytes):
    with self._client.Connection() as sftp:
      sftp.put(file)

  def download(self, target:str) -> bytes:
    with self._client.Connection() as sftp:
      return sftp.get(target)

One quick change and our code is already feeling much more Pythonic. The code is sparse, and not dense, simple not complex, flat and not nested. If you're not on board yet think about how the original code would look with error handling compared to the code following SRP.

Open Closed Principle (OCP)

Definition: Software Entities (classes, functions, modules) should be open for extension but closed to change.

Relevant Zen: Simple is better than complex. Complex is better than complicated.

Since the definition of change and extension are so similar it is easy to get overwhelmed by the Open Closed Principle. I've found the most intuitive way to decide if I'm making a change or extension is to think about function signatures. A change is anything that forces calling code to be updated. This could be changing the function name, swapping the order of parameters, or adding a non-default parameter. Any code that calls the function would be forced to change in accordance with the new signature. An extension, on the other hand, allows for new functionality, without having to change the calling code. This could be renaming a parameter, adding a new parameter with a default value, or adding the *arg, or **kwargs parameters. Any code that calls the function would still work as originally written. The same rules apply to classes as well.

Here is an example of adding support for bulk operations.

Your gut reaction is probably to add a upload_bulk and download_bulk functions to the FTPClient class. Fortunately, that is also a SOLID way to handle this use case.

class FTPClient:
  def __init__(self, host, port):
      ... # For this example the __init__ implementation is not significant

  def upload(self, file:bytes):
      ... # For this example the upload implementation is not significant

  def download(self, target:str) -> bytes:
      ... # For this example the download implementation is not significant

  def upload_bulk(self, files:List[str]):
    for file in files:
      self.upload(file)

  def download_bulk(self, targets:List[str]) -> List[bytes]:
    files = []
    for target in targets:
      files.append(self.download(target))

    return files

In this case, it's better to extend the class with functions than extend through inheritance, because a BulkFTPClient child class would have to change the function signature for download reflecting it returns a list of bytes rather than just bytes, violating the Open Closed Principle as well as Liskov's Substituitability Principle.

Liskov's Substituitability Principle (LSP)

Definition: If S is a subtype of T, then objects of type T may be replaced with objects of Type S.

Relevant Zen: Special cases aren’t special enough to break the rules.

Liskov's Substituitablity Principle was the first of the SOLID design principles I learned of and the only one I learned at University. Maybe that's why this one is so intuitive to me. A plain English way of saying this is, "Any child class can replace its parent class without breaking functionality."

You may have noticed all of the FTP client classes so far have the same function signatures. That was done purposefully so they would follow Liskov's Substituitability Principle. An SFTPClient object can replace an FTPClient object and whatever code is calling upload, or download, is blissfully unaware.

Another specialized case of FTP file transfers is supporting FTPS (yes FTPS and SFTP are different). Solving this can be tricky because we have choices. They are:
1. Add upload_secure, and download_secure functions.
2. Add a secure flag through **kwargs.
3. Create a new class, FTPSClient, that extends FTPClient.

For reasons that we will get into during the Interface Segregation, and Dependency Inversion principles the new FTPSClient class is the way to go.

class FTPClient:
  def __init__(self, host, port):
    ...

  def upload(self, file:bytes):
    ...

  def download(self, target:str) -> bytes:
    ...

class FTPSClient(FTPClient):
    def __init__(self, host, port, username, password):
        self._client = FTPSDriver(host, port, user=username, password=password)

This is exactly the kind of edge case inheritance is meant for and following Liskov's makes for effective polymorphism. You'll note than now FTPClient's can be replaced by an FTPSClient or SFTPClient. In fact, all 3 are interchangeable which brings us to interface segregation.

Interface Segregation Principle (ISP)

Definition: A client should not depend on methods it does not use.

Relevant Zen: Readability Counts && complicated is better than complex.

Unlike Liskov's, The Interface Segregation Principle was the last and most difficult principle for me to understand. I always equated it to the interface keyword, and most explanations for SOLID design don't do much to dispel that confusion. additionally, most guides I've found try to break everything up into tiny interfaces most often with a single function per-interface because "too many interfaces are better than too few".

There are 2 issues here, first Python doesn't have interfaces, and second languages like C# and Java that do have interfaces, breaking them up too much always ends up with interfaces implementing interfaces which can get complex and complex is not Pythonic.

First I want to explore the too small of interfaces problem by looking at some C# code, then we'll cover a Pythonic approach to ISP. If you agree or are just choosing to trust me that super small interfaces are not the best way to segregate your interfaces feel free to skip to the Pythonic Solution below.

# Warning here be C# code
public interface ICanUpload {
  void upload(Byte[] file);
}

public interface ICanDownload {
  Byte[] download();
}

class FTPClient : ICanUpload, ICanDownload {
  public void upload(Byte[] file) {
    ...
  }

  public Byte[] download(string target) {
    ...
  }
}

The trouble starts when you need to specify the type of a parameter that implements both the ICanDownload and ICanUpload interfaces. The code snippet below demonstrates the problem.

class ReportGenerator {
  public Byte[] doStuff(Byte[] raw) {
    ...
  } 

  public void generateReport(/*What type should go here?*/ client) {
    raw_data = client.download('client_rundown.csv');
    report = this.doStuff(raw_data);
    client.upload(report);
  }
}

In the generateReport function signature you either have to specify the concrete FTPClient class as the parameter type, which violates the Dependency Inversion Principle or create an interface that implements both ICanUpload, and ICanDownload interfaces. Otherwise, an object that just implements ICanUpload could be passed in but would fail the download call and vice-versa with an object only implementing the ICanDownload interface. The normal answer is to create an IFTPClient interface and let the generateReport function depend on that.

public interface IFTPClient: ICanUpload, ICanDownload {
    void upload(Byte[] file);
    Byte[] download(string target);
}

That works, except we are still depending on FTP clients. What if we want to start storing our reports in S3?

The Pythonic Solution

To me, ISP is about making reasonable choices for how other developers will interface with your code. That's right it's more related to the I in API and CLI than it is the interface keyword. This is also where the "Readability Counts" from the Zen of Python is a driving force. A good interface will follow the semantics of the abstraction and match the terminology making the code more readable.

Let's look at how we can add an S3Client since it has the same upload/download semantics as the FTPClients. We want to keep the S3Clients signature for upload and download consistent, but it would be nonsense for the new S3Client to inherit from FTPClient. After all, S3 is not a special case of FTP. What FTP and S3 do have in common is that they are file transfer protocols and these protocols often share a similar interface as seen in this example. So instead of inheriting from FTPClient it would be better to tie these classes together with an abstract base class, the closest thing Python has to an interface.

We create a FileTransferClient which becomes our interface and all of our existing clients now inherit from that rather than inheriting from FTPClient. This forces a common interface and allows us to move bulk operations into their own interface since not every file transfer protocol will support them.

from abc import ABC
class FileTransferClient(ABC):
  def upload(self, file:bytes):
    pass

  def download(self, target:str) -> bytes:
    pass

  def cd(self, target_dir):
    pass


class BulkFileTransferClient(ABC):
  def upload_bulk(self, files:List[bytes]):
    pass

  def download_bulk(self, targets:List[str]):
    pass

What does this afford us though...well this.

class FTPClient(FileTransferClient, BulkFileTransferClient):
  ...

class FTPSClient(FileTransferClient, BulkFileTransferClient):
  ...

class SFTPClient(FileTransferClient, BulkFileTransferClient):
  ...

class S3Client(FileTransferClient):
  ...

class SCPClient(FileTransferClient):
  ...

Oh Man! is that good code or what. We even managed to squeeze in a SCPClient and kept bulk actions as their own mixin. All this ties together nicely with Dependency Injection, a technique used for the Dependency Inversion Principle.

Dependency Inversion Principle (DIP)

Definition: High-level modules should not depend on low-level modules. They should depend on abstractions and abstractions should not depend on details, rather details should depend on abstractions.

Relevant Zen: Explicit is Better than Implicit

This is what ties it all together. Everything we've done with the other SOLID principles was to get to a place where we are no longer dependent on a detail, the underlying file transfer protocol, being used to move files around. We can now write code around our business rules without tying them to a specific implementation. Our code satisfies both requirements of dependency inversion.

Our high-level modules no longer need to depend on a low-level module like FTPClient, SFTPClient, or S3Client, instead, they depend on an abstraction FileTransferClient. We are depending on the abstraction of moving files not the detail of how those files are moved.

Our abstraction FileTransferClient is not dependent on protocol specific details and instead, those details depend on how they will be used through the abstraction (i.e. that files can be uploaded or downloaded).

Here is a example of Dependency Inversion at work.

def exchange(client:FileTransferClient, to_upload:bytes, to_download:str) -> bytes:
    client.upload(to_upload)
    return client.download(to_download)

if __name__ == '__main__':
    ftp = FTPClient('ftp.host.com')
    sftp = FTPSClient('sftp.host.com', 22)
    ftps = SFTPClient('ftps.host.com', 990, 'ftps_user', 'P@ssw0rd1!')
    s3 = S3Client('ftp.host.com')
    scp = SCPClient('ftp.host.com')

    for client in [ftp, sftp, ftps, s3, scp]:
        exchange(client, b'Hello', 'greeting.txt')

Conclusion

There you have it a SOLID implementation that is also very Pythonic. I'm hoping you've at least warmed up to SOLID if you hadn't before, and for those of you that are learning Python and not sure how to continue writing SOLID code this has been helpful. This, of course, was a curated example that I knew would lend itself to my argument, but in writing this I was still surprised how much changed along the way. Not every problem will fit this exact breakdown, but I've tried to include enough reasoning behind
my decisions that you can choose the most SOLID & Pythonic implementation in the future.

If you disagree or need any clarification please leave a comment or @d3r3kdrumm0nd on Twitter.