DEV Community: sam

Writing a custom Git command in Python (eventually)

sam — Fri, 26 Mar 2021 17:31:14 +0000

I have a pretty standardized workflow at my day job. I log on, check Jira, and start banging away. I fork a new branch off of our development branch (conveniently named develop) for every new ticket I'm assigned. I like doing things this way, because it keeps the history clean and makes merges and code review easier. (If you've ever had to pull out a feature that's been merged in piecemeal, I'm sure you'll agree.)

This may strike you as a fairly rote series of steps; it certainly struck me as such, so I decided to try writing a custom Git command to help automate it. You can add a custom command to Git by creating an executable file on your $PATH that's named git-$COMMAND_NAME. Git will strip off the git- prefix when you invoke it, e.g. git-my-command will be runnable as git my-command.

My first attempt was a simple Bash script:

#! /usr/bin/sh

new_branch() {
  git switch develop
  git pull origin develop
  git switch -c "$1"
}

new_branch "$@"

I made a .git-commands folder in my home directory and saved this as git-new-branch. I made the file executable and added ~/.git-commands to my $PATH.

It worked. Pretty well.

There were two pain points. First, my editor (Neovim + CoC) didn't pick up on the fact that it was a Bash script, so I didn't get any linting/syntax highlighting/debugging help. I have coc-sh and shellcheck set up, but they were silent. (It is here that I must admit that what I shared above is not the original version, which didn't actually work.) The solution to this issue was to rename the file to git-new-branch.sh. Fine, but then Git thought the command name was new-branch.sh, which sucks. The solution to that was a simple ln -s ~/.git-commands/git-new-branch.sh ~/.git-commands/git-new-branch. Easy enough.

The second problem was trickier. You see, I'm not a very good developer. As such, tickets I'm assigned are frequently re-opened in the course of the QA process. (A hearty shout-out to QA people the world over, and especially at my company, for their tireless diligence and infinite patience.) It's my practice in such situations to fork a new branch off of develop with the same ticket name plus a letter. So, should ticket 1234 be re-opened (after the original was merged into develop and deployed to our development server for QA), I'd open 1234-b, 1234-c, etc. My new command didn't handle this. The issue wasn't disastrous—I'd just get a fresh pull of develop and then a chiding from Git about the pre-existing branch name. I could probably just remember that it was a reopened ticket and add the letters myself, but again, tickets are sometimes re-opened multiple times, and the whole point of this exercise is making my life easier.

There's got to be a better way!

I tried fiddling around with sed and managed to isolate the suffix, but then ran into some limitations, both around Bash's character-handling deficiencies and (more saliently) my own Bash-handling deficiencies, specifically around how to "increment" a character—i.e. going from b to c.

Python to the rescue:

#! /usr/bin/env python3

import re
from subprocess import run
from sys import argv, exit


def branches_list():
    return run(
        ["git", "--no-pager", "branch", "--list"], capture_output=True, text=True
    ).stdout


def next_branch(branch_name):
    p = f"{branch_name}(?:-(\\w))?"
    matches = re.findall(p, branches_list())
    if len(matches) == 0:
        return branch_name
    if len(matches) == 1 and matches[0] == "":
        return f"{branch_name}-b"
    return f"{branch_name}-{chr(ord(matches[-1]) + 1)}"


def main():
    if len(argv) != 2:
        print("Please supply a branch name")
        exit(1)
    branch_name = argv[1]
    run(["git", "switch", "develop"])
    run(["git", "pull", "origin", "develop"])
    run(["git", "switch", "-c", next_branch(branch_name)])
    exit(0)


if __name__ == "__main__":
    main()

(N.B.: subprocess.run was introduced in 3.5; if you're using an older version you'll need to fall back to subprocess.check_output; see the docs for more.)

I threw this into new_branch.py, marked it as executable, deleted the symbolic link to the Bash version and created a new one to this & voilà!

As much as it can feel (to me, at least) like a crutch sometimes, Python legitimately rules for this kind of thing. It was born as a "glue language," and has all the necessary batteries included. I could have reached for Haskell or Rust (Haskell's Ord instance for Char especially would've made some of this nicer) but I didn't want to mess with compiling a "production" build and then figuring out where Stack or Cargo (respectively) put it and so on. (Yes, I could've written a Stack script but I don't really know how to do that and didn't want to bother.)

I guess the lessons here are:

Custom Git commands aren't that hard to write, and we (I) should probably be writing more of them.
While professional development and personal growth are worthwhile goals, sometimes the easy path is the right one.

(One last unsolved irritant: some not-terribly-thorough Googling failed to turn up a way to add Git commands per-repo, as opposed to globally. If anyone has a solution for this, please let me know.)

If anyone's got any custom Git commands they're particularly proud of, feel free to drop them in the comments!

What to Except When You're Excepting: Python error handling dos & don'ts

sam — Fri, 10 Jul 2020 21:16:19 +0000

Exceptions in Python are a great part of the language. They're easy to use, easy to subclass, and pretty fast. Unfortunately, convenience can lead to sloppy thinking and sloppier programming. In my 5+ years of Python programming in various startups, I've encountered a lot of bad exception handling, so here are some recommendations for dos and don'ts.

Do think about your exceptions

Towards the bottom of the post I linked to above, the author provides evidence that try/catch can be slower than if statements when the errors are frequent. His somewhat flippant advice is "to look into more reliable infrastructure," which is helpful in some contexts. I've found, however, that the convenience of Python exceptions, coupled with the dynamically-typed nature of the language itself, leads to developers either taking exceptions for granted, over-using try/catch, or both.

One advantage of statically-typed languages like Haskell and Rust is that they force the programmer to think more deliberately about errors, and provide them with the language constructs to do so in the form of Maybe/Either (in Haskell) or Option/Result (in Rust.)

[For those unfamiliar, Maybe/Option are ways to avoid null, aka undefined aka None aka Hoare's 'billion-dollar mistake'; in Haskell, a value of type Maybe a can either be Just a (e.g. Just "a friend"), or Nothing. Similarly, a value of type Either a b can be either (get it?) Left a (e.g. Left "for dead") or Right b (e.g. Right 1). Both languages use one half of their Either-like types for "success" values, and the other for "error" values; it's common to see things like Result<Int,String> in Rust or Either String Int in Haskell. (I know why Haskell orders it the way it does (ask me in the comments if you care), I don't know why Rust switched the order.)]

Both of these languages have "real" exceptions, but most Haskell and Rust programmers eschew them in favor of Maybe/Either/Option/Result. Why? Because using these types allows you, and people reading your code, to differentiate between errors and exceptions. Errors are anticipatable, semi-routine, unfortunate but not disastrous. Exceptions are exceptional—they shouldn't happen, their very presence represents some serious derivation from the usual course of events.

Even in Python, which doesn't have anything like Maybe/Either, it's still worth thinking about which of your "exceptions" are actually exceptional, and which are merely errors. Viz:

try:
    middle_name = user['middle_name']
except KeyError:
    middle_name = ''


try:
    birthday = user['birthday']
except KeyError:
    birthday = ''

We're treating two pieces of missing information the same way, but they're actually completely different. Lots of people don't have middle names; everybody has a birthday. (Of course, some users may be wary of divulging such information to strangers on the internet, but let's assume you've earned their trust.) It's much clearer to treat different cases differently:

middle_name = user.get('middle_name', '')
birthday = user['birthday']

Now we're treating errors and exceptions with the relative alarm they each deserve.

Do handle specific errors appropriately

This is an extension of the previous point. You can use multiple catches per try, and if there's cause to, you should.

# instead of this...
try:
    do_whatever()
except SomeException as err:
    if isinstance(err, FooException):
        handle_foo_exception(err)
    elif isinstance(err, BarException):
        handle_bar_exception(err)

# ...do this
try:
    do_whatever()
except FooException as err:
    handle_foo_exception(err)
except BarException as err:
    handle_bar_exception(err)

Do write your own exception subclasses

You can facilitate and extend the pattern above by writing and throwing your own exception subclasses. It's easy and aids in clarity.

# instead of this...
def check_birthday(user):
    if not user.birthday:
        raise ValueError('User birthday missing!')
    elif user.birthday.year > 2002:
        raise ValueError('User is too young!')
    elif user.birthday.year < 1900:
        raise ValueError('User is probably dead!')

try:
    check_birthday(user)
except ValueError as err:
    msg = str(err)
    if 'missing' in msg:
        redirect_to_birthday_input()
    elif 'young' in msg:
        redirect_to_site_for_teens()
    elif 'old' in msg:
        handle_probable_fraud()

# ...do this
class UserBirthdayException(ValueError):
    pass

class UserBirthdayMissing(UserBirthdayException):
    pass

class InvalidUserBirthday(UserBirthdayException):
    def __init__(self, problem, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.problem = problem


def check_birthday(user):
    if not user.birthday:
        raise UserBirthdayMissing
    elif user.birthday.year > 2002:
        raise InvalidUserBirthday(problem='young')
    elif user.birthday.year < 1900:
        raise InvalidUserBirthday(problem='old')

try:
    check_birthday(user)
except UserBirthdayMissing:
    redirect_to_birthday_input()
except InvalidUserBirthday as err:
    if err.problem == 'young':
        redirect_to_site_for_teens()
    else:
        handle_probable_fraud()

Sure, it's more boilerplate, but the added clarity and control is worth it. It's especially worth considering if you're doing a lot of validation, or find yourself repeatedly handling a set of errors from an API—write some custom error classes and a response handler function, and your days of grepping through your codebase to find all the places you need to change your error handling because some third-party has switched from colons to tabs in their API responses are over.

Do Make use of exception class hierarchies

Like everything else in Python, exceptions are objects, and they have their own classes and inheritance. catch will handle subclasses of errors appropriately.

class AError(Exception):
    pass

class BError(AError):
    pass

class CError(AError):
   pass


try:
    raise BError
except CError:
    print('Caught CError')  # won't happen
except AError:
    print('Caught AError')  # will happen

Familiarize yourself with Python's builtin exceptions, and don't be shy about using inheritance when defining your own error classes.

Do keep it tight

Quick, tell me which of these functions could throw SomeException:

def f(x):
    try:
        a = do_something(x)
        b = do_something_else(a)
        c = lastly_do(b)
        log_or_whatever(c)
        return c
    except SomeException as err:
        log_exception(err)
        return None

Give up? Me too. How about here:

def f2(x):
    a = do_something(x)
    try:
        b = do_something_else(a)
    except SomeException as err:
        log_exception(err)
        return None
    else:
        c = lastly_do(b)
        log_or_whatever(c)
        return c

Better, no? Keep the scope of your try/catch blocks as narrow as possible. Yes, nested try/catch/else blocks can get annoying, but why not just refactor each of the potential culprits into separate functions?

Don't over-generalize

This should go without saying—by me, because your tooling (pep8/flake8/pylint, etc.) should be saying it at you instead. But I'm saying it anyway, because it bears repeating (and because I've seen—and written!—far too many noqa/pylint: disable=bare-except comments.)

A common excuse I've seen for catch Exception is ignorance—there's a 3rd-party API with bad documentation, or some complicated code with multiple points of failure. Unfortunately, as with the law, ignorance of potential exceptions is not a valid excuse. Do as much research as you can, add logging, and then refactor to catch the actual exceptions you want to catch.

And whatever you do, DO NOT catch BaseException. Trust me.

Lastly,

Don't be afraid to let it ride

Computers are incredibly complex; the world is a terrifying, unpredictable place; chaos is the only constant. Stuff (as it were) will happen. You don't have to catch everything; you don't even have to try. Think about what what handling every exception would even mean: if your DB can't be reached, or a config file doesn't exist, or Jeff Bezos decides overnight to get out of cloud storage and into self storage, what's the point of carrying on like your user can go about their business as normal? Five 9s is the goal, but Potemkin-oriented programming serves no-one. Print that TB, return that 500, move on.

Category Theory You're Already Using

sam — Thu, 27 Feb 2020 23:56:40 +0000

Category Theory has a reputation for being abstract, thorny, complicated, difficult, high-level, and for nerds. While this characterization is not wholly without merit, the truth is closer to that of many other "advanced" CS topics: the pool gets very deep indeed, but even the shallow end has its charms. In the same way that you don't have to fully grok the depths of the V8 engine to write JavaScript, you don't have to understand, e.g. Kan extensions to use category theory.

Let's take a tour of some categories that, as the title of this post suggests, you're already using in your everyday programming. We'll start basic, and work our way towards the legendary burrito itself, the dreaded monad. I'll describe the categories in terms of Haskell typeclasses, which are analogous to interfaces in languages like Go or Typescript, and also give some non-Haskell examples, because, again: you are already using this stuff!

Semigroup

A semigroup, in the most basic terms, is something you can stick together. In slightly more complex terms:

class Semigroup a where
    (<>) :: a -> a -> a

This looks a bit scary, but all it means that if some type is a semigroup, there's a function called <> that combines two instances of that type together. It's also specified that <> is associative, i.e. b <> (c <> d) returns the same thing as (b <> c) <> d.

You can probably think of some types you use everyday that meet this criterion, but put a pin in it because the ones you're thinking of are probably actually instances of...

Monoid

A monoid is a semigroup plus an identity element. In Haskell terms:

class Semigroup a => Monoid a where
    mempty :: a

mempty is short for monoid empty (because naming things is hard) and is the "identity element" for the monoid in question, such that mempty <> b and b <> mempty are the same as just plain b.

I didn't come up with any examples of semigroups just now, because most things that look like semigroups are actually monoids, due to the presence of the identity element:

[1,2,3] + [4,5 6] == [1,2,3,4,5,6]
[1,2,3] + [] == [1,2,3]
[] + [1,2,3] == [1,2,3]

"abc" + "def" == "abcdef"
"abc" + "" == "abc"
"" + "abc" == "abc"

1 + (2 + 3) == (1 + 2) + 3
0 + 1 == 1
1 + 0 == 1

2 * (3 * 1) == (2 * 3) * 1
1 * 2 == 2
2 * 1 == 2

The last 2 examples highlight a crucial fact about monoids, and categories in general--one cannot simply say that some type "is a monoid" or not. Rather, "being a monoid" involves not just a type, but an identity element and the <> function. Thus, positive integers "are a monoid" with mempty as 0 and <> as +, but also "are a monoid" with mempty as 1 and <> as *. Obviously, x * 0 != x and x + 1 != x; you need all the pieces.

Functor

A functor is, basically, kind of, at first approximation, a container. In different terms, a functor is something whose "contents" can be "mapped over," i.e. changed, possibly to a different type. In Haskell terms:

class Functor f where
    fmap :: (a -> b) -> f a -> f b

This also seems scary, but isn't, really. f a is a functor "containing" stuff of type a, and (a -> b) is Haskellese for "a function from type a to type b," so all fmap does is apply a function to every element in a functor. A lot of other languages drop the "f" and just call it map:

[1,2,3].map(x => x + 1) == [2,3,4]

l = [1,2,3]

def f(x):
    return x + 1

list(map(f, l)) == [2,3,4]

# or

[f(v) for v in l] == [2,3,4]

(N.B: in Python 3.x, map produces an iterator, which we have to then turn into a list. The reasons for this are beyond the scope of this post, but are worth it, believe me.)

DIGRESSIONS INCOMING, FEEL FREE TO SKIP

Digression 1

The definition above is limited to just one a, but we can define functor instances for types that "contain" more than one type! In Haskell, lists can only contain one type, but tuples can contain multiple types (and, in fact, are basically defined as all the types they contain):

one :: (Int, String) = (1, "one")
two :: (Int, String, List Int) = (2, "two", [1, 2])

Similarly, (hash) maps are defined by the types of their keys and values:

strIntMap :: Map String Int = fromList [("one", 1), ("two", 2)]
intListMap :: Map Int [Int] = fromList [(1, [1,2]), (2, [2,3,4])]

(ignore the fromList thing, that's just how Haskell displays hashmaps.)

We define functor instances for types like this by treating the right-most type as our a, and the rest of it as our f:

instance Functor (b,) where
    fmap func (x, y) = (x, func y)
instance Functor (Map k) where
    fmap func m = <definition omitted because it's scary and irrelevant>

This seems like a lot but the upside is that you can use fmap on anything that's a functor instance! You don't have to worry about .map vs .forEach vs .dictMap vs .treeMap vs whatever.

Digression 2

The reason I was so hedgy about equating functors and containers earlier is that there are things that are functors but aren't containers, really, like...functions! Yes, functions are functors! As I mentioned before, Haskell represents functions as a -> b. We can treat this like we treated other multi-type functors, like hashmaps and tuples--(a ->) becomes our f, and b becomes our a:

instance Functor (a ->) where
    fmap f2 f1 = \x -> f2 (f1 x)

\x -> ... is how Haskell represents anonymous/lambda functions, so this takes two functions and returns another function! If this blows your mind, we can retreat briefly to the container analogy. Take a function like \x -> show x (or lambda x: str(x), or x => String(x)). We could implement this function, basically, as a hashmap:

addOneFunc :: Int -> String = \x -> show x
addOneMap :: Map Int String = fromList [(0, "0"), (1, "1"), (2, "2"), (3, "3")...]

Obviously, we would never do this, because it's incredibly inefficient and basically impossible in most programming languages. But we could, and then we'd see that we could use the same functor instance we had before, where Map Int is our f, and String, on the right, is our a. Ta-da!
END OF DIGRESSIONS, YOU CAN START READING AGAIN

Applicative

Full disclosure: applicative is important in Haskell, but, like semigroup, I can't really think of how to show its use in other languages. Briefly, it's kind of a "next-level" functor, where the function(s) are also in a "container":

instance Functor f => Applicative f where
    pure :: a -> f a
    <*> :: f (a -> b) -> f a -> f b

Cool, great. Get a glass of water, and pat yourself on the back for sticking with me this long, because now we're going to move on to the fear of FP neophytes everywhere, the slayer of analogies, the big beast:

Monad

We're In It Now, so let's just dive in and start with the typeclass, as usually presented:

instance Monad f where
   return :: a -> f a
   >>= :: f a -> (a -> f b) -> f b

If return looks like pure, well, that's because it actually is. applicative was discovered and popularized after monad made it into standard Haskell, so for a while pure and return were separate. This has since been rectified, such that current Haskell has

instance Applicative f => Monad f where
    return = pure
    >>= :: f a -> (a -> f b) -> f b

Monads get a lot of pixels dedicated to them. If you get nothing else from this post, I want it to be this: monads aren't scary. Take a closer look at >>=:

>>= :: f a -> (a -> f b) -> f b

For kicks, let's specialize it...

>>= :: [a] -> (a -> [b]) -> [b]

Does it possibly remind you of anything?

Yes: it's just flatMap. That's it, that's the whole typeclass.

I'm not even joking. Yes, it turns out that you can also use monads to do IO in a type-safe and "pure" way, but that's just a bonus. It's really just flatMap, but generalizable over a whole bunch of things besides lists, in the same way that functor is. Every time you use flatMap, every time you write a nested for loop, you're using a monad. Category theory is on your side, whether you know it or not!

Postscript and further reading

Like I said, there are a lot of "monad tutorials" out there. My recommendation is: skip 'em. You know what monads are now, go out and be smug about it with my blessings.

If you do want to read more, written by people who are smarter and more eloquent than I am, I recommend the following:

Category Theory For Programmers -- An absolutely fantastic intro to Category Theory. I've read the first 2/3rds or so a half-dozen times--I've never actually finished it because it goes way far, beyond monads and into adjunctions, kan extensions, and more. Great writing, great drawings, great diagrams. It's great, read it.

Haskell Programming From First Principles -- They call it "haskellbook" for a reason. It's big, it's rigorous, it's the best intro to Haskell book out there.

Typeclassopedia -- all the typeclasses in the "core" Haskell, with their laws and raisons d'etre. Great as a reference, satisfying and edifying just to browse through.

Is Python’s asyncio Worth It?

sam — Sun, 11 Mar 2018 18:54:56 +0000

I’ve been a Python (2) dev for around 5 years now; I’m not here to bash Python. That being said, one of the language’s few failings is its poor concurrency/parallelism story. There have been attempts to fight the GIL before, like Twisted, eventlet, and the standard library’s threading module, but the results have been (IMO) overengineered, overcomplicated, and just not...nice.

Now there’s asyncio, which seems better, but still inferior to goroutines or Clojure’s core.async (which is basically goroutines), or the plethora of options presented by e.g. Rust or Haskell.

My question to the audience is: is asyncio worth it? That is:

1) if you had to start a project from scratch that you knew would involve concurrency and/or parallelism and had sufficient freedom of choice, would you choose Python 3.6 and asyncio?
2) if you had a preexisting Python project that you had to add concurrency/parallelism to, would you choose asyncio (over Twisted, eventlet, gevent, etc.)? Would it be worth porting a Python 2.x project to 3.6 for?