loading...

Python for Bash

_andy_lu_ profile image Andy Lu ・4 min read

This was originally published on my blog.

Do you like bash scripts? Personally, I don't.

So when I need to write bash scripts, I figure out the commands I need, then
glue them together with Python.

It's been a while since I've needed to do this and while I neglected it before,
the subprocess module is the best way to run these commands.

A Quick Intro to Python's subprocess.py

Development Environment

If you are following along with me here, you'll want to be using at least python 3.5. Any version before that and you'll have to use a different API in this module to do the things I'll show you.

The Command

The workhorse of this module is the subprocess.Popen class. There are a ton of arguments you can pass this class, but it can be overwhelming- and not to mention overkill- if you're new to this.

Thankfully, there's a function in the subprocess module that we can interface with instead: subprocess.run().

Here's the function signature with some typical arguments passed in. (I pulled this from the Docs)

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None,
shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None,
text=None, env=None)*)

That looks pretty complicated, but we can actually ignore most of it and still do pretty neat things. Let's look at some examples.

A Basic Example

import subprocess as sp

result = sp.run("pwd")
print(result)

The output:

/this/is/the/path/to/where/my/terminal/was/
CompletedProcess(args="pwd", returncode=0)

The output of this is the path to the directory you ran this script from; exactly what you would expect. Then there's some CompletedProcess object. This is just an object that stores some information about the command that was run. For this guide, I'm ignoring it, but I'll have links at the end where you can read all about it.

But that's it! That's all you need to run some basic bash commands. The only caveat is you'll be lacking some features of a shell.

To overcome this, let's look at the next example.

A Better Example

import subprocess as sp

result = sp.run("ls -lah > someFile.txt", shell=True)
output = sp.run('ls -lah | grep ".txt"', shell=True)

You may have noticed earlier in the function signature that shell=False, but here I set it to True. By doing so, the command I want actually gets run in a shell. That means I have access to redirection and pipes like I've shown.

A note on running things like this: the command you want to execute must be typed exactly the way you would if you were doing it on a shell. If you read through the Documentation, you'll notice there is a way to run commands as by passing in a list of strings, where each string is either the command or a flag or input to the main command.

I found this confusing because if you follow my "Better Example" way, you are never left wondering if you passed in the arguments correctly. On top of that, you are free to use Python to build up a command based on various conditions.

Here's an example of me doing just that.

A "Real World" Example

#!/usr/bin/env python3

###############################################################################
#                                   Imports                                   #
###############################################################################
import subprocess as sp
from datetime import date

###############################################################################
#                                  Functions                                  #
###############################################################################

def getTodaysDate():
  currDate = date.today()
  return f"{currDate.year}-{currDate.month}-{currDate.day}"

def moveToPosts():
  lsprocess = sp.run("ls ./_drafts", shell=True)
  fileList = lsprocess.stdout.decode('utf-8').strip().split("\n")
  hasNewPost = len(fileList)

  if (hasNewPost == 1):
      print("New post detected")

      srcName = "./_drafts/" + fileList[0]
      destName = " ./_posts/" + getTodaysDate() + "-" + fileList[0]

      command = "mv "+ srcName + destName
      sp.run(command, shell=True)

      return [destName, files[0]]

  elif hasNewPost == 0:
      print("Write more!")
  else:
      print("Too many things, not sure what to do")

def runGit(fullPath, fileName):

  commitMsg = "'Add new blog post'"

  c1 = "git add " + fullPath
  c2 = "git commit -m " + commitMsg

  cmds = [c1,c2]

  for cmd in cmds:
    cp = sp.run(cmd, shell=True)

if __name__ == "__main__":
  pathToPost, fileName = moveToPosts()
  runGit(pathToPost, fileName)
  print("Done") 

Since this blog is running thanks to Jekyll, I took advantage of the _drafts folder available to me.

For those of you unfamiliar with Jekyll, _drafts is a folder where you can store blog posts that aren't ready to be published yet. Published posts go in _posts.

The filenames in this folder look like: the-title-of-my-post.md. The filenames for published post that sit in the _posts folder have the same name, but with the year-month-day- attached to the front of the draft name.

With this script, I just have to write a post and drop it into _drafts. Then I open a terminal and run this script. First it looks in _drafts and makes an array of the filenames it found. Anything other than just finding one file will stop the script- I'll improve this one day. With that file name and the help of subprocess.run(), the script moves the draft into _posts, gives it the appropriate name, then commits it to git for me.

Wrap Up

I introduced the subprocess.run() function, gave 3 examples of running bash commands with it, and ended with the script that inspired this post in the first place.

I personally don't have too many uses for bash scripts. When I need one though, I'll definitely be writing it in Python and if it suits your needs, you should too.

Further Reading

Posted on by:

_andy_lu_ profile

Andy Lu

@_andy_lu_

A Python developer with a knack for math. TKD black belt, ask me about 2013. Applied Math Alum @SMU, but I can tell you all about commutative rings too. https://www.pbk.org/

Discussion

pic
Editor guide
 

Great article! Although I think it's worth mentioning that (for me, at least) using Python to call subprocess.run a bunch of times is arguably more work than using a bash script. When done right, bash really isn't that bad! I also try to find the "pythonic" way to do things when I can, mostly using the os and shutil packages to do things like list directories, make directories and files, etc. Using native Python calls tends to make things perform better and make them easier to read through.

 

Thanks, glad you enjoyed it!

I concede to the point that doing all of this with Python is probably more work. I don't think bash is that bad either. The main point of this article was that I'm too lazy to type out essentially 4 commands every time I want to post something to my blog; on top of that, there's some small details regarding file names that need to be observed. Rather than going to learn about doing it in bash and getting the date to be in a certain format, I thought it would be quicker to do it in Python. In the spirit of learning something new, I remembered some subprocess module that is supposed to replace the os.system() function that I am used to. So I thought I would teach myself and write it up for anyone else to benefit from.

I hope that clears it up a bit. I'm not here to make performance claims, just to leave a guide out there for people who have the same problem as me one day. Thanks again for the read!

 

Oh definitely! Having the power of Python mixed with bash or any shell scripting is an awesome combo, especially when you get into anything more than a single if statement really or dealing with fun stuff like date formatting.

 

Thanks for the nice introduction to the subprocess module. Have you looked at the pathlib module? It provides an object-oriented way to interact with the filesystem. For example, instead of parsing the stdout from ls, you can get the directory contents using pathlib:

from pathlib import Path
...
fileList = list(Path(./_drafts).iterdir())

You can also use it to move the draft to the destination.

if len(fileList) == 1:
    ...
    src = fileList[0] # already a Path object
    dst = Path(./_posts, getTodaysDate() + - + src.name)
    src.rename(dst)

So pathlib can replace all the file system subprocess calls here! (But you’d still need subprocess for the git calls, of course. There are libraries for that, but nothing in the standard library.)

It’s part of Python 3 and available on pypi for Python 2.

 

I should be the one thanking you, this is so cool!

 

Can you tell me the difference between these codes?

result = subprocess.check_output('ls ./drafts', shell=True)
lines = result.splitlines()
    for line in lines:
        line = str(line, 'utf-8')
lsprocess = sp.run("ls ./_drafts", shell=True)
fileList = lsprocess.stdout.decode('utf-8').strip().split("\n")
 

I caught a bug in my code! Thank you for pointing it out!

In order for Example 2 to run properly, it should read:

lsprocess = sp.run("ls ./_drafts", shell=True, stdout=sp.PIPE)

Running the snippet in the interpreter really quick confirms it. lsprocess.stdout doesn't even exist without the stdout=sp.PIPE part.

 

Nothing as far as I can tell.

Example 1 uses subprocess.check_output(), but Example 2 uses the CompletedProcess class in subprocess.

What difference am I missing here?

 

This is awesome. Python over bash all day long.
Here's my version, inspired by Andy Lu: github.com/kavishgr/My-Python-Scri...