DEV Community: Emeka Icha

TDD: write better code in less time (a tech talk summary note)

Emeka Icha — Sat, 14 Nov 2020 09:13:29 +0000

Speaker: Evan Dorn
Link: https://www.youtube.com/watch?v=HhwElTL-mdI

Why do some programmers not write test?

Programmers are smart people. And smart people can make progress without process. Professionals in other fields first design, plan and prepare before they implement. So should programmers. TDD is a way to do this.

Note: Below points are taken verbatim from the talk. You should really watch the talk. It is awesome.

TDD(Test Driven Development) defined

Decide what the code will do.
Write a test that will pass if the code does that thing.
Run the test, see it fail. (A very important step.)
Write the code.
Refactor.

TDD Provides

design and plan before you code.
document your design.
proof that code implements design.
encourages design of testable code.

Testable code is

modular.
decoupled design.
methods of limited scope.
limits cyclomatic complexiblity.

Conclusion

Processes might feel tedious initially but they are necessary to transition from a 'programmer' to a software engineer. And they pay off in better productivity, cleaner codebase and better designed system.

Also see: TDD on wikipedia

A simple testing checklist

Emeka Icha — Wed, 04 Nov 2020 20:54:25 +0000

photo credit: Chris Reid from Unsplash.

Code without tests is broken as designed.

But testing is hard. Do you find it hard to know what to test in a piece of code? I know I do sometimes. Do you wish you had some mental or physical checklist to make it easier?

Here is one that you may find useful. Feel free to take what you may of it and remix to create your own flavour. Also please do share tips and tricks on testing you have found useful (for you see, I'm but a newbie in this thing).

Anyway, the checklist. I have put them in question form. Let your test cases answer them with those nice green checks we all love to see.

Does the code work?

By 'work', I mean does the code do what it is meant to do, according to the requirement or specification? Add some test cases that question or verifies your assumptions about the problem you are solving. A passing test is a witness you can call to defend your implementation. At least, for now.

Does the code fail? On wrong inputs?

Okay, so the code works well given the right inputs. But does it fail when given the wrong ones? And if yes, how does it fail? Does it fail gracefully or does it leave behind a trail of broken wares after failing? Beware of code that treats good and bad inputs the same. It is a ticking time bomb waiting to explode in the face of your users and make them very very unhappy.

Does it handle edge cases well?

What is an edge case? Well, let's just say for the sake of simplicity that it is the values at the extreme ends of the input type for the code. Also, they are the less-likely-to-show-up inputs to the code. But they can cause a scene when they show up. For example, if the code accepts strings as inputs, how does it handle an empty string? On the other end, how about very long strings? Or more interesting, what happens if a non-latin string shows up to the party?

Are the tests fast?

If not, why not? A unit test should be fast, I think. A usual culprit for slow unit tests is some network request going on. Find them and mock them.
There might also be other reasons for a slow test. A large test case for example.

Conclusion

There it is. A simple checklist for testing that I bet, is not hard to follow. Thanks for reading.

Fun little fact: I wrote this article and lost it when I refreshed the page. I had assumed (and wrongly so) that the editor was auto-saving as I wrote along. If only I had tested that assumption :))

Automate the most boring thing with Python's subprocess module

Emeka Icha — Thu, 21 Mar 2019 23:51:16 +0000

You have heard of the subprocess module in the Python Standard Library, haven't you? How it:

allows you spawn new processes, connect to their input/output/error pipes, and obtain their return codes.

Not only that, it also intends to replace older modules and functions like os.system and os.spawn*, which to be honest, I have no experience using.

Basically, the subprocess module allows you to launch other programs from your code and drive them to whatever end you desire. Think of doing fun things like:

controlling a browser from your code via the webbrowser module or even a third party library like selenium
automating server deployments or administration tasks with Fabric
Running an android emulator from your code.
Yeah, I mean, you get the idea!

The documentation has all the information you need if you desire a deep dive. If that is you, thank you for coming :)

Those of us who just want to stand by the riverbank and scoop some water are the ones reading this sentence and the ones below it.

Jokes aside, what brought us to this point is that I have a couple of repositories I collaborate on with other colleagues. All these repos live in the Projects subdirectory inside my home directory. The goal is to be able to run a git pull on all those repos at the beginning of my work day automatically. The other alternative will be to cd into a project directory, run git pull, cd out and into the next one, etc. That, my friends, is very boring!

Without much further ado, here is the code:


from subprocess import check_output, CalledProcessError
from pathlib import Path

project_dir = Path.home() / 'Projects' / 'work'

for d in project_dir.iterdir():
    if d.is_dir(): # this check is not really needed
        try:
            out = check_output(["git", "pull", d], cwd=d)
            res = out.decode()
            print(res.splitlines()[-1])
        except CalledProcessError as exc:
            pass

In a nutshell, I am going through all the directories(repos) in my work directory, running a git pull on them. I am also decoding the output(by default output is in bytes) and printing the last element in the output list(calling res.splitlines turns the output into a list).
If there is an error, I pass it like it didn't happen.

That is all, believe me! I can run that little script from anywhere and have my repos updated.
Maybe I can even take this further by checking the output to see if there was really an update on the repo. If so, I can open that particular repo in a code editor or on a browser. But what's the point?

Automate the boring stuff with Python 3's pathlib module.

Emeka Icha — Sat, 09 Mar 2019 14:50:07 +0000

The Pathlib module was introduced in Python 3.4 with the aim:

to provide a simple hierarchy of classes to handle filesystem paths and the common operations users do over them.

In this article, I intend to go through some of these common operations. Towards the end, I show how I keep the downloads folder on my computer clean and organized using the Pathlib module and some cron job. Let us get started.

As per the documentation:

If you’ve never used this module before or just aren’t sure which class is right for your task, Path is most likely what you need. It instantiates a concrete path for the platform the code is running on.

That is understood. Let's import the Path class then.

In [9]: from pathlib import Path

If you are following along with an interactive shell like Ipython or any other, it is usually a good idea to dir the imported class or object to get a clue on the attributes and methods the class provides.

Get the user home directory

# returns PosixPath('/home/mekicha')
# This is equivalent to os.path.expanduser('~')
home_dir = Path.home()

Get a directory in the home directory

# returns PosixPath('/home/mekicha/Downloads')
# Equivalent to: home_dir.joinpath('Downloads')
downloads_dir = home_dir / 'Downloads'

downloads_dir.exists() # True

Now, at this point, you might be thinking: "Is this all there is to this click-baiting article that promised to show me practical examples of using the Pathlib module?". Before you close your tab in fury at the waste of time and bandwith, I offer a calm answer to your urgent question: "No, friend, that is not all".

In fact, if that was all, I would have just pointed you to the documentation where such and more examples abound.
I just provided the examples above as some form of a ground work for the better things ahead(hopefully).
Now, what do you think about the following examples.

Given a directory, calculate the total size of all files

from pathlib import Path

def get_size(directory: Path):
  """ Given a directory, return total size of files in bytes """
  total_size = 0
  for file in directory.iterdir():
     total_size += file.stat().st_size  # in bytes
  return total_size

if __name__ == '__main__':
  # Let's get the size of my home directory
  directory = Path.home()
  size_in_byte = get_size(directory)

  # Nice to have: convert the size to megabyte or gigabyte

  mega = 1 << 20
  giga = 1 << 30 
  if size_in_byte / giga < 1:
    if size_in_byte / mega < 1:
      print(f'file size = {size_in_byte} bytes')
    print(f'file size = {size_in_byte / mega} megabytes')
  print(f'file size = {fize_in_byte / giga} gigabytes'

Given a directory, return the file with the biggest size

def get_max_file(directory: Path):
  return max((f.stat().st_size, f) for f in directory.iterdir())

# In the same vein, let's get last modified file in a directory

def get_last_modified(directory: Path):
  return max((f.stat().st_mtime, f) for f in directory.iterdir())

Given a directory, return a list of subdirectories in the directory

def get_dir_list(directory: Path):
  return [child for child in directory.iterdir() if child.is_dir()]

Organizing my computer's download folder

If you have read this far, hopefully, this is the beginning of what you came for.

My download folder used to be a mess. Files of different formats a lie side by side with one another in a helpless disorderly manner, each one crying for my attention, and perhaps feeling neglected when I did not click on them. Not anymore. Today, help has come their way.

I am going to put the files in their places, where they should belong. Images should go to the images sub-folder, documents(think pdf, doc, xls) should live in the documents folder and it is right for zipped files to dwell in the zipped folder. I will assign a cron agent to do the job.

The cron agent would be setup to run once every day, at a time when I should be asleep.

Here is the code:

#!usr/bin/env python3

import time
import logging
from pathlib import Path

logging.basicConfig(level=logging.DEBUG)

logger = logging.getLogger(__file__)


def organize_directory(directory: Path):
    """ Take a path to a directory and organizes the files there
        into subdirs.
     """

    sub_dirs = ['images', 'docs', 'zipped','media', 'others']
    images = ['.png', '.jpeg', '.jpg', '.gif']
    zipped = ['.zip', '.gz', 'tgz']
    docs = ['.doc', '.docx', '.pdf', '.xls', '.xlxs', '.PPT', '.txt']
    media = ['.mp4', '.mkv', '.avi', '.mp3']

    # create base subdirs.
    # Map each subdir to its subdir path
    dir_map = {}
    for subdir in sub_dirs:
        subdir_path = directory.joinpath(subdir)
        subdir_path.mkdir(parents=True, exist_ok=True)
        dir_map[subdir] = subdir_path

    start_time = time.monotonic()
    for child in directory.iterdir():
        child_name = child.name
        if child.is_file():
            if child.suffix in images:
                target = dir_map['images'].joinpath(child_name)
                child.replace(target)
                logger.info(f'moved {child} to images folder')
            elif child.suffix in docs:
                target = dir_map['docs'].joinpath(child_name)
                child.replace(target)
                logger.info(f'moved {child} to docs folder')
            elif child.suffix in zipped:
                target = dir_map['zipped'].joinpath(child_name)
                child.replace(target)
                logger.info(f'moved {child} to zipped folder')
            elif child.suffix in media:
                target = dir_map['media'].joinpath(child_name)
                child.replace(target)
                logger.info(f'moved {child} to media folder')
            else:
                target = dir_map['others'].joinpath(child_name)
                child.replace(target)
                logger.info(f'moved {child} to others folder')

    end_time = time.monotonic() - start_time
    logger.info(f'Organizing directory took {end_time} seconds')



if __name__ == '__main__':
    download_dir = Path.home().joinpath('Downloads')

    organize_directory(download_dir)

Again, the documentation entries for iterdir, mkdir and replace are very clear on what they do.

Running the above script gave me this:

INFO:download_folder_agent.py:Organizing directory took 0.01179114996921271 seconds

That's not all. It has also given me a much better organized downloads folder.

Setup the cron job

Setting up the cron job was fairly easy.

Here are the steps I took that gave me what I wanted:

Open the crontab editor by typing: crontab -e
Look up the crontab schedule expression on [crontab-guru][crontab guru]. I choose everyday at 5 am. 0 5 * * *
Adding the following job entry in the file: `0 5 * * * python3 /home/mekicha/agent/download_folder_agent.py >> /home/mekicha/agent/log 2>&1
I saved the changes and exited the file. 5.Voila! Everything was working.

Actually, I first set the schedule to run every minute * * * * *, checked that it was working before I committed the above schedule.

I set the schedule to run every day by 5 am, which is not really ideal. It's not like I am downloading stuff every now and then. But I will leave it at that for now. Feel free to choose whatever seems best to you.

If you are wondering what does this 2>&1 thing mean, this stackoverflow answer was what I read when I was wondering that too.

Okay, I think this is a good place to bring this whole business to an end.
In the next episode of automating the boring stuff, I will show how I keep the downloads folder clean by removing files that have outlived their usefulness.

Thank you!

P.S. This automate the boring stuff article series is an imitation of a book by the same name.