DEV Community: Mark Edosa

How to Download Multiple Files with Pure Python: A Beginner Guide

Mark Edosa — Thu, 29 Aug 2024 09:42:09 +0000

Introduction

Lately, I've been interested in compilers and thankfully, the excellent Standford University did a great job of providing materials (handouts, assignments, and so on) on the subject here.

The problem is, the files are pretty much (which is a good thing) but I didn't want to click click click. Much less repeat these actions for other similar pages. Another issue was laziness. Well, you get the idea. Finally, I didn't want to create a virtual environment, install beautiful soup, and requests library for something relatively trivial. Too much work!

So to save me stress and probably you, since you're here, here's what I did.

Example 1: Downloading Text Files

Here's the first page. It contains a list of C codes accompanying the book foundations of computer science by Al Aho and Jeff Ullman.

Here's the entire code. Don't worry, the explanations are in the comments. Please, read the code along with the comments.

# Import the HTMLParser to extract links
from html.parser import HTMLParser

# import this for making HTTP requests
import urllib.request as request


# The link to the page. Very obvious huh?
url = 'http://i.stanford.edu/~ullman/fcsc-figures.html'


# Make the get request to the web page
# Read the response as a text string (utf-8 encoded)
with request.urlopen(url) as response:
    content = response.read().decode('utf-8')


# You want to subclass the HTMLParser and override the
# handle_starttag method. We're interested in HTML links <a>
# Each link contains a tuple ('href', URL) so we're taking the second element.
# We need only URLs ending with .txt
class ExtractLinks(HTMLParser):
    links = []
    base_url = 'http://i.stanford.edu/~ullman'

    def handle_starttag(self, tag, attrs) -> None:
        if tag == 'a':
            for attr in attrs:
                link = attr[1]
                if link.endswith('txt'):
                    self.links.append(f'{self.base_url}/{link}')


# Create an instance of your class and feed it the HTML text
# feed huh? great naming I must say!
parser = ExtractLinks()
parser.feed(content)

# This isn't so necessary. 
# I just wanted a sorted copy of the links attribute.
# I don't dislike mutation. The X-men were super cool.
links: list[str] = parser.links[:]
links.sort()

# Lastly, iterate through the links and make a get request to each link
# Read the response as text.
# Split the link and extract the filename. The filename is the last element of your split.
# Rename if you want. I wanted a .c extension instead of a .txt
# Open a file for writing (as text) and we are good to go.
for link in links:
    with request.urlopen(link) as response:
        file = response.read().decode('utf-8')
        filename = link.split('/')[-1]
        filename = filename.replace('.txt', '.c')

        with open(filename, 'wt', encoding='utf-8') as f:
            f.write(file)

Here's what it will look like:

Example 2: Downloading Multiple Binary Files (pdf, zip)

Same scenario as above with slight modifications.

# import the HTMLParser to extract links
from html.parser import HTMLParser

# Import this for making HTTP requests
import urllib.request as request

# For something fancy
import os

# Like before, make the get request to the web page
# Read the response as a text string (utf-8 encoded)
url = 'https://web.stanford.edu/class/archive/cs/cs143/cs143.1128/'

with request.urlopen(url) as response:
    content = response.read().decode('utf-8')


# This time I only want pdf and zip files.
# Well, one pdf link started with http but I didn't need it.
class ExtractLinks(HTMLParser):
    links: list[str] = []

    def handle_starttag(self, tag, attrs) -> None:
        if tag == 'a':
            for attr in attrs:
                _, link = attr # another way to extract tuple elements
                if link.startswith('http'):
                    continue
                if link.endswith('.zip') or link.endswith('.pdf'):
                    self.links.append(link)


# We've been here before! Pompeii by Bastille.
parser = ExtractLinks()
parser.feed(content)
links: list[str] = parser.links[:]
links.sort()


for link in links:
    # A link might be handouts/something.pdf or assignments/10/pdf
    parts = link.split('/')

    # Collect the first and last parts to aid file organization
    directory, filename = parts[0], parts[-1]

    # The file path is the combined directory/folder and the filename
    filepath = f"{directory}/{filename}"

    # If a file has been downloaded, why repeat the task?
    # So skip it.
    if os.path.exists(filepath):
        print('Skipping', filepath, "....")
        continue

    # Create the full URL
    link_url = url + link

    # urllib didn't like the spaces in some of the links
    # so replace it with the encoded form.
    link_url = link_url.replace(' ', '%20')

    with request.urlopen(url_link) as response:
        # read yields a binary string. This is what we want here.
        # No decoding is necessary.
        file = response.read()


        # Okay. So, the directory may not have been created.
        # But EAFP. Easier to Ask for Forgiveness than Permission.
        # Try creating the file. It fails if the directory does not exist.
        # In the except block, create the directory.
        # Finally, the write mode should be 'wb' - write binary.
        try:
            with open(filepath, 'wb') as f:
                # Keep track of what's happening for the sanity sake
                print(f"writing {filepath}")

                f.write(file)
        except OSError as e:
            print(e) # Directory does not exist error.
            print('======== creating it ===========')

            # Create the directory
            os.mkdir(directory)

            # This may potentially fail but not because of unavailable 
            # directory. But, it's okay since I'm not saving the planet.
            with open(filepath, 'wb') as f:
                f.write(file)

Here we go again :)

I had fun!

Introduction to Python Programming - Classes

Mark Edosa — Fri, 23 Feb 2024 19:40:34 +0000

Table of Contents
Introduction
Creating a Class
Creating Instance Objects
Class and Instance Variables
Using Dunder init for Class Invariance
Class and Instance Methods
Static Methods
Data and Implementation Hiding
Inheritance
- Single Inheritance
- The super() Function
- Multiple Inheritance
Printing and Debugging An Object
Data Classes
Summary

Introduction

Classes are a classic feature of objected-oriented programming, a programming paradigm that structures code as objects with attributes (variables) and behavior (functions or methods). The attributes and functions usually form a single logical unit. For example, you may define a Person class to represent the attributes and behavior of a person.

You can also see a class as a blueprint for creating objects, similar to how architectures are blueprints for making concrete buildings. The objects are called instances of that class.

Unlike object-oriented languages like Java, Python lets you use classes or not. You can choose between using the variables and functions individually or group them under a class.

Before proceeding, I assume you understand basic Python programming, including variables and functions.

Creating a Class

Use the class keyword followed by the class name and a colon. For example:

class Restaurant:
    # ...

While you can use any valid naming pattern to define your class names, by convention, classes are defined using Pascal cases. For example, Animal, MyBook, ItalianRestaurant, etc.

Next, include the attributes and methods that are either attached to the class itself or the instances of the class. For example:

class Restaurant:
    """A restaurant class"""

    count: int = 0

    def __init__(self) -> None:
        self.name: str = ''
        self.address: str = ''
        self.type: str = ''
        self.capacity: int = 1
        self.spacing: float = 1.0


    def get_name(self) -> str:
        return self.name

    def set_name(self, name: str) -> None:
        self.name = name

    @classmethod
    def increase_count(cls) -> None:
        cls.count += 1


    @staticmethod
    def describe() -> None:
        print('My awesome Restaurant')

The Restaurant class has the following:

A docstring """A restaurant class""" accessible through __doc__ attribute
A class attribute count, an integer directly attached to the Restaurant class and shared by all instance objects.
An __init__() method where instance variables are created and or initialized
Five instance attributes or variables - name, address, type, capacity, and spacing are attached to each unique instance of the Restaurant class.
Two instance methods - get_name() and set_name(). An instance method must take itself (self) as its first argument.
A class method increase_count() which is attached to the Restaurant class. A class method must take the class itself as its first argument.
An independent static method describe() which is in the class basically for the sake of grouping. This means that static methods can exist as regular functions outside the class. Note that you can access a static method via the class object and an instance object.

In Python, a class, like its instances, is also an object.

Defining a class automatically a class object. You can access its attributes using the ClassName.name syntax. For example:

# Print some of the class attributes
print(Restaurant.count)
print(Restaurant.__doc__)
print(Restaurant.increase_count)
print(Restaurant.describe)

# You can also add new attributes to a class
# However, do not do this. It is better to include
# your variables in the class definition
Restaurant.age = 10
print(Restaurant.age)

# Delete the added attribute 
del Restaurant.age
# print(Restaurant.age) # Throws AttributeError

# call the class method
Restaurant.increase_count()
print(Restaurant.count)

# call the static method
Restaurant.describe()

0
A restaurant class
<bound method Restaurant.increase_count of <class '__main__.Restaurant'>>
<function Restaurant.describe at 0x000002882CECE3E0>
1
My awesome Restaurant

0x000002882CECE3E0 is the current memory address of the function Restaurant.describe(). The memory address usually changes each time you run the Python program.

Creating Instance Objects

You can also create an instance object using the ClassName() syntax. For example:

italian_rest = Restaurant()

# Print the instance object
# Prints the memory address at the time of creation
print(italian_rest)
# Check the type
print(type(italian_rest))

# Create another independent instance of Restaurant
chinese_res = Restaurant()
print(type(chinese_rest))

# Instances are independent or unique
print(italian_rest == chinese_res)

<__main__.Restaurant object at 0x00000229E1AB78F0>
<class '__main__.Restaurant'>
<class '__main__.Restaurant'>
False

Checking the types shows that both italian_rest and chinese_rest objects are Restaurant objects. Also, you can use the built-in isinstance() function to determine if an object was created from a class. For example:

print(isinstance(italian_rest, Restaurant))
print(isinstance(chinese_res, Restaurant))
print(isinstance(chinese_res, list))

True
True
False

The instance variables and methods (including static methods) can be accessed using the instance.name syntax. For example:

print(italian_rest.capacity)
print(italian_rest.spacing)

italian_rest.describe()
print(italian_rest.name) # empty string
italian_rest.set_name('Italian restaurant')
print(italian_rest.name)

1
1.0
My awesome Restaurant

Italian restaurant

Class and Instance Variables

Looking at the Restaurant class, count is a class variable while name, address, and so on are instance variables.

All instances of a class share class variables while instance variables are unique to each instance object created from the class. Each time you create a new instance of Restaurant, a call to Restaurant.increase_count() or self.increase_count() within the __init__() method increases count by one.

class Restaurant:
    """A restaurant class"""

    count: int = 0

    def __init__(self) -> None:
        self.name: str = ''
        self.address: str = ''
        self.type: str = ''
        self.capacity: int = 1
        self.spacing: float = 1.0

        Restaurant.increase_count()
        # self.increase_count()

    # ... Elided for brevity


italian = Restaurant()
chinese = Restaurant()

print(Restaurant.count) # 2
print(italian.count) # 2
print(chinese.count) # 2

Note that while count also appears to be accessible via the instances - chinese and italian, it's not an instance variable. The __dict__ attribute (a dictionary) of an instance contains the variables attached to an instance.

print(italian.__dict__)
print(chinese.__dict__)

{'name': '', 'address': '', 'type': '', 'capacity': 1, 'spacing': 1.0}
{'name': '', 'address': '', 'type': '', 'capacity': 1, 'spacing': 1.0}

count is not in __dict__. You could "hide" or override a class variable by adding the same variable directly to an instance object. Don't do this:

italian.count = 3

print(italian.count) # 3
print(chinese.count) # 2
print(italian.__dict__)
print(chinese.__dict__)

A new instance variable count on the italian instance now overshadows the class variable count.

3
2
{'name': '', 'address': '', 'type': '', 'capacity': 1, 'spacing': 1.0, 'count': 3}
{'name': '', 'address': '', 'type': '', 'capacity': 1, 'spacing': 1.0}

Deleting the instance count makes the class count re-appear.

del italian.count
print(italian.count) # 2
print(italian.__dict__)
# {'name': '', 'address': '', 'type': '', 'capacity': 1, 'spacing': 1.0}

In Python, instance variables, like class variables, must be initialized upon creation inside the __init__() method. The string variables such as name are currently empty strings '' while capacity and spacing are int and float types, respectively. You could modify the instance variables after creation. For example:

from pprint import pp # pretty print

print(italian.name)
print(italian.address)

italian.name = 'Francesco T. Burgers'
italian.address = "21st Navy Avenue, Italy."
italian.capacity = 80
italian.type = 'Italian restaurant'
italian.spacing = 18.2

print(italian.__dict__)
print(italian.name)
print(italian.address)
print(italian.capacity)



{'name': 'Francesco T. Burgers',       
 'address': '21st Navy Avenue, Italy.',
 'type': 'Italian restaurant',
 'capacity': 80,
 'spacing': 18.2}
Francesco T. Burgers
21st Navy Avenue, Italy.
80

Modifying each instance variable feels like a lot of work. Luckily, the __init__() can take any number of arguments you can use to set instance variables. For example:

class Restaurant:
    # ...

    def __init__(self, name: str, address: str, type: str,
                 capacity: int = 1, spacing: float = 1.0):
        self.name = name
        self.address = address
        self.type = type
        self.capacity = capacity
        self.spacing = spacing

        Restaurant.increase_count()

    # ...

You can now create an instance and set appropriate values upon creation. Recreating the italian restaurant you get:

italian = Restaurant('Francesco T. Burgers', '21st Navy Avenue, Italy.',
                     'Italian restaurant', 80, 18.2)

print(italian.name)
print(italian.address)
print(italian.capacity)
print(italian.type)
print(italian.spacing)

Francesco T. Burgers
21st Navy Avenue, Italy.
80
Italian restaurant
18.2

How about the chinese instance:

chinese = Restaurant('Jet Li Spot', '31st Broadway Palace',
                     'Chinese restaurant', 80, 16.2)


print(chinese.name)
print(chinese.address)
print(chinese.capacity)
print(chinese.type)
print(chinese.spacing)

Jet Li Spot
31st Broadway Palace
80
Chinese restaurant
16.2

Using Dunder init for Class Invariance

You can also set constraints on instantiating a class in the __init__() method. For example, it might not make sense for a restaurant to have no name or address or have a capacity or spacing of zero. You can include checks to ensure that nobody can create such a class.

The concept is known as establishing class invariance. A state where an object of a class meets certain expectations that never change throughout its lifetime. For example, we use the __init__() to ensure Restaurants are never created with empty strings or with zero(0) and negative spacing and capacity.

class Restaurant:
    """A restaurant class"""

    count = 0

    def __init__(self, name: str, address: str, type: str,
                 capacity: int = 1, spacing: float = 1.0):

        invalid_name = self._invalid_str(name)
        invalid_address = self._invalid_str(address)
        invalid_type = self._invalid_str(type)

        if invalid_name or invalid_address or invalid_type:
            raise ValueError('Please provide a non-empty string for name, address and type')

        if capacity <= 0 or spacing <= 0:
            raise ValueError('capacity and spacing cannot be zero or less')


        self.name = name
        self.address = address
        self.type = type
        self.capacity = capacity
        self.spacing = spacing

        self.increase_count()

    # ...

    def set_name(self, name):
        if self._invalid_str(name):
            raise ValueError('Name must be string and cannot be empty')
        self.name = name


    def _invalid_str(self, value):
        return not isinstance(value, str) or value == ''

    # ... Elided for brevity

The set_name() method also has a check to ensure non-violation of the class invariance.

# Works
Restaurant('Jet Li Spot', '31st Broadway Palace',
                     'Chinese restaurant', 80, 16.2)

# Invariance violated
# Fails: empty name, address
Restaurant('', '', 'Italian', 18, 17.3)
# ValueError: Please provide a non-empty string for name, address, and type

# Invariance violated
# Fails: capacity is 0
Restaurant('Jet Li Spot', '31st Broadway Palace',
                     'Chinese restaurant', 0, 16.2)
# ValueError: capacity and spacing cannot be zero or less

Class and Instance Methods

A class method, like a class variable, is associated with a class. The method has an annotation @classmethod on top of it. Its first argument is the class itself. See the increase_count(cls, by) method defined earlier. You can use the word cls as the first argument since the class keyword is reserved.

You can use ClassName.class_method() within or outside the class or self.class_method() within the class. For example:

class Restaurant:
    # ... Elided for brevity

    def __init__(
        #  ... Elided for brevity
        ):

        # ... Elided for brevity

        # Restaurant.increase_count()
        self.increase_count()

    @classmethod
    def increase_count(cls):
        cls.count += 1


Restaurant.increase_count()
chinese.increase_count()
italian.increase_count()

print(Restaurant.count) # 5

As you've seen above, each instance can also call a class method. However, all class instances share the result of the call.

Static Methods

A method marked with @staticmethod is a static method. As stated earlier, a static method can exist as a function outside a class. While a class or instance can call a static method, for example, MyClass.static_method() or my_instance.static_method(), the static method does not take the class cls or instance self as its first argument.

Essentially, it is not necessary to have a static method in a class. For example, the describe() method could be a regular function in the module:

def describe() -> None:
    print('My awesome Restaurant')

class Restaurant:
    # ...

Data and Implementation Hiding

Unlike Python, some programming languages like C#, Java, and Cpp have public, private, and protected variables or methods. This helps programmers hide data (private variables) and implementation details (through private methods). To mimic this behavior, many Python programmers, by convention, prefix "private" variables and methods with underscores (_). For example:

class MyClass:

    def __init__():
        self._my_private_variable = 0


    def _my_private_method(self):
        pass

While all instance methods and variables (whether prefixed or not) can still be accessed or modified. However, many people follow the convention. That is usually enough.

Inheritance

A class can inherit attributes and behavior from another class. This is known as inheritance in object-oriented programming. You usually want to do this to extend or modify the behavior of a class. The class inheriting is referred to as a "child" or "sub" or "derived" class while its parent, the "parent" or "base" class.

In Python, a base class can have one or multiple children or subclasses (single inheritance). A child class can have multiple parents (multiple inheritance). The syntax for inheritance in Python is class Child(ParentA, ..., ParentZ):

Single Inheritance

A class class can inherit from a single parent. All classes you create implicitly inherit from the built-in object class. This means that class Restaurant: is equivalent to class Restaurant(object):. For example:

class Restaurant(object):
    # ...

Some people like to be explicit, but I don't think this is necessary.

Add a cook_meal() method to the Restaurant class. You will expect the implementation of this method in an italian instance to be different from a chinese instance or any other instance since they serve different meals.

class Restaurant:
    # ...

    def cook_meal():
        # raise NotImplementedError so that each
        # subclass must override this method
        raise NotImplentedError

While you could use an if statement that checks the type of a restaurant, the easiest way to specialize the cook_meal() is to create subclasses representing each kind of restaurant. For example:

class ItalianRestaurant(Restaurant):

    def cook_meal(self):
        self._make_italian_dishes()


    def _make_italian_dishes(self):
        """Meant to be private"""
        print("I am making Carbonara, Lasagna, and Pizza!")


class ChineseRestaurant(Restaurant):

    def cook_meal(self):
        self._make_chinese_dishes()


    def _make_chinese_dishes(self):
        print("I am making Kung Pao Chicken, Fried Rice, and Hot Pot!")


class NigerianRestaurant(Restaurant):

    def cook_meal(self):
        self._make_nigerian_dishes()


    def _make_nigerian_dishes(self):
        print("I am making Egusi soup, Pounded yam, and Pepper soup")

You can create new instances like you would a Restaurant class. The variable initialization stays the same. For example:

italian = ItalianRestaurant('Francesco T. Burgers', '21st Navy Avenue, Italy.',
                     'Italian restaurant', 80, 18.2)

chinese = ChineseRestaurant('Jet Li Spot', '31st Broadway Palace, China.',
                     'Chinese restaurant', 80, 16.2)


print(type(italian))
print(isinstance(italian, Restaurant))
print(isinstance(italian, ItalianRestaurant))
print(issubclass(ItalianRestaurant, Restaurant))

print(type(chinese))
print(isinstance(chinese, Restaurant))
print(isinstance(chinese, ChineseRestaurant))
print(issubclass(ChineseRestaurant, Restaurant))

<class '__main__.ItalianRestaurant'>
True
True
True
<class '__main__.ChineseRestaurant'>
True
True
True

Like the isinstance() function, you can use the issubclass() function to check if a class is a subclass.

You can now call the cook_meal() of each instance to get specialized meals. For example:

italian.cook_meal()
chinese.cook_meal()

I am making Carbonara, Lasagna, and Pizza!
I am making Kung Pao Chicken, Fried Rice, and Hot Pot!

Notice that each subclass has its own _make_<country>_dishes() unique to that class. The other variables and methods are inherited as is:

print(chinese.address)
print(chinese.type)
print(chinese.spacing)
print(chinese.get_name())
chinese.describe()

31st Broadway Palace, China.
Chinese restaurant
16.2
Jet Li Spot
My awesome Restaurant

Inheritance can be as deep as you want. That is to say that a child class can also be the base class of another class. For example:

class RomanRestaurant(ItalianRestaurant):
    pass


roman = RomanRestaurant('Roman T. Burgers', '21st Navy Avenue, Rome.',
                            'Italian restaurant', 80, 18.2)

# Type and instance checks
print(type(roman))
print(isinstance(roman, Restaurant))
print(isinstance(roman, ItalianRestaurant))
print(isinstance(roman, RomanRestaurant))

# Subclass checks
print(issubclass(RomanRestaurant, ItalianRestaurant))
print(issubclass(RomanRestaurant, Restaurant))

<class '__main__.RomanRestaurant'>
True
True
True
True
True

Too many levels deep can be problematic. Remember to keep it simple.

The super() Function

What if you need to add new instance variables to a derived class or call a parent class method within a derived class? Enter the super() function. For example:

class ParentClass:

    def __init__(self, name, age) -> None:
        self.name = name
        self.age = age


    def talk(self):
        print(f'My name is {self.name} and I am {self.age} years old!')


    def increase_age(self, by = 1):
        self.age += by



class ChildClass(ParentClass):

    def __init__(self, name, age, some_other_variable) -> None:
        # call the ParentClass __init__()
        super().__init__(name, age)

        self.some_other_variable = some_other_variable


    def talk(self):
        if self.age >= 2:
            super().talk()
        else:
            print('I cannot talk yet!')



father = ParentClass('John Doe', 945)
son = ChildClass('Peter Doe', 1, 'Whatever you like')

father.talk()
son.talk()
son.increase_age()
son.talk()

My name is John Doe and I am 945 years old!
I cannot talk yet!
My name is Peter Doe and I am 2 years old!

ChildClass derives from ParentClass.
Using super(), the ChildClass calls ParentClass's __init__() method to initialize the name and age instance variables. It (ChildClass) then creates and initializes some_other_variable specific to itself.
Within the overridden talk() method, the ChildClass calls the parents ParentClass's talk() method through the super() function if age is 2 or greater.

The super() function gives you access to the immediate base class. Let's add a GrandChildClass to the mix.

class GrandChildClass(ChildClass):

    def __init__(self, name, age, some_other_variable) -> None:
        super().__init__(name, age, some_other_variable)

    def talk(self):
        super().talk()
        print('I am a grandchild')


grandson = GrandChildClass("JC", 2, "Whatever")

grandson.talk()

My name is JC and I am 2 years old!
I am a grandchild

The super() in the grandchild points to ChildClass.

Multiple Inheritance

As stated earlier, a derived class can inherit from multiple base classes. For example, the StudentWorker class inherits from both Student and Worker classes.

class Student:

    def __init__(self, school) -> None:
        self.school = school

    def read(self, hours=1):
        print(f'I have read for {hours} hours!')

    def describe(self):
        print(f"I am a student at {self.school}")


class Worker:

    def __init__(self, occupation) -> None:
        self.occupation = occupation

    def work(self, hours=1):
        print(f'I have worked for {hours} hours!')

    def describe(self):
        print(f"I work as a {self.occupation}")


class StudentWorker(Student, Worker):

    def __init__(self, school, occupation) -> None:
        Student.__init__(self, school) # same as super().__init__(school)
        Worker.__init__(self, occupation)

    def work_and_read(self):
        super().work()
        super().read()

A StudentWorker has the variables and methods defined in both base classes.

sw = StudentWorker("John Hopkins University", "Bartender")
sw.read()
sw.work()
sw.work_and_read()

I have read for 1 hours!
I have worked for 1 hours!
I have worked for 1 hours!
I have read for 1 hours!

work(), read(), and work_and_read() are unique to each class. So, the outcome is straightforward.

But what happens if the parents have the same methods? For example, both Student and Worker classes have a describe() method. What happens if you call sw.describe() or super().describe()?

If you guessed "I am a student" you are right! "I am a worker" is not.

sw.describe()

I am a student at John Hopkins University

However, you would get "I am a worker" if you swap the order of inheritance. That is StudentWorker(Worker, Student). The reason is that Python maintains a list of classes it checks (depth-first, then left-right) when searching for a method. This concept is known as Method Resolution Order (MRO).

The first item on this list is the class itself, next are the base classes ordered according to the class definition, and lastly, the object class, the mother of all classes. The mro() method gives you access to this list. For example:

print(StudentWorker.mro())

[<class '__main__.StudentWorker'>, <class '__main__.Student'>, <class '__main__.Worker'>, <class 'object'>]

Therefore, when you call sw.describe() or super().describe() or any method in fact, Python will check in the following order and pick the first match:

StudentWorker
Student and its parents (since Student has its own MRO)
Worker and its parents (since Worker has its own MRO)
object

If Python does not find a method after these checks, it raises an AttributeError. See the article on Exceptions for more details.

Instead of swapping the order of inheritance, you could use the base class directly. For example, Worker.__init__(self, occupation) and Worker.describe(self) will initialize the occupation variable and call the Worker's describe() method, respectively.

Printing and Debugging An Object

When you print an instance using either print(), str(), format(), or repr(), you will get a somewhat cryptic representation of that class or instance. For example:

sw = StudentWorker()

print(sw)
print(str(sw))
print(repr(sw))
print(format(sw))

<__main__.StudentWorker object at 0x0000027314327D40>
<__main__.StudentWorker object at 0x0000027314327D40>
<__main__.StudentWorker object at 0x0000027314327D40>
<__main__.StudentWorker object at 0x0000027314327D40>

Luckily, you can get better and friendlier output by overriding the following methods inherited from the base object class:

__repr__(self) - which is called by the repr() function as well as the REPL at the command line
__str__(self) - which is called by both the str() and print() functions
__format__(self, format_spec) which is called by the format() function and the string .format() method

Let's override only __repr()__ and __str()__ since the format() function uses __str__() if it is present.

class StudentWorker(Student, Worker):
    # ... Elided for brevity

    def __repr__(self) -> str:
        return f"StudentWorker(school={self.school}, occupation={self.occupation})"


    def __str__(self) -> str:
        return f"Studies at {self.school} and works as a {self.occupation}"

We now see a different outcome when we print the object using the same functions mentioned earlier.

sw = StudentWorker("John Hopkins University", "Bartender")

print(repr(sw)) # repr() calls __repr__
print("StudentWorker:", sw)
print(str(sw))
print(format(sw)) # format() calls __str__
print(f"He is serious! He {sw}") # format() calls __str__

StudentWorker(school=John Hopkins University, occupation=Bartender)
StudentWorker: Studies at John Hopkins University and works as a Bartender
Studies at John Hopkins University and works as a Bartender
Studies at John Hopkins University and works as a Bartender
He is serious! He Studies at John Hopkins University and works as a Bartender

Overriding __repr__() is useful for debugging. StudentWorker(school=John Hopkins University, occupation=Bartender) is better than <__main__.StudentWorker object at 0x0000027314327D40>
Overriding __str__() gives a friendlier message.

Data Classes

Data classes are useful for data grouping, representation, or transfer. A data class is a regular class decorated with dataclasses.dataclass. Like regular classes, it can take any number of variables (with types specified) and methods and supports single or multiple inheritance.

However, unlike the regular classes, a data class contains default implementations of __init__() and __repr() methods. For example:

from dataclasses import dataclass, field

@dataclass
class Person:
    name: str
    age: int

    # All keyword variables must come after the positional variables
    # Keyword variables: variables with default arguments
    occupation: str = 'none'

    # hobbies: list = [] # FAILs no mutable argument allowed
    hobbies: list[str] = field(default_factory=list)


john = Person('John Doe', 45, 'Bartender')

print(john)

Person(name='John Doe', age=45, occupation='Bartender', hobbies=[])

name and age are positional variables
occupation is a keyword variable with a default value of none.
hobbies is a list initialized to [] by a call field(). Direct initialization of mutable data types like dict and list is not allowed.

The __init__() and repr() implemented for you might look like these:

# ... Within the Person class
def __init__(self, name, age, occupation='none', hobbies=[]):
    self.name = name
    self.age = age
    self.occupation = occupation
    self.hobbies = hobbies


def __repr__(self) -> str:
    result = f'{type(self).__name__}(' # Person(

    # ', 'join() takes a generator expression
    # Generator expression yields variable=value
    result += ', '.join(f"{k}={v!r}" for k, v in self.__dict__.items())
    result += ')' # close it

    return result

# ... Within the Person Class

You can get a glimpse of the implemented __init__() by looking at its signature.

import inspect

print(inspect.signature(john.__init__))

(name, age, occupation='none', hobbies=[]) -> None

Feel free to drop the __repr__(self) code above in any regular class you create. It just works! The type(self)__name__ ensures that a subclass returns the appropriate type, which is itself.

Every other thing we've discussed for regular classes applies to data classes. Check out the official Python documentation to learn more about data classes.

Note that Python also supports Enumerations (classes that provide unique values). Check the official documentation also on how to use enumerations.

Summary

In this article, you learn how to create a class and instances of classes. You also learned about class and instance variables and methods. You learned about single and multiple inheritance and how Python searches for methods (and variables). Finally, you saw how to create a data class.

Thank you for reading!

Introduction to Python Programming - Errors and Exceptions

Mark Edosa — Fri, 23 Feb 2024 19:40:07 +0000

Table of Contents
Introduction
Common Exceptions
- AssertionError
- AttributeError
- ImportError
- ModuleNotFoundError
- IndexError
- KeyError
- KeyboardInterrupt
- NameError
- IndentationError
- TypeError
- ValueError
- ZeroDivisionError
- File Errors
Handling Exceptions
Including an Else Clause
Adding a Finally Clause
Raising Exceptions
Adding More Information to Exceptions
User-Defined Exceptions
Summary

Introduction

As you write code, errors may occur due to wrong syntax or program execution. The former is known as a syntax error, the latter exception. An unhandled exception will cause your program to crash. Therefore, it is necessary to understand and handle them.

This article focuses on exceptions. Syntax errors are already visible mistakes.

Common Exceptions

Some exceptions are visible to the user. These include NameError, KeyError, IndexError, and so on. Other exceptions are not. These include but are not limited to StopIteration and GeneratorExit exceptions.

Below are some of the exceptions you are likely to encounter.

AssertionError

When an assert statement fails, it raises an assertion error. For example:

assert (4 + 5) == 8

Traceback (most recent call last):
  File "C:\articles\python\introduction\codes\exceptions.py", line 1, in <module>
    assert (4 + 5) == 8
AssertionError

You can customize the message associated with an assertion error. For example:

assert 7 == 8, 'Not sure 7 is equal to 8'

Traceback (most recent call last):
  File "C:\articles\python\introduction\codes\exceptions.py", line 3, in <module>
    assert 7 == 8, 'Not sure 7 is equal to 8'
AssertionError: Not sure 7 is equal to 8

Use the assert statement to ensure your function meets a particular condition during unit testing with a package such as pytest.

AttributeError

Attribute errors occur when you try to access a property (or method) that does not exist in an object. For example:

class Person:
    pass

person = Person()
person.age

Traceback (most recent call last):
  ...
AttributeError: 'Person' object has no attribute 'age'

Another example

# Empty dictionary
records = {}

# Works! items() method exist
records.items()

# Throws as AttributeError exception
records.age()

Traceback (most recent call last):
  ...
AttributeError: 'dict' object has no attribute 'age'

Ensure you are using attributes that exist to avoid this error.

ImportError

An import error usually occurs when you try to import a non-existent variable, function, or module from another module. For example, try to import Fish from the abstract base class (abc) module:

from abc import Fish

...
ImportError: cannot import name 'Fish' from 'abc'
...

ModuleNotFoundError

A ModuleNotFoundError is a subclass of ImportError. When a module does not exist or is not reachable, Python raises a ModuleNotFoundError. For example, I don't have a module called sky:

import sky

...
ModuleNotFoundError: No module named 'sky'

The solution is to ensure that a file sky.py exists within your project directory.

IndexError

An IndexError is raised when you supply a sequence (e.g. list, tuple, etc.) with an index that is out of range. Slice indices are silently truncated to fall in the allowed range. For example:

favorites = ['Peter', 'Stewie', 'Brian', 'Quagmire', 'Joe']

# These works
print(favorites[1]) # Stewie
print(favorites[1:2000])
print(favorites[1000:2000]) # []

# This fails: index is from 0 - 4
print(favorites[10])

Stewie
['Stewie', 'Brian', 'Quagmire', 'Joe']
[]

Traceback (most recent call last):
...
IndexError: list index out of range

KeyError

A KeyError occurs when you try to access a dictionary key that does not exist. For example:

oldest_person = { 'name': 'Methuselah Doe', 'age':  1000 }

# Ok!
print(oldest_person['name'])

# Oops!
print(oldest_person['occupation'])

Traceback (most recent call last):
...
KeyError: 'occupation'

To avoid KeyError, use the dictionary's .get() method. get() returns None for a non-existent key. For example:

print(oldest_person.get('occupation')) # None

Also, .get() optionally takes a default value that gets returned when the key does not exist. For example:

print(oldest_person.get('occupation', 'Musician')) # Musician

KeyboardInterrupt

When you hit the interrupt key (Control-C or Delete), Python raises a keyboardinterrupt exception. For example:

# Simulate a long-running program
import time

while True:
    print(time.time())

    # add some delay
    time.sleep(1)

1707814961.0006871
1707814962.0126708
1707814963.0137038
1707814964.014804
Traceback (most recent call last):
  ...
KeyboardInterrupt
^C

NameError

Python raises a NameError when you use a global or local name or variable (defined within a function) that you have not created. For example:

print(favorite_food)

Traceback (most recent call last):
...
NameError: name 'favorite_food' is not defined

The solution is to ensure you define a variable before using it. For example:

favorite_food = "Whatever works!"

# Works!
print(favorite_food)

IndentationError

An indentation error is a syntax error that occurs when your code is not properly indented. Thankfully, most code editors will help you avoid this kind of error. In the example below, I inserted two spaces before calling print()

  print('Hello')

# IndentationError: unexpected indent

TypeError

A TypeError occurs when you perform an operation or apply a function to an object of inappropriate type. For example, summing two dictionaries or converting an inappropriate string to an integer will raise a TypeError.

class Person:
    pass

# Fails!
map(lambda x: x * 2, Person())
# TypeError: 'Person' object is not iterable

# Fails!
print({} + {})
# TypeError: unsupported operand type(s) for +: 'dict' and 'dict'

ValueError

A ValueError exception occurs when an operation or function receives an argument with the right type but an inappropriate value. A classic example is when you try to convert an inappropriate string to an integer.

print(int('abc'))

# ValueError: invalid literal for int() with base 10: 'abc'

ZeroDivisionError

A ZeroDivisionError exception occurs when you attempt to divide (or modulo) a value by zero. For example:

print(4 / 0)
# ZeroDivisionError: division by zero

print(3 % 0)
# ZeroDivisionError: integer modulo by zero

The base class of ZeroDivisionError is the ArithmeticError. The ArithmeticError is also the parent class of OverflowError, an exception raised when the result of an arithmetic operation is too large to be represented.

File Errors

Other exceptions you might encounter include:

FileExistsError: Occurs when trying to create a file or directory which already exists
FileNotFoundError: Raised when a file or directory is requested but doesn’t exist.

with open('hello.txt', mode='rt', encoding='utf-8') as f:
    print(f.read())

# Side note: The program never gets here
print('I will never get here if hello.txt does not exist')

...
FileNotFoundError: [Errno 2] No such file or directory: 'hello.txt'

Do check the Python documentation for more types of exceptions.

Handling Exceptions

To prevent your program from crashing you need to handle possible exceptions. In Python, you use the try-except statement to handle exceptions. A classic example is asking for a user's age until he provides the expected value:

while True:
    try:
        age = int(input("Please enter your age: "))
        if age >= 18:
            break
        else:
            print("The age you provided is less than 18!")
    except ValueError:
        print("Oops!  That was no valid number.  Try again...")

Please enter your age: 12
The age you provided is less than 18!
Please enter your age: weerr
Oops!  That was no valid number.  Try again...
Please enter your age: 45

Let's handle the FileNotFoundError you saw earlier.

try:
    with open('hello.txt', mode='rt', encoding='utf-8') as f:
        print(f.read())
except FileNotFoundError as e:
    print(e)

# Our program continues because we handled the exception above
print('I will run even if hello.txt does not exist')
print('Me too!')

[Errno 2] No such file or directory: 'hello.txt'
I will run even if hello.txt does not exist
Me too!

A try statement can have multiple except blocks or clauses. Also, an except clause may have many exceptions in a tuple with parenthesis. Here is a contrived example modified from Python documentation:

def divide(a, b):
    return a / b


try:
    f = open('myfile.txt')
    s = f.readline()
    i = int(s.strip())
    divide(10, 0)
except OSError as err:
    print("OS error:", err)
except ValueError:
    print("Could not convert data to an integer.")
except ZeroDivisionError as err:
    print("Zero division error:", err)
except Exception as err:
    print(f"Unexpected {err=}, {type(err)=}")

You can shorten the code above to

try:
    f = open('myfile.txt')
    s = f.readline()
    i = int(s.strip())
    divide(10, 0)
except (OSError, ValueError, ZeroDivisionError) as err:
    print(err)
except Exception as err:
    print(f"Unexpected {err=}, {type(err)=}")

A call to open() might raise a FileNotFoundError or any other exception that is a subclass of OSError, so we use OSError as a catch-all for these exception types.
Converting an invalid string to an integer will raise a ValueError. We handle that directly.
A call to divide(10, 0) raises a ZeroDivisionError exception. We handle the exception. A try-except statement can handle any exception raised from a function.
All built-in, non-system-exiting exceptions are subclasses of the Exception class, so we handle any other form of exception that might occur. The Exception class derives from the BaseException class, the mother of all exception classes.

Consider using specific exception types such as FileNotFoundError before using the general ones like OSError.

Including an Else Clause

You can add an optional else clause to the try-except statement. The else clause must follow all except clauses. Use it for code that must run if the try clause does not raise an exception. For example:

for arg in sys.argv[1:]:
    try:
        f = open(arg, 'r')
    except OSError:
        print('cannot open', arg)
    else:
        print(arg, 'has', len(f.readlines()), 'lines')
        f.close()

Using the else clause is better than adding additional code to the try clause because it avoids accidentally catching an exception from another code you did not intend to protect with the try-except statement.

Adding a Finally Clause

Also, you can include an optional finally block if you need to run some form of "clean up" code. Unlike the else clause, the code in the finally block will always run regardless of an exception. For example:

def divide(x, y):
    try:
        result = x / y
    except TypeError:
        print(f"divide({x}, {y}): You cannot divide {x} by {y}")
    except ZeroDivisionError:
        print(f"divide({x}, {y}): You cannot divide {x} by zero.")
    else:
        # Only runs if the try block succeeds
        print(f"divide({x}, {y}): result is {result}")
    finally:
        # Always runs
        print("executing finally clause")

Dividing with unsupported types such as strings will raise a TypeError which is handled in the first except block. The finally block runs. The program ends.
The second except block catches a ZeroDivisionError. The finally block runs. The program ends.
The else clause runs when divide() succeeds. The finally block runs. The program ends.

divide('hello', 'world')
print() # for space
divide(4, 0)
print() # for space
divide(3, 2)

divide(45, 2): You cannot divide hello by world
executing finally clause

divide(4, 0): You cannot divide 4 by zero.
executing finally clause

divide(3, 2): result is 1.5
executing finally clause

Raising Exceptions

There are instances where you want to raise exceptions if a user's input does not meet the condition you have specified. You can do so using the raise statement. For example:

def divide_only_ints(num1: int, num2: int):
    if isinstance(num1, int) and isinstance(num2, int):
        print(f'num1 / num2 = {num1 / num2 :.2f}')
    else:
        raise TypeError(f'num1 and num2 must be ints!\nGot {num1} and {num2} instead')

# Works!
divide_only_ints(4, 3) # num1 / num2 = 1.33

# Fails
divide_only_ints(4.5, 12.3)

Traceback (most recent call last):
...
TypeError: num1 and num2 must be ints!
Got 4.5 and 12.3 instead

raise Exception is the same as raise Exception(). You can optionally include a custom message. For example, raise Exception('My custom message').

With the raise keyword, you can re-raise or propagate an exception if you don't want to handle it. For example:

try:
    divide_only_ints("a", "b")
except TypeError:
    print('An exception occurred! Sending it your way!')
    raise

An exception occurred! Sending it your way!
...
TypeError: num1 and num2 must be ints!
Got a and b instead

You can also propagate multiple exceptions from an except block using a raise or raise ... from statement. This propagation is known as exception chaining. For example:

try:
    divide_only_ints("a", "b")
except TypeError:
    raise ValueError('Just because I can')

Or using raise ... from:

try:
    divide_only_ints("a", "b")
except TypeError as err:
    raise ValueError('Just because I can') from err

Traceback (most recent call last):
  ...
TypeError: num1 and num2 must be ints!
Got a and b instead

During the handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
ValueError: Just because I can

Adding More Information to Exceptions

The section requires Python >= 3.11

Sometimes, you want to add more information to an exception before re-raising it. You can use the exception's .add_notes() method for such. For example:

try:
    divide_only_ints("a", "b")
except TypeError as err:
    err.add_note('I used the divide_only_ints() function')
    err.add_note('The left operand was the letter a')
    err.add_note('The right operand was the letter b')
    print('An exception occurred! So Sending it your way!')
    raise

An exception occurred! So Sending it your way!
Traceback (most recent call last):
  ...
TypeError: num1 and num2 must be ints!
Got a and b instead
I used the divide_only_ints() function
The left operand was the letter a
The right operand was the letter b

User-Defined Exceptions

Finally, you can define your exceptions by deriving from the Exception class directly or indirectly. For example:

class DivideByIntOnlyError(Exception):
    """An example of a user-defined exception."""

    def __init__(self, *args: object) -> None:
        super().__init__(*args)

    def add_note(self, __note: str) -> None:
        print('About to start adding notes for useless logging purposes')
        return super().add_note(__note)


def divide_only_ints(num1: int, num2: int):
    if isinstance(num1, int) and isinstance(num2, int):
        print(f'num1 / num2 = {num1 / num2 :.2f}')
    else:
        raise DivideByIntOnlyError(
            f'num1 and num2 must be ints!\nGot {num1} and {num2} instead')

try:
    divide_only_ints("a", "b")
except DivideByIntOnlyError as err:
    err.add_note('I used the divide_only_ints() function')
    err.add_note('The left operand was the letter a')
    err.add_note('The right operand was the letter b')
    print('An exception occurred! So Sending it your way!')
    raise

About to start adding notes for useless logging purposes
About to start adding notes for useless logging purposes
About to start adding notes for useless logging purposes
An exception occurred! So Sending it your way!
Traceback (most recent call last):
  ...
DivideByIntOnlyError: num1 and num2 must be ints!
Got a and b instead
I used the divide_only_ints() function
The left operand was the letter a
The right operand was the letter b

Note that overriding the __init__() and add_note() methods within DivideByIntOnlyError is unnecessary in most cases. I've included it here for demo purposes only.
Also note that While you can define Exception classes to do anything any other class can, it is best to keep it simple.

Summary

In this article, you saw examples of exceptions like IndexError and KeyError and how they may occur. You also saw how to handle exceptions using the try-except statement with or without else or finally. Next, you learned how to raise exceptions, chain them, and add more details to the exceptions. Lastly, you learned how to define your exception by subclassing the Exception class.

Thank you for reading.

How to Extract Content From Word Documents Using BeautifulSoup

Mark Edosa — Sat, 10 Feb 2024 09:15:08 +0000

Introduction

This article shows you how to extract content or sections from Word documents using beautifulSoup and regular expressions. I assume you understand Python programming and regular expressions.

A Word Document

Behind the scenes, a Word (docx) document is a zip file containing a collection of XML files. Using Python, you can easily view these files. For example:

import zipfile

f = zipfile.ZipFile('test.docx')
print(f.namelist())
# [..., 'word/document.xml', ..., 'word/settings.xml', ...]

I created a Word document containing several "Lorem Ipsum" text as the body and "Introduction to Python programming" as the title.

The main XML file, word/document.xml, contains the text content you would normally see while the rest XML files are related to settings, styling, and so on.

Viewing the Content

To view the content of document.xml, you can use the BeautifulSoup() class from the bs4 package.

Here, I defined a function read_doc_to_soup() to make it easier.

Note that you have to install the lxml package along with the beautisoup4 package to get BeautifulSoup() working with XML correctly.

def read_doc_to_bsoup(filename: str) -> BeautifulSoup:
    with zipfile.ZipFile(filename) as file:
        document = file.read('word/document.xml')
    return BeautifulSoup(document, 'xml')


soup = read_doc_to_bsoup('test.doc')
print(soup.prettify())

A sample output is shown below.

<?xml version="1.0" encoding="utf-8"?>
<w:document mc:Ignorable="w14 w15 w16se w16cid w16 w16cex w16 ...

   ...

   <w:r w:rsidRPr="0002422F">
    <w:rPr>
     <w:rFonts w:ascii="Times New Roman" w:cs="Times New Roman" w:hAnsi="Times New Roman"/>
     <w:b/>
     <w:bCs/>
     <w:sz w:val="24"/>
     <w:szCs w:val="24"/>
     <w:lang w:val="en-US"/>
    </w:rPr>
    <w:t>
     Introduction to Python Programming
    </w:t>
   </w:r>
  </w:p>

  ...

  <w:sectPr w:rsidR="00F152B4" w:rsidRPr="0002422F">
   <w:pgSz w:h="16838" w:w="11906"/>
   <w:pgMar w:bottom="1440" w:footer="708" w:gutter="0" w:header="708" w:left="1440" w:right="1440" w:top="1440"/>
   <w:cols w:space="708"/>
   <w:docGrid w:linePitch="360"/>
  </w:sectPr>
 </w:body>
</w:document>

The text we're interested in is usually within a w:t XML tag. Notice that the text "Introduction to Python Programming" is wrapped within a w:t XML tag.

Extracting The Content

To get the first text, simply use the .find() method of beautiful soup like so

first_text = soup.find('w:t')

print(first_text) # <w:t>Introduction to Python Programming</w:t>

# To extract the text without the tags
print(first_text.get_text()) # Introduction to Python Programming

To get all text tags, use the .find_all() method.

content = soup.find_all('w:t')
print(content)

[<w:t>Introduction to Python Programming</w:t>, 
<w:t xml:space="preserve">Lorem ipsum </w:t>, <w:t>dolor</w:t>, 
<w:t xml:space="preserve"> sit </w:t>, <w:t>amet</w:t>, 
<w:t xml:space="preserve">, </w:t>, <w:t>consectetur</w:t>, 
<w:t xml:space="preserve"> </w:t>, <w:t>adipiscing</w:t>, <w:t xml:space="preserve"> </w:t>, 
<w:t>elit</w:t>, <w:t xml:space="preserve">. </w:t>, 
<w:t>Pellentesque</w:t>, <w:t xml:space="preserve"> </w:t>, <w:t>metus</w:t>, ...]

To extract the text without the tags, you can use a list comprehension. For example

content = [node.get_text() for node in soup.find_all('w:t') if node is not None]
print(content)

# ['Introduction to Python Programming', 'Lorem ipsum ', 'dolor', ' sit ', 
# 'amet', ', ', 'consectetur', ' ', 'adipiscing', ' ', 'elit', '. ', 
# 'Pellentesque', ' ', 'metus', ' ', 'elit', ', ', 'consectetur', ' id ', 
# 'mollis', ' non, ', 'fringilla', ' in eros. Mauris ', 'aliquam', ' ', 
# 'quis', ' ', 'odio', ' id tempus. ', 'Aliquam', ' ', 'erat', ' ', 
# 'volutpat', '. Donec id ', 'iaculis', ' ipsum. In ', 'tincidunt', ' ', 
# 'massa', ' non ', 'aliquam', ' ', 'dignissim', '. Donec semper ', ...]

Processing The Content

While you have successfully extracted the content, you'll notice that in some cases, the text does not make complete sentences. While some texts are within <w:t xml:space="preserve">, others are within plain <w:t>. This makes working with Word documents a huge problem.

The solution to the problem is entirely dependent on the structure of the document. Therefore, you'll have to spend time understanding the structure of your document(s).

In the simplest case, you could choose to combine all text into a single string and then split by period (.).

Example: Extracting Research Objectives from Research Proposals

Thankfully, research proposals have a fairly regular structure across various fields. Just for fun, I wanted to extract the objectives from a series of research proposals.

In cases where all the objectives are on a single line, the function below just works!

import re

def process_simple(filename: str, target: str = 'To determine'):
    """Extract objectives using a target keyword."""
    soup = read_doc_to_bsoup(filename)
    wt = soup.find_all('w:t')

    results = [re.sub(r'\d+', '', t.get_text()).strip()
               for t in wt if t.get_text()]

    results = list(filter(lambda x: x, results))

    return list(filter(lambda x: x.startswith(target), results))

Extract the content as you saw above
Remove digits and empty strings from the text array
Collect the texts that start with "To determine".

In a more complicated scenario, the objectives spanned multiple lines, section numbers were different, and so on. The above function failed woefully. Luckily, I found that across objectives spanning multiple lines, each text within the <w:t> tag was part of the preceding <w:t xml:space="preserve"> tag.

The code below walks through the XML soup and combines all <w:t> tags with the previous <w:t xml:space="preserve"> tag. It also works for cases where the objectives are on single lines.

def process_xml_soup(soup: BeautifulSoup):
    # extract content
    wt = soup.find_all('w:t')

    results = []

    # Keep track of cursor positions
    current = 0
    next = current + 1
    stop = len(wt) - 1

    while current < stop:
        cur_elem = wt[current]
        next_elem = wt[next]

        # get the current text
        current_text = cur_elem.get_text().strip() or ''

        # Are we between <w:t xml:space="preserve"> and <w:t>?
        if cur_elem.has_attr('xml:space') and not next_elem.has_attr('xml:space'):

            # Join subsequent <w:t> to the preceding <w:t xml:space="preserve">
            while not next_elem.has_attr('xml:space') and next < stop:
                current_text += f' {next_elem.get_text().strip() or ""}'

                # Advance the cursor until
                # we see another <w:t xml:space="preserve">
                next += 1
                next_elem = wt[next]

        # Remove too many spaces
        current_text = re.sub(r'\s+', ' ', current_text.strip())

        # make objectives title consistent
        # That is, if you find 3.1 objectives make it 3.1 OBJECTIVES
        current_text = re.sub(r'(\d\.\d\.?\d?) objectives',
                              r'\1 OBJECTIVES', current_text, flags=re.I)

        # save the text
        results.append(current_text)

        # Move the cursor forward
        current = next
        next = current + 1

    results = [c.strip() for c in results]

    return ' '.join(list(filter(lambda x: x, results)))

Research objectives are usually sandwiched between different sections depending on the field/department. For example, between "AIM" and "HYPOTHESIS".

The following functions complete the extraction.



class NoneObject:
    """Create a NoneObject to avoid return None from find_last()."""
    def start(self):
        return None

    def end(self):
        return None


# Find last uses NoneObject to maintain API consistency
def find_last(text: str, target: str):
    """Find the last occurrence of a target string."""

    # finditer returns an iterator of match objects or None
    result = list(re.finditer(target, text))

    # Let's avoid None checks in any function that uses the find last
    # by using NoneObject
    return result[-1] if len(result) >= 1 else NoneObject()


# Use find last to extract a section
def extract_section(text: str, start: str, end: str, verbose: bool = True):
    """Extract a section of a text."""
    start = find_last(text, start)
    end = find_last(text, end)

    if verbose:
        print(start)
        print(end)

    return text[start.end():end.start()].strip()


# Account for different headers after the list of objectives
def gen_end_regex():
    """Generate a series of regexes to match the section after the objectives."""

    end_titles = ['HYPOTHESES', 'HYPOTHESIS',
                 'RESEARCH QUESTIONS', 'NULL AND ALTERNATE HYPOTHESIS',
                 'CHAPTER TWO LITERATURE REVIEW', 'CHAPTER THREE LITERATURE REVIEW']
    fdgts = r'\d?\.\d?\.?\s*'
    regex = ''

    for i, w in enumerate(end_titles):
        regex += (fdgts + w)
        if i != len(end_titles) - 1:
            regex += '|'

    return regex


def extract_objectives(text: str, verbose: bool = True):
    """Extract the objectives section"""
    start = r'\d?\.?\d?\.?\s*OBJECTIVES[:;]?|\d?\.?\d?\.?\s*Objectives|OBJECTIVES OF THE STUDY'
    end = gen_end_regex()

    return extract_section(text, start, end, verbose)


def seperate_objectives(objectives: str):
    """Separate or split the objectives."""
    objectives = re.sub(r'\d\.?', '', objectives).removeprefix('OBJECTIVES ')
    objectives = list(filter(lambda x: x.strip(), objectives.split('To ')))
    objectives = list(
        map(lambda x: 'To ' + x.strip().replace('.', '').strip(), objectives))

    return objectives


def process(filename: str, verbose: bool = True):
    """Run all functions on a word document."""
    soup = read_doc_to_bsoup(filename)
    document = process_xml_soup(soup)
    objectives = extract_objectives(document, verbose)
    objectives = seperate_objectives(objectives)

    return objectives

In research proposals, each section title or heading is often defined in two places - within the table of contents and the body of the document. find_last() finds the last occurrence within the body of the document.
The re.finditer() returns a match object which has two methods .start() and .end(). These methods return the start and end index respectively. In the absence of a match, instead of returning None, I use the NoneObject() to mimic the result of re.finditer().
extract_section() uses find_last() to do its job.
extract_objectives() and separate_objectives() functions are fairly obvious, I think :)
Everything is combined within the process() function.

Testing

Create a Word document objectives.docx containing the following content:

OBJECTIVES
- To determine the effect of exercise on weight loss in elderly people.
- To determine if the effect of exercise on weight loss in elderly people is influenced by factors such as location and gender.
- To determine the effect of dieting on weight loss in elderly people.
- To determine if the effect of dieting on weight loss in elderly people is influenced by factors such as location and gender.

from pprint import pp

pp(process('objectives.docx'))

['To determine the effect of exercise on weight loss in elderly people',      
 'To determine if the effect of exercise on weight loss in elderly people is '
 'influenced by factors such as location and gender',
 'To determine the effect of dieting on weight loss in elderly people',       
 'To determine if the effect of dieting']

Summary

In this article, you saw how to extract text from a Word document. You also saw a sample case that combined various methods including regex to extract the objective section of a research proposal.

Thank you for reading.

How to Traverse (Walk) Multiple Directories Using Python

Mark Edosa — Sat, 10 Feb 2024 09:00:31 +0000

Introduction

This article shows you how to create a directory walker that extracts files from multiple directories and subdirectories. The walker initially uses a combination of recursion and a for loop and then uses a generator/iterator to save time and memory. I assume you are familiar with basic Python programming, functions, generators, and iterators. Let's get started.

Directory Structure

A directory is typically represented as a tree data structure where the topmost directory sits as the root node. The root node can contain zero, one or more subdirectories (child nodes), and files (leaf nodes). The subdirectories can also contain other directories and files.

The diagram below shows a Desktop (root) directory containing three subdirectories - Work, Music, and Videos.

To extract the contents of all directories, you will need to visit each directory using a loop or recursion. These two methods have their advantages and disadvantages. However, this article will not discuss them. You can check out this post on StackOverflow for more information.

The Path Class from the PathLib Module

The pathlib module provides utilities for working with the filesystems of different operating systems. The Path class, an alternative to os.path, provides a high-level, friendly way to assess file paths while avoiding the many ceremonies associated with the os.path module. For example:



from pathlib import Path

current_dir = Path('.')

print(current_dir) # .

Assessing and navigating subdirectories is as easy as it gets. For example:



# List all subdirectories
subdirs = [x for x in current_dir.iterdir() if x.is_dir()]
print(subdirs) # [WindowsPath('Desktop/Music'), ...]

## List all songs within Desktop
all_songs = list(current_dir.glob('**/*.mp3'))
print(all_songs) # [WindowsPath('Desktop/Music/Hip Pop/song3.mp3'), ...]

# Assuming my current directory is "Desktop"
work_dir = current_dir / "Work"
print(work_dir) # Desktop\Work or Deskop\Work (Linux)

comedy_dir = current_dir / "Videos" / "Comedy"
print(comedy_dir) # Also OS specific

# Print a tuple containing parts of the directory
print(comedy_dir.parts) # ('Desktop', 'Videos', 'Comedy')

If you simply want to extract a specific file type, you can use the .glob() method mentioned above. However, if you want more control over the process, then consider using a loop or a generator function.

You can also check if the path is a directory using the .is_dir() method or if a path exists using the .exists() method. For example:



print(current_dir.is_dir()) # True

dont_know = current_dir / "Created"
print(dont_know.exists()) # False

Walking Directories

Using A Loop and Recursion

I assume you've created the example directory structure in the diagram above. Let's look at a sample code.



from pathlib import Path
from typing import Callable


def song_strategy(filename: str) -> bool:
    """Check if the file is an mp3 that does not contain certain words."""

    # ignore file if any of these words are in the file name
    ignore_words = ['slow', 'jazz', 'old']

    if filename.endswith('mp3'):
        res = [kw for kw in ignore_words if filename.lower().find(kw) != -1]
        return len(res) == 0

    return False


def collect_files(root_dir: Path, file_strategy: Callable[[str], bool]) -> list[Path]:
    """Collect files by walking multiple directories."""

    files: list[Path] = []

    # directories to ignore
    ignore = {'work', 'videos', 'reggae'}

    def inner(root_dir):
        for x in root_dir.iterdir():
            if x.is_dir() and (x.parts[-1].lower() not in ignore):
                # recursion
                inner(x)
            else:
                # use the strategy
                if file_strategy(x.parts[-1]):
                    files.append(x)

    inner(root_dir)

    return files

# The current path is "Desktop"
print(collect_files(Path('.'), song_strategy))

collect_files() takes a root_dir path and a file_strategy() function that filters files.
A strategy, song_strategy() is an example of a file_strategy() function that selects only mp3 files. You can easily add others!
collect_files() uses an inner() function to recurse through each directory and collects the selected file in an array of files.
Note that a recursion must have a termination condition to prevent stack overflow. In this example, the if x.is_dir() ... ensures that recursion ends once all directories have been transversed.

While collect_files() works, its time and space efficiency drastically reduces as the number of directories to transverse increases, especially when dealing with thousands of directories.

Using A Generator Function

A solution to the above problem is to replace collect_files() with an alternative function that uses a generator. The function is defined below.



def collect_files_generator(root_dir: Path, file_strategy: Callable[[str], bool]):
    """Collect files by walking multiple directories using a generator"""
    # directories to ignore
    ignore = {'work', 'videos', 'reggae'}

    def inner(root_dir):
        for x in root_dir.iterdir():
            if x.is_dir() and (x.parts[-1].lower() not in ignore):
                # yield from a generator
                yield from inner(x)
            else:
                if file_strategy(x.parts[-1]):
                    yield x

    yield from inner(root_dir)

collect_files_generator() is a generator that uses a sub-generator inner().
inner() is a sub-generator that recursively uses itself.
collect_files_generator() is better than collect_files() because it produces values one by one (or lazily) instead of storing them in files (memory) before returning them.

While I assume that you already know how a generator works, this post is a great recipe on how yield from works!

Since a generator produces an iterator, you can control how you retrieve each element from the iterator. For example



# loop through the iterator

for file in collect_files_generator(Path('.'), song_strategy):

    print(file)

# Or



# Automatically extract the iterator content as a list

print(list(collect_files_generator(Path('.'), song_strategy)))

Summary

In this article, you saw how

directories are traversed using loops and recursions,
the space and time efficiency can be improved using a generator function rather than a normal function,
directories are represented and how to use the Path class from the pathlib module,
to use a strategy function to filter files.

Thanks for reading.

Introduction to Python Programming - Working With Files

Mark Edosa — Thu, 02 Nov 2023 09:51:05 +0000

At some point in your program, you may want to work with files stored locally on your machine or from a network. Python represents files as file (or file-like) objects. This representation provides you with functions that enable the manipulation of files.

File Object APIs
Opening/Creating a File
Writing to Files
Reading from Files
Example: Copying a File
Working with JSON Files
Working with Structured Datasets
Working with Temporary Files
Working with File-like Objects
- StringIO and BytesIO
- Network Response
- The Terminal (Standard Input and Output)
Getting File Attributes
Working with Directories
Example: Back Up Recently Modified Files
Summary

Typically, these file (or file-like) objects include:

Text file objects which let you read/write text strings. Examples include:
- files opened in text mode (r or w)
- standard input represented by sys.stdin
- standard output represented by sys.stdout, and
- a string created with sys.StringIO()
Binary file objects that let you read/write buffered byte strings. Examples include:
- files opened in binary mode ('rb', 'wb' or 'rb+') and
- a byte object created with io.BytesIO() and gzip.GzipFile
Raw binary file objects are used behind the scenes to build binary and text streams. An example includes a file opened with mode=rb and buffering=0 e.g. open("myfile.jpg", "rb", buffering=0).

You can use the word stream and buffered file interchangeably.

File Object APIs

The table below shows some of the properties of file objects.

Property	type	Description
`buffer`	io.BufferedRandom	An in-memory temporary storage for the actual file text/binary contents.
`closed`	bool	Whether the file is opened or closed.
`encoding`	str	The file encoding applicable only to text files, gives the name of the encoding that the text will be decoded or encoded with. Always specify the encoding when you open a text file for reading/writing. Else, Python will use the default locale encoding and that might not be what you want.
`errors`	str	How should Python handle file errors? Defaults to `strict`. More info
`line_buffering`	bool	If `True`, Python clears the file's internal buffer when a call to `write()` contains a `\n` or a `\r\n`.
`mode`	str	Whether the file is opened for reading(r), writing(w) and so on.
`name`	str	The name of the file.
`newlines`	str	Controls how line endings are handled. It can be `None, '', '\n', '\r', and '\r\n'`.
`write_through`	bool	If `True`, calls to `write()` are guaranteed not to be buffered. Any data written to a text file is directly handed to its underlying binary buffer.

Use sys.getdefaultencoding() or locale.getencoding() to get your system's or locale's default encoding respectively. Note that locale.getencoding() works in Python >=3.11.

The table below shows some text/binary file or stream methods.

Method	Description
`close()`	Flush and close the stream.
`flush()`	Flush the write buffers of the stream if applicable.
`read()`	Read and return all (parameter `size=-1`) or `n` characters/bytes from a stream.
`readable()`	`True` if you can read the file. If `False`, `read()` will raise an error.
`readline()`	Read and return one line from the stream.
`readlines()`	Read and return a list of lines from the stream.
`reconfigure()`	Reconfigure this text stream using new settings for encoding, errors, newline, line_buffering and write_through.
`seek()`	Useful for random file access.
`seekable()`	Return `True` if the stream supports random access. If False, `seek()`, `tell()` and `truncate()` will raise an error.
`tell()`	Return the current stream position.
`truncate()`	Resize the stream to the given size in bytes (or the current position if size is not specified).
`writable()`	Return `True` if the stream supports writing.
`write()`	Write the given bytes/text string to the stream, and return the number of bytes/characters written.
`writelines()`	Write a list of lines to the stream. Add `\n` yourself if you need it.

Opening/Creating a File

The open() function is the most common way to open a file object for reading/writing. For example

# Open a file for reading in text mode
# Be explicit about the mode and encoding
text_file = open('your_text_file.txt', mode='rt', encoding='utf-8')

# <class '_io.TextIOWrapper'>
print(type(text_file))

# Open a file for reading in binary mode
binary_file = open('your_binary_file', mode='rb')

# <class '_io.BufferedReader'>
print(type(binary_file))

The open() in write mode creates a new file if it does not exist. If the file already exists, the open() call will clear its content. For example

# Open a file for writing in text mode
text_file = open('lyrics.txt', mode='wt', encoding='utf-8')

# Open a file for writing in binary mode
binary_file = open('your_binary_file', mode = 'wb')

The table below lists the available file modes and their description.

Mode	Description
`r`	Open for reading (default).
`w`	Open for writing, truncating the file first.
`x`	Create a new file and open it for writing. Raises an error if the file already exists.
`a`	Open for writing, appending to the end of the file if it exists.
`b`	Binary mode.
`t`	Text mode (default)
`+`	Open a disk file for updating (reading and writing). For example, `r+t` opens a file for reading and writing in text mode.

Files are stored as raw bytes. Files opened in binary mode return contents as bytes objects without any decoding. In text mode, the underlying raw bytes are decoded into text (using the default locale encoding or the encoding you specified) and then returned.

Writing to Files

To write to a file, open it for writing with any of w, w+, a, a+, x, x+, or r+ in a text (t) or binary (b) mode and use either .write() or .writelines(). For example

# Open the file for writing. 
# Add + so that we can also read from the file
lyrics = open('lyrics.txt', mode='w+', encoding='utf-8')

# Write something nice
lyrics.write('Best day of my life\n')
lyrics.write('I had a dream a dream so big and loud\n')
lyrics.write('I jumped so high I touched the clouds\n')
lyrics.write('Wo-o-o-o-o-oh, wo-o-o-o-o-oh\n')

# Persist the content into the file without closing it
lyrics.flush()

# Confirm if the write succeeded
lyrics.tell()

# Rewind to first position
lyrics.seek(0)

# Read the content
print(lyrics.read())

more_lines = [
  'I stretched my hands out to the sky\n',
  'We danced with monsters through the night\n',
  'Wo-o-o-o-o-oh, wo-o-o-o-o-oh\n'
]

# Add more lines using .writelines()
lyrics.writelines(more_lines)

# Flush and close the file
lyrics.close()

Reading From Files

To read a file, open it for reading with any of r, r+, w+, a+, or x+ in a text (t) or binary (b) mode and use either .read(), .readline() or .readlines(). For example

lyrics = open('lyrics.txt', mode='rt', encoding='utf-8')

# Read all contents
# The file cursor position goes to the end
all_content = lyrics.read()
print(all_content)

# Reset the cursor/position to the beginning
lyrics.seek(0)

# Read the first ten characters
first_10_chars = lyrics.read(10)
print(first_10_chars)

# Read 20 characters more
another_20_chars = lyrics.read(20)
print(another_20_chars)

# Where is the file cursor position
print("The file cursor is at position", lyrics.tell())

# Read the rest content starting from where read(10) stopped
the_rest_content = lyrics.read()
print(the_rest_content)


# Reset the cursor/position to the beginning again
lyrics.seek(0)

# Read only the first line
first_line = lyrics.readline()
print(first_line)

# Read the rest lines
the_rest_lines = lyrics.readlines()
print(the_rest_lines)

# Reset
lyrics.seek(0)


# print each line in a for loop
for line in lyrics.readlines():
    print(line)

# Reset
lyrics.seek(0)


# Read from the file object directly
for line in lyrics:
  print(line)


# Close the file
lyrics.close()

Example: Copying a File

# copy.py
"""A script for copying files. Usage: python copy.py file1 file1_copy"""
import sys

def copier(source: str, destination: str):
  """Copy the contents of the source to the destination
    Args:
      source - The source file.
      destination - The destination file.
  """
  with open(source, mode='r', encoding='utf-8') as srcfile:
    with open(destination, mode='w', encoding='utf-8') as destfile:
      for line in srcfile:
        destfile.write(line)


if __name__ == '__main__':
  if len(sys.argv) != 3:
    print('Usage: python copy.py file file1_copy')
  else:
    _, src, dest = sys.argv
    copier(src, dest)

The with statement, an example of a "context manager" closes a file. So there is no need to call .close() yourself.

Working with JSON Files

You can read/write to a JSON file in the following ways:

To write, use json.dump(python_object, file_opened_for_writing) to write a python object to a file opened for writing or use file_opened_for_writing.write(json.dumps(python_object)).
To read, use json.load(file_opened_for_reading) to read from a binary file or text file object opened for reading.

Note that you must open a JSON file in utf-8 mode when reading or writing to it.

import json

# Some menu I'd love to try :)
menu = [
  dict(name='alien_fish', price=2000), 
  dict(name='alient_vegetables', price=100),
  dict(name='some_alien_dish', price=200_000)
]

# Open a JSON for writing and dump the menu there so 
# that I won't ever forget
with open('menu.json', mode='w', encoding='utf-8') as jsonfile:
  json.dump(menu, jsonfile)

# Open a JSON file for reading
with open('menu.json', mode='r', encoding='utf-8') as jsonfile:
  same_menu = json.load(jsonfile)

# Sure? yes
assert menu == same_menu, "Oops!, The menu isn't the same"

Working with Structured Datasets

You've already seen how to read and write CSV files in the previous article on modules. For other types of structured data such as CSVs, Excel, Stata, SPSS, databases, and so on, you'll want to use an external package like pandas.

The pandas io API lists several methods for reading/writing various data files.

Working with Temporary Files

You can also securely create one-off files or file-like objects for transfer or demo purposes. For example, you can store a user-uploaded file as a temporary one, process it, and then copy the processed contents to a new one for permanent storage.

To create a temporary file, use tempfile.TemporaryFile(). For example

import tempfile

# TemporaryFile() creates a temporary file-like object
with tempfile.TemporaryFile(mode='w+b') as tpfile:
  print(tpfile)
  print(tpfile.file)

  tpfile.write(b'I will not stay long')
  tpfile.seek(0)
  print(tpfile.read())

The tempfile.TemporaryFile() call returns a file object (on POSIX platforms) or a file-like object on other platforms.
The default mode parameter is 'w+b'.
The binary mode helps to maintain consistency on all platforms regardless of the data stored.

If you need a temporary file to have a name and if you also need to control the file deletion, use tempfile.NamedTemporaryFile() instead. For example

import tempfile

# NamedTemporaryFile() always returns a file-like object 
# whose file attribute is the underlying true file object.
with tempfile.NamedTemporaryFile(mode='w+b', delete=True, delete_on_close=True) as named_tpfile:
  print(named_tpfile)
  print(named_tpfile.name)
  print(named_tpfile.file)

  named_tpfile.write(b'Set delete=False or delete_on_close=False ')
  named_tpfile.write(b'to control when I\'m deleted\n')

  named_tpfile.seek(0)

  print(named_tpfile.read())

Working with File-like Objects

StringIO and BytesIO

You can also create a file directly from text strings or bytes using io.StringIO() or io.ByteIO(), respectively. For example

import io

text_message = 'Hello this is a file created from raw strings'
file_from_raw_strings = io.StringIO(text_message)

print(type(file_from_raw_strings)) # <class '_io.StringIO'>

# print the text contents
print(file_from_raw_strings.getvalue())

byte_message = b'Hello! This is a file created from bytes'
file_from_bytes = io.BytesIO(byte_message)

print(type(file_from_bytes)) # <class '_io.BytesIO'>

# print the byte contents
print(file_from_bytes.getvalue())

This post on stackoveflow highlights the uses of io.StringIO().

Anywhere you need a file for processing, data transfer over the network, demo purposes, etc. For example, to make a CSV 'file' for testing with pandas:

# Copied from Stackoverflow :)
# https://stackoverflow.com/questions/7996479/what-is-stringio-in-python-used-for-in-reality
import io
import pandas as pd

f = io.StringIO("id,name\n1,brian\n2,amanda\n3,zoey\n")

# Pandas takes a file path or a file-like object 
df = pd.read_csv(f)

Network Response

The urllib.request.urlopen() method returns an HTTP response as a binary file-like object. Therefore, you can treat the response like a binary file. For example

import urllib.request

url = 'https://pandas.pydata.org/docs/user_guide/index.html'

with urllib.request.urlopen(url) as response:
  print(type(response)) # <class 'http.client.HTTPResponse'>

  content = response.read()

  print(type(content)) # <class 'bytes'>

  # Convert the bytes to text
  text_content = content.decode('utf-8')

  print(type(text_content)) # <class 'str'>

Here's an example from the Python documentation. The http response is copied directly to a named temporary file for further processing (since the delete was set to False).

import shutil
import tempfile
import urllib.request

with urllib.request.urlopen('http://python.org/') as response:
  with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
    shutil.copyfileobj(response, tmp_file)

# delete was set to False, so this is allowed
with open(tmp_file.name) as html:
  pass

The shutil module offers several high-level operations on files and directories. Do check it out!

The Terminal (Standard Input and Output)

Python provides the standard input and standard output as text streams through sys.stdin and sys.stdout.

The example below shows a program that reads from sys.stdin and writes to sys.stdout and a file.

import sys

stop_signals = {'q', 'quit', 'exit', 'stop'}

with open('from_stdin.txt', mode='w', encoding='utf-8') as file:
  # Read continuously from the terminal
  for line in sys.stdin:
    line = line.strip()

    if line in stop_signals:
      break

    # Print writes to sys.stdout by default
    # Therefore being explicit here is redundant
    print(line, file=sys.stdout)

    # Write to a given file using print instead of .write()
    # Flush: persist immediately
    print(line, file=file, flush=True)

You can use the fileinput module to replicate the example above by replacing sys.stdin with fileinput.input(encoding="utf-8"). Check out the module documentation for more information.

You can also use the input() function to take a user input from the terminal or sys.stdin. For example

name = input('what is your name: ')

print(f'Your name is {name}')

Getting File Attributes

To assess file attributes such as size, last access time, last modified time, and so on, use the os.stat() function. The required argument is either the file path or the file descriptor (returned by a call to .fileno()). For example

import os
from datetime import datetime

# Using the file stats
file_stats = os.stat('eggs.txt')

# Using the file descriptor of an opened file
with open('eggs.txt', mode='r', encoding='utf-8') as file:
  file_stats_2 = os.stat(f.fileno())

# They are the same
assert file_stats == file_stats_2

print(file_stats)
print(f'The file size is {file_stats.st_size} bytes')
print(f'The file was last accessed on {datetime.fromtimestamp(file_stats.st_atime)}')
print(f'The file was last modified on {datetime.fromtimestamp(file_stats.st_mtime)}')

Use datetime.fromtimestamp() to convert timestamps/seconds into human-readable format

st_size is the file size in bytes
st_atime is the time of most recent access expressed in seconds.
st_mtime is the time of the most recent content modification expressed in seconds.
st_ctime is the time of the most recent metadata change expressed in seconds.

Note that the os.path module provides similar functions such as os.path.getsize(), os.path.getatime(), os.path.getmtime(), and os.path.getctime().

Working with Directories

The os module provides several functions to work with directories. Some of these include:

os.getcwd() - Get the current working directory
os.listdir(directory) - list the content of a given directory. Default to ..
os.chdir(directory) - change/enter into a given directory
os.mkdir(directory) - Make a directory
os.remove(path) - Remove a file in the given path
os.unlink(path) - Remove a file in the given path
os.rmdir(directory) - Remove a directory. Throws an error if the directory does not exist or is not empty.
os.removedirs(name) - Remove directories recursively.

Example: Back Up Recently Modified Files

The program below backs up files modified within the last 24 hours. The explanation of the code is in the comments :)

#!/usr/bin/env python3

"""A Python script to backup files that have been created/modified in the last 24 hours 
   Adapted from a bash script from the Linux Commands & Shell Scripting Course on EDX
"""

import sys
import os
from os import path
from datetime import datetime, timedelta
import tarfile
import shutil

if __name__ == '__main__':
  # Make sure we get exactly two arguments
  # The first argument is the filename. So we check for three(3) arguments
  args = sys.argv
  if len(args) != 3:
    print('Usage: python backup.py target destination')
    sys.exit(1)

  # Unpack the arguments. 
  _, target, destination = args

  # Make sure both target and destination are valid directories
  if not path.isdir(target) or not path.isdir(destination):
    print('Invalid directories provided')
    sys.exit(1)

  # Let's save some variables

  # Get the current time and yesterday's
  current_ts = datetime.now()
  yesterday_ts = current_ts - timedelta(hours=24)

  # The absolute path to the base/root directory
  orig_abs_path = path.abspath(path.curdir)

  # The absolute path to the destination directory
  dest_abs_path = path.join(orig_abs_path, destination)

  # list of files to backup 
  to_backup = []


  # Go into the target directory
  os.chdir(target)

  for file in os.listdir(path.curdir):
    # Get the file stats
    file_stat = os.stat(file)

    # If the last modified date is greater than yesterday
    # add it to the backup list
    if datetime.fromtimestamp(file_stat.st_mtime) > yesterday_ts:
      print(f'Adding file: {file} to list')
      to_backup.append(file)

  # Open a tar.gz file for writing
  backup_file_name = f'backup-{current_ts.timestamp()}.tar.gz'
  with tarfile.open(backup_file_name, "w:gz") as tar:
    for name in to_backup:
      tar.add(name)

  # Move the tar.gz from target to destination directory
  print(f'moving {backup_file_name} to destination')
  shutil.move(backup_file_name, dest_abs_path)

You can extend the program to use zip instead of tar.gz by replacing the appropriate line with the following:

# ...

import zipfile

backup_file_name = f'backup-{current_ts.timestamp()}.zip'

# Write to a zip file
with zipfile.ZipFile(backup_file_name, mode='w') as zf:
  for name in to_backup:
    zf.write(name)

# ...

Do check the Python documentation to learn more about working with zip files, gzip files and shutil.

Summary

In this article, you saw how to read and write to files whether permanent or temporary, saved on disk, from the network, or the terminal. You also caught a glimpse of how to work with directories and how to compress files using tarfile and zipfile modules. Thank you for reading!

Introduction to Python Programming - Organizing Your Code With Modules

Mark Edosa — Thu, 05 Oct 2023 14:44:08 +0000

Introduction
An Example Module
Importing From Modules
Using Modules as Scripts
Some Built-in Modules

Introduction

When writing a Python program, things can quickly get out of hand when using a single file.

While functions can help you group code within a Python file or script, sometimes they are not enough. Organizing related variables and functions in separate files enables code cleanliness, maintenance, and consistency. Each file is a module.

The file name is the module name with the suffix .py appended.

You can import variables and functions from one module into another or the main module (entry point of a Python program). Also, you can run the module as a script from a Python REPL. This way you have flexibility while keeping your code clean and organized.

Note: Within a module, a global variable __name__ (a string) exists and represents the module name.

An Example Module

Programmers typically write their tutorials in markdown. The code sections are often within three backticks. Let's say we want functions that extract and execute each markdown file code block.

You could extract all codes from that markdown file you downloaded from Github.

Or you could ensure that your code runs as expected in the markdown file you created. Like the one I used for this article.

Here's an example module.

Note, I've replaced line.startswith() arguments due to Devto's template errors. See this article's image above for the correct arguments.

# parse.py

import sys

def extract_code_blocks(markdown_file):
    """
      Extracts the code blocks from a markdown file
      Args:
        markdown (file) - The file from which we extract the code blocks
      Returns:
        code_blocks (list) - A list containing the code blocks
    """
    start_of_code_block = False
    code_blocks = []
    codes = ""
    with open(markdown_file, encoding='utf8') as file:
        for line in file:

            # the start of a code block. Move to the next line
            if line.startswith("see_post_image") or line.startswith("see_post_image"):
                start_of_code_block = True
                continue

            # the end of a code block
            if line.startswith("see_post_image"):
                code_blocks.append(codes)
                codes = ""
                start_of_code_block = False
                continue

            # We are now in a code line. Add to the code string
            if start_of_code_block:
                codes += line

            # There's nothing else to do
            continue

    return code_blocks

def write_to_file(filename):
    """
    Write the extracted codes to a file
    Args:
      filename: The filename.md to read from. filename.py will be written to
    """
    code_blocks = extract_code_blocks(filename)
    output, _ = filename.split('.')

    with open(f'{output}.py', 'w') as f:
        f.writelines(code_blocks)


def execute_codes(filename):
    """
    Execute the extracted codes of a markdown file
    Args:
      filename: The filename.md to read from
    """
    for code in extract_code_blocks(filename):
        try:
            # NOTE THAT EXECUTING UNTRUSTED CODES (FROM OTHERS)
            # USING EXEC OR ANY OTHER MEANS IS VERY DANGEROUS. 
            exec(code)
        except Exception as e:
            print(e)


# Run as a script
if __name__ == "__main__":
    args = sys.argv

    if len(args) == 3:

        _, command, filename = args

        if command not in {'execute', 'extract'}:
            print("Usage: python parse.py [execute|extract] file.md")
        elif command == 'execute':
            execute_codes(filename)
        elif command == 'extract':
            write_to_file(filename)
    else:
        print("Usage: python parse.py [execute|extract] file.md")

Importing (from) a Module

Using the import keyword, you can use the functions extract_code_blocks(), write_to_file(), and execute_codes() in another file or module in the following ways:

import the entire module import parse and then use parse.extract_code_blocks() to assess extract_code_blocks() or the other functions.
import all functions (and variables) in the module using asterisks from parse import *. Be careful about doing this to avoid surprises.
import one function from parse import extract_code_blocks.
import more than one function from parse import extract_code_blocks, write_to_file, execute_codes.

# test.py

#1 Import the module
import parse

# use the module functions (or variables if any)
print(parse.write_to_file)
print(parse.execute_codes)
print(parse.extract_code_blocks)

#2 Import everything from the module at once using *
from parse import *

# use them
print(write_to_file)
print(execute_codes)
print(extract_code_blocks)


#3 Import a single function
from parse import write_to_file

# use
print(write_to_file)


#3 Import many functions at one
from parse import write_to_file, extract_code_blocks, execute_codes 

# use them
print(write_to_file)
print(execute_codes)
print(extract_code_blocks)

Be careful when sharing codes between modules so as not to introduce circular dependency. A situation where modules A and B both depend on each other.

You can also import the modules from a Python REPL. To do so quickly, you need to open a Python terminal in the same directory as the module.

Using a Module as a Script

As stated earlier, a global variable __name__ (a string) is the module name. For example

# Another module or REPL
import parse

print(parse.__name__)

When running the module as a script, the value of __name__ is __main__. That means you can detect if your module is a script. For example, the section below enables you to run the parse.py as a Python script.


# Run as a script
if __name__ == "__main__":
    args = sys.argv

    if len(args) == 3:

        _, command, filename = args

        if command not in {'execute', 'extract'}:
            print("Usage: python parse.py [execute|extract] file.md")
        elif command == 'execute':
            execute_codes(filename)
        elif command == 'extract':
            write_to_file(filename)
    else:
        print("Usage: python parse.py [execute|extract] file.md")

The sys.argv contains the arguments passed to a Python script.

To run a module as a script, use python module name one_or_more_arguments_separated_by_spaces in the terminal. For example, run python parse.py execute markdownfile.md to execute all Python code blocks in markdownfile.md. Also, running python parse.py extract markdownfile.md will write the Python code blocks to a file named markdownfile.py.

Some Built-in Modules

The Python standard library consists of packages, modules, and submodules to assist you with common tasks. Here are a few worth mentioning.

Module	Description
`asyncio`	Contains functions for performing asynchronous operations
`csv`	Contains functions for reading and writing CSV files
`datetime`	Contains functions for working with dates and times
`functools`	Contains functions for functional programming
`inspect`	Inspect your python functions or objects
`itertools`	Contains functions for working with iterators
`json`	Contains functions for working with JSON objects
`math`	Contains mathematical functions
`os`	Contains functions for working with the operating system
`pdb`	Useful for debugging your Python codes
`pprint`	Useful for pretty printing
`re`	Contains functions for working with regular expressions
`statistics`	Contains some basic statistics functions
`sys`	Contains functions for working with the operating system
`tkinter`	Useful for creating graphical user interfaces
`timeit`	Time the execution of your functions
`unittest`	Contains functions for testing your Python codes
`urllib`	Contains functions for working with urls

You can find more of these modules here.

Example: Reading and Writing CSV Files

The code below will read a CSV file.

import csv

with open('your_csvfile.csv', 'r') as csv_file:
    reader = csv.reader(csv_file, delimiter=',')
    # If it has headings
    headings = next(reader)

    # The rest of the rows
    for row in reader:
        # do something with each row
        # row[0] contains the first column's cell value
        # row[1] contains the second column's cell value
        # row[2] contains the third column's cell value
        # ...
        # row[n] contains the (nth - 1) column's cell value
        print(row)

The function below will write to a CSV file. rows is a list of dictionaries while header is a list containing the dictionary keys.

import csv

def write_csv(filename: str, header: list[str], rows: list[dict[str, any]]):
    with open(filename, 'w', newline='') as csvfile:
        writer = csv.DictWriter(csvfile,
            fieldnames=header,
            quoting=csv.QUOTE_ALL)
        try:
            writer.writeheader()
            writer.writerows(rows)
        except BaseException as e:
            print(e)


# example
header = ['name', 'age']
data = [dict(name='M', age=11), dict(name='J', age=12)]
write_csv('test.csv', header, data)

Summary

You have seen how to organize your code in separate files called modules. You also saw how to import the content of a module into another module or REPL using the import keyword. The __name__ variable can help detect whether your Python module is running as a script. The Python standard library comes with many packages and modules.

Introduction to Python Programming - Function Basics

Mark Edosa — Sat, 04 Mar 2023 11:44:10 +0000

Table of Content

Introduction
Defining Functions
Calling a Function
Function Parameters and Arguments
Return Statement
Built-In Functions
Lambda Functions
An Example: Validate a Full Name

Introduction

At the end of this article, you will have learned the following:

how to define and use your functions
how to use function parameters and return statements
built-in functions
lambda functions for one-time-only tasks
how to create a function to meet the requirements of a "real-life" scenario.

I assume that you've read the previous articles in this series. Also, you are either a beginner programmer or an experienced programmer learning. Let's get started :)

Let's say you want to calculate the body mass index (BMI) of 20 people. Maybe under different conditions or in other parts of your program. Typically you might be tempted to go straight to the calculation.

Let's assume the height and weight of the first person were 1.71m and 68kg, respectively.


# calculate BMI

# height in meters
height = 1.71

# weight in kilograms
weight = 68

# bmi = weight/height square
bmi = weight / height**2

print(bmi)

How about the second person? You copy the code to where you need it.


# calculate BMI

# height in meters
height = 1.89

# weight in kilograms
weight = 92

# bmi = weight/height square
bmi = weight / height**2

print(bmi)

The third person? Or in a different file? You copy again.

That is way too much repetition that you want to avoid. Luckily, Python functions allow you to group or organize your codes.

An inherent benefit of using functions, apart from logic grouping, is the expression of your intention. Function names are a great way to express your intent to yourself and others who will read your code. For example, let's rewrite the above code that calculates BMI.


# define the function
def calculate_bmi(height, weight):
    bmi = weight / height**2
    return bmi

# call the function and save its output in 
# the result variable
result = calculate_bmi(1.71, 68)

print(result)

You don't have to look at the codes line by line to know what it is doing. It calculates BMI.

In Python, two categories of functions exist - Built-In functions provided by the standard library and user-defined functions (the ones you define yourself). Let's start with user-defined functions since they will enable us to understand Built-In functions.

Defining Functions

To define a function, start by using the def keyword. Followed by the name of the function, a parenthesis, a colon (:), and then the function body. The format is:

def function_name():
    # Do something here in the function body


def second_function(parameter_1, parameter_2, ..., parameter_N):
    # do something here

    return something # to get the result

The function name must follow the rules on variable naming. The parenthesis can contain 0 or more implicitly-defined variables called parameters or arguments. The function body can contain a return statement depending on whether the function generates an output.

Let's examine the calculate_bmi function we created above. It has a name (stating the obvious) and has two parameters (height and weight). It computes the BMI within its body (bmi = weight / height**2) and uses the return keyword to give back the result of the calculation (return bmi). Here's the function again for reference.

def calculate_bmi(height, weight):
    bmi = weight / height**2
    return bmi

Calling a Function

To use (call) a function, write its name and include a parenthesis with or without arguments (or parameters). For example, call the calculate_bmi function like so:

calculate_bmi(1.91, 80)

If the function returns a value, we can (optionally) save the value in a variable.

my_bmi = calculate_bmi(1.91, 80)
print(my_pmi)

Note: Any function that does not explicitly return a value will return None by default.

Function Parameters and Arguments

As earlier stated, parameters/arguments are variables implicitly created based on function definition. They act as the function's inputs.

A function can take multiple parameters as in the calculate_bmi function. Also, a function can have no parameter. For example:

# define a function that takes no argument/parameter
def print_hello_world():
    print("Hello World!")

# call the function
print_hello_world()

One or more arguments can have default values. A function uses the default values if you call it without specifying the argument. For example, sum_num is a function that adds two numbers. By default, those numbers are 5 and 10.

def sum_num(num1 = 5, num2 = 10):
  result = num1 + num2
  return result

We can use sum_num in several ways.

# although no arguments are passed in
# num1 is 5 and num2 is 10. The defaults.
fifteen = sum_num()

# returns 25 since num1 is 13 and num2 is 12
twenty_five = sum_num(13, 12)

# specify num1 (20) and use num2's default value (10)
thirty = sum_num(20)

But wait! How do I specify num2 alone? This question brings us to the concept of positional and named arguments.

Positional and Named (Keyword) Arguments

Let's start by defining a different version of sum_num say sum_num_v2.


# function with 1 positional required argument. 
# num1 and num2 are optional named/keyword arguments
def sum_num_v2(num1, num2 = 5, num3 = 10):
  return num1 + num2 + num3

You can use the return keyword in the same line as the calculation.

We can pass arguments to a function based on the position of the arguments or based on the names (keywords) of the arguments. The function sum_num_v2 has one positional argument num1 and two named arguments num2 and num2. The required position argument num1 must always be in the first position during the function call while the named arguments can either take position 2 or 3. For example

# num1, num2, num3 take their positions - 1,2 and 3 respectively
sum_num_v2(10, 20, 30)

# This is the same as using the positions above
sum_num_v2(10, num2 = 20, num3 = 30)

# The order of num2 and num3 have been switched
# We get the same result as above since
sum_num_v2(10, num3 = 30, num2 = 20)

# we could omit num2 and num3 since they
# have default values.
sum_num_v2(10)

Calling a function without its required arguments throws an error.

# sum_num_v2() # fails :(

In recent Python versions, you can be strict about the number and placement of positional and keyword arguments. For example, consider the describe_me function below:

def describe_me(name, age, city=None, zipcode=None):
  # do nothing
  pass

You can call the function in two ways:

with four positional arguments describe_me("Johnny Depp", 1020, "Caribbean", "123456")
with two positional and two keyword arguments describe_me("Johnny Depp", 1020, city="Caribbean", zipcode="123456")

What if you wanted to be very specific? You can use \ (positional) and/or * (keyword) to restrict the function's use to a single way like so:


def describe_me(name, age, /, *, city=None, zipcode=None):
  pass

# Fails
# describe_me("Peter Griffin", 48, "Quahog", "00093")
# TypeError: describe_me() takes 2 positional arguments but 4 were given

# works!
describe_me("Peter Griffin", 48, city="Quahog", zipcode="00093")

The first two arguments must be positional arguments while the last two must be keyword arguments. You can't call the function with four positional arguments.

Likewise, you can also restrict a function to use only keyword arguments like so:

def I_dont_take_positions(*, city=None, zipcode=None):
  pass


# Fails
# I_dont_take_positions() takes 0 positional arguments but 2 were given
# I_dont_take_positions("Quahog", "00093")

# works!
I_dont_take_positions(city="Quahog", zipcode="00093")

# works! the order does not matter remember?
I_dont_take_positions(zipcode="00093", city="Quahog")

Return Statement

Earlier, you saw the return keyword in the calculate_bmi, sum_num, and sum_num_v2. These functions did some calculations and used the return statement to give back the results.

# Store the calculation in the result variable
result = sum_num_v2(10, 20, 30)

print(result)

In other cases, you can return nothing (None) from a function if you use the return keyword alone. For example, the calculate_bmi function requires the weight and height parameters. What if they're absent? What if you give strings instead?

One way to defend your function is to return early if the parameters are not valid:


# helper function: check if parameter n is a number 
def is_number(n):
  if isinstance(n, int) or isinstance(n, float):
    return True

  return False

# A safer, better version of calculate_bmi

def calculate_bmi_v2(weight, height):
  # Do not calculate BMI if weight or height is not a number
  # return early
  if not is_number(weight) or not is_number(height):
    return 

  bmi = weight / height**2
  return bmi

# works well
print(calculate_bmi_v2(68, 1.61))

# returns None
print(calculate_bmi_v2("Hello", 1))

print(calculate_bmi_v2(1, "Hello"))

# returns None
print(calculate_bmi_v2("Hello", "World"))

Note that in the absence of a return statement, a function returns None implicitly.

You can return a single value or multiple values from a function. To do so, separate the return values with a comma. For example, the generate_123 function below returns three different values-1,2,3.


def generate_123():
  return 1, 2, 3

The result of the function call is a tuple containing the values.

one_two_three = generate_123()

print(one_two_three)

You could also unpack the return values individually.

# Use tuple unpacking
one, two, three = generate_123()

print(one, two, three)

You can use the return statement within an if-statement, a for loop, or a while loop. The function will exit. For example:


def contains_only_even_numbers(numbers):
  for num in numbers:
    # If it's an odd number return False
    if num % 2 != 0:
      return False

  # If we get here, the numbers parameter contains only even numbers
  return True

outcome = contains_only_even_numbers([2, 4, 6])
print(outcome)  # True

outcome = contains_only_even_numbers([1, 2, 4]) # False
print(outcome)  # False

Built-In Functions

Python provides many functions in its standard library to make development easier.

These functions allow you to perform common operations without writing them yourself. For example, you can replace the functions sum_num and sum_num_v2 with the sum function provided by Python.

numbers = [1, 2, 3, 4, 5]

total = sum(numbers)

print(total) # 15

The table below shows some common Built-In functions:

Function Name	Description	Example
`int`	Convert a string/float to an integer	`int(4.8)`
`float`	Convert a string/integer to a float	`float("11.82")`
`str`	Convert any type to a string	`str(1)`
`type`	Check the type of a variable/literal	`type(5)`
`map`	Apply a function to every element of a sequence/collection	`map(your_function, collection)`
`filter`	Apply a predicate to a collection	`filter(your_function, collection)`
`any`	Check if any item in a collection is `True`. Use with `map`	`any(map(predicate, collection))`
`all`	Check if all item in a collection are `True`. Use with `map`	`all(map(predicate, collection))`
`enumerate`	Add indices to a sequence/collection	`enumerate(numbers)`
`zip`	Combine collections	`zip(numbers, letters)`
`min`	Get the minimum value in a sequence/collection	`min([1,2,3])`
`max`	Get the maximum value in a sequence/collection	`max("xab")`
`sorted`	Sort a sequence/collection in a customized way	`sorted([2, 1, 5], reverse=True)`
`reversed`	Reverse the content of a sequence	`list(reversed([2, 4, 5, 1]))`
`open`	Open a file for reading/writing/appending.	`open("my_file.txt", mode="r", encoding="utf8")`
`help`	Provide helpful message on variables, functions, etc.	`help(print)`

A predicate function returns a boolean-True or False

Let's use some of the functions. Start by creating some collections names and numbers

# some names
names = ["John Doe", "Peter Griffin", "Ghost Busters", "Super Mario"]

# numbers 1 - 100
numbers = list(range(1, 101))

map: apply a function to every element of a sequence/collection/iterable (list, tuple, etc.)

Create two functions square and length. square raises a number to power 2. length gives the length of a string.

def square(num):
  return num**2

def length(name):
  return len(name)

Square all the numbers within the numbers list.

squared_numbers = map(square, numbers)

# squared_number is wrapped (a map object)
print(squared_numbers)

# Convert to a list/tuple to view the actual content
squared_numbers = list(squared_numbers)
print(squared_numbers)

Get the length of each name in the names list.

# length names are wrapped (a map object)
length_names = map(length, names)

# Convert to a tuple and view the content
print(tuple(length_names))

# CAREFUL HERE: # tuple is empty because we already used the content
# To avoid this scenario, save the content of the map object.
print(tuple(length_names))

The map function returns a map object, a collection that's not evaluated instantly. The operation runs when you use list or tuple on the map object. Note You shouldn't use list or tuple on the same map object twice. Else, the content of the second call will be empty.

Instead of storing and converting a map object, you can loop through its content. For example:

for length_of_name in map(length, names):
  print(length_of_name)

filter: apply a predicate function to each element of an iterable/sequence

Let's say you only want some names. For example, names that have 13 letters or less. Or names that start with the letter P. Use the filter function here.

def less_or_eq_13(name):
  return len(name) <= 13

def starts_with_p(name):
  return name.lower().startswith("p")

names_lte_13 = filter(less_or_eq_13, names)

pnames = filter(starts_with_p, names)

for name in names_lte_13:
  print(name)

# Just like `map`, `filter` returns a filter object that is used only once
# For multiple uses, save the content in a variable
print(list(names_lte_13))

any and all

def is_multiple_of_5(n): return n % 5 == 0

def is_odd(n): return n % 2 != 0


# check if any of the elements is a multiple of 5
mult_5 = any(map(is_multiple_of_5, numbers))
print(mult_5)

# Check if numbers contain only odd numbers
all_odds = all(map(is_odd, numbers))
print(all_odds)

enumerate

enumerate associates an index with each element of collection/sequence. enumerate returns an enumerate object that can be converted to a list of tuples.

indices_and_names = enumerate(names)

# enumerate object
print(indices_and_names)

# Convert to a list and print
print(list(indices_and_names))

Like map objects and filter objects, you can loop through an enumerate object. For example:


for index, name in enumerate(names):
  print(f"{name} is at position {index}")

zip

Using zip, you can combine sequences to create a zip object. Consequently, the zip object can be converted into a list of tuples or used in a for loop.

For example, you can combine the names list with the length_names to have both a name and its length side by side.

# Create a zipped object
names_length_zipped = zip(names, length_names)

# loop through
for name, length in names_length_zipped:
  print(f"{name} has {length} letters")

You can combine the zip step and the for loop.

# Do the above in a single step
for name, length in zip(names, length_names):
  print(f"{name} has {length} letters")

Detour: Implementing the Map and Filter Functions Yourself

The map and filter functions are similar to the map_ and filter_ defined below.

# many people prefer def map_(f, items)
# I chose to write mine in full for demo purposes
def map_(function, items):
  result = []
  for item in items:
    # Call the function and append its output to result
    result.append(function(item))

  return result


# many people prefer def filter_(f, items)
# I chose to write mine in full for demo purposes
def filter_(function, items):
  result = []
  for item in items:
    if function(item):
      result.append(item)

  return result

Note that these functions map_ and filter_ are evaluated immediately. Don't use in real code.

Lambda Functions

If the question "Do I have to define a function each time I use map and filter?" comes to your mind, the answer is NO!

With lambdas you can define a small function where you need it and forget it never existed. Lambdas also provide the following advantages:

Lambda functions are nameless. No need to put thought into naming such functions.
You won't have to litter your Python script with functions you'll only use once. For example, square, length, is_multiple_of_5, is_odd, less_or_eq_13, and starts_with_p ought to be lambdas.

Excited? How do I create a lambda function?

A lambda function has the following structure: lambda *0_or_more_parameters* : *function body*. No parenthesis or explicit return statement is required. Implicitly, Python returns the result of the expression in the function body.

Let's rewrite some of the lines above using lambdas. First, ditch the square and length functions.

# Initially, you used a square function.
# Replace with a lambda that does exactly the same thing
squared_numbers = map(lambda x: x**2, numbers)

# Convert to a list/tuple to view the actual content
squared_numbers = list(squared_numbers)
print(squared_numbers)

# No need for the length function you created above
length_names = map(lambda name: len(name), names)

# Convert to a tuple and view the content
print(tuple(length_names))

The filter function is next. Let's select only names where the length of the name is less than or equal to 13 using a lambda function.

names_lte_13 = filter(lambda name: len(name) <= 13, names)

pnames = filter(lambda name: name.lower().startswith('p'), names)

print(list(names_lte_13))
print(list(pnames))

How about the any and all?

# check if any of the elements is a multiple of 5
mult_5 = any(map(lambda n: n % 5 == 0, numbers))
print(mult_5)

# Check if numbers contain only odd numbers
all_odds = all(map(lambda n: n % 2 != 0, numbers))
print(all_odds)

An Example: Validate a Full Name

Imagine that your boss says that the name of a user of your application must meet the following requirements:

A space should separate first name and last name
First name and last name must be an alphabet
First name and last name must have at least three letters
The function should return True or False
The function must have two versions-one that uses a for loop and another that uses map

To get you started, your colleague creates two functions that must print OK! if your code is correct.


def assert_true(result):
  if result is True:
    print("OK!")
    return

  raise ValueError("incorrect!")


def assert_false(result):
  if result is False:
    print("OK!")
    return

  raise ValueError("incorrect!")

Next, your colleague adds the following calls/tests to prove your function works.


# Test for valid names using both versions 1 and 2

assert_true(valid_name("Mark Edosa"))
assert_true(valid_name("Jet Lis"))

assert_true(valid_name_v2("Mark Edosa"))
assert_true(valid_name_v2("Jet Lis"))


# Test for invalid names using both versions

assert_false(valid_name("a b"))
assert_false(valid_name("Mark"))
assert_false(valid_name("abc"))

assert_false(valid_name_v2("a b"))
assert_false(valid_name_v2("Mark"))
assert_false(valid_name_v2("abc"))

solution

def valid_name(name):

  name_list = name.split()

  if len(name_list) != 2:
    return False

  for name in name_list:
    test = name.isalpha() and len(name) >= 3
    if not test:
      return False

  return True 


def valid_name_v2(name):
  name_list = name.split()

  if len(list_names) != 2:
    return False

  tests = map(lambda name: name.isalpha() and len(name) >= 3, list_names)
  return all(tests)

Conclusion

Functions help you organize your code. They also help you express your intentions through appropriate names.

In this article, you learned how to define and use your functions. You learned about function parameters and return statements. You also saw some Built-In functions in action. And you saw how to use lambda functions for one-time-only tasks. Finally, you (hopefully) created a function to meet the requirements of a "real-life" scenario.

Thank you for reading.

Introduction to Python Programming - A Word-counter Program

Mark Edosa — Thu, 13 Oct 2022 10:50:46 +0000

Introduction

In the last article, we looked at controlling program execution with if-elif-else statements, match statements, for and while loops, pass statements, and try-except statements. In this, we will create some programs.

A Word Counter Program

Building a word counter will allow us to put some of the concepts we have learned into practice.

Assuming you have a string variable declared literally or a text read from a file, you may decide to know the unique words as well as count them. Let us create a variable called text.

text = "A very long text declared or read from a file. \
    The text may contain very long lines but you should not worry because you have python skills. " \
    "Let us begin the conquest."

If you have a file in the same directory/folder as your python file, you can replace the above variable with

    file = open("your_file_name.txt")
    text = file.read()
    file.close()

Since Python is case-sensitive, words like "The" and "the" are not the same. So, convert the text to lowercase.

text = text.lower()

To get the individual words in the text, use the split() method. Optionally, you can sort the text_split list using the list's sort() method.

text_split = text.split()

text_split.sort()

View the unique words in the list using the set() function.


unique_words = set(text_split)

print(unique_words)

{'read', 'from', 'file.', 'because', 'lines', 'very', 'declared', 'you', 'should', 'text', 'a', 'conquest', 'worry', 'may', 'us', 'not', 'have', 'long', 'python', 'let', 'contain', 'skills.', 'the', 'begin', 'but', 'or'}

Notice that some words like file and skills end with a period (.). Let us remove them using the map() function.

# A map returns a "map object"
text_split = map(lambda x: x[: -1] if x.endswith(".") else x,  text_split)

# convert back to list
text_split = list(text_split)

The map() takes a function and an iterable/sequence and applies the function to every element of the sequence. The lambda you see above is used to create a one-time-use function. We will see more of map and lambda in my next article on functions :)

The expression within the lambda function x[: -1] if x.endswith(".") else x is an example of an if expression. You will find it in the previous article. If the word ends with a period take a slice excluding the last [: -1]. If not, give me back the word.

print(text_split)

['a', 'a', 'because', 'begin', 'but', 'conquest', 'contain', 'declared', 'file', 'from', 'have', 'let', 'lines', 'long', 'long', 'may', 'not', 'or', 'python', 'read', 'should', 'skills', 'text', 'text', 'the', 'the', 'us', 'very', 'very', 'worry', 'you', 'you']

Next, create an empty dictionary to hold the words and their count. Let us call the variable summary.

summary = {}

Using a for loop, we will go through the text_split list and if we see a word already in the dictionary, we will increase its count by 1 else, we will add the variable to the dictionary and set its count to 1. For example

for word in text_split:
    if word in summary:
        summary[word] = summary[word] + 1
    else:
        summary[word] = 1

The word in summary checks if the word is already a key in the dictionary.

Here is a shorter way to re-write the code above using the dictionary's get() method.

summary2 = {}

for word in text_split:
    summary2[word] = summary2.get(word, 0) + 1

The summary and summary have the same content.

print(summary == summary2)

True

summary2.get(word, 0) + 1 get the value/count associated with word. If there is nothing there that is, the word is not a key in the dictionary, give me zero (0) as the count. Add 1 to either set the summary[word] to 1 or increment the value contained in summary[word].

Finally, print the content of the summary. The order of the content may differ from yours but they will be the same.

print(summary)

{'a': 2, 'because': 1, 'begin': 1, 'but': 1, 'conquest': 1, 'contain': 1, 'declared': 1, 'file': 1, 'from': 1,'have': 1, 'let': 1, 'lines': 1, 'long': 2, 'may': 1, 'not': 1, 'or': 1, 'python': 1, 'read': 1, 'should': 1, 'skills': 1, 'text': 2, 'the': 2, 'us': 1, 'very': 2, 'worry': 1, 'you': 2}

To obtain a much prettier print, use the pp() method in the pprint module. We have not discussed modules so do not worry about it. Just type it for now.

from pprint import pp

# remember that summary and summary2 have the same content
pp(summary2)

{'a': 2,
'because': 1,
'begin': 1,
'but': 1,
'conquest': 1,
'contain': 1,
'declared': 1,
'file': 1,
'from': 1,
'have': 1,
'let': 1,
'lines': 1,
'long': 2,
'may': 1,
'not': 1,
'or': 1,
'python': 1,
'read': 1,
'should': 1,
'skills': 1,
'text': 2,
'the': 2,
'us': 1,
'very': 2,
'worry': 1,
'you': 2}

Now, here is the full program.

# This could come from a file
text = "A very long text declared or read from a file. \
    The text may contain very long lines but you should not worry because you have python skills. " \
    "Let us begin the conquest"

# Convert to lower case
text = text.lower()

# Create a list of words
text_split = text.split()

# Sort in alphabetical order
text_split.sort()

# Get a set of unique words
unique_words = set(text_split)

print(unique_words)

# Go through each word and remove any period (.)
text_split = map(lambda x: x[: -1] if x.endswith(".") else x,  text_split)

# Convert the map output back to a list
text_split = list(text_split)

print(text_split)

summary = {}

for word in text_split:
    if word in summary:
        summary[word] = summary[word] + 1
    else:
        summary[word] = 1


summary2 = {}

for word in text_split:
    summary2[word] = summary2.get(word, 0) + 1


print(summary == summary2)

# print(summary)

from pprint import pp

pp(summary2)

Conclusion

In this article, we created a word-counter program. If we had read a file, we could include the number of lines and the size of the file in bytes just like the wc command in a linux shell. In the next article, we will deal with functions. Thanks for reading.

Introduction to Python Programming - Control Flow

Mark Edosa — Thu, 06 Oct 2022 05:39:46 +0000

At some point, while writing a program, you would need to make decisions either based on your business logic or based on user input from a client such as a shell or a webpage. Also, you may want some parts of your program to run repeatedly. This article shows you how you can control the flow of your program in Python.

Conditions
- If-Elif-Else Statements
- Nested If Statements
- If-Else Expression
- match statement
Loops
- For Loop
- While Loop
- Break and Continue Statements
Pass statement
Try-Except Statements

Conditions

If-Elif-Else Statements

Use an if or if-elif-else statement when you want to perform an operation or logic based on some conditions. The overall format is:

# Do not run this!
# It's just for illustration

if condition1:
    # Run if the condition is true, avoid the others if they specified
elif condition2:
    # Run here if condition1 is false and condition2 is true
else:
    # Do something else

Note that in Python, indentation matters. In the above statement, there are three blocks or indents - The if, elif, and the else blocks.
Among the three, only the if block is required. The elif and else parts are optional.

Let's say we run a restaurant that only allows people 18 years and above. We may set a variable called age_allowed to the number 18. Then we check if the customer's age matches our expectations. Let's set three variables associated with age. Also, add two string variables as messages to the customers.

# The allowed age
allowed_age = 18

# Age of the first customer
age_1 = 57

# Age of the second customer
age_2 = 17

# Some message to the customer depending on our criterium
not_ok_message = "You are not allowed to enter!"
ok_message = "Welcome to the DELISH haven!"

We do the tests:

# True: prints the ok_message
if age_1 >= allowed_age:
    print(ok_message)


# False: does not print the ok_message
if age_2 >= threshold_age:
    print(ok_message)

Print an "okay message" if the customer's age is higher or equals the allowed age. We used two if statements or blocks, and that's fine. However, what if the test fails? We could use another if block or add an else block. For example:

# True: prints the ok_message
if age_1 >= allowed_age:
    print(ok_message)
else:
    print(not_ok_message)

# False: prints the not_ok_message
if age_2 >= allowed_age:
    print(ok_message)
else:
    print(not_ok_message)

What if we have multiple conditions that are different? We test many conditions using elif blocks. Only the block whose condition passes executes. If available, an else block executes if all tests fail.

Let's say we give out separate seats to customers based on their age group. For customers between 18 and 27, we will give them the first-row seats, those between 28 and 37 will take the second-row seats, and so on. Let's assign a seat to a customer!

# Yeah, a_customer
a_customer = 45

# I like laziness. Let's save typing
message = "Please take the {} seat"

if a_customer < 18:
    print("Sorry, our DELISH only allows customers 18 and above!")

# checks between 18 and 27
elif age <= 27:
    print(message.format("first"))

# checks between 28 and 37
elif age <= 37:
    print(message.format("second"))

# checks between 38 and 47
elif age <= 47:
    print(message.format("third"))

# 81 and above
else:
    print("Well, this part is not always necessary!")

Wondering about the message variable in the above code block? Remember that Python has a string format function/method that lets you do something like this "My name is {}, and I am {} years old".format("Peter Griffin", 90).

One last example, just for fun!

# Reset age_1 and age_2

age_1 = 5
age_2 = 30

pro_footballer = True

if (age_2 > 18) and pro_footballer:
    # runs
    print("Seems appropriate, maybe! I guess you deserve more credit!")
else:
    print("Dude? why not!")

if (age_1 < 10) and pro_footballer:
    print("You are a prodigy! Have you seen the news?")
else:
    print("You probably read too much!")

Nested If Statements

An if statement (top-level) can contain other if statements (child-level). The child-level or contained if statements can contain other if statements, and so on. For example

# Do not run this!
# It's just for illustration
if condition:
    # top level if block
    if another_conditon:
        # second-level if block
        if yet_another_condition:
            # you get the gist
        else:
            # definitely!
    else:
        # second-level else block
else:
    # top/first-level else block

Heads up! Use nested ifs wisely. Nesting too deep is terrible for eggs! Sorry :)

If-Else Expression

An if-else expression has the pattern True if condition else False. This expression is "give me the True part if the condition is true or, give me the False part". For example, chosen_age is age_2 if we want a pro_footballer else it is age_1.

chosen_age = age_2 if pro_footballer else age_1

print("The chosen age is {} years old".format(chosen_age))
# The chosen age is 30 years old

Match Statement

You can use a match statement in place of an if-elif-else with multiple elif-blocks. The format is

match an_expression:
    case option_1:
        # do something if an_expression equals/matches option_1
    case option_2:
        # do something if an_expression equals/matches option_2
    ...
    case option_n:
        # do something if an_expression equals/matches option_n
    case _:
        # do something if an_expression does not match any of the provided options

Only the code in the first block that matches the expression executes. In the case of no match, the default case block case _ runs.

Let's say we have a list of fruits and would like to react to a customer's choice of the best fruit.


best_fruit = 'oranges'

match best_fruit:
    case 'apples':
        print('nice fruit..but not as nice as...whatever!')
    case 'oranges':
        print('even better...Lorem ipsum...Don\'t worry')
    case 'mangoes':
        print('You made a great choice!')
    case 'banana':
        print('We are quite far from the trees...Might you pick something else?')
    case _:
        print('Don\'t know what this is...Try again?')

The code above will print 'even better...Lorem ipsum...Don't worry' because best_fruit is oranges in the example.

If you want to react to multiple options, use the pipe (|) operator. For example


food = 'rice'

match food:
    case 'rice' | 'potatoes' | 'corn':
        print('We are rich in carbohydrates, and you can spend it!')
    case 'carrot' | 'cabbage' | 'lettuce':
        print('What a bunch of vegetables!')
    case _:
        print('Probably an Aliens\' diet!')

If a user selects any of rice or potatoes or corn then 'We are rich in carbohydrates, and you can spend it!' is printed. Conversely, the same code runs if the user chooses carrot, cabbage, or lettuce.

Loops

Using loops, we can repeat specific parts of our code. Or we can go through sequences or dictionaries one item at a time. : The for and the while loops are types of loops in Python.

For Loops

The for statement allows us to go through the elements of a collection (such as list, tuples, dictionary, string) one after the other (iteration). It follows the following pattern:

for element in collection:
    # Do something with the element

The element variable represents each item in the collection during iteration. Note that the element can take any name. Let's see some examples.

We start by creating four variables, my_list, a list of numbers. A list containing tuples list_of_tuples, menu a dictionary, and name a string.


my_list = [1, 2, 3, 4, 5]

list_of_tuples = [(1, 2), (3, 4), (9, 10), (11, 12)]

menu = { "rice": 45, "apples": 33, "garlic": 20, "pepper": 10 }

name = "John Doe"

Let's print two statements using the my_list elements.

for num in my_list:
    print("The number is {}".format(num))
    print(f"num times 2 is {num * 2}\n") # notice the \n

The number is 1
num times 2 is 2

The number is 2
num times 2 is 4

The number is 3
num times 2 is 6

The number is 4
num times 2 is 8

The number is 5
num times 2 is 10

\n is called a newline character. The pattern f"{num}" is equivalent to "{}".format(num).

Using tuple unpacking, we can retrieve the elements of the tuple in the list within the iteration.

for first, second in list_of_tuples:
    print(f"First is {first}, second is {second}")

First is 1, second is 2
First is 3, second is 4
First is 9, second is 10
First is 11, second is 12

First run: first, second = (1, 2). Second run: first, second = (3, 4). Third run: first, second = (9, 10). Fourth run: first, second = (11, 12).

We get a series of letters when we loop through a string. For example:

for letter in name:
    print(letter)

J
o
h
n

D
o
e

Looping Through a Dictionary

Iterating through a dictionary is not so straightforward. We can loop through the dictionary keys using the keys() method and through the values using the values() method, or both (items()).

# Loop through the keys only
for key in menu.keys():
    print(key)

rice
apples
garlic
pepper

Omitting the keys() method yields the same result.

# Same as above
for key in menu:
    print(key)

rice
apples
garlic
pepper

Use the values() method to retrieve only the values:

# Loop through the values only
for value in menu.values():
    print(value)

Use the items() method to retrieve the keys and values!

# loop through the keys and values
for key, value in menu.items():
    print("{}: {}".format(key, value))

rice: 45
apples: 33
garlic: 20
pepper: 10

While Loop

The template is:

while condition_is_true:
    # Keep doing whatever we put here

Let's say we want to print out some numbers 1 to 5. We could set a count variable and increment it within a loop.

count = 1

while count <= 5:
    print(count)
    count += 1  # increment the count

To avoid an infinite loop, the while condition must be evaluated as False at some point. We increment the count variable count += 1 to ensure that the count becomes greater than five eventually.

Operators like += -= \= *= allows us to perform arithmetics and assignment in a single step. For example,
a = a + 1 can be written as a += 1. So also a = a * 7 can be replaced with a *= 7 and so on.

Note that just like the if statement, loops can be nested.

Give me all even numbers between 1 and 20:

# We're starting from zero
count = 0

while count <= 20:
    if count % 2 == 0:
        print(count)
    count += 1  # increment the count

Break and Continue Statements

The break statement helps us control a loop's termination. The continue statement skips an iteration step.

In the example below, we want to print the multiples of 5, specifically from 5 to 20. We set the counter to 5 and then increment by 5. When the counter becomes 20, we exit the while loop using a break statement.

counter = 5

while True:
    print(f'counter is {counter}')
    if counter == 20:
        print("**** stopping here ****")
        break
    counter += 5

counter is 5
counter is 10
counter is 15
counter is 20
**** stopping here ****

To demonstrate the continue statement, create a numbers list like the one below.

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]

If we want the odd numbers between 1 and 11, we will write:


for n in numbers:
    if n % 2 == 0:
        continue # skip even numbers
    print(n)
    if n == 11:
        break  # stop at 11

In the code above, we skipped the even numbers with continue and then exited with break when the current number n was 11,

How do I choose between a for and a while loop? It largely depends on the situation. If you're dealing with a collection, a for loop is preferable most of the time. In other cases, consider a while loop.

Pass Statement

Like comments (#), the pass statement tells Python to ignore parts of a program. However, the pass statement is excellent for ignoring code blocks. This code block may be if-elif-else, a loop block, or a function or class block. For example:


# I'm still thinking of what to do
# So I'll just use a pass so Python won't complain
num_years = 45

if num_years > 80:
    pass
else:
    pass


# I'm not ready to work with num
for num in [1, 2, 3, 4, 5]:
    pass

Try-Except Statement

The try-except statement enables us to perform an alternative action if the intended action fails. For example, we may want to open a file, convert a string to an integer, and so on. These attempts might fail, resulting in an error that may crash our program. The try-except statement solves this problem. The format is:

try:
    # Do something that may fail
except:
    # Do something else. For example, handle the error and inform the user of your program

The example below demonstrates the many uses of try-except. Converting from integers may produce errors. To guarantee the continuation, use a try-except statement.

bad_integer_string = "aa12r"

good_integer_string = "12345"

try:
    good_integer = int(good_integer_string)
    print("Worked! The number is", good_integer)
except:
    print("Failed")

Worked! The number is 12345

try:
    bad_integer = int(bad_integer_string)
    print("Worked??? The number is", bad_integer)
except:
    print("Failed!!!")

Failed!!!

Conclusion

In this article, we looked at controlling program execution with if-elif-statements statements, match statements, for and while loops, pass statements, and so on. Next, we will write some programs. Thanks for reading!

Introduction to Python Programming - Data Structures

Mark Edosa — Fri, 09 Sep 2022 20:18:33 +0000

Overview
Sequences
- List
- Tuples
Set
Dictionary

Overview

Data structures are containers that enable storage, grouping, access, iteration, and manipulation of basic data types. Data structures include Sequences, Set, and Dictionary.

Sequences

A sequence is a container holding data (elements) in an ordered manner. Sequence types include list, str, tuple, and bytes. Among them, only list is mutable while the rest are immutable. The sequence below contains 10 integer elements (n = 10).

The element at the first position takes an index value of 0 while the last takes an index value of n - 1, 9.

Note: str refers to strings.

Operations on Sequences

You can check if an element exists in a sequence using the in keyword or operator. The result is the boolean value True or False. You can also do this for other containers like Set and Dictionary. For example
```
sequence = [99, 22, 43, 40, 85, 66, 17, 81, 2, 8]

40 in sequence # True
82 in sequence # False
```
If you put the code in a Python file (.py), use the print statement to see the output. For example print(40 in sequence). At a Python REPL, you can run the code as is.
You can access the elements of a sequence using the integer indices. This operation, known as indexing is fast. For example, to get the fifth (5) element at index position four (4), you would write
```
sequence[4] #85
```
To retrieve a slice of elements, specify the start and end index.

For example, the code below will return a slice of the first to the fifth element.
```
sequence[0:5] # [99, 22, 43, 40, 85]
```
Note that the sixth element (at index 5) is excluded. You may shorten the above statement to sequence[:5] by omitting the start index. You may also omit the end index in certain cases. Omitting the start index will return a slice from the start index to the rest elements. For example sequence[3:] will yield [40, 85, 66, 17, 81, 2, 8]. Also, omitting the start and end index together (sequence[:]) will return a slice containing all elements.
Sequences also support iteration. You can go through each element in a sequence using a for loop statement (you will see more loops in my next article). For example, in the for loop below, we multiply each element by three (3) and print the results.
```
for element in sequence:
    print(3 * element)
```
Other operations on sequences include counting using the len() function; reversing, appending, and popping using reverse(), append() and pop() methods/functions in the case of mutable sequences. In list and str, you can count the number of times an element appears using the count() method. You can find an element's position using the find() or index() method if the method(s) is available on the sequence. For example
```
len(sequence) # 10

a_string_is_also_a_sequence = "Hello reader!"

len(a_string_is_also_a_sequence) # 13

sequence.index(81) #7 meaning 81 is at index 7

a_string_is_also_a_sequence.count("l") #2, letter l occurs twice
```

Let's look at the sequence types starting with the list data structure.

List

The list data structure is an ordered sequence of elements enclosed in square brackets ([]). The content could be any data type or data structure, including a list. Since lists are mutable, adding or removing elements modifies the list.

Creation

Use the literal square brackets [] or the list() function. Below is a list of names, numbers, two empty lists, and so on.

names = ["John", "Doe", "Peter", "Griffin", "Lois", "Lane"]

numbers = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]

empty_1 = []

empty_2 = list()

# you can convert a sequence to a list using the list function
list_of_letters_abcd = list("abcd")

# The string method split creates a list of strings
list_of_letters_efgh = "e f g h".split(" ")

A list can contain other lists (nesting). For example


nested_list = [names, numbers]

Note: A list can contain a mixture of types e.g. ["a", 1, [], 2.3]. However, you should not do this as it makes working with such a list difficult.

Operations

Indexing (Access) and Slicing

As mentioned above, you can retrieve the elements of a list by indexing and slicing.

# Indexing
# The first element or name
names[0]  # 'John'

# The third name starting from the back
names[-3]  # 'Griffin'

# Slicing

# The first three numbers
numbers[0:3]  # [10, 20, 30]

# same as above(we can omit the start index)
numbers[:3] # [10, 20, 30]

# The third number to the end
numbers[2:] # [30, 40, 50, 60, 70, 80, 90, 100]

Modifying (Mutation)

# Replace the first element 'John' with 'James'
names[0] = "James"

# Replace 'Lois' with 'Mary'
names[-2] = "Mary"

names # ['James', 'Doe', 'Peter', 'Griffin', 'Mary', 'Lane']

Deletion

To delete an element from a list, use the del keyword. For example, del some_list[0] will remove the item in the first position. Afterward, the list is re-ordered so that the element initially at position 2 (index 1) takes position 1 (index 0), and so on.

Copying

Assigning a list variable to a new variable does not create a copy. The new variable refers to the original one. Python does what we call referencing behind the scenes. Referencing saves memory. For example,

numbers_a = numbers

numbers_a == numbers # True

We can check if both lists point to the same object using the is keyword. For example

numbers_a is numbers  # True

numbers_a and numbers are equal and the same. Modification to either will reflect in both variables. Let us add 110 at the end of the list using numbers_a.

# We add 110 to the numbers_a list
numbers_a.append(110)

Print the content of the list through the numbers variable.

# numbers also contains the '110' added to numbers_a
numbers # [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110]

In cases where you need to make a copy of the original list, use either a full slice [:], the list() function, or the copy() method.

#1 Copying with a full slice
a = numbers[:]

# Copying the list function
b = list(numbers)

# Use the copy method
c = numbers.copy()

Concerning content, the new copy is equal to the original list. However, the lists are not identical.

# Do we have the same elements?
a == numbers # True
b == numbers # True
c == numbers # True

# Are we identical?
a is numbers # False
b is numbers # False
c is numbers # False

a and numbers are equal but not the same. They are still equal because they contain the same elements. Variables b and c are also equal but not identical.

Addition and Multiplication

You can combine two or more lists with the + operator. For example

[1, 2] + [3, 4]  # [1, 2, 3, 4]

You can also use the * operator to generate a new list containing repetitions of the elements of the old list. For example

[1] * 5 # [1, 1, 1, 1, 1]

[1, 2] * 5 # [1, 2, 1, 2, 1, 2, 1, 2, 1, 2]

Some List Methods

List methods include but are not limited to append(), pop(), reverse(), index(), and count().

# append: add elements to the end/back of the list

names.append('Owen')
names.append('Wilson')

names # ['James', 'Doe', 'Peter', 'Griffin', 'Mary', 'Lane', 'Owen', 'Wilson']

# pop: remove and return the last element from a list
# Specify an index to remove the item in that position
# The removed item is returned (can be stored in a variable)

names.pop() # 'Wilson'

names  # ['James', 'Doe', 'Peter', 'Griffin', 'Mary', 'Lane', 'Owen']

# pop(index): remove and return an element at a given index name.pop(0) # 'James'

names  # ['Doe', 'Peter', 'Griffin', 'Mary', 'Lane', 'Owen']

# reverse: Reverse the positions of the element

# before reversal
numbers # [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110]

numbers.reverse()

# after reversal
numbers # [110, 100, 90, 80, 70, 60, 50, 40, 30, 20, 10]

# index: get the position/index of an element.
# returns a ValueError if the element is not found
names.index('Lane') # 4

# count: how many "Peter" do we have in the names list
names.count('Peter') # 1

# Use the len function to determine the number of elements in a list
len(numbers) # 10

Tuples

Tuples are very similar to lists except that they are immutable and their elements are enclosed within a parenthesis (). Prefer tuple to list if you do not want the content of your collection to change.

Creation

Use the literal parenthesis () or the tuple() function.

# Literally
empty_tuple = ()

my_tuple = (1, 2, 3)

# you do not always need the parenthesis
my_other_tuple = 4, 4, 5, 5, 5

d = 1,

# from sequences such as string, list
names_tuple = tuple(names)

numbers_tuple = tuple(numbers)

Note that a single element in () without a comma , will not give you a tuple.

not_a_tuple = (2)

type(not_a_tuple) # <class 'int'>

names_tuple # ('Doe', 'Peter', 'Griffin', 'Mary', 'Lane', 'Owen')

numbers_tuple # (110, 100, 90, 80, 70, 60, 50, 40, 30, 20, 10)

my_other_tuple  # (4, 4, 5, 5, 5)

Operations

You can retrieve elements of a tuple via indexing and slicing.

names_tuple[2] # 'Griffin'

names_tuple[:4] # ('Doe', 'Peter', 'Griffin', 'Mary')

However, you cannot modify a tuple.

names_tuple[0] = 'Fail'

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-106-f0d527e00ac0> in <module>
----> 1 names_tuple[0] = 'Fail'


TypeError: 'tuple' object does not support item assignment

Like all sequences, you can count the number of elements in a tuple using the len function.

# count
len(names_tuple) # 6

You can also use the list() function to create a new list from a tuple.

# convert to list
list(names_tuple)  # ['Doe', 'Peter', 'Griffin', 'Mary', 'Lane', 'Owen']

Like in a list, the + and * operators perform similar actions.

# Combine tuples: create a new one
my_tuple + my_other_tuple # (1, 2, 3, 4, 4, 5, 5, 5)

# Repeat the elements and create a new tuple
my_tuple * 5  # (1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3)

Tuple unpacking

You can assign the elements of a tuple to variables using tuple unpacking. For example, the variable my_tuple contains three elements.

my_tuple # (1, 2, 3)

To store these three elements in three different variables, you would write

# unpack my_tuple

first, second, third = my_tuple

first # 1

second # 2

third # 3

Tuple Methods

index() and count() are examples of methods/functions of a tuple.

# How many 5s do we have in the my_other_tuple variable
my_other_tuple.count(5) # 3

# Mary is at which position/index?
names_tuple.index('Mary')  # 3

Check out my previous article on data types where I discussed strings briefly. Note that the bytes sequence is similar to strings. So, I will not discuss it in this article :)

Sets

A set is a mutable collection of unique elements enclosed in curly braces ({}). However, unlike lists, tuples, and strings, sets are not ordered. So you cannot index into or slice sets.

Creation

Use either literal {} or the set function.

letters_set = {'a', 'b', 'c', 'c', 'a', 'b', 'c'}

numbers_set = set([1, 2, 2, 3, 3, 5, 5, 4, 4, 2, 5])

empty_set = set()

not_an_empty_set = {}

type(empty_set) # set
type(not_an_empty_set) # dict

Note that an empty {} gives an empty dictionary and not an empty set. Use set() to create an empty set.

numbers_set  # {1, 2, 3, 4, 5}

letters_set # {'a', 'b', 'c'}

Set Operations

Let's create some sets to perform some set operations.

set_1 = {1, 2, 3, 4, 5}
set_2 = {1, 2, 3}
set_3 = {4, 5}
set_4 = {6, 7}

Union: all elements in both sets

set_1.union(set_2)  # {1, 2, 3, 4, 5}

set_1.union(set_4)  # {1, 2, 3, 4, 5, 6, 7}

Intersection: Only elements that appear in both sets

set_1.intersection(set_2) # {1, 2, 3}

set_1.intersection(set_3) # {4, 5}

Difference: elements present in the first set but not in the second set

set_1.difference(set_2) # {4, 5}

set_3.difference(set_4) # {4, 5}

Set Methods

add, update, union, intersection, and difference are some examples of set methods. See python documentation for more information.

Update: fill a set with the elements of another set

set_1.update(set_4)

set_1  # {1, 2, 3, 4, 5, 6, 7}

Add: Add element(s) to a set

set_1.add(8)
set_1.add(9)
set_1.add(9)
set_1.add(10)

set_1  # {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}

Remove an element from a set


# remove 10. Then remove 1
set_1.remove(10)
set_1.remove(1)

set_1  # {2, 3, 4, 5, 6, 7, 8, 9}

Dictionaries

Just like the English dictionary, a Python dictionary is a collection of keys and their values (key-value pair) enclosed in curly braces ({}). Like sets, dictionaries are mutable and unordered

Note: Dictionary keys must be immutable (for example, strings, and integers). Therefore, a mutable type (such as a list) cannot be a dictionary key. Dictionary values can be of any data type, including a dictionary.

Creation

Use the literal curly bracket {} in the following format { "key": value } or use the dict function.

# Literal
items = { "pen": 30, "chalk": 20, "book": 45 }

empty_dict = {}

# Use the dict function
same_items = dict(pen=30, chalk=20, book=45)

# dict function taking a list of tuples
its_all_i_see = dict([("pen", 30), ("chalk", 20), ("book", 45)])

empty_dict_2 = dict()

items  # {'pen': 30, 'chalk': 20, 'book': 45}

same_items # {'pen': 30, 'chalk': 20, 'book': 45}

its_all_i_see # # {'pen': 30, 'chalk': 20, 'book': 45}

type(items) # dict

key_types = {"string": 1, 1 : 5, (5,): "tuple" }

key_types  # {'string': 1, 1: 5, (5,): 'tuple'}

A mutable key like a list, set, or dictionary will result in an error.

key_types_fail = { [1, 2]: "whatever" }

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-163-59389ba2d9af> in <module>
----> 1 key_types_fail = { [1, 2]: "whatever" }


TypeError: unhashable type: 'list'

Operations

Access and Modification

Use dictionary[key] or dictionary.get(key) to retrieve the value associated with the key.

items["pen"] # 30

items.get("pen") # 30

# Return the value 0 if there is no pen.
items.get("pen", 0)

dictionary[key] throws an error if the key does not exist. This may crash your program. Prefer .get(key) as it returns None if the key is not present. With the get() method, you can also specify a default value to return if the key is present

Use dictionary[key] = value to add a new item to a dictionary or to modify an existing key with a new value. For example:

# Add a new item table to the items
items["table"] = 100

items # # {'pen': 30, 'chalk': 50, 'book': 45, 'table': 100}

# change the value of chalk to 50. The original was 20

items["chalk"] = 50

items  # {'pen': 30, 'chalk': 50, 'book': 45, 'table': 100}

Deletion

# Remove table from the dictionary
del items['table']

Addition and Multiplication

The dictionary type does not support the + and * operators.

del dictionary[key] throws a KeyError if the key is not in the dictionary.

Dictionary Methods

Some dictionary methods are keys, values, clear, items, and update.

same_items  # {'pen': 30, 'chalk': 20, 'book': 45}

# get a list of keys only using the keys() method
same_items.keys()  # dict_keys(['pen', 'chalk', 'book'])

# the values only
same_items.values()  # dict_values([30, 20, 45])

# The items method gives you a tuple of the keys and values of the dictionary
same_items.items() #dict_items([('pen', 30), ('chalk', 20), ('book', 45)])

# clear the dictionary
same_items.clear()

# Now empty
same_items # {}

# add the elements from another dictionary
same_items.update(items)

same_items # {'pen': 30, 'chalk': 50, 'book': 45}

Conclusion

In this article, we have looked at some data structures. In the next, we will look at control flow in Python. Please follow me on Twitter to get notified whenever I release a new article. Thanks for reading!

Introduction to Python Programming - Data Types and Operators

Mark Edosa — Sun, 28 Aug 2022 10:25:00 +0000

TLDR

Numeric types are either by nature whole numbers like 1, 2, 3) or contain decimal places like 2.3, 8.22.
You can perform arithmetic operations on integers and floats using + - * / %. For example: 2 + 2, 9 - 6, 8 * 3, 5 / 2 and 15 % 2.
Booleans such as True and False are used to influence the direction of flow of a program. False values include 0, '', [], {}, set().
Comparison operators (> < >= <= == !=) and logical operators (and or not) yield booleans. For 5 > 3 is True while 9 < 3 is False.
Strings are immutable sequences of characters enclosed in single or double quotes. Examples include "Mary Doe", 'John Doe', and so on.
Common string operations include
- concatenation e.g. "John " + "Doe" gives "John Doe",
- indexing e.g. "John Doe"[0] returns 'J' and "John Doe"[-1] returns 'e' and
- slicing, for example, "John Doe"[0:3] returns "Joh".
Common string methods include replace(), strip(), format(), lower(), upper() and title().
Check the type of a value with the type() function.
int(), float() and str() can be used for basic type conversions.

Data Types

The basic data or value types in python are integers, floats, booleans, and strings. These types are often written literally or stored in variables. Behind the scenes, memory is allocated whenever these types are created in a python program.

There is a type None which represents the absence of a value.

Integers and Floats

In python, numbers might be whole (integers) or might contain decimal places (floats). You can use these types to represent attributes like age, height, and weight and concepts like temperature, price, and so on. Addition information can be found here.

Creation

You can type the number literally. For example, 15, 23, 56, etc., or use the int() or float() function (more on this later).

# Age in years
age = 54

# let's say height in feet
height = 7.2

The line beginning with # is called a comment. Python ignores it. So you can use it to document your code for yourself and others.

To run the code, you can either copy and paste it into a python shell or file. If you used a python file, you need to use the print() function to see the output. For example print(height) will display 7.2 in the python console or REPL.

age and height above are called variables. A variable is a symbol that represents a python data type. Think of it like an address to a house (data type). To create a variable in python, use the assignment operator (=).

A variable's value can change throughout the program. So, we say python is dynamically-typed.

Certain rules apply when naming python variables. For example, python variables cannot start with numbers and can not be any of the reserved keywords. Use help("keywords") to see a list of researched keywords.

Check out this post for more information.

Operations

The most basic operation we perform with these types is arithmetic operations such as addition, subtraction, multiplication, and division. For example:

# Addition (with the + operator)
2 + 6 # 8

# Subtraction
6 - 5 # 1

# Multiplication
4 * 9 # 36

# Division

9 / 2 # 4.5
9 / 3 # 3.0

# Remainder division
8 % 3 # The remainder of 8/3 is 2

# Floor division
9 // 2 # 4

The arithmetic operators include +, -, *, /, %. The modulo (%) operator does remainder division. For example, you can check if a number is even or odd or a multiple of 5 and so on.

To use the integers and floats in several places and to communicate your intent, you need to give them a name or assign them to a variable. This use of variables applies to every other data type.


# create two variables
number_of_fish = 20
amount = 5.2

# use the variables
total_price = number_of_fish * amount # 104.0

# print(total_price)
# >>> total_price

Boolean Data Type

Booleans represent the truthiness of a value. Simply, True and False

Creation

#1 By literal boolean assignment

is_sweet = True
is_bitter = False

#2 From the result of a comparison or logical operation

# comparison
three_is_bigger_than_2 = 3 > 2  # True

is_bitter and is_sweet # False

Operations

The comparison operators include equal (==), not equal (!=), less than (<), greater-than (>), greater than or equal (>=), and less than or equal (<=). For example

3 == 3 # True, 3 is indeed equal to 3

6 != 7 # True, 6 is not equal to 7

8 > 18 # False, 8 is not greater than 18

2 < 10 # True, 2 is less than 10

Quick Exercise

# what's the answer
100 >= 45

number_1 = 45
number_2 = 67
number_1 <= number_2 # What's the answer?

The logical operators include and, or, and not. For an and operation to be true, both sides or operands must be true.

AND Truth Table

	True	False
True	True	False
False	False	False

True and True is True
True and False is False
False and True is False
False and False is False

For an or operation to be true, one side (operand) only needs to be true.

OR Truth Table

	True	False
True	True	True
False	True	False

True or True is True
True or False is True
False or True is True
False or False is False

is_sweet and is_bitter # False
is_sweet or is_bitter # True

Strings

A string is a set of characters (letters, numbers, and symbols) surrounded by single ' or double " quotes.

Strings are useful for identifying and grouping items. Strings are

immutable - a new string is always created after any string operations. The original remains unchanged.
Strings are also ordered sequences - This means that each element or character in a string has a position or index. The first element has an index value of 0.

Creation

# using double quotes
name = "John Doe"
occupation = "Data analyst"

# using single quotes
address = 'somewhere beyond the earth'

# using triple quotes
"""
The triple quotes must be proud of themselves
They keep the new line.
"""

During the creation of nested strings (strings within strings), the inner quotes should be different from the outer quotes. For example


# correct

"John's Paper" # double outer, single inner
'John"s Paper' # single outer, double inner

# wrong
# Why this? Read about escaping strings or escape sequences. Google is your friend :)
'john's paper'
"John"s Paper"

A way to correct the error above is to use the string escape character backslash \.

'john\'s paper'
"John\"s Paper"

There are other escape characters such as the \n (newline character) and \t (tab character).

String Operations

Counting

len(address) # 26

Concatenation (adding strings)

#1 with the + operator
"Hello, my name is" + " " + name + "." + " I am a " + occupation + "!" # 'Hello my name is John Doe. I am a Data analyst!'

#2 placing string literals side by side

"Hello, my name is " "John" # 'Hello, my name is John'

#3. Using the string format method

"My name is {}, and I work as a {}. I live {}!".format(name, occupation, address)

# 'My name is John Doe, and I work as a Data analyst. I live somewhere beyond the earth!'

#4 using the * operator
name * 5 # 'John DoeJohn DoeJohn DoeJohn DoeJohn Doe'

Indexing and Slicing

# indexing
# The letter in the 3rd position is 'h'
name[2] # 'h'

#First position 
name[0] # 'J'

# last position
name[-1] # 'e'

# Slicing

# take from 1st to 4th (exclude the 4th element at index 3)
name[0:3] # 'Joh'

# slice from 4th to the end
name[3:] # 'n Doe'

String methods

String methods can be accessed via the_string.method() pattern. For example, use the replace() method to replace certain characters within a string.

# replace 'Johh' with 'Mary'
new_name = name.replace("John", "Mary")

new_name # Mary Doe

# The original name is not affected
print(name) # 'John Doe'

# or
>>> name
'John Doe'

# upper
name.upper() # 'JOHN DOE'

# lower
new_name.lower() # 'mary doe'

# strip: removes leading and trailing whitespace
"   This string has lots of space. So trim it.   ".strip() # 'This string has lots of space. So trim it.'

# We can chain multiple methods: convert to uppercase and then replace 'DOE' with an empty string
# Then remove the extra white space and slice starting from position 1 to the end
# Then convert the slice to a title-case string

name.upper().replace('DOE', '').strip()[1: ].title() # 'Ohn'

Types and Type Conversions

Types

In python, you can check the value type using the type() function. For example

type("a string") # <class 'str'>

# we created the name variable above
type(name) # <class 'str'>

type(age) # <class 'int'>

# remember that height is a number that contains decimal places
type(height) # <class 'float'>

Behind the scenes, integers, floats, and strings are classes. Therefore, they are represented as <class type>.

Type Conversions

To determine a variable or data type, use the type() function. For example

# create some variables
my_int = 45
my_float = 45.2
my_string = "12.234"


# Check the types

type(my_int)  # <class 'int'>
type(my_float) # <class 'float'>
type(my_string) # <class 'str'>

Use the int() function to convert a value to an integer. Use float() and str() to convert a type to float and string respectively.

Note the int() or float() function can fail if it receives a value it cannot convert (an invalid type)

The input function takes user input from the console and always returns a string. If you ask users for integers/floats such as age and amount, you should convert the values with the appropriate function, that is, int() or float().

now_a_float = float(my_string)

print("my_string: '{}'' is of type '{}'".format(my_string, type(my_string)))
print("now_a_float: '{}'' is of type '{}'".format(now_a_float, type(now_a_float)))

my_string: '12.234'' is of type '<class 'str'>'
my_string_is_now_a_float: '12.234'' is of type '<class 'float'>'

# convert integer to float
float(my_int) # 45.0

# convert float to integer - the decimal place is lost
int(my_float) # 45

# convert float to integer - the decimal place is lost
int(my_string_is_now_a_float) # 12

# convert integer to string
str(my_int) # '45'

# convert float to string
str(my_float) # '45.2'

Summary

You have learned about the fundamental types of python and their operations. You also saw how to convert from one type to another. Thanks for reading!

DEV Community: Mark Edosa

How to Download Multiple Files with Pure Python: A Beginner Guide

Introduction

Example 1: Downloading Text Files

Example 2: Downloading Multiple Binary Files (pdf, zip)

Introduction to Python Programming - Classes

Table of Contents

Introduction

Creating a Class

Creating Instance Objects

Class and Instance Variables

Using Dunder init for Class Invariance

Class and Instance Methods

Static Methods

Data and Implementation Hiding

Inheritance

Single Inheritance

The super() Function

Multiple Inheritance

Printing and Debugging An Object

Data Classes

Summary

Introduction to Python Programming - Errors and Exceptions

Table of Contents

Introduction

Common Exceptions

AssertionError

AttributeError

ImportError

ModuleNotFoundError

IndexError

KeyError

KeyboardInterrupt

NameError

IndentationError

TypeError

ValueError

ZeroDivisionError

File Errors

Handling Exceptions

Including an Else Clause

Adding a Finally Clause

Raising Exceptions

Adding More Information to Exceptions

User-Defined Exceptions

Summary

How to Extract Content From Word Documents Using BeautifulSoup

Introduction

A Word Document

Viewing the Content

Extracting The Content

Processing The Content

Example: Extracting Research Objectives from Research Proposals

Testing

Summary

How to Traverse (Walk) Multiple Directories Using Python

Introduction

Directory Structure

The Path Class from the PathLib Module

Walking Directories

Using A Loop and Recursion

Using A Generator Function

Summary

Introduction to Python Programming - Working With Files

Table of Contents

File Object APIs

Opening/Creating a File

Writing to Files

Reading From Files

Example: Copying a File

Working with JSON Files

Working with Structured Datasets

Working with Temporary Files

Working with File-like Objects

StringIO and BytesIO

Network Response

The Terminal (Standard Input and Output)

Getting File Attributes

Working with Directories

Example: Back Up Recently Modified Files