DEV Community: Hai Vu

A Better History Command

Hai Vu — Tue, 10 Oct 2023 18:08:07 +0000

Problem

Many people using the Linux terminal repeatedly search the command line history with such command:

history | grep cd

While this command is fine, I would like to shorten the command.

Design

I would like to create a command h which will

List the history if there is no argument
Search the history for the arguments when they are supplied

Implementation

After a few trials and error, I finally came up with a shell function, which I add to my ~/.bashrc and ~/.zshrc files:

function h() {
    history | grep -E --color=auto "${@:-}"
}

Usage

When invoked without any arguments, the command h behaves just like the history command.

When invoked with one or more arguments, those arguments will be passed on to the grep command. Examples:

h            # shorthand for history
h ls         # Searches history for the ls command
h -i myfile  # Case-insensitive search for myfile

Twelve Days of Christmas

Hai Vu — Thu, 16 Dec 2021 17:20:12 +0000

Here is a fun bit: I wrote a little Python script to displays the lyric of Twelve Days of Christmas in 12 lines.

#!/usr/bin/env python3
gifts = [
    'a partridge in a pear tree', 'two turtle doves and', 'three French hens',
    'four calling birds', 'five gold rings', 'six geese a-laying',
    'seven swans a-swimming', 'eight maids a-milking', 'nine ladies dancing',
    'ten lords a-leaping', 'eleven pipers piping', 'twelve drummers drumming'
    ]
days = """first second third fourth fifth sixth seventh eighth
    ninth tenth eleventh twelfth""".split()
for limit, day in enumerate(days, 1):
    gift = '\n  '.join(reversed(gifts[:limit]))
    print(f'On the {day} day of Christmas, my true love gave to me\n  {gift}\n')

How to Parse bash Variables

Hai Vu — Sat, 09 Oct 2021 07:08:24 +0000

The Problem

I sometimes need to parse a shell script to pick out variables declarations. Consider the following bash script, myscript.sh:

echo hello
server="server1.example.com"
port=8899
echo done

The Solution

The first step is read the contents of the script:

with open("myscript.sh") as stream:
    contents = stream.read().strip()
print(contents)

echo hello
server="server1.example.com"
port=8899
echo done

Next, we would pick out those lines which contains the variable declaration using regular expression:

import re

var_declarations = re.findall(r"^[a-zA-Z0-9_]+=.*$", contents, flags=re.MULTILINE)
print(var_declarations)

['server="server1.example.com"', 'port=8899']

Finally, we use the csv library to parse these declarations and return a dictionary:

import csv
reader = csv.reader(var_declarations, delimiter="=")
vars = dict(reader)
print(vars)

{'server': 'server1.example.com', 'port': '8899'}

Putting it all together:

import re
import csv

def parse_bash_vars(path: str) -> dict:
    """
    Parses a bash script and returns a dictionary representing the
    variables declared in that script.

    :param path: The path to the bash script
    :return: Variables as a dictionary
    """
    with open(path) as stream:
        contents = stream.read().strip()

    var_declarations = re.findall(r"^[a-zA-Z0-9_]+=.*$", contents, flags=re.MULTILINE)
    reader = csv.reader(var_declarations, delimiter="=")
    bash_vars = dict(reader)
    return bash_vars

Test it out:

parse_bash_vars("myscript.sh")

{'server': 'server1.example.com', 'port': '8899'}

Conclusion

When mentioning the csv library, most thinks of it as a way to parse comma-separated-values files, but it has other uses such as parsing the bash variables or parsing /etc/os-release file.

How to Write a Class Object to CSV

Hai Vu — Wed, 06 Oct 2021 16:32:54 +0000

The Problem

Given a class such as

class Student:
    def __init__(self, student_id, first_name, last_name):
        self.student_id = student_id
        self.first_name = first_name
        self.last_name = last_name

    def __repr__(self):
        return f"Student({self.student_id!r}, {self.first_name!r}, {self.last_name!r})"

    # Other methods...

Given a list of these objects:

my_students = [
    Student(1048, "John", "Adams"),
    Student(2517, "Karen", "Goodman"),
    Student(3131, "Anna", "Karenina"),
]

Our goal is to write them into a CSV file with 3 columns: ID, first, last name. What is a Pythonic way to do that?

The Solution

At first, we might want to create a function which converts a Student object into a tuple and use that:

def student_to_tuple(student):
    return (student.student_id, student.first_name, student.last_name)

with open("students.csv", "w") as stream:
    writer = csv.writer(stream)
    for student in my_students:
        row = student_to_tuple(student)
        writer.writerow(row)

This method works, but if we can modify the Student class, we can do better. By creating an Student.__iter__ method, we can work directly with csv.writer:

class Student:
    def __init__(self, student_id, first_name, last_name):
        self.student_id = student_id
        self.first_name = first_name
        self.last_name = last_name

    def __repr__(self):
        return f"Student({self.student_id!r}, {self.first_name!r}, {self.last_name!r})"

    def __iter__(self):
        return iter([self.student_id, self.first_name, self.last_name])

my_students = [
    Student(1048, "John", "Adams"),
    Student(2517, "Karen", "Goodman"),
    Student(3131, "Anna", "Karenina"),
]

with open("students.csv", "w") as stream:
    writer = csv.writer(stream)
    writer.writerows(my_students)

A couple of notes

the __iter__ performs almost the same task as student_to_tuple, but it returns an iterable instead of a tuple.
The csv.writer will iterate through this Student object to get the cells in a row, thus trigger the __iter__ method
We can now use the .writerows method to write all the objects in one step.

Reading Them Back

Now that we successfully wrote the list of students to a file, how do we read them back?

with open("students.csv") as stream:
    reader = csv.reader(stream)
    for row in reader:
        student = Student(*row)
        print(student)

Student('1048', 'John', 'Adams')

Student('2517', 'Karen', 'Goodman')

Student('3131', 'Anna', 'Karenina')

Conclusion

By adding a simple Student.__iter__ method, we allow a csv.writer object to work directly with Student objects.

How to Parse CSV a Text Variable

Hai Vu — Wed, 06 Oct 2021 14:43:13 +0000

The Problem

Most of us know how to parse a CSV file, but what about other sources such as a text block. For example, given the following block of text:

text = """
uid,alias,shell
501,karen,bash
502,john,tcsh
"""

How do we parse this text. Should we write it to a file first, then read?

The Solution

According to the documentation for the csv library, a CSV reader can handle a file, or anything that supports the iterator protocol, which includes list, or in-memory file objects. Let us explore the first solution, which splits the text into lines.

import csv

text = """
uid,alias,shell
501,karen,bash
502,john,tcsh
"""
reader = csv.reader(text.strip().splitlines())
for row in reader:
    print(row)

['uid', 'alias', 'shell']
['501', 'karen', 'bash']
['502', 'john', 'tcsh']

That was easy. Note that we called .strip() to remove whitespaces surrounding the text block before splitting it into individual lines. Next, we feed the lines to csv.reader and let it does its work.

Alternatively, we can use io.StringIO to turn the text into an in-memory file:

import csv
import io

text = """
uid,alias,shell
501,karen,bash
502,john,tcsh
"""
in_memory_file = io.StringIO(text.strip())
reader = csv.reader(in_memory_file)
for row in reader:
    print(row)

['uid', 'alias', 'shell']
['501', 'karen', 'bash']
['502', 'john', 'tcsh']

Of course, these techniques also work with other kind of reader: csv.DictReader:

import csv
import io

text = """
uid,alias,shell
501,karen,bash
502,john,tcsh
"""
in_memory_file = io.StringIO(text.strip())
reader = csv.DictReader(in_memory_file)
for row in reader:
    print(row)

OrderedDict([('uid', '501'), ('alias', 'karen'), ('shell', 'bash')])

OrderedDict([('uid', '502'), ('alias', 'john'), ('shell', 'tcsh')])

Conclusion

Parsing a block of CSV text is not that hard, you do not need to write it to an external file and that is the beauty of the csv library: it can work with a number of input sources, not just file.

How to Parse /etc/os-release

Hai Vu — Thu, 30 Sep 2021 14:12:42 +0000

The Problem

As part of my daily tasks, I need to parse the /etc/os-release file for information using Python. I have seen various attempts from other people, most of which are too complicated, too cumbersome, so I came up with my own.

Below is a sample contents.

NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

The Solution

The easiest way to parse this file is to you the PyYAML library. However, in some projects, my team does not have access to third-party libraries and have to stay within the Python 3.7's standard libraries.

After several attempts, I came up with a fairly easy solution: Use the csv library. Below is a code snippet which parses /etc/os-release and returns a dictionary.

import csv
import pathlib
from pprint import pprint

path = pathlib.Path("/etc/os-release")
with open(path) as stream:
    reader = csv.reader(stream, delimiter="=")
    os_release = dict(reader)

pprint(os_release)

Here is the output:

{'BUG_REPORT_URL': 'https://bugs.launchpad.net/ubuntu/',
 'HOME_URL': 'https://www.ubuntu.com/',
 'ID': 'ubuntu',
 'ID_LIKE': 'debian',
 'NAME': 'Ubuntu',
 'PRETTY_NAME': 'Ubuntu 20.04.3 LTS',
 'PRIVACY_POLICY_URL': 'https://www.ubuntu.com/legal/terms-and-policies/privacy-policy',
 'SUPPORT_URL': 'https://help.ubuntu.com/',
 'UBUNTU_CODENAME': 'focal',
 'VERSION': '20.04.3 LTS (Focal Fossa)',
 'VERSION_CODENAME': 'focal',
 'VERSION_ID': '20.04'}

How it Works

The file contains lines of key=value, so I use the csv library to read it and specify the equal sign as fields delimiters.
The reader will yield a list of (key, value) tuples
The dict built-in turns that list of tuples into a dictionary

Conclusion

After several attempts, I like this solution the best because it leaves the parsing to a standard library, which I assume well tested.