DEV Community: chaitdwivedi

Pretty print JSON on command line

chaitdwivedi — Thu, 27 Jul 2023 02:17:28 +0000

Use python to pretty print JSON on command line using json std library module

python -m json.tool

Example:

echo '{"foo": "bar", "baz": [1, 2, 3]}' | python -m json.tool

Output:

{
    "foo": "bar",
    "baz": [
        1,
        2,
        3
    ]
}

Swap Base class methods in Python

chaitdwivedi — Wed, 26 Jul 2023 16:39:00 +0000

Problem Statement

Recently I came across this problem where I had to "swap" out a function in a Base class.

I had a base class Base and a bunch of derived classes: Derived0..N

class Base:
  def __init__(self, a, customs=None):
    self.a = a

  def base_func(self):
    print(f'Printing from base: {self.a=}')

class Derived0(Base):
  ...
#
#
#

class DerivedN(Base): 
  ...

I wanted some of the derived classes to use a custom function instead of the base_func, i.e. override the functionality.

def custom_func(self):
  print(f'Printing from fancy custom func: {self.a}')

Solution

Ideally, I should have simply overridden the Base class function, but I did not want to write the same implementation for each derived class.

Instead, I updated the base class to accept a way to override a member function, and let the derived classes pass along their implementation to swap out base functionality.

from functools import partial
class Base:
  def __init__(self, a, customs=None):
    self.a = a
    self.customs = customs or {}

    for func_name, func in self.customs.items():
      current_func = getattr(self, func_name, None)
      if callable(current_func): 
        setattr(self, func_name, partial(func, self))

  def base_func(self):
    print(f'Printing from base: {self.a=}')

Call changed to:

d = Derived0(5, {'base_func': custom_func})
d.base_func()

Prints:

Printing from fancy custom func: self.a=5

Use of partial

I used functools.partial to bind the function call with the object it will be called with, when called via derived class.

Use of self in custom function

self will be required in the custom function to allow access to other attributes of that object.

How to modify a string in Python

chaitdwivedi — Mon, 22 Nov 2021 22:07:32 +0000

You cannot!

Strings in Python are immutable (something that cannot be changed)

Why are Python strings immutable?

Read here

What can you do?

You can create a new modified string.

Examples

Convert all characters to upper case

original = "My String" 
new_string = original.upper() 
print(new_string)  # "MY STRING"

Change one character

Since you can't really change the string, the solution is to convert it a mutable type like list and modify that.

original = "My String" 
original_list = list(original)
original_list[0] = 'm' 
new_string = "".join(original_list)
print(new_string)  # 'my String'

You could also try slicing to speed up the process

Guiding Principles for Writing Documentation

chaitdwivedi — Tue, 09 Nov 2021 22:08:22 +0000

Your documentation should be:

👥 Inclusive and clear

Write docs in an inclusive and clear manner. Your language should be easy to understand and inviting. Try not to make assumptions about user's base level knowledge.

Easy to read documentation is an easy way to increase number of users and contributors to your project.

✅ Complete

Describe all aspects of your project. Try to document all of the common use cases.

💡 Support queries are usually holes in your documentation. If you find yourself answering to questions about your project, which could have been 'searched', you should add that to your documentation.

✨ Up-to-date

This is particularly hard. Outdated documentation might cause more harm than no documentation at all.

For Python projects, doctests does help to a certain extent in keeping documentation up-to-date, but really the onus of maintaining the docs lies on the developer.

💡 Adding documentation as a measure of completeness of coding tasks is very helpful.

💻 Easy to contribute

Add clear instructions on how to contribute to your project. This may include coding guidelines and standards, running test suites etc.

🔍 Easy to find

If working on "inner source" projects, make sure your project is searchable within your organization. Some companies use Confluence/Sharepoint to host docs, you may be able to keep a static landing page that may point to your project's homepage, which might be generated dynamically.

Essentials of README

chaitdwivedi — Wed, 03 Nov 2021 22:23:35 +0000

README.md is a minimum qualifying requirement to call a project well documented. It is basically an introduction to your project.

A README file typically written in Markdown and contains:

📋 Summary

A short description of the problem the project solves.

🌟 Key Features

A list of features that make your project great at solving the given problem.

▶️ Getting Started

This section is basically a usage guide. Include installation and usage instructions here.

This is also a good place to add some code examples as well.

💻 Contribution Notes

Discuss coding style and guidelines. Include steps to run test suite and introduce the contributor to any CI validation (if already setup)

❗ Support

List ways in which your end users can seek help. Email, Slack and JIRA are some good ways of allowing your users and developers to file bugs and ask for help.

💡 Focus on building a community of expert users and developers, so that you don't have to support your project all the time.

©️ License

Read about licenses here.

I found a collection of awesome-readmes which you might enjoy as well!

Why write documentation for your projects

chaitdwivedi — Tue, 02 Nov 2021 22:06:55 +0000

💰 Reduce future development effort

Lack of proper documentation is one of the main reasons why developers rewrite a project/tool. Instead of putting in the effort to understand the code is old now, they just prefer to write new code from scratch.

Documentation helps future you and other developers contributing/building upon your project.

📢 Marketing your project

You want people to use your project. For that to happen, people need to first understand:

Why your project is useful
What problems does it solve

Documentation for your project helps you address these questions.

⭐ Building your personal brand

Within an organization or in the open source community, good documentation can make your work look very professional and outstanding. Users start to associate your name with good documentation if you keep up the practice.

👥 Collaboration

One of the cornerstones of improving code base in open source community or even "inner source" (for proprietary software) is collaboration. Easiest way to invite others to collaborate is by allowing them to make "less intimidating" doc changes.

Contributions from community are aligned with original intentions of the author if the code is well documented.

⬆️ Improve Code quality

Thinking about documentation along with writing code enables you to make good design decisions. If you are unable to define what a module or function does in simple language, you probably need to rethink about the logic.

Rename multiple files with vim and sed

chaitdwivedi — Fri, 28 May 2021 22:31:56 +0000

There are multiple ways to rename a bunch of files in a programmatic manner in Unix-based systems. Here, I will talk about 2 possible ways:

Using vim (a little long winded - but good to build understanding)
Shell one liner - using sed and xargs

Suppose you have a directory with some .html files that you want to rename to .htm and add a suffix (generated) to the base name.

You want to do this:

$ mv file.html file-generated.htm

Using vim

Create a list of files

Use ls and grep to create a list files to update:

$ ls | grep html > exec_me 
$ cat exec_me

genindex.html
index.html
py-modindex.html
search.html

Search and replace to generate commands to execute

The idea is to generate a list of commands that can be executed from the command line to get the desired result.

Perform search and replace in vim's command mode:

:%s/\(.*\)\.html/mv \1\.html \1-generated.htm/g

\1 allows access to data captured in group - \(.*\). Read about vim modes in vim modes explained

Which should change your file list text to:

mv genindex.html genindex-generated.htm
mv index.html index-generated.htm
mv py-modindex.html py-modindex-generated.htm
mv search.html search-generated.htm

Save, quit and execute your file!

chmod 700 exec_me
./exec_me

Using sed and xargs

This technique is very similar to the vim style, except we don't create a new file - but create and execute commands on-the-fly using xargs

ls | grep html | sed 's/\(.*\)\.html/\1\.html \1-generated\.htm/g' | xargs -L1 mv

Conclusion

Scripting in shell is awesome and powerful, but also dangerous. It is like having a shotgun with no one stopping you from shooting yourself in the foot. Have fun with it but don't go deleting your entire code base!

Documenting your way to better tests in Python

chaitdwivedi — Fri, 14 May 2021 04:42:28 +0000

Writing and Reading Code

Writing code is hard, reading code is harder

I participate in a lot of code reviews and one thing I've realized is - developers spend most of their time coding a solution, not enough time explaining it.

There are two ways you could explain how your code works:

Write simple code that is self-explanatory
Write "good" documentation

Value of Good Documentation

In this post, I am going to make a case for how writing good documentation can improve the quality of your code and should be done along with writing self-explanatory code.

Let me illustrate with an example.

I've defined my_function below, that computes some "arbitrary" logic:

def my_function(input_string, input_list=None):
    head = "Start:"
    tail = "" if not input_list else " ".join(input_list)
    output = f"{input_string} {tail}"
    if not input_string.startswith(head):
        output = f"{head} {input_string} {tail}"

    return output.strip()

In a real-world problem, this piece of code might be much more complex.

As a reader, you could read this code and understand what it is doing. However, I can make the reader's job easier by using docstring.

Docstrings: Make code easy to read

Docstring holds immense value when developing code in a large organization and/or in a collaborative community.

It lets any developer understand what's happening in the function/module without them having to read through the entire codebase.

It can also be used with tools like sphinxdoc to generate beautiful documentation for your project.

def my_function(input_string, input_list=None):
    """Sanitize input string and append extra text if required

    The function checks if input_string starts with 'Start:' 
    if not, it will add the string to the input_string 

    It also converts input_list to a string using join 
    and appends to the input_string
    """
    head = "Start:"
    tail = "" if not input_list else " ".join(input_list)
    output = f"{input_string} {tail}"
    if not input_string.startswith(head):
        output = f"{head} {input_string} {tail}"

    return output.strip()

Docstring in the above code explains what the code is doing, but it still feels like a repetition of what is written in the code block.

How do we improve this?
Using doctest!

Doctest: Read, Test and Document better

doctest allows you to not only test interactive Python examples but also makes sure your documentation is up to date.

Let us take a look at improved docstring for the same function using doctest

def my_function(input_string, input_list=None):
    """Sanitize input string and append extra text if required

    >>> my_function('hi')
    'Start: hi'

    >>> my_function('Start: some string')
    'Start: some string'

    >>> my_function('hi', ['other', 'item'])
    'Start: hi other item'

    :param input_string: string to sanitize
    :type input_string: str
    :param input_list: extra items to append, defaults to None
    :type input_list: list, optional
    :return: sanitized string
    :rtype: str
    """
    head = "Start:"
    tail = "" if not input_list else " ".join(input_list)
    output = f"{input_string} {tail}"
    if not input_string.startswith(head):
        output = f"{head} {input_string} {tail}"

    return output.strip()

Here, I have defined some input-output examples for the given function which illustrate what the function is doing. For instance:

my_function('hi', ['other', 'item'])

should return:

'Start: hi other item'

The above documentation tells developers/readers the following:

What the function does
Parameters and their types
Return value and its type
Expected behavior - describes input/output examples

Conclusion

Generating documentation

I used sphinxdoc to generate documentation for the above function:

Practicing TDD

Writing documentation in Python also allows me to follow Test Driven Development, where I first define the behavior in docstring then write the code.

doctest can be run by any testing framework like unittest or pytest

$ pytest --doctest-modules -vv top.py
=================== test session starts ======================
platform darwin -- Python 3.8.2, pytest-6.2.4 
-- /projects/virtualenvs/dev-to/bin/python3
cachedir: .pytest_cache
rootdir: /Users/chaitanyadwivedi/projects/dev-to
collected 1 item                                                                                                                                           

top.py::top.my_function PASSED                          [100%]

==================== 1 passed in 0.05s =======================

How to fix dictionary access bugs in Python

chaitdwivedi — Fri, 09 Apr 2021 05:50:21 +0000

In this post I am going to discuss how accessing a value in a dict can easily introduce a bug in your code and how to fix it.

Access

One of the most common scenarios with dictionaries is accessing a value.

value = my_dict[key]

Problem: If key doesn't exist in my_dict, the code will raise a KeyError

Solution: To avoid this scenario you have three options:

Check if key exists before accessing


    if key in my_dict:
       value = my_dict[key]

Wrap the access in try-except block


    try:
        value = my_dict[key]
    except KeyError:
        ...

Use Python's get function to avoid checking for key in dict
```
    value = my_dict.get(key)
```

Since, get option (3) looks the neatest, I tend to use that the most.

Now, suppose for a function I am writing, I am accessing a value from dict but I only want to proceed if the given key exists in the dict.

I would write something like this:

value = my_dict.get(key)
if not value:
    raise MyError('Found bad key')

# continue with code

At first glance, the code looks fine, but I have introduced a bug in it.

Let me illustrate with an example:

# Assume my_dict as:
my_dict = {
    'key1': 'value1',
    'key3': [],
}

Key	Behavior	Expected?
key1	Regular code execution	Yes
key2	raises `MyError`	Yes
key3	raises `MyError`	No

key2 raises the exception since get function will return None by default for non-existing keys.
key3 exists in the dict however my code will still throw an error because if not [] is also True, leading to unintentional behavior.

Quick Bad Fix

Reading points about key2 and key3 from above, modifying the validation condition seems like the most obvious fix:

value = my_dict.get(key)
if value is None:
    raise MyError('Found bad key')

This, however, only fixes one symptom/effect of the bug that is when the value is an empty data type like '' or []. If the value in my_dict is None the above code will still raise the error.

Fix

Before writing a fix for your code, it is usually a good idea to think about the root cause of the bug, rather than the symptoms that are showing up. In our case, it was not the if condition that was causing the bug - it was the get function.

I only want to proceed if the given key exists in the dict

To encode the described behavior the correct code block would look something like this:

if key in my_dict:
    value = my_dict[key]
    # rest of code
else:
    raise MyError('Found bad key')

Learning

Capturing intent is critical in writing robust code.
First step to fixing bugs should be to list assumptions and work from there.