DEV Community: Siddhesh Agarwal

Formatter for Jupyter notebooks

Siddhesh Agarwal — Sat, 18 Feb 2023 06:14:56 +0000

Introduction

If you are into Data Science or Machine Learning, you have probably come across jupyter notebooks (.ipynb files). The problem I faced when using jupyter notebooks was that the black formatter didn't work on them. I had tried using the

$ black notebook.ipynb

command many times. This article is meant to help with code formatting in Python Notebooks.

nbQA

So, we will be using a python library called nbQA along with code formatters like Black and isort.

Installation

Install the library using:

$ pip install nbqa

Usage

You can use various formatters along with nqba and I will demonstrate how to use a few of them. before trying the formatters, make sure you have installed them already.

black

Format the notebook using black as shown below:

$ nbqa black notebook.ipynb
reformatted notebook.ipynb
All done! ✨ 🍰 ✨
1 files reformatted.

isort

Similarly, format the notebook using isort:

$ nbqa isort notebook.ipynb
Fixing notebook.ipynb

yapf

$ nbqa yapf --in-place notebook.ipynb

autopep8

$ nbqa autopep8 -i notebook.ipynb

mdformat

To format the markdown cells in your notebook, use:

$ nbqa mdformat notebook.ipynb --nbqa-md --nbqa-diff

doctest

To run tests for iPython notebooks using doctypes:

$ nbqa doctest notebook.ipynb

I hope you liked it. That's all for this time.

Packaging and Publishing on PyPI

Siddhesh Agarwal — Tue, 19 Apr 2022 03:17:07 +0000

This post is on "Packaging and Publishing a python library on PyPI".

There is a tutorial on Packaging Python Projects in the official PyPI website, but these docs are... outdated.

What is PyPI?

As the website says:

The Python Package Index (PyPI) is a repository of software for the Python programming language.

This means that PyPI is the official third-party software repository for Python. Anyone (with a PyPI account) can publish a Python Package for the use of the other developers. PyPI hosts these Python packages in the form of sdists (source distributions) or precompiled "wheels".

File structure

You have to maintain a certain file structure before packaging your project.

package_name/
├── src/
|    ├── __init__.py
|    ├── example.py
|    └── py.typed
├── LICENSE
├── pyproject.toml
├── README.md
├── setup.py (or) setup.cfg
└── tests/

So here is what each file does:

src/__init__.py: This file contains all the import statements. We add statements that look like from .example import <function names>
src/example.py: This file contains all the function and class definitions.
src/py.typed: An empty file. Read more here
LICENSE: This file contains the license for the code you are publishing.
- You can choose a license from choosealicense.com
pyproject.toml: This file tells build tools (like build and pip) what is required to build your package. For now, this will do:
```
[build-system]
requires = [
    "setuptools>=42",
    "wheel"
]
build-backend = "setuptools.build_meta"
```
README.md: This file acts as a guide. It gives developers a detailed description of your project.
setup.py (dynamic) or setup.cfg (static) are files that give "setuptools" information about your package (such as the name, version, author etc.). They also inform setuptools which code files to include.
- Writing setup.cfg/setup.py
tests/: This folder is a placeholder for test files.

Packaging Project

First, make sure that the pip installed on the device is of the latest version. To do that, run:

$ pip install --upgrade pip

After this, install build using:

$ pip install --upgrade build

Now, let's package the project. To package the project, run:

$ python -m build

This command will output lots of text and create a build/ folder and a dist/ folder.

The project has been packaged! Now we have to publish it on PyPI.

Publishing on PyPI

The first step to publishing on PyPI is creating an account there. Head over to PyPI and click on register (the text at the top right corner, that I highlighted in green):

You will get a form like this:

Fill out the form and remember the username and password. You will need it while publishing your project.

Now head back to the terminal and run:

$ pip install --upgrade twine

This command will upgrade twine to the latest version if twine is already present. If it isn't present, then it will install the latest version. Now, it is time to upload the package!

$ twine upload dist/*

You will be prompted to enter your PyPI username and password for authentication purposes. After entering them, you will see a URL in the last line of the output. the URL will be of the format https://pypi.org/project/<package_name>

Congratulations, the python package has been published on PyPI!!!

Now, let's pip install the library!

$ pip install <package_name>

print() in python

Siddhesh Agarwal — Sun, 16 Jan 2022 06:26:10 +0000

In this post, I will talk about about the print() statement and it's parameters.

So, I was experimenting with the help() function in python and tried the following command:

>>> help(print)
Help on built-in function print in module builtins:

print(...)
    print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)

    Prints the values to a stream, or to sys.stdout by default.
    Optional keyword arguments:
    file:  a file-like object (stream); defaults to the current sys.stdout.
    sep:   string inserted between values, default a space.
    end:   string appended after the last value, default a newline.
    flush: whether to forcibly flush the stream.

me, after seeing file and flush arguments in the output:

I knew about the sep and end parameters but not about the other 2 parameters. But what are these unpopular-parameters-that-online-tutorials-do-not-talk-about?

Today, I am going to share what they are. But before that, let me explain what the first 2 parameters are.

sep is the string that is inserted between 2 values. For example:

>>> print(1, 2, 3, sep='-')
1-2-3
>>> print(1, 2, 3) # The default value of "sep" is a space
1 2 3

end is the string that is appended after the last value. For example:

print(1, 2, 3, end='-')
print(4, 5, 6) # The default value of "end" is a newline
print("Hello")

Will print:

1 2 3-4 5 6
Hello

Now that I have explained what sep and end are, let's talk about the 3rd parameter. The 3rd parameter is file.

It is a file-like object (stream). The default value of file is sys.stdout (sys is a built-in module). If you don't pass this argument, it will default to stdout and the output will be printed to the standard output. It is the terminal where you execute your code. Standard output is full for stdout (Python is implemented in C that's the reason we get to see stdout in Python). If you specify a value for file, the output will be printed to that file. For example:

print(1, 2, 3, sep="\n", file=open("output.txt", "w+"))

the open("output.txt", "w+") will create a file called output.txt (if it doesn't exist already) and write the output to it. So our output.txt file will look like this:

1
2
3

This allows us to write to a file directly without having to convert to a string. It also allows us to use the sep and end parameters of the print() function without worrying about how to implement them.

Finally, let's talk about the flush parameter. The flush parameter is a boolean value. It "flushes" the internal buffer/stream. Let's see a small example for better understanding:

from time import sleep

# output is not flushed here
print("Hello", end=', ')
sleep(5)
print("world!")

would result in:

Hello, world!

The output looks perfect but there is the problem: the 5-second pause that was supposed to happen between the 2 words! It is not there. Now we run the same code but this time, we clear the output stream:

from time import sleep

# output is flushed this time
print("Hello", end=', ' flush=True)
sleep(5)
print("world!")

Now, when you run the program "Hello, " will be printed first and then after 5 seconds "world!" will be printed.

That's all for this time.

id() in python

Siddhesh Agarwal — Sat, 18 Dec 2021 04:28:59 +0000

In this post, I will try to improve your idea about memory in python using the in-built id() function. For those of you who don't know what id() is:

The id() function returns a unique ID of the object. All objects in python have a unique ID and no 2 different values correspond to the same ID.

So let us begin with a small example:

a = b = 500
print(id(a) == id(b))

Fun fact: In python, id(a) == id(b) is analogous to a is b.

The above code prints True because python creates the variable b with the value 500 and then creates a variable a pointing to the value of b. This implies that a and b are pointing towards the same memory location and hence the same ID.

Now let us raise the bar:

a = 500
b = 500
print(id(a) == id(b))

The above code prints False because:

Python creates a variable a pointing to the value 500 in the memory.
Then it creates another variable b pointing to another value 500 (yeah, both 500 are different).

Hence, both have different IDs because both point towards different memory locations.

I hope this isn't confusing because there is more to come. Guess the output for this:

a = 50
b = 50
print(id(a) == id(b))

Some of you may think "This is the previous question with different values. I know the answer is False" but not so fast.

For small integers (The CPython range is -5 to 256, both inclusive), then integer objects (<class 'int'>) are shared. This is done entirely to save space. The memory imprint of the console would be significantly larger if these objects weren’t sharing their memory.

So the correct answer is True

Okay, okay. Just one more to go. The last one:

a = 500
id1 = id(a)
a = 500
id2 = id(a)
print(id1 == id2)

Well, even though I am re-declaring the same variable with the same value, the answer is most likely to be False. I'll explain to you why. When you re-declare a variable in python, the interpreter works in the same way as a declaration. i.e. It entirely deletes the before existing value and creates a variable with the new value. So when we give a = 500, the second time the interpreter deletes the previously existing value of a and create a new memory location for 500 where a would point towards. Both these IDs are most likely to be different.

SO, the answer is False.

NOTE: If the above example had a number belonging to the inclusive range -5 to 256, the answer would have been True. this is because number belonging to the inclusive range have a fixed memory location.