DEV Community: David Dobrinskiy

nbdev formating and linting

David Dobrinskiy — Fri, 06 Jan 2023 21:04:35 +0000

Introduction
nbqa
black
isort and mypy
Automating your flows with Makefile
pre-commit
- install and configure pre-commit
- create .pre-commit-config.yaml
reproducible
environments with Pipenv/Poetry/virtualenv
Concusion

Introduction

nbdev is a great tool.

However some of you may not be comfortable with the default fastai
codestyle, or maybe your team is
already using a different one.

This post will show you how to integrate your nbdev workflows with
industry standard tools like:

black , The uncompromising code formatter
isort , A Python utility / library to sort imports.
mypy , Optional static typing for Python
pre-commit , A framework for managing and maintaining multi-language pre-commit hooks.

Note

An astute reader may notice that one can enable
black_formatting = True in settings.ini

However,

this will only format the code in the module, not the code in the
notebook.

this does not allow you to use other tools like isort or mypy

nbqa

use nbqa to format and verify
code in your notebooks

nbqa is a tool that allows you to run any tool (formatter/linter) on
your notebooks, without having to worry about the details.

black

Black is a python code formatter that is becoming the de-facto standard

pip install nbqa

nbqa black nbs/

isort and mypy

isort will sort your imports, and mypy will check your type annotations

pip install isort mypy

nbqa isort nbs/
nbqa mypy nbs/

Automating your flows with Makefile

But you are already using nbdev command-line arguments like
nbdev_export and nbdev_test, who wants to remember more commands?

Make is a great tool for
automating your workflows, usually comes pre-installed on most unix
systems.

Create a Makefile in the root of your project and add the following
lines:

Makefile

# Makefile
format:
    nbqa black nbs/
    nbqa isort nbs/
    nbqa mypy nbs/

Now you can run make format to format your notebooks.

Let’s add more commands to our Makefile:

Makefile

format:
    nbqa black nbs/
    nbqa isort nbs/
    nbqa mypy nbs/

export:
    nbdev_export

mypy:
    nbqa mypy nbs/ --ignore-missing-imports --check-untyped-defs

test:
    nbdev_test --n_workers 4

prepare:
    # Export, test, and clean notebooks, and render README if needed
    nbdev_prepare

docs:
    nbdev_docs

And now the cherry on top: Makefile supports dependecies, so we can
chain commands together.

add this to your Makefile:

Makefile

all: format prepare mypy

Running make all will format your notebooks, export them, run mypy,
run tests and update README.md

pre-commit

Want to automate your formatting even more?

Force your team to format their notebooks before committing?

Use pre-commit

install and configure pre-commit

pip install pre-commit # install 

pre-commit install # enable for this repo

create .pre-commit-config.yaml

.pre-commit-config.yaml

repos:
  - repo: https://github.com/fastai/nbdev
    rev: 2.2.10
    hooks:
      - id: nbdev_clean
      - id: nbdev_export

Add nbqa hooks

.pre-commit-config.yaml

  - repo: https://github.com/nbQA-dev/nbQA
    rev: 1.5.3
    hooks:
      - id: nbqa-mypy
        args: ["--ignore-missing-imports", "--check-untyped-defs"]
        additional_dependencies: ["mypy", "types-toml", "types-requests"]
      - id: nbqa-black
      - id: nbqa-isort

control your hooks behaviour

.pre-commit-config.yaml

  - repo: https://github.com/nbQA-dev/nbQA
    rev: 1.5.3
    hooks:
      - id: nbqa-mypy
        args: ["--ignore-missing-imports", "--check-untyped-defs"] # pass args to mypy
        exclude: "directory_to_exclude/.*" # eclude any directory/files from formatting
        additional_dependencies: ["mypy", "types-toml", "types-requests"] # install additional dependencies
      - id: nbqa-black
        exclude: "directory_to_exclude/.*" # eclude any directory/files from formatting
      - id: nbqa-isort
        exclude: "directory_to_exclude/.*" # eclude any directory/files from formatting

reproducible environments with Pipenv/Poetry/virtualenv

Sometimes you may run into reproducibility issues, where your notebook
works on your machine, but not on your team’s.

The best way to avoid it is to use a dependency manager, such as:

But how do you integrate it with nbdev, since it uses settings.ini to
manage dependencies?

In this blog post I show an example of
how to use Pipenv with nbdev.

Concusion

I hope you find this post useful.

These changes may make nbdev more friendly to a team environment.

Please don’t hesitate to post any questions or comments below.

Reproducible environment in nbdev with Pipenv

David Dobrinskiy — Fri, 06 Jan 2023 21:04:15 +0000

Introduction
Some background on Pipenv
Creating a Pipenv environment
Syncing your Pipfile to setup.py
pre-commit hook
Extra reading:

Introduction

If you’re reading this, you’re probably familiar with
nbdev, a system for creating Python libraries
from Jupyter notebooks.

The only trouble I have with it is keeping track of dependencies,
especially when you have multiple projects or expect other people to use
your code.

In this post, I’ll show you how to set up a new nbdev project using
Pipenv, a tool for managing Python
dependencies. This will allow you to easily create a new project,
install dependencies, and run your code without having to worry about
setting up a virtual environment.

Some background on Pipenv

If you’re not familiar with Pipenv, it’s a tool for managing Python
dependencies. It’s similar to
virtualenv,
conda or
poetry, but it’s a bit more user-friendly.

Setting up a new project with Pipenv is easy. You just need to run
pipenv install in the project directory. This will create a new
virtual environment and install all the dependencies listed in the
Pipfile.

If you want to add a new dependency, you can run
pipenv install <package_name>. This will install the package and add
it to the Pipfile, keeping track of the version you installed.

If you want to install a specific version of a package, you can run
pipenv install <package_name>==<version>. This will install the
package and add it to the Pipfile with the specified version.

Creating a Pipenv environment

install pipenv:

pip install pipenv

install some generic dependencies for our project (optional):

pipenv install "requests==2.28.1" "pandas==1.5.2"

Now install the required dependencies for our guide:

pipenv install pipenv-setup "vistir==0.6.1" nbdev --dev

There is now a Pipfile and Pipfile.lock in the root of our project.
Let’s add the following to the Pipfile:

Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "==2.28.1"
pandas = "==1.5.2"

[dev-packages]
nbdev = "*"
pipenv-setup = "*"
vistir = "==0.6.1"

[requires]
python_version = "3.10"

Activate your new environment by running:

pipenv shell

Syncing your `Pipfile` to `setup.py`

Lucky for us, this has already been solved in
pipenv-setup package.

First, install it by running

pipenv install pipenv-setup "vistir==0.6.1" --dev

Now let’s sync our setup.py with Pipfile:

$ pipenv-setup sync --pipfile --dev

No setup() call found in setup.py
can not perform sync

Huh, an error. What’s going on here? Let’s check the setup.py file
that was generated by nbdev:

setup.py

setuptools.setup(
    ...
    install_requires = requirements,
    extras_require={ 'dev': dev_requirements },
    dependency_links = cfg.get('dep_links','').split(),
    ...
    )

The issue is that pipenv-setup expects us to call setup(...), not
setuptools.setup(...).

That is easy enough to fix by changing our function call:

setup.py

from setuptools import setup
setup(
    ...
    install_requires = requirements,
    extras_require={ 'dev': dev_requirements },
    dependency_links = cfg.get('dep_links','').split(),
    ...
    )

Running the command, we get another error:

$ pipenv-setup sync --pipfile --dev

Error parsing setup.py: install_requires is not a list
can not perform sync

Let’s make our setup.py compatible with pipenv-setup by changing
several arguments, namely:

install_requires should be a list
extras_require should be a dictionary
dependency_links should be a list

setup.py

from setuptools import setup
setup(
    ...
    install_requires=[],
    extras_require={},
    dependency_links=[],
    ...
    )

Running again:

$ pipenv-setup sync --pipfile --dev

reformatted setup.py
setup.py was successfully updated
5 default packages from Pipfile synced to setup.py
13 dev packages from Pipfile synced to setup.py

🎉 SUCCESS! 🎉

Our setup.py is now in sync with our Pipfile

now every time someone installs our package, they will get the same
dependencies as we do.

Installation with only the default dependencies:

pip install -e .

OR

pip install your_package_name # if you've already published your package

Installation with all the dependencies:

pip install -e '.[dev]'

updated setup.py example:

setup.py

    install_requires=["requests==2.28", "pandas", "bleak", "rich", "aranet4"],
    extras_require={
        "dev": [
            "black[jupyter]==22.12.0",
            "blacken-docs==1.12.1",
            "isort==5.10.1",
            "jupyter==1.*",
            "mypy",
            "nbdev==2.3.9",
            "nbqa==1.5.3",
            "pre-commit==2.20",
            "types-requests",
            "types-toml",
            "ipykernel",
            "pipenv-setup",
            "vistir==0.6.1",
        ],
    },
    dependency_links=[],

pre-commit hook

Odds are you’ll forget to run pipenv-setup sync every time you add a
new dependency.

To fix this, we can add a pre-commit hook to our project. This will run
pipenv-setup sync every time we commit changes to our Pipfile.

First, install and activate pre-commit in your repo:

pipenv install pre-commit --dev

pre-commit install

Then, add the following to your .pre-commit-config.yaml file:

.pre-commit-config.yaml

repos:
  - repo: https://github.com/Madoshakalaka/pipenv-setup
    rev: v3.2.0
    hooks:
      - id: pipenv-setup
        args: [--dev, --pipfile]
        additional_dependencies: ["vistir==0.6.1"]

Now on every commit, pipenv-setup will check that your Pipfile and
setup.py are in sync.

If you forget to run pipenv-setup sync --pipfile --dev, you’ll get an
error and won’t be able to commit until you fix it.

Extra reading:

New library to develop streamlit apps in jupyter

David Dobrinskiy — Thu, 05 Jan 2023 20:43:22 +0000

Hey there, I created a library streamlit-jupyter to allow for seamless creation of streamlit apps in jupyter notebooks.

A nifty little alternative to voila, one might say.

The way this works is it detects your environment, and if the script is run from a jupyter notebook, then it wraps existing streamlit methods to replace them with ipywidgets alternatives - that way you can code as you would in streamlit, and then displays your page as-you-code.

Then you export your notebook to a .py file, and run it via streamlit as usual (no change in behaviour there)

Install

pip install streamlit_jupyter

How to use it

Take a look at our example notebook

The main idea is for you experiment and develop in your notebook,
visually see all the pieces, and then convert the notebook to .py to
be run by streamlit

start by importing streamlit and patching it with streamlit-jupyter:


import streamlit as st

from streamlit_jupyter import StreamlitPatcher, tqdm

StreamlitPatcher().jupyter()  # register streamlit with jupyter-compatible wrappers

And now develop your notebook as usual, but with the ability to use
Streamlit widgets and components.

See how it works below, and check out the example (it also incudes a nifty alternative to nbconvert that strips out all the magics and other irrelevant stuff)

Demonstration

	Jupyter	Streamlit
Markdown and headings
Interactive data entry
Pick and choose
Dataframes, caching and progress bars
Plots

ps

This is a very early beta and my first try at open source, so please share your thoughts and reach out with any questions/critiques!

DEV Community: David Dobrinskiy

nbdev formating and linting

Introduction

nbqa

black

isort and mypy

Automating your flows with Makefile

pre-commit

install and configure pre-commit

create .pre-commit-config.yaml

reproducible environments with Pipenv/Poetry/virtualenv

Concusion

Reproducible environment in nbdev with Pipenv

Introduction

Some background on Pipenv

Creating a Pipenv environment

Syncing your Pipfile to setup.py

pre-commit hook

Extra reading:

New library to develop streamlit apps in jupyter

Install

How to use it

Demonstration

ps

Syncing your `Pipfile` to `setup.py`