DEV Community: Davit Tovmasyan

How to use the Django shell in Jupyter Notebook?

Davit Tovmasyan — Sat, 28 Mar 2020 11:41:11 +0000

Intro

Running Django, especially Django querysets in jupyter notebooks might be useful if you want to analyze/plot the data that is in your database. Also, sometimes it's hard to execute big querysets in CLI using Django's interactive shell.
In this article, we will see how to connect Django's shell with the Jupyter Notebook and get the advantage of executing querysets in notebook cells.

Django

Django is a web framework for perfectionists with deadlines. It has a shell command that starts the Python interactive interpreter.

Jupyter Notebook

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

Django Extensions

Django Extensions is a collection of custom extensions for the Django Framework. These include management commands, additional database fields, admin extensions and much more. With the help of this python package, we will connect Django's shell with the Jupyter Notebook application.

Setup

I assume that you have python installed on your machine and you are familiar with creating virtual environments.

Create an empty folder for the django project:

$ mkdir django_jupyter && cd django_jupyter

Create a virtual environment inside of the newly created folder:

$ virtualenv .venv -p python3
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /django_jupyter/.venv/bin/python3
Also creating executable in /django_jupyter/.venv/bin/python
Installing setuptools, pip, wheel...
done.

Activate the virtual environment:

$ source .venv/bin/activate

Install all necessary python packages:

(.venv) $ pip install django jupyter ipython django-extensions

Initialize an empty django project:

(.venv) $ django-admin startproject django_jupyter .

Migrate the database:

(.venv) $ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  ...
  Applying auth.0011_update_proxy_permissions... OK
  Applying sessions.0001_initial... OK

To make the use of django_extensions list it in INSTALLED_APPS in settings.py file:

INSTALLED_APPS = [
    ...
    'django_extensions',
]

Usage

Now we will start jupyter notebook application with the command below:

(.venv) $ python manage.py shell_plus --notebook

IMPORTANT! IPython added top-level async/await support, which is running the whole interpreter session inside of a default event loop and starting from Django 3.0 version running application from a thread where there is a running event loop, will cause a SynchronousOnlyOperation error. So, setting DJANGO_ALLOW_ASYNC_UNSAFE to true is mandatory in that case. More on this here.

(.venv) $ DJANGO_ALLOW_ASYNC_UNSAFE=true && python manage.py shell_plus --notebook

Now, when the jupyter notebook web application is on your screen, you can create a new notebook selecting Django Shell-plus as a kernel:

In the brand new notebook import the model you want and start writing and running querysets as you do in Django's shell.

Conclusion

This setup gives us lots of advantages. One of them is the absence of necessity to scroll through the shell history to reuse/change code snippets that we wrote because our code exists in handy Jupyter notebook cells in front of us.

More ☜

How to make your code fast and asynchronous with Python and Sanic

Davit Tovmasyan — Mon, 18 Nov 2019 09:18:38 +0000

Hello everybody. In this article, I'll talk about building simple asynchronous projects with the Sanic framework.

Introduction

Sanic is a very flask-like open-source Python web server and web framework with more than 10K stars that's written to go fast. It allows the usage of async/await syntax added in Python 3.5 (read more), which makes your code non-blocking and speedy.

Sanic has pretty good documentation and it's maintained by the community, for the community.

The goal of the project is to provide a simple way to get a highly performant HTTP server up and running that is easy to build, to expand, and ultimately to scale.

Requirements

Before we start, let's install some packages and make sure that we have everything ready for the development of this project.

Note: Source code is available in my github.com repository. For each step, there is a corresponding commit.

Prerequisites:

Python3.6+
pipenv (you can use any other package installer)
PostgreSQL (for database, can also be MySQL or SQLite)

Packages:

secure is a lightweight package that adds optional security headers and cookie attributes for Python web frameworks.
environs is a Python library for parsing environment variables. It allows you to store configuration separate from your code, as per The Twelve-Factor App methodology.
sanic-envconfig is an extension that helps you bring command line & environment variables into your Sanic config.
databases is a Python package that allows you to make queries using the powerful SQLAlchemy Core expression language and provides support for PostgreSQL, MySQL, and SQLite.

Let's create an empty directory and initialize an empty Pipfile there.

pipenv -- python python3.6

Install all necessary packages using pipenv commands below.

pipenv install sanic secure environs sanic-envconfig

For the database:

pipenv install databases[postgresql]

Choices are postgresql, mysql, sqlite.

Structure

Now let's create some files and folders where we will write our actual code.

├── .env
├── Pipfile
├── Pipfile.lock
├── setup.py
└── project
    ├── __init__.py
    ├── __main__.py
    ├── main.py
    ├── middlewares.py
    ├── routes.py
    ├── settings.py
    └── tables.py

We will use the setup.py file to make the project folder available as a package in our code.

from setuptools import setup

setup(
    name='project',
)

Installing…

pipenv install -e .

In the .env file, we'll store some global variables like the database connection URL.

__main__.py is created for making our project package executable from the command-line.

pipenv run python -m project

Initialization

Let's do our first call in __main__.py file.

from project.main import init

init()

This is the beginning of our application. Now we need to create the init function inside of main.py file.

from sanic import Sanic

app = Sanic(__name__)

def init():
    app.run(host='0.0.0.0', port=8000, debug=True)

Simply creating the app from the Sanic class we can run it specifying host, port and optional debug keyword argument.

Running…

pipenv run python -m project

If you open http://0.0.0.0:8000 on your browser you'll see

Error: Requested URL / not found

We haven't created any routes yet, so it's fine for now. We will add some routes below.

Settings

Now we can modify the environment and settings. We need to specofy some variables in the .env file, read them, and pass to Sanic app config.

.env file.

DEBUG=True
HOST=0.0.0.0
POST=8000

Configuration…

from sanic import Sanic

from environs import Env

from project.settings import Settings

app = Sanic(__name__)

def init():
    env = Env()
    env.read_env()

    app.config.from_object(Settings)

    app.run(
        host=app.config.HOST, 
        port=app.config.PORT, 
        debug=app.config.DEBUG,
        auto_reload=app.config.DEBUG,    
    )

settings.py file.

from sanic_envconfig import EnvConfig

class Settings(EnvConfig):
    DEBUG: bool = True
    HOST: str = '0.0.0.0'
    PORT: int = 8000

Please note that I've added an optional auto_reload argument which will activate or deactivate the Automatic Reloader.

Database

Now it's time to set up a database.

One little note about the databases package before we go ahead:

databases package uses asyncpg which is an asynchronous interface library for PostgreSQL. It's pretty fast. See results below.

We will use two of Sanic's listeners where we will specify database connect and disconnect operations. Here are the listeners that we are going to use:

after_server_start
after_server_stop

main.py file.

from sanic import Sanic

from databases import Database # <- this line

from environs import Env
from project.settings import Settings

app = Sanic(__name__)

def setup_database(): # <- this function
    app.db = Database(app.config.DB_URL)

    @app.listener('after_server_start')
    async def connect_to_db(*args, **kwargs):
        await app.db.connect()

    @app.listener('after_server_stop')
    async def disconnect_from_db(*args, **kwargs):
        await app.db.disconnect()

def init():
    env = Env()
    env.read_env()

    app.config.from_object(Settings)

    setup_database() # <- this line

    app.run(
        host=app.config.HOST, 
        port=app.config.PORT, 
        debug=app.config.DEBUG,
        auto_reload=app.config.DEBUG,    
    )

Once more thing. We need to specify DB_URL in project settings and environment.

.env file.

DEBUG=True
HOST=0.0.0.0
POST=8000
DB_URL=postgresql://postgres:postgres@localhost/postgres # <- this line

And in settings.py file.

from sanic_envconfig import EnvConfig

class Settings(EnvConfig):
    DEBUG: bool = True
    HOST: str = '0.0.0.0'
    PORT: int = 8000
    DB_URL: str = '' # <- this line

Make sure that DB_URL is correct and your database is running. Now you can access the database using app.db. See more detailed information in the next section.

Tables

Now we have a connection to our database and we can try to do some SQL queries.

Let's declare a table in tables.py file using SQLAlchemy.

import sqlalchemy


metadata = sqlalchemy.MetaData()

books = sqlalchemy.Table(
    'books',
    metadata,
    sqlalchemy.Column('id', sqlalchemy.Integer, primary_key=True),
    sqlalchemy.Column('title', sqlalchemy.String(length=100)),
    sqlalchemy.Column('author', sqlalchemy.String(length=60)),
)

Here I assume that you already have a migrated database with a books table in it.

For creating database migrations, I recommend that you use Alembic which is a lightweight and easy-to-use tool that you can use with the SQLAlchemy Database Toolkit for Python.

Now we can use any SQLAlchemy core queries. Check out some examples below.

# Executing many
query = books.insert()
values = [
    {"title": "No Highway", "author": "Nevil Shute"},
    {"title": "The Daffodil", "author": "SkyH. E. Bates"},
]
await app.db.execute_many(query, values)

# Fetching multiple rows
query = books.select()
rows = await app.db.fetch_all(query)

# Fetch single row
query = books.select()
row = await app.db.fetch_one(query)

Routes

Now we need to set up routes. Let's go to routes.py and add a new route for books.

from sanic.response import json
from project.tables import books

def setup_routes(app):
    @app.route("/books")
    async def book_list(request):
        query = books.select()
        rows = await request.app.db.fetch_all(query)
        return json({
            'books': [{row['title']: row['author']} for row in rows]
        })

Of course we need to call setup_routes in init to make it work.

from project.routes import setup_routes # <- this line

app = Sanic(__name__)

def init():
    ...
    app.config.from_object(Settings)
    setup_database()
    setup_routes(app) # <- this line
    ...

Requesting…

$ curl localhost:8000/books
{"books":[{"No Highway":"Nevil Shute"},{"The Daffodil":"SkyH. E. Bates"}]}

Middlewares

What about checking the response headers and seeing what we can add or fix there?

$ curl -I localhost:8000
Connection: keep-alive
Keep-Alive: 5
Content-Length: 32
Content-Type: text/plain; charset=utf-8

As you can see we need some security improvements. There are some missing headers such as X-XSS-Protection, Strict-Transport-Security… so let's take care of them using a combination of middlewares and secure packages.

middlewares.py file.

from secure import SecureHeaders

secure_headers = SecureHeaders()

def setup_middlewares(app):
    @app.middleware('response')
    async def set_secure_headers(request, response):
        secure_headers.sanic(response)

Setting up middlewares in main.py file.

from project.middlewares import setup_middlewares

app = Sanic(__name__)

def init():
    ...
    app.config.from_object(Settings)
    setup_database()
    setup_routes(app)
    setup_middlewares(app) # <- this line
    ...

The result is:

$ curl -I localhost:8000/books
Connection: keep-alive
Keep-Alive: 5
Strict-Transport-Security: max-age=63072000; includeSubdomains
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: no-referrer, strict-origin-when-cross-origin
Pragma: no-cache
Expires: 0
Cache-control: no-cache, no-store, must-revalidate, max-age=0
Content-Length: 32
Content-Type: text/plain; charset=utf-8

Hope this small tutorial helped you to get started with Sanic. There are still many unexplored features in the Sanic framework that you can find and check out in the documentation.

davitovmasyan/sanic-project

Thanks for reading. Go Fast with Sanic and good luck!!!