DEV Community: Chris

How to Setup Autocompletion using Hinterland in Jupyter Notebook

Chris — Sat, 04 Nov 2023 15:38:08 +0000

In this tutorial, I will show you how to setup extensions and enable Hinterland in Jupyter Notebook. There is probably a way to set it up using conda but I will use pip in this tutorial to set it up.

For this tutorial, I will assume following:

using Python 3.9 or higher
running macOS or linux (may require different command for Windows)

Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and narrative text. It is commonly used for data analysis, data visualization, machine learning, and scientific computing, but it can be employed for various other tasks as well. Jupyter Notebook provides an interactive and user-friendly environment for writing and running code, making it popular among data scientists, researchers, and educators.

Hinterland is an extension for Jupyter Notebook that provides code autocompletion suggestions in code cells as you type. It aims to enhance your coding experience by suggesting completions for variable names, functions, methods, and other code elements, which can save you time and reduce potential coding errors. It provides suggestions without you having to press tab key.

Create Virtual Environment

I like to use venv to set up my virtual environment. It goes like this:

$ python3 -m venv <<name of virtual environment>>

For example:

$ python3 -m venv venv

You can activate it by typing this into the terminal

$ source venv/bin/activate

Install Jupyter Notebook

There are different ways to install Jupyter Notebook on your computer. I personally use Anaconda Navigator, but you can install it using pip:

(venv)$ pip install notebook

You can start notebook by:

(venv)$ jupyter notebook

Use Virtual Environment as Kernel

In Jupyter Notebook, a "kernel" is used as a computational engine that executes the code contained within the notebook. Each notebook can be associated with a specific kernel. I will show you how to use our virtual environment as kernel.

First, make sure you have activated venv by source venv/bin/activate. Then we need to install ipykernel.

(venv)$ pip install ipykernel

You can install a kernel with the following command:

(venv)$ python -m ipykernel install --user --name venv

You should replace venv with your python environment.

Now, you should be able to see your kernel when you start up Jupyter Notebook:

Install Nbextensions

When you start up the Jupyter Notebook, you may see an environment like below:

Right now, you may not have the Nbextensions tab. We need that tab to enable extensions, so let's install it now.

(Make sure you are not running Jupyter Notebook.)

You need to install few more Python packages:

(venv)$ pip install jupyter_contrib_nbextensions
(venv)$ pip install jupyter_nbextensions_configurator

Once those packages are installed, run following commands:

(venv)$ jupyter contrib nbextension install --user
(venv)$ jupyter nbextensions_configurator enable --user

The first command copies the nbextensions’ javascript and css files into the jupyter server’s search directory, and edits some jupyter config files.

The second command enables the nbextension.

(You can find more information about these steps here.)

Now, when you start up the Notebook, you should be able to see the Nbextensions tab.

We are almost done. We just need to enable Hinterland now.

Enable Hinterland

Go to Nbextensions tab.

Make sure checkbox beside "disable configuration for nbextensions without ..." is unchecked.

In the filter, type 'hinterland'. Make sure checkbox beside the Hinterland is checked to enable it:

And that is it!

If you create a notebook (make sure to use your virtual environment as a kernel), and start typing things, you will see code suggestion showing up automatically without pressing tab key all the time! :)

How to Set Formatting and Linting on VSCode for Python using Black Formatter and Flake8 Extensions

Chris — Thu, 12 Oct 2023 23:47:38 +0000

In this tutorial, I will demonstrate how you might configure VSCode to use code formatter and linter for Python 3.

A code formatter is a tool or software that automatically and consistently formats source code in a programming language according to a predefined set of style rules or coding conventions.

A code linter (or simply "linter") is a tool that analyzes source code for potential errors, coding style violations, and other issues.

These two tools are optional, but can be an essential part of maintaining code quality and consistency in software development projects, especially in large or collaborative teams where adherence to a common coding style is crucial.

We will use Black as a Python code formatter, and Flake8 as a Python linter.

Install Extensions

Black and Flake8 can be installed using pip, and configured that way. But it seems the built-in support is going to be deprecated soon. They are now migrating to individual Python tool extension. (Check here for more information.)

On Visual Studio Code, go to View > Extensions to see list of extensions. Type "Black Formatter" to download extension from Microsoft. Do the same with "Flake8", also from Microsoft.

(You might want to install "isort" as well - also from Microsoft - if you want your import statements to sort when you save file.)

(If you want to disable these extensions, you can disable the extension per workspace in Visual Studio Code.)

Now we are ready to use these tools.

Create Settings File

Create a new folder in the project root folder, and name it .vscode. Inside newly created folder, create a json file and name it settings.json.

Configure Black Formatter in Settings File

The settings file is in json format, so you need to enclose everything in a curly bracket ({ }). Let's first set the default formatter to use Black; you can do that by typing this into the settings.json:



{
   "editor.defaultFormatter": "ms-python.black-formatter",
}

If you want VSCode to format on save, you need to write this:



{
   "editor.defaultFormatter": "ms-python.black-formatter",
   "editor.formatOnSave": true, // new
}

You can enforce a line length by passing an argument to Black Formatter:



{
   // ...
   "black-formatter.args": [
    "--line-length", "88",
   ],
}

(The line length is 88 by default, which seems to be the recommended length.)

If you want the settings to apply to specific language such as "Python", you can do that like this:



{
   // ...
   "[python]": {
    "editor.codeActionsOnSave": {
      "source.organizeImports": true
    },
   },
}

The above code organize import statements when you save Python files. You can add previous settings here instead so it only applies to Python file (don't forget the commas):



{
   // ...
   "[python]": {
    "editor.codeActionsOnSave": {
      "source.organizeImports": true
    },
    "editor.formatOnSave": true, // new
   },
}

Configure Flake8

When you install Flake8 and enable the extension, it is automatically used without any additional settings applied.

If you have a bug in the program or a typo, and you save a file, you can see that flake8 warns you with a message, such as:

Just like Black Formatter, you can pass Flake8 with an argument. For example:



{
   // ...
   "flake8.args": [
    "--max-line-length", "88",
    "--extend-ignore", "E203"
   ],
}

You can set max length by using --max-line-length.

--extend-ignore can be used to ignore certain errors depending on their codes.

Check these links for more information on the various codes:

And that is basically it! There are more to the settings, and read Black and Flake8 documentation to learn more, but I hope this tutorial gives you an idea how it can be done.

The finished settings might look like this:

Hopefully this tutorial was helpful. Leave me comments if you have any questions :)

How to Deploy Production Django 4 using Google Cloud Run, Cloud SQL, and Cloud Storage

Chris — Sun, 08 Oct 2023 16:35:54 +0000

In this tutorial, I will show you how to deploy Django 4 application on Google Cloud using Cloud Run.

We will cover the following:

Create and connect to Cloud SQL database.

Build Dockerfile image and push to Artifact Registry.

Deploy app on Cloud Run using image from Artifact Registry.

Use Secret Manager to handle Environment Variables securely.

Serve static files from Cloud Storage.

For this tutorial I will be using Django 4.2 and Python 3.10. If you are using different versions, things may be slightly different. Also, I will be using Polls app from official Django 4.2 tutorials for demonstration, but feel free to use your own.

This is not a tutorial on Django, Docker, command line, or GCP. I will assume you know the basics, but I will show you how to put them together! :)

Create Cloud SQL Database (PostgreSQL)

For the tutorial, we will use PostgreSQL database. Go to Cloud SQL, and create an instance.

From the options, choose PostgreSQL. Give it an instance ID, Password, and set Region.

Under "Customize your instance" > "Connections" make sure you select "Public IP" (we will use public ip for this tutorial)

Also Google your ip address, and add as an authorized network. (This is only needed for accessing database from local. You can use Cloud SQL Proxy as an alternative.)

Leave the rest as defaults, and create instance. It will take a minute or so. Once it completes, you will be taken to an Overview page. Take notice of the "Public IP address", and "Connection name". We will need them to connect to database from locally and Cloud Run, respectively.

Next, we need to create users. Go to "Users", and click "ADD USER ACCOUNT" and enter name and password:

Write down this name and password somewhere, these will be DB_USER and DB_PASSWORD, respectively.

Lastly, go to "Databases" and click "CREATE DATABASE". Give the database a name. (Will be our DB_NAME)

And that is it! Now we are ready to connect to Cloud SQL database!

Connect to Cloud SQL Database (Local)

In order to connect to PostgreSQL from Django, we need to install psycopg2-binary library. Type following in the terminal:



$ pip install psycopg2-binary==2.9.9

or
$ pip3 install psycopg2-binary==2.9.9 if you are not using virtual environment.

In settings.py configure to use our Cloud SQL database instead of sqlite3 database:



DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'HOST': env("DB_HOST"),
        'USER': env("DB_USER"),
        'PASSWORD': env("DB_PASSWORD"),
        'NAME': env("DB_NAME"),
    }
}

Make sure DB_HOST, DB_USER, DB_PASSWORD, and DB_NAME is set in the .env file (You can just hard code it, but using environment variables are recommended for production).

DB_HOST: use public ip address
DB_USER: user created ('Admin' for me)
DB_PASSWORD: password created for the user (Admin)
DB_NAME: database name ('django-demo-db' for me)

(I used environs. Check it out if you want to set your project to use environment variables.)

Before we start our application, we need to migrate and create our superuser. Type following in the terminal:



$ python manage.py migrate
$ python manage.py createsuperuser

(If you are not connecting, check if you added your ip address to the trusted network in the previous section.)

Now when we run our app locally, we should be connected to Cloud SQL database!!



$ python manage.py runserver

Now let's deploy our app on Cloud Run.

Push Docker Image to Artifact Registry

Before we go any farther, make sure you have Docker installed. And if you are using Docker Desktop, make sure you start it.

Create Dockerfile and .dockerignore in your project root folder (right beside manage.py).

This is what I used, but you can create your own (For CMD gunicorn, make sure you change mysite and use yours instead. For me mysite is the folder that contains settings.py and wsgi.py files. Yours would probably be different.):



# Dockerfile
FROM python:3.10.4-slim-bullseye

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Set the working directory in the container
WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    netcat postgresql-client mime-support \
    && rm -rf /var/lib/apt/lists/*

# Install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy the Django project code into the container
COPY . .

# Expose the port (Cloud Run uses the PORT environment variable)
ENV PORT 8080
EXPOSE 8080

# Run the Django development server
CMD exec gunicorn mysite.wsgi:application --bind :$PORT --workers 1 --threads 8

.dockerignore:



.env
venv/
db.sqlite3

For production, we are using gunicorn, so let's install that:



$ pip install gunicorn==21.2.0

We also need to create requirements.txt file:



$ pip freeze > requirements.txt

Make sure you update requirements.txt file every time you install a new pip library.

Ok, now let's create our Docker repository on Artifact Registry on GCP.

Give it a name and region:

Once you create your repo, click it from the list. You may need to initialize gcloud and configure Docker. Take a look at SETUP INSTRUCTIONS for more details.

Configuring Docker by typing following in terminal:



$ gcloud auth configure-docker <<your-region>>-docker.pkg.dev

Now that we have set up our repo, let's build our Docker image. Make sure you are in the same directory as your Dockerfile, and type following in terminal:



$ docker build -t <<image-name>> .

We also need to tag it before we push:



$ docker tag <<image-name>> <<region>>-docker.pkg.dev/<project-id>>/<<repo-name>>/<<image-name-you-want>>

Make sure you replace things with your own.

For example:
If I used following:
image-name: django-demo
region: us-central1
project-id: ch-project
repo-name: django-test-repo
image-name-yout-want: django-demo



$ docker tag django-demo us-centra1-docker.pkg.dev/ch-project/django-test-repo/django-demo

Finally we can push our tagged image to repo on Artifact Registry:



$ docker push <<region>>-docker.pkg.dev/<project-id>>/<<repo-name>>/<<image-name-you-want>>

In my case:



$ docker push us-centra1-docker.pkg.dev/ch-project/django-test-repo/django-demo

If you go to GCP Artifact Registry, you should see your image:

Deploy using Cloud Run

Finally, we are ready to deploy our application on Cloud Run.

We are going to use the image that we pushed to Artifact Registry. We also need a service account to use with Cloud Run. This service account will need a following IAM role as a minimum:

Cloud SQL Client

Now go to Cloud Run, and click CREATE SERVICE.

Under "Deploy one revision from an existing container image", select image from Artifact Registry. Select the latest.

Under Authentication, select "Allow unauthenticate invocations".

Select the SQL instance you created from the drop down menu:

Also make sure you include Environment Variables that are inside .env file (Make sure the name matches the ones you use in the settings.py).

The exception is the DB_HOST. We used public IP address when we connected locally, but if we want to connect from Cloud Run, we have to use different value.

DB_HOST: /cloudsql/<<instance-connection-name>>

For example:
/cloudsql/ch-project:us-central1:django-demo-instance

We can get the connection name from the Overview page of SQL instance we created. Don't forget /cloudsql/.

In the later section I will show you a better way to manage Environment Variables using Secret Manager. For now, you can leave the rest as default, and create the Cloud Run service instance.

It will take few seconds, but once it completes, you should see a url:

If you click it now, error message will show. That is because we have to include our url as Allowed Host in settings.py (Make sure you use your own url instead):



# settings.py
ALLOWED_HOSTS = ["127.0.0.1", "localhost", "django-demo-3parkuk6ra-uc.a.run.app"]

Make sure you don't include "https://" or trailing "/" at the end of url.

If you have forms, you will have to include this as well (you may have to uncomment it when you run locally):



# settings.py
CSRF_TRUSTED_ORIGINS = ["https://django-demo-3parkuk6ra-uc.a.run.app"]
SECURE_SSL_REDIRECT = True
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")

Now we need to build image, tag and push again (Make sure you save settings.py).



$ docker build -t <<image-name>> .
$ docker tag <<image-name>> <<region>>-docker.pkg.dev/<project-id>>/<<repo-name>>/<<image-name-you-want>>
$ docker push <<region>>-docker.pkg.dev/<project-id>>/<<repo-name>>/<<image-name-you-want>>

Now go back to Cloud Run, and select EDIT & DEPLOY NEW REVISION. Under Container image URL, select new image you just pushed.

Making sure "Serve this revision immediately" is selected, Deploy:

Congrats! If you check the url, your app should be working now!

Use Secret Manager (Better Way to Manage Environment Variables)

Alright, our app is up and running, but we can make it better by using Secret Manager to keep things more secure. You should create a new secret for each environment variable values that you want to keep secure, such as tokens and passwords. For this tutorial, I will create secrets for SECRET_KEY and DB_PASSWORD.

Go to Cloud Run instance and click "EDIT & DEPLOY NEW REVISION".

First, delete environment variables that you want to convert to secret (can't have duplicates).

Next, scroll down until you see "Secrets" and click "ADD A SECRET REFERENCE":

Click Secret drop down menu and select "CREATE NEW SECRET" to make "Create Secret" modal to appear.

Fill in name (it could be anything as long as unique within project), and under "Secret value", enter just the value of the environment variables:

You can leave the rest as default, and create secret. Then you need to change the "Reference method" to "Exposed as environment variable". Then you need to enter the environment variable name and choose the version. It should look like this:

You can choose the latest as a version, but choosing specific version is recommended. Click "Done" to create it.

Now we just need to repeat the same thing for the password. It should look something like this at the end:

And that is it, now just Deploy the change. If you go to the URL, the app should work exactly the same! Yay!

Setup Cloud Storage

We are almost at the end. The application should be running fine on Cloud Run. However, our application is serving the static files. As more and more people use our app, more instance resources are used up. We will solve this issue by serving static files from Cloud Storage.

First, let's start by creating Cloud Storage bucket. We will create a public bucket that is available for everyone.

Go to Cloud Storage and Create a bucket. Give it a unique name, and choose same region as Cloud Run instance. Make sure "Enforce public access prevention on this bucket" is unchecked, and choose "Fine-grained" as our Access control:

Leave the rest as default, and create a bucket. Inside Bucket details, go to "PERMISSIONS" tab. We need to Grant Access to "allUsers", and give them Storage Object Viewer role:

Save and press "ALLOW PUBLIC ACCESS" button. Now, we are ready to serve static files!

Serve Static Files on Cloud Storage

We need to install django-storages library, and update our requirements.txt file:



$ pip install django-storages[google]==1.14.1
$ pip freeze > requirements.txt

We also need to configure settings.py to use our storage bucket:

If you are using Django >=4.2, type:



# settings.py
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.gcloud.GoogleCloudStorage",
    },
    "staticfiles": {
        "BACKEND": "storages.backends.gcloud.GoogleCloudStorage",
    }
}

If you are using Django < 4.2, type instead (I am using 4.2, so I haven't tested this myself - it is from documentation):



# settings.py
DEFAULT_FILE_STORAGE = "storages.backends.gcloud.GoogleCloudStorage"
STATICFILES_STORAGE = "storages.backends.gcloud.GoogleCloudStorage"

Also for all Django version, type:



# settings.py
# Google Cloud Storage for Static File Serve
GS_PROJECT_ID = env("GS_PROJECT_ID")
GS_BUCKET_NAME = env("GS_BUCKET_NAME")
GS_AUTO_CREATE_BUCKET = True
GS_DEFAULT_ACL = 'publicRead'

STATIC_URL = 'https://storage.googleapis.com/{}/'.format(GS_BUCKET_NAME)

(You can check cloud storage documentation to learn more.)

In your .env file, make sure to include GS_PROJECT_ID and GS_BUCKET_NAME.

Don't forget to also set STATICFILES_DIRS and STATIC_ROOT properly in the settings file. Check documentation for more details.

Ok, now type following into terminal to collect static files:



$ python manage.py collectstatic

Type "yes" to overwrite existing files. It may take few minutes. When it completes, go to Cloud Storage bucket, and you will see your static files! (Notice that there is no 'staticfiles' folder created in your project :))

Congratulation! In the next section, we will make sure to update Cloud Run instance. (You should know how to do it, so go ahead and try!)

Update Cloud Run instance to use Storage Bucket

First make sure everything is saved, and let's build our image again and push to repository (If you have 'staticfiles', you can delete it and add 'staticfiles' to .dockerignore):



$ docker build -t <<image-name>> .
$ docker tag <<image-name>> <<region>>-docker.pkg.dev/<project-id>>/<<repo-name>>/<<image-name-you-want>>
$ docker push <<region>>-docker.pkg.dev/<project-id>>/<<repo-name>>/<<image-name-you-want>>

Now go to Cloud Run instance, and "EDIT & DEPLOY NEW REVISION" while making sure you use the new image.

Also don't forget to include GS_PROJECT_ID and GS_BUCKET_NAME to Environment Variables.

Now Deploy! Everything should work as intended now!

And that is everything guys! It was a long article, but hopefully it was helpful. It was my first article, and writing isn't exactly my strength. I hope everything was clear, but if you have any questions, please leave comments.