In this post I will talk about a few Jupyter apps. They can be used for several different languages and are not limited to Python. This is a strong point I like about the Jupyter programs, as it gives a REPL to any programming language for which a kernel exists for it, such as the IPython kernel for Python. You can read more about IPython in my blog post:
Installation
The whole jupyter suite can be installed by running pip install jupyter or you use a scientific distribution like Anaconda. As I explained in the previous post, you probably want to install the PyQT5 package as well and make sure the QT5 library is installed on your system.
Jupyter QtConsole
Basically the graphical version of IPython. When you type a function here, it shows a tooltip displaying its arguments which is convenient.
The QtConsole is invoked by typing jupyter qtconsole. Some benefits the QtConsole provides over normal terminals are:
-
Multi-line editing: You can make a new line, write part of a code statement in it and be able to erase said line without much of a hassle or keyboard interrupts, along with the additional frustration of pressing Up to recover good lines. It's possible to press Ctrl-Enter to type a second command simultaneously.
To prove a point about this I will demonstrate pasting a long script into the QtConsole. If you have worked with python long enough you must find it frustrating that a single blank line in the script will cause the normal Python interpreter to immediately start executing up that part of the script which usually results in syntax errors. This does not happen in QtConsole or IPython. How convenient.
# Code from https://www.pythonforbeginners.com/code-snippets-source-code/magic-8-ball-written-in-python/
# It's Python 2 but still the point is the blank lines can be pasted into the QtConsole.
# Import the modules
import sys
import random
ans = True
while ans:
question = raw_input("Ask the magic 8 ball a question: (press enter to quit) ")
answers = random.randint(1,8)
if question == "":
sys.exit()
elif answers == 1:
print "It is certain"
elif answers == 2:
print "Outlook good"
elif answers == 3:
print "You may rely on it"
elif answers == 4:
print "Ask again later"
elif answers == 5:
print "Concentrate and ask again"
elif answers == 6:
print "Reply hazy, try again"
elif answers == 7:
print "My reply is no"
elif answers == 8:
print "My sources say no"
- Pager: QtConsole uses a pager (a pager is a program that displays files one page at a time, such as
less) to display long output. - You know from my IPython article that you can run any command with
!cmd. This actually hang the console if you run a program that reads standard input. IPython has some special commands like%man,%clearand%lessthat allow you to run some of these commands within the IPython terminal or QtConsole without hanging.
Some tips:
- To force a line continuation in QtConsle, use Ctrl-Enter.
- To force a command to be run no matter what it is, use Shift-Enter. This works even if the statement wasn't complete yet and would normally make a line continuation if you just used Enter.
-
%qtconsoleopens a QtConsole within an IPython interactive session.
Jupyter notebook
As I explained earlier, Jupiter is a group of projects. Therefore, it's not surprising that the jupyter command has a bunch of subcommands, similar to git.
Jupyter Notebook is a webapp that lets you make IPython notebooks inside your browser and let you run them in there too. It also shows you the result of each code you run, also inside the browser. In a notebook, the code is organized in logical groups called cells. This is useful when you have a bunch of related code snippets that don't form a whole program together.
It can create and edit plain text files as well as interactive notebooks. When creating a text file as opposed to a notebook, it can be used as a text editor for code.
Jupyter Notebook can do pretty much all the things that QtConsole can do. It can render inline images, math, markdown, HTML tables, audio, youtube and stand-alone video, interactive JS widgets, matplotlib plots and even entire webpages. It can render LaTeX math that is outputted, and can also render latex contained inside markdown, using MathJax. In other words, it will render things like \begin{align} and \end{align} just fine. It also has syntax highlighting like QtConsole which is a big plus for any interpreter.
You run Jupiter Notebook by typing jupyter notebook. It will then spawn a web service on your computer listening on localhost:8888 by default. Then it opens a page in your browser shows you the dashboard, which is a list of folders depending on where you started Jupyter from. Think of it as the root of an enviornment. From there you can navigate into folders and create IPython notebooks (.ipynb). The advantage of starting Jupyter Notebook in this way is that you can run many instances of Jupyter Notebook on different ports, if you want. To open a notebook file directly, run jupyter notebook my_notebook.ipynb.
At the dashboard there are three tabs: a Files tab:
A Running tab that shows you what items are being used currently:
And a Clusters tab which needs IPython parallel to work. I might cover IPython parallel in another post.
Notebooks are internally stored as JSON files, therefore they can be tracked with git and other version control systems, but the nbconvert command can export them as HTML, reStructedText (reST), PDF and other formats.
Some options that can be passed to Jupyter Notebook:
- --no-browser: Do not open a browser window with the dashboard after startup
- --no-mathjax: Disables MathJax rendering. Effectively prevents math equations from rendering. This could be useful if the notebook server is not running on your computer and you have a slow internet connection, as rendering is done server-side.
- --ip and --port: Changes IP address and port of the notebook server, respectively.
This demo shows how to create a notebook:
A notebook that's open is colored green. If you select the checkbox next to the notebook it will display a list of operations to perform on the notebook. Notebooks can be shared by multiple users at once. All of them will be able to edit the notebook.
- Duplicate creates a copy the notebook.
- Shutdown disconnects all users of the notebook.
- View and Edit open the notebook.
- The trash icon permanently deletes the notebook after asking for confirmation.
It is possible to connect to a notebook server using the kernel's ID. You would pass the beginning of the kernel's ID to a jupyter program such as QtConsole to use the same IPython kernel that the notebook server is using, by running jupyter qtconsole --existing a9baa0cd (where a9baa0cd is the beginning of the kernel ID).
You can actually connect in a way easier than this. You just run the %connect_info command in a IPython interpreter, save the JSON output to a file, and connect with jupyter <app> --existing <saved-json-file>. <app> could be qtconsole or notebook among other things.
Explanation of kernels
A kernel is, generally speaking, a process started by Jupyter that evaluates a user's code and returns the output back to Jupyter. This is applicable to all Jupyter subprograms; notebook, qtconsole, whatever. We already saw one such kernel, that is IPython, but there is also a Ruby kernel, a Scala kernel, a Node.js kernel, a Go kernel, a PHP and many more that can be found in this list.
A kernel in a different language is usually installed by using the language's package manager but each kernel has its own way of installing itself.
Generally the method of communication between the kernel and some frontend program is called a Decoupled two-process model. Jupyter uses ZeroMQ as a part of this model. The Notebook webapp uses Tornado to serve HTTP requests.
Using notebooks
The notebook editor behaves like a conventional text editor. You have a toolbar and menu of commands to select. I won't bore you by regurgitating what every command in the menu does, but some notable commands I will note here are:
- File > Download As: Gives you many data formats to export the notebook in. Currently there is Asciidoc, HTML page, Markdown (puts the Python code in code snippet tags), PDF (requires xelatex to be installed), a Python file which comments out markdown, reST, and even Reveal.js slides. There's also an .ipynb option if you just want to download the notebook verbatim.
This is what an HTML exported notebook looks like:
- File > Revert to Checkpoint: Contains commands to revert to a previous saved version of the notebook. At the moment I've only found it makes one checkpoint, the most recent save that happened to the notebook.
- Edit > Edit Notebook Metadata: As I mentioned earlier, IPython notebooks are stored in JSON format. There is an entry for metadata in which you can add useful data that you may need to add to the notebook. For example, if you are working with a team, you might want to create a metadata entry representing the topic the notebook is related to.
- Kernel: In this menu there are commands that interrupt the evaluation of a cell, restart the evaluation and switch kernels.
- Help: In this menu there are pointers to a markdown guide, the Python documentation, and documentation for the most popular packages.
- Keyboard icon: This opens a clickable list of all the actions you can perform on a notebook, alongside their keyboard shortcuts if they are defined. It's very helpful for looking for a command you want to do, because it also has a search bar that can filter commands out.
There are three different types of cells in notebooks: Code cells for writing code in, Markdown cells for writing things in markdown and Raw cells.
Inside a code cell you the output from standard output (stdout) has a white background around it, output from standard error (stderr) has a red background around it. Long outputs can be collapsed by clicking on the margin to the right of the blue or green line.
Raw cells are not evaluated at all. They can be used to put text that you don't want to be formatted in anyway and have the interpreter ignore.
Within the notebook there are two ways to use it. One is the Command mode and its active if you see, to quote the documentation, a grey cell border with a blue left margin:
In Command mode there are single-key keyboard shortcuts you can use instead of clicking on menu items. I pasted a list of shortcuts in command mode for easy reference, but you can access this list from Help > Keyboard Shortcuts.
| Key | Action |
|---|---|
| F | find and replace |
| Ctrl-Shift-F | open the command palette |
| Ctrl-Shift-P | open the command palette |
| Enter | enter edit mode |
| P | open the command palette |
| Shift-Enter | run cell, select below |
| Ctrl-Enter | run selected cells |
| Alt-Enter | run cell and insert below |
| Y | change cell to code |
| M | change cell to markdown |
| R | change cell to raw |
| 1 | change cell to heading 1 |
| 2 | change cell to heading 2 |
| 3 | change cell to heading 3 |
| 4 | change cell to heading 4 |
| 5 | change cell to heading 5 |
| 6 | change cell to heading 6 |
| K | select cell above |
| Up | select cell above |
| Down | select cell below |
| J | select cell below |
| Shift-K | extend selected cells above |
| Shift-Up | extend selected cells above |
| Shift-Down | extend selected cells below |
| Shift-J | extend selected cells below |
| Ctrl-A | select all cells |
| A | insert cell above |
| B | insert cell below |
| X | cut selected cells |
| C | copy selected cells |
| Shift-V | paste cells above |
| V | paste cells below |
| Z | undo cell deletion |
| D,D | delete selected cells |
| Shift-M | merge selected cells, or current cell with cell below if only one cell is selected |
| Ctrl-S | Save and Checkpoint |
| S | Save and Checkpoint |
| L | toggle line numbers |
| O | toggle output of selected cells |
| Shift-O | toggle output scrolling of selected cells |
| H | show keyboard shortcuts |
| I,I | interrupt the kernel |
| 0,0 | restart the kernel (with dialog) |
| Ctrl-V | Dialog for paste from system clipboard |
| Esc | close the pager |
| Q | close the pager |
| Shift-L | toggles line numbers in all cells, and persist the setting |
| Shift-Space | scroll notebook up |
| Space | scroll notebook down |
As you can see, there are ways to change the heading of a markdown cell, the type of the cell and toggling line numbers for all cells.
The other mode is called Edit mode, which lets you edit the cell. and it's shown with a green cell border and a prompt showing in the editor area:
Keyboard shortcuts for edit mode:
| Key | Action |
|---|---|
| Tab | code completion or indent |
| Shift-Tab | tooltip |
| Ctrl-] | indent |
| Ctrl-[ | dedent |
| Ctrl-A | select all |
| Ctrl-Z | undo |
| Ctrl-/ | comment |
| Ctrl-D | delete whole line |
| Ctrl-U | undo selection |
| Insert | toggle overwrite flag |
| Ctrl-Home | go to cell start |
| Ctrl-Up | go to cell start |
| Ctrl-End | go to cell end |
| Ctrl-Down | go to cell end |
| Ctrl-Left | go one word left |
| Ctrl-Right | go one word right |
| Ctrl-Backspace | delete word before |
| Ctrl-Delete | delete word after |
| Ctrl-Y | redo |
| Alt-U | redo selection |
| Ctrl-M | enter command mode |
| Ctrl-Shift-F | open the command palette |
| Ctrl-Shift-P | open the command palette |
| Esc | enter command mode |
| Shift-Enter | run cell, select below |
| Ctrl-Enter | run selected cells |
| Alt-Enter | run cell and insert below |
| Ctrl-Shift-Minus | split cell at cursor |
| Ctrl-S | Save and Checkpoint |
| Down | move cursor down |
| Up | move cursor up |
If the IPython kernel dies for whatever reason, Jupyter will restart the kernel automatically after giving you a notification that it happened. You don't need to worry about losing any work, because notebooks are automatically saved periodically.
The notebook will not shut down if you close the browser tab. You have to shut down the notebook from the Resources tab in the dashboard.
Notebook trust
Notebooks that you get from the internet are not marked as trusted by jupyter until you run them, or trust them with jupyter trust notebook-name.ipynb. An untrusted notebook will not render HTML or Javascript output, because it could contain an exploit for a vulnerability in jupyter.
The way trust works is that it computes a signature of the notebook, similar to a file hash for files. If the signature is not in Jupyter's database, then the notebook is untrusted. Trusting the notebook or running it effectively puts the signature in its database.
Terminals inside Jupyter Notebook
Not only can Jupyter Notebook create notebooks, it can also spawn terminals as well. The terminals are run from the notebook server, which if you only ran jupyter notebook is your own computer. And this isn't an interactive python session: It's a full-fledged shell running bash or whatever default shell your user is configured to use. The user being, of course, the user account that ran jupyter notebook.
It functions as a normal terminal would, and you can run any shell command you can normally run on it. The same terminal can be opened by multiple users and when someone types in it, the other users' copy of the terminal updates with it.
Again, closing the browser tab hosting the terminal does not shut down the terminal. To shut it down you need to click on "Shutdown" in the Running tab, or type exit in the terminal (I don't know if logout will terminate your OS user account session too, and it's inconvenient for me to test).
Copy-Paste with the keyboard does not seem to work, not even with Ctrl-Shift-C/Ctrl-Shift-V.
I advise against using the terminals if you are running a notebook server for a bunch of clients, because anyone who can access the notebook server can spawn a terminal and take complete control of the user account it's running as. It's worth mentioning Jupyter Notebook will stop you from running it as root user, unless you pass --allow-root on the command line.
Running a notebook server
In order to serve many users at once with notebooks, similar to how a web server serves users files, you need a few things. First you will need a Jupyter configuration file, and if you don't have one you can create it with jupyter notebook --generate-config. This creates a configuration file called jupyter_notebook_config.py in the .jupyter directory in your home folder. In other words, ~/.jupyter/jupyter_notebook_config.py.
If you want to password-protect your notebook server, you should set up a password with jupyter notebook password. It will then prompt you for a password to enter like this:
$ jupyter notebook password
Enter password: ****
Verify password: ****
[NotebookPasswordApp] Wrote hashed password to /Users/you/.jupyter/jupyter_notebook_config.json
It is also possible to login to a notebook server using a token that's passed in the HTTP request. The first time you login with a token, the webapp asks you to set a password to login with. It isn't required to set a password but if your notebook server will be handling sensitive data then it's recommended to set a password. Changing the NotebookApp.allow_password_change option in the configuration file to False prevents this behavior. By default it's set to True.
After you set a password you should configure an SSL certificate and private key for the notebook server to use. The path the certificate should be put in the NotebookApp.certfile configuration option and the path to the key should be put in the NotebookApp.keyfile option.
You should then let the notebook server bind to all available IP addresses on the server by setting NotebookApp.ip = '*'. You could also set the NotebookApp.ip configuration option to a fixed IP address, but if your notebook server will be accessed publicly (for example you're hosting it on your website, meaning users will type a DNS name of the notebook) then you should set it to '*' so you don't have to worry about changing it every time your IP address changes. To change the port number from 8888, change the NotebookApp.port option accordingly.
NotebookApp.open_browser should be set to False because you don't want to open a browser tab on your server when Jupyter Notebook starts.
Maybe you want your users to access the notebook server from a prefix other than '/', for example '/notebook/'. You can change the prefix Jupyter Notebook uses with the NotebookApp.base_url option.
And we're done
Well, time to start using Jupyter Notebook more because it clearly isn't just for Python, it can be used for many different languages and it can help make your intentions in writing code more clear with the rich display support.
If you see anything incorrect in this article please let me know so I can correct it.
Image by Christian Bodhi from Pixabay
Top comments (0)