DEV Community: Mike Lockhart

Use Jupyter Notebooks for bash scripting

Mike Lockhart — Mon, 10 Mar 2025 20:18:48 +0000

I'm experimenting with using Jupyter to capture Linux server commands in a Notebook while I figure out doing GitLab Geo setup. It's complicated, so I want to keep good notes, but copy/paste from the terminal to a text editor is a chore I'd rather not deal with. Instead I would like to keep a literate-programming style notebook. I learnt recently that there is a Jupyter kernel for running bash instead of python, and that would be ideal.

This post describes how to set up using Jupyter Notebooks on a remote host, and then running and editing notebooks using Visual Studio Code's support for Jupyter.

You can also do this without VSCode, by SSH tunnelling a local web browser to the host. That would be better in cases where users' /home volume size is restricted. I briefly discuss this too.

Install Python, Jupyter, and the Bash kernel

First we need an environment for running Jupyter. I did try this globally, since I'm using a dedicated host, but it works better as a virtual environment ("venv").

This server runs Ubuntu, so the software package installation is specific to that OS.

Install the python-venv package

In Ubuntu, Python is separated into multiple .deb packages, and the python virtual environment functions are in pythonX.Y-venv. Don't worry about the specific version in the package name: if you install python3-venv then APT will get the latest version available:

sudo apt install python3 python3-venv

Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3 is already the newest version (3.8.2-0ubuntu2).
The following NEW packages will be installed:
  python3-venv
0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Need to get 1228 B of archives.
After this operation, 11.3 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://us-west1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 python3-venv amd64 3.8.2-0ubuntu2 [1228 B]
Fetched 1228 B in 0s (17.2 kB/s)       
Selecting previously unselected package python3-venv.
(Reading database ... 197373 files and directories currently installed.)
Preparing to unpack .../python3-venv_3.8.2-0ubuntu2_amd64.deb ...
Unpacking python3-venv (3.8.2-0ubuntu2) ...
Setting up python3-venv (3.8.2-0ubuntu2) ...
Processing triggers for man-db (2.9.1-1) ...

Create a python virtual environment

Make a directory for keeping your notebooks, and navigate to it:

NB=~/lab/notebooks
mkdir -p $NB; cd $NB

Create a virtual environment using Python's venv module, installed above. I'm following the usual practice of putting the venv within the project directory, rather than my preferred practice of putting venvs in ~/lib/venv:

python3 -m venv venv

You can now activate the venv for the current shell session:

source venv/bin/activate

Install Jupyter and bash_kernel

Now that the venv is activated, install Jupyter and the bash_kernel into it:

pip install -U pip jupyter bash_kernel
python -m bash_kernel.install

At this point we can start Jupyter with the bash kernel.

Running Jupyter notebook

Launch the notebook, without automatically spawning a web browser, since that makes no sense on the remote host:

jupyter notebook --no-browser

Jupyter will print status information to the terminal. Among it are details for connecting securely to the server to view and run notebooks:


[I 2025-03-08 21:30:42.929 ServerApp] Serving notebooks from local directory: /home/mjl/lab/notebooks
[I 2025-03-08 21:30:42.929 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2025-03-08 21:30:42.929 ServerApp] http://localhost:8888/tree?token=REDACTEDTOKEN
[I 2025-03-08 21:30:42.929 ServerApp]     http://127.0.0.1:8888/tree?token=REDACTEDTOKEN
[I 2025-03-08 21:30:42.929 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2025-03-08 21:30:42.933 ServerApp]

    To access the server, open this file in a browser:
        file:///home/mjl/.local/share/jupyter/runtime/jpserver-7453-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/tree?token=REDACTEDTOKEN
        http://127.0.0.1:8888/tree?token=REDACTEDTOKEN

Note that a fresh token is generated each time that Jupyter starts.

Tunnel to Jupyter and open in a local web browser

Typically, the next step when running Jupyter on a remote host, is to "tunnel to the host" and then connect a local web browser through the tunnel to view the notebook server. The SSH command for that is:

ssh -N -L localhost:8888:localhost:8888 username@remote_host

After that is connected, use your local web browser to view the URL printed by Jupyter above. If the tunnel held, you are in and have a web interface to Jupyter Notebook.

Running bash in a notebook

You can create notebooks on the remote host using the Bash kernel, and run bash commands in cells of the notebook, executing on the remote host!

What's great about this is that the bash kernal maintains state, just like any other Jupyter kernel. So the environment persists between cells, as it is the same process executing each cell, unlike with Python-kernel cell-magics which fork a sub-process to execute the shell commands.

So this makes the bash kernel for Jupyter ideal for exploring and documenting complicated Linux server setup.

Connect to the host from VSCode

A nicer way to do this is with VSCode Remote SSH. It has these advantages:

Automatically forward ports, so port 8888 will be browsable locally, without the SSH command above
Nicer / more powerful editing than Jupyer in a browser
Open the notebook directly and connect to the server running on the host
Absolute filenames and directories that appear in the shell output can be clicked on and opened with VSCode
View the Jupyter console output, and the notebook, together in the same window
Navigate an outline of the notebook (like Jupyter Lab), browse files in the remote filesystem, run a remote terminal
Other IDE features, like IntelliSense for shell aliases and functions from within the Notebook!

If you followed along the above with SSH in a normal Terminal application, but now want to try using VSCode, then exit the Jupyter server with ⌃-C (Ctrl-C), and open a new VSCode Remote SSH connection to the host, from VSCode:

F1 Remote-SSH: Connect Current Window to Host...

(If the host is in your SSH configuration, it'll be selectable from the quick-pick)

Install Python and Jupyter VSCode extensions

You must install these VSCode extensions in the Remote-SSH session. They are necessary for the integration to work:

Python (Microsoft)
Jupyter (Microsoft)

Launch Jupyter inside VSCode

Having installed the extensions, we can now open a new notebook in VSCode.

Open the VSCode integrated terminal and change to the directory with the notebooks:

cd ~/lab/notebooks

Make sure to activate the environment:

source venv/bin/activate

Then start Jupyter again, like before:

jupyter notebook --no-browser

The Jupyter server output will contain the new authentication token information, for this instance of Jupyter.

Making new notebooks and running cells

With Jupyter running inside VSCode, there's an automatic port-forwarding for Jupyter.

Open a local web browser through the VSCode tunnel

VSCode automatically detects listening TCP ports and tunnels them for us. We can connect our local web browser through the tunnel that VSCode provides, without need for special SSH parameters. Review the PORTS panel to confirm the forwarded ports, but Jupyter usually opens port 8888 by default:

http://localhost:8888/?token=PUT_TOKEN_HERE

Use the connection details shown on the TERMINAL panel for the token.

Open a Notebook directly in VSCode

You can create and interact with notebooks directly in VSCode:

Create a new buffer.
Save the buffer with a .ipynb filename extension. The file re-loads as a Notebook, in VSCode's Notebook editor.
Select the Bash kernel from the Select a kernel button in the top-right.
1. Choose Jupyter kernel... from the quick-pick list.
2. Then choose Bash (venv/bin/python).
Add a code cell by pressing A and enter some bash code in it. Try uname -a to convince yourself it is the remote host.
Run the cell to test that it works, using the ⇧-return keyboard shortcut.

Enjoy editing and executing bash cells with the full features of the VSCode editor and any other extensions that you would like to add.

Side-note: Storage volume consumption by VSCode and Jupyter

Do note that after installing all of this, there will be quite a lot of data in your $HOME directory. The VSCode server components added 510MB:

du -sh ~/.vscode-server/* | sort -h

21M /home/mjl/.vscode-server/code-e54c774e0add60467559eb0d1e229c6452cf8447
60M /home/mjl/.vscode-server/data
195M    /home/mjl/.vscode-server/cli
234M    /home/mjl/.vscode-server/extensions

Breaking down the 234MB of extensions:

du -sh ~/.vscode-server/extensions/* | sort -h

8.0K    /home/mjl/.vscode-server/extensions/extensions.json
116K    /home/mjl/.vscode-server/extensions/ms-toolsai.jupyter-keymap-1.1.2
512K    /home/mjl/.vscode-server/extensions/ms-toolsai.vscode-jupyter-slideshow-0.1.6
556K    /home/mjl/.vscode-server/extensions/ms-toolsai.vscode-jupyter-cell-tags-0.1.9
21M /home/mjl/.vscode-server/extensions/ms-toolsai.jupyter-2025.1.0-linux-x64
25M /home/mjl/.vscode-server/extensions/ms-toolsai.jupyter-renderers-1.1.0
38M /home/mjl/.vscode-server/extensions/ms-python.debugpy-2025.4.0-linux-x64
48M /home/mjl/.vscode-server/extensions/ms-python.python-2025.2.0-linux-x64
103M    /home/mjl/.vscode-server/extensions/ms-python.vscode-pylance-2025.3.1

Also the python virtual environment, and Jupyter itself, are pretty significant:

du -sh ~/lab/notebooks/venv

247M    /home/mjl/lab/notebooks/venv

$ du -sh ~/lab/notebooks/venv/share/* | sort -h

12K /home/mjl/lab/notebooks/venv/share/applications
12K /home/mjl/lab/notebooks/venv/share/man
40K /home/mjl/lab/notebooks/venv/share/icons
2.2M    /home/mjl/lab/notebooks/venv/share/python-wheels
23M /home/mjl/lab/notebooks/venv/share/jupyter

Be mindful of this storage consumption! Running Jupyter within VSCode on remote hosts is probably something only to be done in environments where you can afford this extra ¾ GB storage.

It's also installed per user, so a shared host — such as a university mainframe — will quickly balloon the /home volume.

Saving and Restoring GCP instance snapshots

Mike Lockhart — Mon, 10 Mar 2025 20:12:00 +0000

This post lists the full steps for how to snapshot/restore a Google Cloud Platform (GCP) compute instance, illustrating a few things that might relieve some frustration about performing this "simple operation" in the Googleplex.

Recently I was making a lot of changes to some GitLab virtual machines as I learnt about setting up GitLab Geo, and often I would make mistakes and need to start over. This is very doable with virtual machines managed through infrastructure-as-code with Terraform and Ansible, as we do with the GitLab Environment Toolkit, but it does take about 25 minutes to spin up each new virtual machine and install GitLab.

Twenty-five minutes is amazing compared to doing it all by hand, but I felt it would be faster to use snapshots of the machine instead:

Build new machines for Geo primary and secondary
Snapshot them before making the tricky changes
and then if one breaks, restore the snapshot.

The virtual machines are GCP compute instances, so I needed to learn how to do this with the gcloud compute command-line interface to GCP.

Snapshot a GCP compute instance

In GCP, one manages the disks, snapshots, and instances separately. To "snapshot and restore an instance", one really snapshots the disk(s) and then swaps in new disks created from the snapshots.

There are 6 steps:

1. Create a snapshot from the instance's disk

This can be done with the instance online, multiple times.

gcloud compute disks snapshot \
  DISK_NAME \
    --snapshot-names SNAPSHOT_NAME

2. To restore a snapshot, first create a new disk from the snapshot

Depending on the size of the disk, this might take a while, so it's good to do this only when you discover that you need to restore a snapshot.

gcloud compute disks create \
  NEW_DISK_NAME \
    --source-snapshot SNAPSHOT_NAME

3. Stop the instance

While the snapshot and new disk can be made with the instance still running, you must stop the instance to swap disks.

gcloud compute instances stop \
  INSTANCE_NAME

4. Detatch the current disk from the instance

Instances can only have one boot disk.

gcloud compute instances detach-disk \
  INSTANCE_NAME \
    --disk DISK_NAME

5. Attach the new disk to the instance

gcloud compute instances attach-disk \
  INSTANCE_NAME \
    --disk NEW_DISK_NAME \
    --boot

6. Start the instance

gcloud compute instances start \
  INSTANCE_NAME

Consistency with multiple disks

If the instance has multiple disks, then — to maintain consistency of data — it may be necessary to stop the instance for multiple disk snapshots. That would depend on the application and the how the data are distributed over the disks. Also be mindful of which is the --boot disk.

Useful snapshot and disk commands

Some more commands are helpful in working with disks and snapshots:

gcloud compute disks list
gcloud compute disks delete DISK_NAME
gcloud compute snapshots list
gcloud compute snapshots delete SNAPSHOT_NAME

Why do this? It's too complicated!

Once you have a snapshot, you can create a new instance from it. But creating new disks from snapshots and swapping them in has the advantage that the compute instance itself is kept, with the same IP address, labels, and other attributes.

If it were a spot instance, or you must not stop the old instance, then creating a new instance may be better.

The gcloud commands allow this separation of concern, so that you can do snapshot operations and create new disks and instances, without stopping the current instances. In a production environment that's the more common scenario: you can't always stop an instance. But it is still possible, and it's conceptually only 3 steps to do it.

Tech projects at start of 2023

Mike Lockhart — Sun, 22 Jan 2023 21:35:52 +0000

I was ruminating about what's on my wish-list for tech tasks that I'd like to complete this year, and then it struck me that there's a lot going on to update my blog with.

This is my first blog post that I'm also posting to Dev.to. Socials can be here: I don't run a commenting system on my own web site.

The stuff I've been working on lately, but not blogging about, is this:

Getting my bootstrap to work on macOS and openSUSE (testing in Tumbleweed) (works on Mac, openSUSE is 90% complete)
Ansible playbook for workstation setup (works for Mac! Now need to add a branch for SUSE)
Figuring out how I want to install my dotfiles (I am leaning towards a bare Git Repo)
Playing with tmux
Switching to mastodon
Getting ready for a short (15 minute) demo at the next virtual GitLab Meetup

I'm all over the place!

GitLab Meetup

The thing that's most urgent at the moment is actually preparing for that Meetup. It's a virtual meetup, happening at 12:30PM Melbourne time on Tuesday February 14^th (2023-02-14 01:30 UTC). My part is very small, and I'm going to be doing a short demo based upon my GitLab Profile blog posts. The key here will be to keep it short and sweet. I do want to accomplish some key things:

Do it on a self-managed GitLab CE rather than gitlab.com (because the meetup seems to be talking about CI/CD in CE)
Explain the custom profile project naming convention
Show updating a git repo from within a CI job (and link to my code for reference)
Quickly demonstrate building and publishing a custom docker image with the tools needed, and overcoming a common access problem
Keep the actual mechanics of RSS parsing very simple: the point is to demo GitLab, not the RSS parsing, although the examples use it
I also want to make sure that I'm following GitLab's Handbook on public speeking

That's a lot just in itself, actually! You can see that I've been busy documenting all this in issues and snippets. I only need to pull it all together into something that can be quickly groked by someone else….

Switching to Mastodon

I've joined the recent exodus from the bird app. The latest round of layoffs included those who do the mediation of toxic tweets. That was the final straw for me, though I've had an on/off relationship for a few years now. My new handle is @milohax@hachyderm.io. I've started tooting there and I feel about ten times more comfortable. I've started making connections around technical topics, which is that server's speciality. I also have an alt for my Australian interests: @milohax@mastodon.au, though I'm not very active there yet. I'll see how it goes having an alt, I'm not sure if it's the right thing to do or not.

So, all the links in this blog that went to the bird now go to Hachyderm, and from there to the rest of the fediverse. It's so much nicer and fixes a lot of the problems that have made centralised social media so toxic. As more of the people I follow make the same move, I'll need to visit those places less and less. I'm looking forward to it.

Booting up a new workstation

My work upgraded my MacBook Pro to a new model with an M1-Max chip, and I got to keep the 2019 MacIntel MBP! It's very nice to have my own Macintosh after 38 years, and about 25 years after doing my major dev project at UTas on a Macintosh LC II. I do like the Mac, even though moving back to one in 2016 was an adjustment. While I miss using KDE as my daily driver, macOS does have some very nice creature comforts, just ones that I wouldn't have paid for myself. Now I don't need to, so I'm looking into configuration management.

I also want to keep configurations for my Linux desktop, which runs KDE on openSUSE, the best Linux for devolopment work, IMO.

So to do this, I'm updating my dotfiles, and writing a new ansible workstation playbook to install software packages for macOS and openSUSE. It's based off Jeff Geerling's mac-dev-playbook. To install the base requirements for Ansible, I'm updating a bootstrap project, which also clones my bash library and fish library. That's a whole lot of things to pull together, but I'm working through it gradually. Some of this effort will translate to my work as well, as I'd like to put my dotfiles and bash library onto test servers using Ansible.

Tmux is really nice

I have read about tmux in the past, and it looked really good, but I never had opportunity to use it at my last job. For this job though, I am routinely running up temporary High Availability GitLab deployments to test customer scenarios. A modest-sized 3k reference deployment installs with 32 hosts using our GitLab Environment Toolkit (the same size as a banking application I once supported!) and this is ideal for tmux's windows and panes, and synchronized typing. At the moment it's a new hobby, but it will give me some material to write about.

Blogging

Last year I only made one post. I'd like to make more, though I'm aware that I've made promisses before ;-)

My wish is to be semi regular, and to federate my posts on Dev.to and the fediverse. It least I know that I have a lot to write!

Share your most embarrassing shell pipeline

Mike Lockhart — Sun, 15 Sep 2019 00:43:24 +0000

Sometimes airing your code smells can be cathartic, and motivate you to fix it.

Here's possibly the most embarrassing shell pipeline I currently have in my Bash dotfiles:

            \grep -Rn "alias ${1}=" ${DOTFILES}/source \
                | awk -F: '{print "⍺:\t" $3 "\t--> " $1 ":" $2}' \
                    | sed "s/${1}=//g"

Yes, I'm combining all the filters, grep,awk, and sed to print out an information of where a shell alias is defined in my files. It's part of my describe function for handling aliases:

$ describe gdfs
⍺:      alias 'cpulimit -l 2 -p $(pgrep -f "crash_handler_token=")&'    --> /Users/mjl/.dotfiles/source/40_osx.sh:28

I'm sure all of this can actually be done in awk alone (especially the sed part), but I haven't figured a way to make it so.