DEV Community: Attila Molnar

Search in your Jupyter notebooks from the CLI, fast.

Attila Molnar — Mon, 15 May 2023 19:57:20 +0000

My colleagues and I have written a large number of Jupyter notebooks. It has been a common problem to search efficiently within them.

Jupyter notebooks are JSON files, so using traditional search methods such as grep are tedious. Without a bit of tinkering it is 😉

Initially, I found a script on the internet called "nbgrep", but it did not work out for us. So, I wrote my own version.

It requires jq for JSON processing and GNU parallel for concurrent searches in the notebooks.

These are awesome tools anyway and can be very handy for data scientists. jq makes it easy to write queries against JSON files, while parallel can be used to execute any kind of code concurrently, or even on multiple machines (using ssh) in a very simple way.

They are easy to install:

Debian and friends:

sudo apt-get install jq parallel

MacOS

brew install jq parallel

You can find my script as a gist here, or if you cannot install parallel for some reason, here is the non-parallel version. I wrote a non-parallel version specifically for this post, so please notify me if something is wrong.

So when you run it as:

nbgrep 'read_[a-z]'

you will get something like:

./foo/bar.ipynb
        df_a = pd.read_csv(
        df_b = pd.read_csv(
./foobar/barfoo.ipynb
        G = obonet.read_obo(url)

(I had to rename most of the result)

The script:

#!/bin/bash
set -euo pipefail

catch() {
  echo "ERROR $1 occurred on $2"
}
trap 'catch $? $LINENO' ERR

pattern="${1? You must provide a search pattern}"

jupyter-search() {
  file="$1"
  pattern="$2"

  matches=$(< "$file" jq '.cells[].source[]' -r \
    | grep -P "$pattern" \
    | xargs -I '%' echo -e "\t%"
  )

  if [ ! -z "$matches" ]
  then
    echo "$file"
    echo "$matches"
  fi

}
export -f jupyter-search

find . \
  -type 'f' \
  -iname '*.ipynb' \
  -not -path '*/.ipynb_checkpoints/*'\
  | parallel jupyter-search {} "$pattern"

Now let's see how the script works.

#!/bin/bash
set -euo pipefail

#!/bin/bash just tells the kernel where to find the interpreter for the script. set -euo pipefail is bash's "strict mode". Without it, bash will not stop the script execution on an error -e or on encountering undefined variables -u. -o pipefail will make sure that if any error happens (non-zero exit code) in a pipeline, then the whole pipeline will be considered erroneous.

catch() {
  echo "ERROR $1 occurred on $2"
}
trap 'catch $? $LINENO' ERR

By trapping errors this way, we can see on which line the error occurred.

pattern="${1? You must provide a search pattern}"

This is the search pattern that you want to find in the notebooks. It can be any Perl-like regex pattern. This line will also provide a helpful error message if the search pattern is not provided.

jupyter-search() {
  file="$1"
  pattern="$2"

  matches=$(< "$file" jq '.cells[].source[]' -r \
    | grep -P "$pattern" \
    | xargs -I '%' echo -e "\t%"
  )

  if [ ! -z "$matches" ]
  then
    echo "$file"
    echo "$matches"
  fi
}

This is a bash function definition. This function has two arguments: $file, read from the first positional argument, and the search $pattern, read from the second one. Let's concentrate on the search part:

  matches=$(< "$file" jq '.cells[].source[]' -r \
    | grep -P "$pattern" \
    | xargs -I '%' echo -e "\t%"
  )

Here, the output of the pipeline (commands connected by pipes |) will be assigned to the matches variable.

The first command of the pipeline reads the notebook into the jq JSON processor, which extracts all code cells. These are piped into a grep command which applies the given $pattern as a Perl-like regexp -P. The last command in the pipeline will tabulate the matches found by grep.

The if statement at the end of the function will print the results, given that $matches is not an empty string.

export -f jupyter-search

parallel will execute the given code in a sub-shell, which will not inherit the variables from the parent shell (the shell executing the script itself). Therefore, it is necessary to export the previously defined function, so that it will be accessible in the sub-shell created by parallel.

find . \
  -type 'f' \
  -iname '*.ipynb' \
  -not -path '*/.ipynb_checkpoints/*'\
  | parallel jupyter-search {} "$pattern"

This code will find all regular files (excluding symlinks, directories and device files) with the .ipynb extension in a case-insensitive manner. It will search recursively in the directory where you started the script, omitting only the .ipynb_checkpoints directories. The found notebook files are streamed into the parallel command which applies the jupyter-search function on them with the given search $pattern in as many parallel processes as your CPU core number. All of these processes will send their results onto your standard output, but parallel tidies them up so it will not be messy.

So in conclusion we have a very fast way to search in Jupyter notebooks from the command line. I hope some of you will find it helpful. If you have ideas about how to improve it, I am open to suggestions.

Solution-diffusion model in Rust

Attila Molnar — Sun, 14 May 2023 18:49:09 +0000

I decided to go on an adventure. I will write a simple biological cell simulator in Rust. This is my first time to write a simulator and also my first time using Rust.

In the early stage it will be useful only for the better understanding of already known biological concepts. However, later hopefully it will help to experiment with evolutionary and pharmacological questions. My long term goal is to simulate multiple Escherichia coli bacteria in a complex environment. These bacteria are very well studied, thanks to their significance in human and animal health. They can be both beneficial as probiotics or harmful as pathogens.

I have choosen Rust, because it is a fast, low level language. It has comparable performance to C and C++, but unlike them it has a built-in and streamlined solution for package management and documentation generation. I was able to use them in 5 minutes as a first timer.

However the most telling indicator, that I was able to write the most simple proof of concept in around an hour. Somehow the language abstractions are similar to how I like to model problems in my head, which is deeply subjective, but still it strengthened my decision.

I plan to work in baby steps, so first I programmed a very simple diffusion model.

If you did not hear about diffusion, its basic explanation is easy. Given you have a water tank and you infuse it with oxygen or with some other soluble substance, then the particles of that substance can traverse freely in any direction.

So given time, they will be more or less dispersed in the water tank uniformly. However, if you divide the water tank with a membrane, then you will have the same result, but after a longer time period. The time delay will be dependent on the membrane permeability for that given substance. The lower the permeability, the longer time you must wait.

I wrote pseudo-code-like python script to provide the simplest model for this problem.

def membrane_permeate(a_solvent_o2_count, b_solvent_o2_count, o2_permeability):
    o2_forward = int(round(a_solvent_o2_count * o2_permeability))
    o2_reverse = int(round(b_solvent_o2_count * o2_permeability))

    a_solvent_o2_count = a_solvent_o2_count - o2_forward
    b_solvent_o2_count = b_solvent_o2_count - o2_reverse

    a_solvent_o2_count = a_solvent_o2_count + o2_reverse
    b_solvent_o2_count = b_solvent_o2_count + o2_forward

    return a_solvent_o2_count, b_solvent_o2_count


a_solvent_o2_count = 10000
b_solvent_o2_count = 10
o2_permeability = 0.3
for cycle in range(10):
    a_solvent_o2_count, b_solvent_o2_count = membrane_permeate(
        a_solvent_o2_count,
        b_solvent_o2_count,
        o2_permeability,
    )
    print(f"{cycle},{a_solvent_o2_count},{b_solvent_o2_count}")

The Rust implementation logically is almost the same, but for 3 equal volume solvent spaces across 2 membranes between them.

Well it is almost a gram negative bacteria, only a few parts are missing. XD

(image source wikipedia)

Also you should note that I came up with the term "exosol". So on your quiz you should write what your teacher said, which will be most likely "extracellular environment" which is long and boring but will be the right answer.

Back to the topic: I fell in love with rust, because with the trait language feature I was able to program diffusion in a very close way to how I would reason about it:

"Diffusion is based on the trait of membranes that they can permeate molecules between solvents, which they separate"

and one of the most important attributes of every code is how easy it is to reason about it.

So let's see the code:

First I defined the Solvent struct, which has only one field, the oxygen molecule count. Rust structs seems very similar to C structs, but differ in some important ways. The most important difference is regarding this code is the ability in Rust to implement functions bound to a struct.

struct Solvent {
    o2_count: i128,
}

Then I defined the Membrane also as a struct, has a sole field, its permeability for molecular oxygen, expressed as a simple factor.

struct Membrane {
    o2_permeability: f32,
}

Then I defined the Permeate trait, which has a method with 2 solvents as it parameters.

trait Permeate {
    fn permeate(&self, solvent1: &mut Solvent, solvent2: &mut Solvent);
}

Then I implemented the Permeate trait for the Membrane struct. I just calculated the amount of oxygen flux in both directions by multiplying the oxygen count in the given simulation cycle with the membrane permeability factor.

impl Permeate for Membrane {
    fn permeate(&self, solvent1: &mut Solvent, solvent2: &mut Solvent) {
        let o2_forward = (solvent1.o2_count as f32) * self.o2_permeability;
        let o2_reverse = (solvent2.o2_count as f32) * self.o2_permeability;

        solvent1.o2_count = solvent1.o2_count - (o2_forward.round() as i128);
        solvent2.o2_count = solvent2.o2_count - (o2_reverse.round() as i128);

        solvent1.o2_count = solvent1.o2_count + (o2_reverse.round() as i128);
        solvent2.o2_count = solvent2.o2_count + (o2_forward.round() as i128);
    }
}

Lastly in the main function I declare the players: extrasol, periplasm, cytosol, outer_membrane and inner_membrane. So in a simulation loop I can calculate the diffusion across the two membranes.

The most important two lines are:

outer_membrane.permeate(&mut extrasol, &mut periplasm);
inner_membrane.permeate(&mut periplasm, &mut cytosol);

That is all it takes to calculate the diffusion in every simulation cycle. It is so tidy.

However here is the full main function:

fn main() {
    let mut extrasol = Solvent { o2_count: 10000 };

    let mut periplasm = Solvent { o2_count: 10 };

    let mut cytosol = Solvent { o2_count: 5 };

    let outer_membrane = Membrane {
        o2_permeability: 0.3,
    };

    let inner_membrane = Membrane {
        o2_permeability: 0.3,
    };

    println!("cycle,extrasol_o2,periplasm_o2,cytosol_o2");
    for cycle in 1..=10 {
        println!(
            "{},{},{},{}",
            cycle, extrasol.o2_count, periplasm.o2_count, cytosol.o2_count
        );
        outer_membrane.permeate(&mut extrasol, &mut periplasm);
        inner_membrane.permeate(&mut periplasm, &mut cytosol);
    }
}

So it was a nice experience. In the next version I will step up my game by developing a more sophisticated diffusion model, because now I am very far away from a strong model. One of the most obvious limitations is that in reality those solvents have vastly different volumes. Also the membrane surface areas should be considered. So I will further develop the model using Fick's law. Which I will explain in the next post.

But nevertheless I have a very simple model as a starting point on my way to a nicer bacteria model and I also learnt a bit of Rust, which was joyful.

This project is free and open source under GPL 3.0 license,and can be found on GitHub.

Inspect your node application on a different machine

Attila Molnar — Thu, 09 Aug 2018 15:41:39 +0000

Problem

You wrote a node app, which works on your workstation but not on your server, and you plan to use your Chrome to debug the remote app.

Warning

Debugging anything on a public server, especially on a production server, is an extremely typical way to a monday morning mistake. If your server runs in a container or in a virtual machine, you can use Docker or Vagrant to run an equivalent instance on your local machine, and the method below will still work, it will be even faster without the extra networking overhead.

Solution

Thankfully our problem is fairly easy to solve with the all powerful SSH command, which among other breathtaking features allows you to forward TCP connections from your local machine to your host or vica-versa. So we will forward connections directed to the node debugging port on our workstation to our host.

1) On the host start your app with the --inspect switch

$ node --inspect myapp

In the output you should seek a line like this.

Debugger listening on ws://127.0.0.1:9229/e9580cde-b1ee-4ed1-a23b-e8178390b1b9

It tells us two important things. First the IP 127.0.0.1 ensures us that the debugger won't accept any connection from the network, which may have some shady characters lurking around, especially if you are doing this on the Internet against my warning, then let me hold your beer in awe.

Second the 9229 number just after the IP is the port number used by the debugger, eager to accept a connection from our local chrome.

2) Type the following command on the local machine.

 ssh -L9229:localhost:9229 user@remote_hostname_or_ip

It will forward any TCP connections directed to your local 9229 port to the same port on the host.

3) If everything is fine, now you can debug your remote app the same way as the local one. Just put this line to your URL bar.

chrome://inspect/#devices

The following page will be presented to you.

4) Just click on the inspect

5) Hunt the bugs.

image source: xkcd comic

Bypassing the DNS hierarchy with the dig command

Attila Molnar — Thu, 11 Jan 2018 14:05:09 +0000

Topics

Temporarily switching domain name servers (NS from now).
Tracing domain name resolutions.
Finding a way to interrogate NS directly, bypassing the DNS hierarchy.

First Use case

You change your name server settings at your domain name registrar, but you do not see the effect immediately, lets say this problem emerges from the capability of your resolving NS of choice to cache domain name <--> IP (or other raw record) mappings.¹

Solution

Instead of temporarily changing the name server settings on your whole system, you can use the dig command to override the name server for one request. The super power of dig is that it mimics NS requests.

synopsis:

dig [@address of the NS] [domain name]

example:

$ dig @208.67.222.222 wikipedia.org

output:

; <<>> DiG 9.10.6 <<>> @208.67.222.222 wikipedia.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4381
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;wikipedia.org.                 IN      A

;; ANSWER SECTION:
wikipedia.org.          600     IN      A       91.198.174.192

;; Query time: 75 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Thu Jan 11 11:43:32 CET 2018
;; MSG SIZE  rcvd: 58

If you are interested only in the resulting IPs and raw data.

$ dig +noall +answer @208.67.222.222 github.io | rev | cut -f 1 | rev

output:

151.101.1.147
151.101.65.147
151.101.129.147
151.101.193.147

Second Use case

Similarly to the first case the domain name <--> IP mapping is cached but not by the resolving NS, rather some other NS in its delegation chain.

Solution

We will use some other neat features of dig, first we will trace how our query is delegated in the NS hierarchy, then bypass this hierarchy.

Usual hierarchy:
resolving NS --> Root NS --> TLD NS --> registry operator NS --> NS of a company.

$ dig +trace @208.67.222.222 en.wikipedia.org

; <<>> DiG 9.10.6 <<>> +trace @208.67.222.222 en.wikipedia.org
; (1 server found)
;; global options: +cmd
.                       518400  IN      NS      a.root-servers.net.
.                       518400  IN      NS      b.root-servers.net.
.                       518400  IN      NS      c.root-servers.net.
.                       518400  IN      NS      d.root-servers.net.
.                       518400  IN      NS      e.root-servers.net.
.                       518400  IN      NS      f.root-servers.net.
.                       518400  IN      NS      g.root-servers.net.
.                       518400  IN      NS      h.root-servers.net.
.                       518400  IN      NS      i.root-servers.net.
.                       518400  IN      NS      j.root-servers.net.
.                       518400  IN      NS      k.root-servers.net.
.                       518400  IN      NS      l.root-servers.net.
.                       518400  IN      NS      m.root-servers.net.
;; Received 239 bytes from 208.67.222.222#53(208.67.222.222) in 63 ms

org.                    172800  IN      NS      a2.org.afilias-nst.info.
org.                    172800  IN      NS      b0.org.afilias-nst.org.
org.                    172800  IN      NS      b2.org.afilias-nst.org.
org.                    172800  IN      NS      c0.org.afilias-nst.info.
org.                    172800  IN      NS      d0.org.afilias-nst.org.
org.                    172800  IN      NS      a0.org.afilias-nst.info.
org.                    86400   IN      DS      9795 7 2 3922B31B6F3A4EA92B19EB7B52120F031FD8E05FF0B03BAFCF9F891B FE7FF8E5
org.                    86400   IN      DS      9795 7 1 364DFAB3DAF254CAB477B5675B10766DDAA24982
org.                    86400   IN      RRSIG   DS 8 1 86400 20180124050000 20180111040000 41824 . BCA4iv0QFRRxn9WDpnUvHcuI7AnIMMuq/RhfnkyaSEA+XnYrcxHx2Tom UJDobnWucc5pDI/wLzdNFGP37VYUIfHKlnY8Fv9yFDWA5/nP2Wtdi591 SNrNFSmS+XXCwWIOMl7r6coouzGY1MRkmVXnbLjwN5FqvLqquAwFBWpa a5flfMnO7K5M2trNhYXUTo2BVB/ZnWNox+ynPjJB1xspkfS/K4Fv0BXx S6i6LMQk/rYEo4RIB/2JzPj+GWLTTSqD/LtQSA6f6kFFnDkywBSWSuBq hBGAxyrP0LWYbNuUBn8gP9H1kiWgcn8sgaHbcQlk4uKs9snB55myTQYw 7KLIUg==
;; Received 818 bytes from 192.33.4.12#53(c.root-servers.net) in 36 ms

wikipedia.org.          86400   IN      NS      ns0.wikimedia.org.
wikipedia.org.          86400   IN      NS      ns2.wikimedia.org.
wikipedia.org.          86400   IN      NS      ns1.wikimedia.org.
h9p7u7tr2u91d0v0ljs9l1gidnp90u3h.org. 86400 IN NSEC3 1 1 1 D399EAAB H9PARR669T6U8O1GSG9E1LMITK4DEM0T  NS SOA RRSIG DNSKEY NSEC3PARAM
h9p7u7tr2u91d0v0ljs9l1gidnp90u3h.org. 86400 IN RRSIG NSEC3 7 2 86400 20180201111010 20180111101010 1862 org. A3vUwzoJIFJbiWgQK1/ACwB6ZyvIK99ulaAJAoalHwrKn1JnecZ7Sina rEhvKs4vL+FWLvwLNfDWthfaFsO0++eaVFFQ3A6tWvx8PQk2nSKsGdEI Ri6Nld7lJIkL3DCo0PY+0+WwJFnhUGXTtr+B0uPSGA+jSBsQWYTmJnQI 2dM=
hhdfcaa81hb1nfs0h6sd0pq6ctib1781.org. 86400 IN NSEC3 1 1 1 D399EAAB HHE2JLM2DDBRLOQF1RVBUAGVETJ6EPFD 
hhdfcaa81hb1nfs0h6sd0pq6ctib1781.org. 86400 IN RRSIG NSEC3 7 2 86400 20180130153401 20180109143401 1862 org. YK0/K+mWIph03k8zsmD90XnmsxU2bdm60usXSMR8QnbsS4aJpmzGvGOQ FZOnOuUAv58++rtlXU5SI0cQ5vZ0L/iTGW3iXlD9WBQoa/e45E7vtzbK Kn2fhMSLVLWzn377i8AWOgpC4LNHPKIyA1o1NRtdvxkhASNbY5PQWaQI oN8=
;; Received 642 bytes from 199.19.56.1#53(a0.org.afilias-nst.info) in 299 ms

en.wikipedia.org.       600     IN      A       91.198.174.192
;; Received 89 bytes from 91.198.174.239#53(ns2.wikimedia.org) in 52 ms

Every table is a response from a different NS. The columns are:

The domain name to look up
The time-to-live value [ seconds ] how much time the requester should cache this result.
The Class of the network where the domain name is mapped, which is almost always the internet IN. If you see something like CH, then you will meet history itself.
Type of the record.
The raw data of the record, like the IP address.

The most important lines for us have this form:

;; Received [number] bytes from [address]#[port](domain name) in [milliseconds] ms

like:

;; Received 89 bytes from 91.198.174.239#53(ns2.wikimedia.org) in 52 ms

They expose the address and port of the name server, which we can use to bypass the name resolution hierarchy, which can be useful to identify caching issues.

Applying this:

$ dig -p 53 @208.80.154.238 en.wikipedia.org

; <<>> DiG 9.10.6 <<>> -p 53 @208.80.154.238 en.wikipedia.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31772
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;en.wikipedia.org.              IN      A

;; ANSWER SECTION:
en.wikipedia.org.       600     IN      A       91.198.174.192

;; ADDITIONAL SECTION:
en.wikipedia.org.       600     IN      AAAA    2620:0:862:ed1a::1

;; Query time: 135 msec
;; SERVER: 208.80.154.238#53(208.80.154.238)
;; WHEN: Thu Jan 11 13:55:53 CET 2018
;; MSG SIZE  rcvd: 89

I used the -p switch to set the port explicitly, however 53 is the default port for domain name requests.

Installation

Linux

It is almost always provided by your default package manager or use Nix.

MacOS

You can use MacPorts, Homebrew or Nix.

Windows

As Jack Fletcher pointed out in the discussion you can use Chocolatey.

Online

@nslookuptool shared a link to a nice web interface for dig.

DEV Community: Attila Molnar

Search in your Jupyter notebooks from the CLI, fast.

Solution-diffusion model in Rust

Inspect your node application on a different machine

Problem

Warning

Solution

Bypassing the DNS hierarchy with the dig command

Topics

First Use case

Solution

Second Use case

Solution

Installation

Linux

MacOS

Windows

Online

Further reading