DEV Community: Max Myroshnychenko

Jupyter notebooks on gitlab pages

Max Myroshnychenko — Fri, 30 Sep 2022 15:52:16 +0000

Problems

Some plotting packages' output (holoviews, Altair) are currently not always saved as a part of the notebook itself.
Sharing results with collaborators takes an extra step of extracting plots and putting them together in something like pdf or html

Solution: jupyter-book plus gitlab pages

Automatically save the output of a notebook in a common, shareable format. Figures are included, and not just matplotlib based ones! Here is a small demo

Set up jupyter-book on your computer

Initialize jupyterbook in your projects folder. Be in the folder above the projects folder:

pip install -U jupyter-book cookiecutter sphinx myst_nb
jupyter-book create projects --cookiecutter

Type through the prompts. Example is in screenshot above

Daily use

Create some jupyter notebooks in your my_cool_project folder
Add your newly created notebooks' names to the file toc.yml in the folder my_cool_project
Compile jupyter code to html locally:

cd projects/my_cool_project
jb build my_cool_project

Set up for jupyter-book on gitlab

Create an empty repository my_cool_project in gitlab
Upload the newly created folder my_cool_project to gitlab my_cool_project repository
- Follow the section "Push an existing directory" that appears on gitlab when you create an empty project
- You may have to substitute the git init step with:
```
git init                              
git symbolic-ref HEAD refs/heads/main
```
Sync to gitlab

 git add * && git commit -m 'I did some work' && git push

A blue build icon should appear on gitlab and then change to a green checkmark. That means your pages are available at https://<YourUsername>.gitlab.io/<RepoName> (e.g. https://mmyros.gitlab.io/my_cool_project/)

Gitlab pages CI debug

If a green checkmark never appears, or https://.gitlab.io/RepoName does not work, edit .gitlab-ci.yml to contain the following, then commit and push like usual.

image: python:3.7                                                                                                                                                                                                  

pages:                                                                                                                                                                                                             
  script:                                                                                                                                                                                                          
    - mv my_cool_project/_build/html/ public/                                                                                                                                                                      
  artifacts:                                                                                                                                                                                                       
    paths:                                                                                                                                                                                                         
      - public                                                                                                                                                                                               

workflow:                                                                                                                                                                                                          
  rules:                                                                                                                                                                                                           
    - if: $CI_COMMIT_REF_NAME =~ /-wip$/  # Pipelines for branch or tag names that include -wip don't run                                                                                                          
      when: never                                                                                                                                                                                                  
    - if: '$CI_PIPELINE_SOURCE == "push"'

My pycharm setup

Max Myroshnychenko — Fri, 30 Sep 2022 14:34:21 +0000

Debugger

Check Drop into debugger on failed tests

Then you can use debugger:

Use left panel to go up and down the stack of your functions
Click Console to type and debug as if you were inside your functions # Console startup
Settings -> Console, uncheck Show console variables by default
Settings -> Console-> Python Console:
- in Starting script, add your database connection:

import sys

print('Python %s on %s' % (sys.version, sys.platform))

sys.path.extend([WORKING_DIR_AND_PYTHON_PATHS])
from can_dj import *
import logging
logging.getLogger("datajoint").setLevel(logging.ERROR)

Disable reader mode

in Editor-> Reader Mode, uncheck Enable reader mode

Things I wish I knew before starting a relational database

Max Myroshnychenko — Mon, 10 Jan 2022 19:43:33 +0000

Why relational DAG

Checks on data inegrity means you can pass your project to the maintainers
Lower barrier to try out ideas on old data

Ability to try new features without breaking existing workflow

No-code parallization with linear scaling (up to n channels sppedup for preprocessing)

Workflow graphics

No-extra-work unit testing (in theory)

No extra work for usage documentation. This frees up devs to document features/science

Engineering process pointers

Write two schemas at a time, one you're thinking about now and its child
First, populate a few recording sessions, not all datasets
Once the current schema and its child seem to work fine, get them populating and move on the the next node.
Work on whole analysis simultaneously - no need to wait for all step to finish.
Can start next as soon as a few keys have populated.
My data entry stage is during recording when I make the filename
Keep Datetime as primary key for restrictions on populate calls (Not so sure anymore)
If you work on children and parents simultaneously, parents may not show up for a while. Wait until mysql processes them first.

Organizational principles

Separate phases of analysis: ingestion/condensing, chunking, computation, organization, plotting
baby steps at the top, cram many at the bottom - bottom is high overhead due to multiplication of keys (eg events), whereas dropping top nodes means recalculate all children
For populate calls, restrict by list of sessions if you suspect that some are not populating due to errors
if not all items are showing up at the end of populate, repeat with reserve=False
start with reserve=False for schemas that require little cpu

Gotchas
Fractional primary keys must be stored as decimal (10,30)

Organizing jupyter notebooks (OLD)

Max Myroshnychenko — Tue, 01 Sep 2020 23:08:52 +0000

Jupyter notebooks are great, but I hit a few grave problems once I accumulated a handful.

Some problems you may encounter:

Remembering what was in each one is a chore
It's tempting to pile on more and more code into the same notebook. They grow in complexity and size
They break overtime. If they rely on my outside functions, changes in these functions will break the notebooks.
Since they are not typically included in an IDE, spotting errors such as those in problem 3 is nearly impossible unless you re-run them regularly
Notebook anxiety. As consequence of problems 3 and 4, you are not likely to rerun them, because something surely broke down while you weren't looking!
Some plotting packages' output (holoviews, Altair) are not always saved as a part of the notebook
Sharing results with collaborators takes an extra step of extracting plots and putting them together in something like pdf or html

Ideal solution would:

Automatically save the output of a notebook in a common, shareable format
Include figures, and not just matplotlib based ones
Keep a running list of what's in each notebook
Re-run changed notebooks from scratch to make sure all cells are still runable
Make it easy and stress-free to re-run even ones that didn't change
One command to rerun all relevant notebooks in a project.

Solution: jupyter-book

Jupyter book addresses all of these problems and then some.
For example output, see my small demo. It's very easy to set up - really it's just two commands: jupyter-book create coolproject and jupyter-book build coolproject, but let's go through them in more detail.

How-to

= 1 = Make a new repository (let's call it coolproject) and clone it. Navigate to it in a terminal and then:

pip install jupyter-book
jupyter-book create coolproject

This will make the coolproject folder inside your repo. Navigate to it in your terminal and fire up jupyter:

jupyter notebook

= 2 = Work on some jupyter code. Say you made awesome_code.ipynb, and it's related to the topic wowtopic. Also create wowtopic.ipynbthat only has some text in a markdown cell. You'll use this to introduce the topic. It can be empty for now.
= 3 = In addition to awesome_code.ipynb you created in step 2, you'll find the file toc.yml inside this new folder. Edit it to add awesome_code.ipynb. Since it's related to wowtopic, you'll want toc.yml to have:

- file: wowtopic
  sections:
    - file: awesome_code

= 4 = Compile jupyter code to html: Go to the top folder of your repository and issue the command

jb build coolproject

That's it! Now, you have instantly shareable html representation of all your notebooks.

View results

To recap, the top-level folder that contains your repo is called coolproject. It has a subfolder coolproject. In it, you have a bunch of jupyter files and toc.yml that organizes them into topics.

The last command above makes all of them into a shareable, interlinked group of html files. Open the file coolproject/coolproject/_build/html/index.html with Chrome or Firefox and never get lost in jupyter files again!

There is an html file in that folder for every jupyter file you put in toc.yml. Share any of them with your collaborators. No need to save figures separately. You can even hide code so it doesn't get in the way. Perfection!

Pandas tools you didn’t know you needed, part 2: Groupby

Max Myroshnychenko — Thu, 19 Mar 2020 14:59:47 +0000

Very often, you need to do something to groups of rows of a dataframe that match some condition, for instance a certain mouse or brain region. The most intuitive solution is to use a for loop.

For instance, let's start with this dataframe:

	experiment	brain_region	neuron	Mean ISI
0	bucket_1_m026_1568757659_	pfc	0	3.08281
1	bucket_1_m026_1568757659_	pfc	1	8.37044
2	bucket_1_m026_1568757659_	pfc	2	38.5265
3	bucket_1_m026_1568757659_	pfc	3	31.795
4	bucket_1_m026_1568757659_	pfc	4	3.43186

We can loop over experiments, brain regions, and neuron like so:

log_mean_isi = []
for neuron in df['neuron'].unique():
    for brain_region in df['brain_region'].unique():
        for experiment in df['experiment'].unique():
            sub_df=df[
                (df['experiment']==experiment) & 
                (df['brain_region']==brain_region) & 
                (df['neuron']==neuron) 
            ]
            log_mean_isi.append(np.log(sub_df['Mean ISI']).values)
print(np.hstack(log_mean_isi))

[ 1.12584188  2.19215951 -1.98074317 -0.25722232  2.1247061   0.0063244 0.92415014 -1.93038317  3.6513468   2.28266164  1.39365976 -3.23681353  3.459308    1.48402753 -0.65003803  1.4140086   1.23310109 -0.8740564  1.29957486 -0.60594385 -0.61921541  4.51176028  0.5622578   3.48668087 -1.57394565 -1.15482997  3.62078013 -0.64878302 -2.08398518  0.28354575 -0.92326873 -2.23523802 -1.15944804 -1.16315081  1.29454247 -0.89566237  0.72300259 -1.38527266  1.3694298   2.95919545  3.07555102 -1.38317331  2.3510588  -1.68488011  2.15864159  2.92067342  0.20249933  1.76685816 -1.14606501 -2.54545905  2.17651238  2.49333615 -1.08762839 -1.10690211  1.79675315  1.08438564 -1.61687289  3.64009558  2.50376142  3.53577889 -2.31447026 -0.9560716  ...]

However, nested for loops are evil for many reasons. Let's use the groupby command:

log_mean_isi=[]
for index, sub_df in df.groupby(['neuron', 'brain_region', 'experiment']):
    log_mean_isi.append(np.log(sub_df['Mean ISI']).values)
print(np.hstack(log_mean_isi))

or even in one line:

mean_isi_list=[np.log(sub_df['Mean ISI']).values for index, sub_df in df.groupby(['neuron', 'brain_region', 'experiment'])]
print(np.hstack(mean_isi_list))

Much more readable and efficient! Furthermore, the index part can be used for reconstructing the modified dataframe. I might make a post on that in the future.

Pandas tools you didn’t know you needed, part 1: Apply

Max Myroshnychenko — Thu, 19 Mar 2020 14:59:07 +0000

Dataframe rows can contain arrays rather than a single value. This is the case with Datajoint's "blobs." Let's examine an instance of such a dataframe I fetched from a datajoint table:

	fid	brain_region	single_unit	single_unit_phy	spike_time	waveform	snr
0	bucket_1_m026_1568757659_	pfc	0	12	[ 1.71006667 2.14576667 3.35306667 3.6982 22.4359 22.80743333	[]	[]
					26.00536667 26.61663333 28.39046667 29.45536667]
1	bucket_1_m026_1568757659_	pfc	1	15	[17.8181 37.5422 37.90253333 39.36106667 46.77196667 60.7908	[]	[]
					77.62783333 78.71806667 82.33186667 93.15203333]
2	bucket_1_m026_1568757659_	pfc	2	19	[ 1.552 115.68683333 115.69416667 160.0344 342.9736	[]	[]
					346.51526667 346.8301 348.25513333 348.2767 348.29066667]
3	bucket_1_m026_1568757659_	pfc	3	37	[ 64.1048 145.53183333 185.3421 187.57793333 281.31683333	[]	[]
					304.80466667 326.5742 339.3119 348.08556667 350.2595 ]
4	bucket_1_m026_1568757659_	pfc	4	40	[ 0.66776667 3.9761 4.8187 16.72106667 22.01286667 25.6975	[]	[]
					29.069 29.25703333 30.07223333 31.55446667]

We'd like to get the interspike intervals.
In this case,
np.diff(df['spike_time'])
will not work as it will attempt to work across rows:

array([array([16.10803333, 35.39643333, 34.54946667, 35.66286667, 24.33606667, 37.98336667, 51.62246667, 52.10143333, 53.9414 , 63.69666667]), array([-16.2661 , 78.14463333, 77.79163333, 120.67333333, 296.20163333, 285.72446667, 269.20226667, 269.53706667, 265.94483333, 255.13863333]), array([ 62.5528 , 29.845 , 69.64793333, 27.54353333, -61.65676667, -41.7106 , -20.2559 , -8.94323333, -0.19113333, 1.96883333]), ..., array([29.80516667, 38.70003333, 40.26796667, 53.5446 , 45.7118 , 55.4116 , 55.9258 , 79.12003333, 77.03993333, 83.48073333]), array([-11.45576667, -11.35606667, -9.69276667, -23.187 , -22.03416667, -31.00213333, -29.06093333, -51.92653333, -51.33696667, -57.59203333]), array([ 72.3652 , 92.1018 , 157.1723 , 173.74453333, 185.13646667, 188.13473333, 187.84116667, 187.3681 , 206.6341 , 206.1082 ])], dtype=object)

You can use a for loop, but there is an easier way.

To get our operation of interest to work row by row without looping, use apply:

df['ISI']=df['spike_time'].apply(np.diff)
df['ISI']

	fid	brain_region	single_unit	single_unit_phy	spike_time	waveform	snr	ISI
0	bucket_1_m026_1568757659_	pfc	0	12	[ 1.71006667 2.14576667 3.35306667 3.6982 22.4359 22.80743333	[]	[]	[ 0.4357 1.2073 0.34513333 18.7377 0.37153333 3.19793333
					26.00536667 26.61663333 28.39046667 29.45536667]			0.61126667 1.77383333 1.0649 ]
1	bucket_1_m026_1568757659_	pfc	1	15	[17.8181 37.5422 37.90253333 39.36106667 46.77196667 60.7908	[]	[]	[19.7241 0.36033333 1.45853333 7.4109 14.01883333 16.83703333
					77.62783333 78.71806667 82.33186667 93.15203333]			1.09023333 3.6138 10.82016667]
2	bucket_1_m026_1568757659_	pfc	2	19	[ 1.552 115.68683333 115.69416667 160.0344 342.9736	[]	[]	[1.14134833e+02 7.33333333e-03 4.43402333e+01 1.82939200e+02
					346.51526667 346.8301 348.25513333 348.2767 348.29066667]			3.54166667e+00 3.14833333e-01 1.42503333e+00 2.15666667e-02
								1.39666667e-02]
3	bucket_1_m026_1568757659_	pfc	3	37	[ 64.1048 145.53183333 185.3421 187.57793333 281.31683333	[]	[]	[81.42703333 39.81026667 2.23583333 93.7389 23.48783333 21.76953333
					304.80466667 326.5742 339.3119 348.08556667 350.2595 ]			12.7377 8.77366667 2.17393333]
4	bucket_1_m026_1568757659_	pfc	4	40	[ 0.66776667 3.9761 4.8187 16.72106667 22.01286667 25.6975	[]	[]	[ 3.30833333 0.8426 11.90236667 5.2918 3.68463333 3.3715
					29.069 29.25703333 30.07223333 31.55446667]			0.18803333 0.8152 1.48223333]

This get the mean without having to resort to looping over rows. If you have a more complex procedure in mind, simply define it before using apply:

def inverse_isi(spikes):
    """
    Inverse mean ISI of spikes is an alternative measure of firing rate
    """
    return 1/np.mean(np.diff(spikes))

and then apply it in the same way

df['ISI-based firing rate']=df['blob_column'].apply(inverse_isi)

Pandas tools you didn't know you needed, putting it together: groupby+from_records

Max Myroshnychenko — Thu, 19 Mar 2020 14:55:29 +0000

Say we are trying to fetch unique instances of several conditions:

index_cols = 'brain_region', 'stim_type', 'frequency', 'phase_delay_frac', 'stim_amplitude', 'phase_type', 'brain_region'

Each of these dataframe columns contains several unique entries repeated multiple times. Eg brain region takes the values of pfc and v_hpc.
Let's use an example key to examine:

# Fetch an entire day's worth of events:
summary = (SpikePhase * MazeTrials & key).fetch(format='frame').reset_index()
key = dict(fid='bucket_5_m036_1570900150_', single_unit=1, phase_type='theta_hilbert_bessel')
for brain_region in summary['brain_region'].unique():
  for stim_amplitude in summary['stim_amplitude'].unique():
    for phase_delay_frac in summary['phase_delay_frac'].unique():
      for frequency in summary['frequency'].unique():
        for phase_type in summary['phase_type'].unique():
          for stim_type in summary['stim_type'].unique():
             # specify primary key for fetching and populating
             group_key['stim_amplitude'] = stim_amplitude
             group_key['phase_delay_frac'] = phase_delay_frac
             group_key['frequency'] = frequency
             group_key['stim_type'] = stim_type
             group_key['phase_type'] = phase_type
             group_key['brain_region'] = brain_region
# fetch specific to treatment
             spikes_phases = (MazeTrials *
                              * EventRelatedSpikesAndPhases 
                              & group_key
                              ).fetch(format='frame')

becomes simply:

# Iterate over indices of interest:
for index, dff in summary.groupby(index_cols):
    # make a dictionary holding values of index
    group_key = {key: value for (key, value) in zip(index_cols, index)}
    # fetch specific to treatment
    spikes_phases = (MazeTrials * EventRelatedSpikesAndPhases & group_key).fetch(format='frame').reset_index()

Note the absence of nested for loops!
Once we have all treatment-specific events, we can do some computation on them within this loop and build the list of computed results. In the last line of the snippet below, we will concatenate them into a dataframe:

    # ...continued from above
    # do some computation here
    spikes_phases['log_stim_amplitude'] = np.log(spikes_phases['stim_amplitude'])
    # add to our list of dataframes (preallocated outside the loop):
    group_results.append(spikes_phases)
# Concatenate dicts into a dataframe:
group_df=pd.DataFrame.from_records(group_results).set_index(index_cols)

The columns of group_df now have our index_cols, original data column, and our computed column:

> group_df.columns
Index(['brain_region', 'stim_type', 'frequency', 'phase_delay_frac', 'stim_amplitude', 'phase_type', 'spike_phase','log_spike_phase'])