DEV Community: Rodolfo Ferro

Running Jupyter notebooks in parallel

Rodolfo Ferro — Fri, 09 Sep 2022 20:09:54 +0000

As we have shared with you before, there are several tools to run Jupyter notebooks on our computer. Some of these tools include certain functionalities that, beyond just executing our notebooks from the start to the end, these functionalities allow us to take advantage of the computer's resources to even carry out parallel executions. In this post we will show you how you can run some notebooks in parallel and we will compare the execution times of some of these tools.

Why run notebooks in parallel?

First of all, we need to understand what advantages it has and in what cases it helps us to be able to run notebooks in parallel. Among some of the advantages that exist, we highlight the following:

We can run different tasks or processes simultaneously by using multiple computing resources during parallel notebook executions.
Parallel notebook executions will allow us to save time, allowing general process executions to be carried out in less time by executing independent processes simultaneously (as mentioned in the previous point).
Parallel notebook executions can generally help us to keep a more organized code structure because of the requirements to be able to perform those executions simultaneously.

Now that we know some of the benefits, a natural interrogative would be to look for a practical use case where running notebooks in parallel would be useful, so let's take a look at a use case.

Use case: Parallelizing ML model training

Let's imagine that we are working on a data process in which the obtaining, cleaning and preparation of data has been previously carried out. We are now at a crucial point in our process: we need to fit a model (or more than one) to our data. Ideally, we have an idea of which models might work, however, what we would like to do is try different models (with different parameters) so that, based on the results obtained, we can choose the one that best fits our data.

A first approach might be to work in a Jupyter notebook, running the entire code from the start to the end of the notebook, adding the corresponding code to load and prepare the data, and later to model and evaluate the results for that data. Then, a copy of the notebook would be created to test with another model, and so on until you have n notebooks for the n different models to be tested. The entire process would look like this:

This process could be executed in parallel, since the notebooks are independent of each other and each one returns its own results. Furthermore, to add good code execution practices and a better work structure, since the data that enters a model is the same for each one, we can separate the data from the modeling and make our execution more efficient:

Again, running model training from a notebook for each model can be done in parallel to solve our problem in this use case.

As we have seen, running notebooks in parallel can be very useful, and even more so by adding a good work structure. Now it is time to know about some tools that will allow us to carry out these actions at a technical level.

Comparison of different tools

Some of the Jupyter notebook tools stand out for their versatility and simplicity of use to run an entire notebook, cell by cell. The extra benefits about some of these tools come when considering executions in parallel.

Here we will share the results after testing and evaluating some of these tools. Note that to make this comparison fair, it takes into account the use of the same code for all executions and we also use Python's time module to measure the execution time. The notebooks used for benchmarking can be found here and correspond to the african_microbiome_portal_data repository. Serial execution cases (each notebook sequentially) are evaluated first, followed by parallel notebook execution cases.

Papermill

As a first option, we will use Papermill, which has a Python API that allows us to run different notebooks using some functions:

# This requires:
# pip install papermill

import papermill as pm
from glob import glob

for nb in glob('*.ipynb'):
    pm.execute_notebook(
        input_path=nb,
        output_path=nb,
    )

Execution time: 1:58.79 total

Ploomber serial

As a second option, we will use Ploomber with serial execution, which also has a Python API that allows us to execute different notebooks using the NotebookRunner function:

# This requires:
# pip install ploomber

from ploomber import DAG
from ploomber.products import File
from ploomber.tasks import NotebookRunner
from ploomber.executors import Parallel

from pathlib import Path
from glob import iglob

dag = DAG()


for path in iglob('*.ipynb'):
    NotebookRunner(Path(path), File(path), dag=dag, papermill_params=dict(engine_name='embedded'))


if __name__ ==  '__main__':
    dag.build(force=True)

Execution time: 59.324 total

Papermill with ploomber-engine

As a third option we will use Papermill again, but now with the ploomber-engine, which adds debugging and profiling features to Papermill:

# This requires:
# pip install papermill ploomber-engine

from glob import glob

for nb in glob('*.ipynb'):
    pm.execute_notebook(
        input_path=nb,
        output_path=nb,
        engine_name='embedded',
    )

Execution time: 51.256 total

Ploomber in parallel (the fastest option!)

Finally, as a fourth option we use Ploomber again but now with its parallel notebook execution feature:

# This requires:
# pip install ploomber

from ploomber import DAG
from ploomber.products import File
from ploomber.tasks import NotebookRunner
from ploomber.executors import Parallel

from pathlib import Path
from glob import iglob

dag = DAG(executor=Parallel())


for path in iglob('*.ipynb'):
    NotebookRunner(Path(path), File(path), dag=dag, papermill_params=dict(engine_name='embedded'))


if __name__ ==  '__main__':
    dag.build(force=True)

Execution time: 25.440 total (the fastest option!)

Conclusion

As we could see, executing notebooks in parallel can be very useful because this allows us to carry out several independent processes at the same time and thus reduce processing time. To do so, there are different tools such as Papermill and Ploomber, which are similar in that each one has its Python API in order to be integrated into our workflows, but different in terms of performance as we have seen; being that the parallel executor of Ploomber allows the parallel execution of notebooks in record time. It should be noted that if you are more familiar with working with Papermill, ploomber-engine in its integration with Papermill gives it parallel execution capacity in a very good time too.

Even more advantages using Ploomber: Ploomber Cloud

Ploomber and plumber-engine can be run remotely in our servers via Ploomber Cloud, so you don't need to worry about a robust infrastructure that can support your runs, we take care of it. You just need to prepare your pipelines with Ploomber and we do the magic for you!

Do you have a particular use case? We'd love to hear what you're working on and help you. Send us a message through our Slack channel or drop an email to rodo@ploomber.io.

References

A simple linear regression with TensorFlow 2.0

Rodolfo Ferro — Thu, 08 Sep 2022 23:54:17 +0000

Linear regression

A very simple approach to perform a linear regression with a single neuron using Keras.

Package import

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')

Random data generation

We create our samples.

x = np.linspace(0, 50, 51)
x

array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
       13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.,
       26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38.,
       39., 40., 41., 42., 43., 44., 45., 46., 47., 48., 49., 50.])

And for each sample add some random noise for the $y$ value.

y = x + 10 * np.random.random((len(x)))
y

array([ 4.59333333, 10.1888939 ,  3.10342612,  3.78295369, 10.57957367,
        8.31186454, 11.97466834,  8.38350117, 14.58165746, 12.7179244 ,
       16.04800824, 16.04253531, 17.7570093 , 15.36504093, 23.32077078,
       19.38063338, 25.72030949, 20.81364232, 18.55875267, 25.1340618 ,
       28.48036   , 21.73467374, 30.81790828, 28.56736033, 28.83225669,
       28.18684725, 34.95836113, 29.90731219, 30.90521404, 38.67280311,
       33.28501437, 40.01292045, 33.16216509, 34.99748693, 35.87077378,
       35.66317699, 36.37898628, 42.26194454, 47.36216501, 45.62434907,
       46.47169133, 48.05329522, 45.83970327, 50.68483177, 53.01284414,
       54.96997999, 46.42993498, 50.41667051, 53.26256812, 58.97071734,
       55.3635401 ])

The generated data looks like this:

plt.scatter(x, y, label='Generated  data')
plt.legend()
plt.show()

Modelling

The model has just a single neuron that will model the linear equation $y = mx + b$.

The trained weight will correspond to the slope $m$ of the equation and the bias to the intersection value $b$.

model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(1, input_shape=[1]))
model.compile(loss='mean_squared_error', optimizer=tf.keras.optimizers.Adam(0.1))

model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 1)                 2         
=================================================================
Total params: 2
Trainable params: 2
Non-trainable params: 0
_________________________________________________________________

We proceed to fit the model.

history = model.fit(x, y, epochs=200)

Epoch 1/200
2/2 [==============================] - 0s 4ms/step - loss: 653.8051
Epoch 2/200
2/2 [==============================] - 0s 2ms/step - loss: 399.1392
Epoch 3/200
2/2 [==============================] - 0s 2ms/step - loss: 205.5257
Epoch 4/200
2/2 [==============================] - 0s 2ms/step - loss: 83.4503
Epoch 5/200
2/2 [==============================] - 0s 2ms/step - loss: 23.8983
Epoch 6/200
2/2 [==============================] - 0s 2ms/step - loss: 14.8980
Epoch 7/200
2/2 [==============================] - 0s 2ms/step - loss: 30.8598
Epoch 8/200
2/2 [==============================] - 0s 2ms/step - loss: 53.7728
Epoch 9/200
2/2 [==============================] - 0s 2ms/step - loss: 69.0219
Epoch 10/200
2/2 [==============================] - 0s 2ms/step - loss: 70.0617
Epoch 11/200
2/2 [==============================] - 0s 2ms/step - loss: 58.6065
Epoch 12/200
2/2 [==============================] - 0s 2ms/step - loss: 43.6842
Epoch 13/200
2/2 [==============================] - 0s 2ms/step - loss: 26.3015
Epoch 14/200
2/2 [==============================] - 0s 2ms/step - loss: 15.8109
Epoch 15/200
2/2 [==============================] - 0s 1ms/step - loss: 12.0864
Epoch 16/200
2/2 [==============================] - 0s 2ms/step - loss: 13.2904
Epoch 17/200
2/2 [==============================] - 0s 2ms/step - loss: 17.2666
Epoch 18/200
2/2 [==============================] - 0s 2ms/step - loss: 20.2788
Epoch 19/200
2/2 [==============================] - 0s 1ms/step - loss: 21.0044
Epoch 20/200
2/2 [==============================] - 0s 2ms/step - loss: 19.3135
Epoch 21/200
2/2 [==============================] - 0s 2ms/step - loss: 16.5099
Epoch 22/200
2/2 [==============================] - 0s 1ms/step - loss: 13.8078
Epoch 23/200
2/2 [==============================] - 0s 1ms/step - loss: 12.4786
Epoch 24/200
2/2 [==============================] - 0s 1ms/step - loss: 11.5450
Epoch 25/200
2/2 [==============================] - 0s 2ms/step - loss: 12.2430
Epoch 26/200
2/2 [==============================] - 0s 2ms/step - loss: 12.6910
Epoch 27/200
2/2 [==============================] - 0s 2ms/step - loss: 13.0053
Epoch 28/200
2/2 [==============================] - 0s 1ms/step - loss: 12.8281
Epoch 29/200
2/2 [==============================] - 0s 1ms/step - loss: 12.4289
Epoch 30/200
2/2 [==============================] - 0s 1ms/step - loss: 11.6947
Epoch 31/200
2/2 [==============================] - 0s 1ms/step - loss: 11.3287
Epoch 32/200
2/2 [==============================] - 0s 2ms/step - loss: 11.3578
Epoch 33/200
2/2 [==============================] - 0s 1ms/step - loss: 11.3632
Epoch 34/200
2/2 [==============================] - 0s 1ms/step - loss: 11.5163
Epoch 35/200
2/2 [==============================] - 0s 1ms/step - loss: 11.5517
Epoch 36/200
2/2 [==============================] - 0s 2ms/step - loss: 11.4354
Epoch 37/200
2/2 [==============================] - 0s 2ms/step - loss: 11.2493
Epoch 38/200
2/2 [==============================] - 0s 2ms/step - loss: 11.0766
Epoch 39/200
2/2 [==============================] - 0s 2ms/step - loss: 11.0697
Epoch 40/200
2/2 [==============================] - 0s 2ms/step - loss: 10.9828
Epoch 41/200
2/2 [==============================] - 0s 1ms/step - loss: 11.0126
Epoch 42/200
2/2 [==============================] - 0s 1ms/step - loss: 11.0186
Epoch 43/200
2/2 [==============================] - 0s 1ms/step - loss: 10.9801
Epoch 44/200
2/2 [==============================] - 0s 1ms/step - loss: 10.9340
Epoch 45/200
2/2 [==============================] - 0s 1ms/step - loss: 10.7914
Epoch 46/200
2/2 [==============================] - 0s 2ms/step - loss: 10.7687
Epoch 47/200
2/2 [==============================] - 0s 2ms/step - loss: 10.6795
Epoch 48/200
2/2 [==============================] - 0s 2ms/step - loss: 10.6936
Epoch 49/200
2/2 [==============================] - 0s 1ms/step - loss: 10.6416
Epoch 50/200
2/2 [==============================] - 0s 1ms/step - loss: 10.6058
Epoch 51/200
2/2 [==============================] - 0s 1ms/step - loss: 10.5609
Epoch 52/200
2/2 [==============================] - 0s 1ms/step - loss: 10.5371
Epoch 53/200
2/2 [==============================] - 0s 1ms/step - loss: 10.4772
Epoch 54/200
2/2 [==============================] - 0s 3ms/step - loss: 10.4357
Epoch 55/200
2/2 [==============================] - 0s 1ms/step - loss: 10.4067
Epoch 56/200
2/2 [==============================] - 0s 2ms/step - loss: 10.3866
Epoch 57/200
2/2 [==============================] - 0s 2ms/step - loss: 10.3872
Epoch 58/200
2/2 [==============================] - 0s 1ms/step - loss: 10.3384
Epoch 59/200
2/2 [==============================] - 0s 2ms/step - loss: 10.3408
Epoch 60/200
2/2 [==============================] - 0s 1ms/step - loss: 10.3051
Epoch 61/200
2/2 [==============================] - 0s 1ms/step - loss: 10.2206
Epoch 62/200
2/2 [==============================] - 0s 1ms/step - loss: 10.1830
Epoch 63/200
2/2 [==============================] - 0s 2ms/step - loss: 10.2140
Epoch 64/200
2/2 [==============================] - 0s 2ms/step - loss: 10.1338
Epoch 65/200
2/2 [==============================] - 0s 1ms/step - loss: 10.1010
Epoch 66/200
2/2 [==============================] - 0s 1ms/step - loss: 10.0534
Epoch 67/200
2/2 [==============================] - 0s 2ms/step - loss: 10.0265
Epoch 68/200
2/2 [==============================] - 0s 1ms/step - loss: 9.9996
Epoch 69/200
2/2 [==============================] - 0s 1ms/step - loss: 9.9761
Epoch 70/200
2/2 [==============================] - 0s 1ms/step - loss: 9.9521
Epoch 71/200
2/2 [==============================] - 0s 2ms/step - loss: 9.9255
Epoch 72/200
2/2 [==============================] - 0s 1ms/step - loss: 9.8958
Epoch 73/200
2/2 [==============================] - 0s 4ms/step - loss: 9.8855
Epoch 74/200
2/2 [==============================] - 0s 2ms/step - loss: 9.8750
Epoch 75/200
2/2 [==============================] - 0s 2ms/step - loss: 9.8222
Epoch 76/200
2/2 [==============================] - 0s 2ms/step - loss: 9.7902
Epoch 77/200
2/2 [==============================] - 0s 2ms/step - loss: 9.7729
Epoch 78/200
2/2 [==============================] - 0s 2ms/step - loss: 9.7519
Epoch 79/200
2/2 [==============================] - 0s 2ms/step - loss: 9.7254
Epoch 80/200
2/2 [==============================] - 0s 2ms/step - loss: 9.7084
Epoch 81/200
2/2 [==============================] - 0s 2ms/step - loss: 9.6988
Epoch 82/200
2/2 [==============================] - 0s 2ms/step - loss: 9.6648
Epoch 83/200
2/2 [==============================] - 0s 2ms/step - loss: 9.6532
Epoch 84/200
2/2 [==============================] - 0s 2ms/step - loss: 9.6367
Epoch 85/200
2/2 [==============================] - 0s 2ms/step - loss: 9.6036
Epoch 86/200
2/2 [==============================] - 0s 2ms/step - loss: 9.6125
Epoch 87/200
2/2 [==============================] - 0s 2ms/step - loss: 9.5628
Epoch 88/200
2/2 [==============================] - 0s 2ms/step - loss: 9.5510
Epoch 89/200
2/2 [==============================] - 0s 2ms/step - loss: 9.5662
Epoch 90/200
2/2 [==============================] - 0s 2ms/step - loss: 9.5147
Epoch 91/200
2/2 [==============================] - 0s 2ms/step - loss: 9.5191
Epoch 92/200
2/2 [==============================] - 0s 1ms/step - loss: 9.5031
Epoch 93/200
2/2 [==============================] - 0s 1ms/step - loss: 9.4601
Epoch 94/200
2/2 [==============================] - 0s 2ms/step - loss: 9.4431
Epoch 95/200
2/2 [==============================] - 0s 2ms/step - loss: 9.4269
Epoch 96/200
2/2 [==============================] - 0s 2ms/step - loss: 9.4388
Epoch 97/200
2/2 [==============================] - 0s 2ms/step - loss: 9.3951
Epoch 98/200
2/2 [==============================] - 0s 2ms/step - loss: 9.4153
Epoch 99/200
2/2 [==============================] - 0s 1ms/step - loss: 9.3706
Epoch 100/200
2/2 [==============================] - 0s 1ms/step - loss: 9.3629
Epoch 101/200
2/2 [==============================] - 0s 4ms/step - loss: 9.3394
Epoch 102/200
2/2 [==============================] - 0s 2ms/step - loss: 9.3237
Epoch 103/200
2/2 [==============================] - 0s 2ms/step - loss: 9.3269
Epoch 104/200
2/2 [==============================] - 0s 2ms/step - loss: 9.3081
Epoch 105/200
2/2 [==============================] - 0s 2ms/step - loss: 9.2812
Epoch 106/200
2/2 [==============================] - 0s 2ms/step - loss: 9.2657
Epoch 107/200
2/2 [==============================] - 0s 2ms/step - loss: 9.2867
Epoch 108/200
2/2 [==============================] - 0s 2ms/step - loss: 9.2592
Epoch 109/200
2/2 [==============================] - 0s 2ms/step - loss: 9.2355
Epoch 110/200
2/2 [==============================] - 0s 2ms/step - loss: 9.2726
Epoch 111/200
2/2 [==============================] - 0s 2ms/step - loss: 9.2075
Epoch 112/200
2/2 [==============================] - 0s 2ms/step - loss: 9.2053
Epoch 113/200
2/2 [==============================] - 0s 1ms/step - loss: 9.1904
Epoch 114/200
2/2 [==============================] - 0s 2ms/step - loss: 9.1810
Epoch 115/200
2/2 [==============================] - 0s 1ms/step - loss: 9.1972
Epoch 116/200
2/2 [==============================] - 0s 2ms/step - loss: 9.1841
Epoch 117/200
2/2 [==============================] - 0s 2ms/step - loss: 9.1773
Epoch 118/200
2/2 [==============================] - 0s 2ms/step - loss: 9.1964
Epoch 119/200
2/2 [==============================] - 0s 2ms/step - loss: 9.1943
Epoch 120/200
2/2 [==============================] - 0s 1ms/step - loss: 9.1598
Epoch 121/200
2/2 [==============================] - 0s 1ms/step - loss: 9.1316
Epoch 122/200
2/2 [==============================] - 0s 1ms/step - loss: 9.1139
Epoch 123/200
2/2 [==============================] - 0s 2ms/step - loss: 9.1245
Epoch 124/200
2/2 [==============================] - 0s 2ms/step - loss: 9.1076
Epoch 125/200
2/2 [==============================] - 0s 1ms/step - loss: 9.0872
Epoch 126/200
2/2 [==============================] - 0s 1ms/step - loss: 9.1018
Epoch 127/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0746
Epoch 128/200
2/2 [==============================] - 0s 4ms/step - loss: 9.0583
Epoch 129/200
2/2 [==============================] - 0s 2ms/step - loss: 9.1542
Epoch 130/200
2/2 [==============================] - 0s 2ms/step - loss: 9.1178
Epoch 131/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0470
Epoch 132/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0456
Epoch 133/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0353
Epoch 134/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0327
Epoch 135/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0275
Epoch 136/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0258
Epoch 137/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0356
Epoch 138/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0048
Epoch 139/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0333
Epoch 140/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0226
Epoch 141/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0144
Epoch 142/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0049
Epoch 143/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9832
Epoch 144/200
2/2 [==============================] - 0s 1ms/step - loss: 8.9752
Epoch 145/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0019
Epoch 146/200
2/2 [==============================] - 0s 1ms/step - loss: 9.0445
Epoch 147/200
2/2 [==============================] - 0s 4ms/step - loss: 8.9637
Epoch 148/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9682
Epoch 149/200
2/2 [==============================] - 0s 1ms/step - loss: 8.9812
Epoch 150/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9747
Epoch 151/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9863
Epoch 152/200
2/2 [==============================] - 0s 1ms/step - loss: 8.9593
Epoch 153/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9487
Epoch 154/200
2/2 [==============================] - 0s 1ms/step - loss: 8.9373
Epoch 155/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9841
Epoch 156/200
2/2 [==============================] - 0s 1ms/step - loss: 9.1398
Epoch 157/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0232
Epoch 158/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9637
Epoch 159/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9288
Epoch 160/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9415
Epoch 161/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9729
Epoch 162/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9534
Epoch 163/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9379
Epoch 164/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9516
Epoch 165/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9502
Epoch 166/200
2/2 [==============================] - 0s 2ms/step - loss: 9.0555
Epoch 167/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9306
Epoch 168/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9888
Epoch 169/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9640
Epoch 170/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9570
Epoch 171/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9267
Epoch 172/200
2/2 [==============================] - 0s 1ms/step - loss: 8.9134
Epoch 173/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9231
Epoch 174/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9520
Epoch 175/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9192
Epoch 176/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9162
Epoch 177/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9310
Epoch 178/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9030
Epoch 179/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9056
Epoch 180/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9027
Epoch 181/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9037
Epoch 182/200
2/2 [==============================] - 0s 2ms/step - loss: 8.8955
Epoch 183/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9029
Epoch 184/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9420
Epoch 185/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9328
Epoch 186/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9198
Epoch 187/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9586
Epoch 188/200
2/2 [==============================] - 0s 2ms/step - loss: 8.8910
Epoch 189/200
2/2 [==============================] - 0s 2ms/step - loss: 8.8981
Epoch 190/200
2/2 [==============================] - 0s 1ms/step - loss: 8.9199
Epoch 191/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9941
Epoch 192/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9090
Epoch 193/200
2/2 [==============================] - 0s 2ms/step - loss: 8.8938
Epoch 194/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9167
Epoch 195/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9029
Epoch 196/200
2/2 [==============================] - 0s 2ms/step - loss: 8.8897
Epoch 197/200
2/2 [==============================] - 0s 2ms/step - loss: 8.8889
Epoch 198/200
2/2 [==============================] - 0s 1ms/step - loss: 8.8904
Epoch 199/200
2/2 [==============================] - 0s 2ms/step - loss: 8.8932
Epoch 200/200
2/2 [==============================] - 0s 2ms/step - loss: 8.9216

And we can plot the loss during the training.

plt.plot(history.history['loss'])
plt.show()

Model prediction

There are two ways to generate the adjusted model. The first one will be simlpy to use the .predict() method directly over the $x$ samples:

y_pred_model = model.predict(x)

plt.scatter(x, y, label='Generated data')
plt.plot(x, y_pred_model, label='Predicted with model', color='c')
plt.legend()
plt.show()

The second (and my favorite) way is to understand the guts inside the network and access the information to literally replicate the model.

In this case we acces the first (and only) layer:

layer = model.get_layer(index=0)
layer

<tensorflow.python.keras.layers.core.Dense at 0x7fe81910e828>

Then, we get and print the weights and biases:

weights = layer.get_weights()
weights

[array([[1.014319]], dtype=float32), array([4.2396894], dtype=float32)]

As we previously mentioned, the only weight will correspond to the slope and the bias to the intersection point. In order to replicate the linear equation we simply do:

m, b = weights[0][0], weights[1]
print(m)
print(b)

[1.014319]
[4.2396894]

y_pred_params = m * x + b
y_pred_params

array([ 4.23968935,  5.25400829,  6.26832724,  7.28264618,  8.29696512,
        9.31128407, 10.32560301, 11.33992195, 12.35424089, 13.36855984,
       14.38287878, 15.39719772, 16.41151667, 17.42583561, 18.44015455,
       19.4544735 , 20.46879244, 21.48311138, 22.49743032, 23.51174927,
       24.52606821, 25.54038715, 26.5547061 , 27.56902504, 28.58334398,
       29.59766293, 30.61198187, 31.62630081, 32.64061975, 33.6549387 ,
       34.66925764, 35.68357658, 36.69789553, 37.71221447, 38.72653341,
       39.74085236, 40.7551713 , 41.76949024, 42.78380919, 43.79812813,
       44.81244707, 45.82676601, 46.84108496, 47.8554039 , 48.86972284,
       49.88404179, 50.89836073, 51.91267967, 52.92699862, 53.94131756,
       54.9556365 ])

plt.scatter(x, y, label='Generated data')
plt.plot(x, y_pred_params, label='Line fitted using parameter values', color='c')
plt.legend()
plt.show()

If you want to run the code above directly on Google Colab, please follow this link.

Setup your Raspberry Pi Model B as Google Colab (Feb '19) to work with Tensorflow, Keras and OpenCV

Rodolfo Ferro — Fri, 22 Mar 2019 17:26:14 +0000

Read this post in a better format, visit my blog.

Description: Setup OpenCV, Tensorflow and Keras as in Google Colab but in your Raspberry Pi, LOL.

Motivation (The struggle is real!)

The other day I was happily training some neural networks I built with Keras using the Tensorflow backend on Google Colab. After I finished training like 4 or 5 different deep neural nets, I downloaded the trained models into my Raspberry Pi 3 Model B and realized that it was not able to compile any of these models... But Google Colab is able to compile them...

It looked pretty weird to me at the beginning, but soon I noticed that it might be caused by the versions (of Python, Tensorflow and Keras) I was using. My Raspberry Pi was running Python 3.4 with Tensorflow 1.1.0 (compiled from scratch) and Keras 2.1.0, meanwhile Google Colab is running the following:

Anyway, after a couple of days trying some installations, after reading a lot of posts and Stack Overflow answers, I ended up solving everything and writing this blog post, which I hope it becomes very helpful for those having this struggle with these installations on their Raspis.

Let's get started

Before any installation, please be sure that you're working on Raspbian Stretch. You can download it from the official site and write the OS image into your memory card using Etcher. (Yes, I totally broke my Raspbian Jesse before noticing that this will work fine and easy with Raspbian Stretch.)

After doing the basic configuration on your Raspberry Pi (connecting it to Wi-Fi, enabling ssh connection, and everything else), we are ready to start the installation of main packages.

First of all, we will need to install pip:

wget https://bootstrap.pypa.io/get-pip.py
sudo python3 get-pip.py

Creating a virtual environment

We will be working on a local environment, so we will need to install virtualenv and virtualenvwrapper. To install them using pip we just need to run in our terminal:

sudo pip install virtualenv virtualenvwrapper

We need to add the following lines to ~/.profile. Let's open it using nano by doing sudo nano ~/.profile and append these lines to the end:

# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh

We now close the file and save it (CTRL+X, Y, Y), and we need to "source it" in our terminal:

source ~/.profile

Now that we have the virtual environment setup, we will create a new virtual environment named tfk (which stands for Tensorflow & Keras). To do so, we simply run:

mkvirtualenv tfk -p python3

This should create a virtual environment with our Stretch's default Python 3 version (3.5.3) and it should be also activated. For any case, to activate/deactivate the virtual environment:

# Activate environment after sourcing ~/.profile
workon tfk

# Deactivate environment
deactivate

Installing OpenCV

We will be installing OpenCV for any image pre-processing that will feed our convolutional neural networks (at least for me it results very useful).

Let's install some dependencies before OpenCV:

sudo apt-get install libcblas-dev libatlas-base-dev libjasper-dev 
sudo apt-get install libhdf5-dev libhdf5-serial-dev
sudo apt-get install libqtgui4 libqt4-test

After installing the dependencies, inside the virtual environment install OpenCV:

pip install opencv-contrib-python

This should install OpenCV in the virtual environment, and we should be able to test it to verify the installation:

Installing Tensorflow and Keras

To install Tensorflow in our Raspberry Pi, we will use pip and install it directly from a whl file that we will get from https://github.com/lhelontra/tensorflow-on-arm. Specifically, we will be installing the 1.13.1 version.

To install Tensorflow directly from this source, we can simply run:

pip install https://github.com/lhelontra/tensorflow-on-arm/releases/download/v1.13.1/tensorflow-1.13.1-cp35-none-linux_armv7l.whl

This should have done the magic, at least with Tensorflow.

Now, to proceed with Keras we will be installing some (more) dependencies and an update:

sudo apt-get install libblas-dev liblapack-dev gfortran
sudo apt-get install python3-dev python3-setuptools
sudo apt-get install python3-numpy python3-scipy python3-h5py

sudo apt-get update

Finally we can install Keras using pip inside our virtual environment:

pip install keras

This should have done the magic, now with Keras.

Let's verify our installations:

Conclusion and installation of other packages

So far, we have installed some basic packages inside our virtual environment for Python 3, as the versions used in Google Colab (possibly a better name for the virtual environment should have been gcolab or something...). This will allow us to compile and run models in our Raspberry Pi, that were trained in the cloud using Google Colab.

To install other packages that are also useful we can keep using pip inside our virtual environment, for instance:

# To install pandas, matplotlib, scikit-learn, flask, and requests
pip install pandas matplotlib scikit-learn flask requests

I hope this post turns out to be helpful.

Building a traffic fine scraper with Python

Rodolfo Ferro — Mon, 07 Jan 2019 09:44:18 +0000

To see this post in a better format, check my blog.

Motivation

The other day I was remembering that during my visit to Mexico City back in June '17 (for my participation on the first edition of the PythonDay México), a friend of mine hosted me at his house during my time there. The thing is that during my stay we were talking about building something that could be useful, so we decided to scrape a website in which car owners from Mexico City would be able to verify their traffic fines.

The website is the following: http://www.multasdetransito.com.mx/infracciones/df-distrito-federal

As you may have noticed if you opened the site, it contains an input to write down the car plate from which you can search the traffic attached to that plate (if it has any).

My friend told me that only a small group of Mexico City residents knew about that site, so it might be useful to create something in which they can consult this info in an easier way.

After this, I was like: "Dude, today I held a workshop on how to build a TwitterBot in 30 minutes for the PythonDay, what if we build a TwitterBot that inputs the plate in the site and then scrapes the info from it automatically?". He said yes. With this, we built MultaBot a TwitterBot that used to read a car plate and scrape for its traffic fines.

We started with this project as a private project, but the other day I was invited by the Division of Law, Politics and Government of my university to give a tech talk about how automation can be useful for future jobs not for only people in tech industry, but also for politicians and lawyers. When I remembered about this, I thought that it would be a nice example, so I wanted to improve on its base and ended with this example script that now I want to share with you.

Experimentation

I'll explain the whole code, so you'll understand what's happening every tep of the way. To begin, I'll import Selenium and the Python data pretty printer:

 1 from selenium import webdriver
 2 from pprint import pprint

Selenium will allow us to automate the control of a web browser, so the idea will be to use it to automatically open the website, input the car plate and after the traffic fines are loaded, extract and parse them for the pprint output.

After this, I created a function to launch the browser and scrape the info off of the car plate. If verbose is set to True, the function will print the details of each traffic fine:

 5 # Browser opener function:
 6 def launch_browser(placa, verbose=False):
 7     # Browser launch with chromedriver:
 8     browser = webdriver.Chrome()
 9
10     # Set url and open it on Chrome:
11     url = 'http://www.multasdetransito.com.mx/infracciones/df-distrito-federal'
12     browser.get(url)
13
14     # Search plate:
15     multas = busca_placa_df(browser, placa, verbose)
16
17     # Close browser:
18     browser.quit()
19     return multas

Inside the function I set the url to be scraped, then I called function named busca_placa_df (which means search_plate_df, df stands for Federal District in Spanish) and then I closed and quit the opened browser for this search.

Then, I built the main function that inputs data and then scrapes the traffic fines:

22 # Plate finder function:
23 def busca_placa_df(browser, placa, verbose=False):
24     # Find ID and input test data:
25     browser.find_element_by_id('plate').clear()
26     browser.find_element_by_id('plate').send_keys(placa)
27
28     # Look for button and click it:
29     browser.find_element_by_xpath('//*[@id="request"]/div[2]/div/a').click()
30
31     # Search result and return it:
32     resultado = browser.find_element_by_id("secciones")
33     if resultado.text == "INFRACCIONES":
34         resultado = "¡No tienes adeudos!"
35     else:
36         resultado = resultado.text
37
38         # Search infraction tables:
39         if verbose:
40             infractions = browser.find_elements_by_id("tablaDatos")
41             if len(infractions):
42                 for index, infraction in enumerate(infractions):
43                     raw_data = infraction.text.split('\n')
44                     folio, fecha = raw_data[1].split(' ')[0:2]
45                     situation = ' '.join(raw_data[1].split(' ')[2:])
46                     motive = ' '.join(raw_data[2].split(' ')[1:])
47                     items = raw_data[3] + ' , '
48                     its = [item.split(' ')[-1] for item in items.split(', ')]
49                     art, fac, par, inc = its[:4]
50                     sanc = raw_data[4]
51
52                     data_dict = {
53                         'Folio': folio,
54                         'Fecha de Infracción': fecha,
55                         'Situación': situation,
56                         'Motivo': motive,
57                         'Fundamento': {
58                             'Artículo': art,
59                             'Facción': fac,
60                             'Párrafo': par,
61                             'Inciso': inc,
62                         },
63                         'Sanción': sanc
64                     }
65                     print("Infracción {}:".format(index + 1))
66                     pprint(data_dict)
67                     print()
68 
69     return resultado

As you see, in lines 24-26 I clear the data in the input field with id=plate from the HTML source, then input the car plate that I want to look for, and after this I look for the button by its XPath and click it (lines 28-29).

I now want to know the results for the search, for this I look for the element with id=secciones in the HTML of the response (lines 31-32) and first verify if it has any debts by looking if the resulting text is only "INFRACCIONES", if so I returned a message saying t has no debts (lines 33-34). In the other case, in lines 35-36 I save the result with the text it contains (which specifies the number of traffic fines for the plate).

If the verbose flag is turned on, then the scraper will get all the info of each traffic fine (lines 38-40). For this, it will look for the element with id=tablaDatos in the HTML, which contains all the info of each traffic fine and if it is not empty (line 41) it will iterate for each traffic fine and then extract all the info them (lines 42-50). I now build a dict with the response to serve it in a nice way using pprint (lines 52-67) and return the result (line 69).

To test the script I just called the function launch_browser with a demo car plate (that coincidently has 2 traffic fines) and print out the results (lines 72-74):

72 if __name__ == '__main__':
73     multas = launch_browser('MKJ3940', True)
74     print(multas)

If you run this in a script, you'd get an output like this one (note that it prints the 2 traffic fines I mentioned 😝):

Conclusion

At the beginning, what we thought would be a project just for fun to work together that night –like a mini hackathon– ended up being a useful project that could be helpful for citizens in Mexico City.

As you have seen, Python has tools that help to automate some things in very cool ways. I mean, if you run the script as it is provided, you should see a browser window to open automatically, then the text gets written alone in the input field and the results are shown by their own. Isn't this really awesome?

Please let me know what you build with Selenium and Python in the comments! 💻🐍🤙🏼

DisAtBot - How I Built a Chatbot With Telegram And Python

Rodolfo Ferro — Tue, 12 Dec 2017 23:13:04 +0000

Originally guest posted by Rodolfo Ferro on Sun 10 December 2017 in Tools @ PyBites

Rodolfo recently joined our Code Challenges and built Disaster Attention Bot (DisAtBot), a chatbot that helps people affected by natural disasters. In this article he shows how he built this bot with Telegram and (of course) Python. Show him some love because who knows, this could be a life saver (pun intended)! We are delighted to have him show this interesting project he submitted for Code Challenge 43 which earned him a book on chatbots. /Rod please share ...

"¿Quién convocó a tanto muchacho, de dónde salió tanto voluntario, cómo fue que la sangre sobró en los hospitales, quién organizó las brigadas que dirigieron el tránsito de vehículos y de peatones por toda la zona afectada? No hubo ninguna convocatoria, no se hizo ningún llamado y todos acudieron"

"El jueves negro que cambió a México"
– Emilio Viale, 1985.

A bit of context...

Since September 19th, 2017 Mexico has been hit by several earthquakes (The Guardian, CNN). This made me wonder how we could better handle the reporting of damaged zones, people buried under the rubble of buildings, injured people in need of medical attention and other situations.

Verificado 19s was an immediate solution to follow up reports from social media and visualize the info on an online map. This required a lot of real-time (24/7) monitoring of posts on social media from people that were located in the affected areas. And that data was updated every ~10 minutes.

So I started thinking about a way to optimize this process for future situations, not only for earthquakes, but for disaster situations in general. This incentivized me to work on this bot for Pybites Code Challenge 43 - Build a Chatbot Using Python.

So DisAtBot was born

DisAtBot automates the process of reporting incidents via messaging platforms, such as Telegram, Facebook Messenger, Twitter, etc. At this time it only supports Telegram, but I hope to expand it to other social media. If you'd like to contribute, see the Contribute section at the end.

You can find DisAtBot at:

Telegram: https://t.me/DisAtBot
The official repo: https://github.com/RodolfoFerro/DisAtBot

The idea was to have a simple flow that allowed disaster reporting to be quick and easy. The general process of DisAtBot is as follows:

The idea is that any user can interact with the bot by selecting options from button menus in the conversation. This greatly speeds up incidents reporting.

The next step would be opening a ticket which will be stored in a database, for the corresponding government instance/public organization/NGO/etc. to validate and send assistance. When no more help is needed, or the situation is under control, the ticket is closed.

Setup

First clone the repo. I used Python 3.6 and the following packages:

To install all dependencies create a virtual env and run:

pip install -r requirements.txt

Then cd into the scripts folder and run the bot as follows:

python DisAtBot.py

Design

The focus of the initial version was the creation of menu buttons for an easy interaction with the user. The second –and main– issue addressed was the conversation handler. A finite state machine was needed to preserve the desired flow and the responses for each state.

I won’t go too deep into the explanation, but the code below will show how I tackled this.

First of all, Telegram’s library has several methods to create button menus for user responses during the conversation flow. The idea is to create a Keyboard Markup to handle responses through buttons. This can either be Inline (buttons will appear in the conversation window) or as a Reply Keyboard (buttons will be displayed under the textbox to write messages).

An example can be seen in the menu function:

def menu(bot, update):
    """
    Main menu function.
    This will display the options from the main menu.
    """
    # Create buttons to select language:
    keyboard = [[send_report[LANG], view_map[LANG]],
                [view_faq[LANG], view_about[LANG]]]

    reply_markup = ReplyKeyboardMarkup(keyboard,
                                       one_time_keyboard=True,
                                       resize_keyboard=True)

    user = update.message.from_user
    logger.info("Menu command requested by {}.".format(user.first_name))
    update.message.reply_text(main_menu[LANG], reply_markup=reply_markup)

    return SET_STAT

As you can see, the keyboard variable is a list that contains the four buttons to be displayed. The layout can be set by nesting lists inside. In this case the Report and Map buttons are in the first row, while FAQ and About buttons are in the second row. This looks like:

Continuing with the code, a ReplyMarkup is needed to handle the button responses. It specifies the layout of the menu: if only one menu is displayed, if it needs to be resized, etc.

A logger is used for the bot, and the update.message.reply(...) function is used to update the displayed text according to the response from the user. The SET_STAT variable returned in this function is a (predefined) integer to return the state at that time, and to follow the flow.

We now understand the menu creation and handling. The reason of using buttons is that we want a quick interaction because the bot is used in an emergency situation.

The conversation handler - Telegram's ConversationHandler - takes care of setting the state or step of the flow we're currently at, the finite state machine I mentioned earlier. Note that each state also needs to handle its respective information (button responses, etc.)

This code shows the conversation handler:

def main():
    """
    Main function.
    This function handles the conversation flow by setting
    states on each step of the flow. Each state has its own
    handler for the interaction with the user.
    """
    global LANG
    # Create the EventHandler and pass it your bot's token.
    updater = Updater(telegram_token)

    # Get the dispatcher to register handlers:
    dp = updater.dispatcher

    # Add conversation handler with predefined states:
    conv_handler = ConversationHandler(
        entry_points=[CommandHandler('start', start)],

        states={
            SET_LANG: [RegexHandler('^(ES|EN)$', set_lang)],

            MENU: [CommandHandler('menu', menu)],

            SET_STAT: [RegexHandler(
                        '^({}|{}|{}|{})$'.format(
                            send_report['ES'], view_map['ES'],
                            view_faq['ES'], view_about['ES']),
                        set_state),
                       RegexHandler(
                        '^({}|{}|{}|{})$'.format(
                            send_report['EN'], view_map['EN'],
                            view_faq['EN'], view_about['EN']),
                        set_state)],

            LOCATION: [MessageHandler(Filters.location, location),
                       CommandHandler('menu', menu)]
        },

        fallbacks=[CommandHandler('cancel', cancel),
                   CommandHandler('help', help)]
    )

    dp.add_handler(conv_handler)

    # Log all errors:
    dp.add_error_handler(error)

    # Start DisAtBot:
    updater.start_polling()

    # Run the bot until the user presses Ctrl-C or the process
    # receives SIGINT, SIGTERM or SIGABRT:
    updater.idle()

It might seem a bit confusing at first, but it boils down to:

The conversation handler has the states of the flow.
It also has entry points (such as the start function), and fallbacks (such as the cancel and help functions).
It also contains some error handlers.
A global LANG variable is used, since the implementation
I forgot to mention - support interacting in English or Spanish! To support this I created dictionaries for each interaction in both languages.

If you want to check the full code of this bot, check out the scripts directory where you'll find the main script and the language dictionaries.

Some other features implemented are geolocation handling and About / FAQ sections. But the best way to know about this project is by watching it in action (for a live demo go to 8:30):

Future work

For future development I am thinking about adding a map. The system already creates a GeoJSON file from the locations acquired.

As mentioned I am considering expanding this to other platforms like Facebook Messenger and Twitter. Another good thing to add would be a website explaining the main use cases of the bot, maybe a wiki –kinda– site?

If you have any other ideas or suggestions feel free to contact me or:

Contribute

If you're interested in contributing to this project, feel free to take a look at the repo's CONTRIBUTING file. I'd be very pleased if this project would grow out to something used in real life to alleviate the dramatic consequences of natural disaster, which always seem to hit when least expected.

Keep Calm and Code in Python!
– Rod

Sentiment analysis on Trump's tweets using Python 🐍

Rodolfo Ferro — Tue, 12 Sep 2017 03:51:41 +0000

Knight's tour solved with Python 🐍

Rodolfo Ferro — Sat, 15 Apr 2017 11:58:17 +0000

My @JuliaSetBot: A (python) bot that Tweets fractals. ðŸ¤–

Rodolfo Ferro — Fri, 14 Apr 2017 08:46:29 +0000

PythonÂ is a languageÂ that can be easily learnt andÂ usedÂ for many things. A cool thing about Python is that many interesting stuff has already been developed. An example of this isÂ Tweepy, which is an easy-to-use Python library for accessing the Twitter API.

I really like to tryÂ integrating different technologies, so I gave myself the task of trying to build a Twitter bot using Tweepy. Te result: I built a bot that everyday at 11:00 UTC (6:00 inÂ Mexico) it generates a random complex number, then it iterates it in a function in order to generate a Julia set, and finally the bot Tweets the fractal generated. You can check it out atÂ @JuliaSetBot.

An example of what it does is embed here:

// Detect dark theme var iframe = document.getElementById('tweet-852476953542225920-708'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=852476953542225920&theme=dark" }

Awesome right? ðŸ¤–

Hi, I'm Rod!

Rodolfo Ferro — Fri, 14 Apr 2017 08:32:51 +0000

I'm a math student, and a science & technology enthusiast. I have served as a Google Student Ambassador on 2014-2015. Right now, I'm on charge of a science section of a radio program at my university (University of Guanajuato), and I'm a team member in CdeCMx.

I have been coding for 6 years, mostly in Python ðŸ’™, but I also use R and C/C++. I like to learn about what seems interesting to me, from the theory to simulate particles interactions to the creation of bots that tweet fractals.

You can find me on Twitter as @FerroRodolfo, and see a bit more of what I like to do on my blog or my Github.

Nice to meet you.