DEV Community: Florian Rohrer

Awesome Data Science with Python

Florian Rohrer — Sun, 13 Jan 2019 16:45:47 +0000

I have created a list of useful python packages for data science.

r0f1 / datascience

Curated list of Python resources for data science.

Awesome Data Science with Python

A curated list of awesome resources for practicing data science using Python, including not only libraries, but also links to tutorials, code snippets, blog posts and talks.

Core

pandas - Data structures built on top of numpy.
scikit-learn - Core ML library.
matplotlib - Plotting library.
seaborn - Data visualization library based on matplotlib.
pandas_summary - Basic statistics using DataFrameSummary(df).summary().
pandas_profiling - Descriptive statistics using ProfileReport.
sklearn_pandas - Helpful DataFrameMapper class.
missingno - Missing data visualization.
rainbow-csv - Plugin to display .csv files with nice colors.

Environment and Jupyter

General Jupyter Tricks
Fixing environment: link
Python debugger (pdb) - blog post, video, cheatsheet
cookiecutter-data-science - Project template for data science projects.
nteract - Open Jupyter Notebooks with doubleclick.
papermill - Parameterize and execute Jupyter notebooks, tutorial.
nbdime - Diff two notebook files, Alternative GitHub App: ReviewNB.
RISE -…

View on GitHub

Sometimes, I have also linked to Youtube Talks, other Github Repos that contain short examples, etc.

Want to contribute? Let me know.

How do you ensure knowledge transfer in your company?

Florian Rohrer — Sat, 25 Aug 2018 22:19:23 +0000

Well, the title says it all. How do you ensure knowledge transfer in your company? How does collegue A know, what collegues B, C and D have been up to? And how can collegue A use the other collegues' insights, and source codes if need be?

Do you have a centralized collection of documents? A Forum, a Wiki?

I would be interested to know some established best practices of your workplace.

A simple way to anonymize data with Python and Pandas

Florian Rohrer — Mon, 13 Aug 2018 09:01:02 +0000

Recently, I was given a dataset that contained sensitive information about customers and that should not under any circumstance be made public. The dataset resided on one of our servers which I deem to be a reasonably secure location. I wanted to copy the data to my local drive, in order to work with the data more comfortably and at the same time not having to fear that the data is less save. So, I wrote a little script that changes the data, while still preserving some key information. I will detail all the steps that I have taken, and highlight some handy tricks along the way.

The Task

The task is to prepare a dataset, such that it can later be used for machine learning purposes (e.g. classification, regression, clustering) without containing any sensitive information. The final dataset should not be too different from the original one and should reflect the initial datasets' distributions.

Let's go

I will be using a Jupyter notebook as my environment. First, let's import all the necessary libraries.

import pandas as pd
import numpy as np
import scipy.stats
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn_pandas import DataFrameMapper
from sklearn.preprocessing import LabelEncoder
# get rid of warnings
import warnings
warnings.filterwarnings("ignore")
# get more than one output per Jupyter cell
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
# for functions we implement later
from utils import best_fit_distribution
from utils import plot_result

I am going to assume that you are already familiar with most of the libraries used here. I just want to highlight three things. sklearn_pandas is a convenient library that tries to bridge the gap between the two packages. It provides a DataFrameMapper class that makes working with pandas DataFrames easier as it allows for changing the encoding of variables in fewer lines of code. With from IPython.core.interactiveshell ... I change a Jupyter Notebook default, such that more than one output is displayed. A nice blog post about other handy Jupyter tricks is here. Finally, we are going to put some code into a file called utils.py, that we will be placing next to the Notebook.

df = pd.read_csv("../data/titanic_train.csv")

For our analysis, we will be using the training portion of the Titanic Dataset.

df.shape
df.head()

Now that we have loaded the data, we are going to strip all the personally identifieable information. The columns ["PassengerId", "Name"] contain such information. Notice that ["PassengerId", "Name"] are unique for every row, so if we build a machine learning model, we would drop them anyways later on. Similar arguments can be made about ["Ticket", "Cabin"], which are almost unique for every row.

For purposes of this demonstration, we will not deal with missing values. We simply disregard all observations, that contain missing values.

df.drop(columns=["PassengerId", "Name"], inplace=True) # dropped because unique for every row
df.drop(columns=["Ticket", "Cabin"], inplace=True) # dropped because almost unique for every row
df.dropna(inplace=True)

The result looks like this.

df.shape
df.head()

Next, to strip even more information, and as a preprocessing step for later, we are going to encode Sex and Embarked with numeric values. Sex will be coded 0,1, Embarked will be coded 0,1,2. The class LabelEncoder() does most of the work for us.

encoders = [(["Sex"], LabelEncoder()), (["Embarked"], LabelEncoder())]
mapper = DataFrameMapper(encoders, df_out=True)
new_cols = mapper.fit_transform(df.copy())
df = pd.concat([df.drop(columns=["Sex", "Embarked"]), new_cols], axis="columns")

DataFrameMapper comes from the sklearn_pandas packages and accepts a list of tuples where the first item of the tupels are column names and the second item of the tuples are transformers. For our purposes, we use LabelEncoder(), but any other Transformer would be accepted by the interface as well (MinMaxScaler() StandardScaler(), FunctionTransfomer()).
In the last line, we join the encoded data with the rest of the data. Note that you can also write axis=1, but axis="columns" is way more readable and I encourage everybody to use the latter version.

df.shape
df.head()

df.nunique()

Anonymizing by sampling from the same distribution

Above I printed the number of unique values per column. We will go ahead and assume that everything with less than 20 unique values is a nominal or categorical variable, and everything with equal to or more than 20 unique values is a continuous one. Let's put the nominal/categorical variables in one list and the other ones in another list.

categorical = []
continuous = []

for c in list(df):
    col = df[c]
    nunique = col.nunique()
    if nunique < 20:
        categorical.append(c)
    else:
        continuous.append(c)

for c in list(df): iterates over all columns. Instead of list(df), one could also write df.columns.tolist(). I am still torn which one I like more.

Here is the core idea of this post: For every categorical variable, we will determine the frequencies of its unique values, and then create a discrete probability distribution with the same frequencies for each unique value. For every continuous variable, we will determine the best continuous distribution from a pre-defined list of distributions. How this is done is explained below. Once we have determined all the probability distribution (discrete and continuous), we sample from these distributions to create a new dataset.

Treatment of nominal/categorical variables

This is the simpler case and requires only 3 lines of code.

    for c in categorical:
        counts = df[c].value_counts()
        np.random.choice(list(counts.index), p=(counts/len(df)).values, size=5)

First, we determine how often a unique value occurs in a variable. This is the empirical probibility function. Then we use this probibility function and pipe it to np.random.choice() to create a new random variable that has the same probibility function.

Treatment of continuous variables

Luckily, there is a StackOverflow thread about a similar issue. The main solution is the following. For every continuous variable do:

Create a histogram using a pre-defined number of bins
Go though a list of continuous functions, and fit every function to that histogram. This fitting process also yields the parameters for the function.
The one function that has the smallest error (the smallest residual sum of squares), between itself and the histogram is the one we will use for approximating the continuous variable.

The author of this solution put everything neatly into two functions. I created a third one and put everything in a file called utils.py. We will use the functions from utils.py in our notebook.

best_distributions = []

for c in continuous:
    data = df[c]
    best_fit_name, best_fit_params = best_fit_distribution(data, 50)
    best_distributions.append((best_fit_name, best_fit_params))

# Result
best_distributions = [
    ('fisk', (11.744665309421649, -66.15529969956657, 94.73575225186589)),
    ('halfcauchy', (-5.537941926133496e-09, 17.86796415175786))]

The best distribution for Age is fisk and the best distribution for Fare is halfcauchy. Let's look at that result.

plot_result(df, continuous, best_distributions)

Not too bad, I'd say.

Putting the codes into one function

def generate_like_df(df, categorical_cols, continuous_cols, best_distributions, n, seed=0):
    np.random.seed(seed)
    d = {}

    for c in categorical_cols:
        counts = df[c].value_counts()
        d[c] = np.random.choice(list(counts.index), p=(counts/len(df)).values, size=n)

    for c, bd in zip(continuous_cols, best_distributions):
        dist = getattr(scipy.stats, bd[0])
        d[c] = dist.rvs(size=n, *bd[1])

    return pd.DataFrame(d, columns=categorical_cols+continuous_cols)

Now we have a function, that we can use to create, say, 100 new observations.

gendf = generate_like_df(df, categorical, continuous, best_distributions, n=100)
gendf.shape
gendf.head()

As a post processing step, one can also round the continuous variables. I chose not to do that. What I did do however, was I deleted all the column names, as this might also leak some infomation about the dataset. I simply replaced them with 0,1,2,... .

gendf.columns = list(range(gendf.shape[1]))

Finally, everything is saved to disk.

gendf.to_csv("output.csv", index_label="id")

Final Remarks

One drawback of this approach is that all the interactions between the variables are lost. For example, let's say in the original dataset, women (Sex=1) had a higher chance of surviving (Survived=1), then man (Sex=0). In the generated dataset, this relationship is no longer exsistent. Any other relationship between the variables that might have existed, are lost as well.

I hope you find this blog post helpful and would like to hear your thoughts and comments. All the codes shown here are also available on github.

Write a simple but impactful script

Florian Rohrer — Thu, 24 May 2018 21:29:26 +0000

Basic challenge (simple): Write a script that produces all possible 4-digit numbers (0000...9999), then put them into a random order and save the output into a text file. Make sure you include leading zeroes where necessary and make also sure that each number is on a separate line in the text file.

Bonus challenge (for daring programmers only): Do the basic challenge and then do the following: Call the generated file leaked_pins.txt, and email it to one of your co-workers. In the description of the email, say that all the PIN numbers have been leaked, and they should be cautious and check, whether their PIN is contained in the file or not.

Share your code below. If any of you tries the bonus challenge, I am happy to hear any stories :)

Sharing your Advent of Code Results

Florian Rohrer — Sat, 09 Dec 2017 10:13:31 +0000

Hi, it has been over a week of coding, so I wanted to ask if anyone would like to share their Advent Of Code results? If so, post a link to your public Github repo in the comments. Here is my Github Repo.

Also, to spark the competition even further, I've created a private leaderbord, that anybody interested can join. Go to here and enter the code 28346-d8f8d749. Let's see how many people we get on board.

Best, Flo

Replacing Function Calls In Python

Florian Rohrer — Sat, 25 Nov 2017 17:33:18 +0000

I recently came across a handy function in Python that I would like to share with you. Consider the following piece of code:

# This code is not modifiable

# a.py
from scipy.spatial.distance import cdist

def test_function():
    coords = [(0,0), (3,4)]
    y = cdist(coords, coords, metric="euclidean")
    print(y)

This function uses an external distance function called cdist() and calculates the euclidean distances of some (hardcoded) list of points. The resulting distance matrix between every pair of points is printed to the console. Up until now you used this function in your code, and everything was fine.

# Your code = modifiable
from a import test_function

test_function()

# Prints
# [[ 0.  5.]
#  [ 5.  0.]]

However, you decide that from now on you don't want to use the euclidean distance anymore. You want to use some other function instead of cdist(). There is a problem: You cannot modify the above code. Your collegue wrote this above code and he is out of town. But you still have to use this function. A complete rewrite is also not possible, because you think the function is way too complex.

The solution: mock.patch()

# Your new code 
from a import test_function
from unittest import mock

def rogue_cdist(x1, x2, metric):
    print("rogue_cdist called")

with mock.patch('a.cdist', wraps=rogue_cdist):
    test_function()

# Prints
# rougue_cdist called
# None

mock.patch() works by temporarily changing the object that a name points to with another one. You can use mock.patch() with a context manager as shown above. In the above example, your new rogue_cdist() function gets called, and now you are able to fill it with whatever distance metric you like.

I think this is very clever and clean and allows you to do modifications on existing code, without the need for the infamous copy-pasterino coding style.

Challenge: Write your worst program

Florian Rohrer — Mon, 30 Oct 2017 18:46:40 +0000

Hi everyone,
As you may know, there is no way in JavaScript to determine whether or not a number is a multiple of another number, so I wrote this function.

function isMultiple(big, small){
    return (big/small + "").indexOf('.') < 0;
}

Now you can use this in your code as well. You're welcome. Got any other tips for me? I am looking for functions, that can ease my programming life. Any programming language is appreciated.

EDIT: Some quick edit, because there seems to be confusion. This introductory text is supposed to be sarcastic. I know there is a % operator. I know this function does not make sense and that it is super inefficient. The challenge is, that you should provide pieces of code that are (on purpose or not) weired or way too complicated or inefficient or useless.

When can you say you know a language?

Florian Rohrer — Sat, 21 Oct 2017 15:12:36 +0000

When people ask me how many programming languages I know, I am usually hesitant and say something like "Many." and then tell them the projects I have worked on and the languages they are written in. My CV lists eight languages that I am familiar with. But at what point can you say you know a programming language?

If somebody says they know C, how much can you trust this statement? C has so many pelicularities and as soon as you start to dig into something a little deeper, things get complicated very fast (link1, link2, link3, link4, link5).

If somebody says they know Java, for me, this raises more questions than it answers. Have they built a desktop GUI, developed mobile apps? Have they worked with threads, developed web applications, distributed applications? Have they used reflection or aspect-oriented programming? These are all things that can be done in Java.

Does knowing a language mean I can read code that others have written in that language. If I am nasty, and put a little time and effort into it, I am sure you can't read the code I produce or get a totally wrong impression of what it does (link). Does knowing mean I know how to get my programs past the compiler?. If this is the case, I'd say you can learn any language on a lazy sunday afternoon and put it in your CV.

You might say that years of experience matters the most. But what does years of experience mean exacty? If you say you have "2 years" of Python experience, did you use Python literally 730 days in a row? Or did you use it once every week, when you needed a quick script, to delete some files? Or did you build a website (2 years ago) using a Python framework where you copied most of the code from the tutorials?

When can you say you know a language? Or are we asking the wrong question?

Challenge: Write a program that never stops

Florian Rohrer — Thu, 19 Oct 2017 10:14:42 +0000

Write an endless loop, write some confusing gotos...

The rules are simple: Write a program that never stops. Language: Any. Try to keep it short and confusing :D

Here is an example:

for(i=0; i<10; i++){
    for(j=0; i<10; j++){
        System.out.println("Blubb.");
    }
}

Post your funny, creative solutions below.

Only code goes into a repository, right?

Florian Rohrer — Thu, 21 Sep 2017 09:22:05 +0000

I was wondering how you deal with API keys, config files, precompiled binaries, test data or other environment specific data or data that is too large? Also, those Word documents, PDFs and Emails, that should be kept too.

If it is a small project, the easiest would be to commit almost everything to the repository, and keep some of my files in other places like Dropbox or Google Drive or just simply save them on the local hard drive. But this does not scale to larger projects or projects where more people are involved.

There has to be a better way. Is there? How do you manage your files?

4 Useful Things in Python

Florian Rohrer — Mon, 18 Sep 2017 17:29:51 +0000

Just some more things that I thought of. Since my last post received some attention, I thought I write a little more about things that I came accross in Python. Coming originally from Java, most of those things came as a surprise to me.

Tilde operator

# You can index lists from the front
a = ['a', 'b', 'c', 'd', 'e']

print(a[0]) # 'a'
print(a[1]) # 'b'

# You can also index them from the back
# starting with -1 and going backwards
print(a[-1]) # 'e'
print(a[-2]) # 'd'

# The tilde operator when applied to a number `x` calculates `-x - 1`, 
# which can be useful for list indexing 
#
# x  ~x  
# 0  -1
# 1  -2
# 2  -3
# 3  -4 
# 4  -5 

print(a[~0]) # 'e' = 0th element from the back
print(a[~1]) # 'd' = 1st from the back

SO link

[0] and 0

# Consider this function
def foo(data):
    """Prints the status of my list."""
    if data:              # Don't do this
        print("The list contains some items.")
    else:
        print("We got an empty list.")

foo([]) # ok -> "We got an empty list."
foo([1,2,3,4]) # ok -> "The list contains some items."

# Empty lists are treated as false in Python.
# However, False and 0 are also false, which can lead to some 
# unexpected behavior if your client, does not know that you 
# expect him to pass a list.

foo([0])   # still ok -> "The list contains some items."
foo(0)     # NOT ok -> "We got an empty list."
foo(False) # NOT ok -> "We got an empty list."

def foo(data):
    if len(data) > 0:     # Safer version
        print("The list contains some items.")
    else:
        print("We got an empty list.")

Default values

Default values in Python should not be mutable, such as a list or a dictionary. Default values are evaluated when the def statement they belong to is executed. That is, they are evaluated only once.

# Don't do this
def foo(bar=[]): 
    bar.append(1)
    return bar

foo() # returns [1] --> ok
foo() # returns [1,1] --> not ok = the same list as before
foo() # returns [1,1,1] --> also not ok = again, the same list

# Do this instead
def foo(bar=None):
    if bar is None:
        bar=[]
    ...

A lot of ways to call the same function

def draw_line(length, type="solid", color="black"):
    print(length, type, color)

draw_line(5)                                # 1) 5 solid black
draw_line(length=5)                         # 2) 5 solid black
draw_line(5, "dashed")                      # 3) 5 dashed black
draw_line(5, "dashed", "green")             # 4) 5 dashed green
draw_line(5, type="dashed", color="green")  # 5) 5 dashed green

# Can we take this a step further? Definitely.
draw_line(type="dashed", length=5)          # 6) 5 dashed black

args = [5] # could also be a tuple (5,)
draw_line(*args)                            # 7) 5 solid black

args = {"length": 5, "color": green} 
draw_line(**args)                           # 8) 5 solid green

Some explaination

More on functions

# Pre-fill functions with the arguments you use often

def draw_line(length, type="solid", color="black"):
    print(length, type, color)

from functools import partial

draw_green_line = partial(draw_line, color="green")
draw_green_line(5)                          # 5 solid green

draw_unit_line = partial(draw_line, length=1)
draw_unit_line()                            # 1 solid black

# If you want to use keyword arguments but you do not want
# to provide default values, you can do that as well

def draw_line(*, length, type, color): # all three are required
    ...

# Allow some positional arguments, but forbid 3) and 4)
def draw_line(length, *, type="solid", color="black"): # only length is required
    ...

Any more ideas on how to call a function? Know any more pythonic coding things, that you found helpful in the past? Comment below.

Relevant mystery link

What is the most confusing thing to you in Python?

Florian Rohrer — Tue, 12 Sep 2017 21:12:26 +0000

Just some traps and pitfalls that took me a while to get used to in Python.

List methods vs. built-in methods

# sort(), sorted(), reverse(), reversed()
# sort() modifies the existing list and is a function of the list object
# sorted() returns a new list and is a built-in function

a = [5,2,8,1,9,3]
sorted(a) # ok -> [1, 2, 3, 5, 8, 9]
a.sort()  # ok -> [1, 2, 3, 5, 8, 9]
a.sorted() # ERROR

# however, min() and max() are only built-in functions
a.min() # ERROR
min(a) # ok -> 1

# unless you work with numpy, where min() and max() do exist :S
import numpy as np
n = np.array([5,2,4,9,1,7])
n.min() # ok -> 1

string.join()

# I have to call .join() on a string to concatenate a list = confusing.

# concatenating a list of strings
", ".join(["apple", "banana", "cherry"]) # 'apple, banana, cherry'

# even uglier, when list is not a string
", ".join([str(s) for s in [1,2,3,4]])   # '1, 2, 3, 4'

List of lists


# The wrong way
a = [[]] * 5
# [[], [], [], [], []]
a[0].append(1)
# [[1], [1], [1], [1], [1]]

# The right way
a = [[] for _ in range(5)]

'is' and '==' on ints

# Internally some int objects are cached. When you compare 2 ints, you should use ==. 
# However, using 'is' can also work in some cases.

x = 2000
y = 2000
x is y # False -> good, because this should not be allowed anyways.

x = 5
y = 5
x is y # True -> But this works....
# See also
# https://stackoverflow.com/questions/306313/is-operator-behaves-unexpectedly-with-integers

Did you find any interesting or funny things in Python, that confused you the first time you saw them? Share them in the comments below.