DEV Community: Jürgen Hermann

SVG Thumbnails in Windows 11 with Microsoft PowerToys

Jürgen Hermann — Sun, 27 Aug 2023 21:37:49 +0000

Install the PowerToys from the Microsoft Store so you get the Explorer extensions as shown in the video below.

For avid Inkscape users, this is a great addition and gets you the same level of convenience any Linux system offers by default. 😏 🐧

https://youtu.be/ZAGDNwk7Bd8?si=BCxTVhc04yNb8HD8

🗣 Communication Hack – Use Labelled Links

Jürgen Hermann — Tue, 31 Mar 2020 09:09:08 +0000

Do your readers a favor and use the “Copy as Markdown for Chrome & Firefox” add-on to post web links.

When you post links to some information bit on the web or your intranet, using raw technical links typically reduces information content. For example, a link featuring just a post number (…/post/0815) is purely technical, and takes up a lot of space while having almost zero information content – it just tells your audience the target site via the domain, and that is all.

🌐 🚑 Copy as Markdown for Chrome & Firefox to the rescue.

The above link was created like this:

Using that context menu item copies the link including its label (or the current selection if there is one, like shown above) as markdown text into your clipboard. Markdown is a universal format that can be pasted into lots of channels: Rocket.Chat and other web chats, blog sites like this one, many wikis (like Atlassian Confluence), etc.

In Confluence, the trick to know is the “Insert Markup” menu item in the editor (Ctrl-Shift-D), enabling you to paste the markdown text almost directly into any page – just make sure the format drop-down above the text area says markdown and not Confluence wiki markup.

So do your audience a favor, install that add-on, and in future provide them with readable links that allow them to judge whether they need to click-through on that link.

Software Development is the Metamorphosis of Caffeine into Code

Jürgen Hermann — Sun, 29 Mar 2020 20:58:39 +0000

A list of my personal coffee gear – a long answer to What type of coffee you like?

Being able to brew a decent cuppa joe at home is more essential than ever, in times without access to my company's professional WMF coffee machines – about 7.5k€ if you buy them, a little out of my price range, and see the note on cleaning further below. 🏡 🍵 😋

Let's start with a simple list of my gear, including current prices (from amazon.de).

Product	Price
Aerobie AeroPress A80	29€
Accessory: Fellow Prismo	29€
Kitchen Timer	14€
Digital Scale	14€
Kettle: Uarter 2000W	42€
Coffee Grinder: Graef CM 800	111€

The mentioned prices should give you a hint whether you want to go down this specific rabbit hole. Keep in mind that people spend a whole lot of money on their hobbies, or coffee-2-go for that matter – so just regard drinking a decent cup as your hobby, then it's all normal and not snobbish at all. 😀

The Coffee

Finding the brand of coffee you like is all up to you – taste is subjective by nature, and that also includes milk and sweeteners.

However, there are a few guiding principles: get it as fresh(ly roasted) as possible, get whole beans, grind them right before brewing your cup, and get 100% Arabica only (cheap coffee is of the Robusta variant, or a blend). Spoiled coffee beans are uneven in shape and color, and are brittle with a lot of crumbling – avoid. One kg of non-specialty coffee beans will go for around 15€ in Germany, and a little more for fair-trade brands.

The Coffee Maker

The AeroPress is hailed by many as the way for brewing single cups of coffee. It is certainly cheaper and way more sanitary than the usual fully automatic coffee machines for end-consumers, which normally develop a nice population of mildew and other gunk in their insides, unless you religiously clean them every day. For the AeroPress, you need 15 seconds of rinsing under running water.

I evolved to the AeroPress via several simple electric coffee makers, an espresso machine, a fully automatic coffee maker, back to the simple things with various french presses, to finally settle on what I use now.

As for dosage, I typically go for 25 g coffee per cup of about 400 ml beverage. With the AeroPress, you get a pretty concentrated brew, but adding more hot water after filling the cup gets you to your preferred strength. And there's a lot of other variables involved: dark or light roast, grind size, water temperature, extraction time. You have to experiment in the beginning to get to your personal favorite recipe.

Because volumetric measurement definitely torpedoes repeatability of recipes, and extraction time is quite important, you also need these two things (possibly also as scales with an integrated timer):

You'll also find plenty of videos on the net showing you how to use an AeroPress, including all the variants and special recipes people came up with.

The Fellow Prismo is an accessory to replace the original filter holder, featuring a pressure-sensitive valve and including a stainless steel filter. Using it you can make espresso-style coffee with the AeroPress, but I got mine because it is a way to use the so-called ‘inverted’ style of AeroPress recipes, without the involved hazard of juggling hot brown water around.

The Grinder

Using a good grinder is just as essential as using good coffee. You can ruin great coffee with a bad grinder, e.g. one that produces particle sizes from dust to grainy, all at the same time. And no grinder on this world can make stale or spoiled coffee beans any good.

I have the CM 80 as shown above, but the available model is the CM 800 featured in the list at the top.

If you get no workout otherwise, a very good manual burr grinder is also an option. 💪

The Kettle

I replaced my plain old steel kettle by a glass one with temperature control (in 5°C steps) and a few other gimmicks. I really like the glass, especially when filling it – no narrow gauges you can only see from one specific angle, or similar things.

Closing Thoughts

I never was a friend of coffee shops and their inherent waste of resources and, well, money. I rather invest that money and some time to drink my first coffee at home, or take it with me in a steel vacuum bottle or cup. Working for a company offering great free coffee helps too.

I can also recommend watching James Hoffmann on YouTube. Yes, he is a “coffee professional” (i.e. a barista by trade), but his tips are still quite grounded and useful for mere mortals, like a recent test of water filters. He's also a funny guy, watch his tests of various coffee drinks for a good laugh.

Photo credits: Adafruit Industries [CC BY-NC-SA 2.0]

🌠 Listing My GitHub Stars

Jürgen Hermann — Sun, 29 Mar 2020 14:01:41 +0000

starred is a Python tool to create a repository that lists all your GitHub stars by topic.

Having a list of your stars organized by topic helps you as well as others to explore that database of curated know-how from another angle than what GitHub's web interface provides. On the “Stars” tab of your profile page, you can filter by language / technology or a keyword, and sort the list by history, popularity, or activity.

The starred CLI tool gets all your stars via the GitHub API and writes them to a README file that is then committed into a dedicated repository in your account.

The end result looks like this:

This also makes it easy to search for repositories and their descriptions, by using the in-page search of your bowser (Ctrl-F).

While we're on the topic of searching for keywords in your stars, the oh-my-stars tool does just that.

But that has to wait for another post…

Image credits: John Vermette

Packaging Python Applications for Debian

Jürgen Hermann — Sat, 28 Mar 2020 05:00:00 +0000

Easily deploy any Python application in form of an ‘omnibus’ Debian packaxge, i.e. one that contains all the application's dependencies, just like in a Java WAR. A basic understanding of Debian packaging, the Linux command prompt, and Python tooling is assumed.

Introduction

In this article, I'll show how to use dh-virtualenv to create self-contained Debian packages to deploy a Python application. The resulting package is very similar to a executable JAR that you can start via java -jar, in that it contains all the moving parts except Python itself, without influencing or being influenced by version requirements of other applications. This also frees you from being restricted to the dependencies and their versions found on your target platforms, and makes porting to several different target environments easier.

The advantage of using a Debian package for deployment as opposed to the native Python tool chain is that you are less dependent on typical development tools and services, i.e. to deploy to QA or production environments you need neither Internet access nor any compiler suites (for extension packages). To achieve the same with direct use of virtualenv and pip, you'd need to have an in-house PyPI repository accessible from production networks, and also release any extension packages as wheels pre-built for the target platform. Removing and updating an application is also much easier with Debian packages.

To use dh-virtualenv, you just have to extend your existing application project with a debian subdirectory – project meta-data like pip requirements and so on will be leveraged to build the final package, i.e. common tasks are delegated to the standard Python eco-system.

Note that just like with any other form of omnibus packaging, you take over the responsibility to release security updates of the contained dependencies in a timely manner.

How does it work?

dh-virtualenv is a debhelper plugin that extends the normal Debian tool chain for package building with the ability to create a Python virtualenv (an isolated Python environment), and then wrap that into the final Debian package.

Depending on the details of the application, you often also have to provide some kind of configuration of the software itself, and possibly some means to run it as a service. This can be done in several ways:

add a debian/«pkg».install descriptor to add configuration files to the Debian package.
provide a Puppet recipe or Ansible playbook that deploys the package and integrates it into the system.
embed (default) configuration into the application's Python package (via the include_package_data option of setuptools).

All those can be combined, e.g. provide defaults via Python package data, and then add external configuration that only provides values specific to the concrete host installation.

A real-world example is the devpi supervisor ERB template that serves both the purpose of passing configuration to the application process (via command line options), and also starting and controlling that process (i.e. handle demonization and automatic startup on boot).

Installing the build tools

Unsurprisingly, you need to install dh-virtualenv to use it. Since it is architecture independant, you can choose to use a recent release offered by Debian sid whatever your build platform is.

If this is your first time to build a Debian package, you also need to add the basic tools for that:

sudo apt-get install build-essential debhelper devscripts equivs

Finally, to take advantage of the available template for easily adding an inital debian directory, install the cookiecutter tool. Note that you can opt to build packages in a Docker container instead, with only Docker as a requirement on your build host.

Packaging an example project

To add the necessary debian directory with minimal effort, you can use the dh-virtualenv-mold cookiecutter. The following commands basically repeat what the integration test script of that project does, namely instantiate a Python project and then add debianization on top of it.

To provide common defaults to cookiecutter, it makes sense to have a ~/.cookiecutterrc file similar to the one I use.

Let's first create a sample project:

mkdir -p ~/tmp/dh-venv-blog
cd ~/tmp/dh-venv-blog
cookiecutter --no-input \
    "https://github.com/borntyping/cookiecutter-pypackage-minimal.git"
cd cookiecutter_pypackage_minimal/
python3 setup.py build

You can of course also use one of your own, then just check that out instead. Next, we add the debian directory:

cookiecutter --no-input \
    "https://github.com/Springerle/dh-virtualenv-mold.git"
dch -r "" # insert proper date & distro

Using --no-input causes the template's defaults to be accepted – it avoids answering all the template's prompts. After all, this is just a demo not requiring sensible inputs. Take the time to have a look at what's in the debian directory.

You're now able to build the package and if that succeeds, print the contained meta data:

dpkg-buildpackage -uc -us -b
dpkg-deb -I ../pyvenv-foobar_*.deb

The last command should show you something like this:

Package: pyvenv-foobar
Version: 0.1.0
Architecture: amd64
Maintainer: Jürgen Hermann <jh@web.de>
Installed-Size: 12877
Pre-Depends: dpkg (>= 1.16.1), python3, python3-venv
Section: contrib/python
Priority: extra
Homepage: https://github.com/jschmoe/foobar
Description: A Python package and its dependencies packaged up 
    as DEB in an isolated virtualenv.

Finally, install the new package via dpkg -i, or upload it to a repository and use it from there with apt.

Real-world examples

These are examples of dh-virtualenv packaging for non-trivial applications:

The last two show how to integrate Python web applications into systemd, instead of using supervisor like the devpi example. The jupyterhub one also demonstrates the integration of a Python project with server-side Javascript (in a NodeJS environment).

Release of rudiments v0.4.0

Jürgen Hermann — Fri, 27 Mar 2020 20:59:04 +0000

New release of my Python package with fundamental elements for any Python project, like configuration handling.

rudiments is a Python package that collects ‘miscellaneous’ functionality that is unspecific in nature and shared among many projects. The package provides runtime support for my Springerle cookiecutters, which enable you to quickly create new Python projects that work out of the box.

The new release 0.4.0 fixes some minor problems regarding newer Click versions, and path handling under Windows. It also drops Python 2.7 support, so Python 3.5 or higher is required now.

See the rudiments documentation for details on using the contained features.

Image credits: annola on Flickr

Embedding Graphs Into Your Sphinx Documents

Jürgen Hermann — Wed, 25 Mar 2020 05:00:00 +0000

Embed GraphViz ‘dot language’ graphs into your documentation, and hot-link the nodes to any HTTP resource.

The sphinx.ext.graphviz extension allows you to directly embed GraphViz ‘dot language’ graphs into your document files. They are then rendered to PNG or SVG images, which get added to your generated HTML documentation. Using SVG allows you to hot-link your nodes to any HTTP resource.

Before use, you have to activate the extension with just a few changes to your docs/conf.py

extensions = [
    # …
    'sphinx.ext.graphviz',
]

# …

# -- GraphViz configuration ----------------------------------
graphviz_output_format = 'svg'

This is an example for what you can then add to your documentation:

As long as the nodes have a href attribute, the SVG rendering contains them and thus node labels become clickable hyperlinks.

And here's the related markup that needs to be added to one of your .rst files:

.. graphviz::
    :name: sphinx.ext.graphviz
    :caption: Sphinx and GraphViz Data Flow
    :alt: How Sphinx and GraphViz Render the Final Document
    :align: center

     digraph "sphinx-ext-graphviz" {
         size="6,4";
         rankdir="LR";
         graph [fontname="Verdana", fontsize="12"];
         node [fontname="Verdana", fontsize="12"];
         edge [fontname="Sans", fontsize="9"];

         sphinx [label="Sphinx", shape="component",
                 href="https://www.sphinx-doc.org/",
                 target="_blank"];
         dot [label="GraphViz", shape="component",
              href="https://www.graphviz.org/",
              target="_blank"];
         docs [label="Docs (.rst)", shape="folder",
               fillcolor=green, style=filled];
         svg_file [label="SVG Image", shape="note", fontcolor=white,
                   fillcolor="#3333ff", style=filled];
         html_files [label="HTML Files", shape="folder",
                     fillcolor=yellow, style=filled];

         docs -> sphinx [label=" parse "];
         sphinx -> dot [label=" call ", style=dashed, arrowhead=none];
         dot -> svg_file [label=" draw "];
         sphinx -> html_files [label=" render "];
         svg_file -> html_files [style=dashed];
     }

For all this to work, you need the GraphViz suite of tools installed on the machine that renders the documentation.

A JupyterHub Showcase: DevOps Intelligence

Jürgen Hermann — Sun, 22 Mar 2020 06:00:00 +0000

Originally published in 2019 as DevOps Intelligence in the Jupyter Blog.

DevOps Intelligence turns data from software development and delivery processes into actionable insight, just like BI does for the business side. Jupyter is the ideal instrument for that, with its combination of powerful coding environments and a user interface facilitating experimentation with ultra-short feedback cycles.

A Jupyter-based setup supports risk analysis and decision making within development and operations processes – typical business intelligence / data science procedures can be applied to the ‘business of making and running software’. The idea is to create feedback loops, and facilitate human decision making by automatically providing reliable input in form of up-to-date facts. After all development is our business — so let's have KPIs for developing, releasing, and operating software.

Typical Use-Cases

Here are some obvious application areas where data analysis can be helpful on the technical side.

Migration processes of all kinds (current state, progress tracking, achievement of objectives)
Inventory reporting for increased transparency and support of operational decisions
Automate internal reporting processes to free up scarce assets and human expertise

Platform Architecture

A simple JupyterHub setup can enable you to do analysis on your already available but under-used and hardly understood data, without any great investment of effort or capital. By adding a single JupyterHub host, you can use the built-in Python3 kernel to access existing internal data sources.

The following diagram shows what role JupyterHub can play in an existing environment.

To make such a deployment easy, the 1and1/debianized-jupyterhub project provides a JupyterHub service including a fully equipped Python3 kernel as a single Debian package – only Python3, NodeJS, and Chromium packages must be installed in addition to the jupyterhub one. If you raised an eyebrow on Chromium being in that list, it's used by JavaScript-based visualization frameworks to render PNG images.

Including a NginX-powered SSL off-loader, the complete setup can be done in under an hour.

Use-Case: Migration Reporting

At the time of this writing (early 2019), a widespread challenge is migration from Oracle Java to other vendors, and also to start migration from Java 8 to newer versions (Java 11). If you do that at scale across many machines and teams, you definitely need some kind of governance, and constant feedback on the current status and the rate of progress.

What follows is an excerpt from a productive notebook, with anonymized data about AdoptOpenJDK deployments. That data was originally retrieved from a system called “Patch Management Reporting”, which collects information about installed packages for all hosts in the data center. We're in the yellow “Data Sources” box of the above figure here.

First off, we read the data and show the value sets of categorical columns, plus a sample.

import numpy as np
import pandas as pd

raw_data = pd.read_csv("../_data/cmdb-aoj.csv", sep=',')
print('♯ of Records: {}\n'.format(len(raw_data)))

for name in raw_data.columns[1:]:
    if not name.startswith('Last '):
        print(name, '=', list(sorted(set(raw_data[name].fillna('')))))

print(); print(raw_data.head(3).transpose())

♯ of Records: 104

Distribution = ['Debian 8.10', 'Debian 8.11', 'Debian 8.6', 'Debian 8.9',
                'Debian 9.6', 'Debian 9.7', 'Debian 9.8']
Architecture = ['amd64']
Environment = ['--', 'DEV', 'LIVE', 'QA']
Team = ['Team Blue', 'Team Green', 'Team Red', 'Team Yellow']
Installed version = ['11.0.2.9-83(amd64)', '11.0.2.9-85(amd64)', '8.202.b08-66(amd64)',
                     '8.202.b08-83(amd64)', '8.202.b08-85(amd64)']

                                    0                   1                   2
CMDB_Id                     108380195           298205230           220678839
Distribution              Debian 8.11          Debian 9.6         Debian 8.11
Architecture                    amd64               amd64               amd64
Environment                       DEV                  --                 DEV
Team                         Team Red            Team Red            Team Red
Last seen            2019-03-18 06:42    2019-03-18 06:42    2019-03-18 06:42
Last modified        2019-03-18 06:42    2019-03-18 06:42    2019-03-18 06:42
Installed version  11.0.2.9-83(amd64)  11.0.2.9-83(amd64)  11.0.2.9-83(amd64)

Next comes the usual data cleanup. The Distribution column is a bit diverse, and not everyone has Debian codenames and associated major versions memorized. The map_distro function fixes that.

def map_distro(name):
    """Helper to create canonical OS names."""
    return (name.split('.', 1)[0]
        .replace('Debian 7', 'wheezy')
        .replace('Debian 8', 'jessie')
        .replace('Debian 9', 'stretch')
        .replace('Debian 10', 'buster')
        .replace('squeeze', 'Squeeze [6]')
        .replace('wheezy', 'Wheezy [7]')
        .replace('jessie', 'Jessie [8]')
        .replace('stretch', 'Stretch [9]')
        .replace('buster', 'Buster [10]')
    )

Together with other cleanup steps, the mapper function is applied in a dfply pipeline. The result can be controlled by showing a sample of data points with unique version numbers.

from dfply import *

cleaned = (raw_data
    >> mutate(Version=X['Installed version'].str.split('[()]', 1, expand=True)[0])
    >> mutate(Environment=X.Environment
              .fillna('--').str.replace('--', 'N/A').str.upper())
    >> mutate(Distribution=X.Distribution.apply(map_distro))
    >> drop(X.CMDB_Id, X['Last seen'], X['Last modified'], X['Installed version'])
)

print((cleaned >> distinct(X.Version)).transpose())

                       0            13            14            62  \
Distribution   Jessie [8]  Stretch [9]    Jessie [8]   Stretch [9]
Architecture        amd64        amd64         amd64         amd64
Environment           DEV          N/A           DEV           DEV
Team             Team Red    Team Blue      Team Red     Team Blue
Version       11.0.2.9-83  11.0.2.9-85  8.202.b08-83  8.202.b08-85

                        68
Distribution    Jessie [8]
Architecture         amd64
Environment            DEV
Team             Team Blue
Version       8.202.b08-66

Time to present the cleaned up data, starting with a table of teams and their number of installed packages. In the production notebook, an API of the corporate identity management is used to enrich the table with contact data of the team leads. Having the organizational data available also makes it possible to filter or aggregate the data by business units.

counts = cleaned.groupby(['Team']).size()
print(counts.reset_index(name='Count'))

          Team  Count
0    Team Blue     42
1   Team Green     16
2     Team Red     45
3  Team Yellow      1

To create a heatmap of how diverse a team's version spectrum is, we calculate percentages of versions per team.

percentage = cleaned.groupby(['Team', 'Version']).size().reset_index(name='Count')
percentage = percentage.assign(Percent=
    percentage.apply(lambda x: 100.0 * x[-1] / counts[x[0]], axis=1))
print(percentage.head(3))

        Team       Version  Count    Percent
0  Team Blue   11.0.2.9-85      1   2.380952
1  Team Blue  8.202.b08-66      1   2.380952
2  Team Blue  8.202.b08-85     40  95.238095

HoloViews makes creating the heatmap including a label overlay a breeze.

import holoviews as hv
hv.extension('bokeh')
publish = 1  # publishing or interactive mode?

heatmap = hv.HeatMap(percentage[['Version', 'Team', 'Percent', 'Count']]).opts(
    hv.opts.HeatMap(
        title='Version Distribution by Team',
        width=480, xrotation=25,
        zlim=(0, 100), cmap='kbc_r', clipping_colors=dict(NaN='#ffffe0'),
        colorbar=True, tools=['hover'], toolbar=None if publish else 'right',
    )
).sort()

label_dimension = hv.Dimension('Percent', value_format=lambda x: '%.1f' % x)
labels = hv.Labels(heatmap, vdims=label_dimension).opts(
    hv.opts.Labels(
        text_color='Percent',
        text_font_size='10pt',
        text_font_style='bold',
    )
)

chart = heatmap * labels

if publish:
    import time
    import phantomjs_bin
    from IPython.display import HTML, clear_output

    %env BOKEH_PHANTOMJS_PATH={phantomjs_bin.executable_path}
    chart_img = 'img/devops/aoj-heatmap.png'
    hv.save(chart, chart_img)
    chart = HTML('<img src="{}?{}"></img>'.format(chart_img, time.time()))
    clear_output()

chart

From the heatmap, you can easily glance whether a team uses predominantly one version, and how recent the used versions are.

Conclusion

Using a platform powered by Project Jupyter and a big chunk of the scientific Python stack lets you easily mold your data into the shape you need, and then choose from a wide range of visualization options to bring your message across.

👍 Image credits: Devops-toolchain

Using Python3's ‘venv’ with tox

Jürgen Hermann — Sat, 21 Mar 2020 22:27:28 +0000

Using the built-in Python module in favor of virtualenv to create your testing or project automation environments.

tox is a generic virtualenv management and test command line tool, especially useful for multi-environment testing. It has a plugin architecture, with plenty of both built-in and 3rd party extensions.

This post assumes you are already familiar with tox and have a working configuration for it. If not, check out its documentation.

In order to make tox use the built-in virtual environment venv of Python 3.3+, there is a plugin named tox-venv that switches from using virtualenv to venv whenever it is available.

tox-dev / tox-venv

Use Python 3 venvs for Python 3 test environments

Typically, venv is more robust when faced with ever-changing runtime environments and versions of related tooling (pip, setuptools, …).

To enable that plugin, add this to your tox.ini:

[tox]
requires = tox-venv

That merely triggers tox to check (on startup) that the plugin is installed. You still have to add it to your dev-requirements.txt or a similar file, so it gets installed together with tox. You can also install tox globally using dephell jail install tox tox-venv – see the first post in this series for details.

The end result is this (call tox -v to see those messages):

py38 create: …/.tox/py38
  …/.tox$ /usr/bin/python3.8 -m venv py38 >…/log/py38-0.log

And there you have it, no more virtualenv package needed. 🎉 🎊

Add level setting to dashboard

Jürgen Hermann — Sat, 21 Mar 2020 11:42:43 +0000

On the dashboard, in the meta data summary box after the "Manage" button, please include the level setting. Otherwise it is tedious to impossible to maintain them after the fact, or get an idea whether your level settings are consistent across posts.

Sort Order of Posts in a Series?

Jürgen Hermann — Sat, 21 Mar 2020 10:37:28 +0000

I'm a bit surprised about the (in a timeline) 2nd post Blogging About My Repositories to Guide Interest being the first in the generated list of posts. Any hints to fix this?

Maybe related to the original date of posts at the source? If that, it might be a worthwhile feature to allow explicit ordering (series: #«PRIO» «TITLE», with prio defaulting to the UNIX timestamp used currently), or else sort by order of publishing on dev.to instead of any sources.

Using R-style Data Pipelines in Notebooks

Jürgen Hermann — Tue, 17 Mar 2020 05:00:00 +0000

Mutate Pandas data frames simply and elegantly with data pipelines.

Overview

This post shows how mutating data frames can be written more elegantly (and thus understood more easily) by using data pipelines. R users know this concept from the dplyr package, and Python offers a similar one named dfply.

Setting the Stage

We start off with some global definitions…

import numpy as np
import pandas as pd

The sample data (about OS package deployments) is read into the raw_data dataframe from a CSV file.

raw_data = pd.read_csv("../assets/data/cmdb-packages.csv", sep=',')

print('♯ of Records: {}\n'.format(len(raw_data)))

for name in raw_data.columns[1:]:
    if not name.startswith('Last '):
        print(name, '=', list(sorted(set(raw_data[name].fillna('')))))

raw_data.head(3).transpose()

♯ of Records: 146

Distribution = ['Debian 8.11', 'Debian 8.6', 'Debian 8.9', 'jessie']
Architecture = ['amd64']
Environment = ['', 'Canary', 'DEV', 'LIVE', 'QA']
Team = ['Automation', 'Big Data', 'Email', 'Ops App1', 'Ops Linux', 'Persistence', 'Platform']
Installed version = ['41.15-2(amd64)', '42.28-2(amd64)', '42.44-1(amd64)', '45.11-1(amd64)', '48.33-1(amd64)']

	0	1	2
CMDB_Id	274656589	153062618	282201163
Distribution	jessie	jessie	jessie
Architecture	amd64	amd64	amd64
Environment	DEV	DEV	LIVE
Team	Ops App1	Ops App1	Ops App1
Last seen	2019-02-18 11:43	2019-02-18 11:56	2019-02-18 12:04
Last modified	2019-02-18 11:43	2019-02-18 11:56	2019-02-18 12:04
Installed version	42.28-2(amd64)	42.28-2(amd64)	48.33-1(amd64)

The map_distro() helper function is used in the following sections to clean up the Distribution column.

def map_distro(name):
    """Helper to create canonical OS names."""
    return (name.split('.', 1)[0]
        .replace('Debian 7', 'wheezy')
        .replace('Debian 8', 'jessie')
        .replace('Debian 9', 'stretch')
        .replace('Debian 10', 'buster')
        .replace('squeeze', 'Squeeze [6]')
        .replace('wheezy', 'Wheezy [7]')
        .replace('jessie', 'Jessie [8]')
        .replace('stretch', 'Stretch [9]')
        .replace('buster', 'Buster [10]')
    )

Data Cleaning With Pandas

This code cleans up the imported data using the Pandas API.

To get sensible version statistics, we split off the auxiliary information in the version column (anything after -), leaving just the upstream part of the version string. The environment classifier is also cleaned up a little, and distributions are mapped to a canonical set of names. Some unused columns are dropped.

Finally, a subset of unique version samples is selected.

data = raw_data
data = data.assign(Version=data['Installed version'].str.split('-', 1, expand=True)[0])
data = data.assign(Environment=data.Environment.fillna('UNDEFINED').str.upper())
data = data.assign(Distribution=data.Distribution.apply(map_distro))
data = data.drop(columns=['CMDB_Id', 'Last seen', 'Last modified', 'Installed version'])
data = data.drop_duplicates(subset='Version', keep='first')

data.transpose()

	0	2	26	45	62
Distribution	Jessie [8]	Jessie [8]	Jessie [8]	Jessie [8]	Jessie [8]
Architecture	amd64	amd64	amd64	amd64	amd64
Environment	DEV	LIVE	LIVE	UNDEFINED	DEV
Team	Ops App1	Ops App1	Platform	Email	Platform
Version	42.28	48.33	41.15	45.11	42.44

Data Cleaning With Pipelines

This does the exact same processing as the code above, but is arguably more readable and maintained more easily:

It has less boilerplate, and makes the use of pipelined processing transparent.
Each step clearly states what it does to the data.
When steps are copied into other pipelines, the X placeholder ensures you use the data of this pipeline (the code is more DRY).

from dfply import *

piped = (raw_data
    >> mutate(Version=X['Installed version'].str.split('-', 1, expand=True)[0])
    >> mutate(Environment=X.Environment.fillna('UNDEFINED').str.upper())
    >> mutate(Distribution=X.Distribution.apply(map_distro))
    >> drop(X.CMDB_Id, X['Last seen'], X['Last modified'], X['Installed version'])
    >> distinct(X.Version)
)

piped.transpose()

The result is identical to the pure Pandas code, as expected.

	0	2	26	45	62
Distribution	Jessie [8]	Jessie [8]	Jessie [8]	Jessie [8]	Jessie [8]
Architecture	amd64	amd64	amd64	amd64	amd64
Environment	DEV	LIVE	LIVE	UNDEFINED	DEV
Team	Ops App1	Ops App1	Platform	Email	Platform
Version	42.28	48.33	41.15	45.11	42.44

To learn more about dfply, read the dplyr-style Data Manipulation with Pipes in Python blog post, which has more examples.