DEV Community: Samuel Chan

My journey in data visualization, and how it took me to Altair

Samuel Chan — Thu, 27 Oct 2022 10:44:16 +0000

It is remarkable how far we've come in terms of python tooling for charting and visualization. I came into the python scientific computing ecosystem from R (user since 2015) and Matlab (user since 2013), and my experience has been largely pleasant. I wish I could say I've never looked back since, but the truth is that I have. For all the envy of Python's enormous ecosystem, scientific graphing in Matplotlib never felt as natural as it is with Matlab. This is a cruel irony as Matplotlib borrows heavily from the syntax of Matlab, a fact that isn't exactly subtle when you learn that the "Mat" in "Matplotlib" refers to Matlab and the library retains many of Matlab's plotting terminologies (i.e "markers").

My introduction to `ggplot2`

Attached above are some of the images I've created in R, with the trusty ggplot2 library.

I remember how I came across Hadley Wickham's ggplot2. I was relatively new to R and similar to Matlab, R has a rich feature set of built-in graphing utilities, catering to scientific computing users who needed to model complex simulations to statistical user who want a one-liner to produce a faceted plot (R users fondly call them "trellis plot"). Similar to Matlab, one didn't have to bother with importing external packages as R's core plotting library offers a lot out of the box.

Then I came across a challenge by Harvard's IQSS and I was rather intrigued. The challenge was about reproducing this plot on the Economist and the materials presented an introduction to ggplot2. I completed the challenge that weekend, and went ahead to reproduce a few more plots I found from the Economist as practice exercise, including this one.

ggplot2 is fantastic. It is ergonomic, natural, and intuitive. It is all of the above because it is an implementation of the grammar of graphics, which aims to bring a set of grammar to the art of the visualization.

ggplot2 was great, and I duly put my newfound skills to use by combining it with the shiny framework to produce visualization-heavy web dashboards like this one. When I taught data visualization, I only wanted to teach the ggplot system because it felt methodical, logical and highly organized.

The teaching experience never quite translate equally to the python universe. It wasn't just because ggplot2 is highly declarative -- the user is concerned only with "what" she wants to achieve, not "how" she wants it achieved. It wasn't just because ggplot2 adheres to a coherent system, known as the grammar of graphics. It wasn't just because ggplot2 is highly instructive -- it picks default values in ways that are seemingly minor but when compounded over many decisions, can mean the difference between an abhorrent mess and a convincing medium of message.

Altair

I know many Python developers, myself included, begrudgingly switch between the different ecosystems to get the best of what R and Python offer. Then I stumbled upon Altair in 2018, when Altair was in its early days (last few releases before the v2.0; current version is 3.x). I created my first web app with Altair and it is a project that I still maintain as it is actively used among our teaching staffs at Algoritma Data Science Education Center, a company I co-founded 6 years ago.

Altair is Jake Vanderplas's attempt to provide a Python wrapper over Vega-Lite / Vega.js (which in turn, is built on D3.js).

It adheres to the same grammar of graphics system. It is declarative. It is extensible and customizable.

On top of that, it does something that ggplot2 library couldn't: it emits JavaScript code, which means your graphics can be embedded in just about any HTML file. It runs in the browser and can support browser events like onMouseOver, onMouseOut, onClick etc, so charts created with Vega have that added interactivity and event binding that R plots couldn't.

This year, when Anaconda announced PyScript, I started a playlist that I initially named build with PyScript in which I build one web app per video, showcasing the different possibilities that come with it. In many of the interactive web dashboards throughout the PyScript series I have relied on the trusty Altair library:
- Visualizing MiBand runs
- Building an interactive data visualization dashboard

These videos demonstrates the utility of PyScript along with Altair to create user-friendly front-ends that are truly interactive and fun-to-use. Viewers love them, but I started to get comments that request for a separate series that cast the light on Altair. One that more centrally focus on the building blocks of Altair as a data visualization library.

With this, I spent the last two months conceptualizing and developing a video series that goes into the art and science of data visualization in Python, adopted from the data visualization curriculum developed by the University of Washington. The series will consist of 5 to 8 long-form videos, each focusing on a key aspect of data visualization and the accompanying mental models that go with it.

Data Visualization in Python (2022)

Here's part 1 of the video:
Data Visualization in Python: Altair 4.2 (altair-viz) Tutorial

Teaching visualization with Altair has been highly satisfying, and the data scientists where I work at are in agreement of its versatility; documenting my journey here so there is a point of reference and book-keeping for my future self has been equally refreshing.

Now, readers, if you'd love sharing with me, how do you decide on your data visualization / charting toolkit and how does it look like?

Interactive Visualization of my Mi Band ⌚️ runs w/ PyScript

Samuel Chan — Fri, 15 Jul 2022 13:24:18 +0000

I’ve heard this fantastic quote from a podcast that goes something alone the lines of

Covid, it’s safe to say, will be here temporarily and then goes away completely. But some of the damages it causes, will stay with us probably permanently, certainly with far longer consequences, long after covid has moved on.

It could have been Balaji Srinivasan, it could have been somebody else.

But it was the wake up call I needed. Being in this line of work as the rest of you, I am quite used to being glued to my chair working away for up to 10 hours a day. Then the pandemic struck.

I transported my work machines from office to home second week into the pandemic. This is promptly followed by 14-hour work days, and a complete obliteration of my health. I put on weight, and my already pitiful fitness level tanked to virtually non-existent.

Hearing that line, the thought that covid would take a few years off my lifespan sounds completely logical, and even inevitable.

So I started running. And my best friend and co-founder at work bought me Mi Band, a fitness tracker. "A form of encouragement" is how he calls it.

I kept at it for more than a year, and this week I thought to export the data and visualize my runs. The goal is to develop a responsive, mobile friendly dashboard that summarizes my running progress, milestones and personal best(s).

I chose to do it in PyScript and deployed the web app using GitHub pages. Since I have a video series on building apps with PyScript, I naturally recorded the whole development process and wanted to share this with my audience.

The tutorial features data wrangling and cleaning process using the pandas library, before delegating the visualization to altair, which is a wrapper over the popular vega-lite (vega) charting library. As with all videos in the PyScript tutorial series, we deploy it through GitHub Pages.

Here's the live dashboard:
onlyphantom.github.io/miband/

Here's all the code and data:
https://github.com/onlyphantom/miband

Here's each video in the PyScript tutorial series:

Part 1 PyScript Demo / Tutorial
Part 2 Deploying a PyScript App w/GitHub
Part 3 Build a PyScript interactive dashboard w/Altair
Part 4 Build a PyScript guestbook webapp, CRUD
Part 5 Building an interactive map w/ Folium & PyScript
Part 6 Dashboard to visualize Mi Fit runs with PyScript + Altair (⌚ -🏃)

So if also use a fitness tracker / wristband, and don't mind picking up pandas or altair for some personal dashboard-ing, follow along the tutorial! It's extremely satisfying to have a longer-term perspective on your run performances and I hope it helps, even if just a little bit, in helping you stay in shape.

Part 6 Dashboard to visualize Mi Fit runs with PyScript + Altair (⌚ -🏃)

See you on YouTube!

Todo + gamification with Taskwarrior & Taskquant

Samuel Chan — Wed, 06 Apr 2022 13:59:21 +0000

If you live and breathe in the terminal, you want as little context switching as possible. For many things (like web browsing and spreadsheet editing) a graphical user interface seems inevitable no matter how great the alternative is (i.e lynx browser).

For everything else, there is almost always a better option, one that allows you to stay within the terminal and gasp optionally works with VIM bindings. As I review my current work and processes, it strikes me that a task manager (todo app) is one such use-case.

A good todo app is stunning, aesthetically pleasing, and joyous to use.

But a great todo app should be almost the entirely opposite of a good todo app. It shouldn't even be joyous to use, lest you spend too much time in it. I want a todo app that is so minimal, so unobtrusive that it's barely there. If I find the todo app enjoyable, I'm spending too much time in it instead of doing the task.

Taskwarrior

Taskwarrior is a free, open source TUI that puts your TODO in the terminal and is as unobtrusive as to-do apps get.

Taskwarrior video introduction

It has built-in reporting features (task burndown.weekly), a tagging system (+meeting +report), a context switcher (task context office), project hierarchies (task project:Startup), a robust date system that blows anything, free or proprietary, out of the water, and a scriptable urgency formula for each task.

There are many other real-life examples of algorithms that are used to approximate qualities: the Apgar Score that assesses the health of newborns, Google's Page Rank that measures the importance of a web link, or credit ratings. These examples all approximate, but cannot perfectly represent, a complex concept.

Taskwarrior uses a polynomial expression to calculate urgency. This is a simple sum of terms, where each term represents some measurable quality of a task.

Custom Attributes

The best part of Taskwarrior is its support for user-defined attributes (UDA) beyond what it offers out of the box. No matter what productivity system (e.g GTD system) I choose, I can make it work the way I want to with a bit of customization.

One theme of my productivity system over the past many years is a way to quantitatively measure my productivity with a score attribute. Different apps may come and go, but to make it work for me, I need the score attribute as it's integral to how I manage my periodic reviews and longer-term reflections. It also pushes me to work on large projects since they come with bigger rewards (at least psychologically).

How do I add that to Taskwarrior? Simple. Add the following lines to the end of your .taskrc file:



uda.score.type=numeric
uda.score.label=Score 🏆 
urgency.uda.score.coefficient=2

Now all your task has an extra, optional score field (🏆)!

Productivity Scoresheet

One thing I often do in my periodic review is to collect data-points on my activities over the last period. This includes data from my work machines, my Mi Band, and Apple Health.

For example, I know how many hours and minutes I've spent on one specific app, or task, over the last calendar year.

For Taskwarrior, all that is left to do is to roll out a Python app that I can install and run. This package weighs less than 8kb (which is really nothing) and is available on PyPI. It installs under one second:



pip install taskquant

After which you can issue tq to get a productivity breakdown. Add the -w flag to have it grouped by week, in a rolling sum fashion:



tq -w 
+-------+-------+------------+
| Week# | Score | Cumulative |
+-------+-------+------------+
|  10   |  56   |     56     |
|  11   |  33   |     89     |
|  12   |  26   |    115     |
|  13   |  12   |    127     |
+-------+-------+------------+

If you want to hack away at your productivity system, the project is available on my GitHub:

TaskQuant

Bonus: Watch the whole process in action

I thought it'd be interesting to show how I build out the CLI tool so I've uploaded a recording of that on YouTube as well. If you're a data analyst from the pandas world, you may learn a thing or two about:

group by
aggregation
filling missing dates (padding)
tabular summary
colorful CLI
All without any dependencies except tasklib (the Taskwarrior libary itself). This brings the package down to 6.8kb of code, and with its 1 second installation time, is similarly unobtrusive. Gets you your productivity scores, and get out of the way. If you'd like to watch it, here are the links:
Building Taskquant ep.1
Building Taskquant, full playlist

What's next

We'll add terminal-based graphs and charts, and maybe more variety in terms of reporting format. No, there can never be too many ways to see how you're not doing enough. If you want to follow along the journey, drop by and 👋 on YouTube.

If you're feeling more driven than that, go ahead and install Taskwarrior and Taskquant and start knocking out some to-do! Collaborators welcomed!