DEV Community: Jeff Palmer

Experimental Visualizations of Large Open Source Projects

Jeff Palmer — Thu, 17 Jun 2021 22:08:02 +0000

Once I completed my interactive Git history, I started to think about a smaller visualization project that would allow me to experiment with new ways of visualizing development histories. I was looking through some photos of naturally occurring phenomena for inspiration, when I came across this image of particles moving in a cloud chamber:

“Trajectories in a Cloud Chamber” Image from Gordon Fraser/CERN (CC BY 4.0)

There was something about it that sparked an idea in my head: “What if developer changes across the history of a source repository could be represented like this?”

I started brainstorming possible transformations to the git source repository domain (author, date, number lines added/deleted, etc.) that might end up with a structure that was visually similar to that of particles in a cloud chamber. I considered simulating particles with forces over time, gravitational effects, etc. Nothing was off of the table.

I eventually decided to start with a basic mapping, to try to get a sense of how that would look. (I can never never really guess how complex visualizations will look without seeing something based on actual data.)

My basic mapping algorithm looked like this:

Create a point for every contributor, starting in the center of a two-dimensional space
For every contributor’s commit (in date order):
- Move that contributor’s point according to:
- The date of the commit = a direction around a radial timeline, potentially inverting the direction based on timezone offset
- The number of lines changed = distance to move
- Create a dot in this adjusted position

So basically every commit would “push” a contributor point around in 2d space.

My guess at the beginning was that I would need to use some type of curve interpolation to make things interesting. I was expecting to need Bézier curves to make things look gracefully organic.

When I rendered a first version (based on the Git VCS project history), this was the result:

That was a lot more interesting than I had expected!

And, as a bonus, it actually seemed to have some of the organic feeling that I associated with the cloud chamber images that got me started. I decided to put the other more complex mapping strategies to the side to see where this simple version would lead.

The first change that I made was to spread out the origin points for each contributor, based on the date of their first contribution. I decided to lay out these points on a circular timeline, because I’m a sucker for round visualizations.

I also realized that I needed some way of introducing color, but I wasn’t sure yet how to do that. I started sorting the contributors by number of lines changed, and then used that to apply a linear color scale, just to see what would happen. Surprisingly, it wasn’t terrible.

Okay, so at this point the visualizations were looking like something that might actually come out of a cloud chamber, so I decided to add some controls to allow me to tweak the various parameters that shaped the rendering.

Once I added these controls I realized that I needed to spend some time optimizing my code. The rendering was taking a very long time in-between changes, making iterative refinement a challenge.

I spent some time looking into what was going on and realized that the quick hacks that I had used to assemble the first version were not scaling to the larger repositories that I was actually working with. I had to do some rethinking of how the render positions were calculated, and I also moved to using HTML canvas to avoid creating hundreds of thousands of DOM elements during SVG rendering.

I then changed the color mapping to be based on the age of the commit, relative to “now,” so that newer commits were brighter and older commits faded away over the course of the project history, which worked well for animation (see below). With a little more tweaking and experimentation, I started to generate interesting results.

Here is the Kubernetes project history, with my standard render parameters:

Kubernetes (2015-2021) - Radial timeline (clockwise) - One rotation per year

This is the Git project repository (one of my favorites), with the entire project shown as a single timeline rotation. The long looping structure on the right side is the project maintainer, Junio C Hamano.

Git (2005-2021) - Radial timeline (clockwise) - One rotation for entire project duration

Here’s the TensorFlow project, but with the radial timeline replaced with a linear one:

TensorFlow (2015-2021) - Linear timeline (left to right)

By reducing how fast the angle changes based on commit date, you can smooth out the contributor curve which leads to an interesting effect, especially with a repository the size of the Linux kernel.

Linux (2005-2021) - Radial timeline (clockwise) - Flattened annual commit date curves

Finally, I thought it would be interesting to try to animate these, so that it would be possible to see the evolution of project histories over time. I had never created an animation from a canvas rendering before, but I figured that there must be tools out there that would make it easy.

It turns out that capturing an animation was more work than I expected. 😭

I started by further optimizing my canvas rendering in order to get silky smooth animations, but then I realized that I wasn’t able to capture 4k video at 60 fps while rendering in realtime. At that point I decided to create the videos frame-by-frame using the h264-mp4-encoder library that generates mp4 files directly from a sequence of frames. That actually worked really, really well.

You can see the final result of these efforts in the following video:

Thanks for reading! If you have any thoughts or feedback you’d like to share, please feel free to contact me on Twitter or via my blog.

Git: An Interactive Development History

Jeff Palmer — Tue, 18 May 2021 22:12:16 +0000

In April I set out to create an interactive visualization of the Git distributed version control system, and I’ve finally reached a point where it’s ready for others to see. Here’s a screenshot of the current version:

This became one of my larger projects, so I thought I’d share my experiences building it.

Motivation

Git is a distributed version control system that allows software developers to safely store historical changes and collaborate. Version control is a critical part of software development, and the introduction of Git in 2005 has had a major impact on how teams manage software version history.

I’ve been around long enough to have developed in the world before Git, and have managed large source repository migrations to it from other tools. While it has its share of warts, like all tools, I’ve always found its abilities to far outmatch any difficult situations I’ve encountered. As a way of saying thanks to the Git community, I wanted to build something to recognize the efforts of the open source contributors that have turned Git into what it is today.

At the same time, I had been creating static data visualizations from software development data and thought that I should try to create something that was more interactive. I had recently read Data Sketches, by Nadieh Bremer and Shirley Wu, and was inspired by the stories behind the creation of their (often complex) interactive visualizations. Their experiences felt very close to mine, but they had pushed to an interactive end product, whereas I had yet to do so.

So, I decided to expand on my previous exploration of the Git development history and create a tool that would support the interactive exploration of the history of Git’s development.

Goals

In the past when I’ve seen visualizations of development histories, they tend to be things like force-directed simulations of module evolution over time, or contributor graphs like those provided by GitHub:¹

And, while these are interesting, they’re not quite what I had in mind. I was hoping to create something that encouraged interactive exploration of the data, providing additional details depending on what was selected.²

To get things started, even though I didn’t have a concrete idea of what I was going to build, I decided on the following project goals:

Provide some type of visualization of the entirety of the development history
Allow specific developers to be selected to understand their efforts
Provide interactive point-in-time and cumulative statistics
Show developer geographic distribution
Annotate the timeline with historical project milestones

These provided some direction to my overall goal, which was to experiment with alternative ways of visualizing development.

Visual Inspiration

I started by doing an exploration of existing visualizations as a source of inspiration. Here’s a sampling of the many great ones that I found:

Paul Button’s Items of Interest
Josh Gowen’s The Data Double
Mike Bostock’s 2018 Box Office
Susie Lu’s Strong Openers and Late Bloomers remake of the NYT original
Many, many of the works from The Pudding when it comes to telling a story with data

With these ideas in my head I started gathering data.

Getting the Data: Part One

Before I could do anything I needed to convert the Git project commit history into something that I could work with. Getting the Git repository was the easy part, thanks to the very nature of how Git works. Once I had the repository, I then needed author, time, and size data for every change (there were around sixty-two thousand changes in the project history at that time).

Somewhat surprisingly, there wasn’t an “out-of-the-box” way to export commit history that included the number of files changed, as well as total lines added or removed. You can get use git log to get something close, but I wasn’t able to find an incantation that provided the detail that I was looking for in an aggregated way. I ended up building a quick ruby script to parse the output of git log, aggregating file change details so that for each commit I’d have total files changed, lines added and removed, in addition to the other commit data like author, date, tag, etc.

I also looked into using git annotate on the entire repository to understand the historical ‘legacy’ of various lines of code, but that turned out to be cost prohibitive to run for every commit. I decided to generate this information for the start of every year as a starting point to understand if there was anything actually interesting there.

Exploring the Data

As I’ve mentioned in the past, my visualization workflow usually starts in Observable. Their reactive notebooks make data exploration really easy. I generally upload the data, parse and transform it as necessary, and then use the Vega Lite API (probably Plot soon) to interactively graph and explore the dataset.

I started by trying to understand how contributors and contributions changed over the life of the project. For example, this chart shows net lines of code changed per month by contributor for the top 10% of contributors over the entire project:

Top 10% of contributors

The sorting is sort of arbitrary, but you can see Linus Torvalds (he created the very first version of Git) at the top, with Junio C Hamano next (the project maintainer - he’s been very, very busy!). You can also see that there’s been a pretty consistent stream of new major contributors over the entire life of the project; active development has never really slowed down that much.

Next I thought I’d created a stacked area chart showing contributions over time:

Early stacked area experiment

That was interesting. You can see overall periods of activity increase, and how individual contributors participated over time. It was during this investigation that I came across the notion of a streamgraph, which seemed to present the data similarly while using some additional logic to improve the aesthetics of the final visual.

The problem with both the stacked area chart and the streamgraph is that they make comparing quantities difficult, but for this visualization my goal wasn’t to be able to accurately discern who had more changes in a single month - I was interested in presenting the project history in a way that encouraged interaction. The streamgraph felt like it might support that goal, so that’s what I decided to use as the main history navigation element.

Finally, I took a look at how code “ownership” had changed across the entire repository based on the yearly git annotate output.

Git contributor ‘legacy’

While that was interesting, it wasn’t really that surprising. It seemed natural that ownership would dilute over time as more people participated. I decided to shelve that data for another day.

Imagining the Layout

At this point I started sketching (yes, on paper) what the overall layout might look like. In my mind I knew that I wanted a static sidebar and a scrollable timeline that would provide a way to select project time periods. Statistics about the currently selected person or time period would be in the sidebar, and I was hoping to also have a globe or map that showed the distribution of developers for a selected period of time. (I had never created any type of choropleth, so I was interested in giving that a try, if possible.)

I was also hoping to have a way to see zoomed-in views of the timeline so that I could highlight the first several weeks, etc. I didn’t have any idea how to lay that out, so I did some exploration of scrollytelling to see if that might work. Ultimately I decided not to go down that path, as the scrollytelling approach was so completely foreign to me I had no idea how to fit it into the interaction model I had in my head. I will definitely return to it, however, when I have a more focused story to tell, as I think it can be really compelling.

Finally, I began to consider using circles to represent total contributions for a contributor. I had the idea of animating contributor circles as various time periods were selected, perhaps with a force directed simulation. I had no idea whether or not that would work/be a good idea, but since I was brainstorming at that point it all went on the list.

I came up with a rough three column layout, with the leftmost column for statistics, the contributor circles in the center, and finally the right column dedicated to a scrolling timeline streamgraph that would also somehow support annotations. With that loose structure, I was ready to start.

Let me add one caveat to all of this - I am not an experienced frontend developer. Development of modern browser-based applications, especially regarding how to make a visualization responsive and accessible, is a real challenge. I lean very heavily on the tools that I use to help me with these areas, to the extent that they are able. I am also not a visual designer, which further complicates projects like this. All that is to say that if you notice anything that could be improved, please let me know. I am very open to feedback.

Tools and Libraries

I have been a fan of Svelte for a while now, and I’ve also been experimenting with TailwindCSS, although when using Svelte the benefits aren’t as obvious. I ended up using it mostly for the structure it provided regarding font size consistency, spacing, etc. I also decided to try experimenting with Vite once the SvelteKit project moved over to it. I had been using Snowpack, but after a quick experiment and learning that it still used Rollup under the hood, I decided to try it.

In addition to the frontend tech used to structure and build the site, the visualization was all done in (unsurprisingly?) D3.js,³ while all of the data transformation was done in Arquero. Arequero is amazingly performant — I went into this expecting to have to pre-aggregate all of the data, but Arquero was fast enough that I haven’t yet needed to do that.

Getting the Data: Part Two

As I started building the site, I realized that I wanted to do two things to further cleanse/prepare the data:

Remove all email addresses from the source data set and replace them with unique ids.
Retrieve GitHub account information for contributor avatars and locations so that I could create a high-level contributor map.

The unique id was required to be able to unambiguously refer to contributor data once everything was loaded.

I wanted to remove the email address information because that just seemed like the right thing to do, even though that information is available directly in the Git repository.

Getting useful GitHub account information was a little more of a stretch. First of all, GitHub account information isn’t available when you retrieve commit data via the GitHub REST API, so you have to find some other means of getting it. I ultimately realized that GitHub provides detailed information for the top 500 contributors to a project, which seemed like a decent starting point since I wasn’t planning to show all contributor data (for performance and legibility reasons).

Ha ha, joke’s on me. Turns out that the GitHub account information is completely divorced from the reality of the Git repository emails, etc. This makes sense, as the Git repository history represents fixed information from a point in time (email address, name, etc.), whereas GitHub is a service that people keep up-to-date. As a result, there’s no “easy” way to join these data sets, and I was forced to join them by hand using commit counts, email matching, etc. I only tried to match up the first 250 contributors, but even so, that was not a fun afternoon.

Once that was done, I realized that out of almost 2,000 contributors, I could get location information for maybe 5%; almost no one had any location data in their GitHub profiles. Well, that was that — the idea of creating a map of worldwide contributor distribution was a non-starter. At least I could show contributor avatars for my trouble.

To minimize data duplication, I wrote a node script that used Arquero to load both of these data sets and join them, generate unique ids, and then write them back out as normalized data that would be re-joined in memory at runtime.

Challenges

I bumped into several issues while putting everything together. Most of the visualizations were pretty straightforward to create, but the two that caused the most grief were the streamgraph and the contributor circles.

Streamgraph

The streamgraph consists of two main parts: the visualization of contributor activity, and the timeline annotations.

Getting the data into the streamgraph wasn’t so bad (after browsing some Observable examples), but I quickly realized that it wouldn’t be possible to represent all contributors as separate curves without bringing the browser to its knees.

I decided to segment the contributors into two groups: the top 20% of contributors by commit volume, and everyone else. The top contributors would be represented directly, while the remaining contributors would be aggregated and displayed as a single “Other Contributors” band. This is unfortunate, but at the time I couldn’t think of another way to address the issue. It’s possible that I might be able to make things more performant by using Canvas, but that wouldn’t change the fact that the small 80% contributor band would be comprised of about 1,600 contributors. That seems like an unmanageable number of details to put into a small space without some type of zoom-based interaction.

The timeline annotations were also a challenge because it’s possible for two milestones to overlap, and I don’t have any logic for handling that. One possibility is to aggregate milestones when they overlap, but I haven’t yet had a chance to try that out.

Contributor Circles

During my research into force simulations in D3, I came across the idea of configuring the simulation such that a circle is attracted to a certain point on a timeline, but is free to move in the other dimension. I thought that I’d try to anchor the contributor circle to the date of the contributor’s first contribution, and the allow the simulation to move contributor around until things stabilized. I’d also increase the size of the circle to represent cumulative lines of code changed by contributor, so that the farther in the timeline you went, the more that a contributor’s circle would expand to represent overall project contributions.

I did try animating the circles when time period changed, but for some reason I wasn’t able to get them to settle down into a stable configuration, so I went with the approach of running the simulation without animation and simply showing resulting positions after 500 iterations. That seems to work okay, but I’d like to revisit the animation at some time in the future, even though my gut tells me it’d probably be very distracting.

The other issue I faced with was putting labels on the larger contributor circles. For a while I toyed with the idea of centering the name in the circle, but because SVG text elements don’t wrap, getting them to look reasonable was difficult. I ended up using SVG textPath to curve names around the bottom of the circle (but had to avoid the path part of the spec since it wasn’t supported by Chrome, Edge, and Safari).

Color Palettes

I ended up creating several color palettes during development, which was perhaps one of the most difficult parts of this entire project. When I started, the color palette was based on D3’s interpolatePurples, which was fine for prototyping but wasn’t anything I was going to be able to keep:

After many, many iterations I landed on a light and dark mode palette that didn’t seem terrible. You can see how they turned out below:

Mobile

Once all of the layout and charting was completed, I decided to see how things looked on a mobile device. Not surprisingly they weren’t great. I had designed an interactive visualization that was meant for a desktop style of interaction, a fact that was obvious if you tried to use a mobile device to do anything. I tried to add some code to handle smaller resolutions - one of the biggest challenges was getting the SVG elements and HTML font sizes to align depending on the size/aspect ratio of the viewport. That wasn’t great, so I ended up putting a warning on the site if you visit from a small-ish device resolution, but with the ability to proceed after the disclaimer. I would like to revisit how to redesign to support mobile in the future.

What’s Left?

Even though I had some issues during development, I am generally happy with how this initial version turned out. I’d like to be able to solve the timeline annotation spacing issue, and I think I have an approach to do so in my head.

What I’d really like to add, however, is a more comprehensive set of project milestones. I’d really like this to be useful as a way to understand important aspects of Git’s development history, but at this point I haven’t been able to capture a lot of milestones outside of the ones that can be directly derived from the data.

With that in mind, if you have any interesting information on the history of the Git project that you believe would make an interesting milestone, please don’t hesitate to contact me and I will happily consider adding it.

Thanks!

…for taking the time to read a little about this project. If you have any thoughts or feedback you’d like to share, please feel free to contact me on Twitter or directly on this site.

Which appear to be incorrect, now that I’m actually looking at them. 😬 Junio C Hamano is the third Git contributor, but in this chart doesn’t have any recognized contributions until some time after 2007-ish. I assume this is due to the mismatch between GitHub accounts and the identities in the Git commit history, which I’ll cover in more detail later. ↩︎
I find the GitHub visual presentation of historical data to be particularly lacking, as it immediately segments the data in a way that makes it hard to understand as a whole, while providing almost no tools for further investigation. Honestly this seems like a missed opportunity for them, but I am obsessed with this stuff so maybe I’m the outlier. They have so much historical development data available to them it would seem like an interesting path to explore. ↩︎
As an aside, I’ve seen some online opinions stating that D3 is not a good choice for visualization in 2021, and while I do agree that if you’re putting basic charts on a page it’s probably overkill, for anything of any complexity it’s really an amazing toolkit. The key is to treat it as such. Amelia Wattenberger and Russell Goldenberg cover this well in their recent Svelte Summit talk: Declarative Data Visualization: Creating a bar chart race. ↩︎

Visualizing Open Source Project Commit Histories

Jeff Palmer — Fri, 02 Apr 2021 03:05:59 +0000

Motivation

Recently, I came across this excellent interview with the graphic designer Peter Saville describing the origin of Joy Division’s famous cover for Unknown Pleasures.

I hadn’t realized that the image was a scientific visualization of the data from a pulsar (actually data from the first discovered pulsar PSR B1919+21), but now that I know the history I somehow find it even more compelling.

I decided to recreate the visual form of the image, but needed an appropriate data source. Since I’d been experimenting with the visualization of open source commit histories recently, I thought I’d start with that data and see how things looked.

Approach

One of the challenges with data visualization (at least for me) is dealing with the mismatches that inevitably happen between the image I had in my mind and the one that the data actually creates. I guess that’s pretty common, as both Nadieh Bremer and Shirley Wu both mention it in Data Sketches. (The Data Sketches book is excellent, by the way. If you’re interested in data visualization you should definitely get it.)

Anyway, my intuition was that, if you aggregated the changes made over the history of the project, there would be some interesting patterns that would fit the pseudo-3D layout of the original image. So I set about to pull some data from various projects to see if that was true.

Exploring the Data

I usually start a new visualization project by putting a dataset into Observable and then interactively experimenting with various visualizations to better understand the shape of the data. Because this is exploratory visualization, I want to be able to quickly create many different types of visualizations to open/close exploration options. I use Vega Lite (specifically the Vega Lite API JS wrapper) to do this. Take a look at this introduction if you’re interested in seeing how the combination of Observable and Vega Lite can really accelerate data exploration.

I first exported the git commit history from the Visual Studio Code repository, taking care not to normalize the commit times. I wanted the author’s local commit times so that they could be aggregated in a way that represented an author’s day, not the project-wide perspective. I was also curious as to whether there might be interesting information in the timezone offsets, so I exported the author commit time and parsed it by hand into these various components.

I also needed to aggregate the commits in some way that made it possible to layer timespans on the y-axis. I started by counting the number of commits (circle radius) in each minute of the day (x-axis), and then combining the year and month as the y-axis. This looked promising:

Initial visualization of VS Code commits

That is an amazingly consistent lunch hour! 😆

Unfortunately, once I put this data into d3 to actually create the visualization, the results were not very compelling:

That’s waaay too much detail!

Even though I could see that there was a clear pattern to the data, aggregating by minute obscured it by providing too much detail. I experimented with a variety of aggregation windows until I landed on 30 minutes, which seemed to best capture the pattern that I saw in Observable while not smoothing things out to such an extent that it was no longer visually interesting.

Results

A couple of titles later and I had the final version. Each line represents a full month of commits, broken into 30 minute bins.

Visual Studio Code - Monthly Commits by Hour

I thought that the clear separation between morning and afternoon was pretty interesting, and I was honestly surprised at the consistency over time. I realized that the reason for this was that, even though Visual Studio Code is an open source project, it’s primarily a Microsoft-driven effort.

I decided to run the commit history of some other open source projects through the same pipeline, unmodified, to see how they compared.

React

React - Monthly Commits by Hour

I started with React. React’s repository has 14,010 commits over its eight year commit history, compared to Visual Studio’s 79,457 (as of this writing).

I thought it was interesting to see how React seems to have had a period of time where contributions leveled off, and then increased dramatically again for another several months.¹

The React team also seems to have a few more night owls working on it, with the far left side showing pretty regular post-midnight contributions.

TensorFlow

TensorFlow - Monthly Commits by Hour

Then I decided to take a look at Google’s TensorFlow. 108,084 commits, starting in November of 2015.

The distribution here seems closer to that of Visual Studio, although without the very distinct morning/afternoon separation.

I’d also guess that there’s something automated happening in the early morning. You can see it shift by an hour forward and backward every few months. Everyone’s favorite villain, daylight saving time. 😭

Git

Git - Monthly Commits by Hour

Finally I pulled in the Git commit history, which I had just been digging through for my recent Git commit history visualization. 62,574 commits, starting in April of 2005.

What I like about this is that it’s clear that the project has gone through some big changes over time. At the beginning, there was a lot of late night hacking happening, and the contributions seem pretty consistent over that time period.

There’s a major change in contributions, however, starting about one third into the project history. You can see the number of contributions increase dramatically compared to the first third, and stay that way.

It also seems not to have such a clear “company project” vibe, as the contribution times are all over the place.

Conclusion

For the small-ish amount of work that I put into this, I found the results to be pretty interesting, and I’m glad I did it. Being able to compare various projects against one another was something that I hadn’t planned to do, but once the tools had been created, that was effectively free.

I hope you found these interesting. As always, if you have any feedback please let me know on Twitter or feel free to contact me here.

If I wasn’t creating a stylized visualization, I think it’d be helpful to have a margin histogram for each month showing the total contributions for that month. At the moment the aggregate contributions for the month are hard to see. ↩︎

Prioritizing with Cost of Delay

Jeff Palmer — Tue, 30 Mar 2021 21:52:06 +0000

Photo by Jp Valery

I can’t begin to count the number of times I have heard development teams say things like this:

“We don’t have time to fix that.”
“Product won’t let us fix our technical debt.”
“All we do is build new features.”

Which I mentally translate to:

We don’t know how to explain the impact of our technical projects.

This post is an attempt to explain how quantifying Cost of Delay can help teams successfully understand and prioritize their technical project backlog.

The Prioritization Problem

Prioritization is key to ensuring that a company is investing in efforts that will sustainably grow the business. Product teams spend a tremendous amount of time working to understand and quantify project impacts so that financial benefits are maximized (or at least they should!).

However, it’s not always the case that there is a corresponding understanding of the technical backlog — infrastructure projects, security upgrades, etc. Depending on the organizational structure, these projects may be in separate tools, spreadsheets, or simply in people’s heads. Even if there is a central technical project list, project benefits are often described in technical terms and may require in-depth system knowledge to appreciate. These inconsistencies make prioritizing technical projects within development a real challenge.

Outside of development, this lack of understanding becomes a bigger problem. When the Product team has already quantified the financial impacts of six months of feature development, hand-wavy technical project lists aren’t going to cut it.

With that in mind the question becomes: How can the impacts of technical projects be quantified such that they can be directly compared to other projects?

Introducing Cost of Delay

The term “Cost of Delay” captures the idea that you can quantify the financial impact of delaying work. It is measured in dollars / time unit, and should represent the total financial impact to the organization if a project is delayed for a single time unit.

I first came across the concept in the writings of Donald Reinertsen, shown here describing how he came to recommend quantifying the Cost of Delay to his customers.

Why Cost of Delay?

I have found Cost of Delay to be an excellent means of prioritizing projects because it provides a consistent way to compare the impact of any type of project. Since Cost of Delay is measured in dollars / time unit, there’s no ambiguity when comparing impacts. By quantifying Cost of Delay it’s possible to have a much more focused conversation about technical initiatives, and these can then be compared directly according to financial impact.

Calculating Cost of Delay for a Project

If you want to calculate the Cost of Delay for a project, you need a model that describes the impact of the project to the organization in financial terms. There are many ways to do this. Projects can reduce operational costs, reduce security risks, generate revenue, eliminate time-wasting development friction, etc. Depending on the needs of the organization, models can be basic so as to provide high-level directional guidance, or detailed to the point that historical operational cost data is included.

Personally, I prefer to start with a simple model if at all possible. I’ve found that the prioritization conversation that happens across projects has the most impact, so I try to get there as quickly as possible.

Example: Reducing Build Time

Let’s work through an example of how to calculate Cost of Delay for a single project: reducing build times.

Scenario : Say you are part of a 100 person development organization where everyone is working in a monorepo. Build times average about 20 minutes, and people are tired of having to wait for builds during the Pull Request review process. Is it worth spending time to make the build faster?

Because the goal of this project is to reduce build times, that impact can be modeled in (at least) two areas:

Reduction in continuous integration provider costs
Reduction in developer “downtime” (i.e., waiting for builds to complete, context switching, etc.)

If you make some assumptions about how many PRs each developer creates per week, and how many reviews on average each of those PRs requires, you can create a model that calculates the amount spent on CI infrastructure as well as the cost of having developers wait for builds to complete. (A spreadsheet containing this model is available here.)

Note: the goal of this model is not to perfectly reflect reality, but to represent the project’s impacts given a shared set of organizational assumptions. This consistency will make it possible to compare the impacts of one project to others.

Based on this model, keeping the same build infrastructure and reducing build times by five minutes would result in a savings of nearly $1500 / week, or about $78,000 annually. Reducing build times by 10 minutes would have a savings of $3000/week, or $156,000 annually.

Additionally, because this model scales with the number of developers, if the team were to start hiring aggressively (and who isn’t?) the weekly Cost of Delay would increase over time as more engineers join the organization. (You might choose to include this growth in the model, depending on the conversation.)

Yes, there are valid questions as to the amount of downtime that developers actually incur when waiting for builds to complete, and you could certainly model things differently. To me that’s the beauty of a model with documented assumptions — your team can collaboratively determine which assumptions make the most sense for your organization.

Bonus: Using the Model to Explore Options

Once a model has been created for a project, it can sometimes be used to explore various implementation options. For example, for this project there are at least two options to reduce build times:

Have someone with a detailed understanding of the build process work to manually optimize the build
Throw hardware at it

Since you now have a model that can give you a high level understanding of how these might work out in practice, this can be quite enlightening.

For example, if it turns out that you can reduce build times by five minutes using a larger instance (say one that is 4 times more expensive), then while your CI provider costs might increase from $288 to $864, the overall cost to the organization would decrease from $6479 to $5628. In that case spending more money might actually save you money.

Prioritizing Projects with Cost of Delay

Okay, so once you have the Cost of Delay for a project, how can that be used to prioritize?

The recommended way to do this is to divide the Cost of Delay by the estimate of how long it will take to do the work. This is often referred to as CD3 ( C ost of D elay D ivided by D uration), and using this combination of factors to order projects will optimize the value delivered over that time period. (For a detailed explanation as to why this optimizes value in this fashion,¹ see Reinertsen’s The Principles of Product Development Flow.)

For example, here are the CD3 values for several projects:

ID	Project	Cost of Delay	Duration	CD3
1a	Reduce build time by 5 minutes (move from Large to 2X-large)	$1,714/week	1/5 week	8570
1b	Reduce build time by 10 minutes (Manually optimize build)	$2,997/week	1 week	2997
2	Increase log retention from 7 to 30 days	$37,860/week	1 week	37860

Even with this simple list there are some interesting results.

Due to the large time impacts missing logs have on problem investigation efforts, the Cost of Delay for increasing log retention is very large relative to the other projects on this list. CD3 says that that project should be done first (if a single team has to choose). Only once that project is completed should the team experiment with other instance types to reduce build times, even though that build time reduction effort is only expected to take a day.

Is this really the best order to complete these projects? If reducing build times by 5 minutes only takes a day, shouldn’t that get done first? Let’s see.

If the team tried out new instance sizes first (project 1a), and then increased the log retention (project 2), they would incur the Cost of Delay of the both project 1a and project 2 for the first day, and then the cost of project 2 for the remainder of its implementation. That’s ($1,714 / 5) + ($37,860 / 5) + $37,860 = $45,775.

If the team were to complete the projects in the order indicated by CD3, the costs would be $37,860 + $1,714 + ($1,714 / 5) = $39,917. The CD3 ordering saves $5,859 while delivering the same results in the same time period.

Conclusion

I hope I’ve provided some insights into how Cost of Delay can help you understand project impacts and assist in project prioritization. Once your organization is able to clearly describe project impacts in financial terms, it becomes much easier to advocate for technical initiatives and discuss priority trade-offs with other teams.

If you have any feedback, please let me know on Twitter or feel free to contact me here.

tl;dr: it minimizes the area under the project cost curve. ↩︎

The 2013 Felton Volume Chart

Jeff Palmer — Fri, 26 Mar 2021 22:13:57 +0000

This is the final post in my series on how I created the various visualizations in my Visual Studio Code 2020 Year in Review. In this post I cover how I recreated the final visualization type from Nicholas Felton’s 2013 Annual Report, the “volume chart,” shown below.

Volume chart detail from 2013 Annual Report(Nicholas Felton)

This stacked set of charts breaks down each type of interaction and displays the aggregated volumes of those interactions by hour (y-axis) for each week in the year (x-axis). There is a histogram at the top of the visualization that provides summary context for the hourly measures below. Finally, a yellow highlight color represents the top 25% of volume measures for that type.

The approach I took to build this visualization was to break it into two Svelte components: WeeklyVolumeHeader and HourlyVolume. WeeklyVolumeHeader generates the weekly summary histogram SVG at the top of the visualization, and HourlyVolume generates an SVG for each of the volume visualizations. This separation made it easy to repeat the volume visualization with a different slice of the dataset for each event.

Before all of that, however, the data needed to be loaded.

Transforming the Data

As I previously mentioned, all of the data used to create this visualization was obtained via queries to the ClickHouse GitHub archive that they have generously made available to the public.

I used the following query to aggregate event counts by week and hour, and then save the data into a JSON file for use in this visualization.

select
  toStartOfWeek(toTimeZone(created_at, 'CET')) as week
  , toHour(toTimeZone(created_at, 'CET')) as hour
  , countIf(1, event_type = 'PullRequestEvent') as pull_requests
  , countIf(1, event_type = 'IssuesEvent') as issues
  , countIf(1, (event_type = 'CreateEvent' or event_type='DeleteEvent')) as branches
from github_events
where repo_name='microsoft/vscode'
  and toYear(toTimeZone(created_at, 'CET'))=2020
  and event_type in ('CreateEvent', 'DeleteEvent', 'PullRequestEvent', 'IssuesEvent')
group by week, hour
into outfile '/data/volume-weekly-activity-by-hour.json'
format JSON;

This query produces a JSON file that has two top-level entries, meta and data. The data entry contains the query results, and is what needs to be processed to create usable data for visualization. It looks like this:

[
  {
    "week": "2019-12-29",
    "hour": 12,
    "pull_requests": "0",
    "issues": "4",
    "branches": "0"
  },
  {
    "week": "2020-08-02",
    "hour": 13,
    "pull_requests": "0",
    "issues": "87",
    "branches": "0"
  },
...

Ideally, the data would look like this:¹

[
  {
    "week": "2019-12-29",
    "hour": 12,
    "event": "pull_requests",
    "count": 0
  },
  {
    "week": "2019-12-29",
    "hour": 12,
    "event": "issues",
    "count": 4
  },
  {
    "week": "2019-12-29",
    "hour": 12,
    "event": "branches",
    "count": 0
  }
]

You can see that the original row that contained a column for every event has been “unrolled” into three rows, and the type of event is itself now a column. This type of format, where every observation is represented as a row in the dataset, is often referred to as “tidy” data. It was first described in Tidy Data, by Hadley Wickham, and has since become a popular way of preparing/structuring data for analysis within the data science community.

As more and more teams have been working with tidy data, a number of tools/libraries that simplify the process of “tidying” data have been developed. For TypeScript, I’ve been using a library called Arquero (developed by the folks at the UW Interactive Data Lab), and I’ve been really pleased with it so far.

Tidying the Data with D3 and Arquero

To get things rolling, the data needs to be loaded from the JSON file into memory. I used d3.json to load the JSON data from its file on disk, and then ran a series of data transformations on the data using Arquero to get it into the format I described above.

import * as d3 from "d3";
import * as aq from "arquero";
async function loadData() {
  const rawData = await d3.json("/data/volume-weekly-activity-by-hour.json");
  aq.addFunction("d3_parse_date", d3.timeParse("%Y-%m-%d"), {override: true,});
  const allData = aq
    .from(rawData.data)
    .fold(["pull_requests", "issues", "branches"], {
      as: ["event", "count"],
    })
    .derive({
      count: (d) => aq.op.parse_int(d.count),
      week: (d) => aq.op.d3_parse_date(d.week),
    });
  return allData;
}

In line 8 I extend the standard Arquero operators so that I can use the d3.timeParse function instead of relying on the wonky default Javascript Date constructor that Arquero uses internally (and warns you about).

Lines 11-20 actually transform the data. Line 12 converts the JSON array into the Arquero table format, and lines 13-15 use the fold verb to convert column names and values into rows.

Once that’s done, lines 16-19 convert strings to integers or dates, with line 18 using the d3_parse_date custom operator that was defined above, and the cleanup is complete.

In Svelte, you can await the result of an asynchronous call using the #await construct.² This makes it easy to retrieve the data asynchronously directly in the markup, like so:

<div class="p-5">
  {#await loadData() then data}
    <WeeklyVolumeHeader {data} {width} />
    <HourlyVolume
      data={data.filter((d) => d.event === "pull_requests").objects()}
      {width}
      {highlightColor}
      {normalColor}
      title="PULL REQUESTS"
    />
    <HourlyVolume
      data={data.filter((d) => d.event === "issues").objects()}
      {width}
      {highlightColor}
      {normalColor}
      title="ISSUES"
    />
    <HourlyVolume
      data={data.filter((d) => d.event === "branches").objects()}
      {width}
      {highlightColor}
      {normalColor}
      title="BRANCHES"
    />
  {/await}
</div>

With that done, it was time to start building the weekly summary component.

The Weekly Volume Header Histogram

This histogram is a slightly modified version of the two histograms that I described in earlier posts. The new element in this version is the addition of a vertical dashed line indicating the month boundary.

Each of these separators is a dashed line from the x-axis to the measurement at the time of the start of the month. However, because the solid line represents a histogram that is aligned with week boundaries, it’s necessary to find the weekly measure into which the start of the month falls. It turns out that you can use a d3.bisector to efficiently find these entries, which can then be used to find the height at the start of the month.

A bisector function is used to search a sorted array for the location at which a new value should be inserted to keep all array values properly sorted. This search takes advantage of the sorted nature of the array and uses a binary search to find the insertion point, which is very efficient as the number of entries being searched increases.

Here’s how it’s put into action:

// generate a set of month boundary dates
$: months = d3.timeMonth.range(
  weeklyTotals[0].week,
  weeklyTotals[weeklyTotals.length - 1].week
);
const dateFinder = d3.bisector(xAccessor);
function findHeightAtDate(data, date) {
  const index = dateFinder.left(data, date) - 1;
  return yScale(yAccessor(data[index]));
}

Provided with any date, findHeightAtDate will provide the height of the histogram at that date. This can then be used to draw the dashed line that corresponds to the start of each month:

<svg viewBox="0 0 {width} {height}">
  {#each months as month}
    <line
      class="dashed-line"
      x1={xScale(month)}
      x2={xScale(month)}
      y1={yScale(0)}
      y2={findHeightAtDate(weeklyTotals, month)}
    />
  {/each}
  <path class="total" d={yLine} />
</svg>

which results in this:

That is quite close, but there’s a problem. The start of March happens to coincide with the transition element between histogram measurements, and as a result the dashed line extends above transition line. This problem is one of my own devising. 😬

I decided that the easiest thing to do³ was to try to figure out a heuristic that would allow me to figure out when I needed to blend two histogram measurements together. In this case, the week boundaries are all on Sundays, so I was able to simply calculate an average measurement for any month that started on a Sunday.

function findHeightAtDate(data, date) {
  const index = dateFinder.left(data, date) - 1;
  // if the current date is a sunday (the start of the week) we'll
  // have some issues with the Felton line transitions, so work around
  // that by averaging neighboring measure heights
  if (date.getDay() === 0) {
    return (
      ( yScale(yAccessor(data[index])) +
        yScale(yAccessor(data[index + 1]))) / 2
      );
  } else {
    return yScale(yAccessor(data[index]));
  }
}

This result was good enough for me.

On to the volume visualization component.

Creating the Volume Visualization

It turns out that the HourlyVolume component was a very straightforward application of several d3 scales. In the end this visualization required five separate scales, each of which was responsible for mapping a certain aspect of the volume measurement to its visual representation. These included:

xScale, yScale to map weeks and hours to coordinates
radiusScale to map count of events to circle size
colorScale to map count of events to a color fill
opacityScale to map count of events to an opacity⁴

Sizing the circles

I’ve covered how to create xScale and yScale scales in previous posts, so I’ll move straight to the radiusScale.

When using a filled circle to visually represent a quantity, it’s important not to overstate amounts accidentally. With a circular representation this is easy to do because the area of a circle varies as the square of the radius. To address this, when using a filled circle the scale for the radius should use d3.scaleSqrt, which will scale a value according to the square root of the parameter.

I also wanted to ensure that outliers in the aggregated data didn’t end up overwhelming or obscuring parts of the the volume visualization. The easiest way to do that is to calculate a value that represents the high end of the collection of measurements without outliers, and use that instead of the maximum value. One way to do this is to calculate the 99th percentile for the dataset and use that as the maximum value for the domain of the scale. This can be done using d3.quantile.

Finally, once the scale has been created, the clamp function can be used to ensure that values that fall outside of the specified domain are clamped to the extents of the domain. This ensures that the largest radius the scale will generate is the one that is specified with maxRadius.

const maxRadius = 2.5;
const boundPercentile = 0.99;
// find an upper limit for the radius scale based on boundPercentile
// this allows us to use .clamp(true) to keep outliers from
// overwhelming the visual
$: sizeLimit = d3.quantile(data, boundPercentile, sizeAccessor);
$: radiusScale = d3
  .scaleSqrt()
  .domain([0, sizeLimit])
  .range([0, maxRadius])
  .clamp(true);

💡 Note: I typically put a set of constant declarations for these types of fine-tuning variables at the top of my components, so that I can tweak them as necessary based on the data I’m actively trying to visualize.

Coloring the circles

Finally, in the original volume chart all aggregate values that were in the busiest 25% of activity were colored yellow, whereas in my VS Code visualization only the busiest 20% were highlighted. Both of these can be easily implemented via d3.scaleThreshold. This scale type transforms a domain into sections based on specified thresholds, which makes it perfect for performing the described color mapping.

$: p80 = d3.quantile(data, 0.8, sizeAccessor);
$: colorScale = d3
  .scaleThreshold()
  .domain([p80])
  .range([normalColor, highlightColor]);
$: opacityScale = d3.scaleThreshold()
  .domain([p80])
  .range([0.5, 1.0]);

(Both normalColor and highlightColor are Svelte component properties and are passed in from the parent component.)

This creates a scale that is split at the 80th percentile point in the domain (p80). The left and right sides of this point are colored according to the range [normalColor, highlightColor], meaning that any value that is above the 80th percentile in the domain will be highlighted, and any point below it will not be.

The opacityScale is the same, except that it maps to two opacity values instead of colors.

Once all of these elements are in place, the actual drawing of the volume chart is pretty minimal:

<svg viewBox="0 0 {width} {height}">
  {#each weeks as week}
    {#each getHoursForWeek(data, xAccessor(week)) as hour}
      <circle
        cx={xScale(xAccessor(hour))}
        cy={yScale(yAccessor(hour))}
        r={radiusScale(sizeAccessor(hour))}
        fill={colorScale(sizeAccessor(hour))}
        opacity={opacityScale(sizeAccessor(hour))}
      />
    {/each}
  {/each}
  <line class="baseline" x1={0} x2={width} y1={0} y2={0} />
  <text class="title" x={0} y={15}>{title}</text>
  <text class="time-label" x={width} y={15}>AM</text>
  <text class="time-label" x={width} y={height - 5}>PM</text>
</svg>

where getHoursForWeek is defined as the following, because apparently you can’t compare Date objects for equality. Joy!

function getHoursForWeek(data, week) {
  const result = d3.filter(
    data,
    (d) => xAccessor(d)?.valueOf() === week.valueOf()
  );
  return result;
}

The Final Version

Once this is all in place, the final version of the volume chart looks like this:

Final volume chart version

The code I used to write this post is available on GitHub.

If you have any feedback, please let me know on Twitter or feel free to contact me here.

Yes, I could have written the query in such a way as to generate something much closer to this format from the beginning. I didn’t do that because of time constraints, and I’ll definitely try harder to export tidy data directly going forward. ↩︎
This is the short form of the #await construct, which if fine when I’m prototyping ↩︎
After trying to ignore it by increasing the stroke-width. 😆 ↩︎
Strictly speaking, the opacityScale was not required, as I could have not been lazy and chosen a dimmer color for this visual. ↩︎

Visualizing the Change History of the Git Repository

Jeff Palmer — Wed, 24 Mar 2021 01:16:41 +0000

The Change History of the Git Repository from 2005-2020

Recently I decided to continue my experiments visualizing the data produced by the software development process. This time I decided to dig into the git commit history of Git itself, with the goal of creating something interesting and creative that still provides information about the subject matter.

This visualization is a radial representation of all changes stored in the Git git repository from inception to the end of 2020. Time flows clockwise and outward, starting from the first commit on 7 April 2005, until 31 December 2020. Circles represent the net lines of code added to the project on a single day.

Detail View

It was a fun exercise and I’ll probably try to apply this approach to some other popular open source projects. I hope you enjoy it!

As always, if you have any feedback please let me know on Twitter or feel free to contact me here.

The 2013 Felton Histogram - Variant 2

Jeff Palmer — Mon, 15 Mar 2021 20:05:32 +0000

Nicholas Felton’s 2013 Annual Report contained two distinct histogram variants. This post covers how I created the second variant, which looks like this:

Histogram (Variant 2) detail from 2013 Annual Report(Nicholas Felton)

This is similar to the first variant that I covered in my last post, with the following changes:

Two measures are plotted on the same x-axis, with their y-axes facing in opposite directions
Irregular stippling is used to differentiate the lower measure
Measure series labels are added, while value labels are removed

Plotting with Multiple Axes

This histogram variant adds a second measure that has its y-axis pointing downward, in opposition to the y-axis of the top measure. Since the y-axis in SVG increases from top to bottom, the range of the top scale will be from the y midpoint to the top margin (from x-axis upwards), and the range of the bottom scale will be from the y midpoint to the height minus the bottom margin (from x-axis downwards).

$: xScale = d3
  .scaleTime()
  .domain([dateParser("2020-01-01"), dateParser("2021-01-01")])
  .range([margins.left, width - margins.right]);

// The top scale goes from midpoint to top margin
// Also add 10% of the max top value to the end of the domain
// to ensure there's headroom on the chart
$: topScale = d3
  .scaleLinear()
  .domain([0, d3.max(data, topAccessor) * 1.1])
  .range([
    (height - margins.bottom - margins.top) / 2 + margins.top,
    margins.top,
  ])
  .nice();

// The bottom scale goes from midpoint to bottom margin
// Also add 10% of the max bottom value to the end of the domain
// to ensure there's footroom (?) on the chart
$: bottomScale = d3
  .scaleLinear()
  .domain([0, d3.max(data, bottomAccessor) * 1.1])
  .range([
    (height - margins.bottom - margins.top) / 2 + margins.top,
    height - margins.bottom,
  ])
  .nice();

With these scales, the same line drawing techniques from the last post can be used to create the two measure lines.

First, the line paths were created using the generateFeltonLine method:

$: topLine = d3.line()(
  generateFeltonLine(
    augmentedData,
    xScale,
    xAccessor,
    topScale,
    topAccessor
  )
);

$: bottomLine = d3.line()(
  generateFeltonLine(
    augmentedData,
    xScale,
    xAccessor,
    bottomScale,
    bottomAccessor
  )
);

Then they were placed within the SVG along with some CSS classes for styling:

<svg viewBox="0 0 {width} {height}" ...>
<path class="top-line" d={topLine} />
<path class="bottom-line" d={bottomLine} />
...
</svg>

Note: Much of the boilerplate in the code has been omitted to keep things short. These omissions are represented by ... in the source snippets. See the GitHub repository for the working implementation.

Generating Irregular Stippling

The original histogram used a form of stippling to differentiate the bottom measure from the top measure. My goal was to find an algorithm to generate realistic-looking stippling, generate a polygon that represents the area to be shaded, and then use the polygon as a mask to determine which points should be shown.

Aesthetically pleasing random point distribution

It turns out that generating stippling that looks good is actually somewhat difficult. Luckily I was able to lean on the community to find an existing implementation of Bridson’s Algorithm, which can be used to approximate human stippling. Thanks (again) Mike Bostock!

I added a version of this generator function to my code and was able to use it to generate a set of points like so:

import { poissonDiscSampler } from "../utils.ts";

// Use the generator to generate all of the points
// so that they can be iterated over by svelte
function generatePoissonPoints(width, height, radius) {
  const sampler = poissonDiscSampler(width, height, radius);
  let points = [];
  let a: [float, float] = sampler();
  while (a) {
    points.push(a);
    a = sampler();
  }
  return points;
}

// generate points covering the entire bottom of the SVG
// they'll be filtered later
$: points = generatePoissonPoints(
  width,
  (height - margins.top - margins.bottom) / 2,
  10
);

(The actual radius value of 10 that I used was the result of some visual trial and error.)

If you were to plot these points without any filtering, you get this:

Stippling without polygon containment checks

The stippling looks great! Now we just need to get rid of everything outside of the measure area.

Creating the bounding polygon

Unsurprisingly, D3 has the ability to perform polygon containment checks out of the box via polygonContains. Since I had already created a function to generate the set of points that constitute the measure line, creating a polygon from those points was straightforward.

function generateClosedFeltonPolygon(data, xScale, xAccessor,
yScale, yAccessor) {
  const lineSegments = generateFeltonLine(data, xScale, xAccessor,
    yScale, yAccessor);
  return [
    // First point - on the origin
    [xScale(xAccessor(data[0])), yScale(0)],
    // Now the generated points for the measure lines
    ...lineSegments,
    // Bring the line back to the x-axis
    [xScale(xAccessor(data[data.length - 1])), yScale(0)],
    // Now close the polygon by going back to the origin
    [xScale(xAccessor(data[0])), yScale(0)],
  ];
}

$: closedPoly = generateClosedFeltonPolygon(
  augmentedData,
  xScale,
  xAccessor,
  bottomScale,
  bottomAccessor
);

Once the polygon was created, it could be used to filter the point array.

<svg ...>
  ...
  {#each points as point}
    {#if d3.polygonContains(closedPoly, [point[0], height / 2 + point[1]])}
      <circle
        class="stipple"
        cx={point[0]}
        cy={height / 2 + point[1]}
        r={2.5}
      />
    {/if}
  {/each}
</svg>

At this point, the filtered histogram looked like this:

Stippling after implementing polygon containment checks

Adding Labels

The final step was to add measure labels above and below the x-axis:

<text
  class="top-label"
  text-anchor="end"
  x={width - margins.right}
  y={(height - margins.top - margins.bottom) / 2 +
     margins.top - textHeight / 2}>Top</text
>
<text
  class="bottom-label"
  text-anchor="end"
  x={width - margins.right}
  y={(height - margins.top - margins.bottom) / 2 +
     margins.top + textHeight}>Bottom</text
>

Unfortunately the lower label was somewhat obscured by the stippling:

Measure labels - detail

I experimented with CSS-based text shadows in an attempt to create an outline that would mask the nearby points, but that didn’t work as well as I hoped it would. I did a little more research and eventually came across an article describing the feMorphology filters available in SVGs. (It’s still amazing to me what’s built into SVG!)

These filters can be used to define a series of image transformations, which in this case were used to implement a very clean text outline (see the linked article for a detailed description of how this actually works).

<svg ...>
  <filter id="outline">
    <feMorphology
      in="SourceAlpha"
      result="DILATED"
      operator="dilate"
      radius="5"
    />
    <feFlood
      flood-color="rgba(55, 65, 81, 1)"
      flood-opacity="1"
      result="BACKGROUND"
    />
    <feComposite
      in="BACKGROUND"
      in2="DILATED"
      operator="in"
      result="OUTLINE"
    />
    <feMerge>
      <feMergeNode in="OUTLINE" />
      <feMergeNode in="SourceGraphic" />
    </feMerge>
  </filter>

  <text
    class="top-label"
    filter="url(#outline)"
    text-anchor="end"
    x={width - margins.right}
    y={(height - margins.top - margins.bottom) / 2 +
       margins.top - textHeight / 2}>Top</text
  >
  <text
    class="bottom-label"
    filter="url(#outline)"
    text-anchor="end"
    x={width - margins.right}
    y={(height - margins.top - margins.bottom) / 2 +
       margins.top + textHeight}>Bottom</text
  >
</svg>

Once the SVG text elements were altered to include a reference to the filter via filter="url(#outline)", the label text in the stippled area became significantly more legible:

SVG text outline via feMorphology filters - detail

Final Result

At this point, the final version of the histogram was complete:

Final histogram version - Variant 2

The code I used to write this post is available on GitHub.

If you have any feedback, please let me know on Twitter or feel free to contact me here.

The 2013 Felton Histogram - Variant 1

Jeff Palmer — Tue, 09 Mar 2021 20:51:43 +0000

Nicholas Felton’s 2013 Annual Report contained two distinct histogram variants. This post covers how I created the first variant, which looks like this:

Histogram (Variant 1) detail from 2013 Annual Report (Nicholas Felton)

A pretty typical histogram. However, the styling aspects are what I found most interesting:

Truncated measure labels with a scale factor in the upper-right corner
Slanted measure transition lines that join the entire series of measurements into a single line

I’ll cover how I achieved both of these below.

Truncated Measure Labels

This was pretty straightforward, as there’s a math function that can be used directly to determine how many significant digits you need to represent a number: Math.log10. The log10 function calculates the base 10 logarithm of a number, which can be used to calculate the number of digits within it. For example,

Math.log10(623) = 2.7944880466591697

The scale factor is calculated by using the whole part of the result (using Math.floor) as the exponent in a call to Math.pow.

// We know we're showing 12 months so divide into 24 intervals
labelXOffset = (width - margins.left - margins.right) / 24;
// Calculate the scale factor used to get the most-significant digit
const yMin = d3.min(data, yAccessor);
const scale = Math.pow(10, Math.floor(Math.log10(yMin)));
const formatter = d3.format(".0f");

The minimum value was used to ensure that the labels weren’t unnecessarily “flattened” by a scale that was off by a factor of 10 (or more), and D3’s formatting functions were used to discard the fractional part of each measure.

Inside the SVG, the measure labels were added using text elements:

{#each data as datum}
  <text
    class="label"
    x={xScale(xAccessor(datum)) + labelXOffset}
    y={yScale(yAccessor(datum)) - labelYOffset}
    >{formatter(yAccessor(datum) / scale)}</text>
{/each}

The title and scale indicator was added using additional text elements:

<text class="title" x={margins.left} y={margins.top + textHeight}>
  TOTAL MONTHLY MEASUREMENT
</text>
<text
  class="scale-label"
  x={width - margins.right}
  y={margins.top + textHeight}>&times;{formatter(scale)}
</text>

Measure Transitions

I implemented two separate line styles while building this visualization. The first used the built-in D3 line generation tools, and the second used a custom line generator.

First Attempt Using D3 Curves

At first I tried to use the line function to generate a segmented line that represented the measures in the histogram. There is a curve function that can be used to specify how points in a line should be joined, and in particular curveStepAfter seemed like it might do the job.

💣 However, in order for this to work I needed to ensure that there was one extra entry in the data, or D3 wouldn’t create the final flat value segment. I simply copied the last point and set the date to one month in the future.

// Massage the data so that there are enough points to complete the path
// Copy the final entry in the array
const myClonedData = R.clone(data[11]);
// Now set the date to one month after
myClonedData.month = "2021-01-01";
const myData = [...R.clone(data), myClonedData];

// First attempt - using 'curveStepAfter'
const yLine = d3
  .line()
  .x((d) => xScale(xAccessor(d)))
  .y((d) => yScale(yAccessor(d)))
  .curve(d3.curveStepAfter)(myData);

This resulted in the following:

First attempt using D3 line and curveStepAfter

This was actually a pretty good starting point, and was what I used until I had time to revisit the visualization.

Second Attempt Using Custom Line Generator

Once I had a little more time to revisit my first attempt, I decided to implement the slanted segment connectors that were used in the original.

I had to replace the use of the line built-in x and y functions with a custom function that took the spacing between measured values into consideration. My goal was to allocate 5% of each measure line at the start and end to be used for the transition. As a result, each line segment was drawn by creating a horizontal line for the measure itself, and then creating a line from the end of that measure to the start of the next. This logic is contained in the following function:

function generateFeltonLine(data, xScale, xAccessor, yScale, yAccessor) {
  // this is only correct because of 0-based arrays
  // and # segments = # points - 1
  const segments = data.length;

  // Calculate displayed segment width and connector width
  const segmentWidth =
    xScale(xAccessor(data[1])) - xScale(xAccessor(data[0]));
  const connectorWidth = segmentWidth * 0.05; // 5% on each side

  // start with the first point, as it (and the last point)
  // are special cases
  let result = [
    [ xScale(xAccessor(data[0])), yScale(yAccessor(data[0])) ]
  ];

  // now all of the interior points
  for (let i = 1; i < data.length - 1; i++) {
    result.push([
      xScale(xAccessor(data[i])) - connectorWidth,
      yScale(yAccessor(data[i - 1])),
    ]);
    result.push([
      xScale(xAccessor(data[i])) + connectorWidth,
      yScale(yAccessor(data[i])),
    ]);
  }
  // add the final point
  result.push([
    xScale(xAccessor(data[data.length - 1])),
    yScale(yAccessor(data[data.length - 1])),
  ]);
  return result;
}

Once that was done I could create the measure line as follows:

// Second attempt - Felton-style connectors
const yLine = d3
  .line()(
    generateFeltonLine(myData, xScale, xAccessor, yScale, yAccessor)
  );

Resulting in a much-improved (in my opinion) final version of the visualization:

Final histogram version - variant 1

The code I used to write this post is available on GitHub.

Please feel free to contact me if you have any questions or comments.

The 2013 Felton Bar Chart

Jeff Palmer — Thu, 18 Feb 2021 23:51:42 +0000

This bar chart was one of the more straightforward visualizations to implement, and it was a good place for me to get my feet wet with Svelte and D3.

For reference, Nicholas Felton’s original bar chart looks like this:

Bar chart detail from 2013 Annual Report (Nicholas Felton)

Simple and effective.

In order to recreate this, several things needed to be done:

Set up some sizing values to define the spacing of individual elements in each line.
Calculate the height of each line, which is then used to determine the height of the SVG.
Create a scale for the x-axis which will be used to draw both the solid and dashed lines.
Parameterize the highlight color so that the chart can be re-used with different colors.

I’ll cover how I did these things below.

Caveats

First of all, I’m sort of assuming that you have some idea of how Svelte works. If you don’t and are curious, you can take 30 minutes to work through the Svelte Tutorial. While there are a bunch of other great resources available, I’ve found that Svelte is simple enough that the tutorial is actually a really great starting point.

Also, I’m not going to cover the details of how Svelte and D3 interact. The key takeaway is that Svelte handles reactivity and DOM manipulation, and D3 is used for things like scales, etc. Paul Butler’s YouTube series about combining D3 with Svelte is a great starting point, as are Matthias Stahl’s presentations on Svelte and D3 and How do you set up visualization with Svelte.

SVG Sizing and Some CSS Gymnastics

In order to create an SVG that will maintain its aspect ratio when placed into containers of various sizes, you need to use a viewBox attribute to set the coordinate system of the SVG. This had to be one of the most confusing things I ran into when getting started. In fact, I’m still not sure that I have completely internalized the specifics of how SVG sizing/scaling works, so I’ll simply refer you to the resources I used to get started:

Understanding SVG Coordinate Systems and Transformations ( amazing )
How to Scale SVG
SVG: Font Size

So, one of the first things I needed to do was calculate the height of the overall bar chart based on its contents. I ended up explicitly setting the font size in my TypeScript code and using that to perform calculations with the text height, like calculating offsets, line heights, etc.

// Sizing in SVG user space
const width = 1200;
// Font size in user space
let textHeight = 30;

This needs to be made available to the CSS, but because Svelte doesn’t perform template substitution within the CSS <style> section, you need to do a little hoop jumping to get things to work. You can add a style attribute on the SVG (or any container) that defines CSS custom properties, like so:

<svg style="--label-size: {textHeight}px; --highlight-color: {highlightColor};">
  ...
</svg>

Once you’ve done this you can write CSS that uses those variables in your <style> section:

<style>
svg {
  width: 100%;
  height: auto;
}
.label {
  font-size: var(--label-size);
}
.highlight {
  fill: var(--highlight-color);
  stroke: var(--highlight-color);
  font-weight: bold;
}
.measure {
  stroke-width: 5px;
}
.dash {
  stroke-dasharray: 4px 12px;
  stroke-width: 3px;
}
</style>

This makes it straightforward to change the font size if I’m using the chart in a different context. You can even make textHeight a Svelte component property and pass it in as necessary. That said, ideally I’d be able to control all of this styling via CSS, so I’m still trying to figure out if you can do that. If you have suggestions on how to better manage consistent font sizes across both CSS and SVG text elements, please let me know.

Calculating the SVG Height

Okay, so with the textHeight available, we can calculate the height of an individual line, and then the overall SVG height.

Each line in the visualization is comprised of two parts: the name and metric value on top, and the bar visualization of the metric value below. The spacing of a line is the height of the text line plus some padding below the text that contains the bar visualization. I used linePadding and barOffset to keep track of these spacing elements.

// Margins within the SVG
let margins = {
  top: 50,
  bottom: 50,
  left: 50,
  right: 50,
};

// How many lines should be displayed?
const topN = 10;

// Space between the bottom of the text and the next line
const linePadding = 40;

// Space between the bottom of the text and the metric bar
// *must be less than linePadding*
const barOffset = 15;
const lineHeight = textHeight + linePadding;

// Compute the overall height
const height = topN * lineHeight + margins.top + margins.bottom;

Once we’ve calculated the height we can finally add the viewBox attribute to set the aspect ratio of the SVG.

<svg viewBox="0 0 {width} {height}">
  ...
</svg>

💣 One thing to keep in mind with SVGs is that the x-axis runs from left to right, but the y-axis runs from top to bottom, opposite of the coordinate system that most of us were taught in geometry. That means that (0, 0) is in the upper-left part of the SVG, and (100, 100) will be in the lower right. While not really a problem for this simple chart, this definitely messed with my head on another chart that had multiple y-axes pointing in different directions.

Drawing the Data

Now we can move on to drawing the elements of the bar chart. The structure of the data that I used in this example is as follows:

const data = [
  {
    name: "Elias Vančo",
    total_actions: 260,
  },
  {
    name: "Sommer Wolfe",
    total_actions: 337,
  },
  ...
];

Many D3 functions take accessor functions for data arrays, so I like to declare helper functions to access individual data elements so that I can easily change them later, if necessary.¹

const metricAccessor = (d) => d.total_actions;
const titleAccessor = (d) => d.name;

Creating the X-Axis Scale

D3 makes it easy to map values in your data domain to the coordinate system of the SVG (the range) through the use of scales. There are a number of different scales from which to choose, and for this visual I used scaleLinear to perform a basic linear mapping from domain to range.

import * as d3 from "d3";

// Grab the maximum value from the data for later use
const xMax = d3.max(data, metricAccessor);

// Create a D3 scale that maps domain values (from 0 to xMax)
// to SVG coordinates, respecting margins
const xScale = d3
  .scaleLinear()
  .domain([0, xMax])
  .range([margins.left, width - margins.right]);

// Calculate the index-based offset from the top of the SVG
// Note: Add textHeight to shift text below the starting point
// as text is placed according the the baseline
const yOffsetFn = (i) => lineHeight * i + margins.top + textHeight;

xScale is a function that calculates the absolute x-axis SVG positions for any data value in the domain. yOffsetFn serves the same purpose as xScale but calculates an index-based vertical offset for the y-axis to be used for line spacing.

Creating the Bar Elements

Once I had a scale that mapped domain values to SVG coordinates, I could iterate through the data to draw the bar chart.

First, I sorted the data and pulled the topN values into a separate variable using Ramda.

import * as R from "ramda";
// Sort and extract the first topN elements from the sorted collection
const diff = (a, b) => metricAccessor(b) - metricAccessor(a);
const topParticipants = R.take(topN, R.sort(diff, data));

I then used the Svelte #each construct to iterate through the topParticipants collection to generate SVG elements. In these elements I made use of the xScale and yOffsetFn functions to calculate absolute SVG positions.

<svg ...>
{#each topParticipants as participant, i}
  <text
    class="label"
    text-anchor="start"
    x={xScale(0)}
    y={yOffsetFn(i)}
  >{titleAccessor(participant)}</text>
  <text
    class="label highlight"
    text-anchor="end"
    x={xScale(xMax)}
    y={yOffsetFn(i)}
  >{metricAccessor(participant)}</text>
  <line
    class="dash"
    x1={xScale(0)}
    x2={xScale(xMax)}
    y1={yOffsetFn(i) + barOffset}
    y2={yOffsetFn(i) + barOffset}
  />
  <line
    class="measure highlight"
    x1={xScale(0)}
    x2={xScale(metricAccessor(participant))}
    y1={yOffsetFn(i) + barOffset}
    y2={yOffsetFn(i) + barOffset}
  />
{/each}
</svg>

The first text element is the left-aligned participant name, using xScale(0) for the x position.

The second text element is the right-aligned count of total actions. To do this I needed to calculate the maximum x position via xScale(xMax), and set the text-anchor attribute of the text element to end. That moved the element all the way to the right, but lined up the end of the text with the provided x coordinate.

The first line element is the dashed-line placeholder for the measure. The y position is offset by the additional barOffset to give the text above it a little breathing room. This line is drawn across the entire x-axis to produce visually consistent dashed lines across the entire chart. Since SVG elements are drawn in the order they are defined, with later elements overlapping earlier ones, this placeholder is hidden by the next element and there are no fancy line start/end calculations required.

The second line element is the bar representing the actual measure. It is drawn on the same y position as the previous line, but is slightly thicker to completely obscure the dashed line.

Once all of these elements were in place the entire visual was basically done. I did notice one detail that bothered me after a quick visual inspection:

Right side dash mis-alignment

The final dot at the end of the dashed placeholder lines didn’t match up to the right end of the scale. This was easily remedied by drawing the dashed line from right to left instead of left to right (by swapping the x1 and x2 values), which resulted in any left side alignment issues being hidden under the metric bars.

Here’s the final result:

Final bar chart version

The code I used to write this post is available on GitHub.

Please feel free to contact me if you have any questions or comments.

If this were really a general purpose chart library, you’d need to pass these functions in to this component, in the same way that generic sorting algorithms take comparison function parameters. ↩︎

Creating the Visual Studio Code 2020 Year in Review

Jeff Palmer — Thu, 18 Feb 2021 23:45:40 +0000

At the end of January I tweeted out a visualization that I had put together over the course of the preceding week to celebrate Visual Studio Code Day:

2020 Visual Studio Code Year in Review

I received some positive feedback from the community, which was really great, and I also got some questions about how I’d made it. I promised to provide more detail in a blog post (on a blog that I didn’t actually have 😆), so here we are. Better late than never, I suppose.

In this series of blog posts I’m going to cover why I made a 2020 VS Code Year in Review visualization, the inspiration behind it, the tools I used to build it, and some of the finer details of its construction. I hope you find it interesting!

Motivation

At the beginning of the year I was thinking about starting a new project that would allow me to experiment with a bunch of different technologies that I had been interested in. Specifically, I had been using Svelte for the last year and had really enjoyed it, and after watching Paul Butler’s YouTube series about combining D3 with Svelte I was eager to give it a try. Additionally, I had just heard about CSS Grid and thought that might be interesting to experiment with as a framework for structured visualizations.

I had also just read Everything You Ever Wanted To Know About GitHub (But Were Afraid To Ask)¹ and was excited by the prospect of digging into that dataset. The idea that the entire GitHub Archive dataset was available in a ClickHouse database that could be easily queried via SQL seemed too interesting to pass up.

Finally, over the past year I had learned to love Visual Studio Code and had been using it daily (sorry Emacs!). After learning that VS Code was one of the most active open source projects on GitHub, I thought it might be interesting to use that project as a lens into the GitHub dataset. Once I started digging around I realized that Visual Studio Code Day was on January 26th, which left me about a week to pull something together. This timing seemed ideal, and at the same time doable, so I set that as my deadline.

Understanding the Data

Before I started coding I took a look at what data was available in the ClickHouse GitHub Archive. I wanted to be able to get data directly from ClickHouse and avoid having to write a bunch of glue code to pull in information from other sources. (For example, I thought of some really interesting git commit history metrics that could be surfaced, but they would have required the processing and integration of data outside of ClickHouse.)

At the end of the Alexey’s article there’s a section on how the data was processed and loaded. All of the GitHub events that were captured are described, as well as how they were mapped to the ClickHouse schema. I ended up choosing three types of events that all GitHub users would be familiar with:²

Pull Requests
Branches
Issues

Pull requests and branches seemed interesting as they’re the meat and potatoes of how work gets done on GitHub and have a direct relationship with code contributions.

Issues, on the other hand, seemed like another story. I took a look at the VS Code repo and saw that they had 4.4k issues open, and over 100k closed! That is a ton of issue volume, and I was a little worried that there I was dealing with garbage data of some kind. Then I realized that, even if there is some bot activity causing those bug counts to be so high, that’s the reality of the project. So, issues it was.

Timezone Considerations

I started by writing some ad-hoc SQL queries to understand data distributions across these events and to look for any oddities in the data. Part of this was to understand who the top contributors were, and how activity was spread across the hours of the day. It became pretty clear that I was likely dealing with a timezone-related issue, as the majority of project effort seemed to be happening in the middle of the night (according to my timezone, at least).

Once I took a look at the profiles of the top project contributors I noticed something. For example, Benjamin Pasero, the primary Pull Request participant in 2020, had the following information on his GitHub profile:

I am a software engineer at Microsoft in Zurich, Switzerland since 2011. Our team started VS Code when it was still called Monaco.

Okay, so it seemed likely that a large part of the team was in Europe. That would explain why the hourly workload appeared as it did. I re-pulled the hourly data, this time converting to CET during aggregation. The results looked a lot more like a normal workday, so I decided to use CET as the primary timezone for all data. Luckily ClickHouse makes this easy.

For example, here’s the SQL to determine the date in 2020 with the most closed pull requests:

select
  toStartOfDay(toTimeZone(created_at, 'CET')) as most_closed_date
  , count(*) as total_closed_actions
from github_events
where repo_name='microsoft/vscode'
  and toYear(toTimeZone(created_at, 'CET'))=2020
  and event_type='PullRequestEvent'
  and action='closed'
group by most_closed_date
order by total_closed_actions desc
limit 1;

At this point I knew that I’d be able to get whatever data I needed out of ClickHouse. Now I needed to figure out how to represent it visually.

Design Inspiration

So, I had a timeline and a number of technical goals that I wanted to achieve. Given the short timeline I knew that I wouldn’t be able to successfully apply several new technologies while at the same time creating a compelling and innovative novel visualization. I needed to find an existing visualization to use as a template for the visual structure.

The Felton 2013 Annual Report

I’d always been a fan of Nicholas Felton’s work focused on the “quantified self,” and in particular found his 2013 Annual Report to be particularly compelling. I thought that there was likely a pretty direct mapping between the communication metrics he presented and those that I could calculate for the VS Code software development process.

Detail page from the 2013 Felton Annual Report (Nicholas Felton)

Additionally, the report’s distinctive visual style would force me to pay attention to the micro-decisions that had made when during its creation, which would be a great way to structure my experiments with Svelte and D3.

Mapping the Specifics

With the visual framework established, it was time to take a closer look to figure out which parts mapped best to the GitHub data.

I decided to map the original data metrics as follows, using pull requests as an example:

Original Metric	VS Code Metric
Total participants	same
Top N participants	same
Date with most interactions	Date with most opened PRs
Days without interactions	Date with most closed PRs
Aggregate send/receive activity by hour	Aggregate PR open/close activity by hour
Total sent	Total opened PRs
Total received	Total closed PRs
Aggregate activity by month	same
Total senders	Total PR openers
Total receivers	Total PR closers
Various type-specific metrics	Reopened PRs³
Longest gap in interactions	same

Getting the Data

At this point I knew what data I needed, so I started writing SQL. One of the benefits of one time development is that you can make a mess and not really care. Well, I certainly made a mess of my data retrieval code, but it got the job done.⁴

GitHub Archive Data in ClickHouse

Once I started getting query results, I needed to find a way to automatically save them to a format that could be easily parsed and integrated. It turns out you can write ClickHouse SQL statements that will save query results as JSON data:

select
  count(distinct(actor_login)) as total_participants
from github_events
where repo_name='microsoft/vscode'
  and toYear(toTimeZone(created_at, 'CET'))=2020
  and event_type in ('CreateEvent', 'DeleteEvent', 'PullRequestEvent', 'IssuesEvent')
into outfile '/data/total-participants.json'
format JSON;

Which will generate a JSON file that looks like this:

{
    "meta":
    [
        {
            "name": "total_participants",
            "type": "UInt64"
        }
    ],

    "data":
    [
        {
            "total_participants": "13749"
        }
    ],

    "rows": 1,

    "statistics":
    {
        "elapsed": 0.001163885,
        "rows_read": 611192,
        "bytes_read": 11183744
    }
}

This seemed perfect until I remembered that I was running the ClickHouse client in Docker⁵ which meant that the generated files were ending up in the Docker container. Five minutes of Googling later and I’d remembered how to mount local directories in Docker and had added it to my wrapper script. Now my data was being written to a local data directory on my Mac. 🎉

#!/bin/sh
LOCAL_DATA_DIR="$HOME/data"
docker run -it --rm \
    --mount type=bind,source=$LOCAL_DATA_DIR,target=/data \
    yandex/clickhouse-client:latest --secure "$@"

Ultimately this is how I obtained all of the data. Everything was saved off and then fed to TypeScript for parsing and further structuring/consolidation. Fine for a one-off, but as a result I couldn’t create visualizations for any other repository, even though I realized that that would be pretty interesting. I’ll likely revisit the data retrieval parts of this project in the near future to make that possible.

Augmenting User Data

As I mentioned earlier, I wanted to avoid writing a bunch of glue code to integrate data from other sources. In the end I really only needed to do this to transform GitHub usernames into real names (and I didn’t really need to do that—I just wanted the Top N lists to look a little more “human”). Given that I was only dealing with the top twenty-ish contributors, I ended up just doing this by hand using the GitHub /users/{username} REST API endpoint. I manually ran the Top N user queries and stored the responses in a map, which I then joined to other data as necessary via TypeScript.

Putting It Together

So at this point I had all of the data that I needed, and it was time to start building the visualizations.

CSS Grid

The layout heavy lifting was done by CSS Grid, which auto-sized the content in the All Events, Pull Requests, Issues, and Branches rows so that everything matched perfectly across columns. This was an absolute joy to work with, and once I got the initial structure into place I never really had to think about it again. I suppose most developers familiar with CSS are used to grids by now, but this was my first time using them and it certainly won’t be the last.

As an example, this was the css grid structure for the columns to the right of the Volume visualization:

.three-by-ten {
  display: grid;
  grid-template-rows: repeat(10, min-content);
  grid-template-columns: repeat(3, 370px);
  grid-auto-flow: column;
}

I then simply had to drop div elements into that container and everything flowed as expected. The only challenge that I remember having was wrapping my head around the column/row flow model at the very beginning. I found this guide to be a really helpful reference.

Svelte and D3

Developing these visualizations using Svelte and D3 was the bulk of my effort. There were a lot of lessons learned which I’m going to cover in a series of other posts:

Formatting Libraries

Aside from the visualization work done in D3, I used a couple of libraries to format some of the metrics:

To Words: Converts numbers into words.
Humanize Duration (TypeScript): Converts durations into words

For most everything else I used some variant of D3 number formatters or Date.toLocaleString and Date.toLocaleTimeString.⁶

Typeface

The typeface used in the report is Source Sans Pro, designed by Paul D. Hunt. I tried a variety of typefaces during development and found that I preferred the number styling of Source Sans Pro, especially the dotted zero. I enabled this via:

body {
  font-variant-numeric: slashed-zero;
}

In order for this to work as expected, I needed to use a non-Google typeface CDN due to apparent optimization on Google’s side. I ended up using Brick because they host unmodified versions that support OpenType features and have an elegantly simple API.

Color Palette

Finally, I thought it would be interesting to use the color palette of the most popular VS Code theme as the color palette of the visualization. It’s pretty easy to see themes ranked by installs, and at that time One Dark Pro was the most-installed non-icon theme.

Example of the One Dark Pro theme (theme site)

As it turns out the color palettes used in One Dark Pro and the original report are quite similar, so it was an easy mapping that ended up looking great.

Done!

With that last change everything was ready. I shared the result on Twitter, sending it out the night before hoping that it might get seen before things got started. Things were a little quiet until the next morning when someone on the VS Code team retweeted my message, at which point I got some really great feedback.

Thanks for taking the time to read a little about this project. If you have any thoughts or feedback you’d like to share, please feel free to contact me on Twitter or directly on my site.

Milovidov A., 2020. Everything You Ever Wanted To Know About GitHub (But Were Afraid To Ask), https://gh.clickhouse.tech/explorer/ ↩︎
I had hoped to include project releases as well, perhaps as an overlay on the various visualizations to provide additional insight into major project activities over the course of 2020, but I just wasn’t able to get something compelling completed in time. I’ll probably revisit that information in the future, as I feel like you can already see the build-up to major releases in the data, so it would be nice to be able to provide that additional context. ↩︎
There wasn’t an obvious “reopened” metric for branches, so I thought it might be interesting to see how many branches were opened by dependabot, the dependency management assistant that’s now part of GitHub. ↩︎
TODO: Fix data retrieval 😭 ↩︎
I forgot to mention that all of my data investigation takes place in Emacs, running sql-mode with the ClickHouse extension. This is a little wonky, and because I’m running on a Mac I needed this wrapper script to make things work as expected. ↩︎
I fully acknowledge that it is very weird to convert all data to CET and then choose to format all times with American AM/PM nonsense. My only excuse is that in the interest of time I discarded all locale considerations. 😬 ↩︎

My Development Setup and Tools

Jeff Palmer — Tue, 16 Feb 2021 19:35:53 +0000

Photo by Yancy Min

I’ve always found it interesting to see how others get things done, so here’s a little bit about my development setup, and how I like to go about building things.

Hardware

I’ve been programming since I was about 12 years old, and I’ve learned some hard lessons about ergonomics in that time. At this point I optimize for a comfortable and quiet development environment, and that’s pretty much it. I use a Jarvis standing desk along with a Herman Miller Aeron chair. My Mac is effectively silent, and I have a 27 inch 4K Dell monitor that makes screen reading very comfortable. I had a couple of different lighting options but have landed on the BenQ ScreenBar which attaches to the top of my monitor and provides a surprisingly satisfying low-profile task lighting experience.

Programming for extended periods led me to suffer from RSI, and in an attempt to alleviate these issues I started exploring the mechanical keyboards rabbit hole. My first experiment was to build an Atreus, which helped me learn about keyboard construction and the ins and outs of QMK and custom layouts. It’s a great keyboard that I still really love, but the lack of ability to “tent” the keyboard to reflect my wrist orientation was keeping me from really getting the full ergonomic benefit. So, I looked for a similar keyboard that was physically split and ended up with my current model, the Keeb.io Iris. It works well for me since the separate sides can be easily adjusted to match the angle of my wrists, and it runs QMK so migrating my custom layout from the Atreus was a snap.

I also swap between an Apple Magic Trackpad that I’ve had for ten years, and a Logitech MX Ergo Trackball. The Logitech is amazing but since it is a right-handed model it can’t be swapped to my left side; eventually my right hand will bother me enough that I need to pull out the trackpad and switch to my left hand.

Okay, on to the software side.

Software

If you were raised on the command line like I was, HomeBrew is a requirement to even get started with software development on MacOS. I couldn’t live without it, to the point that I have created a HomeBrew Bundle of my “must-have” packages that I use whenever I’m configuring a new account. Takes a little of the guesswork out of getting started. The other tool that I’ve come to really love is Oh My Zsh. I used to be a bash user, but at some point a co-worker suggested that I give zsh a try and I’ve never looked back.

For a long time I was an Emacs user, but these days Emacs has basically become an Org Mode hosting environment. I use Org to keep track of everything (after experimenting with a lot of alternatives), so that’s not going anywhere. For everything else I have moved to Visual Studio Code (with an emacs keybinding extension, of course; I’m not a masochist). I’m still amazed at how fast VS Code became the amazing experience that it is. Really great work.

I have some tips in the off-chance that they’re helpful:

Use a typeface that was developed specifically for programming. I use JetBrains Mono, but also really like Fira Code. They’ve been crafted with particular attention to the types of letter and symbol combinations that are common in programming, and in my opinion they really improve the day-to-day experience of reading code. Make sure to enable ligatures.
Disable mode-specific extensions globally, and only enable them in the workspaces where you need them. This avoids issues with competing extensions which can happen if you experiment with a lot of technologies.
Adjust the explorer zoom level as indicated in this Stack Overflow answer to change the size of the explorer to something that better suits how you want things. I have my explorer smaller than the default because I don’t actively use it, but when I do I want to see a lot of information.

The other tool that I find myself reaching for frequently on a Mac is Docker. Homebrew is great when you’re running things on your own machine, but when you need to start thinking about deployment, Docker makes that process a lot easier.

Finally, I use Hugo for blogging with my forked version of the Blogophonic theme that allows it to be installed like a normal Hugo theme.

Miscellany

I’ve been doing a lot of front-end focused work over the past year or so and it’s been really great to see how the Javascript ecosystem has evolved. The biggest impact for me was to start using Svelte for all of my front-end development. It is amazing. I highly recommend that you check it out.

There are a ton of javascript charting libraries out there (I’m partial to Vega Lite), but recently I’ve been focused on building custom information visualizations. The tool that I use to do that is D3.js. When you pair D3 and Svelte’s reactivity it’s a really powerful and satisfying development experience.

I’m a big fan of Bulma when I want to quickly prototype something. I’ve recently started using Tailwind CSS and, although I haven’t used it enough to have a strong opinion, I’m liking what I have seen so far.

So, that’s everything that I thought it might be interesting to cover. If you have any questions or thoughts, or would like to know more about an area that I didn’t cover, please don’t hesitate to contact me!