DEV Community: Patrick DeVivo

Exploring Git Repos With MergeStat 🔬

Patrick DeVivo — Wed, 16 Feb 2022 01:08:22 +0000

mergestat is an open-source tool that allows users to run SQL queries on the contents and history of git repositories.

We recently added a summarize command, which prints out a summary report of various aspects of a repo. For instance, mergestat summarize commits will output commit stats grouped by author.

You can pass --start and --end flags to generate these stats for various periods of time, and filter by a path pattern to only match commits that modify certain files.

For instance, if we wanted to see a total summary of commits and authors in the Redis codebase, we could run:

mergestat summarize commits --repo https://github.com/redis/redis

Commits             10,950
Non-Merge Commits   9,533
Files Δ             2,307
Unique Authors      711
First Commit        14 years ago (2009-03-22)
Latest Commit       2 days ago (2022-02-14)


Author                 Commits   Commit %   Files Δ   Additions   Deletions   First Commit                 Latest Commit
antirez                5,999     54.79%     1,311     553,839     284,018     14 years ago (2009-03-22)    2 years ago (2020-06-25)
Salvatore Sanfilippo   1,038     9.48%      336       62,815      15,823      12 years ago (2011-04-11)    2 years ago (2020-06-25)
Pieter Noordhuis       510       4.66%      209       44,927      16,860      13 years ago (2010-03-03)    9 years ago (2013-07-10)
Oran Agra              416       3.80%      654       50,531      16,960      7 years ago (2015-03-31)     3 days ago (2022-02-13)
Yossi Gottlieb         189       1.73%      263       19,059      5,047       10 years ago (2012-12-24)    20 days ago (2022-01-27)
zhaozhao.zz            153       1.40%      49        1,607       670         5 years ago (2017-09-20)     10 months ago (2021-05-12)
Matt Stancliff         151       1.38%      126       10,492      3,057       9 years ago (2014-02-10)     8 years ago (2015-02-24)
artix                  122       1.11%      13        11,982      3,735       5 years ago (2018-01-12)     3 years ago (2020-02-18)
Binbin                 96        0.88%      183       4,707       1,556       10 months ago (2021-04-28)   3 days ago (2022-02-13)
Itamar Haber           81        0.74%      76        5,930       1,042       7 years ago (2016-01-20)     a year ago (2021-03-24)
Wen Hui                77        0.70%      60        2,293       522         2 years ago (2020-07-15)     7 days ago (2022-02-09)
sundb                  59        0.54%      103       4,442       2,109       a year ago (2020-10-28)      5 days ago (2022-02-12)
Guy Benoish            56        0.51%      229       28,525      24,365      6 years ago (2016-10-31)     2 years ago (2020-05-03)
guybe7                 56        0.51%      445       32,647      6,003       2 years ago (2020-08-18)     8 days ago (2022-02-08)
WuYunlong              54        0.49%      38        504         133         5 years ago (2017-07-15)     a year ago (2021-02-09)
yoav-steinberg         46        0.42%      85        4,309       2,359       2 years ago (2020-07-16)     5 days ago (2022-02-11)
Huang Zhw              45        0.41%      39        1,323       546         a year ago (2021-03-06)      3 months ago (2021-11-28)
Madelyn Olson          45        0.41%      83        4,131       1,499       2 years ago (2020-07-21)     16 days ago (2022-01-30)
Madelyn Olson          38        0.35%      32        2,205       1,391       4 years ago (2018-06-19)     2 years ago (2020-05-21)
Dvir Volk              34        0.31%      9         873         270         6 years ago (2016-03-31)     5 years ago (2018-02-14)
Viktor Söderqvist      34        0.31%      91        5,597       1,578       a year ago (2021-01-13)      8 days ago (2022-02-08)
Wang Yuan              32        0.29%      47        2,520       649         2 years ago (2020-08-08)     a month ago (2022-01-18)
dejun.xdj              30        0.27%      9         140         92          4 years ago (2018-04-10)     4 years ago (2018-08-22)
Geoff Garside          29        0.26%      9         219         109         11 years ago (2011-06-17)    10 years ago (2012-10-17)
antirez                25        0.23%      30        433         106         12 years ago (2010-04-22)    11 years ago (2011-06-02)
...686 more authors

We see that the first commit was roughly 14 years ago and there have been over 700 authors in the lifetime of the project (noting that sometimes we double count if different email addresses are used by the same author).

We also see that antirez, Salvatore Sanfilippo, is the most prolific committer by absolute number of commits (and likely additions/deletions as well). This makes sense, as he started the project back in 2009. However, we also note that his last commit was roughly 2 years ago. Again, this makes sense given his announcement to leave Redis, also around 2 years ago.

By passing in a --start flag, we can limit the report to only look at commits in the past year:

mergestat summarize commits --repo https://github.com/redis/redis --start "-1 year"

Commits             906
Non-Merge Commits   903
Files Δ             1,428
Unique Authors      173
First Commit        a year ago (2021-02-15)
Latest Commit       2 days ago (2022-02-14)


Author                        Commits   Commit %   Files Δ   Additions   Deletions   First Commit                 Latest Commit
Oran Agra                     119       13.13%     521       20,237      7,941       a year ago (2021-02-15)      3 days ago (2022-02-13)
Binbin                        96        10.60%     183       4,707       1,556       10 months ago (2021-04-28)   3 days ago (2022-02-13)
Yossi Gottlieb                58        6.40%      121       3,383       1,864       a year ago (2021-02-15)      20 days ago (2022-01-27)
Huang Zhw                     45        4.97%      39        1,323       546         a year ago (2021-03-06)      3 months ago (2021-11-28)
yoav-steinberg                44        4.86%      81        4,235       2,294       a year ago (2021-03-03)      5 days ago (2022-02-11)
Wen Hui                       41        4.53%      43        1,409       328         a year ago (2021-02-22)      7 days ago (2022-02-09)
guybe7                        41        4.53%      441       31,325      5,702       a year ago (2021-02-23)      8 days ago (2022-02-08)
sundb                         40        4.42%      90        4,142       1,870       a year ago (2021-02-19)      5 days ago (2022-02-12)
Madelyn Olson                 30        3.31%      61        3,165       1,039       a year ago (2021-02-24)      16 days ago (2022-01-30)
Viktor Söderqvist             24        2.65%      83        3,675       1,035       a year ago (2021-02-15)      8 days ago (2022-02-08)
Wang Yuan                     20        2.21%      38        1,882       449         a year ago (2021-03-25)      a month ago (2022-01-18)
Meir Shpilraien (Spielrein)   17        1.88%      46        3,088       698         3 months ago (2021-11-28)    14 days ago (2022-02-02)
chenyang8094                  16        1.77%      44        4,113       863         9 months ago (2021-06-16)    5 days ago (2022-02-12)
Ozan Tezcan                   12        1.32%      49        1,200       235         4 months ago (2021-11-11)    18 days ago (2022-01-29)
YaacovHazan                   12        1.32%      44        1,114       285         a year ago (2021-03-01)      2 days ago (2022-02-14)
Meir Shpilraien (Spielrein)   9         0.99%      18        2,107       316         a year ago (2021-03-16)      5 months ago (2021-10-06)
zhaozhao.zz                   9         0.99%      20        493         124         10 months ago (2021-05-17)   a month ago (2022-01-17)
Itamar Haber                  8         0.88%      78        712         262         6 months ago (2021-09-09)    a month ago (2022-01-09)
ZhaolongLi                    8         0.88%      6         18          38          9 months ago (2021-06-07)    7 months ago (2021-07-25)
Bonsai                        7         0.77%      17        86          21          a year ago (2021-02-28)      10 months ago (2021-05-19)
Harkrishn Patro               7         0.77%      33        1,551       211         a year ago (2021-02-17)      17 days ago (2022-01-30)
DarrenJiang13                 6         0.66%      11        40          42          10 months ago (2021-05-19)   5 months ago (2021-10-18)
Yoav Steinberg                6         0.66%      652       175,144     86,772      5 months ago (2021-10-10)    5 months ago (2021-10-11)
perryitay                     6         0.66%      58        1,539       313         8 months ago (2021-07-11)    a month ago (2022-01-20)
David CARLIER                 5         0.55%      4         52          5           6 months ago (2021-09-11)    a month ago (2022-01-19)
...148 more authors

And we see over 100 unique contributors, with other maintainers now the primary committers.

It's important to note that in general, commits are not fungible - and absolute counts of them don't tell us much beyond, well, counts of commits. It may be useful to compare relative counts within the context of a project (like we do here), but using commit counts to tell us about "involvement" or "importance" of a contributor can be problematic, for many reasons.

If this is interesting, you can read more about our mission or try getting started with our CLI.

We'll be adding more summary report functionality for:

Blameable Lines
GitHub PRs
GitHub Issues
Files Modified
..and more!

Daily Commit Counts as a Terminal Graph w/ Emojis 🚀

Patrick DeVivo — Mon, 06 Jul 2020 22:19:42 +0000

Here's a use case of a project I recently made public. It's called gitqlite and lets users run SQL queries on git repositories.

Display a terminal chart of commit counts for a repo (facebook/react, here) by day, for the past N days (30 in this case):

cat query.sql | gitqlite --repo https://github.com/facebook/react
+------------+-----+-------+----------------------------
|    DAY     | DOW | COUNT |          COMMITS           
+------------+-----+-------+----------------------------
| 2020-07-06 | Mon |     6 | ✅✅✅✅✅✅               
| 2020-07-05 | Sun |     0 |                            
| 2020-07-04 | Sat |     0 |                            
| 2020-07-03 | Fri |     0 |                            
| 2020-07-02 | Thu |     2 | ✅✅                       
| 2020-07-01 | Wed |    13 | ✅✅✅✅✅✅✅✅✅✅✅✅✅ 
| 2020-06-30 | Tue |     8 | ✅✅✅✅✅✅✅✅           
| 2020-06-29 | Mon |     1 | ✅                         
| 2020-06-28 | Sun |     0 |                            
| 2020-06-27 | Sat |     0 |                            
| 2020-06-26 | Fri |     2 | ✅✅                       
| 2020-06-25 | Thu |     2 | ✅✅                       
| 2020-06-24 | Wed |     0 |                            
| 2020-06-23 | Tue |     5 | ✅✅✅✅✅                 
| 2020-06-22 | Mon |     0 |                            
| 2020-06-21 | Sun |     0 |                            
| 2020-06-20 | Sat |     0 |                            
| 2020-06-19 | Fri |     2 | ✅✅                       
| 2020-06-18 | Thu |     0 |                            
| 2020-06-17 | Wed |     1 | ✅                         
| 2020-06-16 | Tue |     2 | ✅✅                       
| 2020-06-15 | Mon |     1 | ✅                         
| 2020-06-14 | Sun |     0 |                            
| 2020-06-13 | Sat |     1 | ✅                         
| 2020-06-12 | Fri |     3 | ✅✅✅                     
| 2020-06-11 | Thu |     1 | ✅                         
| 2020-06-10 | Wed |     0 |                            
| 2020-06-09 | Tue |     0 |                            
| 2020-06-08 | Mon |     0 |                            
| 2020-06-07 | Sun |     0 |                            
+------------+-----+-------+----------------------------

The SQL in query.sql looks like:

WITH RECURSIVE
days(day) AS (
    SELECT strftime('%Y-%m-%d', DATETIME('now'))
    UNION ALL
    SELECT strftime('%Y-%m-%d', (SELECT DATETIME(day, '-1 day'))) FROM days
    LIMIT 30
)
SELECT
    days.day,
    substr('SunMonTueWedThuFriSat', 1 + 3*strftime('%w', days.day), 3) as dow,
    IFNULL(count, 0) AS count,
    CASE count IS NULL WHEN true THEN '' ELSE printf('%.' || (IFNULL(count, 0)) || 'c', '✅') END AS commits
FROM days LEFT JOIN (
    SELECT strftime('%Y-%m-%d', committer_when) AS day, count(*) AS count FROM commits GROUP BY day
) c ON c.day = days.day

More examples are available in the README

Kubernetes' 2k+ TODO Comments

Patrick DeVivo — Sun, 15 Dec 2019 16:24:24 +0000

This Medium story takes a look at the 2,000 TODO comments in the Kubernetes' source code; where the average age of a TODO comment is 2.3 years. Check it out if you're interested!

https://medium.com/@augmentable/looking-at-kubernetes-2k-todo-comments-b2db42dc7fdb

Understanding the TODOs in your Codebase

Patrick DeVivo — Sat, 09 Nov 2019 02:39:06 +0000

I've been working on a side project called tickgit, which is an experiment in code-based project management.

I put together a small UI which can present the TODOs in public github repos, see facebook/react for instance: https://todos.augmentable.dev/?repo=https://github.com/facebook/react

I'm wondering if this is a useful point-of-view into a codebase, as a way of summarizing items that need to get done. What do you think? Could you see it being useful for the codebases you work on?

I'm thinking about adding some search functionality and parsing out the age of TODOs as well, to have a sort order available...

Managing tickets and todos within your codebase - does it make sense? Looking for feedback

Patrick DeVivo — Fri, 25 Oct 2019 03:12:54 +0000

Hey everyone - I've begun working on a side project called tickgit, which I'm hoping is something that could be useful for anyone writing code and using a git repo (presumably many folks here!)

I would love any feedback, and am looking for ways to make this tool useful. It's fairly rudimentary in its current state: augmentable-dev/tickgit but I have high hopes for making it a great developer experience for managing tickets, todos, checklists (and whatever other project-manage-y type things) might exist in a codebase or git repo.

Please share your thoughts! The README has much more detail, and you can run a basic version of it today (installable via homebrew).

the tldr for how it works currently:

$ tickgit todos ~/Desktop/facebook/react
...
TODO: does clang define __GNUC__ ?
  => /Users/.../Desktop/facebook/react/scripts/perf-counters/src/portability.h:34:3

TODO: FIXME! Without this implemented properly, the JIT
  => /Users/.../Desktop/facebook/react/scripts/perf-counters/src/portability.h:133:3

TODO: it's awkward to create a bundle for this but if we don't, the package
  => /Users/.../Desktop/facebook/react/scripts/rollup/bundles.js:454:7

TODO: this is too permissive.
  => /Users/.../Desktop/facebook/react/scripts/rollup/validate/eslintrc.umd.js:20:7

TODO: type it.
  => /Users/.../Desktop/facebook/react/scripts/shared/inlinedHostConfigs.js:25:27

123 TODOs Found 📝

for getting a snapshot and pointers to TODOs.

# rocketship.tickgit

goal "Build the Rocketship 🚀" {
    description = "Finalize the construction of the Moonblaster 2000"

    task "Construct the engines" {
        status = "done"
    }

    task "Attach the engines" {
        status = "pending"
    }

    task "Thoroughly test the engines" {
        status = "pending"
    }
}

and

$ tickgit status
=== Build the Rocketship 🚀 ⏳
  --- 1/3 tasks completed (2 remaining)
  --- 33% completed

  ✅ Construct the engines
  ⏳ Attach the engines
  ⏳ Thoroughly test the engines

as an example of defining and managing simple tickets in your codebase