DEV Community: Andrew Nesbitt

"Ecosyste.ms 2023 End of Year Update"

Andrew Nesbitt — Sat, 23 Dec 2023 17:24:00 +0000

This year has been another very productive one for Ecosyste.ms, we've designed, built, deployed and maintained 16 significant services to support, sustain, and secure critical digital infrastructure.

Some of the stats from the biggest services:

Packages:

Registries: 59
Packages: 8,647,405
Versions: 93,009,665
Namespaces: 1,284,530
Maintainers: 1,412,065
Database size: 245GB
Background jobs processed: 688,025,502

Repos:

Hosts: 784
Repositories: 183,755,645
Owners: 10,543,833
Tags: 188,198,256
Manifests: 190,349,152
Dependencies: 14,051,505,152
Database size: 2.49 TB
Background jobs processed: 727,979,154

Timeline:

Events: 7.43 Billion
Database size: 8.69 TB

Some new services we deployed this year:

Issues - An open API service for providing issue and pull request metadata for open source projects.
Commits - An open API service providing commit metadata for open source projects.
Summary - An open API service for producing an overview of a list of open source projects.
Docker - An open API service providing dependency metadata for docker projects.
Papers - An open API service providing mapping between scientific papers and software projects that are mentioned in them.
OST - An open API service for discovering and reviewing projects for opensustain.tech.

We also helped launch ClimateTriage.com with Opensustain.tech to help people discover a meaningful way to contribute to open source projects focused on climate technology and sustainability powered by the whole range of ecosyste.ms services.

And we published some very large open data releases, check them all out on the blog: https://blog.ecosyste.ms/2023/08/24/open-data-releases.html

We experienced some growing pains over the year, increases in API usage and the pure amount of data being indexed (over 10TB of data across all services) required upgrades and changes to infrastructure to keep it running well but things are quite stable now.

We also used up the majority of our grant from Plaintext group and are exploring ways to continue funding further development of features and level up the services.

A big project we have planned starting in the new year is to work with OSC to build tools to measure the impact of funding open source software and tracking changes in metrics over time of open source software projects that are part of OSC.

We are always looking to work with more partners to help them add support for more ecosystems to their products as well as expanding the number of ecosystems and repository hosts that we index.

Making 24 Pull Requests more inclusive for 2018

Andrew Nesbitt — Thu, 29 Nov 2018 20:38:43 +0000

For the past 6 years, 24 Pull Requests has been encouraging developers to give back to the open source projects they use. The idea was simple: “Send 24 pull requests between December 1st and December 24th”, giving little gifts of code throughout December.

Since the initial launch of the project in 2012, open source usage has exploded with millions of people contributing to projects around the world but one area where open source continues to struggle with is in the diversity of contributors.

One problem that compounds the diversity issue is that tools and metrics that measure open source contributions often only acknowledge code contributions, ignoring all the other ways that people can contribute to open source projects.

The GitHub contribution graph is a prime example, which only recognises a handful of kinds of contribution. This can make it difficult for people to get the recognition they deserve, potentially hampering their motivation and even hurting their job prospects.

GitHub profiles don’t show the full picture of open source contribution

It took me 6 years to realise that 24 Pull Requests has been part of that problem too. In previous years, you could only participate if your contributions could be presented in the form of a pull request. This is particularly difficult for people are contributing by:

Organising an event
Answering questions on Stackoverflow
Supporting and mentoring new contributors
Speaking at a conference or user group
Managing the governance aspects of a project
Writing blog posts about how to use a project
Mailing stickers to contributors
Donating money to help pay a projects infrastructure costs
and many other ways of contributing without writing code…

Those kinds of tasks are often under taken by women, people of colour and other underrepresented groups, only making things worse as their contributions become invisible to many people.

So this year we’re making the biggest change to 24 Pull Requests since the start of the project: you’ll be able to record every kind of contribution you make to open source, not just pull requests.

There’s a new form on the site that allows you to record the contributions you’ve made each day that wouldn’t usually make sense as a pull request:

The new 24 Pull Requests contribution form

Those non-pullrequest contributions are treated exactly the same as pull requests across the site, showing up on your profile and the homepage, in other words all contributions are considered equal.

User profile page showing a mix of pull requests and non-pull request contributions

With this change, we hope to use 24 Pull Requests to encourage all kinds of contributions to open source projects, it also opens up the ability to track contributions to open source projects that aren’t hosted on GitHub.com.

In summary, here’s how to track your contributions this year:

For contributions that are pull requests, everything will work the same as previous years, just login and sync your pull requests from the dashboard.

For contributions that you’ve made to an open source project that aren’t pull requests, you can record them in the new contribution form.

See you on December 1st!

Untangle your GitHub Notifications with Octobox

Andrew Nesbitt — Sun, 25 Nov 2018 11:37:15 +0000

As an active contributor and maintainer of many open source projects on GitHub, I struggle to keep up with the various in-progress issues and pull requests across multiple repositories and find it hard to see an overview of everything I’m involved in. If you manage or contribute to any number projects on GitHub, I’m betting you probably have the same issue.

This is because GitHub Notifications are marked as read and disappear from the list as soon as you load the page or view the email of the notification. This makes it very hard to keep on top of which notifications you still need to follow up on.

Most open source maintainers and even some GitHub staff end up using a complex combination of filters and labels in Gmail to manage their notifications from their inbox. If, like me, you try to avoid email, then you might want to try Octobox.

Octobox is designed with exactly that problem in mind. Firstly it adds an extra “archived” state to each notification so you can mark it as “done”. If new activity happens on the thread/issue/pr, the relevant notification will pop back into your inbox. You can also star notifications that are special to you.

This puts the control of when to clear notifications back in your hands, you can work through your inbox at your own pace. It also means you can always find old notifications, which, on GitHub, disappear a few days after you’ve read them.

The other problem that GitHub power users struggle with is the sheer number of notifications they get, some users are getting 200+ notifications every single day. Octobox helps to tackle this in a number of ways:

Filters: you can filter the notifications in your inbox in pretty much any way you can imagine; by repository, organization, type, action, state, CI status and reason and keep notifications from bots alongside your regular labels, author and assignees.
Multi-select: Archive and mute a whole bunch of notifications at once, allowing you to stay on top of even the noisiest repositories with ease.
Search: Combine a wide range of powerful search filters help you get straight to the notification you’re looking for and focus on just what you need, then pin your favorite searches to the sidebar for easy access.
Keyboard shortcuts: Quickly navigate, triage and manage your notifications like a pro using Gmail-inspired keyboard shortcuts for every function, no mouse required.

Octobox started life as a side project in December 2016, since then it’s grown to help over 10,000 people manage over 4 million notifications and the project has been downloaded almost half a million times by developers hosting their own versions. It’s now one of the most popular open source tools on GitHub.

Last month Ben and I announced that we’re going to start working full time to make Octobox a truly sustainable open source project that can financially support itself and the community.

Today we’re expanding the scope of Octobox and **launching on the GitHub Marketplace**, offering new, paid enhancements for private repositories on Octobox.io with a two week free trial period.

You can also get the same enhancements for private repositories by donating or becoming a sponsor on Open Collective, same price but the support goes directly to the community.

We’ve also got a whole host of new features planned for the coming months, including:

Snoozing notifications until later
Gmail-style automated notification filters
Localization and internationalization
View comment threads right in the Octobox interface
Team discussions and direct user-to-user messaging
Highlighting important notifications to you
Allow replying to an issue/pull request directly from Octobox

To keep up with everything that’s going on and even contribute directly to the project, check out the open source project on GitHub.

Untangle your GitHub notifications with Octobox

Exploring Unseen Open Source Infrastructure

Andrew Nesbitt — Fri, 24 Feb 2017 15:41:44 +0000

Whilst working on Libraries.io, I often stumble across libraries that appear to be used by an incredible amount of open source projects but often don't have any of the usual signs of being a popular project on GitHub.

Take debug_inspector for example:

25 stars
21 commits
4 contributors
5 watchers
4 forks
2 open issues
Last commit over 2 years ago

At face value if the GitHub page you'd be forgiven for mistaking it as a small, project that's barely used, when in fact it's listed as a dependency in over 111,000 open source projects!

Libraries.io has a number of different pages for exposing interesting and unexpected lists of libraries, including ones with a low bus factor and ones that have been yanked from their package manager, so I thought I'd add one to show the most unappreciated but highly used libraries.

This afternoon I shipped the Unseen Open Source Infrastructure page: https://libraries.io/unseen-infrastructure

To paraphrase Arfon Smith on Request for Commits #3, “Stars on GitHub are a measure of attention, more akin to a Like on Facebook than a measure of quality or usage”, this page shows hundreds of projects that are depended upon by at least 1,000 other open source repositories but have less than 100 stars.

Any of these projects could be the next left-pad or Heartbleed where an underlying, critical library is highly used but has very little attention paid to it. An unnoticed security issue or abandoned project could potentially could result in hundreds of thousands of affected software applications.

You can help these projects and the communities that depend upon them by reviewing the code for these libraries, helping out with open issues, sharing them on social media and thanking the maintainers for their hard, often unrewarding work to keep things running behind the scenes.

Something that Ben and I are planning to do a lot more of over the next year and beyond is help to highlight and support the open source software that is critical to today's technology infrastructure.

Another area that needs exploring is system level package managers like apt and yum, which contain even more important and often overlooked libraries that often aren't hosted on a social platform like GitHub but are still critical to world of software.

If you'd like to get involved, the whole project is open source and we'd love to help you get started contributing, or if you'd like to build tools on top of all this data, check out the Libraries.io REST API

As always, follow us on Twitter at @teabass, @benjam and @librariesio for more updates.