DEV Community: Scarf

Alex Biehl: Open Sourcing a Tool to Generate Haskell Server Stubs

Christina Gorton — Mon, 25 Jul 2022 14:37:04 +0000

Scarf Sessions is a new stream where we have conversations with people shaping the landscape in open source and open source sustainability. This post will give a recap of the conversation I had with our guest Alex Biehl

Alex is a software engineer at Scarf who recently open sourced a tool to generate Haskell server stubs called Tie.

For the full interview please see the video posted on our YouTube channel.

Stream Recap

Can you talk more about what programming languages the engineering team at Scarf uses?

Alex: “We use a different mix of languages. Though there are two things that stand out for the main products, which are Haskell for the backend like data processing metrics, pipeline and all. And the other one is of course, JavaScript for the front-end, like the UI of the product.

“Also one of the defining aspects of our infrastructure is that we use something called Nix to assemble all the infrastructure bits. We use it to build out our binaries, build the containers from those binaries, help manage and distribute our artifacts for deployment.”

Can you describe the difference between Haskell and other programming languages?

Alex: “Haskell is known as a functional programming language. One of the legends of Haskell is that if it compiles, it works okay. Which is, of course, not always true, but it's a good approximation for a lot of things.

“And coming from that background our founder, Avi, decided that it would be a good foundation for a product.”

Did you work in open source before joining Scarf?

Alex: “I've always been an active open source contributor. Most of the time in the Haskell ecosystem. I’ve been contributing to a lot of Haskell infrastructure as well.

“I was a maintainer for Haddock. It's the de facto standard documentation tooling for Haskell. And I also contributed to the core build system called Cabal and also contributed a few small patches to GHC itself, as well as, all the ecosystem around it."

Christina: “I'm always interested in how people get started in open source.

"Was it something that you were just using these things, so you decided to contribute to the project?

“Or were you specifically like, “Hey, I want to do something with open source."

“How did you get started?"

Alex: “I think I'm kind of a pragmatist in that regard. I noticed something isn't working and now you have the chance to either, you know, build it yourself or piggyback off what's already there and improve it.

“And the pragmatic choices of course, to not reinvent the wheel and improve what's there and iterate on that. And that's what brought me to open source and what brings me to contributing to open source even to date.“

Where did the idea for Tie come from?

Scarf Redesign

Alex: “At scarf we are in the process of revamping our UI for the product.
And we thought it would be a good time for us to also not only revamp the user interface, but also revamp the underlying parts that serve those UIs. Right now the API is written in a very ad hoc way.

“We had a chance to revamp the API, we looked at what we had and analyzed all the shortcomings and disadvantages that we saw. And we found that defining your API in Haskell itself is prone to the usual developer habits.

“Something that developers do is build out patterns and mechanisms to type out lines of codes. They reduce duplication and they make things. They consolidate things to make it shorter, to make it more correct, to avoid repeating themselves.

“It happens that the API itself gets less and less explicit because it's hidden behind all those structures that we build in order to easily write the API itself.

“That made it very hard for us to actually find out what the API itself is. Like what is the actual route for this particular endpoint? You have to look it up and dig deep in order to find out. And another issue we found is we often ended up with something I call “scope creep”. Sometimes types that are meant for the public API contract of the API make it into the domain and business logic.

“Which means that in the long term you are coupling your API contract, which you want to support for your customers, to the internal logic of your product. And every time you want to make a seemingly internal change to your logic you might accidentally change the public contract as well."

Developing Tie

“This brought me to two realizations. First, we want an explicit human readable API contract, which we can share with our customers.
Second, I wanted to settle it on co-generation. That means I don't want to type out these very boring interface types or contract types myself. I really want a tool to have it generated for me so that I never ever accidentally make a change in the Haskell code that would break the contract. With those goals in mind the most common choice today for these kinds of things is Open API. Open API denotes domain specific language to model web APIs.

“Where Tie comes into play is that it understands Open API. And Tie takes the Open API format and generates Haskell code from it. It does this in such a way that we only have to implement the actual list for the specific routes and don't have to deal with all those serialization logics and peculiarities.“

What did you find hard or unexpected about open sourcing Tie?

Alex: “One of the things that I fell really short on at first was writing good documentation.

I hadn't even created a good example in the repository when I first open sourced the project. It didn’t have a “How to use” section although we were already using it ourselves in production.

“It was born as a weekend project after two or three days of work and we were putting it into production already after that weekend. So I quickly had other support issues I had to work on before I had time for documentation.

Christina: “This is why bigger teams have dedicated people for documentation, I totally understand. And to Alex's credit he very quickly messaged me because I have been an open source community manager in the past and he asked me how do I make a good readme?

“I think that's a really good point though. Often maintainers or someone who really wants to open source a tool they were using and it was helpful for them don't have time for everything else, like the documentation.
‍Do you have any advice for anyone putting anything out?

Alex: “Don’t let the perfect be the enemy of good."

"You should add documentation and all the parts that got lost on the initial phase of the project. But you know, what's more important I think is, getting initial feedback on the project itself.”

What’s in the future for Tie?

Alex: “Tie is not finished. It doesn't support all of Open API’s setup. The standard is vast so this is the call for contribution right here.

“Help us improve it, file issues. We can definitely help and guide any changes and improvements.”

What else are you working on?

Alex: My pet peeve is proper co-generation for contracts for external systems, for example databases. People like to use ORMs and generators.”

Christina: “Can you explain what ORMs are?”

Alex: “Object relational mappings method. Like an active record in Ruby, Django, etc. And I want a tool that takes your migration folder and generates for you the types very similar to what Tie does. I want the tool to generate types that reflect the records, the structure of the tables in the database then also allows you to basically ingest SQL.

And generate small records around that SQL which are typed. So that you can use your SQL to write theories, to define tables, to alter tables and so on, but still have the good old type system to help so that you don't mix up your types”

For more information on Tie please visit our repository and consider contributing to the project to help improve it.

Are you helping to shape the landscape in open source? Interested in joining us on a Scarf Sessions Stream? Email Christina at christina@scarf.sh.

To learn more about Scarf:
Website

To follow Alex:
Twitter
GitHub

To follow Christina:
Twitter

Tanner Linsley: Building Sustainable Open Source Projects

Christina Gorton — Thu, 21 Jul 2022 14:00:26 +0000

Tanner is the Co-founder & VP of UI/UX at Nozzle. He is also the creator and maintainer of the open source projects, TanStack. He has been active in open source for over 5 years contributing to the TypeScript, JavaScript, and React communities. He joined us to explain how he got started in open source and how he has made working in open source sustainable.

For the full interview please see the video posted on our YouTube channel.

Stream Recap

What would you attribute to the growth of TanStack?

Tanner: "React Table and React query. I built React table and people started to like it, which was my initial entry into open source. React Query was a bigger push into a lot more users. It was a very general purpose library for React. So it drew a very large audience. I thought it was pretty great. Apparently other people did too."

How did you promote your open source projects?

Tanner mentioned that he literally just put his work out on Twitter when he was ready to make it open source.
Tanner: “I published it at 10:00 PM. I got on Twitter and I said, Hey, everybody, check this out. You know? That's kind of how I've done a lot of my releases too. I like doing it that way. Like I don't need to make too big of a fuss over it.”

What is React Location?

Tanner: “Location is an alternative router for React that is not React Router, I guess, because there's not a lot of them out there. Remix came out and they had a lot of great asynchronous routing features in Remix. I wanted to bring those into a routing experience and I needed more control over search params to the extent that no other router I've seen has.

"So I built this custom router from the ground up to have really good search param support and control over URLs.”

Getting Started in Open Source

Tanner expressed that he generally creates projects that he needs for his own work and then open sources them for others to use.

Avi: “That makes a lot of sense. I very much resonate with the experience of building tools that you need for yourself or tools that your company needs as a way to build things that are valuable to other people in general. If it's useful to you, it might be useful to others too and that is exactly how I got my first projects that had any kind of traction beyond just me using them. They were tools that I wanted to build as well. “

Avi mentioned that people ask him how to come up with ideas for an open source project or product and he tells them to build something they find useful first.

Tanner: “I think that's kind of the wrong way to approach open source anyway. I’d rather solve a real problem. And do something that's actually gonna move the needle for users or a product then be like, “oh, I wonder what I could build in open source.

"I feel like you kind of come up with half baked ideas and you're probably just innovating to innovate. I don't think there's as much drive and inspiration when you do that.“

Do you have any advice for people looking to open source their projects?

Tanner: “I can only give advice on what works for me and the best way that I can do it is you have to remove yourself from your code. You are not your code

“If people look at your code and think it's crappy or judge it, honestly you should be just as judgey of your own code as they are. But it's not you, it just represents, you know, an attempt at solving a problem and it's always an attempt, right?

“Things are going to get better. I publish things to be open source because I want them to get better. And I know that by pushing them into the open source ecosystem, I get a lot of eyeballs on it. And in a way, outsource a lot of the difficult tasks of problem solving to those who care about it as much as I do.“

“Nothing is ever going to be perfect enough to release into the open source world as a true product. It's not gonna be ready until you release it and get other people interested in it to iterate on it with you. It's a chicken and egg problem, but you definitely have to put forth the effort, get rid of your ego and, and just kinda say here it is, you know?“ - Tanner Linsley

Avi: “Yeah. I very much agree with that. And I think that translates to not just open source. For anything that you're putting out there in the world, the feedback is the good thing. And the sooner you get the thing out there, the sooner you get that feedback and the sooner you can build something actually great.“

Why are you trying to build a sustainable open source project in addition to your start-up?

Tanner: “I don't really feel like I'm trying to build a business around my open source stuff. What I am trying to build though is a sustainable model for those problems and solutions. And no one ever said that money didn't help solve sustainability, right? You get money in the mix, it just helps a lot of things.

"Right now I am able to sponsor a lot of devs that help me out. I definitely want those projects to live on and that's a great way to do it, because I'm just one person.

"So there's some really bright minds that have kind of organically joined TansStack to carry things forward. ‍

Dominik Dorfmeister is one who basically just takes care of react query these days. The next version of React Query that's coming out very soon is basically his brain child.“

“I want to see that happen for more projects even if they're not mine. I want TanStack to be like an open source home, if you will, for great projects where they can come and actually survive and get taken care of.“ - Tanner Linsley

You have been fortunate to have some great sponsors. What have you done to encourage sponsorship?

Tanner: “It's going to sound really simple, but I think companies really like to be seen. People like to be seen for their contributions and I don't know if you've seen my little sponsor bubble pack that I have on tanstack.com but I think it's really cool and I think my sponsors think it's really cool too.

"The bigger your bubble is, the more you're helping out. And in a way, I've tried to sell that almost as like a partnership real estate space.

"You can change your image and you can tell me where you want it to link to and point to. And it's not just on the GitHub repo, but it's on my website. It's on every project page. It's very prominent. It has been a great way to let people feel like they are being seen for their sponsorships.”

How do you handle burnout as a maintainer? Do you have advice for other maintainers?

Tanner: “When I feel like I am burning out I go work on my startup instead. I just ship something for my product. We have real customers at Nozzle so shipping something new there makes you feel really good. You have to find a balance with your hobbies. I think you can burn out on any hobby and open source is just like a hobby.”

For more information on Tanner Linsley and his open source projects be sure to visit tanstack.com. If any of his projects have helped you in the past, consider becoming a sponsor.

Stefano Maffulli: An Exploration on Standards for Open Source Packaging and Distribution

Christina Gorton — Wed, 20 Jul 2022 19:41:19 +0000

Stefano is the Executive Director of the Open Source Initiative(OSI) and a long time advocate of open source. He joined us to discuss standards that should exist around open source packaging and distribution. He also talked more in depth about what the OSI is currently up too.

For the full interview please see the video posted on our YouTube channel.

Stream Recap

How or why did Avi and Stefano first connect?

Stefano reached out to Avi after reading the NewStack article “Where Does Open Source Fit into Russia’s War with Ukraine?”.

The article quoted Avi, who talked about how Scarf was able to leverage its distribution data to uncover who was downloading and using our packages. Scarf noticed the Russian government was downloading some of its packages and moved to block the traffic.
“Scarf will be blocking all package and container downloads originating from Russian Government sources until further notice.”

Stefano was excited to see a tool that maintainers could leverage and feel empowered to stop bad actors using their software.

Stefano: "As maintainer of a project or as a developer of some open source software, you have no ways of limiting the nasty usage of your software. It feels like you have no way to empower, you know, using your powers to say, look, I don't want my software to be used for nefarious purposes.

"And, that's why, you know, while we were having these conversations about whether the open source movement can have a say to help populations that are being attacked by an oppressor. That's how you and I met."

What can open source developers do about bad actors?

Both Avi and Stefano agreed this was a complicated topic. What kind of powers do maintainers have?

Open source software is pervasive but what tools do maintainers have to stop bad actors?

Stefano: "We feel like we must use the tools that we know how to use, which is our licenses and, copyright and contract law."

Stefano also mentioned that the current tools developers rely on are not powerful enough because for example bad actors are not going to obey the law.

Stefano: "On one hand, you enable dissidents with strong encryption. Yyou enable free speech.
On the other hand, you also empower terrorist organization to go, you know, behind the surveillance. So I think a lot of the conversations that I hear about the, role of developers in civil activism is relying on tools that are not really powerful enough in my opinion, like contract law or copyright."

However distribution data is another tool maintainers could use.

“You don’t have to do business with people who you don’t politically or socially agree with. And distribution seems to be one of those areas where we actually can give open source projects and maintainers a bit more leverage over their own work.”- Avi Press

Maintainers can also be creative with how they block the usage of their software. Many maintainers and open source communities stop bad actors by refusing to provide support.

“Open source is about collaboration but you can pick and choose who you collaborate with.” - Stefano Maffulli

How do you see distribution playing a role in any facet of open source?

Avi believes distribution plays a role in every facet of open source.

Avi: "Yeah, I think that distribution plays a role in pretty much every one of these different sections when it comes to things like security. How do we respond to CVEs when we find them? Knowing what organizations rely on a given vulnerable package can make it a lot easier to you know, to do damage control and tell people that they need to upgrade proactively."

"I think when we talk about, “how do we make sure that open source developers are building financially sustainable projects or secure projects?” It really comes down to having the distribution data. And having observability into that can really enable a lot of these opportunities.

"I think distribution touches just about every aspect of this. And we're really just starting to scratch the surface on kind of the various ways, which this can be powerful. And I think the same thing for all the political activism aspects that we were talking about. This just gives you another tool in the tool chain that you can use to, be creative as was said earlier."

What kind of standards, if any, should exist for open source packaging and open source distribution?

Stefano reiterated that the OSI are the stewards of the open source definition. They don’t write it but maintain it for the community. He believes more conversations in terms of the development of standards for distribution should be encouraged amongst stakeholders.

Avi made the point that these conversations are important. For example, many package registries do not require two factor authentication. This means a single person, who has the power to push a new package version to millions of devices overnight, could easily have their password leaked. He reiterated that it is vital for the OSS community to have best practices and standards around these kinds of situations.

What are the latest problems or questions when it comes to licensing, as it pertains to the OSI?

“The one thing that has always been fascinating to me is the impact of new technologies on the open source definition.” - Stefano Maffulli

The OSI is starting to investigate the impact of artificial intelligence on open source. Stefano talks about how AI is a weird blend of software and data that blurs the line between what users own and no longer own when sharing their content.

Stefano proposes questions like:

What kind of licenses should be on top of an application that uses AI?
What is the right of the user and the right of the developers? He reiterates that changes to technology always bring new challenges to existing standards and definitions.

What do you think the next 20 years of open source looks like?

Stefano: “Our role in the next 20 years is to continue educating and advocating the benefits of open source and to continue to build bridges so that these open source communities can continue to evolve and thrive around new challenges.”

Avi agreed collaboration within open source will help us to continue developing processes and systems that keep open source sustainable and secure.

He mentioned that Scarf is now a sponsor of the OSI.

To that end, we encourage others to consider becoming a sponsor or donating as well. Together we can all work to advocate the benefits of open source for the next 20 years and beyond.

Encourage Community with a Good ReadMe

Christina Gorton — Thu, 02 Jun 2022 16:28:53 +0000

Community is important for the health of an open source project. During my time as an open source community manager and developer advocate, I have seen how a project’s ReadMe can help with discoverability and encourage community contributions when they are well written. I have also seen how projects can turn away users when they don’t take the time to create a good ReadMe.

What Makes a Good ReadMe?

When you are looking to promote your open source project and encourage developers to use it there are a few things your ReadMe could include.

Project Name and Description

First, a ReadMe should be succinct. More detailed documentation about your product or project can be moved to a separate page or website. A few things you can add at the beginning of your ReadMe are:

Product name and logo if available
A short description that can include:
- What your application does,
- Why you used the technologies you used
- Some of the challenges you faced and features you hope to implement in the future.

Examples of project’s that do this well are:
Haystack by Deepset AI

Jina by Jina AI

If your ReadMe is long or contains several sections, a Table of Contents(TOC) is helpful.

There are several tools you can use to create a TOC in your GitHub ReadMe.

First, you can manually generate a TOC with Markdown. You can follow the markdown in this cheatsheet.

To automatically generate a TOC, you can use an online tool like the GitHub Wiki TOC generator. I used this tool to create the TOC in Forem’s Selfhost project when I was a developer advocate there.

Their open source project, Forem, is another good example of a project that uses a TOC to help their community navigate the many sections of their ReadMe.

Installation Guide

As mentioned previously, in-depth documentation should be added to a separate page, but most users appreciate a "Quick Start" or installation guide, so they can test the project quickly.
You will want to provide a quick step-by-step description of how to set up and run the development environment.

Fluent-bit is on example of a project that has a quick start section and links out to more detailed build and install instructions.

Grouparoo is another good example of including a quick reference to running their application, but also linking out to a fuller version.

Usage Examples

Usage examples help your community visualize how your project can be implemented and better explains your project’s use cases.

Kestra includes a demo app users can play within their ReadMe.

Httpie includes several different examples of their HTTP client, including a GIF to help visualize.

Contributor Guide

A contributor guide will encourage your community to participate in the health and maintenance of your project.

A contributor guide is useful for both maintainers and contributors. Guidelines support good pull-requests and issues and encourages good communication between contributors and maintainers, which saves everyone a lot of time and hassle.

Add a License

An open source license makes it easier for other people to contribute to your project. Depending on the license, it will also encourage or discourage the free use and distribution of your project. A public project is not free to use, distribute, modify, or contribute back to by default. You need to add a license to explicitly give others the right to do so.

For more information on open source licensing, see the following resources.

Licensing a repository
The Legal Side of Open Source
The Linux Foundation: Open Source Licensing Basics for Software Developers (LFC191)

Nice to Haves

Credit Authors or Contributors

In addition to a contributor guide, a section giving credit to the authors of the project and/or highlighting project contributors shows your appreciation for your community by centering the work they have done.

One example of a project that highlights their contributors with a “Contributors” section is Plausible. They use a tool like contrib.rocks or Open Collective to generate a contributors list and display an image on their ReadMe.

Additional Things to Check

If the project will be open to the public, you should check the Insights tab in your repository to view the Community Standards.

This tab provides a checklist you can follow to your community the best experience with your project. Try to include:

Maintaining an open source project is both exciting and exhausting. A ReadMe isn’t always at the top of mind when a developer or team open sources their code. Yet, taking the time to craft a good ReadMe encourages community and the health of your project. It will also save you time in the future by answering users questions from the start. Not everything in this list is necessary for a ReadMe, but even the addition of a few of these items will make an impact on your project.

A Different Approach to Measuring Open Source Community Health

Christina Gorton — Tue, 31 May 2022 15:15:13 +0000

What is Community Health?

Community is important to the success of open source software. The health of a community can be measured in different ways depending on the stakeholders involved. To understand and grow a community, project founders and maintainers need visibility into various technical, social, and even financial metrics. These metrics are important because they can help projects form data driven decisions. *But what metrics should we be using? *

This post is inspired by several discussions I have seen around the failings of current open source metrics and the tools used to measure them. This post will discuss what metrics we currently use and what I believe we should be using instead.

Current Metrics

On LinkedIn’s publishing platform Pulse, William Tracz wrote about Open Source vanity metrics and the problem with using them to measure a project's success. Tracz lamented they could not currently see a better way to gather valuable data about open source projects.

What are these current metrics?

Like Tracz mentions, many of the metrics open source maintainers and contributors rely on are considered “vanity metrics”. This term does not mean these metrics are not useful but they often do not give a full picture of a communities health.

Examples of vanity metrics in open source are:

Number of GitHub stars
Number of active commits
Number of new and closed issues
Releases

and more.

Often these are metrics founders and maintainers care about as they provide a way to quickly show “success” but may not actually help when it comes to important business outcomes and decisions.

Tools You Can Use to See Metrics

Orbit

Orbit is a tool you can integrate with GitHub to see metics like:

Pull request opened
Pull request merged
Issue opened
Issue comment created

and more.

As an open source community manager I used Orbit to quickly track new comments from contributors and maintainers so I could respond in a timely manner. I also used it to see when community members were no longer active. This helped me reach out and get feedback.

CHAOSS Metrics

CHAOSS is a Linux Foundation project focused on creating analytics and metrics to help define community health.

Common Room

Common Room is a growth platform that focuses on a business’ community. The platform helps companies connect their community engagement, product usage, and customer data through the use of artificial intelligence.

Bitergia

Bitergia is an analytics tool for software development teams, open source program offices, and InnerSource programs. Open source projects can use this tool to improve decision making and reporting by analyzing software development community, activity, and performance of open source projects.

Merico

Merico provides tools to help engineering teams understand and optimize their productivity and code quality.

What Metrics Should We Use?

“Tracking different types of metrics is essential for free and open source communities. Metrics give project insights into specific efforts and help get a feel of the community’s general perception. For that, tools that can pull data from various sources and develop a visualization of this data will help projects make informed decisions.”
Linux.com Editorial Staff, Measuring the Health of Open Source Communities

In the article quoted above and in William Tracz’s article the need for a tool that can pull more valuable data is apparent. But what data should projects be looking at and what tool could they use?

Beyond Vanity Metrics

What if you could see a project’s impact, identify potential customers, and understand/showcase growth without sacrificing your end-user’s privacy?

Scarf Gateway helps you understand how users are interacting with your open-source project at every step of their journey

Instead of answering questions like “how many followers do we have”, Scarf’s data digs deeper in to a project’s community health and answers questions like:

How many new companies tried our OSS this month?
How many unique users did our OSS have this month compared to last month?
Was the spike in downloads we saw last week genuine growth, or a thrashing CI pipeline from an existing user?
Should I actually spend the time to fix the bug for that one Windows user?

We can also have visibility into how OSS software is distributed.
Instead of wondering who is using our software, Scarf’s data answers questions like:

What percent of our user base has adopted our latest patch?
What cloud providers are the most popular with our users?

Scarf can also help uncover community metrics like your projects discoverability, usage, churn, and commercial reliance. Community Managers can use this data to proactively connect with users and grow a healthy and vibrant OSS community.

This data compliments the metrics you can get from tools like Orbit and answers questions like:

What is the geographical breakdown of our user-base?
What percentage of our user-base is corporate usage vs hobby usage?
What time zones should we cater to with our online event planning?

For a project’s documentation it can answer:

What percent of our user-base is on Windows vs Linux vs Mac?
Which section of our docs are users finding most confusing?

A New Way

This type of rich usage data moves beyond the typical vanity metrics the open source community has typically relied on. While those metrics can tell you a lot about a project, they do not give the full picture of a community's health.

With the addition of tools like Scarf Gateway, maintainers now have the power to make more informed decisions. When maintainers have hard data around who, where, and why companies and individuals are using their software they can better prioritize features, patches, releases and more.

Python Wheels vs Eggs (And How Data-Driven Decisions Must Become The Norm in Open-Source)

Avi Press — Tue, 10 May 2022 16:10:21 +0000

Originally posted on Hackernoon.

The world of software development is always evolving, and every now and then, you get to a moment in time when the path forks ahead of you. As developers, we have a choice as to which path to take, but each comes with specific consequences that impact the effectiveness and impact of the code going forward.

Currently, we’re going through one of those moments when it comes to Python Eggs. A recent tweet from Dustin Ingram showed that eggs accounted for less than 1% of built distribution uploads in December 2021.

Dustin Ingram

@di_codes

Welp, this chart definitely makes it look like it's time to deprecate .egg uploads to @pypi. Less than 1% of built distribution uploads in Dec 2021.

Do you still build or install eggs, or maintain a tool that supports eggs? I'd love to hear from you: github.com/pypa/warehouse…

17:59 PM - 26 Jan 2022

This data kickstarted a vibrant debate as to whether Python eggs should be deprecated. This conversation has a lot that we can learn from in the open-source community, and so we thought we’d explore the lessons in this post.

Python Wheels vs Eggs

The .egg format for Python packages was first introduced in 2004 and has served an important purpose most of the time since then. However, when Python Wheels were introduced in 2012, the weaknesses of the egg format became more well known, like the lack of support for clean uninstallations or upgrades, and the fact that only a single version of a project can be installed on any single directory.

Python Wheels provided meaningful improvements on these, as well as additional improvements in terms of the distribution format, a richer file naming convention, versioning, and better internal organization.

In light of this, many are debating whether deprecating uploads of new eggs is the right path to take.

How should developers make this decision?

The Importance of Data-Driven Decisions

As a developer, we have limited time and resources with which to accomplish our objectives. Our efficiency relies a lot on prioritization – choosing what to spend time on and, more importantly, what not to spend time on. So, when we have to consider whether deprecating eggs is the right decision, we want to do so with the right data backing that decision up.

As a result, many have pointed to these upload statistics as a clear sign that Python eggs are no longer the force they once were. Indeed, this data enables a vastly more informed discussion on the decision than if it was absent.

However, that doesn’t tell the full story. Even if the proportion of uploads is immaterial, we still don’t have much clarity on how the eggs are being used and relied on. This download data may exist somewhere, but it’s not generally available so that we can do proper analysis.

This is indicative of a wide range of other key decisions that currently get made on the basis of anecdotal experience, gut intuition, and personal preferences. Without the data being readily accessible, we are impairing our ability to make informed and unbiased decisions.

As a community, we should be pushing for better visibility, so that we can service users better and unlock the leverage that we have built up over time.

Introducing Scarf Gateway for Python

This data is crucial to our long-term decisions as developers because it helps us understand what aspects of our code are truly driving results for our users. It’s with this in mind that we’re excited to have launched Python support for Scarf Gateway, which will bring better visibility for any and all Python packages.

This is an important step toward securing the data that we need as an open-source community to make better decisions on all kinds of key matters; from infrastructure to packaging formats to vulnerability assessments and beyond.

For every decision point that you arrive at as a developer, having timely and accurate data on how your projects are being used helps you to sidestep your own biases and push towards the path that is making the largest impact for your key stakeholders.

Data-driven decision-making is how the open-source world can work smarter, not harder.