Introduction
Perl's failure to grow with the times and needs of modern developers is one of a handful of core reasons our language has seen a precipitous drop in popularity, especially over the past 10 years. It would behoove us to think hard about what we need to offer in order to remain more than a hobby language. Recently there's been a renewed effort by some to address the feature gap issue, however the process and conversation lacks much and I believe that has resulted in inferior results as well as discord amongst community members.
Ultimately we are seeking a process that returns results with enough gravitas that the bulk of the community accepts them and with enough relevance so that we move the needle on Perl's declining fortunes. What follows is a proposal of what they might look like.
A Short History of How the Perl Language Changes
I've been involved with Perl since the mid 1990's and for the bulk of that time language change in Perl has been primarily driven by remarkable individuals who start and build an important project and which project nearly always languishes when they leave. I say 'project' here but I mostly mean 'projects on CPAN' because for the most part of the 21st century the core Perl language did not evolve in any meaningful way via the existing process. Yes, we iterated and did make useful improvements to things like UTF8 support but the use of Perl in the wild in 2020 looked a lot like what I saw back in the early 'Aughts. This fact alone should be a most clear indicator that the current process is broken. However we did have big, splashy and useful projects on CPAN such as Moose, PSGI and to a lesser extent projects like DBIC, Catalyst, Dancer and Mojolicious which did create a sort of mini Perl renaissance that started in the later 'Aughts and extended into the early 'Teens. Unfortunately once the founder of these projects moved on to other things they for the most part fell into life support mode at best (hopefully getting security fixes and changes needed to make them work when things upstream broke). This 'founder syndrome' issue can be found lurking behind nearly all the biggest Perl projects as well as flops over the last several decades. This is the case because we never developed a true, community based, process for introducing change into the Perl language. Nearly everything of importance that has happened in Perl has happened because of the extraordinary effort of a single person, or at best a tiny group. This is not a sustainable process and it's a haphazard approach to change at best. Sometimes we get lucky and we get something like Moose, other times we get results that cause discord in the community. Extraordinary individuals will always find a place in Perl to shine but the core language cannot and should not depend on such happenstance if we wish to develop a sustainable and widely supported change process.
The Proposal
Development Process for Adding a New Feature to an Existing Programming Language
1. Non-Technical Value and Goals Document
Before delving into technical specifics, establishing the non-technical values and goals for the proposed feature is crucial. This document provides guidance for the decision-making process, ensuring alignment with the programming language's vision and objectives. The non-technical value and goals document should encompass:
- Mission Statement: A concise statement clarifying the purpose of the new feature and how it contributes to the language's advancement. This should be a short 'elevator pitch' style statement and should include answering the questions "Why?", "Why now?" and "Why in core?".
For example a possible mission statement for the project of moving signature support into core might go something like: "Method signatures exist in many other popular programming languages and have been found to improve code safety, readability, assist in maintenance and refactoring as well as provide a self documenting facet to code writing. Although for years people have played with various approaches to adding signatures to Perl via CPAN modules, which gives us a good idea of what works and what doesn't, this really belongs in core instead of CPAN so that we can improve code interoperability and make learning easier by providing a more complete core language."
The mission statement is important because you will often need to reflect on it to resolve conflicts and prevent scope creep.
- Core Values: Enumerate the guiding principles that should inform the feature's development. Consider values like simplicity, compatibility, security, performance, inclusivity, and transparency.
Possible core values for the signatures project might have been: "Deliver a feature set informed by learnings from other languages as well as prior art on CPAN", "Maintain capability with older projects so that one can introduce signatures slowly into an existing codebase", "Cope with less popular related Perl features such as subroutine attributes and prototypes".
Core values are important especially in promoting inclusivity for the project. It helps for developers to feel that things which are important to them are being taken into account or at least discussed and removed from the project via a fair, open and transparent decision making process.
- Scope: What is and what is not part of the project. This should be a clear list which acts as a limiting factor on what you are doing. For example, a possible scope for the signatures project might have been:
"Introspection of a subroutines signature will not be provided in version 1".
"Variable attributes will not be supported in signatures".
Of all the sections of this first part, "Scope" is the most likely to be altered as time and research goes into the project. In fact it is encouraged to add to the scope of limits as decisions are made. This helps keep further choices in properly alignment and also helps newcomers to the project understand and catch up with decisions already made.
2. Survey and Research
Comprehensive research provides the foundation for informed decision-making. Conduct an exhaustive survey within the community to understand current methods for addressing the issue the feature intends to solve. Additionally, explore solutions from other programming languages to extract insights. Compile and analyze your findings to identify trends and pain points.
Materializing this into the signatures project, this would have included doing a survey of CPAN and listing all the various attempts at adding signatures, making a feature matrix based on that list and then looking at what other popular scripting languages like Ruby, Javascript and Python are doing, again fitting that into the feature matrix. We'd then have had a document we could have used to generate a desired feature list for core Perl signatures and then prioritized those features based on developer value versus difficulty of implementation.
This research is also incredible valuable because it often identifies other issues and areas of concern which can lead to the creation of other new feature request projects.
As a side note, this is partly where I think the "Cor" (Perl core class) project went off the rails. When it was first announced, there was a call for comments but the announcement also included a significant technical spec of what the announcer thought was a good starting point. This very rapidly turned into a project based on Object::Pad. This meant that people who had different ideas were nearly immediately excluded from the process and the project merely attracted a small group of people who just happened to like the proposed technical specification. People that showed up and objected got told they didn't have to use it if they didn't like it. We basically missed a huge opportunity to step back, survey what Perl programmers were really doing in the wild, reflect on what popularly languages are currently doing and develop something both in line with Perl programmer expectations while cherry picking the best ideas from other languages. It also didn't help that we never had a pre technical discussion about project values that would have included considering issues of backward compatibility with existing projects. Basically it jumped nearly straight to code. However there's still time to step back and think all these things thru and possibly develop something better. I hope that is what happens.
3. Community Building
A thriving community is essential for the feature's success. Identify and engage key stakeholders, including experienced users, potential champions, maintainers, and contributors. Strive for diversity in perspectives to ensure comprehensive insights.
Identification of Key Stakeholders or Representatives: For a given project it is important to figure out which groups of people need to be involved, what groups form important stakeholder cohorts. There needs to be at least one person representing that cohorts needs and interests.
Transparency: Emphasize using scheduled forum issues for discussions rather than relying solely on ad hoc meetings on platforms like IRC. This ensures that discussions are well-documented and accessible to all interested parties.
For some reason it seems like a lot of developers believe all they have to do is have an open git repo and maybe a channel on IRC and that's enough to be able to claim the project is fair, open to all and fully transparent in terms of decision making. That is far from the case. Communities need to be built. In many cases stakeholders in overlapping projects need to be made welcomed so that they feel safe to provide a point of view. Meetings need to be properly announced and organized so that people with a viewpoint know when and how to contribute.
Currently this is a process we massively fail at. I see lots of important projects like Cor being managed in extremely ad hoc ways. Sometimes debates happen on Github issues, sometimes in the P5P mailing list (which is its own failure as I will note below), sometimes on one of several IRC channels, often at random times of the day. Notes are often not taken or shared in any public place regarding important decisions. This basically makes it impossible for someone who is not a party insider already contribute meaningfully. In addition there is no published process for how choices are made and disagreements resolved (see section 5 below).
Just a short digression here on the Perl Porters mailing list. It's basically the worst possible way we could be doing this, and the fact it's been 'the way' since the beginning is clearly a core reason for our failure to grow the language. The mailing list serves more of a gatekeeping and silencing function than it foster's productive conversation. It exists to stop people from getting involved. Don't believe me, go take a look at the archives: https://www.nntp.perl.org/group/perl.perl5.porters/ and follow any random thread. Anything remotely contentious rapidly turns into a very deeply nested tree of people shouting back and forth, often repeating things said on other branches over and over. Sometimes there's good comments but it's impossible to find them in the mess. Most of the time these conversations just end when the people involved get tired of taking about it. It's never clear what, if any, decision was made. I'd also add the the majority of younger programmers are more familiar with other types of discussion boards and insisting on using a mailing list is likely pre filtering out an entire younger generation of developers. If there's one thing we do, killing the porters mailing list and replacing it with something modern and sane would do much to move the needle.
I'll add one last thing, although community building is listed as item three, there needs to be a minimum 'quorum' community from the start in order to develop the non technical requirements and to make decisions on project scope and values as well as write the plan for how future decisions will be made (via a voting process or some other means).
4. Consideration of Backwards Compatibility and User Acceptance
At the project's inception, discuss critical aspects such as:
Backwards Compatibility: Assess the potential impact of the feature on existing codebases. Detail how the proposal will facilitate smooth transitions and support developers in migrating their code seamlessly.
User Acceptance: Outline strategies for soliciting early feedback from the community. Describe how the proposal will actively incorporate user input and concerns to create a feature that resonates with users' needs.
In general this is one of the things we are not terrible at. Historically we've been careful to release new features in Perl under feature flags and pragma that start off as opt in rather than requiring an opt out. However it would behoove a project of significance to decide early on how user acceptance will play into helping shape future iterations of the feature. There should be some objective way to measure the acceptance (for example are we seeing people using the new features in CPAN modules or in the source code of important Perl projects?). That user acceptance metric should be built into the project in such a way that failure to meet acceptance metrics forces the project to step back and reassess why.
5. Decision-Making Process
To foster respectful discussions and representation, establish a structured decision-making process that respects the basics of 'due process', Notification, opportunity to be heart and a fair and transparent protocol to resolve differences.
Discussion: Initiate a discussion platform for stakeholders to voice their thoughts and opinions. Emphasize neutrality and respect in communication. Discussions should be publicly announced and scheduled. Ad hoc decision making via back channels (on IRC at 4AM for example) should be highly discouraged.
Moderation: Appoint a moderator to ensure all stakeholders' viewpoints are heard and discussions remain focused. I would also say that a project leader is unlikely to be a good moderate especially if they have very strong view on how the project should evolve.
Constructive Feedback: Encourage evidence-based constructive criticism. Encourage participants to present well-reasoned arguments.
Consensus and Resolution: When discussions mature, aim for consensus. In cases where consensus is elusive, introduce a transparent voting mechanism with predefined acceptance criteria.
Key Stakeholders: Acknowledge that key stakeholders might include leaders of other open-source projects within the language ecosystem that overlap with the new feature. Ensure their input is valued and incorporated.
This if anything is one place I note we fail tremendously. People will have different viewpoints based on what they value. These viewpoints may all be reasonable. But since we generally fail to adopt clear project values from the start we end up with a lot of back and forth that generally never gets properly resolved. It usually ends up 'someone with an alternative viewpoint gets tired of getting attacked and decides to go away, possible then moving to a different programming language'. We need a way to start a conversation, run it and conclude it.
6. Defining Done
Although we accept that many projects will happen in iterations each iteration needs a clear definition of done. That could be a clearly defined and widely agreed upon minimal viable product. It could also include things like "successfully transitioned an existing large open source project in Perl to use the new feature". It might include the production of a test specification suite.
I recognize this probably sounds like a lot more process than many of you want the here about. To that all I have to say is 'is the current process working'?
Top comments (4)
This is a great essay. What communication channels do you propose (noting that the convo is split among a gazillion ones, with the IRC being the least likely to be used and the more likely (after mailing lists) to lead to chaos?
Many projects use a forum and proposal system like one built into GitHub. It’s not ideal but that couple with public meetings that are timely announced and well moderated could make a difference
You know we have the entire PPC (Proposed Perl Change) process, right?
github.com/Perl/PPCs
I'm aware of it. It's broken and Porters isn't even really following it.
It's broken from 'email your elevator pitch'.
1) The mailing list is a terrible way to manage this. I already talked about that.
2) It does not define a process for fairly handling what happens after the email goes out.
This process document was written so that the gatekeepers of the porters group could pretend they made a fair and open process and that therefore the choices they made have moral authority. This is far from the case.
I said before you don't get to say 'ok here's a mailing list and a repo, anyone can join', and that means its a fair and open community.
Communities need to be built, and a basic quorum of decision makers that properly represent all the major stakeholders assembled. Conversations need moderation and decisions need to happen in a way that fairly gives all parties due process. That is not what happens. Read any random thread on the porters mailing list and its obvious this is a bad way to run a language development process.