DEV Community: Allison

Launched a lightweight deprecation monitoring tool

Allison — Thu, 19 Dec 2024 20:16:48 +0000

If you've got a Rails app that's not on Rails 8 yet try out our new deprecation monitoring tool! It's like rollbar but for deprecations instead of exceptions.

Link to our blog post here: https://www.infield.ai/post/deprecation-monitoring

Mike McQuaid on 15 years of Homebrew

Allison — Tue, 18 Jun 2024 14:20:31 +0000

This week we’re talking to Mike McQuaid, project leader and longest tenured maintainer of Homebrew, a package manager for macOS and Linux used by tens of millions of developers worldwide. After ten years at GitHub, Mike is now CTO of Workbrew, a startup for managing a fleet of machines running Homebrew. Mike spoke with us from Edinburgh, Scotland.

Once a Maintainer is written by the team at Infield, a platform for managing open source dependency upgrades.

What was your first exposure to computers and eventually writing software?

As a kid my first exposure was at my school. We had BBC Micros, which is what we grew up with in the UK. This was in the early 90’s. And my dad noticed that pretty much any task, no matter how boring, if it involved computers I would happily do it. He was in finance and back before you could get online stock prices, every Saturday he would get the Financial Times and there would be the big broadsheet with the stock prices, and I had some little spreadsheet program and I would basically just go through and enter all the stock prices into his spreadsheet and spend like two hours on a Saturday afternoon doing that. And for me that was delightful. And my dad was like, OK, what's going on with this kid?

And then I just evolved my own interest through PC gaming and fiddling around trying to get the games to work on my computer. Eventually I went off to university and did a computer science and business degree and got a job as a software engineer. This was 17 years ago.

What was the environment like at your first job?

My first job was for BT, British Telecom. It’s sort of similar to AT&T if you’re in the States. And this was back in the days when it wasn’t that AI was going to possibly destroy your job, it was offshoring. Why would you be British and interested in writing software, because in two years all the jobs will be in India because they’re just as good for half the money, etc. etc. That was my first experience at a massive company.

I’d been dabbling with open source for a few years, and I learned pretty quickly that the open source way of doing things - it goes down well in the open source community, but at your real job you get pulled into your manager’s office, like what have you done? So that didn’t work super well with me. After that I left and was the first employee at a startup called Mendeley in London. I’d been hacking on KDE, the Linux desktop environment, for a few years at that point. Hired a bunch of people from the KDE community there, stayed there about a year, did some consulting around Qt for KDAB, a Swedish company. And then I kind of got this sense that maybe the cross-platform desktop app world was not future-proof, you know, starting to smell things. I did a bit of research. I wanted to work for a company using Ruby on Rails based in the Bay Area and pretty quickly came upon Github. I applied, and they said “Not now, you’re too fresh.” So I went to work for this company AllTrails. And then several applications later, I got a job at Github where I worked for ten years.

I guess that’s a nice segue back to the open source stuff since I left there last year and started my own company with some former Github people called Workbrew, and we’re building product solutions for bigger companies around Homebrew.

I’m having this Zoom from a KDE desktop. So I’m curious how did you get into working on it?

So I did Google Summer of Code the summer after I graduated, which was in 2007. Back then you could have journals, and you could post journals to your blog. And I basically built that integration so you could post journals to your blog. I built blogging support for WordPress and a couple of other blogs as my summer code project. And then I ended up sticking around and being involved with like bits and pieces on the KDE side for, you know, maybe a kind of couple years. And then I moved to Mac and I was doing some KDE porting to Mac and then I sort of gave up on it as a lost cause.

I remember a period in the middle there where it was like you're gonna run all of your Linux apps everywhere - I think I tried to install Amarok or something on Windows, and it sort of worked. Not the way the world ended up going.

No, it never worked quite as nicely not on Linux, unfortunately.

What was your first exposure to Ruby?

During university I had started dabbling with Linux and using various Linux distros, submitting patches to things and modifying things on my system or whatever. I’d seen a tiny amount of Ruby but mostly I got into KDE and playing with lower level Linux bits and pieces. Ruby didn’t come until later on when I was at the consultancy and I heard about this project called Homebrew from a friend of a friend in London. And that was the first serious Ruby that I wrote in 2009 or so, and I’ve been using it for the majority of my career ever since.

We hear a lot about how intimidated people are to get involved with open source. So when you heard about Homebrew, what made you get over that hump and get involved?

Hopefully without sounding too arrogant, I never felt particularly intimidated to get involved with open source projects. I think particularly in the Linux era and the pre-Github era, the barriers to entry were so high that it felt like if you managed to get through those barriers people were generally like, we're going to be your friend now. And if you're in my city, I'll take you out for dinner. I’d been to a few open source conferences by this point and everyone was very friendly and welcoming and accepting.

So with Homebrew, one of the guys I hired at Mendeley was friends with this other guy who worked at another startup in London. And he was the guy that created Homebrew, Max Howell. I think having that connection and in the first year or so, being involved with Homebrew, meeting up and getting beers with Max and talking about our thoughts, it felt easy. What sucked me into Homebrew initially was just this idea of scratching your own itch - I’m using this, I need this, and I’m going to add a new thing because it’s easier if it’s in here and then everyone else can use it. And then over time it becomes less about me building for myself and more about, what does everyone who uses this project need and how can I help them?

How do you manage the roadmap? We talk to some maintainers where the projects, especially the huge ones, are very formally run. Others are still very informal. How do you think about that?

Yeah, I would say we’re on the moderately anarchistic end in terms of feature roadmaps and stuff like that. At least until you get to open source projects run entirely by corporations, maintainers have no ability to make anyone work on anything they don’t want to. I'm the longest running maintainer, I've been the project leader. But if I say like, hey Bob, I want you to ship this thing by the end of the month, then Bob can just be like, nah. And I don't really have any ability to do anything beyond that.

So in some ways it was quite similar when I was a principal engineer at GitHub, because when you're in those roles, you have no direct reports and so you don’t have twenty engineers to do what you tell them, but at the same time, you have an awful lot of cultural influence. So instead of it being like, I will tell you what to do, it becomes like, hey, I've had an idea, who wants to join me on this journey? And there's some people with within Homebrew particularly who would like to be doing more Homebrew stuff. So I guess a big change for me in the last few years is that I try to push as much of my personal Homebrew to do list into issues that are tagged ‘help wanted’. Sometimes those are done by me, and sometimes those are done by random people in the community. There are five open issues in the main Homebrew repo right now and all of them are things that five years ago I would have maybe just had in my Apple notes somewhere. But I learned that there are a lot of people that might go “hey, that’s a good idea” and jump in and get involved.

Do you guys have any kind of regular meetings among the core maintainer team?

Once a year we try and get as many maintainers as possible to meet together in person for our AGM. We try and collocate that with FOSDEM, which is in Brussels every year in early February, to try to make a bit of a weekend of it. We're trying to do more events now. We're doing a hackathon focused on performance and security next month. And historically we’ve had some vaguely regular Zoom calls, but it’s hard to sort out the timing for those. Most of our private communication is happening in Slack. That's where we have the conversation about what do you think we should be doing? Or, there’s this issue right now. Could someone jump on and help with that?

So this list of issues here, I see this one about Sorbet. How does that conversation, like “we should add type checking to Homebrew”, get shepherded through?

That's a pretty good example actually. You inadvertently stumbled on one of the more amusing but contentious ones. All of our governance documentation is public but if you’re not a Homebrew maintainer, it can be a bit of a struggle to get through it all. We have the project leadership committee, which is essentially managing the money and the governance structure. And historically that was also maintainers. But now two of the five people in that are not maintainers and never have been. Then we have the technical steering committee, which is managing roadmaps and deciding these types of things. And then we have a project leader, which is like an elected position as well, which is me. And I'm the only person who's ever been the project leader so far.

But the way it works in reality is that the technical steering committee exists to help resolve conflicts in technical direction I’ve been unable to resolve myself with the other maintainers. And that's what happened with the Sorbet stuff. Interestingly, a few years ago when it was initially proposed, I was not a fan of the idea. Now, however, I'm actually a big proponent of Sorbet. I think we should double down on it and use it even more.

What convinced you?

Well sometimes the way it can go with open source projects is that someone gets an idea, that one person is very enthusiastic. And people get excited and say yeah we want to get involved, great. But what can happen in the worst case is that none of the people who say they’re going to step up actually step up, and the person who pushed it to begin with got bored and wandered off, and now you’re left with these problems. And then it becomes someone else’s problem, often mine, to clean up the mess. So I thought that was the way it was going to go with Sorbet. But it turned out that over time actually more people got interested in it, and more people got involved. And personally over at Workbrew we started using it to solve a bunch of problems, and at GitHub a bit as well, and so now having used it to solve problems I’m like ok, this is better than I thought it was.

So in the 15+ years that you’ve been working in open source, do you feel that the way that open source projects are run have changed? What have you noticed in terms of open source culture over the last 15 years?

I think the really big thing in the last 15 years is where and how open source is happening. So 15 years ago I'm not sure whether node.js existed - certainly npm if it existed was pretty minor. Whereas now most engineers out there are writing JavaScript, right? If you're a new engineer coming out of a bootcamp or learning a new language, it would be a strange choice for you not to learn JavaScript and do that on the front end and the back end and then I'm full stack and yadda, yadda, yadda. So I think that has been interesting just because the JavaScript ecosystem has its own culture and way of doing things. You have millions of dependencies and often those are quite small and maintained by like a single person. Not fewer, big chunky open source projects. Like KDE is essentially a big umbrella project that probably has, I don't know how many active contributors it has, but it wouldn't surprise me if it's hundreds or thousands or whatever. And it's carved into a wide variety of pieces. Whereas I feel like open source in general has become a bit more decentralized and there's a lot more small projects of one or two people here and there.

Things have also moved away from the Linux world where it used to be. Like when I was at university, open source and using Linux were almost one to one, right? If you were a big C# Microsoft stack Windows type person, it would be relatively unusual for you to use any open source at all. But obviously over time, with GitHub and Microsoft acquiring them and Microsoft themselves getting a lot more into open source, it's like everything is open source now in some respect but what open source means has also changed. There used to be the assumption that open source projects are community run and maybe they're loosely affiliated with a company or whatever. In the earlier days of GitHub, if you looked at the repos with the most stars or whatever, Homebrew was often up there. Whereas now it's all like VSCode and next.js. Almost all the projects that are up there are ones that have major corporate backing basically. That makes open source a very different world and makes it harder to be an indie, volunteer-run project.

Are there any other open source projects that you’re impressed by or watching?

Ruby on Rails always continues to impress me. I'm using it again at the Workbrew startup that I'm building, and it’s the first time I've built a completely greenfield app from scratch. We’ve got a couple apps I guess, but one’s in Rails and one’s in Go and you know, Go is really good at what it does, but writing it doesn't make me happy in the same way Ruby does. There's been such an amount of time and effort and care and love that's gone into making it very pleasant for developers to use.

gemcompat: an open source db of undocumented gem incompatibilities with rails

Allison — Thu, 04 Apr 2024 16:47:37 +0000

TLDR: We created an open database of every gem version that’s silently incompatible with Rails. Check it out at https://github.com/infieldai/gemcompat

Our company is built on Rails, and part of our work involves upgrading Rails apps for our customers. In doing this we come across incompatibilities between packages that aren’t captured in gemspecs. Commonly a package will leave an open-ended upper bound in their gemspec (like activerecord >= 6.1) which can cause issues if the next version of Rails comes out and breaks something. I once tracked down an incompatibility between two packages that only surfaced as colors in images embedded in PDFs being inverted. These can waste a lot of time.

Last month we shared a script that checks an application against these kinds of issues for Rails 7.1. It was based on research we did reading a few thousand ruby changelogs looking for messages like “Added support for Rails 7.1.” People found it useful so we thought we’d turn it into a more complete project.

We’ve open sourced all the incompatibilities we know about across Rails 6.1, 7.0, and 7.1 at https://github.com/infieldai/gemcompat/. You can check your own codebase like this:

gem install gemcompat

gemcompat –package rails –target-version 7.1 –lockfile Gemfile.lock

Hope this is useful! We’d love you to try it out and let us know when you come across an incompatibility we don’t have. You can open a Github issue with anything you find and we’ll add it to the DB.

Undocumented Gem Incompatibilities with Rails 7.1

Allison — Thu, 29 Feb 2024 17:19:31 +0000

TLDR: Use this script to check whether your app relies on any gems that are silently incompatible with Rails 7.1.

Upgrading Rails means first upgrading other dependencies that block the way. Some of these will have explicit incompatibilities documented in their gemspecs. If you try to run bundle update rails without upgrading these gems you'll see an error that bundler couldn't resolve the upgrade.

Other gems leave an open-ended rails requirement in their gemspec. This means bundler will allow a new version of Rails alongside your current version of those gems, but there's no guarantee from the maintainer that the two are compatible. This can lead to subtle bugs that don't get caught until production.

For example, take the popular data-migrate gem. Its gemspec requires activerecord >= 6.1 with no upper bound. Looking at the changelog, though, you'll see that support for Rails 7.1 wasn't added until version 9.2.0. Older versions will hit this exception when someone tries to run migrations under the latest Rails, even though bundler installs the package with no warning.

These "silent" incompatibilities are often documented in the maintainer’s changelog even though they’re not available to bundler. My startup Infield keeps a database of every ruby package and its changelog, which we parsed to find any mention of adding Rails 7.1 support. Then we filtered those results to only new package versions where bundler would have already allowed Rails 7.1 in the previous version. We found 23 of these:

Rails 7.1 compatibilities

Here’s a script to check for these in your app - just point it at your Gemfile.lock.

Hope this is useful!

Infield dependency management system launches support for javascript and python

Allison — Fri, 19 Jan 2024 19:00:29 +0000

TechCrunch: Infield wants to make open source dependency management trivial

The Limitations of Dependabot in Managing Open Source Dependencies

Allison — Thu, 09 Nov 2023 17:07:09 +0000

Open source software development has revolutionized the way we build and maintain software. Something like 96% of codebases now include open source components. However, one of the significant challenges for any software team relying on open source software is managing dependencies. Dependabot, a tool that automatically scans your project's dependencies and alerts you when a dependency contains a known security vulnerability or has a new version available, is one of the most widely used tools for managing dependencies. While it's a valuable addition to any developer's toolkit, it's important to understand that Dependabot is not in and of itself a complete dependency management solution. We’ll explore that here.

1. Lack of Contextual Understanding

Dependabot lacks the ability to assess the impact of a dependency update on your specific repo. This can lead to problems like compatibility issues between a version of a dependency that Dependabot suggests upgrading to and a different dependency that your app relies on. If an incompatibility wouldn’t be caught by your package manager (for example two packages conflicting in a global namespace) it won’t be caught by Dependabot.

Without understanding the specifics of your codebase, Dependabot might initiate PRs that would break your app, creating work for your team to sort through which PRs are safe to merge and which ones are just noise, or potentially risky.

2. Limited Ability to Handle Breaking Changes

Dependabot primarily relies on the version numbers in your manifest files and semantic versioning to determine when to update dependencies. It is the responsibility of the developers on your team to find the changelog associated with the new release of the dependency, review that changelog for breaking changes that apply to your codebase, make the necessary code adjustments to accommodate the new version, test it, and deploy it. There can be complications with every step of this process.

Some open source project maintainers do not keep a changelog in source code, or don’t adhere to semantic versioning. For some packages, no changelog may exist at all. Once you find a changelog, researching the changes and assessing which changes apply or do not apply to your codebase can take a significant amount of time. For example a breaking change might drop support for a language version that your app relies on. In order to address that change, you first need to upgrade your language version, which may be a project in itself that relies on other intertwined dependencies. Even once all of the documented breaking changes are addressed, there may be undocumented incompatibilities between your specific codebase and the new package version. We’ve run into some wild ones, like a patch version of a popular APM provider’s SDK clashing with another popular logging platform, but only under the production environment.

3. It Won’t Dig You Out of a Deep Dependency Hole

The more stale your application, the more outdated dependencies you’ll have and the more likely it is that there are breaking changes between the versions you’re using and the versions that Dependabot will try to get you to. When you first enable Dependabot, it will start opening pull requests to update your outdated dependencies within minutes, up to a maximum of five PRs. But without a holistic strategy to dig out of the hole, you may find yourself overwhelmed and drowning in failing PRs.

Getting your application to a reasonable state where Dependabot can start generating useful, automated updates often requires some amount (and sometimes a large amount) of upfront planning and upgrade work. Dependabot is not a good solution for an app that’s been left unmaintained.

4. False Positives and Negatives

Dependabot's automated approach sometimes generates false positives and false negatives. It might recommend updates that appear necessary but end up causing issues, or it might not alert you to critical updates that could enhance security or performance. Relying solely on Dependabot's notifications can lead to a false sense of security, and it may neglect important updates that are not clearly indicated by version numbers in maintainer documentation.

5. Incomplete Testing

Dependabot can automatically generate pull requests to update dependencies, but it doesn't ensure these updates won't break your code. This can create a risk when accepting automatic updates. Developers should run their regular test suite after accepting Dependabot-generated pull requests to ensure everything works as expected, but if test coverage is limited, this can create a false sense of security that a Dependabot-generated PR will not break anything. Dependabot works best on apps with robust test coverage.

Dependabot can be a very useful tool when an organization has built significant muscle around researching, changing, and merging PRs generated by its automated system. Absent such guardrails, a more comprehensive platform like Infield can improve your team’s overall dependency management posture and keep you in good shape going forward.

Once a Maintainer: James Lamb

Allison — Mon, 16 Oct 2023 13:03:56 +0000

Welcome to Once a Maintainer, where we interview an open source maintainer and tell their story.

This week we’re talking to James Lamb, Staff Machine Learning Engineer at Spothero and maintainer of LightGBM, a machine learning framework out of Microsoft. James is a prolific contributor to the open source and data science communities and the co-organizer of the MLOps meetup in Chicago.

Once a Maintainer is written by the team at Infield, a platform for managing open source dependency upgrades.

How did you become a software developer?

My high school did not have computer programming, and I didn’t write any code in high school. I really liked hip hop, and I wanted to be in marketing at a record label. So I went to college for marketing. At the school that I went to, you could get a double major in the business school by taking one extra class. And from high school, I already had three credits for Econ. So purely out of laziness, I was like, I guess I'll double major in Econ. So I started taking Econ classes and I really, really liked it. So I went to graduate school. I happened to be at one of the two universities in the US at the time, Marquette, that had a terminal master's degree in Economics, so I was really lucky to get into that.

My thesis project was a time series forecasting project where I was using methods from finance to forecast the sales of frozen dinners at Dominic's grocery stores in Chicago. It was like fit a model to some data, make a couple weeks ahead forecast, save the results to an Excel spreadsheet. That's what I was using. Add another week of data, retrain, repredict. Over and over and over again like hundreds of times. And I was staring, like staring at this project with, you know, like four months to go until graduation. I was like, do I try to learn R? People do R in Econ. Or do I just grind it out in the computer lab and just like point, click, copy, in Excel over and over again. And I actually chose to grind it out. But as soon as I graduated, I was like, I'm never doing that again. I have to learn how to code.

My first job was an economics job and while I was there I started learning R on the side from the Johns Hopkins Data Science Coursera courses, which were amazing. Those courses taught me to write my first function. I created my first plot programmatically. I read data into a program for the first time, used git for the first time. But I wanted to write code to be a better economist. So I kept taking online classes, Coursera, Codecademy, DataCamp, whatever I could find.

Eventually I did end up getting a data science job at a startup here in Chicago. And that just put me on a whole other path. I learned a lot more really fast. That was the place I wrote my first CI pipeline, first time I put a machine learning model to production, my first container image, etc.

Did you join the engineering team or did they have a separate data science team?

This is another place in my career where I just was really fortunate, right place, right time. I joined this company called Uptake. They had just gotten a large round of funding, and they were looking to hire like 35 data scientists of varying experience levels, all at once, full time in Chicago. Data science was the product at this company - we took in sensor data from industrial equipment like mining trucks, wind turbines, locomotives, stuff like that. And we built applications that were powered by models that predicted when that stuff would break so that you could do maintenance ahead of time. Maintenance ahead of time is always cheaper than “this thing is broken right now, we need to fix it” type maintenance.

So when I got in there, I joined a data science team, but we had like 70 full time data scientists in a head count of like maybe 700 people. It was a huge part of the business. And so within that team, we did a lot. When I joined, I was the most junior member of the team and I was joining as a data scientist. The expectation was that I would write R and a little bit of Python, and bash scripts to produce reports and statistical models. And then by the time I left there, I really wasn't producing models anymore. I was doing something more like what you'd expect from a staff engineer. I was writing code, but also writing long form architecture proposals. I was translating the data science team's requirements into engineering requirements. You know, that could be understood by engineers who weren't familiar with machine learning.

That feels like a major transition to make, from self taught to “I can do what I want with data”, to “I’m ready to help an entire team do data science engineering in the right way”. I’m curious how you do that.

You know I was fortunate in a lot of ways, a lot of opportunities that just kind of like came to me and and I took them. I was one of the more junior members of a team of very, very talented people. I had to learn to keep up with with some of those people. Absolutely brilliant. I was lucky to be in this group of really curious, just like nice, friendly, but very, very smart people who were never bothered if you were like, hey, I saw you did this thing. Like, what does that actually mean? What is Docker? How does Docker work?

At the same time I did another master's degree in data science from UC Berkeley.

Did they encourage you to do that degree or did you go to them and say, hey, I want to do this degree, and I would like to do it while I'm still working here?

No one that I worked with said you should go get a graduate degree, but they were very supportive when I did it. I was actually encouraged by a mentor of mine, who I first got in touch with doing my economics degree. At the time there wasn’t a ton of research into the exact topic I was interested in, which was applying these financial models outside of finance to a supply chain setting. There was one set of papers that I cited heavily in my work, and I reached out to one of the authors of the papers, Dr. Shoumen Palit Austin Datta. I just said hey thank you, I’m a student at Marquette, I cited your work, here it is. And amazingly, he emailed me back and had questions for me and we developed this relationship over email. He recommended and really pushed me to UC Berkeley.

It sounds like you had great mentorship on both the commercial side and the academic side, which is really rare.

I feel very fortunate and you know, very privileged to be able to take these risks. I had a strong family support system. I came out of college with less debt than the average person because my parents helped me with money for college. So I was able to take these bets, you know, like I'm going to spend all this time on Coursera courses or take out another loan to go to Berkeley. And I had the time, too. I didn't have kids that I was taking care of or elderly relatives or other obligations. So I was very fortunate in that way too. That’s a really big part of the story.

This idea of having the time to work on open source comes up often, so that’s a bit of a natural transition. What was your first exposure to open source software, and how did you start to get involved in it?

When I was at Uptake, one of my coworkers, Yuan Tang, he was already a prolific open source contributor. He was one of the maintainers of the XGBoost project. He had written the first official guide to TensorFlow in Mandarin. He had worked on just a ton of projects.

He was really passionate about open source, and he organized a little hack night for us coworkers at Uptake one night after work. I still remember, he had a little R package called dml, it was an R package for calculating distance metrics between data sets. And it was on CRAN. You know, it was like a real project. But it was small enough that it was basically only him contributing to it. And so he took the time to write up manageable sort of good first issue types of issues. And then he got the company to buy us a little bit of food and drinks. And one night after work, we just went into a conference room and he gave like a 10 minute presentation that was like, here's how to contribute to open source projects on GitHub. Specifically, here's how it's different from working at your job. Here's what a fork is, for example. You don't really think about that maybe when you work in a company. Here's the etiquette, here's the mechanism, here’s what continuous integration is.

And then for the night, he paired us up and we each picked an issue and we just worked on it. And he sort of walked around the room like a teacher and we could raise our hand and be like, hey, I have a question about this. And we all came out of that night having made our first open source contribution.

Getting that first one was what I needed to break through that mental barrier of “What do I have to contribute?” To hear him say you know, most of my first contributions were writing unit tests, fixing documentation, removing unused imports. I thought OK, I can do that. And so I started doing it. One of the tricks that I use is to take something I learned something at work, for example I learned if you combine together multiple run statements in a docker file, the resulting image is smaller because there's fewer layers. Once I learned that, I was like I'm going to go look in all the repos in the Apache GitHub project and see if there’s any there that are publishing images that I could go make that change. It’s a super small, well defined thing, and I can go in and they’ll be grateful. I don't have to understand how all of Apache Pulsar works to just like cut 50 megabytes out of their container image. I repeated that pattern a few times, where I would learn something at work, then go look at high profile projects and see if I could apply it.

It’s interesting in the data science world it sounds different from contributing to a JavaScript library or something, where it’s coming from an implementation of an idea that may be in an academic paper, versus here’s a new calendar widget or something. It seems more intimidating to me to jump into a project like that, like LightGBM, where there's a sort of deep academic foundation to it that you feel like you might need to understand. You describing here how you got into some of those projects with these well defined changes is really interesting.

This feeling of needing to understand the entire project to make a contribution, it was really important for me to get over that and to realize that's not true. There are a ton of these helpful contributions that are appreciated by maintainers that do not require you to understand the whole project, and and forming pattern recognition of what those are can reduce the amount of time it takes to get contributions out there and start forming a relationship with maintainers. By the time I started contributing to LightGBM I had done maybe dozens but not hundreds of these small open source contributions. I needed to be taught the the etiquette for for open source, where I was originally changing things that didn't need to be changed, or I was making arbitrary style-related changes that I had no business making in a project. I had to learn those lessons. And so by the time I came to LightGBM, I understood the etiquette and how to contribute.

As you said, it’s an academic project that was then turned into software. The people running it were researchers, they were not software engineers by training. They had PhDs in machine learning and statistics. The original LightGBM paper was a C library with a C API. Shortly after that, a Python package was added on top that takes in Python data structures and then calls the underlying C stuff through that C API. When I started getting involved with the project around 2017, there was an R package that was contributed by an outside contributor, not a maintainer, but it was kind of sitting there.

The maintainers were not R programmers. They knew it a little bit, but it wasn't getting the attention that the rest of the project did. So I came in there and I had just been learning about R packaging and I saw a bunch of things right away that I felt I could help with. R is very particular about how you structure your documentation, how you import third party dependencies, all this stuff. So I started just making submitting pull requests related to the R package, and I tried to explain in each of those what I was doing because I knew the maintainers weren’t R programmers. And again, I was very fortunate timing-wise. The project was sort of at an inflection point in its popularity, and the maintainers were feeling the strain of all these feature requests, bug reports. And they saw me showing up, making contributions to the R package. They invited me to the project. I got commit rights on the repo. I joined the private maintainer Slack, this was 2017 or so, and over time I've just gotten more and more involved with the project. At this point I am its most active maintainer. I’ve learned so so much from it.

There’s this reciprocal nature between you doing more academic work that allowed you to contribute to the open source project to begin with, because you had this R expertise that the original creators of the package didn't have. But then on the flip side, you're saying that you learned so much from the project that you didn't know. That’s really interesting.

The other thing this project prepared me for really well is for working remotely in my day to day job. The entire time I've been involved in this project, I have been the only American and I think there are only two of us involved with the project who are native English speakers. Most of the other maintainers are not native English speakers and they are not in Western Hemisphere time zones. They're in Beijing, Moscow. There was one in France, one in Australia and so I had to learn through this project how to communicate effectively in writing asynchronously. I had to learn to anticipate the questions I was going to be asked and be proactive in putting all my assumptions into writing to reduce the number of cycles. As I’ve gotten into more senior engineering positions where more of my job is writing words than writing code, that’s helped me a lot.

The human side of open source is something that comes up again and again as a reason for people feeling like they can’t contribute. How do you think about that?

When I was at Uptake I was sort of their unofficial Head of Open Source which wasn’t real but it meant that I was given a little bit of budget to run a biweekly hack night after work, get some food and drinks. We even ran an event for Hacktoberfest that was open to the public in our space where my coworkers and I walked around and helped people make contributions. I tried to break down that barrier for people feeling like they’re not good enough or not knowing where to start. I show them look, here’s a real PR I made to Apache Spark or something where I changed the word “three” to the word “several” because the list that it was referring to had four things in it. Something where the audience can say “Oh, well I could have done that.” That’s all it takes, you know?

I’ve definitely had some rough experiences with open source, socially. People can get very gruff or defensive. And if I hadn’t had those great first experiences, or been on the other side of it as a maintainer, that might have discouraged me. There’s a book that I want to mention, this book by Nadia Eghbal called Working in Public that saved me from burning out. I know that sounds dramatic, but I mean it. About two years ago I was ready to quit open source, or at least walk away for a few years, until I read her book. Cannot recommend it highly enough.

Why don't we end with what are some other open source projects that you think are really interesting right now?

I have been getting really into the Trino projects. Trino was Apache Presto, then Facebook got overly aggressive with the way it was placing its employees into positions of power in the Presto project, and so the creators of Presto forked it to a new project called Trino. It’s an open source query engine that I find really interesting.

I've just started getting into a project called JAX. JAX is a machine learning and deep learning framework that also has some really nice lower level primitives for just in time compiling Python functions, and the coolest thing that it offers is automatic differentiation of functions which is a really important thing in machine learning. This JAX project lets you come with basically arbitrary Python code that describes what is good or bad to a machine learning model and use it to train a model. It's really cool.

There's a whole other class of projects that I'm really interested in and wish I could spend more time on. Those are all projects around storing machine learning models in a language agnostic way. So that is like the ONNX project, the PFA project, the PMML project. I'm really interested in those. They’re like data interchange formats for machine learning models. I haven’t spent enough time with them recently.

Upgrading Rails at Alice

Allison — Thu, 05 Oct 2023 13:33:18 +0000

Alice is a startup in New York City that helps hospitality, retail, and healthcare employees get an extra week of pay, for free. Employees connect their credit and debit cards, Employers connect their payroll systems, and Alice automates pre-tax benefits—no open-enrollment, no saving receipts, no use-it-or-lose-it—just extra money in paychecks. Because Alice handles sensitive data, they place a high premium on security and maintain HIPAA, SOC II, and PCI compliance with a lean Engineering team.

Backlogged Upgrades

Luke Rodgers is a Senior Engineer at Alice and has been there for 6 years. For the majority of Luke’s time at the company, Alice aspired to keep Rails and other core application dependencies up to date, but this work continued to slip on the backlog. There were several reasons these upgrades kept slipping:

Poor Visibility: The Rails upgrade was perceived to be a nebulous and massive project. Without clear scope and visibility into future incompatibilities caused by the Rails upgrade, it was challenging to determine where to start or how to progress. According to Luke, “A Rails upgrade can feel like you’re heading into a jungle.”
Staffing Issues: This ambiguity informed a belief that only the most senior engineers on the team could be involved in the project. Allocating these resources to a dependency upgrade of unknown complexity would come at a huge cost to the company. “Luke’s focus is precious to us,” said Avi Karnani, Alice’s CEO.
Cost Implications: There were very real opportunity costs to assigning senior engineers like Luke to the Rails upgrade project, but there was also a very real and direct cost to alternatives: “A vendor had given us a quote for $200,000 to upgrade Rails, which seemed like a lot of money. Maybe that even contributed to our sense that this is just some giant, untamable beast.”
‍
These challenges combined to push the Rails upgrade further down the priority list, leaving it lingering in the backlog for years.

Upgrade Debt's Tipping Point

Over time, the issues caused by the lingering Rails upgrade accelerated and started to cascade into other areas of the business:

Security / Compliance: The first was Alice’s security and compliance posture. As new security patches came out, Luke and his team were unable to easily patch vulnerable dependencies because they were blocked by major, complex upgrades like Rails. “We would need to monkey patch a gem because there was a security patch that we couldn’t get to.” Although this fixed the problem in the short term, it added considerable maintenance debt to the codebase and increased the overall time to remediate.

Planning Costs: The amount of work associated with dependency upgrades increases at an accelerating rate over time because a codebase's dependencies are always shifting. Said another way, dependency upgrades are a moving target. For the team at Alice, attempts to manually research the Rails upgrade “got stale over time because we actually removed this other gem or these gems got upgraded for some other reason. So it's this shifting landscape that every time someone picked it up again, they'd have to sort of start fresh," said Luke.

Employer Brand: Great engineers generally want to work on the latest versions of tools. As Luke pointed out, this is also a useful signal to potential hires that a company “isn’t just a feature factory. That the executives understand the importance of paying down tech debt.”
‍

Discovering Infield

Luke's introduction to Infield came via a post on Hacker News.The post highlighted Infield's Upgrade Path feature, which seemed to remove two major points of friction for Alice getting the Rails upgrade completed. The first was that Infield gave the team a clear, step-by-step plan for the upgrade, potentially saving weeks of work and significant costs. The second was that the plan updated dynamically, allowing different team members of varying seniority levels to “snap into maintenance mode without having to reload a bunch of context.”

Over the course of the following 4 weeks, Luke and his team chipped away at Alice’s Upgrade Path, implementing the breaking changes that Infield surfaced, clearing the blocking dependencies, and finally upgrading Rails to the latest version. Infield’s impact was felt throughout the business:

Time and Cost Efficiency: With Infield's guidance, the team saved around $8,000 to $30,000 in planning and troubleshooting time.

Leveling up Developers: The clarity and direction provided by Infield made it feasible for other developers on the team to contribute to the upgrade process, democratizing work that had previously been reserved for long-tenured engineers. This also gave Alice the infrastructure to build a culture of maintenance into the Engineering organization.

Enhanced Security: By bringing and keeping dependencies up-to-date, it became much easier and faster to remediate security vulnerabilities on an ongoing basis. According to Luke, accepting a security patch on a supported package is much easier than "figuring out if we are actually affected [by a vulnerability]. Okay, dig into the gem and look at whether we're using the affected code, go right to monkey patch, test the patch (if it’s even possible to). It just means you're spending way more time having to worry about vulnerabilities versus ‘there's a new version, let's upgrade.’”
‍
On an ongoing basis, Alice plans to use Infield to keep dependencies up-to-date across their Rails app. Because Infield also gives the team visibility into maintenance status (abandoned packages, unsupported versions, etc.), they can use Infield to queue up application maintenance work that can be addressed in the course of a standard maintenance rotation. As Luke put it, Infield will help Alice avoid upgrade lag on an ongoing basis, “if I have a few hours to do some upgrades now, what should I prioritize? Infield helps me do that immediately.”

‍

Rails 7.1 is coming out soon. Infield will upgrade your app with minimal back and forth.

Allison — Thu, 05 Oct 2023 00:24:41 +0000

Infield keeps open source dependencies up to date with a combination of software and human experts. We take on the responsibility of getting you onto the latest versions of your dependencies, safely, even when there are breaking changes. Once you're up to date we'll keep you there as new versions are released. If you've ever considered outsourcing dependency upgrades to a consultant we offer a similar white glove service but with software that makes the process safer and more reliable. Right now we support Ruby and Javascript (with Python and Java coming soon).

Here's how the process works:

You grant access to your github or gitlab repo and let us know your priorities (e.g., large framework upgrade; no high+ CVEs)
Our software calculates an upgrade plan and estimates the complexity.
You get an ongoing series of small, incremental PRs

A little about us - we've been using Rails professionally since 2011 and React since the beta. We've seen all sorts of complicated tech debt and abandoned dependencies, upgraded apps as large as ~500K LOC, and can likely do this faster and more safely than your team.

Send a note to allison@infield.ai if you're interested!

No changelog for your open source project? We'll make one for you for Hacktoberfest!

Allison — Thu, 28 Sep 2023 20:05:16 +0000

Our company helps dev teams keep their open source dependencies up to date. One thing we do is plan and implement large complex upgrades (e.g. Rails) using software and people. Along the way we've used LLMs to categorize thousands of breaking changes in changelogs and Github releases. We've run into lots of projects that have a) no changelog at all, or b) undocumented changes. We're all human!

So for Hacktoberfest, we'd like to offer to all Ruby and Javascript maintainers: if you need help creating or augmenting a changelog, let us help! You can write to founders@infield.ai and we'll assign you a contributor from our team. We'll be working throughout the month of October. Cheers!

Automated open source dependency upgrade plans

Allison — Mon, 18 Sep 2023 18:45:05 +0000

I’ve been writing and using open source software for 20 years. My earliest memory of an open source upgrade is Ubuntu 5.04 in 2006. Back then upgrades were so unstable that I spent an evening backing my laptop up to a few dozen CDs in case something went wrong.

As OSS has evolved from a social movement for user freedom to dominating commercial software development, the relationship between open source maintainers and the companies that rely on them has grown more complicated. Open source software is constantly changing. Maintainers release new versions for bug fixes, security patches, and new features. Documentation may be lacking, or non-existent. Upgrading these dependencies is toilsome work.

I spent a year as a consultant helping teams with large, outdated apps. I upgraded their apps and set them up with ongoing maintenance programs to stay up to date going forward. I found myself doing the same things over and over. The right way to tackle a large framework upgrade is to break it apart into a series of small steps that are individually safe and accumulate in being upgraded. Breaking upgrades apart lets you parallelize them, reducing drag on the team. It’s also safer - making these changes individually ahead of time makes it much easier to identify a regression, decreases the blast radius of any missed breaking change, and lowers the chance that your final framework upgrade goes wrong.

But breaking major upgrades apart is hard planning work that includes risky unknowns - things like other blocking dependencies that might need to be upgraded first, breaking changes you weren’t aware of, and abandoned dependencies that may be incompatible with the newer version you’re targeting. It can also be completely overwhelming. Many teams that have fallen several major versions behind don’t know where to start. Though it takes an engineer to do this work, very little of it is actually coding.

We founded Infield with the mission to make open source software as easy to upgrade as Chrome. Today, we’re launching our upgrade management system and releasing our first automated upgrade product, which we call Upgrade Path.

Upgrade Path breaks complex upgrades into incremental, safe steps. We do this by connecting to your code, reading the changelog for every package you rely on, and running an optimization algorithm that solves for the best path. The result is a plan not just for your framework (e.g., Rails) but also for all your other dependencies that need to be upgraded along the way.

Upgrade Path takes care of all the planning and research work involved in upgrading. By unblocking engineers from the toilsome work of ordering and releasing upgrades, they are free to spend more time on feature development.

Upgrade Path also serves as a project management and coordination tool for teams across the organization - the Product, Security, and Platform Engineering teams that rely on each other to maintain and ship apps. Each team can view the upgrades that need to be done and those that have been completed, and they can make notes on how the work that Upgrade Path suggests applies to their specific codebase. In this way it serves as a record of the upgrades that have been done over time, showing progress toward making the enterprise more secure, agile, and ready to make use of the latest and greatest that open source has to offer.

Over time, we hope to automate away more and more of the toil involved in keeping open source dependencies on their latest versions. In this way we can make our own contribution to the world I first entered 20 years ago, and make open source a more sustainable and reliable part of every company’s codebase.

Once a Maintainer: Felienne Hermans

Allison — Fri, 15 Sep 2023 13:33:45 +0000

Welcome to Once a Maintainer, where each week we interview an open source maintainer and tell their story.

This week we’re talking to Felienne Hermans, professor of computer science at the Vrije Universiteit Amsterdam and creator of Hedy, a progressive programming language designed for kids 10 and up and for teachers to use in the classroom.

Once a Maintainer is written by the team at Infield, an app that helps engineering teams manage complex open source dependency upgrades. We spoke with Felienne from her home in the Netherlands.

How did you become a programmer?

Probably when I was about six, we had a computer in the house. Immediately I enjoyed playing with the computer, making the alphabet and stuff. Then after a while, probably when I was about eight or nine, I started programming in BASIC. There was no Internet, so I just had booklets from the library, copying and pasting basic codes around. I also had friends that liked programming, so we hung out together rotating houses. I loved building electronics as well. They had these kits from Velleman, which sounds very Dutch. You would get a little package and it would be like make your own dice and then it would have two buttons and six LED lights and then you press the button and then you randomly throw two dice with LED lights. I would save all my pocket money and to buy these kits. And I would also open my computer and add extra memory, stuff like that. I would say the regular things, but I realize now for many people this is not regular.

Do you remember your first computer?

Yes, it was a Commodore 64.

Did you learn programming in high school?

We did have a computing course, but it was very basic and boring and taught by a German teacher that didn't know programming. But my school also had a computer club, and the club wasn't run by the boring German teacher, it was run by the math teachers. So every Thursday during lunch break you could go to the math teacher’s office and all the kids that liked computers would be there, and we would exchange illegal games on CD-ROMs and things like that.

Then I went to do a major in computer science at a university engineering school, and then I did a PhD in software engineering, and then I became a university professor afterwards. So I would say in all but gender, I tick all the boxes.

What led you to decide to go down the academic path instead of the more corporate path?

I don't really know. It sort of happens I think. I mean, I was interested in doing a PhD, but I wasn't actively job hunting or anything. I was actually looking for a non-academic job, but when I was finishing up my undergrad degree I participated in this programming competition, and one of the judges would later become my PhD supervisor. I was chatting with him after the competition. We didn't win, we came in second, but he was like, “I'm really impressed with what you've been doing. Do you have a job?” I'm like, “Oh, I'm writing my thesis” and he said “Well, if you're interested, we have this job which is about building programming languages”, which I was really interested in. So then I applied and I got that position.

What was your first exposure to open source?

So even in university as an undergrad already there was quite some exposure to open source. Many of my professors were these traditional old gray white dudes and open source was very important to them. They would always bitch about how Microsoft is bad, and Windows is bad, and if you'd send them a Word document they would get really angry at you. This is ridiculous, proprietary software. Their world is very angry. And then I couldn't happen but notice that the stuff I liked using, like Windows and Mac OS, they were commercial, so they were bad, but I liked their software because it works really well, right? Then there’s Linux. I took a course on Linux. I had to use Linux. Nothing would ever work. And this is the early days of the Internet. So you want to download a movie and it took forever.

So to me, open source means being an old dude that's always angry at the inevitable state of things. Like, yes, Internet Explorer is bad, but everyone's using it right? So they didn't seem to me to be grounded in reality. And also their software was bad, it didn't work. So I really started believing that open source just didn’t work, you can’t build in open source. I didn't see myself in that world because my research has always been about making software that works for people. This is why I'm in computer science.

You also see that the leaders of open source back then were also like this. When I was a student, we had Richard Stallman lecture in the university. We go to the lecture and a student asks a question and he answers “That is not a question.” And then he just continues. So yeah, it never really got better after that.

So how did you decide to focus on programming education?

So when I finished my PhD, I was very much like, I don't know what I want. I don't know if I want to be in academia because it's far from having an actual impact in the world. I was sort of struggling. I started to teach in a high school one day a week. I thought I would do something valuable for my community because they really need teachers, and I can do programming. So I’m just gonna teach 12 year olds. And I thought, you know, how hard can it be? They're 12 year olds. I literally wrote a book on programming. And then it was quite hard. Actually, very hard. I was failing spectacularly.

These were all smart kids. They were in a good high school. So I cannot say it’s the kids. It's definitely me. I'm not teaching well. First I thought, you know, maybe I can explain it like this or I can explain it like that. But ultimately, I thought, a 12 year old and a professional language like Python, that's just not going to work. It's too hard. You have stuff like Scratch which is a visual language, a bit like Mindstorms with the blocks. But the 12 year olds, they think they're like adults. And you give them Scratch and they say it’s like a toy. This is for my 8 year old brother. I want to do an actual grown up language. But then you give them Python and it’s just not reasonable at that age, with their attention span and the level of logical thinking that they can do. So there was a gap there.

And that’s where I thought well, a 12 year old doesn't need everything of Python. They don't need methods, they don't need objects, they don't need functions, they don't need parameters or arguments. You can just print “Hello”. That's enough at the beginning. So I made this baby Python. That meets them where they are. And then we gradually amp up the complexity until it’s a subset of real Python. So secretly, we bring them up to Python.

Do you still use Scratch for a kid who has no exposure at all?

No, because mainly because I have one class and some of them have already done Scratch in elementary school. For the kids that have never done it, if you first give them Scratch while you give the other ones a grown up language, this is not neutral, right? These are the girls. These are the kids from lower socioeconomic families where there's not a computer, or they don't have parents bringing them to a computer club. So if you give some kids Scratch and some Hedy, my programming language, you are reinforcing that they don’t have the talent. So we think it’s better to start everyone at the same time.

How do you think your high school teaching has informed your university-level teaching? Has it?

That's a great question. We get more and more people that teach in universities that want to use Hedy, even though I built it for 12 year olds. Doing stuff in small steps and making sure your learners are not overwhelmed, it really works well with any age. I don't teach programming in the university now. I actually teach didactics, because I teach computer science teachers. But if I taught a programming course, I would definitely use this approach for a few weeks in university. I mean, MIT also uses Scratch initially and then they go to C, right? There's nothing wrong with using a tool that's also for younger learners if it's a context that makes sense.

What’s the general consensus around writing your own language and your own platform versus working within the curriculum as it was? How do people react to you personally when you say I dealt with this problem by writing my own language?

It depends who you ask, right? Teachers are generally very positive. It solves our problem. For other people, there is always criticism on the exact thing that we're doing. They want to know why don’t we do level two or three, and why do you support this or that. So on that level, people have opinions, but the idea to make a language that is specifically for this group that is too old for Scratch and too young for Python, I think there was a missing link and the idea of doing a new language isn’t so weird.

The end of Hedy is Python. Level 18 of Hedy is a subset of Python. So it’s not like I’m teaching them this fake thing that will never help them. It’s more like we want to go to the same place, you and me. I’m just helping you get there. And I think that’s powerful.

Why did you choose Python?

Well, one reason is that we already teach Python, so I could reuse the rest of my lesson plan. Python is used in most elementary introductory programming across levels. So it's a good place to go because there's a lot of resources there. I like Python. It’s also just the most common language used for teaching at this moment in time.

How do you encourage your university students to get involved in open source?

So I do still supervise undergrads in computer science, and for their final project many of them want to do something with Hedy. So they come to me for that. And it’s always this interesting exploration. Some of them want to do user research. They want to understand where students are struggling. Many of them also want to build something. And everything we have open on Github, we look at it and I say, “This could be your undergrad project.” Of course some are too big for an undergrad, or some too small. But we look at it together. I’ve had maybe a dozen undergrad university CS students contribute to Hedy, and that’s very nice.

Open source can be a good experience because it's very motivating. They build something that has actual users and they know that some of their peers that graduate with other professors, they build a prototype, but they just hand it into the professor. and that’s it. I had this one student, she built a new feature into the language and then we deployed it, and after five minutes already hundreds of programs had been created with her work. She's looking at the statistics, 100 projects and this is just five minutes, right? And ultimately, she did data analysis on 900,000 programs that used her new feature. That’s really, really cool.

How do we get more women into open source?

I don’t know, man. As I said, I had never contributed. I didn’t plan to build a big open source project. I just put it on Github because that’s sort of the right thing to do. I built this as a prototype for me and my students. And the real reason I put it on Github was it integrated really nicely with Heroku, so I could easily deploy. That was the actual reason. I didn’t plan it.

One thing that we do that I think is very good for any novice, we have a weekly Zoom meeting with contributors. So we say if you want to contribute, come to a meeting, just so that you see our faces. These are the people that will interact with you in a pull request and you see that we are people. We also see that you are a person. Sometimes people say, you know, I'm really new and then we know to treat them in a different way. I think that is something that works really well and I haven’t seen many open source projects do this. It takes a lot of my time. I think it’s worth it because this is how we attract people. It’s also how we keep people, because they come and say “I’m working on this, I tried something really hard and I’m stuck.” On Github you can’t do that, make half a request and then say you’re stuck.

We also have really strong guidelines on how you can contribute. I don’t want to say a code of conduct because people have very strong opinions about that, but it says if you see code that's written in a way that maybe you don't like, probably there's a reason. Like, assume this person had a reason. We just had a new contributor who says they’re super excited about helping. And then the first thing they do is start a discussion with 12 points - we should do this, we should do that. So I reached out to him on our Discord and said I’m sorry, that’s just not the way we speak to each other here. We shouldn’t do anything, and for most of your points there are very good reasons. I’m happy to tell you those reasons. I'm sure there are also other projects on the Internet where your style of communication is welcome and maybe even preferred. But that's not my project and I do think that has something to do with my gender.

It's just such an interesting question in open source. It is your project. It definitely started as your project. At what point does it kind of become the community’s project?

The nicest thing is when we’re in this meeting and someone else says “That is not the Hedy way.” If you enforce these norms, also if you radiate the norms, you can have the norms. And also live by the norms yourself, especially as a core contributor. You have to document them, but you also have to do them yourself.

Initially, I was really trying my best to keep different people in the projects happy, including my own husband. He was also submitting pull requests to my repository. I'm like, I love you, but I don't want this. He saw an error and wanted to fix it. But for me, even though I told you about this bug over dinner, I didn't ask you to fix it and it's just gonna make stuff more complicated. You can come to the meetings like a normal person.

What are some other projects you think are interesting? Or people doing interesting work?

We do use a lot of components. We use a tool called Weblate. Weblate is a tool that allows you to translate website content or other content, doesn't even have to be website content. It's also open source. You can log in with your GitHub, you can connect it, you go on a web interface and you just get a string that's in English and an empty box and you type your language there. People hit save and it creates a pull request and it will show up in their Github repository. We used to have our own hacks where people would upload translations and e-mail it to me and then I would have to put it in the system, it had all sorts of problems.

The other nice thing about Weblate is that some people are just really excited about translating. “Oh, I will make it my mission to translate everything into Vietnamese.” I think it's not always like perfect quality, but it's something to get you going - now we have a Vietnamese version, then people start using it in Vietnam, and then someone with a little bit more content knowledge, they updated it. Without this tool maybe we would have maybe ten languages, definitely not the 47 we have now. And it’s not a lot of effort to set up. There's no reason that whatever you're doing isn't translated into multiple languages. It supports JSON and YAML and Markdown. If your website isn't properly localized, it's because you didn't do five minutes of work with that. It's a very cool project. I love that.